INET Conferences

Evaluation Frameworks for ICT-Based Distance Learning

Sam LANFRANCO <lanfran@bellanet.org>
Bellanet Secretariat and York University
Canada

Abstract

This paper views the ICT-based electronic venue as a learning workspace and social process arena. It focuses on how to (1) conceptualize, (2) plan, and (3) evaluate learning in the presence of the ICT-enabled electronic venue. It broadens the scope of the inquiry well beyond the narrow issue of how to evaluate technology-based delivery tools. The paper presents a meta-framework for the design and evaluation of learning strategies. This meta-framework operates at one level above any number of specific design and evaluation tasks. Its role is to help create appropriate design and evaluation templates, given the learning process and the learning situation at hand.

Contents

• Introduction
• The learning venue: getting the context right
• The electronic venue and real learning
• The learning entity's virtual persona
• Planning and evaluating ICT-based learning:
• The planning and evaluation framework
• Conclusion
• Notes

Introduction

The rapid development and deployment of information and communication technology (ICT) in the later two decades of the 20th century has produced an electronic venue (a.k.a the internet) which serves both as a virtual workspace and a social process arena. Nowhere will this workspace and social process area be more important in the future than for individual, organizational and community learning.

The launch of Sputnik four decades ago brought the quest for technical knowledge to the fore. Persistent global, social and environmental problems have brought organizational and social issues to the fore. Technical, social and organizational knowledge has become increasingly more important than natural resources.(1)

The start of the third millennium, as with each new century, is a time for taking stock and speculating about the future. A century ago we did poorly in predicting what the future had in store. There was no expectation of two great and many small wars, a myriad of nationalist revolutions, the advances and failures in medicine, the great depression, the atomic bomb and AIDS or the advent of the ICT revolution.

The end of the 19th century was marked by optimism, faith in technology and a belief that nature had been discovered and would be conquered. The start of the 21st century is marked by anxiety and suspicion about technology and a nature only partially subject to discovery, even less subject to conquest.

There is a great hunger and a greater need to know more. To know more about nature, society, self and others. There is a hunger for knowledge and wisdom, for the good of self, society and the planet. Knowledge and learning are seen as pivotal in the restoration of faith and optimism in our vision of the future. Learning has become embedded as a cardinal characteristic of every social process, a goal for individual action, and part of the fabric of institutions large and small. Knowledge is being touted as the resource of the future.

We are entering an "Information Age" where, in a reductionist fashion, information is touted as the building block of knowledge. This view is reflected in the language used to describe the relationship between the electronic venue and information. The venue is touted as an "information highway." The metaphors and images see it as an "information warehouse" -- a place to store or get information. It is an "information pipeline," spewing forth a cornucopia of information, or an "information vacuum" sucking information from the weak and vulnerable.

We are confronted with opportunities, challenges and risks concerning how we nurture, diffuse, acquire and use knowledge.

In this paper we will view the ICT-based electronic venue as a learning workspace and social process arena, as well as a place where learning, wisdom and power combine to sustain some things and change others. We focus on what this electronic venue means for individual, organizational and community learning.

To expand on the title of the paper, what follows is about how to (a) conceptualize, (b) plan, and (c) evaluate learning in the presence of the ICT-enabled electronic venue. One will quickly note that we have broadened the scope of the inquiry and gone well beyond the narrow issue of how to evaluate this or that technology based delivery tool, for this or that learner.

This is for two reasons. First, there are thousands of efforts directed at designing or testing specific tools for the delivery of education and training in the electronic venue.(2) Second, we think of those efforts as akin to trying to design a better, safer and more efficient car. Our concerns are more akin to the design of the transportation system and its role in the structure and function of the city. One does not get to this second stage by aggregating automobiles, one needs a different perspective and different vantage point. The same holds for learning and the electronic venue.

The learning venue: getting the context right

This paper presents a meta-framework for the design and evaluation of learning strategies in the presence of the electronic venue. By "meta-framework" we mean that it operates at one level above any number of specific design and evaluation tasks. Its role is to help create appropriate design and evaluation templates, given the learning process and the learning situation being addressed.

It is not meant to be used as "the" framework in the sense that it is applied as a rigid template across situations. Its use is as a common scaffolding for the design of appropriate policy, delivery and evaluation strategies. It is "scalable" in the sense of "fractal nesting." By this we mean its essential properties are the same across the design of policy, delivery and evaluation. They are the same at the level of the individual lesson (in a course setting) and the curriculum, and for the individual learner as well as the "learning organization" and the "learning community." This will become clear below.

The meta-framework is meant to apply across three apparently different but in fact quite similar learning processes. The first is the individual learner, the second is the learning organization and the third is the learning group (or community). This makes it relevant for both formal and information education and training. It is relevant for learning organizations and organizational change. It is relevant for community and social learning, as well as learning within intentional groups (workgroups, support groups). Of course, its relevance is with regard to how each of these activities takes place with a significant presence in the electronic venue.

The approach takes learning as its vantage point, rather than education and training. It treats education and training as embodying "intent." It treats learning as the embodiment of "outcome." In simple terms, the task is to achieve outcomes, and evaluate them against intent. This focus on learning casts considerable light on the design of education and training while guarding against treating education and training as the primary outputs.

There are several reasons for this. First, for many of the questions around education and training, they are inputs, not outputs. Second, it guards against viewing ICT solely as a delivery tool. It highlights the fact that the electronic venue is both a place in which learning takes place and (as a new space) a place to learn about. Lastly, the pace of technological change and the degree of interdependence between social and natural systems is making learning an ongoing part of individual, organizational and community life. Specific knowledge skills will have a short "shelf-life" (or half-life) and will only be appropriate "in context." Lastly, learning has to be understood as a process and not just an outcome.

These factors necessitate a strategy toward ongoing (lifelong) individual, organizational and community learning. They also caution against using industrial models as a starting point, treating ICT as a tool and education as a product. This is a trap when it comes to getting the context right for assessing the role of ICT in learning policy, delivery and evaluation, whether for skills training or for education. This trap feeds the almost interminable and mis-specified debate between those defending face-to-face instruction and those promoting computer-based training.(3)

Issues such as the risk of technology displacing teachers, the ownership of electronic content, and promise or threat from virtual universities are all real. However, approaching them from a "technology-as-tool" perspective handicaps the analysis and serves as an inadequate basis for policy or the design of curriculum and evaluation strategies. While these issues are not dealt with here, this approach is intended to be useful for those debates as well.

The electronic venue and real learning

Our first task has been to shift away from a focus on ICT as delivery tools and toward the notion of an ICT-enhanced (electronic) learning space.(4) A second task is to recover the electronic learning space as part of an overall learning venue, and not to view it primarily as a stand alone or competitive alternative to other (non-electronic) learning venues or tools.

We start by clarifying the term distance learning itself. The term has its roots the 19th century and our traditional notion of correspondence education. The learner was physically distant from the provider. The tools of distance education/learning were the book, the lesson and the postal service.(5)

The educational provider was "here" and the learner was "there" across geographic space. We will identify this process as "asynchronous across space." The process was also asynchronous across time, with books mailed today and read tomorrow, tests done here and graded there.

Until modern times, formal distance education was mainly asynchronous across space and time. Instructional materials were sent, completed, and returned for assessment. In this century, the phone, radio and television have introduced synchronous time delivery (e.g., live TV lectures) which may or may not be synchronous in space.(6) The driving force behind such practices was usually cost, either the cost (and time) for learners to attend lectures or the low marginal cost of mounting "extra sessions" via video monitor. Seldom was the practice justified on pedagogical grounds. When given a choice, many learners prefer the time and place flexibility of working with pre-recorded tapes, in groups or alone, rather than attending scheduled live video "classes" as groups, or individually in a learning center.(7)

For the audio/video synchronous option to be a preferred, it must bring added value and not just relative to the obstacles facing live presentation. It must be  preferred to the asynchronous audio/video option as well. This has become a contested area, as multi-point electronic video is becoming more widely available. This position has not been subjected to much critical analysis. In technology "new" need not equate to "better." "Appropriate" equates with "better," and appropriate is measured with respect to both learning objectives and learning alternatives.

There is a downside to touting each ICT-enabled tool or venue in its own right. While individual uses can be ranked in terms of efficiency, effectiveness and suitability as competitors, they also exist as complementary tools and venues depending upon objectives, opportunities and constraints.

It has usually been necessity that drove distance learning to depend heavily on one or the other vehicle. This constrained (or enabled) it to be mainly asynchronous across time and space, less interactive, and have more of an individual learner focus. When possible it has blended delivery components and learning venues to include local study groups, circuit teachers, resident tutors apprenticeships and mentoring and the like.

The history of distance education tells us that it is less than fruitful to focus on the new electronic venue as a bag of virtual delivery tools which compete with traditional literal tools, where delivery and learning decisions are made across a tools continuum (Fig 1, upper part).

It is more fruitful to think of the ICT-enabled electronic venue as a virtual workspace and social process arena which brings its own opportunities, challenges, promise (and risks) to education, training and learning.

The lower part of Fig. 1 depicts a literal and virtual venue in which learning operates in varying degrees of synchronous and asynchronous fashion both across literal time and space and across the electronic venue. Literal venue choices are located along the vertical axis with respect to time and space (place). Virtual venue choices are located along the horizontal axis with regard to time and space (place).

To illustrate this, consider the following. Face-to-face lectures are synchronous in time and place. Live video is place -- asynchronous but time-synchronous. E-mail is asynchronous with regard to time and space. A chat group or video-conference is synchronous. An e-mail based list, or conference, can reflect learning in an individual or a group setting. What is important here is that the literal and the virtual together represent the real venue for distance learning across time and place.

There is much confusion around this important point. Much of the discussion over the merits of using the electronic venue, or this or that electronic tool, has been couched in a debate about what is "real" and what is "unreal." Much of the discussion wrongly equates "real" with the literal and "unreal" with the virtual. The position that "face-to-face" instruction is real and electronic instruction is not contains a conceptual error independent of the merits of one over the other as a venue or a delivery tool.

The proper distinction is between the literal as a domain bounded by the material world, and the virtual as a domain which transcends the material world, much as religion and the psyche transcend the material world. In this case it has an impact on, and is being felt in, social process. That it is not literal makes it no less real as a dimension of the individual, group and community reality. Learning here is a social process, not a physical one. It is real in the same sense in which we have accepted religion and the psyche as real. Together the literal and the virtual constitute the real.

If it were easy to coin terms and change language, it would be wise to drop the term distance from the discussion. We would use instead a notion of distributed learning. Learning distributed across time, space and venues. Learning is distributed across delivery vehicles and participated in by individuals, groups, organizations and communities.

The rest of this paper will treat the electronic venue as a virtual workspace and social process arena within the context of distributed learning. Distance learning and distributed learning are used interchangeably with education distributed literally and virtually across time space and venues.(8)

The learning entity's virtual persona

We are told that the future requires us to become lifetime learners, as individuals and in groups. Organizations are told to become learning organizations. The rhetoric of the "information age" is about how we -- as individuals, groups and communities -- combine knowledge with wisdom, not about how much raw information we can amass, deliver or process.

At the core of this is the notion of a learning entity as an active agent in its own learning, and -- one might add -- the learning of others. The role of the entity in the learning of others -- in the presence of the electronic venue -- is not discussed in detail here but is being developed elsewhere.(9)

We now present a conceptual framework for the learning entity. Just as the learning venue has literal and virtual components, the "persona" of any individual, organization or community learning entity has a literal component and a virtual component. Learning and behavior occur across a reality which encompasses both the literal and the virtual.

Think of the entity's electronic persona in terms of its own virtual workspace. This includes the entity's electronic infrastructure (intranet, extranet, internet) and its organizational and behavioral presence in those venues.

In Fig 2 we depict the entity as carrying out four tasks across three virtual domains. These features apply to a process (learning) as well as the entity as a learner or an education and training provider.

E-mail, GroupWork, and Digital Objects denote the venues. Administration, research/learning, products/services, and communications denote tasks. [Note: The labels used here are a shorthand for longer and more complete descriptions.]

The e-mail domain facilitates the transmission of digital information and is the command center of the entity's electronic presence. It is linked to its own component parts and any bigger entity of which it is a part, and beyond to the rest of the world, by an ICT-based communications corridor. The e-mail domain, as the entity's electronic presence command center, is where the entity orchestrates its work and presence in the electronic venue.

The Groupwork domain is the venue for (mainly) asynchronous collaboration across time and space. This space is "intranet-like" (with itself and within the larger entity in which it resides). It is also "internet-like" as it links to entities beyond itself. This groupwork domain reflects the suitability of the electronic venue as a place for collaborative work and is of extreme importance for evaluating collaborative learning and work.

The Digital Objects domain supports the creation, use, and distribution of, as well as access to, stored digital objects. This is the most commonly understood feature of the electronic venue, as a place to store or get information.

In simple terms the entity administers itself (north-west quadrant), carries out its product/service mission (south-west quadrant), stays knowledgeable through learning and research (north-east quadrant) and communicates with others in the pursuit of these objectives (south-east quadrant).

In accordance with the principle behind fractal nesting in chaos theory, one can dis-aggregate or aggregate entities by function, mission, or whatever. At each level of aggregation or dis-aggregation the structure of the electronic persona remains the same. At each level an entity, part of an entity or collection of entities will have rights and privileges of access to the digital domains (objects, workspace, central control) depending on entitlements.

In a learning organization or community, entities will have reciprocal obligations to provide access to digital objects (information), and extend participation in groupwork venues.

We now have an conceptual image of the entity's electronic persona. It can be dis-aggregated by quadrant and electronic domain. It can be dis-aggregated (for example) by produce or service. Lastly, it can be aggregated to consider the entity's role as a learning organization or component part within a larger community defined by economic, political, environment or other criteria and concerns.

Planning and evaluating ICT-based learning:

This approach is useful in planning and evaluating distance learning, and for situating it into strategies of lifelong learning, the learning organization, and the knowledgeable community. It underscores the need for the learner and the education and training provider to understand the electronic venue as a workspace and social process arena in its own right, and not just as a delivery vehicle.

For effective learning, it is equally important for society, the community, the organization and the entity to think though their obligation to inform (02i) so that learning processes are properly enabled in terms of access to information, and electronic workspaces.(10)

The task quadrants allow us to focus on the link between learning and "deliverables" in terms of an entity's mission, vision, strategy and logistics. While seeking a "organic" (holistic, linked) learning process, this ability to break out a entity's learning process in the electronic venue is doubly useful. It helps us conceptualize ICT-based learning strategies.

For brevity here we only address the task quadrants. Learning can be directed at: (1) enhancing own-task efficiency and effectiveness (within a quadrant); (2) other-task efficiency and effectiveness (across quadrants); and (3) overall entity efficiency and effectiveness. One can use the electronic venue to enhance administration or use enhanced administration to support product/service delivery or research and learning.

One need not stop at aggregation across quadrants. One can aggregate across entities. This can be vertical aggregation across a hierarchy of entities or horizontal aggregation across a community of entities. Such links and proximity can be literal or virtual.

The planning and evaluation framework

Since the framework applies at the meta level it is hard to explain the framework in more detail without resorting to actual applications. It is compatible with a wide range of evaluation methodologies. Its role is to identify what to evaluate, not how to evaluate it.

Preliminary test applications have been conducted on ICT-based projects funded by Canada's International Development Research Center (IDRC).(11) Several researchers are currently applying parts of the evaluation framework to African telecenters and organizational change in none-profit development organizations.

The meta-framework has sufficient detail to construct a "user-friendly" template for evaluating ICT-based distance and distributed learning. At first glance it may appear more burdensome than it actually is.

The template has three electronic domains in the virtual workspace, four task components across three quadrants, plus two "corridor" activities (access to and from remote sites) in the fourth quadrant.(12) This produces a template for the dis-aggregate entity with sixty cells across four tables. Including the aggregate entity adds fifteen cells, and locating the entity within a horizontal or vertical community raises the total to ninety cells. Each cell represents a descriptive data point and the ninety cells provide an exhaustive list of starting points from which to query the learning process in an ICT-based venue.

This produces a series of tables which can be found in this authors' paper at http://dkglobal.org/meta4.htm, for the sake of brevity and to avoid tables in the mark up of this paper as per the request of the INET'98 Program Committee. [Note: The tables will be available a the INET'98 presentation.]

Each cell can be the source of more than one specific question, data point or evaluation concern. As well, different policy and evaluation concerns can prompt revisions in particular cells. However, the actual task is far less daunting than the table size would suggest. Much of what is required at the level of a core learning activity will have already been answered in the component level analysis.

To evaluate learning and capture lessons learned about the electronic learning venue, priority should be given to analysis at the quadrant and domain level, rather than just at the level of the entity. Traditional evaluation techniques usually start at the level of the core entity and focus on inputs and outputs and, in so doing, fail to understand the role and importance of the electronic venue in the process.

Any evaluation of the learning process at the level of a network or a community should follow, in sequence, evaluations at the component and entity levels. The focus should be on (a) how the learning process collaborates with others in the virtual workspace, (b) how the process uses the knowledge, wisdom and expertise of the wider network or community, and (c) how it uses the electronic venue to share its knowledge, wisdom and expertise with the network or wider community.

In practical terms it is not necessary to explore all ninety cells in an evaluation question. Keeping in mind that this is a meta-level framework, all evaluations start with:

1. the "object" of the evaluation (e.g., a learning process or learning entity),
2. an evaluation methodology, and
3. a set of key concerns (e.g., sustainability, capacity building, performance, etc.)

From the six evaluation tables containing the ninety cells, one can then identify those cells which are relevant to the evaluation at hand. Some will be relevant for data collection. Others will be relevant by suggesting things that the project or entity should have considered but didn't.

Conclusion

This paper presents the electronic venue as a learning workspace and social process arena and focuses on how to (a) conceptualize, (b) plan, and (c) evaluate learning in the presence of this electronic venue. It broadens the inquiry well beyond a narrow focus on technology based delivery tools.

The paper presents a meta-framework for the design and evaluation of learning strategies. This meta-framework operates at one level above any number of specific design and evaluation tasks. Its role is to help create appropriate design and evaluation templates, given the learning process and the learning situation at hand. The resulting approach is compatible with a wide range of planning and evaluation issues and methods

Notes

1. Concerns over a sustainable social and natural environment have broadened the notion of "appropriate technology" to include knowledgeable (and wise) individual and social behavior, and not just appropriate mechanical (tool) technologies for exploiting natural resources.

2. See the Canadian Network for the Evaluation of Education and Training Technologies (EvNet) at http://socserv2.mcmaster.ca/srnet/evnet.htm. [This author is a member of that Canadian funded research network of excellence.]

3. See the F2F discussion in the archives of gkd97 at http://www.globalknowledge.org/

4. Since the electronic venue is also a social process arena, one of the objectives of a learning strategy is to arm (educate, train) individuals as citizens and workers to operate within that arena. They must learn to treat it, along with literal space, as an integral part of their learning and workspaces. We do not focus on this aspect of the learning space in this paper. See: http://dkglobal.org/democracy.

5. Outside formal certification, distance learning can use a variety of settings and tools and need not involve vehicles for the feedback and assessment.

6. By "synchronous in space," we mean transmitting a live lecture by video (or audio) to a remote group of learners, whether this be across campus or across the globe.

7. This has been the experience of the U.S. based National Technical University which broadcasts live and pre-recorded lectures to learner centers in high tech companies which have paid for their employees to take the courses. While the learner is frequently down the hall from the learner center, nine out of ten transmissions are taped for later viewing.

8. We deliberately avoid terms like "virtual reality" whose meaning is ambiguous enough to include "non-real" through to escapist activities (games, browsing, chat groups, etc.) which use the electronic venue. "Virtual university" is another ambiguous term, likely to suffer the same fate as discussants try to define it in terms of technologies rather than learning strategies and objectives.

9. See S. Lanfranco, "The Notion of the Obligation to Inform (02i)" at http://dkglobal.org/02i.htm

10. O2i differs from access to information and the rights of privacy. Access is about a legal right to information about the entity or some governmental process. Privacy rights entitle an entity, or oblige another entity (e.g., gov't) to withhold information.

11. See Michael Graham, ". . ." at http://www.bellanet.org/ldia

12. The checklist tables are at the end of the document at: http://dkglobal.org/meta4