comparing approaches: telecentre evaluation experiences in asia and latin america

Intl. Inform. & Libr. Rev. (2002), 34, 57^78doi:10.1006/iilr.2002.0184Available online at http://www.idealibrary.com on

Comparing Approaches:Telecentre Evaluation Experiences inAsia and Latin AmericaRICARDO GOŁ MEZ* & KATHERINE REILLYy

ABSTRACT

IDRC has telecentre evaluation initiatives in Africa, Asiaand Latin America. Each of these is in the process of estab-lishing evaluation frameworks and each is adopting very dif-ferent approaches. The current paper seeks to build on theoutputs of IDRC’s Far Hills Workshop onTelecentre Evalua-tion (September 1999) to explore the experiences in two ofthe three regions. These explorations will in turn contributeto IDRC’s current initiative to develop a framework for ICTsevaluation.The Far Hills Workshop, and the resulting report entitled

Telecentre Evaluation: A Global Perspective, had as objectives toexplore challenges and opportunities for telecentre evalua-tion; understand and compare emerging evaluation frame-works and methodologies; assess needs and resourcesavailable; identify salient issues a¡ecting telecentre perfor-mance; and provide an opportunity to exchange experiencesand lessons across regions.

r 2002 Elsevier Science Ltd. All rights reserved

UNDERSTANDING TELECENTRES

Telecentres are a new trend in international development. Loosely de-¢ned, telecentres are physical spaces that provide public access to infor-mation and communication technologies, notably the Internet, foreducational, personal, social, and economic development.1 When it

*International Development Research Centre (IDRC), Ottawa, Ontario, Canada.wOttawa, Ontario, Canada.1Go¤ mez, R. & Patrick Hunt (1999) Guiding Principles of Telecentre Evaluation. Paper presented at

the Telecentre Evaluation: A Global Perspective, Far Hills Inn, Quebec, Canada.

1057^2317/02/010097 + 22 $35.00/0 r 2002 Elsevier Science Ltd. All rights reserved

58 R. GOŁ MEZ AND K. REILLY

comes to evaluating what kind of bene¢t or development telecentresbring to a community, we are only beginning to scrape the surface. Re-cently IDRC’s PAN and Acacia Networking initiatives organised an in-ternational meeting in Quebec, Canada, to discuss telecentre evaluationmethodologies. At the meeting a list of criteria for telecentre evaluationwere established.2 Based on this work, we suggest the following guide-lines to assess ICT evaluation that is useful, ¢nancially responsible,builds local capacity, and enables shared learning:

GUIDING PRINCIPLES FOR SOUND TELECENTRE EVALUATION

Building from discussions at Global Telecentre Evaluation Meeting, FarHills Inn, September 1999. Guiding principles to help evaluation beUSEFUL, FINANCIALLY RESPONSIBLE, BUILDS LOCAL CAPACITYand ENABLES SHARED LEARNING:3

1) Participatory. All relevant stakeholders, including users andnon-users, are involved in the evaluation process.

2) Socially Inclusive. Evaluations explicitly address, include andprovide di¡erentiated information about the experiences of sub-groups in a society (gender, age, culture, religion, etc.).

3) Locally Grounded. Evaluations are context sensitive. (By wayof explanation, this may imply relying on available expertise ortaking into consideration local practices, world views andpriorities.)

4) Public and Transparent. Evaluation results are publicized inways appropriate to all relevant stakeholders.Evaluation processes are transparent.

5) Methodologically Appropriate. Choices of methods and toolsare appropriate for the context and use, balancing replicabilitywith usefulness and methodological robustness with practicality.

6) Sustainability Enhancing. Evaluations contribute to makingtelecentres more viable, to enhancing services and to makingthem more relevant.

7) Capacity Building. Lessons learned F both failures andsuccesses F are documented during evaluations and are used toempower telecentre practitioners and users. Training is carriedout where a need is identi¢ed.

2 Ibid.3 Gomez, Ricardo & Katherine Reilly. Comparing Approaches: Telecentre Evaluation Experiences in

Asia, Africa and Latin America. (mimeograph) July 2000.

TELECENTRE EVALUATION EXPERIENCES 59

8) Re£ective of Shared Visions. Evaluations are based on acommon understanding of the telecentre mission, the evaluationprocess, the goal of the evaluation, and how results will be used.

9) Strategically Oriented. The evaluation strategy is incorpora-ted into the project design and re£ects project objectives. It isfocused on clear questions that are important to stakeholders.

10) Gender Sensitive. The evaluation strategy, process, methodo-logy and tools are sensitive to the particular realities and needsof women. Women are consulted in the development andrealization of evaluative processes.

At the Far Hills meeting, several participants noted a distinction be-tween two classes of evaluation criteria. The ¢rst is input measures,which Harris explains, ‘‘are not immediately associated with the bene¢tsthat the user community enjoy, but they are regarded as essentialpreconditions for those bene¢ts to emerge. Consequently, they can beused by operators as guides for establishing telecentres and forconstructing the services that they will provide’’.4 Output measures onthe other hand, ‘‘relate to the achievements of the Telecentre within thecommunity it serves. Output measures are distinguished from inputmeasures in that they relate to the additional bene¢ts that the commu-nity enjoys as a result of utilising the services of the centre’’.5 Hudsonmakes a distinction between formative and summative functions of eva-luation, the former being ‘‘feedback on the project or activity: how wellis it working, what changes or improvements should be made, what waslearned that could be applied in other similar projects, etc.’’ and the laterbeing, ‘‘Did the project achieve its goals? What was learned about howICTs can contribute to social and economic development?’’.6 Formativeevaluation focuses more on process and feedback, while summative eva-luation seeks to discover if a di¡erence was made by a project.These distinctions can be somewhat arti¢cial, however. For example,

how well a project is working is inextricably linked to whether itachieves its goals. A broader basis for understanding is required to com-pare and examine the evaluation work being done by IDRC. This is inpart the intent of the Guiding Principles. But on another dimension, it is

4Harris, R. (1999) Evaluating Telecentres within national policies for ICTs in Developing coun-tries. In Go¤ mez & Hunt (Eds). Telecentre Evaluation: A Global Perspective (Report of an InternationalMeeting onTelecentre Evaluation). Far Hills Inn, Quebec, Canada.

5 Ibid.6Hudson, H. (1999) Designing research for telecentre evaluation. In Go¤ mez & Hunt (Eds).Tele-

centre Evaluation: A Global Perspective (Report of an International Meeting on Telecentre Evaluation). FarHills Inn, Quebec, Canada.


important to understand what the purpose of any evaluation is, and foran organisation like IDRC, which works on various levels, this is key. Inthe case of an international donor agency, evaluations can serve di¡er-ent purposes.Within IDRC, telecentre projects are often carried out in association

with a recipient (often a regional body), which in turn works with var-ious telecentre organizations. Many projects follow a three-tiered cas-cading structure (IDRC ^ recipient ^ telecentres), while in other casesIDRC program o⁄cers interact directly with telecentres. But, whileIDRC’s research is often concerned with broader policy issues and globalimpacts F what could be termed the ‘‘big questions’’ F the immediateinformation needs of telecentre users and operators at the local level aremore specialized.7 In an environment of limited resources, the potentialexists that choices will have to be made between research that answersthe big questions and research that suits local information needs. Or, tobe less absolute, it may be di⁄cult to fully reconcile the need for datathat is relevant for regional studies with the need for data that is rele-vant locally. Thus, there is a possibility that studies executed by IDRCor a regional organization with the assistance F and even input F oflocal groups, will not fully meet the needs of those local groups. Alter-natively, it is possible that specialized research at the local level will notcontribute fully or meaningfully to larger research questions.Research activities which seek to answer the ‘‘big questions’’ are im-

portant for justifying international contributions to telecentre activities,determining regional caveats, understanding both positive and negativeoutcomes, discovering linkages with other development objectives, unco-vering successful practices, and building capacity, among other things.The research will determine whether and how telecentre activities arecarried out in the future, and, as such, it is important that these re-search activities have input from local groups. Evaluation cannot beviewed by primary and secondary recipients as simply a condition offunding. It should be viewed as a means to in£uence the activities ofdevelopment agencies. Evaluations are an opportunity to determine that‘‘This is working for us, but that is not.’’On the other hand, a goal of IDRC’s work is also to enhance the sus-

tainability of individual telecentres. Telecentres are a relatively new phe-nomenon and there exists little guidance for the individual telecentreoperator on how to successfully carry out a project. Thus, evaluation isparticularly important at the local level so that the telecentre operator

7 This is not to say that local organizations do not get involved in the larger questions. In fact,much of IDRC’s work in the are of connectivity has focused on helping local groups contribute toregional or international exchanges.

TABLE I

Comparing approaches to telecentre evaluation

The Evaluation Should Be: Asia: PANTLEG Latin America: TELELAC

1) Participatory Great potential if local story gatherers are empowered. Guiding philosophy isparticipation.

2) Socially Inclusive Individual voices can represent a variety of groups. To be determined.3) Locally Grounded Stories re£ect local experiences. Adapted to the context of the

telecentre user.4) Public and Transparent Publication processes and level of transparency is unclear. Highly public and transparent

processes and methods ofdissemination.

5) Methodologically Appropriate Practical. Not particularly robust. Focus on successes. Flexibility and adaptability area challenge.

6) Sustainability Enhancing Opportunity for learning and exchanges. Goal is to enhance the sustain-ability of telecentres.

7) Capacity Building Emphasis on celebrating success. Capacity-building and training.8) Re£ective of Shared Visions Unclear. Strong collective process.9) Strategically Oriented Open-ended. Methodology must be £exible

to accommodate strategic goals.10) Gender Sensitive Includes women’s voices. Tools to be determined.

TELECENTREEVALUATIONEXPERIENCES

61


can better understand local needs and opportunities, so that problemareas can be identi¢ed and addressed, and so that successful or unsuc-cessful practices can be recorded. In order for telecentre operators torealise these local evaluations, they will require training and support.It is not clear that this area is being systematically addressed F eitherby IDRC or by other institutions. The lack of clarity may emerge fromthe fact that local evaluations are folded into larger comparative or re-gional evaluations. Though information from such studies may be usefulto local operators, it is not su⁄cient. Information needs at the local levelare likely more frequent, detailed and specialized than larger studies canprovide. Thus, it may be necessary to develop appropriate guidelinesand training for evaluation at the local telecentre level. This may be dif-¢cult given the variety of telecentre experiences and communitiesserved. However, it is possible that much knowledge already exists inthe form of business auditing techniques and management reviews.Furthermore, telecentres can provide each other with a great deal of in-formation and support if the appropriate channels are put in place forcommunication.When various levels of partnership emerge, evaluation becomes more

complicated. IDRC is working on di¡erent approaches to telecentre eva-luation in Latin America, Asia and Africa. The activities in each regionare vastly di¡erent and each approach has its strengths and weaknesses.Throughout this paper, the approaches used in Asia and Latin Americawill be described and presented in light of the guiding principles to tele-centre evaluation. The preceeding chart synthesises the ¢ndings of thisanalysis and provides a means to compare approaches. The discussionsalso re£ect the balance that each approach has struck between the indi-vidual needs of telecentres and the need to address larger research ques-tions. The conclusion draws from these results to provide summaryresults, recommendations and future directions.

EXPERIENCES IN ASIA

PANTLEG F An emerging evaluation approachEvaluation of IDRC’s Asian telecentre projects got its impetus duringregional working sessions at the Far Hills workshop. At this time thePAN-Asia Telecentre Learning & Evaluation Group (PANTLEG) wasestablished consisting of two IDRC sta¡, and ¢ve Asian partner organi-zations. The group decided to explore an innovative approach to evalua-tion. In order to measure the impacts of the telecentre initiatives, thegroup investigated the idea of collecting stories from telecentre operators


and users. According to the PANTLEG group:

Stories, whilst anecdotal, o¡er a rich picture of the impact of ICT interventions inlocal, complex and dynamic social settings. They are accessible and veri¢able dur-ing short visits and they acknowledge the often indirect in£uence that developmentinterventions have on the behaviour of their bene¢ciaries. Moreover, stories as eva-luation concede that the bene¢ts of telecentre activities are often detectable onlyafter they have been installed, contrary to traditional approaches to informationsystems, in which expected bene¢ts are usually speci¢ed before the technology isinstalled.8

Because stories must be told to an evaluator, and because a major criter-ion for evaluation as determined by the Far Hills workshop is participa-tion, the group also explored the idea of having the telecentres evaluateeach other. Thus, neighbouring telecentres would exchange, compareand document their experiences, providing an opportunity for both pri-mary learning in the form of telecentre capacity building, and for sec-ondary learning in the form of evaluation capacity building. Theapproach has been dubbed a ‘‘learning evaluation’’.A ¢rst exploration of this methodology has been carried out by

PANTLEG. In November 1999, the group realized a mission to studythe projects of two IDRC partner organizations in India. Two IDRCrepresentatives accompanied Roger Harris of Malaysia, Merlita Openaof the Philippines and Narangerel Dander of Mongolia to visit V. Balajiof the MS Swaminathan Research Centre (MSSRF) and Loyola Josephof the Foundation of Occupational Development (FOOD), both inMadras. This entire group of project co-ordinators visited six village tel-ecentres, one of which acts as an information hub for three others admi-nistered by MSSRF, and one centre of operations hooked to onetelecentre administered by FOOD. The group had discussions with tele-centre users and operators to, ‘‘discover a range of typical stories thatdescribed instances of telecentre use which were perceived by the usersto have led to successful outcomes,’’ and to, ‘‘discover any success storiesconcerning operational problems, and the means that were used to over-come them’’.9 Story collection typically started with village leaders(mostly male), then users (¢rst male and then female users) and ¢nallyoperators (in one case the operators were part of a woman’s co-opera-tive). In all, the group uncovered 24 success stories, each of which, ‘‘owesits outcome to the sensitive and timely delivery of useful informationthat contributed to local knowledge which facilitated something desir-able for the recipient’’.10

8 PANTLEG. (1999) ‘‘Success Stories of Rural ICTs in A Developing Country.’’ IDRC Website,URL: http://www.idrc.ca/pan/telsuccstories_e.htm, (viewed 2 May 2000).

9 PANTLEG (1999) Op. cit.10 Ibid.


The evaluation group summarized their thoughts for the two projectorganizers of FOOD and MSSRF. They then reviewed the impact as-sessment section of an MSSRF project proposal. MSSRF had proposedregular surveys of key indicators throughout their projects in orderto measure improvements attributable to the telecentres. An importantlesson of the exercise for the PANTLEG group was that evaluationof a telecentre project is not straightforward. They suggested that theproposal be changed so that baseline surveys were followed by storygathering and the collection of fresh empirical data, a process moreakin to action research. This approach recognizes that the implementa-tion of the telecentre projects can shift the focus of the research questionand generate new avenues for inquiry. Furthermore, the mix ofapproaches would provide a much richer picture than could comefrom a uniquely survey approach. Another recommendation was thatMSSRF focus less on speci¢c project outcomes which may behard to quantify and measure. Instead, the evaluation group suggestedthat, ‘‘it might be bene¢cial to acknowledge . . . uncertainty of out-comes at the outset and to demonstrate the capability of the projectto tease out unpredictable bene¢ts through the skilful and sensitiveapplication of appropriate methods such as Action Research, Parti-cipatory Rural Appraisal, Rapid Rural Appraisal and OutcomeMapping’’.11

The mission to Asia gathered several stories indicating the positiveimpact that telecentres can have in underdeveloped communities. Forexample, in one instance, a farmer discovered late at night that if acow didn’t receive treatment within a few hours, it would likely die.PANTLEG’s report explains that, ‘‘The farmer was able to summon helpfrom the telecentre sta¡ who searched their networked informationsources for a veterinary surgeon who was close enough to apply treat-ment within a short time. The surgeon was contacted by telephone andhe arrived in time to save the cow’’.12

Story gathering has many positive characteristics. It is highly focusedon the individual, which provides a means to ensure social inclusion. So-cial groups and women have an opportunity to express their experiencesin their own voice. This also makes the approach very locally groundedin that the stories will re£ect the reality of the orators. As thePANTLEG group points out:

The context of the evaluation supported the approach, by allowing individuals toexpress themselves freely and by focusing special attention during each session on

11PANTLEG (1999) Op. cit.12 Ibid.


the women bene¢ciaries. The Group had every reason to believe that the storiesthey heard were genuine and were faithfully transmitted through the translationprocess. However, we acknowledge that the evaluation, depending on its purpose,could in some ways be described as super¢cial.13

Indeed, the applicability of the approach may depend on the situa-tion. By its nature, it is £exible and easily adaptable and stories are ea-sily accessible and veri¢able, however it is not a particularly robustmethodology in terms of replicability and collection of baseline datafor crosscutting comparisons. Furthermore, because the approach canuncover unanticipated impacts, it could be di⁄cult to use for certainstrategic purposes. An open-ended question is much more unwieldy thana set of survey questions, which can constrain the range of answers. Stor-ies, although useful in some contexts, may not be considered su⁄cientlyrobust by government agencies, donors or partners. For some purposes(policy development, telecentre management, economic implications), adi¡erent type of evaluation may be more appropriate.PANTLEG has explored the possibility of using story gathering

throughout a project to complement base-line empirical data gatheredat the beginning of the project. Telecentres, as was earlier pointed out,are implemented with a certain level of unpredictability: if you build itthey will come F and bene¢t. The group asks, ‘‘is our ability to predictoutcomes de¢cient in some way or is it more in the nature of the crea-tivity and resourcefulness of individuals who have been empowered withaccess to information that we should not be expected to predict howthey will conduct themselves with such new found capacities?’’.14 Withtelecentres the expected outcomes, as measured against projectobjectives or current knowledge, may not re£ect the actual impacts Fpositive or negative. For program evaluation this is problematic as fund-ing agents and policy makers may want a measurable impact. This sug-gests that learning evaluations might need to be supplemented withother kinds of data, including more quantitative information. In thiscase, story telling could be considered an indicator of areas that warrantmore systematic research.The need to choose an appropriate methodology underlines the im-

portance of involving telecentre users and operators in the planningand realisation of evaluations. The users and operators will have a posi-tion on what they consider to be a methodologically correct approachand a good strategic orientation. Furthermore, by providing an oppor-tunity for the users and operators to work out these positions, the eva-luation will become more re£ective of a common vision, and the

13 Ibid.14 PANTLEG (1999) Op. cit.


capacity of the telecentre community will be enhanced. During PAN-TLEG’s visit to India, exchanges took place at the project level in away that allowed program implementation teams to learn from eachother. The mission also allowed the program implementers and IDRCsta¡ members to collect information in a participatory manner byworking with the telecentres. However, the telecentre operators did nothave an opportunity to learn from each other. It would have been inter-esting to foster evaluation through exchanges at the telecentre level. Thiswould allow telecentre operators and users to exchange information onservice provision and uses. Furthermore, it would make evaluationsmore participatory, strategic and re£ective of the local vision of the tele-centre community.Before these groups can be expected to carry out e¡ective evaluations,

training will be required. Story telling requires facilitation and inter-viewing skills. The evaluator would not simply listen and record stories,but would also need to e¡ectively manage the information that is uncov-ered. In such an open-ended process, the stories told could unearth somedi⁄cult questions for the stakeholders. In this case, the evaluator maybe called on to suggest solutions to the problem or moderate disputes.The approach also requires that evaluation results be used in an appro-priate manner. Stories are sometimes personal, so there should be somediscussion of the ethical aspects of how the information is used. Closelyrelated to this is the imperative that the purposes and results of thestudy be made public in a manner appropriate to all stakeholders.Training on the approach would also help to overcome another poten-

tial pitfall. When two telecentres or two project co-ordination teams arebrought together there is a risk that successes will be emphasized overfailures, as was seen in the initial PANTLEG experiment. This particu-lar project was operating successfully at the time of the evaluation, andas a result, the evaluation produced a list of positive outcomes. However,it should be noted that the language of the PANTLEG report on themission to India does emphasize successes.15 Furthermore, two telecen-tres established by MSSRF had been closed and replaced by two alter-native centres prior to the mission. The reasons for these closures shouldhave been examined to bring out lessons for use in other centres. Whilethe PANTLEG group wanted to focus on how telecentres can positivelyimpact development, ignoring failures can lead to problems with pro-gram implementation and the sustainability of telecentres. Closely re-lated to this experience is the concern that learning evaluations may beat risk of emphasizing successes over failures because of inter-groupcompetition, a desire to show hospitality, or incentives to gain funding

15 PANTLEG (1999) Op. cit.


or other support. An emphasis must be placed on honest sharing of in-formation for the sake of learning.The PANTLEG group continues to develop its approach to evalua-

tion.16 Since their visit to India, PANTLEG has recognised ¢vecommon causes of concern in telecentre operation and use including:standards for telecentre operation, policy environments for telecentrepropagation, cultural context of telecentre deployment, sharing ofknowledge between PANTLEG partners, and extending capabilities tonon-PANTLEG individual and organizations. In order to address thesepoints, PANTLEG has adopted comparative evaluation and exchange ofparticipants, among other things, as crosscutting objectives for theirwork. A study mission to Sarawak has been proposed to evaluate theIDRC-supported remote community telecentre research project beingcarried out by the Universiti Malaysia Sarawak. A goal of the missionwill be to further develop the evaluation methodology and to ‘‘de¢ne apilot implementation of evaluation methodologies and techniques suita-ble for the Sarawak project and capable of enhancing the Group’s capa-city for evaluating telecentre activities’’.17 They are also working towardsdeveloping adaptable and replicable learning evaluation methods thatcould eventually become part of a training program.

PANTLEG IN LIGHT OF THE GUIDING PRINCIPLES FORTELECENTRE EVALUATION

1. Participatory FThe PANTLEG approach raises a question asto the focus of the evaluation and choice of evaluator. Storycollection was carried out by project leaders and researchersrather than telecentre operators and users. In order to be fullyparticipatory the approach should empower telecentres, inaddition to project co-ordinators, to carry out evaluations andinclude them in their planning processes.

2. Socially Inclusive F PANTLEG was aware of the need to besocially inclusive in the gathering of stories and made an e¡ort tospeak to various groups. By focusing on the individual, theapproach provides a way to portray the experiences of variousgroups within a community F as long as all voices have anopportunity to be heard.

16 PANTLEG (2000) ‘‘An agreement for joint activities to further the PANAsia programme fortelecentres.’’ (draft).17 PANTLEG (1999) Op. cit.


3. Locally Grounded F This approach is very locally groundedin that it seeks the views of local telecentre operators and userson the impacts of the technology and the centre. The storiesgathered will re£ect the local experiences and attitudes towardsthe telecentre and uncover local distinctions and needs.

4. Public and Transparent F PANTLEG listened to stories in apublic forum which makes the research process somewhattransparent. It is not clear, however, if the purpose, implicationsand results of the evaluation were made public to the studygroups.

5. Methodologically Appropriate F The applicability of thisapproach may depend on the situation. By its nature, it is £exibleand easily adaptable however group facilitation and interviewingskills are required which may make it inappropriate for certaingroups. It is a very practical approach in that stories are easilyaccessible and veri¢able, however it is not a particularly robustmethodology. There is a risk that survey groups will emphasisesuccesses over failures in exchanges with other groups because ofinter-group competition, a desire to show hospitality, or incentivesto gain funding or other support. An emphasis needs to be placedon honest sharing for the sake of learning. Also, PANTLEG hasexplored the possibility of using story gathering throughout theproject to complement base-line empirical data gathered at thebeginning of the project.

6. Sustainability Enhancing F This approach is very applicableto the promotion of quality and sustainability in that it providesan opportunity for learning exchanges and incrementalimprovement. However, this will require that telecentreoperators, in addition to project co-ordinators, are trained onevaluation techniques. Also, telecentre operators and users wouldbene¢t from the exchange of experiences, successes and failures,solutions, and plans.

7. Capacity Building F PANTLEG is still working on developingadaptable and replicable learning evaluation methods that couldtranslate into a training program. Information exchanges buildcapacity both in terms of contributions to project and telecentreadvances, and also in terms of evaluation capacity building.Serious re£ection on the failures and successes of users andoperators is empowering however, this goal would be furtherrealized if telecentre operators were trained to carry outevaluations. PANTLEG should also put more emphasis on theoften di⁄cult task of understanding failures so that they can beavoided in the future.


8. Re£ective of Shared Visions F With reference to points 1and 5, there remain questions around the interaction oftelecentres, project co-ordination groups, and donors inthe realisation of the PANTLEG evaluations. It is not clear thattelecentre user and operators have played a role in planning theevaluations. Also, stories are sometimes personal, so there needs tobe some re£ection on the ethical aspects of how the information isused.

9. Strategically Oriented F Because this approach, in part, seeksto uncover unanticipated impacts, it could uncover informationthat challenges stakeholders. Storytellers may have their ownagendas and thus the evaluator(s) cannot necessarily control thestrategic orientation of the evaluation. An open-ended question ismuch more unwieldy than a set of survey questions, which canconstrain the range of answers. Thus, this approach may bedi⁄cult to harness for strategic purposes. Stories can provide richinformation on telecentre impacts, but may not be consideredsu⁄ciently robust by government agencies, donors or partners.For some purposes (policy development, telecentre management,economic implications), a di¡erent type of evaluation may bemore appropriate.

10. Gender Sensitive F It is important that women’s voices areheard during the story collection process. When this is done, thisapproach allows women to express experiences without constraint.In turn stories can demonstrate how women are experiencing thetelecentre.

EXPERIENCES IN LATIN AMERICA

TELELAC - Exploring evaluation collectivelyIDRC’s involvement with telecentre evaluation in the Latin Americanand Caribbean context has increasingly focused on its support of theTele-centre Evaluation Network in Latin America and the Caribbean Project also knownasTELELAC (http://www.idrc.ca/pan/projects_e.htm#latinamerica) in ad-dition to a small number of pilot telecentre implementation projects.18 Thisproject, which is being implemented by the Fundacio¤ n ChasquiNet (http://www.chasquinet.org) of Ecuador, aims to support the development and

18Other telecentre projects in Latin America currently include Telecentres, Citizenship andMunicipal Governance in Mexico, InforCauca Community Telecentres in Colombia, andAshaninka F Internet by Radio for Indigenous Communities (Capacity Development forInternet use in Latin America and the Caribean F LAC) in Peru.


strengthening of local, national and regional telecentres initiatives in LatinAmerica and the Caribbean. The project’s goals emerged out of a processthat started with a survey and study conducted by IDRC sta¡ followed bya discussion on the needs and expectations of telecentre operators and re-searchers on an electronic mailing list ( January^September, 1999). Fromthis process it was determined that telecentre monitoring and evaluationis not strong in the region and in particular that:

K Monitoring was often regarded as a project closure activity;K Monitoring during projects was mostly used as a means to correcterrors or introduce e⁄ciencies;

K Evaluations have been either part of needs identi¢cation exercises,or carried out by external parties;

K There was a need for a regional forum to communicatingexperiences, review methodologies, publicise results, and articulateregional and local dynamics.19

Thus the major question faced by the TELELAC group was how to goabout developing a collective evaluation process so that telecentres inthe region could learn from each other through their experiences. Theproject’s activities were elaborated ¢rst at the Far Hills meeting, andthen during a planning meeting held in Papallacta, Ecuador in March,2000 that included members of Chasquinet and other organisationswhich are actively involved in the project. The project is pursuing threemain activities:

1) research, analysis, diagnostics, documentation and creation ofmaterials to be used by telecentre in Latin America and theCaribbean, as well as collection of historical telecentre experiences;

2) the development of a collective space for hands on learningwhich will o¡er tools and services to strengthen telecentres; and

3) an opportunity to develop, experiment and share a set of methods,techniques, and instruments for monitoring and evaluating thework of telecentres.20

Main goals of the project are to establish strategic alliances with se-lected bene¢ciary groups (especially marginalized groups), build usercapacity, and train community facilitators. The philosophy of the projectis based on democracy of access and knowledge, solidarity and mutual

19 ChasquiNet. (2000b) Bienvenidos al Proyecto Telelac. ChasquiNet. Available: http://www.chas-quinet.org/telelac [viewed 5 June 2000].20 ChasquiNet. (2000a) Proyecto Telelac: Lecciones Aprendidas por los Telecentros y Fortalecimiento de susAcciones al Servicio de la Sociedad Civil. ChasquiNet. Available: http://www.tele-centros.org/proyecto[May 4, 2000].


support, participation and transparency, proactive participation, and re-spect of diversity and gender equity.A set of guidelines was established at the Papallacta meeting to deter-

mine what level of participation would be necessary and acceptable foreach component of the project.21 There is a co-ordination team consist-ing of Chasquinet members responsible for de¢ning the structure andoverseeing the implementation of the project. The ¢ve-member assess-ment committee is made of up representatives of interested organizationsand is responsible for ensuring that the goals of the project are met tothe satisfaction of the members of TELELAC. The operational teamconsists of Chasquinet members as well as other interested parties andis responsible for the implementation of the project. All activities are tobe carried out with reference to the TELELAC network through a pro-ject electronic mailing list and the existing TELELAC electronic mail-ing list. Each group or team also has a dedicated electronic mailing listto facilitate internal communication.The research component includes a quantitative inventory and qua-

litative analysis of telecentre initiatives in the region, and research todetermine the ‘‘state of the art’’ in telecentre operations. Materials arebeing developed for use by telecentres with emphasis on collecting anddocumenting telecentre histories and experiences. The project is alsostressing the importance of articulating local and regional dynamics gi-ven the close relationship between communities and telecentre opera-tions. Two major research themes within this activity are to determineways to adapt new technologies for communities with limited means,and to examine methods for diverse groups to in£uence telecommunica-tions policy.The second main project activity is the development of an on-line tele-

centreresourceclearinghouse (locatedathttp://www.tele-centros.org)whichwill be used to strengthen collaborativework and exchange informa-tion. It will include a resource centre and toolkit with information and in-structionalmaterials relevant to telecentre operators andusers.Finally the evaluation component will draw on the research com-

ponent, as well as other investigations, to develop methodologies andtechniques for monitoring and evaluating the work of telecentres. Thesematerials will become part of the online toolkit. It is hoped that themonitoring and evaluation techniques will help operators and users tocreate a sustainable and meaningful telecentre institution. The metho-dology chosen for these activities is highly participatory focusing on:

K learning through practice;

21ChasquiNet. (2000b) Op. cit.


K using a multidisciplinary team for the development of toolsinvolving, ‘‘grassroots communities, operators and promoters oftelecentres working directly with grassroots groups, researchersand academics who are using these technologies,’’; and

K documenting and systematising the experiences of these groups tofacilitate their dissemination and use.22

For the TELELAC group the development of monitoring and evaluationmethodologies and techniques follows a transparent and participatoryprocess of research and development. A major strategy is to focus oncommon priorities for monitoring and evaluation as determined througha collaborative and collective process of on-line discussions. The evalua-tion tools developed cover diagnostic tools, planning, management, andoperations. The ¢rst step is to identify, analyse and adapt existing tele-centre evaluation materials, with special attention to social, economic,cultural and gender parameters. This process is being complementedby the collection of existing telecentre evaluation results. An analyticalframework and evaluation methodology will then be drafted, and re-vised after discussion with the TELELAC group.A three-day workshop will be held to train a multi-disciplinary team

of facilitators. These individuals will then undertake three pilot evalua-tions to test and re¢ne the methodology, materials and instruments de-veloped. A major goal of the pilot tests is to examine the adaptabilityand £exibility of the methodology to meet the speci¢c needs of eachpilot telecentre. Finally, a telecentre evaluation resource kit will beprepared and shared using the Tele-Centros site. This will include amonitoring system, an evaluation guide, and a training module for eva-luation facilitators. The kit will also include a pool of evaluation experts,a continually updated set of lessons learned, bibliographies, and exam-ples of methods and models used in various telecentres.The development of the project plan has been completed but accord-

ing to Michel Menou,23 one of the researchers participating in the pro-ject, work has yet to commence on the monitoring and evaluationmethodologies. In analysing the TELELAC project, it is important toremember the following two points. 1) The TELELAC project is work-ing on the development of a telecentre evaluation methodology, and notthe evaluation of telecentres as such (except in the three pilot studies).And 2), The shape of the evaluation methodology to be developed

22 Ibid.23Menou, M. (1999) Impact of the Internet: some conceptual and methodological issues, or howto hit a moving target behind the smoke screen. In Go¤ mez & Hunt (Eds).Telecentre Evaluation: AGlobal Perspective (Report of an International Meeting on Telecentre Evaluation). Far Hills Inn, Quebec,Canada.


cannot be predicted, especially since it is to be developed through a par-ticipatory process. On the other hand, the development process and phi-losophy is likely to be re£ected in the ¢nal product. In Menou’s view, theevaluation methodology to be developed should ensure that,

The communities (groups of stakeholders) who are going to use/bene¢t from the telecentre should identify the expected socialchanges, specify how they can be observed, and validate theobservations. They should appoint representatives to a Monitoringand Evaluation (M&E) committee which will assist the telecentreoperators. Panels and group meetings should be implemented atregular (quarterly) intervals in order to discuss what is happeningin relation to ‘‘key problems’’ illustrating the expected changes.External back up will be provided as appropriate and feasible, butmost of the M&E should be carried out from within thestakeholders communities whose representatives should be trainedfor that purpose, together with telecentre operators.24

Menou also envisions telecentre evaluation to focus on enhancing thee¡ectiveness of the operation at the telecentre level; he sees telecentreevaluation as feeding into wider project evaluations. Finally, he alsopoints out that gender considerations will be a part of all phases of theproject. According to Menou, ‘‘If designation of committee members or‘‘key informants’’ does not secure fair representation of a particulargroup, or women, this would need to be corrected, possibly by duplicat-ing relevant parts of the M&E with adhoc panels.Whenever the specialissues under review so require, women should be involved separately’’.25

This participatory approach can produce rich information and pro-vides an opportunity for the people involved in the process to determinethe research methodology. In this project, evaluation has meant tryingto ¢gure out where things are at; it is a process of ‘‘ubicacio¤ n’’ (localiza-tion). It is of particular note that this process is focused on the telecentrelevel and will allow telecentre users and operators to take ownership ofdeveloping and administering the evaluation process. However, re£ect-ing on the much more structured approach taken by the ELSA project,several questions arise. As discussed in the previous section, ELSA seesthe importance of collecting baseline ‘‘pre-connectivity’’ information be-fore the opportunity has passed. The urgency of this important taskbegs the question: Will the participatory forums undertaken by TELE-LAC translate into a viable approach to evaluation and locally andregionally applicable results?

24 Ibid.25 Ibid.


Another question surrounds the application of the methodology bytelecentre users and operators. Based on the philosophy of the projectand the attitudes of its researchers, we can predict that the resultingmethodology will be convergent with the 10 Guidelines for TelecentreEvaluation. However, in the end, the strength of the approach will de-pend on how it is applied by telecentre users and operators. On the onehand, the existence of the Tele-Centros virtual community will help toensure that this is done e¡ectively at the telecentre level, and that ex-periences are shared at the regional level. But, the project will need toprovide adequate training and documentation on the approach. In par-ticular, since a goal of the methodology should be responsiveness to stra-tegic orientations and local distinctions, information and training willneed to be provided to advise the development of evaluation plans. Theevaluation toolkit should also include complementary information to en-sure its proper and e¡ective application. This should cover participation,social inclusion, transparency, and gender. This information should ulti-mately contribute to an evaluation that is methodologically appropriate,re£ective of shared visions, sustainability enhancing, and strategicallyoriented.Setting these concerns aside, the TELELAC approach has many mer-

its. It is transparent and participatory, locally grounded, highly public,facilitates sustainability, and is supportive of capacity building e¡orts.This collective process will culminate in a collective result. By providingan opportunity for participation and capacity building at the telecentreand partner levels, the project is developing capacity for evaluationwhich will result in better local information and data for use in broaderstudies. Furthermore, this process of discovery will help the telecentrecommunity to take ownership of the evaluation and research processes.The questions being asked are justi¢able, but the results of theTELELAC project will be watched with interest.

TELELAC IN LIGHT OF THE GUIDING PRINCIPLES FORTELECENTRE EVALUATION

1. Participatory F The process for developing the TELELACevaluation methodology is participatory and a criterion for theimplementation of the project is that it be transparent andparticipatory. While it does not necessarily follow that theresulting methodology will be participatory, there is a strongindication F based on project philosophy, and on discussionswith project members F that this will be a major consideration.


2. Socially Inclusive F Until the actual methodology isdeveloped it is di⁄cult to judge whether it will be sociallyinclusive, but once again, based on the project philosophy andinterviews, it appears that this will be the case. It will alsodepend on how the methodology is applied by the individualtelecentres, however the existence of the Tele-Centros networkcould help ensure that this goal is met.

3. Locally Grounded F TELELAC aims to develop anevaluation methodology that can be adapted to the context ofthe telecentre user. It will be challenging to develop a tool thatwill allow for aggregation or comparison of regional data, thatis also £exible enough to accommodate local contexts, needs,visions and strategies.

4. Public and Transparent F The process for developing theevaluation methodology is highly public and transparent. Allmaterials will be posted on the tele-centros web-site. However,once again, alignment with this criterion will be di⁄cult tojudge until the methodology is developed and will depend onindependent applications. The Tele-Centros web-site will providea means for telecentre groups to share their evaluationexperiences and results.

5. Methodologically Appropriate F The development of amethodologically appropriate framework is the main goal of theevaluation portion of the TELELAC project. A major challengewill be to create a methodology that is £exible and adaptable tothe speci¢c needs of each telecentre, yet still provides a basis forregional comparisons.

6. Sustainability Enhancing F It is the intention of theTELELAC project to enhance the sustainability of telecentres.This will be done by developing evaluation capacity through theestablishment of appropriate methodologies and training, andfostering learning through the Tele-Centros virtual space.

7. Capacity Building F Not only is the TELELAC project acapacity building exercise as such, but it will result in a set ofresources and training exercises that will build that capacity ofthe participants in the project. Furthermore, the tele-centrosvirtual space will provide resources that will allow telecentrepractitioners and users to develop their capacity to moree¡ectively use and operate telecentres. It will also provide aspace for telecentre operators and users to exchange lessonslearned.

8. Re£ective of Shared Visions F The development of theTELELAC evaluation methodology is highly re£ective of shared


visions based on the results of surveys and group discussions.According to Menou’s views, the methodology to be establishedshould also support this type of process, however, once again thisis yet to be determined.

9. Strategically Oriented F The methodology developed mustbe £exible enough to be applied towards various ends. It willneed to provide information for users and the community, formanagement, for outside agencies etc. Strategic orientation willdepend on how the methodology gets applied and this should betaken into consideration in presentations of, or training on thematerial. The pros and cons of the methodology should behighlighted and situation-based alternatives should be provided.

10. Gender Sensitive F Gender issues are being taken intoconsideration by the project team. Once again, the gendersensitivity of the evaluation technique to be developed is yet tobe seen, but current indications are that the criteria will beincorporated. It may be necessary to provide gender sensitivitytraining in order to ensure that the criterion is properly appliedat the telecentre level.

CONCLUSIONS

By comparing the approaches to evaluation being developed by IDRCprojects in Asia and Latin America against the Guidelines for TelecentreEvaluation we can see several regional and institutional di¡erences.PANTLEG is highly focused on the individual and on people, makingthe approach participatory, socially inclusive, locally grounded and sen-sitive to gender concerns. TELELAC has focused on the need to developcollective processes for fostering capacity and sustainability, and on dis-semination strategies.There are common threads that can be drawn from these experi-

ences. Re£ecting on the discussion of evaluation purposes presented inthe introduction, this paper constitutes evaluation of the evaluation sys-tem, or meta-evaluation. Meta-evaluation can be a part of the ongoingprocess of the three regional experiences as they work to develop theirapproach, methodologies and tools. In all three cases, the developmentof methodologies and tools is an ongoing process with PANTLEG work-ing on story-telling techniques, and TELELAC focusing on collectivediscussions of appropriate tools. Training goes hand in hand withthe development of an approach and again, this was an issue in all threecases. PANTLEG will need to provide facilitation training for telecentreoperators, and TELELAC should prove training materials to


accompany its online tool kit in order to ensure that tools are properlyimplemented. Finally, both approaches share a need to disseminate re-sults. In the case of TELELAC, dissemination channels have been estab-lished, but in the other case, strategies are weak in this area.Both approaches have provided an opportunity to re£ect on the pur-

pose of evaluations in the context of telecentres and the role of donoragencies in facilitating them. The ¢gure below compares the main focusof the evaluation strategy in each of the three cases. PANTLEG has fo-cused on collecting data from individual telecentres, with participationof both program o⁄cers and telecentre users and sta¡. However, todate, recipients and IDRC sta¡ have used this information, not indivi-dual telecentres. TELELAC is working on developing a methodologyand communication strategy to facilitate evaluations at the local level,primarily for the bene¢t of telecentre operators and users. However,the project’s on-line data gathering and communications strategies willprovide a means to gather information of use for bigger researchquestions.

Focus of the evaluation strategy

Evaluations have both academic and more practical drivers and thisis particularly true of the new ¢eld of telecentre evaluation. Donorshave a responsibility not only to support the useful and ¢nancially re-sponsible realisation of telecentre projects, but they also have a role toplay in analysing and theorising on the impact of telecentres. We needto recognize that this is an inherently political task. As Menou pointsout, ‘‘Policy and decision makers who are approving or supportingprograms for the development of the Internet, would like to havesome sort of evidence in support of the claims by the proponents ofthemselves that it will bring about all the said bene¢ts’’.26 While re-search on telecentres is being done by way of evaluation, it is importantto remember that the methodologies we use play a role in the researchresults. People have a right to have a voice in the results of telecentre

26Menou (1999) Op. cit.


evaluations because the research generated will determine the future di-rection of telecentre activities.But at the same time, it is important to remember that individual tel-

ecentres have specialized needs and will require training and support tocarry out local, specialized evaluations. Local research programs arealso important for the identi¢cation of best practices and impacts. Their¢ndings should be incorporated into the objectives of programs in theregion to be re£ected in future program evaluations. To facilitate thistype of research, IDRC and its partners need to develop guidelines ortechniques for local small-scale evaluation that are easy to implementat a low cost, and champion their application in telecentres throughprogram work. Better local information can not only facilitate betterproject results, but can also provide a basis for better program evalua-tions and research. Finally, IDRC should contribute to the use of evalua-tion approaches that facilitate the participation of partners andtelecentres in the research process so that they have a role in determin-ing the parameters of their work.This analysis of IDRC telecentre evaluation experiences in Asia and

Latin America has demonstrated the usefulness of the Guidelines forTelecentre Evaluation as they provide a common framework for asses-sing each experience. The guidelines can be used to assess other telecen-tre experiences and allow us to determine if telecentre evaluation isbeing useful, ¢nancially responsible, if it is building local capacity andenabling lesson sharing. IDRC is also using the guidelines as a startingpoint as it works towards developing a centre-wide framework for ICTevaluation.

FURTHER READING

ChasquiNet. (2000c) Informe de la Reunio¤ n del ProyectoTelelac, Papallacta, Ecuador, May 26^30.Gomez, R., Hunt, P. & Lamoureaux, E. (1999) Telecentre evaluation and research: a global

perspective. In Go¤ mez & Hunt (Eds).Telecentre Evaluation: A Global Perspective (Report of an Interna-tional Meeting onTelecentre Evaluation). Far Hills Inn, Quebec, Canada.Graham, M. (1997) ‘‘Use of Information and Communication Technologies in IDRC Project:

Lessons Learned.’’ IDRC Website, URL: http://www.idrc.ca/acacia/outputs/op-eval.htm, [viewed24 May 2000].

comparing approaches: telecentre evaluation experiences in asia and latin america

Documents