ict grants: monitoring & evaluation€¦ · 2 dbe1 ict grants monitoring & evaluation 2008 usaid...
TRANSCRIPT
-
1
Decentralized Basic Education 1: Management and Governance
ICT Grants:
Monitoring & Evaluation
July 2008
This report is one of a series of special reports produced by Research Triangle Institute (RTI),
Implementing Partner for the USAID-funded Improved Quality of Decentralized Basic Education
(IQDBE) program in Indonesia
Special Monitoring Report
-
2
DBE1
ICT GRANTS
Monitoring & Evaluation
2008 USAID DBE1 – Management and Education Governance Indonesia Stock Exchange Building Tower 1, 29
th Floor
Jl. Jend. Sudirman Kav. 52-53, Phone (62 21) 515 2772 • Fax (62 21) 515 5859
Jakarta 12190, Indonesia http://pdms.dbeindonesia.org
http://www.dbe-usaid.org
http://pdms.dbeindonesia.org/http://www.dbe-usaid.org/
-
3
DBE1 – ICT Grants Program Monitoring Activity
Outcomes and Recommendations
April/May 2008
Purpose of the Activity The purpose of the DBE1 ICT Grants Program Monitoring Activity is to provide an insight
into the DBE1 ICT Grants Program to date, document lessons learned, provide
recommendations and propose next steps for its implementation, especially in regard to the
feasibility to conduct round II of the program with DBE1 cohort 2 districts, and in regard to
necessary activities to capture and document lessons learned from existing grant programs.
I. BACKGROUND
1. Task Order Requirements:
Grants to support RFP objectives and wider community
Grants to include private sector
Grants to be competitive
Grants to show sustainable models for profit-driven education hotspots (business plans)
Grants to use business and/or tech volunteers
Grants programs of DBE123 to be different from each other to show approaches
2. DBE1 ICT Grants Program – Key Aspects:
One program for both education hotspot (ICT Access Grants) and grants targeting
education management and governance (EMG) (ICT Innovation EMG Grants)
Program objectives:
o Leverage ICT to contribute to more effective management and governance of
education
o Leverage ICT to contribute to more effective information and communication
among education stakeholders and the wider community
Two tier system:
Tier 1 grants range between $2,000 and $4,000 in funding.
Tier 2 grants range between $10,000 and $40,000 in funding.
Consortium requirements: At least one private sector actor and one DEO agency
Cost-sharing requirement
Grant program as an opportunity to broker relationships and build capacity
o Sensibilization and Awareness raising activities,
o Regional forums to broker consortium,
o Assistance (manuals, in-person) in grant application development,
o Program (objective alignment, coherence, M&E) and budget review
assistance (budget integrity, unit estimations, business planning) for short
listed applicants,
o Review and Assistance of applicant financial management practice for grant
management,
o Grantee resource Toolkit.
-
4
3. Related DBE1 Indicators:
IR 3: Increased use of Information Resources to Enhance Education Management and
Governance:
o IR 3-24: Increased capacity of education stakeholders in target districts to use
ICT in education management and governance.
o IR 3-25: Number of grants awarded to district governments in collaboration
with the private sector to develop and implement activities that increase use
of ICT and are suitable of wider application.
o IR 3-26: Number of grants awarded to district public institutions in
collaboration with the private sector to develop and implement ”education
hotspots” that have sustainable business plans and are suitable of wider
application.
For the purpose of this grants program and internal tracking, IR 3-24 has been further
differentiated into three key aspects:
1. Increased capacity for use of ICT among education stakeholders
2. Increased access to functioning ICT infrastructure for education stakeholders
3. Increased access to education information and services for education stakeholders
4. Grantees to Date (Table 1):
14 Grants.
South Sulawesi (4), West Java (2), Banten (1), Central Java (3), East Java (3), North
Sumatera (1).
USD 246,284 committed.
7 Tier 1; 7 Tier 2 grantees.
7 ICT Innovation EMG Grants and 7 ICT Access Grants (hotspot).
Grantee main activities (Figure 1).
Figure 1. Distribution of Grantee Main Activities
-
5
Table 1. Grantees and Grantee Main Activities to Date
NO GRANT CATEGORY
GRANTEE NAME
(PRIVATE SECTOR
PARTNER)
LOCATION
GRANTEE MAIN ACTIVITIES
Inte
rnet
Café
(W
arn
et)
Lib
rary
MIS
EM
IS/M
IS
Educa
tion P
ort
al
eR
eso
urc
es
IT C
ente
r/
Connect
ivity/
Netw
ork
IT M
ain
tenance
/ Serv
icin
g
Sch
ool Connect
ivity
Tra
inin
g (
DEO
)
Tra
inin
g (
Oth
ers
)
1. Tier-1 EMG
Turatea Computer Center
Janeponto - South Sulawesi
2. Tier-2 EMG
YPK Amanah Pangkep - South Sulawesi
3. Tier-1 Hotspot
Indo Komputer Soppeng - South Sulawesi
4. Tier-1 Hotspot
PT Rekayasa Teknologi Informasi
Enrekang - South Sulawesi
5. Tier-1 Hotspot
PT Tridata Cakrawala
Tuban - East Java
6. Tier-1 Hotspot
Yayasan Tarbiyah Islamiyah
Sukabumi - West Java
7. Tier-2 EMG
CV Trisatya Pratama
Karawang - West Java
8. Tier-2 Hotspot
CV Almagada Jaya
Tangerang - Banten
9. Tier-2 EMG
PT Indomaya Wira Sejahtera
Karanganyar - Central Java
10. Tier-1 Hotspot
PT Indomaya Wira Sejahtera
Karanganyar - Central Java
11. Tier-2 EMG
CV Cosmo Jaya Klaten - Central Java
12. Tier-2 EMG
PT ITS Kemitraan Surabaya - East Java
13. Tier-1 EMG
PT ITS Kemitraan Surabaya - East Java
14. Tier-2 Hotspot
PT Indoukm Insis Tapanuli Utara – North Sumatra
Grantees Main Activities 6 5 4 5 2 8 1 3 13 7
5. Problem Statements according to Grant Applications:
Grant applications included the requirement of a problem statement that would drive the
purpose/objectives of the proposed grants program. Problem statements from the 14 grant
proposals are summarized below:
There are inefficiencies in education management and governance (information-based
decision making, financial modeling, planning)
o There is limited capacity of DEO staff to make use of ICT for education
management and governance ( available computers are underutilized)
-
6
o There are communication gaps between DEO and education stakeholders
(critical information arrives only in part and late)
o There are communication gaps between different DEO agencies
o There are information discrepancies between different DEO agencies
(fragmentation of management information systems, no access to easy and
secure communication and information exchange channels)
o There is underutilization of existing infrastructure (hardware, software or
connectivity limitations/failures)
There is a lack of self-study and/or complementary teaching and learning resources
in “digestible” formats
There is limited capacity among teachers and relevant DEO counterparts in
developing electronic teaching and learning resources to complement existing
materials
There are inefficiencies in DPL management (user registration, collection processing,
cataloguing, retrieval, user transactions and monitoring are done manually, using
paper-based systems)
Education stakeholders lack structured avenues/opportunities to access (education)
information and related services
Community members lack structured avenues/opportunities to access reliable
(education) information/data and services (problem of social inclusion)
II. GRANT IMPLEMENTATION MONITORING FINDINGS
The following are key findings of the April 2008 DBE1 ICT Grants Program monitoring
activity, which included interviews with DBE1 staff managing and administrating the grants,
review of grantee documentation, as well as site visits and interviews to several grantees:
USAID and RTI grant regulations require heavy pre-award determination efforts.
This resulted in considerable delays in grant implementation.
USAID and RTI procurement regulations require significant efforts in procurement.
This resulted in considerable delays in grant implementation for most grantees.
During the rather lengthy pre-award determination and procurement process several
of the private sector consortium leaders changed their status or company name which
required updates to grant application materials and led to even further delays.
Due to the rather lengthy pre-award determination and procurement process, several
grantees had to withdraw their grant applications because of changes in personnel or
regulations that prohibited them to continue the process and participate in the
program.
A final 14 of 22 selected grantees signed a grant award agreement with
DBE1/USAID and are currently in the implementation process.
Grantee consortia interviewed during the field visits show, in all cases, very good
collaboration among partners, that is district public libraries.
-
7
Current grantees show a wide variety of activities and different approaches, both in
establishing education hotspots (ICT Access Grants), as well as in using technology
to improve management and governance of education (ICT Innovation EMG Grants).
Capacity building is an integral part of most grantees’ programs and activities.
Status of grant implementation is varied. Some grantees have already completed most
of their activities and are now in the ongoing monitoring phase of their grants
program, others are still in the procurement stage for equipment (e.g. to set up an
education hotspot).
Grantee capacity for sound M&E and reporting is very limited.
III. RECOMMENDATIONS & NEXT STEPS
1. Recommendations:
The key recommendation in regard to the DBE1 ICT Grants Program is to capitalize and
focus on the existing 14 grantees, rather than expanding the grants program to more districts
and actors at this stage of the program.
Lessons learned from the program to date highlight challenges DBE1 and grantees were
facing in regard to processes and procedures of grant award pre-determination and equipment
procurement, which affected effective and timely processing and implementation. At the same
time, the 14 existing grants already show a wide variety of activities and in each a number of
different approaches that provide a unique opportunity to identify lessons learned, study best
practices and serve as examples for future programs and activities in this area, which was a
key objective for the overall grants program.
2. Key Questions to Move on:
The recommendation above, therefore, highlights the need for sound and detailed monitoring
and documentation of the existing 14 grants program. In this context, the following two key
questions need to be addressed in planning next steps:
(1) What do we need to do (now and in the coming months/years) in terms
supporting and monitoring grantee activities?
(2) What systems do we need to put in place and what information do we need to
collect (and when) to write a substantial evaluation report of the grants program
in two years time.
3. Proposed Next Steps:
To address these questions, on the basis of the information gathered and the recommendation
made, the following next steps are being proposed:
To confirm lessons learned and findings from the April 2008 site visits:
1. Upon receipt of first progress report from each grantee, review for completeness and
compliance with general reporting requirements.
2. Upon receipt of first progress report from each grantee, carefully review all grantees
elaborations on Monitoring and Evaluation (M&E) of their grants program:
-
8
a. Were indicators “customized” on the basis of DBE1 guidance (per DBE1 ICT
Grantee Toolkit and related socialization/grantee briefing activities).
b. Does the grantee indicate understanding on what indicators are and how they
are supposed to provide progress information?
c. Does the grantee demonstrate capacity in implementing appropriate M&E
activities, such as surveys or training evaluations, e.g.?
To address challenges in regard to M&E and reporting:
3. Provide personalized support to each grantee to clarify gaps or aspects that were
found to be unclear, e.g., on
d. Required content for their progress reports (drawing on examples in the grant
implementation toolkit, which all grantee have received upon award).
e. The difference between the requirements to fulfill grant progress payment
milestones and the requirements in regard to monitoring, evaluating and
documenting their progress in achieving grant program objectives.
f. Clarification on the key issues to be addressed by their program (re-iteration
of connection between problem statement and program objectives).
g. Customization of M&E indicators to their individual programs (on the basis
of the examples provided in the grantee toolkit).
To manage and document grantees’ progress and the overall program,
4. Carefully track each grantee’s progress on activity status and reporting to DBE1, as
well as each grantee’s adherence to an appropriate M&E framework, that addresses
information needs for a later overall grants program evaluation.
In this regard, the following tools have been provided, developed together with DBE1 EMIS
& ICT Specialist and M&E Specialist, to support implementation of next steps and its key
task to carefully monitor grantee activities over the next 2 years:
1. A tool to facilitate tracking of grantee milestones and progress payment disbursement,
including an overview on all grantees’ activities, and tools for documentation of grant
program implementation status on monthly or quarterly basis. This is Appendix 1is
attached to this report.
2. An updated ICT Grants Program M&E Framework (on the basis of what had been
developed in July 2007). This is Appendix 2 is attached to this report.
3. A tool to track overall ICT grants program indicators, as well as customized grantee
indicator tracking sheets for each individual grantee. This is Appendix 3 is attached to
this report.
It is anticipated that this support to grantees in areas of M&E and careful tracking and
documentation on grant program activities and grantees progress towards program objectives
will provide DBE1 with the information needed to write a substantial evaluation report of the
grants program in two years time, highlighting different approaches and models on the use of
ICT to strengthen management and governance of education and to benefit the wider
community.
-
9
APPENDIX 1
Tracking of Grantee Milestones
and Progress Payment
Disbursement
-
10
INVENTORY OF GRANTS Instructions:
1. Please complete yellow highlighted cells.
Country: 2. Please add specific comments in a footnote
Project Name:
Project Start & End Date
RTI Project Number:
Contract/Agreement Number:
Grant Manager/Email:
PAS:
Number of GrantsTotal Committed
AmountTotal Obligated Amount
$0,00 $0,00
Simplified Cost Reimbursement
Grants (SIGs) Issued
Grantee Name Start
& End Date
Total Grant Amount Obligated Amount as
of 06/30/08
Grantee Name
Start & End Date
Total Grant Amount Obligated Amount as
of 06/30/08
Grantee Name
Start & End Date
Total Grant Amount Obligated Amount as
of 06/30/08
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
$0,00 $0,00 $0,00 $0,00 $0,00 $0,00
Through June 30, 2008
Fixed Obligated Grants (FOGs) Issued Standard Cost Reimbursement
Grants (STGs) Issued
Total Grants Budget
(Dollar Value)
-
11
ICT GRANT RECORDS As of May 31, 2008
NO GRANT USAID GRANTEE NAME LOCATION Grant Period
No APPROVAL IN RP IN USD IN RP IN USD IN RP IN USD
1 0209604-G-07-001 April 2007 Turatea Computer Center Janeponto - South Sulawesi (Apr 17,2007 - Mar 17, 2008) 25.252.500 2.806 18.300.000 2.033 6.952.500 773
2 0209604-G-07-002 April 2007 YPK Amanah Pangkep - South Sulawesi (Apr 30,2007 - Apr 30,2009) 415.152.500 46.128 99.493.375 11.055 315.659.125 35.073
3 0209604-G-07-003 May, 2007 Indo Komputer Soppeng - South Sulawesi (Aug 1,2007 - Aug1, 2008) 31.243.000 3.471 21.730.000 2.414 9.513.000 1.057
4 0209604-G-07-004 May, 2007 PT Rekayasa Teknologi Informasi Enrekang - South Sulawesi (Apr 1, 2007 - Dec 31, 2007) 25.925.000 2.881 7.777.500 864 18.147.500 2.016
5 0209604-G-07-005 June, 2007 PT Tridata Cakrawala Tuban - East Java (June 18, 2007 - Feb 14, 2008) 69.754.000 7.750 61.304.000 6.812 8.450.000 939
6 0209604-G-07-006 June, 2007 Yayasan Tarbiyah Islamiyah Sukabumi - West Java (June 27,2007 - May 28,2008) 60.670.400 6.741 37.430.000 4.159 23.240.400 2.582
7 0209604-G-07-007 06-Sep-07 CV Trisatya Pratama Karawang - West Java (Sept 6, 2007 - Sept 4, 2009) 292.518.451 32.502 21.576.553 2.397 270.941.898 30.105
8 0209604-G-08-008 13-Nov-07 CV Almagada Jaya Tangerang - Banten (August 22,2007 - July 22, 2009) 300.930.000 33.437 119.927.250 13.325 181.002.750 20.111
9 0209604-G-08-009 13-Nov-07 PT Indomaya Wira Sejahtera - T2 Karanganyar - Central Java (Oct 10, 2007 - Sept 10, 2009) 258.532.500 28.726 17.270.500 1.919 241.262.000 26.807
10 0209604-G-08-010 13-Nov-07 PT Indomaya Wira Sejahtera - T1 Karanganyar - Central Java (Sept 10, 2007 - Aug 19, 2008) 27.377.000 3.042 4.809.300 534 22.567.700 2.508
11 0209604-G-08-011 7-Jan-08 CV Cosmo Jaya Klaten - Central Java (Jan 7,2008 - Jan 7, 2010) 347.461.000 38.607 26.125.000 2.903 321.336.000 35.704
12 0209604-G-08-012 4-Jan-08 PT ITS Kemitraan - T2 Surabaya - East Java (Jan 17, 2008 - Jan 15, 2010) 311.557.100 34.617 - - 311.557.100 34.617
13 0209604-G-08-013 22-Feb-08 PT ITS Kemitraan - T1 Surabaya - East Java (Feb 4, 2008 - Feb 4, 2009) 50.184.000 5.576 - - 50.184.000 5.576
TOTAL 2.216.557.451 246.284 435.743.478 48.416 1.780.813.973 197.868
GRANT AMOUNT REALISATION GRANT Obligated Amount
-
12
PROVINCE: SOUTH SULAWESI
1. TURATEA COMPUTER CENTER Bank Voucher
NO SCHEDULE AMOUNT REALIZATION REALISATION Balance NOTES
OF PAYMENTS PAYMENTS AMOUNT
1 17-Apr-07 3.800.000 30-Apr-07 3.800.000 - MAK/0000464/BV
17-Apr-07 6.550.000 30-Apr-07 6.200.000 350.000 1 unit Laptop MAK/0000465/BV
2 17-Jul-07 4.500.000 18-Jul-07 4.500.000 - MAK/0000598/BV
3 17-Sep-07 3.800.000 4-Oct-07 3.800.000 - MAK/0000734/BV
4 27-Nov-07 3.800.000 3.800.000
5 17-Mar-08 2.802.500 2.802.500
TOTAL 25.252.500 18.300.000 6.952.500
2. INDO KOMPUTER
NO SCHEDULE AMOUNT REALIZATION REALISATION Balance NOTES
OF PAYMENTS PAYMENTS AMOUNT
1 14-Aug-07 5.280.000 20-Aug-07 5.280.000 - MAK/ 0000644/BV
14-Aug-07 6.520.000 29-Oct-07 7.600.000 (1.080.000) Equipments MAK/ 0000750/BV
2 14-Sep-07 8.850.000 15-Jan-08 8.850.000 - MAK/ 0000852/BV
3 15-Jan-08 6.300.000 6.300.000
4 15-May-07 2.725.000 2.725.000
5 15-Jul-07 1.568.000 1.568.000
TOTAL 31.243.000 21.730.000 9.513.000
3. PT REKAYASA TEKNOLOGI INFORMASI
NO SCHEDULE AMOUNT REALIZATION REALISATION Balance NOTES
OF PAYMENTS PAYMENTS AMOUNT
1 16-May-07 7.777.500 7.777.500 - MAK/ 0001110/BV
2 19-Jul-07 10.370.000 10.370.000
3 18-Dec-07 7.777.500 7.777.500
TOTAL 25.925.000 - 7.777.500 18.147.500
4. YPK AMANAH (SIG with advance)
NO ADVANCES AMOUNT REALIZATION REALISATION OUTSTANDING NOTES
MONTH PAYMENTS AMOUNT ADVANCES
1 July 2007 15.625.000 17-Sep-07 13.525.375 - MAK/000708/BV
13-Feb-08 1.270.000 MAK/000910/BV
2 14-Dec-07 7.700.000 laptop & printer MAK/000798/BV
3 Dec 2007 4.180.000 20-Feb-08 3.634.000 546.000 MAK/000926/BV
4 1-Feb-08 62.500.000 10 unit computer MAK/000893/BV
5 15-May-08 6.264.000 personel fee oct 07 - Mar 08 MAK/001202/BV
6 19-May-08 4.600.000 1 unit generator Yamaha MAK/001208/BV
7
8
TOTAL 19.805.000 99.493.375 546.000
-
13
APPENDIX 2
Updated ICT Grants Program
M&E Framework
-
14
DBE1 ICT Grants Program – M&E Framework
I. DBE1 – Existing Results Framework
II. DBE1 – Proposed Results Framework
III. SIR 3-24.1 : Increased capacity for ICT use among Education Stakeholders – Indicators
IV. SIR 3-24.2: Increased Access to Education Information – Indicators
V. SIR 3-24.3: Increased Access to ICT - Indicators Framework explanation: This framework tries to outline a systematic approach of
integrating indicators and results with a focus on the grantee level.
This framework provides a set of suggested indicators for each main grantee activity, such as
training, establishment of a connectivity or network between diknas and depag, development
of a web-based education information system, or establishment of an education hotspot/public
Internet access point.
Each indicator in this framework is presented with an indicator definition, method and
frequency of data collection, method of calculation, data proof, the key person responsible for
tracking it, and with information on who will need to report it to DBE1 how often and in what
format.
The indicator tables below are to be customized for each grantee and used, as appropriate,
for the explanatory note accompanying the grantee indicator tracking sheets developed in
parallel to this framework.
-
15
I. DBE 1 – EXISTING RESULTS FRAMEWORK
USAID Program Objective: More effective decentralized education management
and governance
USAID IR3: Number of districts using enhanced ICT for data and information reporting,
assessing best practices, and enhanced communication in the field of education
DBE1 IR 3: Increased use of information resources to enhance education management and
governance
DBE1 Intermediate Result 3-24: Increased capacity1 of education stakeholders in target
districts to use1 ICT in education management and governance
DBE1 Intermediate Result 3-25: Number of grants awarded to district governments in
collaboration with the private sector to develop and implement activities that increase use of
ICT and are suitable of wider application
DBE1 Intermediate Result 3-26: Number of grants awarded to district public institutions in
collaboration with the private sector to develop and implement ”education hotspots” that have
sustainable business plans and are suitable of wider application.
II. DBE1 – PROPOSED RESULTS FRAMEWORK Below are DBE1 Sub-Intermediate Results (SIR) that are being proposed to differentiate
critical aspects of Indicator 3-24 for the purpose of the ICT Grants Program:
SIR 3-24.1: Increased capacity for ICT use among education stakeholders
SIR 3-24.2: Increased access to education information
SIR 3-24.3: Increased access to ICT
The following are proposed indicators to track progress in achieving each of the three
proposed SIRs.
1 We understand the capacity to “use” ICT as a function of having access to ICT infrastructure, the
skills to make use of the infrastructure and access to appropriate content (education information).
-
16
III. SIR 3-24.1: INCREASED CAPACITY FOR ICT UE AMONG EDUCATION STAKEHOLDERS –
INDICATORS
A. ICT-RELATED TRAINING
The indicators below are for those grantees that organize ICT-related training in some form for a variety/specific group of education stakeholders.
# Indicator Indicator Definition Data Collection Method
Method of Calculation Data Proof Who How often
When
Data reported to DBE1
By whom
How often When Format
3-24.1.a
# of education stakeholders attending an ICT-related training course
“Education stakeholders” are defined as members of the following 5 user groups: education administrators (from diknas, Kandepag, or DEB), school administrators (school principal, vice-principal, other school administration staff), teachers, students, and parents.
a. Capture number of trainees attending at least one training session, disaggregated by user groups
Count number of people that attended the training.
a. Training participants log
e.g. Training Manager
based on training schedule/ work plan
after every training session
e.g. Project Manager
Quarterly (or by progress reporting period) if a training had been started in that quarter
Based on start of training program
Progress Report including summary on the training program carried out during the last reporting period and a summary on the indicator calculation, including copies of the training manuals and participant lists.
“ICT-related” training is any training that teaches basic computer or Internet skills, computer networking, servicing, and maintenance skills, electronic resource production and evaluation skills or general data entry and management skills.
Disaggregate by gender and user group2
A “training course” is considered a course that constitutes at least 4 contact hours.
2 For the purpose of this M&E framework, 5 different user groups have been identified and are made up of different education stakeholders, that is, education administrators
(diknas [incl. DPL staff], depag, DEB) are one user group, school administrators (principal, vice-principal, other school admin staff) are another, teachers, students and
parents each are separate user groups as well.
-
17
# Indicator Indicator Definition Data Collection Method
Method of Calculation Data Proof Who How often
When
Data reported to DBE1
By whom
How often When Format
3-24.1.b
% of education stakeholders successfully completing an ICT-related training course
“Education stakeholders” are defined as members of the following 5 user groups: education administrators (from diknas, Kandepag, or DEB), school administrators (school principal, vice-principal, other school administration staff), teachers, students, and parents.
a. Record training participation at every training session.
Divide number of education stakeholders that successfully completed the course, by number of education stakeholders that signed up for the course. Move decimal two places to the right and add a percentage sign (e.g. 15 of 75 training participants signed up for the course AND passed the exam: 15/75=0.2 or 20%).
a. Participant training log
e.g. Training Manager
a. based on training schedule
a. after every training session
e.g. Project Manager
Quarterly (or by progress reporting period) if a training had been completed in that quarter
Based on start of training program
Progress Report including summary on training program carried through and outcome of the indicator calculation, in the last reporting period, with copies of training logs, and testparticipation logs
“Successful” completion entails regular attendance and passing of an end-of-course test (either internationally, nationally, or institutionally standardized)
b. Record names of training participants that sit for the test and pass the test
Disaggregate by gender and user group3
b. Test participation log
b. once b. after every course
“ICT-related” training is any training that teaches basic computer or Internet skills, computer networking, servicing, and maintenance skills, electronic resource production and evaluation skills or general data entry and management skills.
A “training course” is considered a course that constitutes at least 4 contact hours.
3 For the purpose of this M&E framework, 5 different user groups have been identified and are made up of different education stakeholders, that is, education administrators
(diknas [incl. DPL staff], depag, DEB) are one user group, school administrators (principal, vice-principal, other school admin staff) are another, teachers, students and
parents each are separate user groups as well.
-
18
# Indicator Indicator Definition Data Collection Method
Method of Calculation Data Proof Who How often
When
Data reported to DBE1
By whom
How often When Format
3-24.1.c
% of education stakeholders continuing on to another ICT-related training course.
This indicator captures the percentage of education stakeholders that successfully completed a paid or free ICT-related training course of at least 4 contact hours, and are taking an additional paid or free ICT-related training course of at least 4 contact hours.
Compare log of trainees that start a new training course with the test log from previous courses.
Divide number of education stakeholders continuing with another training course, by number of education stakeholders successfully completing training. Move decimal two places to the right and add a percentage sign (e.g. 85 of 280 training participants continue on to another training: 85/280=0.31 or 31%)4
a. Training logs
e.g. Training Manager
once At the beginning of each training course.
e.g. Project Manager
Quarterly (or by progress reporting period)
Based on start of training program
Progress Report including summary on training program carried through during last reporting period, summary on indicator calculation, accompanied by copies of training logs, and test participation logs
“Education stakeholders” are defined as members of the following 5 user groups: education administrators (from diknas, Kandepag, or DEB), school administrators (school principal, vice-principal, other school administration staff), teachers, students, and parents.
Disaggregate by gender and user group2
b. Test logs
“ICT-related” training is any training that teaches basic computer or Internet skills, computer networking, servicing, and maintenance skills, electronic resource production and evaluation skills or general data entry and management skills.
A “training course” is considered a course that constitutes at least 4 contact hours.
4 Rounded for 2 decimals
-
19
# Indicator Indicator Definition Data Collection Method
Method of Calculation Data Proof Who How often
When
Data reported to DBE1
By whom
How often When Format
3-24.1.d
% of stakeholders reporting to have medium or advanced level computer and(/or) Internet skills
This indicator tracks education stakeholders that respond to a survey, indicating that they have at least medium level, if not advanced level computer and/or Internet skills. The survey will provide adequate explanation on the skills each level entails to facilitate the self-assessment5
Carry out a separate baseline and a training impact evaluation, e.g., by providing questionnaires to a representative sample of education stakeholders.
Divide the number of education stakeholders that were surveyed and that responded to this question, indicating at least medium or advanced level computer and Internet skills, by the total number of education stakeholders that responded to this question. Move the decimal two places to the right and add a percentage sign (e.g. 65 of 250 respondents report at least medium level computer and Internet skills: 65/250=0.26 or 26%).
Filled-in questionnaires, and summary on data analysis; focus group guides and notes from focus group discussions if applicable
e.g., Project Manager
Twice Once at project inception, once at the end of the grants program
e.g .Project Manager
twice Based on grant implementation schedule
First progress report and final report with summary on evaluation procedure and instruments, summary on baseline results, disaggregated by user group and gender, accompanied by copies of original filled-in questionnaires from education stakeholders, and a summary on each focus group discussion if applicable.
“Education stakeholders” are defined as members of the following 5 user groups: education administrators (from diknas, Kandepag, or DEB), school administrators (school principal, vice-principal, other school administration staff), teachers, students, and parents.
Questionnaire-based surveying may be complemented with focus groups with different education stakeholders to find out in more detail about their current ICT use, skills, knowledge and needs.
Disaggregate by gender and user group2
5 We still need to define the specific criteria defining basic, medium and advanced level computer and Internet skills. Maybe Tita has something that is used in an Indonesian
context already, if not, I may have some international examples
-
20
# Indicator Indicator Definition Data Collection Method
Method of Calculation Data Proof Who How often
When
Data reported to DBE1
By whom
How often When Format
3-24.1.e
% of education stakeholders reporting that the training has provided them with critical skills for their work or studies
This indicator tracks the impact of the training in terms of education stakeholders’ work or studies. It tracks education stakeholders responding to a questionnaire and indicating that the ICT-related training they received was “very critical” or “somewhat critical” for their work and studies.
Capture data with the help of a self-assessment questionnaire given to a representative sample of education stakeholders that participated in an ICT-related training, conducted as part of the grants program.
Divide the number of education stakeholders that were surveyed and that responded to this question, indicating that the training has provided them with “very critical”, or “somewhat critical” skills for their work and studies, by the total number of education stakeholders that responded to this question. Move the decimal two places to the right and add a percentage sign (e.g. 160 of 250 respondents report that the training has provided them with critical skills: 160/250=0.64 or 64%).
Filled-in questionnaires and summary on data analysis; focus group guides and notes from focus group discussion if applicable.
e.g., Project Manager
once Once at the end of the grants program
e.g., Project Manager
once Based on grant implementation schedule
Final report with summary on evaluation procedure and instruments, summary on evaluation results, disaggregated by user group and gender, accompanied by copies of original filled-in questionnaires from education stakeholders, and a summary on each focus group discussion if applicable.
“Education stakeholders” are defined as members of the following 5 user groups: education administrators (from diknas, Kandepag, or DEB), school administrators (school principal, vice-principal, other school administration staff), teachers, students, and parents.
Questionnaire-based surveying may be complemented with focus groups with different education stakeholders to find out in more detail about their current ICT use, skills, knowledge and needs.
Analyze open ended question on what skills and how they are employed applying qualitative data analysis methods.
As part of the evaluation, inquire from education stakeholders which specific skills they learned are critical and have changed their every day work or studies.
Disaggregate by gender and user group2
-
21
# Indicator Indicator Definition Data Collection Method
Method of Calculation Data Proof Who How often
When
Data reported to DBE1
By whom
How often When Format
3-24.1.f
% of education stakeholders reporting that after the training, they have used ICT more frequently in their work than before
“Education stakeholders” are defined as members of the following 5 user groups: education administrators (from diknas, Kandepag, or DEB), school administrators (school principal, vice-principal, other school administration staff), teachers, students, and parents.
Carry out a separate baseline and a training impact evaluation, e.g., providing questionnaires to a sample of education stakeholders.
Divide the number of education stakeholders that were surveyed and that responded to this question, indicating that they use ICT more often for their work or studies than before their participation in grantee-organized training, by the total number of education stakeholders that responded to this question. Move the decimal two places to the right and add a percentage sign (e.g. 134 of 250 respondents report using ICT regularly: 134/250=0.54 or 54%)6
Filled-in questionnaires, and summary on data analysis; focus group guides and notes from focus group discussion if applicable.
e.g., Project Manager
Once Onceat the end of the grants program
e.g, Project Manager
Once Based on grant implementation schedule
Final report with summary on evaluation procedure and instruments, summary on baseline results, disaggregated by user group and gender, accompanied by copies of original filled-in questionnaires from education stakeholders, and a summary on each focus group discussion if applicable.
“Use” is defined as using computers for any work-related activity, using the Intranet or Internet, or using any education information system or portal for work or studies.
Questionnaire-based surveying may be complemented with focus groups with different education stakeholders to find out in more detail about their current ICT use, skills, knowledge and needs.
Disaggregate by gender and user group7
6 Rounded for two decimals
7 For the purpose of this M&E framework, 5 different user groups have been identified and are made up of different education stakeholders, that is, education administrators
(diknas [incl. DPL staff], depag, DEB) are one user group, school administrators (principal, vice-principal, other school admin staff) are another, teachers, students and
parents each are separate user groups as well
-
22
B. ELECTRONIC EDUCATION RESOURCES
The proposed indicators below are for grantees that build capacity among education stakeholders to develop and evaluate electronic education resources (and in parallel build
general computer skills among process participants and improve collaboration).
# Indicator Indicator Definition Data Collection
Method Method of Calculation Data Proof Who
How often
When
Data reported to DBE1
By whom
How often
When Format
3-24.1.g
# of education institutions (schools and/or DEO/Kandepag/DEB sections) involved in the e-resources development process
“Education institutions” are the institutions from which education stakeholders come, that is: diknas, depag, DEB, DPL, universities, schools, school committees, district public libraries, teacher unions, etc.
Make inventory of people that are engaged in the resource development process, track name and education institution affiliation of each person
Count the number of different institutions represented in the resource development process.
List of participants that explicitly highlights their position and institutional affiliation;
signed participant’s list
e.g., Project Manager
once Based on project work plan, e.g. within month 1 of the project inception
e.g. Project Manager
once Based on project work plan, e.g. within 30 days of grant award
Initial or first process report with summary on participants of the resource development process
3-24.1.h
# of resource development meetings held
This indicator tracks the number of meetings held during resource development.
Collect meeting invitations and meeting participant lists
Count number of meetings held within one reporting period.
Meeting invitations; signed participant’s lists
e.g. Project Manager
Regularly by reporting period
Based on project work plan
e.g. Project Manager
Quarterly (or by progress reporting period)
Based on grant implementation schedule
Initial report,, and each progress report and final report, with summary on activities carried out in the last quarter, highlight number of resource development meetings held; copies of signed meeting participants lists
3-24.1.i
% of resource development meetings attended by members of
Trying to assess the level of activity of diknas and depag staff in the resource development process, this indicator intents to track the percentage of resource development
Ensure that at every meeting participants sign a participant list, including
Divide number of meetings at which all depag and diknas representatives, who are involved in the process, were present by the total number of
Signed participant’s lists after every meeting
e.g., Project Manager
Regularly by reporting period
Based on project work plan
e.g. Project Manager
Quarterly (or by progress reporting period)
Based on grant implementation schedule
Progress reports and final report with summary of meetings held
-
23
# Indicator Indicator Definition Data Collection
Method Method of Calculation Data Proof Who
How often
When
Data reported to DBE1
By whom
How often
When Format
all institutions involved
group meetings, at which all depag, diknas and DPK participants were in attendance.
providing their name, position and affiliation with an education institution, such as diknas, depag, university, etc.
resource development group meetings held in the last quarter.
Move the decimal two places to the right and add a percentage sign (e.g. at 3 of 6 meetings, all participants that are part of the process were present: 3/6 = 0.5 or 50%).
and at what date during the last reporting period, accompanied with copies of the originally signed participant lists
3-24.1.j
% of participants in the resource development process reporting to have acquired critical skills and knowledge for their work
This indicator tracks the extent to which being involved in the resource development process has increased skills and knowledge among participants. In detail, it will track what percentage of the participants explicitly express that involvement in the project has provided them with “very critical “ or “somewhat critical” skills and knowledge for their work.
Conduct a survey and/or interviews among the resource development participants
Divide the number of resource development participants that expressed having acquired “very critical” or “critical” skills in the process, by the total number of participants involved in the process.
Disaggregate by user group8 and gender
Survey questionnaires; interview guides and notes
e.g. Project Manager
once Based on project work plan, at the end of the grants program
e.g. Project Manager
once Based on grant implementation schedule
Final report with summary on survey outcomes on skills, knowledge and attitude of resource development participants; with copies of original, filled-in questionnaires.
3-24.1.k
# of education institutions to which final resources have been distributed
This indicator tracks the extent to which the e-resource developed have been distributed to other education institutions.
“Education institutions” are the institutions from which education stakeholders come, that is: diknas, depag, DEB, DPL, universities, schools, school committees, district public libraries, teacher unions, etc.
Track which institutions the resources have been shared with, or sent to.
Count the number of institutions.
Tracking list of resource package recipients
e.g. Project Manager
Once Based on project work plan, at the end of the grants program
e.g. Project Manager
once Based on grant implementation schedule
Final report with list of institutions that have received the resource package(s).
8 For the purpose of this M&E framework, 5 different user groups have been identified and are made up of different education stakeholders, that is, education administrators (diknas [incl. DPL
staff], depag, DEB) are one user group, school administrators (principal, vice-principal, other school admin staff) are another, teachers, students and parents each are separate user groups as
well.
-
24
# Indicator Indicator Definition Data Collection
Method Method of Calculation Data Proof Who
How often
When
Data reported to DBE1
By whom
How often
When Format
3-24.1.l
% of respondents in education institutions having received the resources reporting them useful for their teaching or self-study
This indicator tracks how users value the resources for their teaching or self-study.
“Education institutions” are the institutions from which education stakeholders come, that is: diknas, depag, DEB, DPL, universities, schools, school committees, district public libraries, teacher unions, etc.
Conduct questionnaire survey among members of those education institutions that have received the resources.
Questionnaire-based surveying may be complemented with focus groups
Divide the number of respondents that were surveyed and that responded to this question, indicating that the they find the resources “very valuable”, or “valuable” for their teaching or self-study, by the total number of respondents to this question. Move the decimal two places to the right and add a percentage sign (e.g. 160 of 250 respondents report that the resources are useful: 160/250=0.64 or 64%).
Survey questionnaires; focus group guides and notes if applicable
e.g. Project Manager
Once Based on project work plan, at the end of the grants program
e.g. Project Manager
once Based on grant implementation schedule
Final report with summary on evaluation procedure and instruments, summary on evaluation results, disaggregated by user group and gender, accompanied by copies of original filled-in questionnaires from education stakeholders, and a summary on each focus group discussion if applicable.
-
25
IV. SIR 3-24.2: INCREASED ACCESS TO EDUCATION INFORMATION – INDICATORS
A. EDUCATION PORTAL/SYSTEM DEVELOPMENT
The proposed indicators below are for grantees that develop a publicly accessible electronic management information system, such as a web-
based education information system or library research system, SMS-supported education or library information system/gateway/center, or an
education portal of some kind.
# Indicator Indicator Definition Data Collection
Method Method of calculation Data Proof Who
How often
When
Data reported to DBE1
By whom
How often When Format
3-24.2.a
# of times public system /portal has been accessed
This indicator tracks the frequency of system/portal use on a daily basis.
“Accessed” is defined by a registered user logging in to the system/portal, or, in an open system/portal, by a successful request to the server.
Use of system /portal tracking statistics software. Where use statistics should disaggregate by user group and gender, if individual user profiles are being tracked through registration before use.
Based on system/portal , statistical analysis on use, extract the number of times the system has been access per day.
Calculate average of daily access per reporting period.
Print-out, summary graphics of the statistics on system/portal use
e.g. Project Manager
Tracked daily
End of every day
e.g. Project Manager
Every reporting period, starting with launch of system
Based on grant implementation schedule
Progress report with summary on system statistics, e.g. via graphics, charts or statistical tables
3-24.2.b
% of education stakeholders reporting use of the system/portal
This indicator tracks the percentage of education stakeholders who respond to a survey and who report using the system/portal.
“Education stakeholders” are defined as members of the following 5 user groups: education administrators (from diknas, Kandepag, or DEB), school administrators (school principal, vice-principal, other school administration staff), teachers, students, and parents.
“Use of the system/portal” is defined as accessing the system/portal (
Survey questionnaire send to a sample of education stakeholders (representative of every user group).
Divide the number of education stakeholders that state that they use the system/portal, by the total number of education stakeholders that responded to this question. Move the decimal two places to the right and add a percentage sign (e.g. 20 of 50 respondents use the system: 20/50=0.4 or 40%).
Report on frequency of use by asking respondents that report
Actual surveys (questionnaire) sent back by users and analyzed
e.g., Project Manager
Once per year
Once after 6 months of system/ portal existence, from then on every 12 months
e.g. Project Manager
Twice or three times based on grant timeline
Based on grant implementation schedule
Progress and/or Final report including an analysis of the survey, and samples of filled-in survey questionnaire
-
26
# Indicator Indicator Definition Data Collection
Method Method of calculation Data Proof Who
How often
When
Data reported to DBE1
By whom
How often When Format
online or offline) for at least 5 minutes to either acquire or upload education information of any kind
using the system to indicate at what level of frequency (once, daily, weekly, bi-weekly, monthly, etc.)
Disaggregate by user group9 and gender.
3-24.2.c
% of education stakeholders reporting that the system/portal as increased ease of access to education information
This indicators tracks the percentage of education stakeholders who respond to a survey and report that the system as has made it easier for them to access education information.
“Education stakeholders” are defined as members of the following 5 user groups: education administrators (from diknas, Kandepag, or DEB), school administrators (school principal, vice-principal, other school administration staff), teachers, students, and parents.
Survey, questionnaire
Divide the number of education stakeholders that respond that one of the benefits of the system/portal is “easier access to education information”, by the total number of education stakeholders that responded to this question. Move the decimal two places to the right and add a percentage sign (e.g. 30 of 50 respondents report that the system/portal increased ease of access to information: 30/50=0.6 or 60%).
Disaggregate by user group10 and gender.
Actual surveys (questionnaire) sent back by users and analyzed
e.g Project Manager
Once per year
Once after 6 months of system/ portal existence, from then on every 12 months
e.g. Project Manager
Twice or three times based on grant timeline
Based on grant implementation schedule
Progress and/or Final report including an analysis of the survey, and samples of filled-in survey questionnaire
3-24.2.d
% of education administrators (DEO, Kandepag,
This indicator tracks the percent of education administrators that report the system/portal having improved communication among each other.
“Education Administrators” are
Survey, questionnaire
Divide the number of education stakeholders that respond that one of the benefits of the system/portal is “improved
Actual surveys (questionnaire) sent back by
e.g Project Manager
Once per year
Once after 6 months of system/
e.g. Project Manager
Twice or three times based on grant
Based on grant implementation schedule
Progress and/or Final report including an analysis of the survey, and
9 For the purpose of this M&E framework, 5 different user groups have been identified and are made up of different education stakeholders, that is, education administrators (diknas [incl.
DPL staff], depag, DEB) are one user group, school administrators (principal, vice-principal, other school admin staff) are another, teachers, students and parents each are separate user
groups as well. 10
For the purpose of this M&E framework, 5 different user groups have been identified and are made up of different education stakeholders, that is, education administrators (diknas [incl. DPL staff], depag, DEB) are one user group, school administrators (principal, vice-principal, other school admin staff) are another, teachers, students and parents each are separate user
groups as well.
-
27
# Indicator Indicator Definition Data Collection
Method Method of calculation Data Proof Who
How often
When
Data reported to DBE1
By whom
How often When Format
DEB) reporting that the system/portal has improved communication among each other
representatives from the following institutions: DEO, Kandepag, and District Education Board (DEB).
communication”, by the total number of education stakeholders that responded to this question. Move the decimal two places to the right and add a percentage sign (e.g. 30 of 50 respondents report that the system/portal improved communication: 30/50=0.6 or 60%).
Disaggregate by gender.
users and analyzed
portal existence, from then on every 12 months
timeline
samples of filled-in survey questionnaire
3-24.2.e
% of education administrators (DEP, Kandepag, DEB) reporting that the system/portal has improved time-efficiency in their work
This indicator tracks the percent of education administrators that report the system/portal having improved time-efficiency in their work.
“Education Administrators” are representatives from the following institutions: DEO, Kandepag, and District Education Board (DEB).
Survey, questionnaire
Divide the number of education stakeholders that respond that one of the benefits of the system/portal is “improved time-efficiency”, by the total number of education stakeholders that responded to this question. Move the decimal two places to the right and add a percentage sign (e.g. 30 of 50 respondents report that the system/portal improved time-efficiency: 30/50=0.6 or 60%).
Disaggregate by gender.
Actual surveys (questionnaire) sent back by users and analyzed
e.g. Project Manager
Once per year
Once after 6 months of system/ portal existence, from then on every 12 months
e.g. Project Manager
Twice or three times based on grant timeline
Based on grant implementation schedule
Progress and/or Final report including an analysis of the survey, and samples of filled-in survey questionnaire
3-24.2.f
% of education administrators (DEP, Kandepag, DEB) reporting that the system/portal has facilitated reporting
This indicator tracks the percent of education administrators that report the system/portal has facilitated reporting of education data
“Education Administrators” are representatives from the following institutions: DEO, Kandepag, and District Education Board (DEB).
Survey, questionnaire
Divide the number of education stakeholders that respond that one of the benefits of the system/portal is “facilitated reporting of education data”, by the total number of education stakeholders that responded to this question. Move the decimal two places to the right and add a percentage
Actual surveys (questionnaire) sent back by users and analyzed
e.g. Project Manager
Once per year
Once after 6 months of system/ portal existence, from then on every 12
e.g. Project Manager
Twice or three times based on grant timeline
Based on grant implementation schedule
Progress and/or Final report including an analysis of the survey, and samples of filled-in survey questionnaire
-
28
# Indicator Indicator Definition Data Collection
Method Method of calculation Data Proof Who
How often
When
Data reported to DBE1
By whom
How often When Format
education data
sign (e.g. 30 of 50 respondents report that the system/portal facilitated education data reporting: 30/50=0.6 or 60%).
Disaggregate by gender.
months
B. INTERNET CONNECTIVITY/LOCAL AREA NETWORK
The proposed indicators below are for grantees that focus activities on connecting education stakeholders either via Internet connectivity or
Local Area Networks (IT centers) of some kind.
# Indicator Indicator Definition
Data Collection Method
Method of Calculation Data Proof Who How often
When
Data reported to DBE1 By
whom How often When Format
3-24.2.g
% of DEO and Kandepag sections and/or sub-district offices and/or schools connected
This indicator captures the percentage of DEO and Kandepag sections and/or sub-district offices and/or schools that have been connected under this grants program.
“connected” defines a functioning, dedicated internet or intranet (LAN, WAN) connection between two or more sections/offices/schools
Conduct site observations or a short survey.
Survey should capture whether the section/offices/school had been connected as part of the grants program or had previously been connected.
Divide the number of sections/offices/schools that are connected (either by the DBE grants program or otherwise) by the total number of sections/offices/schools in the district.
Move the decimal two places to the right and add a percentage sign (e.g. 31 of 96 sections/offices/schools are connected: 31/96=0.32 or 32%)11.
Site observation notes or survey questionnaire
e.g. Project Manager
once Based on project work plan, after connectivity has been established
Project Manager
once Based on grant implementation schedule, after connectivity has been established, e.g. in quarterly report July 07
Progress or final report including summary of survey analysis, sample of original questionnaires filled out or copies of original site observation notes.
3-24.2.h
% of education stakeholders reporting to exchange
This indicator tracks the percentage of education stakeholders that report to use the network/connectivity established by the DBE1 grantee, and exchange information with other
Survey, questionnaire
Divide the number of education stakeholders that respond that they use the connectivity/network for information exchange with
Actual surveys (questionnaire) sent back by
e.g. Project Manager
Once per year
Once after 6 months of system/
e.g. Project Manager
Twice or three times based on grant
Based on grant implementation schedule
Progress and/or Final report including an analysis of the
11
Rounded for two decimals.
-
29
# Indicator Indicator Definition
Data Collection Method
Method of Calculation Data Proof Who How often
When
Data reported to DBE1 By
whom How often When Format
information with other education stakeholders
education stakeholders.
“Education stakeholders” are defined as members of the following 5 user groups: education administrators (from diknas, Kandepag, or DEB), school administrators (school principal, vice-principal, other school administration staff), teachers, students, and parents.
“Exchanging information” include activities, such as sending an email, sharing files, saving a document in a shared folder, accessing documents and files from somebody else in shared folders.
other education stakeholders by the total number of education stakeholders that responded to this question. Move the decimal two places to the right and add a percentage sign (e.g. 30 of 50 respondents report that they use the connectivity/ network for information exchange : 30/50=0.6 or 60%).
Report on frequency of information exchange by asking respondents that report using the connectivity/network for information exchange to indicate at what level of frequency (once, daily, weekly, bi-weekly, monthly, etc.)
Disaggregate by gender and user group12.
users and analyzed
portal existence, from then on every 12 months
timeline
survey, and samples of filled-in survey questionnaire
3-24.2.i
% of education stakeholders reporting easy access to critical information
This indicator tracks the percentage of education stakeholders who respond to a survey and report that the connectivity/network established by the DBE1 grantee as has made it easier for them to access critical information.
“Education stakeholders” are defined as members of the following 5 user groups: education administrators (from diknas, Kandepag, or DEB), school administrators (school principal, vice-principal, other school administration staff), teachers,
Survey, questionnaire
Divide the number of education stakeholders that respond that one of the benefits of the connectivity/network is “easier access to critical information”, by the total number of education stakeholders that responded to this question. Move the decimal two places to the right and add a percentage sign (e.g. 30 of 50 respondents report that the connectivity/network increased
Actual surveys (questionnaire) sent back by users and analyzed
e.g. Project Manager
Once per year
Once after 6 months of system/ portal existence, from then on every 12 months
e.g. Project Manager
Twice or three times based on grant timeline
Based on grant implementation schedule
Progress and/or Final report including an analysis of the survey, and samples of filled-in survey questionnaire
12
For the purpose of this M&E framework, 5 different user groups have been identified and are made up of different education stakeholders, that is, education administrators (diknas [incl. DPL staff], depag, DEB) are one user group, school administrators (principal, vice-principal, other school admin staff) are another, teachers, students and parents each are separate user groups as
well.
-
30
# Indicator Indicator Definition
Data Collection Method
Method of Calculation Data Proof Who How often
When
Data reported to DBE1 By
whom How often When Format
students, and parents.
“critical” information is defined as information relevant to respondents’ work.
ease of access to information: 30/50=0.6 or 60%).
Disaggregate by user group13 and gender.
3-24.2.j
% of education stakeholders reporting easy access to ICT
This indicator tracks the percentage of education stakeholders who respond to a survey and report that the connectivity/network established by the DBE1 grantee as has made it easier for them to access ICT in general.
“Education stakeholders” are defined as members of the following 5 user groups: education administrators (from diknas, Kandepag, or DEB), school administrators (school principal, vice-principal, other school administration staff), teachers, students, and parents.
“critical” information is defined as information relevant to respondents’ work.
Survey, questionnaire
Divide the number of education stakeholders that respond that one of the benefits of the connectivity/network is “easier to access ICT”, by the total number of education stakeholders that responded to this question. Move the decimal two places to the right and add a percentage sign (e.g. 30 of 50 respondents report that the connectivity/network increased ease of access to ICT: 30/50=0.6 or 60%).
Disaggregate by user group14 and gender.
Actual surveys (questionnaire) sent back by users and analyzed
e.g. Project Manager
Once per year
Once after 6 months of system/ portal existence, from then on every 12 months
e.g. Project Manager
Twice or three times based on grant timeline
Based on grant implementation schedule
Progress and/or Final report including an analysis of the survey, and samples of filled-in survey questionnaire
C. LIBRARY INFORMATION SYSTEM
The proposed indicators below are for grantees that develop an electronic Library Information System solely for the purpose of increasing
efficiency of library services, such as membership management, library resource management or library transaction management.
13
For the purpose of this M&E framework, 5 different user groups have been identified and are made up of different education stakeholders, that is, education administrators (diknas [incl. DPL staff], depag, DEB) are one user group, school administrators (principal, vice-principal, other school admin staff) are another, teachers, students and parents each are separate user
groups as well. 14
For the purpose of this M&E framework, 5 different user groups have been identified and are made up of different education stakeholders, that is, education administrators (diknas [incl. DPL staff], depag, DEB) are one user group, school administrators (principal, vice-principal, other school admin staff) are another, teachers, students and parents each are separate user groups as
well.
-
31
# Indicator Indicator Definition Data Collection
Method Method of calculation Data Proof Who
How often
When
Data reported to DBE1
By whom
How often When Format
3-24.2.k
% of library staff reporting use of the system
This indicator tracks the percentage library who respond to a survey and who report using the system.
“Use of the system/” is defined as accessing the system (online or offline) for at least 5 minutes to manage library information of any kind
Survey questionnaire send to all library staff.
Divide the number of library that state that they use the system, by the total number of library staff that responded to this question. Move the decimal two places to the right and add a percentage sign (e.g. 20 of 50 respondents use the system: 20/50=0.4 or 40%).
Report on frequency of use by asking respondents that report using the system to indicate at what level of frequency (once, daily, weekly, bi-weekly, monthly, etc.)
Disaggregate by gender.
Actual surveys (questionnaire) sent back by users and analyzed
e.g., Project Manager
Once per year
Once after 6 months of system/ portal existence, from then on every 12 months
e.g. Project Manager
Twice or three times based on grant timeline
Based on grant implementation schedule
Progress and/or Final report including an analysis of the survey, and samples of filled-in survey questionnaire
3-24.2.l
% of library staff reporting that the system has improved time-efficiency in their work
This indicator tracks the percent of library staff that report the system has having improved time-efficiency in their work.
Survey, questionnaire
Divide the number of library staff that respond that one of the benefits of the system is “improved time-efficiency”, by the total number of library staff that responded to this question. Move the decimal two places to the right and add a percentage sign (e.g. 30 of 50 respondents report that the system/portal improved time-efficiency: 30/50=0.6 or 60%).
Disaggregate by gender.
Actual surveys (questionnaire) sent back by users and analyzed
e.g. Project Manager
Once per year
Once after 6 months of system/ portal existence, from then on every 12 months
e.g. Project Manager
Twice or three times based on grant timeline
Based on grant implementation schedule
Progress and/or Final report including an analysis of the survey, and samples of filled-in survey questionnaire
V. SIR 3-24.3: INCREASED ACCESS TO ICT – INDICATORS
A. INTERNET ACCESS POINT
-
32
The proposed indicators below are for grantees that establish public Internet access points of some kind (Warnet, hotspot, internet café).
# Indicator Indicator Definition Data Collection
Method Method of Calculation Data Proof Who
How often
When
Data reported to DBE1
By whom
How often When Format
3-24.3.a
# of education stakeholders using the public access point
This indicator tracks the number of education stakeholders using the public access point.
“Education stakeholders” are defined as members of the following 5 user groups: education administrators (from diknas, Kandepag, or DEB), school administrators (school principal, vice-principal, other school administration staff), teachers, students, and parents.
“Using” is defined as entering the public access point facilities at least once and purchasing and/or consume a specific service (e.g. Internet access) or attend a training course or workshop.
Track visits of education stakeholders with the help of a daily user log;
Track number of users via user logs.
Calculate average of daily access per reporting period.
Disaggregate data by gender and user group15.
User logs e.g. Project Manager
Tracked daily
End of every day
e.g. Project Manager
Every reporting period, starting with launch of public access point
Based on grant implementation schedule
Progress report with overview on public access point use, e.g. attached user logs or print outs thereof
3-24.3.b
# of education stakeholders reporting access to ICT via the public access point to be critical for their work or studies
This indicators tracks the number of education stakeholder that use the public access point and responding to a questionnaire, indicate, that access to computers and/or the Internet via this access point is “very critical” or “somewhat critical” for their work and studies.
“Education stakeholders” are defined as members of the following 5 user groups: education administrators
Capture data with the help of a questionnaire given to public access point users.
Count number of users that respond to this question and say that they find access to ICT (computer/Internet) via the public access point “very critical” or “somewhat critical” to their work and/or studies.
Disaggregate by user group16 and gender.
Filled-in questionnaire
Project Manager
Twice in year one, then yearly
After month 6 and 9 of operation, then yearly
Project Manager
Twice in year one, then yearly
After month 6 and 9 of operation, then yearly
Progress and final report including an analysis of the survey, and samples of filled-in survey questionnaire
15
For the purpose of this M&E framework, 5 different user groups have been identified and are made up of different education stakeholders, that is, education administrators (diknas [incl. DPL staff], depag, DEB) are one user group, school administrators (principal, vice-principal, other school admin staff) are another, teachers, students and parents each are separate user
groups as well. 16
For the purpose of this M&E framework, 5 different user groups have been identified and are made up of different education stakeholders, that is, education administrators (diknas [incl. DPL staff], depag, DEB) are one user group, school administrators (principal, vice-principal, other school admin staff) are another, teachers, students and parents each are separate user
groups as well.
-
33
# Indicator Indicator Definition Data Collection
Method Method of Calculation Data Proof Who
How often
When
Data reported to DBE1
By whom
How often When Format
(from diknas, Kandepag, or DEB), school administrators (school principal, vice-principal, other school administration staff), teachers, students, and parents.
3-24.3.c
% of months in operation, where revenue exceeds ongoing costs
This indicator requires that the monthly revenue exceeds ongoing costs.
“Monthly revenue” includes all income generated by services offered at the public access point, such as internet access, training courses, information research services, printing, etc.
“Ongoing costs” include all costs required to keep the public access point running, including salaries, rents, communication expenses, stationary expenses, minor repair expense on both, building and equipment, all applicable taxes and fees.
Diligently manage accounting records and do a monthly analysis of ongoing costs and revenue cash flows.
Count the number of months, in the last quarter, in which monthly revenue has exceeded on-going costs.
Accounting records
e.g., Project Manager
Monthly After every calendar month
e.g., Project manager
Quarterly (or by reporting period)
Based on grant implementation schedule
Progress and Final Report including detailed overview on on-going costs and revenue generated each month in the past reporting period from selling services such as connectivity, bandwidth, training, computer repair, computer equipment trading, communication services for non-(computer) literate community
-
34
APPENDIX 3
ICT Grants Program Indicators
and Customized Grantee
Indicator Tracking Sheets
-
35
ICT GRANT PROGRAM INDICATOR
KEGIATAN INDIKATOR PETUNJUK
PUBLIC INTERNET ACCESS POINT (Hotspot/Warnet/
Internet Café)
# stakeholder pendidikan yang menggunakan hotspot Mohon didata berapa orang stakeholder pendidikan (pegawai administrasi sekolah/kec/kab/kota, guru, murid, orangtua, dsb) yang menggunakan hotspot/Warnet/Internet Café dalam satu bulan
# stakeholder pendidikan yang yang menyatakan bahwa akses ICT sangat penting bagi pekerjaan atau pendidikan mereka
Dalam periode satu bulan, edarkan questioner secara sederhana kepada stakekholder pendidikan, apakah akses terhadap hotpost/warnet/internet café tersebut (1) tidak penting; (2) kurang penting; (3) penting; (4) sangat penting dalam pekerjaan atau pendidikan mereka. Kemudian buatlah rekapitulasi berapa orang atau persen yang menyatakan tidak, cukup, dan sangat penting
% bulan beroperasi dimana pendapatan melebihi biaya berjalan
Cukup dicatat berapa pengeluaran yang diperlukan untuk mengelola hotpsot/warnet/internet café dan berapa pendapatan yang diperoleh setiap bulanya. Catat mulai bulan keberapa pendapatan melebihi pengeluaran
MIS (LIBRARY, EMIS, SMS Gateway or Education
Portal)
Berapa kali dalam satu bulan public system/portal diakses
Untuk mengisi data ini, cukup didata setiap hari orang-orang yang mengakses portal dan buatlah rekapitulasi dalam satu bulan
% stakeholder pendidikan yang melaporkan penggunaan system/portal
Secara acak (bukan hanya pengguna saja) lakukan wawancara dengan stakeholder pendidikan (pegawai administrasi sekolah/kec/kab/kota, guru, murid, orangtua, dsb) masing-masing sebanyak sebanyak 2) orang, dan buat rekapitulasi berapa di antara mereka yang menggunakan portal yang didanai oleh DBE1.
% stakeholder pendidikan yang melaporkan bahwa system/portal memudahkan akses informasi pendidikan
Dalam tempo satu bulan, edarkan questioner secara sederhana kepada setiap pengunjung, apakah keuntungan dengan adanya system/portal: (1) tidak ada pengaruhnya, (2) tidak tersedia alat bantu, (3) memudahkan; (4) sangat memudahkan dalam mengakses informasi pendidikan.
% administrator pendidikan yang menyatakan bahwa system/portal telah memperbaiki komunikasi diantara mereka
Mohon ditanyakan kepada (masing-masing 2 orang) staf Dinas Pendidikan, perpustakaan daerah, kandepag, dewan pendidikan. Bappeda, kesra pertanyaan sbb: Dibandingkan sebelum ada portal yang didanai DBE1, apakah komunikasi di antara mereka saat ini (1) menjadi lebih baik; (2) tetap sama/tidak ada perubahan.
% administrator pendidikan yang menyatakan bahwa system/portal telah membuat waktu lebih efisien
Mohon ditanyakan kepada (masing-masing 2 orang) staf Dinas Pendidikan, perpustakaan daerah, kandepag, dewan pendidikan. Bappeda, kesra pertanyaan sbb: Dibandingkan sebelum ada portal yang didanai DBE1, apakah keuntungan yang didapat? (1) waktu menjadi efisien; (2) komunikasi dengan institusi lain menjadi efektif; (3) lebih sering mengakses system/portal dibanding sebelumnya
-
36
KEGIATAN INDIKATOR PETUNJUK
% administrator pendidikan yang melaporkan bahwa system/portal telah menfasilitasi mereka dengan data pendidikan
Mohon ditanyakan kepada (masing-masing 2 orang) staf Dinas Pendidikan, perpustakaan daerah, kandepag, dewan pendidikan. Bappeda, kesra pertanyaan sbb: Apakah system/portal telah menfasilitasi mereka dengan data pendidikan? (1) Ya; (2) Tidak. Kemudian buatlah rekapitulasi berapa orang/persen yang menyatakan "tidak" dan berapa "ya"
eRESOURCES (electronic
teaching and learning
materials)
# institusi pendidikan (sekolah, Disdik, Kandepag, Dewan Pendidikan) yang terlibat dalam pengembangan eResources
Mohon dicatat jumlah institusi pendidikan yang terlibat dalam pengembangan eResources
# pertemuan pengembangan eResources Mohon dicatat berapa kali dalam satu bulan pertemuan diadakan
% pertemuan pengembangan eResources yang dihadiri oleh seluruh anggota dari institusi yang terlibat
Mohon dicatat berapa kali dalam satu bulan pertemuan yang dihadiri oleh seluruh institusi pendidikan diadakan
% peserta pengembangan eResources yang melaporkan bahwa mereka telah mendapatkan keterampilan dan pengetahuan yang sangat berharga untuk pekerjaan mereka
Mohon ditanyakan kepada peserta pengembangan eResources pertanyaan sbb: Apakah electronic teaching dan learning material itu meningkatkan ketrampilan dan pengetahuan yang sangat membantu dalam pekerjaan mereka? (1) Ya; (2) tidak. Mohon direkap berapa yang menyatakan "ya" dan berapa yang menyatakan "tidak"
# institusi pendidikan dimana eResources yang final telah didistribusikan
Mohon dicatat jumlah institusi pendidikan yang telah mendapatkan eResources yang telah final (sekolah, dinas pendidikan, Dewan pendidikan, dsb.)
% responden dari institusi pendidikan yang telah memperoleh eResources yang menyatakan bahwa eResources tersebut sangat bermanfaat untuk mengajar atau belajar sendiri
Mohon ditanyakan kepada anggota institusi pendidikan (minimal masing-masing 2 orang) pertanyaan sbb: Apakah eResources tersebut bermanfaat bagi proses belajar mengajar? (1) Ya; (2) Tidak. Rekap jawaban responden dan nyatakan berapa yang menyatakan "Ya" (bermanfaat) dan berapa yang "tidak" (tidak bermanfaat.
IT CENTER/ CONNECTIVITY/ NETWORK
% Seksi dari Diknas/UPTD/Kancam atau sekolah yang telah tersambung
Harap didata, dari kabupaten/kota yang bersangkutan, berapa yang telah mempunyai sambungan internet (untuk melihat % sekolah, bandingkan proporsi sekolah yang punya internet dengan yang belum, % seksi dari diknas, bandingkan jumlah seksi yang punya internet dan yang tidak; demikian selanjutnya untuk KCD/UPTD.
% stakeholder pendidikan yg melaporkan terjadinya pertukaran informasi dengan stakeholder pendidikan lainnya
Mohon ditanyakan kepada staf Diknas/ UPTD/KCD/sekolah (masing-masing 2 orang) yang mempunyai jaringan ICT yang didanai oleg DBE1 : Apakah mereka melakukan pertukaran informasi dengan stakeholder lainya? (1) Ya; (2) Tidak. Jika "ya" berapa kali dalam satu bulan?
-
37
KEGIATAN INDIKATOR PETUNJUK
% stakeholder pendidikan yang melaporkan kemudahan akses informasi yang sangat dibutuhkan
Harap ditanyakan kepada staf diknas/UPTD/KCD atau guru (masing-masing 2 orang) pertanyaan sbb: apakah keuntungan dengan adanya jaringan internet bagi saudara? (1) memudahkan akses informasi yang saya butuhkan dalam bekerja; (2) memudahkan pertukaran informasi; (3) memudahkan berubungan dengan institusi lain (Catatan: Jawaban boleh lebih dari 1).
% stakeholder pendidikan yang melaporkan akses ke ICT secara mudah
Harap ditanyakan kepada staf diknas/UPTD/KCD atau guru (masing-masing 2 orang) pertanyaan sbb: Dibandingkan dengan saat sebelum ada jaringan ICT yang dibangun DBE1, bagaimana akses ke ICT pada saat ini? (1) lebih mudah; (2) sama saja.
TRAINING
# stakeholder pendidikan yang menghadiri pelatihan ICT atau program sejenisnya
Mohon didata berapa orang stakeholder pendidikan (pegawai administrasi sekolah/kec/kab/kota, guru, murid, orangtua, dsb) yang mengikuti pelatihan ICT atau sejenisnya. (Bisa dibandingkan sebelum ada bantuan DBE1 untuk baseline dan sesudah ada bantuan DBE1)
% stakeholder pendidikan yang sukses mengikuti kursus ICT atau program sejenisnya
Mohon didata berapa orang stakeholder pendidikan (pegawai administrasi sekolah/kec/kab/kota, guru, murid, orangtua, dsb) yang mengikuti pelatihan ICT atau sejenisnya sampai selesai dan memperoleh sertifikat dan berapa yang tidak selesai/tidak memperoleh sertifikat.
% stakeholder pendidikan yang melanjutkan kursus ICT atau program sejenisnya
Mohon didata berapa orang stakeholder pendidikan (pegawai administrasi sekolah/kec/kab/kota, guru, murid, orangtua, dsb) yang mengikuti pelatihan ICT atau sejenisnya sampai selesai dan berapa orang yang kemudian melanjutkan kursus lain yang sejenis.
% stakeholder pendidikan yag mempunyai ketrampilan komputer/internet tingkat menengah (medium) atau lanjutan (advanced)
Mohon didata berapa orang stakeholder pendidikan (pegawai administrasi sekolah/kec/kab/kota, guru, murid, orangtua, dsb) yang mengikuti pelatihan ICT atau sejenisnya dan apakah mereka bisa mengoperasikan program berikut ini: (1) Word Processing; (2) Spreadsheet; (3) Power point; (4) Browsing internet; (5) internet/emal; (6) download file; (7) SIM. Categori: menguasi 4 program = Medium; 5 atau lebih
program=advanced
% stakeholder pendidikan yang menyatakan bahwa pelatihan telah meningkatkan skill yang amat penting diperlukan dalam pekerjaan atau studi mereka
Secara acak, tanyakan kepada peserta pelatihan (masing-masing stakeholder 2 orang) pertanyaan sbb: Apakah pelatihan yang mereka ikuti berhasil meningkatkan skill yang diperlukan dalam pekerjaan dan pendidikan mereka? (1) Ya; (2) Tidak. Mohon direkapitulasi berapa persen yang menyatakan "ya" dan berapa yang "tidak"
% stakeholder pendidikan yang menyatakan bahwa setelah training, mereka telah menggunakan ICT lebih sering dibandingkan sebelumnya
Secara acak, tanyakan kepada peserta pelatihan (masing-masing stakeholder 2 orang) pertanyaan sbb: Dibandingkan dengan saat sebelum mengikuti pelatihan, seberapa sering sdr menggunakan ICT? (1) Sama; (2) lebih sering. Mohon direkap berapa persen yang mengatakan "sama" dan berapa yang mengatakan "lebih sering"
-
38
KEGIATAN INDIKATOR PETUNJUK
Education Stakeholders: 1. Education Administrators (Administrator Pendidikan),
2. School Administrators
3. Teachers
4. Students
5. Parents
6. others
Administrator Pendidikan: 1. Dinas Pendidikan
2. Perpustakaan Daerah
3. Kandepag
4. Dewan Pendidikan
5. Bappeda
6. Bagian Kesra
-
39
CUSTOMIZED GRANTEE INDICATOR TRACKING SHEET
JENEPONTO
MAIN ACTIVITY I. LAN NETWORK IN KANDEPAG
Pertanyaan-pertanyaan pokok : Apakah ada peningkatan akses terhadap ICT bagi para