inno appraisal template final 071214 empty

8/2/2019 INNO Appraisal Template Final 071214 Empty

1/8page 1 of 8

Information about the Template and the Project

The INNO-Appraisal initiative aims to contribute to the creation, consolidation and strengthening of an evaluation culture inEurope and to learning about good practice in innovation policy making.

INNO-Appraisal takes stock of innovation appraisal exercises such as evaluations and monitoring, benchmarking and peer reviewreports in the area of innovation policy. It seeks to characterise and assess appraisals and utilise appraisal reports for policylearning. The study is funded by the European Commission, DG Enterprise.

This template is the primary instrument for the collection of information and the assessment of individual appraisals ofinnovation policies and programmes. It will be a part of a repository that will be stored on the public PRO INNO Europewebsite. (http://www.proinno-europe.eu/)

We are not interested in appraisals at the individual project level, rather we are restricting our analysis to appraisals at the policymeasure (programme) level. Our starting point for identifying evaluated measures is the INNO-Policy TrendChart innovationpolicy measures database.

We hope to derive lessons for innovation policy formulation and implementation from the appraisal of the specific policymeasure.

The purpose of this template is to characterise and assess one specific appraisal exercise in the context of one particularinnovation policy measure/programme. (Please do not consider the role of other appraisals in relation to this specific innovationpolicy measure.)

The appraisal covered in this template is

The innovation policy measure assessed by this appraisal is

We kindly ask you, as one of the policy-makers or administrators responsible for the innovation policy measure concerned, to helpus complete these templates.

These short instructions are designed to reduce your workload as much as possible:

1. Most of this form has been pre-filled by our project team, relying on information contained in the relevant appraisal report(s);2. Please complete the fields we have not pre-filled;3. Please briefly check our pre-filled answers and correct them if needed;4. If you do correct a prefilled answer, you may comment on your reason in the box provided;5. You can save this questionnaire at any time and continue to complete it at your convenience. To do so, please use the File >> Savemenu in your reader.6. Please return the completed template by attaching it to an e-mail and sending it to the following e-mail address:

Thank you for your co-operation. You will receive a note once the analysis is complete and has been uploadedto the website. Please feel free to make comments or requests to us any time.

The more policy-makers participate,the greater the benefit for the policy-making community in Europe!

Appraisal Template


2/8

page 2 of 8

Part A: Information about the Respondent

A.1.a. First name

A.1.b. Last name

A.1.c. Affiliation and positionPosition, Unit/Department, Institution/Ministry

A.1.e. E-mail address

A.1.d. Address

A.1.f. Telephone number

B.1.b. Country and name of the measure that the appraisalevaluated (Only if the measure is not listed in B.1.a.)

B.1.a. Trendchart code and title of the corresponding policymeasure

Part B: Corresponding Policy Measure

Part C: Characteristics of the Appraisal(while answering this part please consider the primary policy measure indicated in part B)

C.1.a. Who conducted the appraisal?

You can select more than one option by using the ctrl key.

C.1.b. Participating institutions in the appraisal exerciseIf there is a leading institution, please put it first.

C.1.c. E-mail address of the corresponding evaluator

C.2.a. Tender Procedure

C.3.d. Any Comments?

C.3.a. Timing of the appraisal

C.3.b. Purpose of the appraisal

C.4.c. Any Comments?

C.4.a. Was this appraisal a condition of external/international(co)sponsorship (e.g. EU, World Bank, etc.) of the measure?

C.4.b. If your answer to C.4.a. is Yes, please name the sponsor.

Pre-filled by (Office use only)

Sent by (Office use only)

C.3.c. If your answer to C.3.b. is "Other", please specify.


3/8

page 3 of 8

Part C: Characteristics of the Appraisal (continued from previous page)(while answering this part please consider the primary policy measure indicated in part B)

C.5.a. Doesthe policy measure has a dedicated budget forappraisal?

C.6.a. Was theappraisal, to which this template refers, foreseenand planned for during the design phase of the measure?(please tick if your answer is yes)C.6.b. If your answer to C.6.a is no, what prompted the appraisal?


C.7. Topics covered in the appraisal (Questions that the appraisal aims to answer) Covered in the Appraisal

C.7.a. External Consistency (If the measure's goals were appropriate and consistent with the externalchallenges the measure was meant to address)

C.7.b. Internal Consistency (If the measure's modalities were appropriate and consistent with its goals)

C.7.c. Coherence/Complementarity (Whether the measure was coherent with, and complementary to, otherprogrammes and policy initiatives)

C.7.d. Goal Attainment/Effectiveness (If the goals of the measure were achieved)

C.7.e. Outputs, Outcomes and Impacts (The consequences or results of the measure)

C.7.f. Quality of Outputs (Whether the outputs of projects were of a high quality)

C.7.g. Value for Money/Return on Investment/Cost-Benefit Efficiency (If there were adequate returns oninvestment)

C.7.h. Programme Implementation Efficiency (If the measure was well-managed)

C.7.i. Project Implementation Efficiency (If projects supported by the measure were well-managed)

C.7.j. Input Additionality (If the measure stimulated more investment in innovation than would haveoccurred in the absence of the measure)

C.7.k. Output Additionality (If the measure stimulated more innovation outputs than would have occurredin the absence of the measure)

C.7.l. Behavioural Additionality (If the measure stimulated persistent changes in the behaviours of theparticipants that would have not occurred in the absence of the measure)

C.7.m. Policy/Strategy Development (Implications for future strategy development and policy formulation)

C.7.n. Gender issues

C.7.o. Minority issues

C.7.p. Other:

C.7.q. Any Comments?


C.5.b. Cost of the appraisal as a percentage of the budget of themeasure (or in absolute terms in Euro).


4/8

page 4 of 8


C.8.g. Any Comments?

C.8. Did the appraisal look at the following impacts?

C.8.a. Scientific

C.8.b. Technological

C.8.c. Economic

C.8.d. Social

C.8.e. Environmental

C.8.f. Other:

C.9. Sponsor(s) of the appraisal

C.9.a. Programme Owner/Manager

C.9.b. Other Government DepartmentC.9.c. Other Public Bodies (e.g. court of auditors, parliament committee)

C.9.d. Other:

C.9.e. Any Comments?

C.12. Data analysis method employed in the appraisal

C.12.a. Case Study Analysis

C.12.b. Network Analysis

C.12.c. Econometric Analysis

C.12.d. Descriptive Statistics (e.g. uptake analysis)

C.12.e. Input/Output Analysis

C.12.f. Document Analysis

C.12.g. Context Analysis

C.12.h. Before/After Group Comparison Approach

C.12.i. Control Group Approach

C.12.j. Counter-Factual Approach

C.12.k. Cost/Benefit Approach

C.12.l. Other:

C.12.m. Any Comments?

C.10.a. Publication date of the appraisal report

C.10.b. Start date of the period covered by the appraisal

C.10.c. End date of the period covered by the appraisal

C.11.a. Availability of the appraisal report

C.11.c. URL of the Report

C.11.b. Language of the appraisal report


5/8

page 5 of 8


C.13. Data collection methods and data sources employed in the appraisal

C.13.a. Existing Surveys/Databases

C.13.b. Participant Surveys

C.13.c. Non-participant Surveys

C.13.d. Interviews

C.13.e. Focus Groups/Workshops/Meetings

C.13.f. Peer Reviews

C.13.g. Technometrics/Bibliometrics Search

C.13.h. Document Search

C.13.i. Monitoring Data

C.13.j. Other:

C.13.k. Any Comments?

C.14. Main intended audiences for the appraisal

C.14.a. Policy Makers (Politicians)

C.14.b. Policy Makers (Government Officials)

C.14.c. Programme Management

C.14.d. Auditors/Financial Authorities

C.14.e. Those directly supported by the measure

C.14.f. External/International (co)sponsor of the measure/programme

C.14.g. Potential Users of the Measure

C.14.h. Policy Analysts

C.14.i. General Public

C.14.j. Other:

C.14.k. Any Comments?

Part D: Quality of the AppraisalMethods and Data

D.1.a. Were the Terms of Reference available in the appraisal report?

D.1.b. If your answer to D.1.a. is No, were the Terms of Referencemade available elsewhere?D.1.c. If the Terms of Reference were available, did they clearly state the objective of the appraisal?

2

NA/Not Known

5 - Yes, definitely

1 - No, not at all

3

4

D.1.e. Any Comments?D.1.d. If Terms of Reference are given,does the appraisal address them?


6/8

page 6 of 8

D.2.a. Who designed the Terms of Reference for the appraisal?

D.2.b. Were methodologies and approaches specified in the Terms of

Reference? (please tick if applicable)D.3.b. Any Comments?D.3.a. Was/Is the design of the evaluation

appropriate given the objectives of theevaluation and the nature of the policymeasure?

2

NA/Not Known

5 - Yes, definitely

1 - No, not at all

3

4

Part D: Quality of the AppraisalMethods and Data (continued from the previous page)

D.4.b. Any Comments?D.4.a. Did/Do the methods chosen satisfythe Terms of Reference/purpose of theappraisal? 2

NA/Not Known

5 - Yes, definitely

1 - No, not at all

3

4

D.5.b. Any Comments?D.5.a. Was/Is the application of thequalitative methods satisfactory?

2

NA/Not Known

5 - Yes, definitely

1 - No, not at all

3

4

D.6.b. Any Comments?D.6.a. Was/Is the application of thequantitative methods satisfactory?

D.7.b. Any Comments?D.7.a. Were/Are the information sourcesused in the report well documented andreferenced? 2

NA/Not Known

5 - Yes, definitely

1 - No, not at all

3

4

Analysis

D.8.b. Any Comments?D.8.a. Was/Is the analysis clearly basedon the data given?

2

NA/Not Known

5 - Yes, definitely

1 - No, not at all

3

4

D.9.b. Any Comments?D.9.a. Given the objectives of the

appraisal, does the analysis cover thebroader context (e.g. societal, institutional,policy and economic contexts) sufficiently?

2

NA/Not Known

5 - Yes, definitely

1 - No, not at all

3

4

2

NA/Not Known

5 - Yes, definitely

1 - No, not at all

3

4


7/8

page 7 of 8

E.1.b. Any Comments?E.1.a. Were/Are the conclusions based onthe analysis?

Part E: Conclusions and Recommendations

E.2.b. If your answer to E.2.a is yes, please indicate whether they addressed any of thefollowing areas and the extent to which the recommendations were useful Usefulness

E.2.c. Changes to the design of the programme/measure appraised

E.2.d. Changes to themanagement and implementation of the programme/measureappraised

E.2.e. Changes to the design, management and implementation of future programmes/measures

E.2.f. Changes to the design, management and implementation of contemporaneousprogrammes/measures

E.2.g. Changes to broader policy formulation and implementation

E.2.h. Other:

E.2.i. Any Comments?

E.2.a. Did the appraisal contain any recommendations? (please tick if applicable)

F.1.a. Were/Are the analysis andrecommendations of the appraisalwidely discussed within governmentcircles?

2

NA/Not Known

5 - Yes, definitely

1 - No, not at all

3

4

F.2.a. Were/Are the analysis andrecommendations of the appraisalwidely discussed with participantsand broader stakeholders?

2

NA/Not Known

5 - Yes, definitely

1 - No, not at all

3

4

F.3. What were/are the consequences of the appraisal Tick if applicable Please clarify

F.3.a. Termination of the measure

F.3.b. Major re-design of the measure

F.3.c. Minor re-design of the measure

F.3.d. Expansion/Prolongation of the measure

F.3.e. Re-design of another measure

F.3.f. Merger of measures

F.3.g. If there were no consequences, please say why not?

F.3.h. Any Comments?.

Part F: Dissemination and Implementation of Appraisal Results

4

3

1 - No, not at all

5 - Yes, definitely

NA/Not Known

2

F.4.a. What aspects/methods were particularly useful in thisappraisal for both this measure and for general policy making?


8/8

page 8 of 8

End of the Template

Please save this file by using file >> save menu and return to us via e-mail.

Please send it to aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

Thank you for your time.

Part G: General Comments

G.1.a. Please provide any general comments regarding to thismeasure and appraisal.

INNO-Appraisal ProjectAppraisal Template

Final version as of 14 December 2007

inno appraisal template final 071214 empty

Documents