tca venet | project evaluation gerd beidernikl center for education and economy, research &...
TRANSCRIPT
![Page 1: TCA VENet | project evaluation Gerd Beidernikl Center for Education and Economy, Research & Consulting Graz, Austria 1st project meeting 11. September](https://reader031.vdocument.in/reader031/viewer/2022032123/56649b44550346318e8b75fb/html5/thumbnails/1.jpg)
TCA VENet | project evaluation
Gerd Beidernikl
Center for Education and Economy, Research & Consulting
Graz, Austria
1st project meeting
11. September 2005, Cyprus
![Page 2: TCA VENet | project evaluation Gerd Beidernikl Center for Education and Economy, Research & Consulting Graz, Austria 1st project meeting 11. September](https://reader031.vdocument.in/reader031/viewer/2022032123/56649b44550346318e8b75fb/html5/thumbnails/2.jpg)
2
the goals of this session
To introduce myself and the ZBW
To develop a common understanding WHY we are evaluating our work and HOW we are going to do it
To present the (preliminary) evaluation concept and to get feedback on it
To make sure the words and concepts used are familiar to everybody
To develop a common understanding of how to develop criteria that could be used as criteria for success
Overview on the upcoming steps
![Page 3: TCA VENet | project evaluation Gerd Beidernikl Center for Education and Economy, Research & Consulting Graz, Austria 1st project meeting 11. September](https://reader031.vdocument.in/reader031/viewer/2022032123/56649b44550346318e8b75fb/html5/thumbnails/3.jpg)
3
Introduction
![Page 4: TCA VENet | project evaluation Gerd Beidernikl Center for Education and Economy, Research & Consulting Graz, Austria 1st project meeting 11. September](https://reader031.vdocument.in/reader031/viewer/2022032123/56649b44550346318e8b75fb/html5/thumbnails/4.jpg)
4
my background
Profession: Sociology, further training in the field of evaluation and quality management
Researcher in the „Center for Education and Economy“, Graz, Austria
5 full time employees
Public and private clients
My main working areas: labour market research, research on vocational education and training, evaluation, statistics and surveys
Ca. 20 evaluation projects in the last 4 years
![Page 5: TCA VENet | project evaluation Gerd Beidernikl Center for Education and Economy, Research & Consulting Graz, Austria 1st project meeting 11. September](https://reader031.vdocument.in/reader031/viewer/2022032123/56649b44550346318e8b75fb/html5/thumbnails/5.jpg)
5
project references on EU level
Evaluation of the ongoing Leonardo project „inclusive training for disability care workers“
Self-evaluation of the Equal project „girls crack IT“
Evaluation of the Styrian Territorial Employment Pacts
Evaluation of the article VI project „GO BEST“
Project management of an ongoing Leonardo project „online consulting tool for IT professions“
Member of a proposal development group in the 6th framework programme, priority 7
Reasearch activities for the CEDEFOP: reports on the Austrian training system, comperative country studies, administrating the ERO database (European Research Overview)
![Page 6: TCA VENet | project evaluation Gerd Beidernikl Center for Education and Economy, Research & Consulting Graz, Austria 1st project meeting 11. September](https://reader031.vdocument.in/reader031/viewer/2022032123/56649b44550346318e8b75fb/html5/thumbnails/6.jpg)
6
workflow of this presentation
Evaluation
What does it mean?
What can it do for us?
What do we need?
Concept
Log FrameElements of tender
Gender Mainstreaming
Defining the Log FrameIndicators
Terms of Reference
Next steps
Timeline
At this meeting
After the meeting
Feedback
Feedback
![Page 7: TCA VENet | project evaluation Gerd Beidernikl Center for Education and Economy, Research & Consulting Graz, Austria 1st project meeting 11. September](https://reader031.vdocument.in/reader031/viewer/2022032123/56649b44550346318e8b75fb/html5/thumbnails/7.jpg)
7
Evaluation
>>There are as many different definitions of „evaluation“ as there are evaluators.<<
![Page 8: TCA VENet | project evaluation Gerd Beidernikl Center for Education and Economy, Research & Consulting Graz, Austria 1st project meeting 11. September](https://reader031.vdocument.in/reader031/viewer/2022032123/56649b44550346318e8b75fb/html5/thumbnails/8.jpg)
8
what is “evaluation”
>>Evaluation is the systematic investigation of the merit or worth of an object or process for the purpose of reducing uncertainty in decision making.<< (Mertens, 1998)
3 types of evaluation – regarding the point in time
Ex-ante or predictive evaluation concept
Acompanying or formative evaluation processes
Ex-post or summary evaluation results
Different context, different focus
Generating and securing knowledge
Controlling
Legitimizing actions
Dialogue with stakeholders, etc…..
![Page 9: TCA VENet | project evaluation Gerd Beidernikl Center for Education and Economy, Research & Consulting Graz, Austria 1st project meeting 11. September](https://reader031.vdocument.in/reader031/viewer/2022032123/56649b44550346318e8b75fb/html5/thumbnails/9.jpg)
9
why evaluation? what’s the benefit?
An evaluation offers a systematic way of approaching a topic benefits for project planning
Gives neutral feedback for all project partners
Informs about performance, efficiancy, effectiveness and quality of the project
Makes the project results visible marketing, proof
Possibility to learn in the runtime of a project
Possibility to counteract in case of anticipated problems
Secure Knowledge
Identify best practice and transferable models
![Page 10: TCA VENet | project evaluation Gerd Beidernikl Center for Education and Economy, Research & Consulting Graz, Austria 1st project meeting 11. September](https://reader031.vdocument.in/reader031/viewer/2022032123/56649b44550346318e8b75fb/html5/thumbnails/10.jpg)
10
what this evaluation should be
Support for the transnational cooperation
Answering the question: Do we deliver what we are supposed to deliver?
Neutral reflexion of what is going on
Suggestions for changes and improvement
Ensuring positive project development by giving periodic input
Fostering the coperation and communication
![Page 11: TCA VENet | project evaluation Gerd Beidernikl Center for Education and Economy, Research & Consulting Graz, Austria 1st project meeting 11. September](https://reader031.vdocument.in/reader031/viewer/2022032123/56649b44550346318e8b75fb/html5/thumbnails/11.jpg)
11
what an / this evaluation should not be
A judgement of good and bad, of right or wrong
Evaluators are no project outsiders, they are project partners
Evaluators are no pitbull terriers, lurking for any mistake
Evaluation ≠ project controlling
No analysis of cost efficiancy
No evaluation of the benefits for the target groups
![Page 12: TCA VENet | project evaluation Gerd Beidernikl Center for Education and Economy, Research & Consulting Graz, Austria 1st project meeting 11. September](https://reader031.vdocument.in/reader031/viewer/2022032123/56649b44550346318e8b75fb/html5/thumbnails/12.jpg)
12
What are your expectations?
![Page 13: TCA VENet | project evaluation Gerd Beidernikl Center for Education and Economy, Research & Consulting Graz, Austria 1st project meeting 11. September](https://reader031.vdocument.in/reader031/viewer/2022032123/56649b44550346318e8b75fb/html5/thumbnails/13.jpg)
13
Evaluation concept
![Page 14: TCA VENet | project evaluation Gerd Beidernikl Center for Education and Economy, Research & Consulting Graz, Austria 1st project meeting 11. September](https://reader031.vdocument.in/reader031/viewer/2022032123/56649b44550346318e8b75fb/html5/thumbnails/14.jpg)
14
logical framework
One commonly used apporach: logical framework = LogFrame
The LogFrame is the starting point of the evaluation
Displays the main logic of the evaluation subject
A skeleton of the project that has to be evaluated
Represents the causal relations between different levels of objectives, actions and results
![Page 15: TCA VENet | project evaluation Gerd Beidernikl Center for Education and Economy, Research & Consulting Graz, Austria 1st project meeting 11. September](https://reader031.vdocument.in/reader031/viewer/2022032123/56649b44550346318e8b75fb/html5/thumbnails/15.jpg)
15
LogFrame
GoalsNeeds Ressources / Input
Processes / Mechanisms
Effects / results
Overall objectives
Indikators
Specific objectives
Expected Products
Impact
Outcome
Output
Performance
Relevance
Effectiveness
Efficiency
Secondary effects - unexpected
sustainability
Context effects
Structure
Concept
![Page 16: TCA VENet | project evaluation Gerd Beidernikl Center for Education and Economy, Research & Consulting Graz, Austria 1st project meeting 11. September](https://reader031.vdocument.in/reader031/viewer/2022032123/56649b44550346318e8b75fb/html5/thumbnails/16.jpg)
16
preconditions
There are commonly shared, defined and agreed goals
Goals have to be measureable
There are defined ressources
There are defined processes
The skeleton of the log frame has to be filled based on information in the kick off stage of the project.
Should be kept in mind in every working session of this meeting.
Will be worked out by the evaluation team after the meeting in cooperation with the project management
![Page 17: TCA VENet | project evaluation Gerd Beidernikl Center for Education and Economy, Research & Consulting Graz, Austria 1st project meeting 11. September](https://reader031.vdocument.in/reader031/viewer/2022032123/56649b44550346318e8b75fb/html5/thumbnails/17.jpg)
17
call for tenders
The call for tender asked for
Evaluation of the agreed procedures, tasks and responsibilities formal evaluation
Evaluation of the results and the processes to reach these results content evaluation
Participation at every meeting
Monitoring of the Gender Mainstreaming Process
Periodically reporting
![Page 18: TCA VENet | project evaluation Gerd Beidernikl Center for Education and Economy, Research & Consulting Graz, Austria 1st project meeting 11. September](https://reader031.vdocument.in/reader031/viewer/2022032123/56649b44550346318e8b75fb/html5/thumbnails/18.jpg)
18
main elements of the tender
Evaluation TCA VENet
Formal Evaluation Content Evaluation
Concept Evaluation
Structural Evaluation
Gender Mainstreaming
Process Evaluation
Result Evaluation
VENet ModelEuropean Mainsteaming
Spport SystemInternet Platform
Final Activity
Gender Mainstreaming – cross-sectional topic
![Page 19: TCA VENet | project evaluation Gerd Beidernikl Center for Education and Economy, Research & Consulting Graz, Austria 1st project meeting 11. September](https://reader031.vdocument.in/reader031/viewer/2022032123/56649b44550346318e8b75fb/html5/thumbnails/19.jpg)
19
formal evaluation - concept
Concept analysis
Assessment of the codified project structure and plan
Assessment of management tools
Assessment of communication tools
Data
project documents (e.g. proposal)
meetings
protocols (e.g. steering group)
interviews
![Page 20: TCA VENet | project evaluation Gerd Beidernikl Center for Education and Economy, Research & Consulting Graz, Austria 1st project meeting 11. September](https://reader031.vdocument.in/reader031/viewer/2022032123/56649b44550346318e8b75fb/html5/thumbnails/20.jpg)
20
formal evaluation - structure
Structural analysisFocus: transnational cooperation als system of interacting
organizations resp. individuals
Assessment of flow of communication
Assessment of the cooperation patterns
DataHard facts: contribution on formal level (reports, website,..)
Soft facts:Social network survey (x2, email)
Satisfaction survey (x2, email)
Event questionnaire (x5, at every project meeting)
Interviews
Documents, Internet Platform (?),….
![Page 21: TCA VENet | project evaluation Gerd Beidernikl Center for Education and Economy, Research & Consulting Graz, Austria 1st project meeting 11. September](https://reader031.vdocument.in/reader031/viewer/2022032123/56649b44550346318e8b75fb/html5/thumbnails/21.jpg)
21
Social network analysis (SNA)
SNA – research method that focuses on the relation (communication, cooperation, exchange of information,…) between any given set of actors.
A SNA survey asks the partners to rate their own network behaviour according to different criterias (frequeny of interaction, channels of interaction, topics of interaction,…).
The answers on these questions are producing a picture of the overall network. All individual answers are displayed on the network level.
Benefit
You get SNA plots of the network structure
Key fiugures on well and not well established elements
![Page 22: TCA VENet | project evaluation Gerd Beidernikl Center for Education and Economy, Research & Consulting Graz, Austria 1st project meeting 11. September](https://reader031.vdocument.in/reader031/viewer/2022032123/56649b44550346318e8b75fb/html5/thumbnails/22.jpg)
22
network plots
![Page 23: TCA VENet | project evaluation Gerd Beidernikl Center for Education and Economy, Research & Consulting Graz, Austria 1st project meeting 11. September](https://reader031.vdocument.in/reader031/viewer/2022032123/56649b44550346318e8b75fb/html5/thumbnails/23.jpg)
23
Gender Mainstreaming
GM are the efforts we make to ensure that gender aspects and gender equality is considered in all parts and stages of the project (working plan, objectives, actions,…).
A cross sectional topic
GM aspects are considered in every part of the evaluation
and the evaluation itself
3 main GM approaches
Equal opportunities approach neutral treatment
Diversity approach awareness of the different needs
Affirmative action approach focus on women
![Page 24: TCA VENet | project evaluation Gerd Beidernikl Center for Education and Economy, Research & Consulting Graz, Austria 1st project meeting 11. September](https://reader031.vdocument.in/reader031/viewer/2022032123/56649b44550346318e8b75fb/html5/thumbnails/24.jpg)
24
Gender Mainstreaming
Formal Evaluation Content Evaluation- Gender sensitive language- Roles and functions- GM as a topic at the meetings- GM as cross sectional topic in the project- Gender aspects in documents- Treatment of GM in general- GM experience of the partners-…
- Is GM considered by the partners in
- the analysis of underling
problems
- the definition of goals and
target groups
- the design of the processes
- the implementation
- Do the products reflect GM principles?
- Design / content
- Dissemination
- Benefit
Data: protocols, interviews, survey questions, products,…
![Page 25: TCA VENet | project evaluation Gerd Beidernikl Center for Education and Economy, Research & Consulting Graz, Austria 1st project meeting 11. September](https://reader031.vdocument.in/reader031/viewer/2022032123/56649b44550346318e8b75fb/html5/thumbnails/25.jpg)
25
Content evaluation
Two levels
Processes
Results
Main questions
Have the intended results been reached (quantity and quality)?
Have the processes to reach these results been carried out as intended?
Did the partners contribute as scheduled?
Are there any positive or negative deviations? If so, why?
Which deductions for the following project work can be made?
![Page 26: TCA VENet | project evaluation Gerd Beidernikl Center for Education and Economy, Research & Consulting Graz, Austria 1st project meeting 11. September](https://reader031.vdocument.in/reader031/viewer/2022032123/56649b44550346318e8b75fb/html5/thumbnails/26.jpg)
26
Content evaluation
For each of the five main products
VENet model, internet platform,…..
Data
Documents
Interviews
The products themselves
Check lists resp. progress reports for the project (in cooperation with the project management)
Questions in the email survey
![Page 27: TCA VENet | project evaluation Gerd Beidernikl Center for Education and Economy, Research & Consulting Graz, Austria 1st project meeting 11. September](https://reader031.vdocument.in/reader031/viewer/2022032123/56649b44550346318e8b75fb/html5/thumbnails/27.jpg)
27
deliverables of the evaluation
Short reports before each meeting
Distributed electronicly (internet platform)
Participation at every transnational meeting (1 w.d.)
Presentation of evaluation results and workshop on the findings (1-2,5h)
1 mid-term report
1 final report
![Page 28: TCA VENet | project evaluation Gerd Beidernikl Center for Education and Economy, Research & Consulting Graz, Austria 1st project meeting 11. September](https://reader031.vdocument.in/reader031/viewer/2022032123/56649b44550346318e8b75fb/html5/thumbnails/28.jpg)
28
Feedback on the evaluation model What is your impression?
Does the evaluation provide the information we need?GM in our project?
How do we take the evaluation results up?
![Page 29: TCA VENet | project evaluation Gerd Beidernikl Center for Education and Economy, Research & Consulting Graz, Austria 1st project meeting 11. September](https://reader031.vdocument.in/reader031/viewer/2022032123/56649b44550346318e8b75fb/html5/thumbnails/29.jpg)
29
Defining the LogFrame
![Page 30: TCA VENet | project evaluation Gerd Beidernikl Center for Education and Economy, Research & Consulting Graz, Austria 1st project meeting 11. September](https://reader031.vdocument.in/reader031/viewer/2022032123/56649b44550346318e8b75fb/html5/thumbnails/30.jpg)
30
Defintion
By now the LogFrame is empty
Will be based on the transnational project plan with defined goals, milestones and responsibilities
Next Step: Filling the LogFrame with information
Results from this meeting (e.g. working plan,…)
Documents
In cooperation with transnational coordination
The rough LogFrame will increase in its density as the project goes on
![Page 31: TCA VENet | project evaluation Gerd Beidernikl Center for Education and Economy, Research & Consulting Graz, Austria 1st project meeting 11. September](https://reader031.vdocument.in/reader031/viewer/2022032123/56649b44550346318e8b75fb/html5/thumbnails/31.jpg)
31
Definition
The defined aims of the project are crucial for the evaluation!
Important: If you set up any aims, goals and objectives:
be SMART
S…specific: general aims must be broken down
M…measurable: no untouchable aims
A…acceptance: agreement among all partners
R…realistic: no trivial aims, no unrealistic visions
T…timely fixed and with fixed responsibilities
![Page 32: TCA VENet | project evaluation Gerd Beidernikl Center for Education and Economy, Research & Consulting Graz, Austria 1st project meeting 11. September](https://reader031.vdocument.in/reader031/viewer/2022032123/56649b44550346318e8b75fb/html5/thumbnails/32.jpg)
32
Definition
The LogFrame is the catalogue of the evaluation.
Beneath this we will set up the so called Terms of Reference
Kind of evaluation handbook, the practice version of the tender
Puts down the procedures of the evaluation
Commonly shared evaluation principles
Will beset up by the ZBW
![Page 33: TCA VENet | project evaluation Gerd Beidernikl Center for Education and Economy, Research & Consulting Graz, Austria 1st project meeting 11. September](https://reader031.vdocument.in/reader031/viewer/2022032123/56649b44550346318e8b75fb/html5/thumbnails/33.jpg)
33
Next Steps
![Page 34: TCA VENet | project evaluation Gerd Beidernikl Center for Education and Economy, Research & Consulting Graz, Austria 1st project meeting 11. September](https://reader031.vdocument.in/reader031/viewer/2022032123/56649b44550346318e8b75fb/html5/thumbnails/34.jpg)
34
At this meeting
Consider the LogFrame allready when working out objectives (SMART), products and processes
Evaluation grows as the project grows.
Some short interviews on the 2nd resp. 3rd day of the meeting – transnational coordinators
A concluding event questionaire in the last session of the meeting
Define one main contact person regarding the evaluation in each partner organization
![Page 35: TCA VENet | project evaluation Gerd Beidernikl Center for Education and Economy, Research & Consulting Graz, Austria 1st project meeting 11. September](https://reader031.vdocument.in/reader031/viewer/2022032123/56649b44550346318e8b75fb/html5/thumbnails/35.jpg)
35
After the meeting
ZBW will work out the LogFrame in cooperation with the transnational coordination
Set up the Terms of Reference
Set up dates for the next evaluation steps
Inform the partners about the evaluation
![Page 36: TCA VENet | project evaluation Gerd Beidernikl Center for Education and Economy, Research & Consulting Graz, Austria 1st project meeting 11. September](https://reader031.vdocument.in/reader031/viewer/2022032123/56649b44550346318e8b75fb/html5/thumbnails/36.jpg)
36
Timeline
![Page 37: TCA VENet | project evaluation Gerd Beidernikl Center for Education and Economy, Research & Consulting Graz, Austria 1st project meeting 11. September](https://reader031.vdocument.in/reader031/viewer/2022032123/56649b44550346318e8b75fb/html5/thumbnails/37.jpg)
37
Your responsibilities!
To commit and take active part in the evaluation process
To fill in questionnaires carefully
To submit documents and information when asked for
To return you contribution to the evaluation in time – respect deadlines
To evaluate the evaluation! Feedback is welcome!!
![Page 38: TCA VENet | project evaluation Gerd Beidernikl Center for Education and Economy, Research & Consulting Graz, Austria 1st project meeting 11. September](https://reader031.vdocument.in/reader031/viewer/2022032123/56649b44550346318e8b75fb/html5/thumbnails/38.jpg)
38
Contact information
Thank you!Let‘s get it on!
Mag. Gerd BeiderniklZentrum für Bildung und Wirtschaft
Joanneumring 5/4, 8010 Graz, Austria (valid 1st October 2005)
Mail: [email protected]: +43 / 316 / 72 17 44 -13
Fax: +43 / 316 / 72 17 44 -21Mobile: +32 / 650 / 46 21 061