![Page 1: INSTITUTIONAL EVALUATION – RECENT EXPERIENCES, …](https://reader034.vdocument.in/reader034/viewer/2022050716/624cbec0a1f19413ed56deda/html5/thumbnails/1.jpg)
Hubert Paulmer M.Sc. (Ag), MBA, G.D.I .PHarry Cummings and Assoc ia tes In c .
Dr. Harry Cummings Ph.D. , R.P.P. , M.C.I .PUnivers i ty o f Gue lph / Harry Cummings and Assoc ia tes Inc .
AEA – ANNUAL CONFERENCE 2009 – ORLANDO
NOVEMBER 12 , 2009
INSTITUTIONAL EVALUATION IN THE NON-PROFIT SECTOR:
CHALLENGES AND LESSONS FROM RECENT EXPERIENCES
![Page 2: INSTITUTIONAL EVALUATION – RECENT EXPERIENCES, …](https://reader034.vdocument.in/reader034/viewer/2022050716/624cbec0a1f19413ed56deda/html5/thumbnails/2.jpg)
PRESENTATION - OVERVIEW
Institutional Evaluation – Some Definitions Program Evaluation vs. Institutional Evaluation Frameworks Evaluation Issues Approach Methodology Lessons Challenges Impact of our Evaluations How does Institutional Evaluation help? Is it gaining importance?
![Page 3: INSTITUTIONAL EVALUATION – RECENT EXPERIENCES, …](https://reader034.vdocument.in/reader034/viewer/2022050716/624cbec0a1f19413ed56deda/html5/thumbnails/3.jpg)
DEFINITIONS
Institutional evaluations have been described as “processes which use concepts and methods from the social and behavioural sciences to assess organization’s current practices and find ways to increase their efficiency
(Universalia, 1993)
![Page 4: INSTITUTIONAL EVALUATION – RECENT EXPERIENCES, …](https://reader034.vdocument.in/reader034/viewer/2022050716/624cbec0a1f19413ed56deda/html5/thumbnails/4.jpg)
DEFINITIONS
Institutional assessment is a comprehensive approach for profiling institutional capacity and performance. The approach considers various factors which come to play in institutional development: external environment; institutional factors; and inter-institutional linkages
(Morgan and Taschereau, 1996)
![Page 5: INSTITUTIONAL EVALUATION – RECENT EXPERIENCES, …](https://reader034.vdocument.in/reader034/viewer/2022050716/624cbec0a1f19413ed56deda/html5/thumbnails/5.jpg)
DEFINITIONS
Organizational assessment can be defined as a systematic process for obtaining valid information about the performance of an organization and the factors associated with the performance
(Gaebler, & Osborne, 1993; Meyer 7 Scott, 1992)
![Page 6: INSTITUTIONAL EVALUATION – RECENT EXPERIENCES, …](https://reader034.vdocument.in/reader034/viewer/2022050716/624cbec0a1f19413ed56deda/html5/thumbnails/6.jpg)
TERMS - LITERATURE
Institutional evaluation Institutional assessment Organizational assessment
![Page 7: INSTITUTIONAL EVALUATION – RECENT EXPERIENCES, …](https://reader034.vdocument.in/reader034/viewer/2022050716/624cbec0a1f19413ed56deda/html5/thumbnails/7.jpg)
PROGRAM vs. INSTITUTIONAL EVALUATION
IE is more complex – lot more factors / issues to be looked at
IE needs more time
Stakeholder commitment, involvement and participation is important for both (but for IE it is essential) – “buy in” at all levels required for IE
![Page 8: INSTITUTIONAL EVALUATION – RECENT EXPERIENCES, …](https://reader034.vdocument.in/reader034/viewer/2022050716/624cbec0a1f19413ed56deda/html5/thumbnails/8.jpg)
PROGRAM vs. INSTITUTIONAL EVALUATION
Performance in the context of external environment, organizational capacity and motivation (in program it is against the plan)
IE contributes to the institution’s strategy/vision/mission (PE to program strategy)
Program / projects are driven by logic and relatively linear, where as institutions/ organizations are dynamic and changes are clustered / interactive
![Page 9: INSTITUTIONAL EVALUATION – RECENT EXPERIENCES, …](https://reader034.vdocument.in/reader034/viewer/2022050716/624cbec0a1f19413ed56deda/html5/thumbnails/9.jpg)
FRAMEWORK - SIX BOX MODEL
STRATEGY
INTERNAL RELATIONSHIPS
HELPFUL MECHANISMS
LEADERSHIP
REWARDS
STRUCTURES
Source: Mavin Wiesbord
![Page 10: INSTITUTIONAL EVALUATION – RECENT EXPERIENCES, …](https://reader034.vdocument.in/reader034/viewer/2022050716/624cbec0a1f19413ed56deda/html5/thumbnails/10.jpg)
FRAMEWORK – 7 S MODEL (McKinsey)
Structure Strategy Skills Style Staff Shared Values Systems
![Page 11: INSTITUTIONAL EVALUATION – RECENT EXPERIENCES, …](https://reader034.vdocument.in/reader034/viewer/2022050716/624cbec0a1f19413ed56deda/html5/thumbnails/11.jpg)
FRAMEWORK - UNIVERSALIA / IDRC
OrganizationalCapacity
Strategic LeadershipStructure
HR ManagementFinancial ManagementProgram management
InfrastructureInter-institutional Linkages
Organizational Processes
External EnvironmentAdministrative & Legal
Socio-culturalGeographic
PoliticalStakeholderEconomic
Organizational MotivationHistoryMissionCultureIncentives or rewards
Organizational Performance
EffectivenessEfficiencyRelevance
Financial Viability
Source: CIDA (2006); Universalia (1995)
![Page 12: INSTITUTIONAL EVALUATION – RECENT EXPERIENCES, …](https://reader034.vdocument.in/reader034/viewer/2022050716/624cbec0a1f19413ed56deda/html5/thumbnails/12.jpg)
“Performance is a function of an organization’s enabling environment, capacity and organizational motivation”
(CIDA, 2006)
![Page 13: INSTITUTIONAL EVALUATION – RECENT EXPERIENCES, …](https://reader034.vdocument.in/reader034/viewer/2022050716/624cbec0a1f19413ed56deda/html5/thumbnails/13.jpg)
OTHER MODELS / CONCEPTS
Porter’s – Five Forces Model
Porter’s – Competitive Advantage (cost leadership, differentiation, focus)
Prahalad’s and Hamel’s – Core Competencies
No one method / framework – you can combine frameworks and / or adapt & adopt
![Page 14: INSTITUTIONAL EVALUATION – RECENT EXPERIENCES, …](https://reader034.vdocument.in/reader034/viewer/2022050716/624cbec0a1f19413ed56deda/html5/thumbnails/14.jpg)
KEY EVALUATION ISSUES
Depended on Funding / Grant agreement and / or requirements of main funders
The Most Common Ones Relevance Efficiency Success (effectiveness / impact) Access / ReachPlus Cost-effectiveness
![Page 15: INSTITUTIONAL EVALUATION – RECENT EXPERIENCES, …](https://reader034.vdocument.in/reader034/viewer/2022050716/624cbec0a1f19413ed56deda/html5/thumbnails/15.jpg)
WE ALSO LOOKED AT
Governance Management – style/structure Processes / Mechanisms / Systems Planning Relationships HR Reporting Communications – internal / external - platforms
Project Management
![Page 16: INSTITUTIONAL EVALUATION – RECENT EXPERIENCES, …](https://reader034.vdocument.in/reader034/viewer/2022050716/624cbec0a1f19413ed56deda/html5/thumbnails/16.jpg)
WE ALSO LOOKED AT
Partnerships / Inter-institutional linkages Financial Viability / Sustainability Sources of revenue - trend # of donors & contribution by each donor – trend
![Page 17: INSTITUTIONAL EVALUATION – RECENT EXPERIENCES, …](https://reader034.vdocument.in/reader034/viewer/2022050716/624cbec0a1f19413ed56deda/html5/thumbnails/17.jpg)
ALSO LOOKED AT
External Environment
Organizational Motivation History Mission Incentive / Rewards
Strengths and Weaknesses
![Page 18: INSTITUTIONAL EVALUATION – RECENT EXPERIENCES, …](https://reader034.vdocument.in/reader034/viewer/2022050716/624cbec0a1f19413ed56deda/html5/thumbnails/18.jpg)
APPROACH / METHODOLOGY
Initial meeting
Evaluation Framework / Work Plan – shared with funders / institutions for inputs
Mixed-methods approach
Participatory (where possible) involving key stakeholders –but retaining the external reviewer’s perspective
Note: Too much loosely structured involvement of internal stakeholders with vested interest will create conflicts over agendas, methodologies, working relationships and the wording of reports (Chelimksy & North 1997)
![Page 19: INSTITUTIONAL EVALUATION – RECENT EXPERIENCES, …](https://reader034.vdocument.in/reader034/viewer/2022050716/624cbec0a1f19413ed56deda/html5/thumbnails/19.jpg)
APPROACH / METHODOLOGY
Methods used Document review Key informant interviews Online survey Focus groups Delphi Panel
![Page 20: INSTITUTIONAL EVALUATION – RECENT EXPERIENCES, …](https://reader034.vdocument.in/reader034/viewer/2022050716/624cbec0a1f19413ed56deda/html5/thumbnails/20.jpg)
DATA
Qualitative / Quantitative Rating / Ranking scale scores Comparisons of “similar” organizations Financial numbers Cost data Internal data – projects, events, partners, media
etc.
![Page 21: INSTITUTIONAL EVALUATION – RECENT EXPERIENCES, …](https://reader034.vdocument.in/reader034/viewer/2022050716/624cbec0a1f19413ed56deda/html5/thumbnails/21.jpg)
LESSONS – Points to keep in mind
IA Frameworks can give you a lot of information –but its important to help focus on the dynamics / relationships between various factors
It is important to keep in mind that the focus of IE / IA is on the individual institution and its environment than on a development challenge / program/initiative
Institutional “lens”
![Page 22: INSTITUTIONAL EVALUATION – RECENT EXPERIENCES, …](https://reader034.vdocument.in/reader034/viewer/2022050716/624cbec0a1f19413ed56deda/html5/thumbnails/22.jpg)
LESSONS – Points to keep in mind
Data has to be contextualized and the limitation of both data and process has to be acknowledged / explained
Avoid “insight fatigue”
![Page 23: INSTITUTIONAL EVALUATION – RECENT EXPERIENCES, …](https://reader034.vdocument.in/reader034/viewer/2022050716/624cbec0a1f19413ed56deda/html5/thumbnails/23.jpg)
CHALLENGES
Stakeholder commitment / involvement Long-term vs. short-term All levels
Sensitive information Identifying leverage points of change Comparisons – benchmarking Data / information considered in isolation of
context can be misleading Time / cost factor (how comprehensive)
![Page 24: INSTITUTIONAL EVALUATION – RECENT EXPERIENCES, …](https://reader034.vdocument.in/reader034/viewer/2022050716/624cbec0a1f19413ed56deda/html5/thumbnails/24.jpg)
CHALLENGES
Availability of data (reliance on perceptions of experts)
Measuring effectiveness / impact for “policy oriented” think tanks
![Page 25: INSTITUTIONAL EVALUATION – RECENT EXPERIENCES, …](https://reader034.vdocument.in/reader034/viewer/2022050716/624cbec0a1f19413ed56deda/html5/thumbnails/25.jpg)
IMPACT OF OUR WORK(INSTITUTIONAL EVALUATIONS)
Changes in processes / systems within institutions Revisions in strategy – (work in progress) Focus on work - (narrowed) on themes Project / Partner identification processes Changes in reporting/sharing of project costs
(within the organization)
![Page 26: INSTITUTIONAL EVALUATION – RECENT EXPERIENCES, …](https://reader034.vdocument.in/reader034/viewer/2022050716/624cbec0a1f19413ed56deda/html5/thumbnails/26.jpg)
HOW DOES INSITITUTIONAL EVALUATION HELP?
Insights for the institution – revise strategy, turning point, preparing for funding, staffing, programming, etc.
Reveals capacities and gaps / strengths and weaknesses
Creates an understanding of relationships among departments/units of the institution
Develops an understanding of the inter-institutional linkages
![Page 27: INSTITUTIONAL EVALUATION – RECENT EXPERIENCES, …](https://reader034.vdocument.in/reader034/viewer/2022050716/624cbec0a1f19413ed56deda/html5/thumbnails/27.jpg)
HOW DOES INSITITUTIONAL EVALUATION HELP?
Facilitates capacity development / organizational change
Feedback on governance and management Help institutions to get better / sustainable
![Page 28: INSTITUTIONAL EVALUATION – RECENT EXPERIENCES, …](https://reader034.vdocument.in/reader034/viewer/2022050716/624cbec0a1f19413ed56deda/html5/thumbnails/28.jpg)
THE GROWING IMPORTANCE
Financial resources – scarce Pressure to perform / deliver Changing (global) context ‘Environment’ has become very competitive High inter-dependency / multi-stakeholder linkages Holding a continued presence in the ‘market place’
– this needs a new level of performance
![Page 29: INSTITUTIONAL EVALUATION – RECENT EXPERIENCES, …](https://reader034.vdocument.in/reader034/viewer/2022050716/624cbec0a1f19413ed56deda/html5/thumbnails/29.jpg)
Organizations / institutions have to be dynamic, flexible,
accommodating and forward thinking
![Page 30: INSTITUTIONAL EVALUATION – RECENT EXPERIENCES, …](https://reader034.vdocument.in/reader034/viewer/2022050716/624cbec0a1f19413ed56deda/html5/thumbnails/30.jpg)
H U B E R T P A U L M E R M . S c . ( A g ) , M B A , G . D . I . P .
D r . H A R R Y C U M M I N G S P h . D . , R . P . P . , M . C . I . P
H a r r y C u m m i n g s a n d A s s o c i a t e s I n c .O t t a w a / G u e l p h , C a n a d a
h c a @ w e b . c a
T E L : 6 1 3 - 5 6 7 - 4 5 0 0 / 5 1 9 - 8 2 3 - 1 6 4 7w w w . h c a c o n s u l t i n g . c a
THANK YOU