developing performance measures through a consultative process
TRANSCRIPT
![Page 1: Developing Performance Measures through a Consultative Process](https://reader036.vdocument.in/reader036/viewer/2022062708/5587b20cd8b42a6b408b45d9/html5/thumbnails/1.jpg)
124 Merton St., Suite 502Toronto, Ontario M4S 2Z2
Telephone: (416) 469-9954Fax: (416) 469-8487
www.cathexisconsulting.ca
Developing Performance Measures Through a Consultative Process
June 2013
![Page 2: Developing Performance Measures through a Consultative Process](https://reader036.vdocument.in/reader036/viewer/2022062708/5587b20cd8b42a6b408b45d9/html5/thumbnails/2.jpg)
Purpose of the Presentation
To provide an example of transformative use of performance measurement
![Page 3: Developing Performance Measures through a Consultative Process](https://reader036.vdocument.in/reader036/viewer/2022062708/5587b20cd8b42a6b408b45d9/html5/thumbnails/3.jpg)
We will cover. . .
Background on the Accessibility for Ontarians with Disabilities Act
An overview on the development of performance indicators to measure the extent to which the Act is really making a difference
An introduction to a method that can help with the selection of indicators when there are multiple stakeholders
An overview of the monitoring methods
![Page 4: Developing Performance Measures through a Consultative Process](https://reader036.vdocument.in/reader036/viewer/2022062708/5587b20cd8b42a6b408b45d9/html5/thumbnails/4.jpg)
Background on AODA The Accessibility for Ontarians with Disabilities Act, 2005 (AODA)
was adopted to improve accessibility for the 1.85 million people with disabilities in Ontario
The legislation covers five domains:• Customer Service
• Information and Communication
• Employment
• Transportation
• Built Environment
Standards have been development relating to each of those domains
![Page 5: Developing Performance Measures through a Consultative Process](https://reader036.vdocument.in/reader036/viewer/2022062708/5587b20cd8b42a6b408b45d9/html5/thumbnails/5.jpg)
Questions to be Addressed What dimensions (health, education, recreation, etc)
should be considered when developing performance indicators to measure the AODA’s impact on people with disabilities, older adults and their families
Which indicators can best account for the diversity among people with disabilities, older adults and their families (i.e. types of disabilities, degree of disability, age)
Which indicators will best measure quality of life improvements for people with disabilities, older adults and their families
What data collection tools can be used measure the performance indicators over time and with a limited budget
![Page 6: Developing Performance Measures through a Consultative Process](https://reader036.vdocument.in/reader036/viewer/2022062708/5587b20cd8b42a6b408b45d9/html5/thumbnails/6.jpg)
Key Focus
The extent to which the standards have had a positive impact on Ontarians with disabilities, older adults and their families
![Page 7: Developing Performance Measures through a Consultative Process](https://reader036.vdocument.in/reader036/viewer/2022062708/5587b20cd8b42a6b408b45d9/html5/thumbnails/7.jpg)
Developing the Indicators Understanding the ADO Standards
Literature review
Consultation with people with disabilities, older adults and their families
• Out reach through a number of vehicles
• On-line survey with 426 respondents
• Focus groups in Huntsville, Toronto, Ottawa, Thunder Bay and London with a total of 58 participants
• Social media blog with no response
• Submission (1)
Priority Sort process including a rapid sort and a more thoughtful sort
Finalizing indicators
![Page 8: Developing Performance Measures through a Consultative Process](https://reader036.vdocument.in/reader036/viewer/2022062708/5587b20cd8b42a6b408b45d9/html5/thumbnails/8.jpg)
Findings from the Literature
Not agreed-upon definition – subjective to each individual
Schalock and his colleagues developed a framework that focuses on empowerment rather than disability
Canadian Index of Well-being looks at indicators for that cover all Canadians
Quality of Life Research Unit at the University of Toronto’s Center for Health Promotion: “The degree to which a person enjoys the important possibilities of his or her life”
![Page 9: Developing Performance Measures through a Consultative Process](https://reader036.vdocument.in/reader036/viewer/2022062708/5587b20cd8b42a6b408b45d9/html5/thumbnails/9.jpg)
Quality of Life Research Unit, Center for Health Promotion, University of Toronto
Key Elements of a QOL Framework Being: Who One Is
• Physical Being
• Psychological Being
• Spiritual Being
Belonging: Connection with One’s Environment• Physical Belonging
• Social Belonging
• Community Belonging
Becoming: Achieving Personal Goals, Hopes and Aspirations• Practical Becoming
• Leisure Becoming
• Growth Becoming
![Page 10: Developing Performance Measures through a Consultative Process](https://reader036.vdocument.in/reader036/viewer/2022062708/5587b20cd8b42a6b408b45d9/html5/thumbnails/10.jpg)
![Page 11: Developing Performance Measures through a Consultative Process](https://reader036.vdocument.in/reader036/viewer/2022062708/5587b20cd8b42a6b408b45d9/html5/thumbnails/11.jpg)
Findings from the Consultation
Most respondents indicated that most indicators were important
Provision of health care services was considered to be the most important
Gave advice on wording and consolidation of indicators
Based on the findings seven high level indicators and 26 specific indicators emerged
![Page 12: Developing Performance Measures through a Consultative Process](https://reader036.vdocument.in/reader036/viewer/2022062708/5587b20cd8b42a6b408b45d9/html5/thumbnails/12.jpg)
The Priority Sort Process
Want you to know . . .• What Group Priority Sort is
• Why you might use Group Priority Sort (what the benefits are)
• When it is (and is not) appropriate to use Group Priority Sort
![Page 13: Developing Performance Measures through a Consultative Process](https://reader036.vdocument.in/reader036/viewer/2022062708/5587b20cd8b42a6b408b45d9/html5/thumbnails/13.jpg)
What is Priority Sort?
A participatory priority-ranking exercise
It can be used to:
Define the scope of an evaluation
Select performance measures/indicators
Prioritize strategic planning goals
Define a complex concept
![Page 14: Developing Performance Measures through a Consultative Process](https://reader036.vdocument.in/reader036/viewer/2022062708/5587b20cd8b42a6b408b45d9/html5/thumbnails/14.jpg)
What is Priority Sort?
Has small groups of stakeholders or “experts” rank-order specified items
The outputs are:
a) comparative rankings:
b) rich qualitative data; and
c) engaged participants
Evolved out of Q Methodology and produces similar information as Delphi
![Page 15: Developing Performance Measures through a Consultative Process](https://reader036.vdocument.in/reader036/viewer/2022062708/5587b20cd8b42a6b408b45d9/html5/thumbnails/15.jpg)
Overview of Process
![Page 16: Developing Performance Measures through a Consultative Process](https://reader036.vdocument.in/reader036/viewer/2022062708/5587b20cd8b42a6b408b45d9/html5/thumbnails/16.jpg)
The Priority Sort:Two Step Process
![Page 17: Developing Performance Measures through a Consultative Process](https://reader036.vdocument.in/reader036/viewer/2022062708/5587b20cd8b42a6b408b45d9/html5/thumbnails/17.jpg)
Recent Group Priority Sort: Quality of Life Indicators for People with Disabilities
![Page 18: Developing Performance Measures through a Consultative Process](https://reader036.vdocument.in/reader036/viewer/2022062708/5587b20cd8b42a6b408b45d9/html5/thumbnails/18.jpg)
Rapid Sort Outcome
![Page 19: Developing Performance Measures through a Consultative Process](https://reader036.vdocument.in/reader036/viewer/2022062708/5587b20cd8b42a6b408b45d9/html5/thumbnails/19.jpg)
Sample Instructions: Forced Sort
Now it gets more difficult...
Refine your sort so that there are no more than 6 cards in each category
Try to come to agreement about where each benefit should go
Use blank cards to record any other important indicators (do not sort these)
You have 25 minutes
![Page 20: Developing Performance Measures through a Consultative Process](https://reader036.vdocument.in/reader036/viewer/2022062708/5587b20cd8b42a6b408b45d9/html5/thumbnails/20.jpg)
Forced Sort Outcome
![Page 21: Developing Performance Measures through a Consultative Process](https://reader036.vdocument.in/reader036/viewer/2022062708/5587b20cd8b42a6b408b45d9/html5/thumbnails/21.jpg)
Analysis
![Page 22: Developing Performance Measures through a Consultative Process](https://reader036.vdocument.in/reader036/viewer/2022062708/5587b20cd8b42a6b408b45d9/html5/thumbnails/22.jpg)
Benefits of Priority Sort
Informed decision-making;
Enhanced understanding of the topic among participants;
Strengthened community of stakeholders;
Confidence in the process and the resulting decisions; and
An enjoyable and engaging experience for participants.
![Page 23: Developing Performance Measures through a Consultative Process](https://reader036.vdocument.in/reader036/viewer/2022062708/5587b20cd8b42a6b408b45d9/html5/thumbnails/23.jpg)
When to use Priority Sort
Not the right technique if you are exploring a new area that nobody knows much about.
Not a brainstorming exercise.
Priority Sort captures subjective opinions. If you are seeking objective facts, this is not the right method.
The Priority Sort is most useful when there are divergent opinions.
![Page 24: Developing Performance Measures through a Consultative Process](https://reader036.vdocument.in/reader036/viewer/2022062708/5587b20cd8b42a6b408b45d9/html5/thumbnails/24.jpg)
Recommended Indicators
Agreement on the high level indicators• The over-arching indicators are:
• Being treated with dignity• Feeling an integrated part of school, work and the community
• One high level indicator for each Standard• Customer service: Being served in a way that maintains dignity and
supports inclusion• Employment: Have access to employment consistent with experience,
abilities and training• Information: Have access to information• Transportation: Able to get where you need to go, when you need to go• Design of Public Spaces: Able to move independently around the
community
Reduced the 26 indicators to 12
![Page 25: Developing Performance Measures through a Consultative Process](https://reader036.vdocument.in/reader036/viewer/2022062708/5587b20cd8b42a6b408b45d9/html5/thumbnails/25.jpg)
Recommended IndicatorsCustomer ServiceLevel of confidence that health care providers can provide services in a way that takes into account the person’s disability
• links back to physical being, psychological being and physical belonging in the project’s methodological framework.
Educational institutions provide services in a respectful manner
• links to psychological being, social belonging, practical becoming and growth becoming in the project’s methodological framework.
Emergency response services are provided in a manner that takes into account the person’s disability
• links to physical being, psychological being, and community belonging in the project’s methodological framework.
Recreation/fitness facilities provide services are provided in a way that allow people with disabilities to use and benefit from them
• links to physical being, physical belonging and leisure becoming in the project’s methodological framework.
![Page 26: Developing Performance Measures through a Consultative Process](https://reader036.vdocument.in/reader036/viewer/2022062708/5587b20cd8b42a6b408b45d9/html5/thumbnails/26.jpg)
Recommended Indicators
EmploymentAccommodation is provided in the workplace for people with disabilities
• links to practical becoming in the project’s methodological framework.
Managers and co-workers at all levels accept and make accommodation for people with disabilities
• links to social belonging and practical becoming in the project’s methodological framework.
Career development opportunities are provided in a manner that accommodates people with disabilities
• This indicator links to psychological being and practical becoming in
the project’s methodological framework.
![Page 27: Developing Performance Measures through a Consultative Process](https://reader036.vdocument.in/reader036/viewer/2022062708/5587b20cd8b42a6b408b45d9/html5/thumbnails/27.jpg)
The Recommended IndicatorsInformationInformation about local warnings/emergencies is available to and can be accessed by people with disabilities
• links to physical being, community belonging and practical becoming in the project’s methodological framework.
Accessibility planning includes input from people with disabilities
• links to psychological being and community belonging in the project’s methodological framework.
Websites are designed so that people with disabilities can access them
• links to social belonging, community belonging, and practical becoming in the project’s methodological framework.
![Page 28: Developing Performance Measures through a Consultative Process](https://reader036.vdocument.in/reader036/viewer/2022062708/5587b20cd8b42a6b408b45d9/html5/thumbnails/28.jpg)
Recommended IndicatorsTransportationPublic transportation, taxis and GO trains are equally accessible for people with disabilities as for people without disabilities
• Links to psychological being, and community belonging in the project’s methodological framework
Design of Public SpacesPeople with disabilities are able to get to stores, community centres and other public facilities
• This indicator links to social belonging and community belonging in the project’s methodological framework
![Page 29: Developing Performance Measures through a Consultative Process](https://reader036.vdocument.in/reader036/viewer/2022062708/5587b20cd8b42a6b408b45d9/html5/thumbnails/29.jpg)
Learnings from this Process People with disabilities are interested in participating in consultations related to
accessibility want to see that the ADO is taking the impact of the AODA seriously.
The most effective recruitment occurred through organizations representing or serving the population
The survey provided an excellent sense of what was important for people with disabilities and supported reaching a large number of people
The face-to-face consultation augmented this understanding plus provided input into acceptable wording.
A large number of people with disabilities have access to the technology required to participate in on-line surveys
The steps taken to accommodate people with disabilities, as part of the consultation process, was noted and appreciated by participants.
It is still challenging to find facilities that are fully accessible
When conducting consultations, it is important to set the parameters in a way that keeps the discussion focused, but does not disrespect the importance of the other issues being raised.
![Page 30: Developing Performance Measures through a Consultative Process](https://reader036.vdocument.in/reader036/viewer/2022062708/5587b20cd8b42a6b408b45d9/html5/thumbnails/30.jpg)
Research Questions
To what extent have there been changes in quality of life as it relates to the five AODA Standards for people with disabilities, older adults and their families?
What impact do the Accessibility Standards have on key areas of daily living areas for people with disabilities, older adults and their families?
![Page 31: Developing Performance Measures through a Consultative Process](https://reader036.vdocument.in/reader036/viewer/2022062708/5587b20cd8b42a6b408b45d9/html5/thumbnails/31.jpg)
Suggested Monitoring Method
On-line survey
Interviews
Use of existing data
![Page 32: Developing Performance Measures through a Consultative Process](https://reader036.vdocument.in/reader036/viewer/2022062708/5587b20cd8b42a6b408b45d9/html5/thumbnails/32.jpg)
Monitoring Cycle
Baseline (2013/14)• Recruit monitoring participants
• Survey administered and analyzed
• Interviews conducted and analyzed
• Existing data analyzed
• Baseline report produced
Interim year (2014/15)• Contact with participants
Second monitoring cycle (2015/16)• Survey administered and analyzed
• Interviews conducted and analyzed
• Existing data analyzed
• Second report produced
Interim year (2016/17)• Contact with participants
Third monitoring cycle (2017/18)• Survey administered and analyzed
• Interviews conducted and analyzed
• Existing data analyzed
• Review indicators to ensure continued relevancy
• Third report produced
![Page 33: Developing Performance Measures through a Consultative Process](https://reader036.vdocument.in/reader036/viewer/2022062708/5587b20cd8b42a6b408b45d9/html5/thumbnails/33.jpg)
Strengths
Builds on the consultation for developing the indicators – people expressed interest in being involved on an ongoing basis
Covers all of the selected indicators
Cost-effective
Multiple lines of inquiry
Able to measure change over time
Able to measure statistical significance of change
![Page 34: Developing Performance Measures through a Consultative Process](https://reader036.vdocument.in/reader036/viewer/2022062708/5587b20cd8b42a6b408b45d9/html5/thumbnails/34.jpg)
Limitations
Participants dropping off from the survey
• Annual contact will help mitigate this
• A small token of appreciation can encourage ongoing participation
Self-selecting so cannot be generalized to the total population
• With a large enough sample size and tri-angulation with other lines of inquiry we can speak to trends
![Page 35: Developing Performance Measures through a Consultative Process](https://reader036.vdocument.in/reader036/viewer/2022062708/5587b20cd8b42a6b408b45d9/html5/thumbnails/35.jpg)
Questions?
Accessibility Directorate of OntarioCharene Gillies: [email protected]
Cathexis ConsultingMartha McGuire: [email protected] McGuire: [email protected]