Developing Performance Measures Through a Consultative Process

124 Merton St., Suite 502
Toronto, Ontario M4S 2Z2
Telephone: (416) 469-9954
Fax: (416) 469-8487
www.cathexisconsulting.ca
Developing Performance
Measures Through a
Consultative Process
June 2013

Purpose of the Presentation
 To provide an example of transformative use of
performance measurement

We will cover. . .
 Background on the Accessibility for Ontarians
with Disabilities Act
 An overview on the development of performance
indicators to measure the extent to which the
Act is really making a difference
 An introduction to a method that can help with
the selection of indicators when there are
multiple stakeholders
 An overview of the monitoring methods

Background on AODA
 The Accessibility for Ontarians with Disabilities Act, 2005 (AODA)
was adopted to improve accessibility for the 1.85 million people
with disabilities in Ontario
 The legislation covers five domains:
• Customer Service
• Information and Communication
• Employment
• Transportation
• Built Environment
 Standards have been development relating to each of those
domains

Questions to be Addressed
 What dimensions (health, education, recreation, etc)
should be considered when developing performance
indicators to measure the AODA’s impact on people with
disabilities, older adults and their families
 Which indicators can best account for the diversity among
people with disabilities, older adults and their families (i.e.
types of disabilities, degree of disability, age)
 Which indicators will best measure quality of life
improvements for people with disabilities, older adults and
their families
 What data collection tools can be used measure the
performance indicators over time and with a limited
budget

Key Focus
 The extent to which the standards have had a
positive impact on Ontarians with disabilities,
older adults and their families

Developing the Indicators
 Understanding the ADO Standards
 Literature review
 Consultation with people with disabilities, older adults and
their families
• Out reach through a number of vehicles
• On-line survey with 426 respondents
• Focus groups in Huntsville, Toronto, Ottawa, Thunder Bay and
London with a total of 58 participants
• Social media blog with no response
• Submission (1)
 Priority Sort process including a rapid sort and a more
thoughtful sort
 Finalizing indicators

Findings from the Literature
 Not agreed-upon definition – subjective to each
individual
 Schalock and his colleagues developed a
framework that focuses on empowerment rather
than disability
 Canadian Index of Well-being looks at indicators
for that cover all Canadians
 Quality of Life Research Unit at the University of
Toronto’s Center for Health Promotion: “The
degree to which a person enjoys the important
possibilities of his or her life”

Quality of Life Research Unit, Center for Health Promotion, University of Toronto
Key Elements of a QOL Framework
 Being: Who One Is
• Physical Being
• Psychological Being
• Spiritual Being
 Belonging: Connection with One’s Environment
• Physical Belonging
• Social Belonging
• Community Belonging
 Becoming: Achieving Personal Goals, Hopes and
Aspirations
• Practical Becoming
• Leisure Becoming
• Growth Becoming

Findings from the Consultation
 Most respondents indicated that most indicators
were important
 Provision of health care services was considered
to be the most important
 Gave advice on wording and consolidation of
indicators
 Based on the findings seven high level indicators
and 26 specific indicators emerged

The Priority Sort Process
 Want you to know . . .
• What Group Priority Sort is
• Why you might use Group Priority Sort (what
the benefits are)
• When it is (and is not) appropriate to use
Group Priority Sort

What is Priority Sort?
A participatory priority-ranking exercise
It can be used to:
 Define the scope of an evaluation
 Select performance measures/indicators
 Prioritize strategic planning goals
 Define a complex concept

What is Priority Sort?
 Has small groups of stakeholders or
“experts” rank-order specified items
 The outputs are:
a) comparative rankings:
b) rich qualitative data; and
c) engaged participants
 Evolved out of Q Methodology and
produces similar information as Delphi

The Priority Sort:
Two Step Process

Recent Group Priority Sort: Quality of Life
Indicators for People with Disabilities

Sample Instructions:
Forced Sort
Now it gets more difficult...
 Refine your sort so that there are no more than
6 cards in each category
 Try to come to agreement about where each
benefit should go
 Use blank cards to record any other important
indicators (do not sort these)
 You have 25 minutes

Benefits of Priority Sort
Informed decision-making;
Enhanced understanding of the topic among
participants;
Strengthened community of stakeholders;
Confidence in the process and the resulting
decisions; and
An enjoyable and engaging experience for
participants.

When to use Priority Sort
 Not the right technique if you are exploring a
new area that nobody knows much about.
 Not a brainstorming exercise.
 Priority Sort captures subjective opinions. If
you are seeking objective facts, this is not the
right method.
 The Priority Sort is most useful when there are
divergent opinions.

Recommended Indicators
 Agreement on the high level indicators
• The over-arching indicators are:
• Being treated with dignity
• Feeling an integrated part of school, work and the community
• One high level indicator for each Standard
• Customer service: Being served in a way that maintains dignity and
supports inclusion
• Employment: Have access to employment consistent with experience,
abilities and training
• Information: Have access to information
• Transportation: Able to get where you need to go, when you need to go
• Design of Public Spaces: Able to move independently around the
community
 Reduced the 26 indicators to 12

Customer Service
Level of confidence that health care providers can provide services in a way
that takes into account the person’s disability
• links back to physical being, psychological being and physical belonging in the
project’s methodological framework.
Educational institutions provide services in a respectful manner
• links to psychological being, social belonging, practical becoming and growth
becoming in the project’s methodological framework.
Emergency response services are provided in a manner that takes into
account the person’s disability
• links to physical being, psychological being, and community belonging in the
Recreation/fitness facilities provide services are provided in a way that allow
people with disabilities to use and benefit from them
• links to physical being, physical belonging and leisure becoming in the

Employment
Accommodation is provided in the workplace for people with
disabilities
• links to practical becoming in the project’s methodological
framework.
Managers and co-workers at all levels accept and make
accommodation for people with disabilities
• links to social belonging and practical becoming in the project’s
methodological framework.
Career development opportunities are provided in a manner that
accommodates people with disabilities
• This indicator links to psychological being and practical becoming in
the project’s methodological framework.

The Recommended Indicators
Information
Information about local warnings/emergencies is available to and
can be accessed by people with disabilities
• links to physical being, community belonging and practical becoming
in the project’s methodological framework.
Accessibility planning includes input from people with disabilities
• links to psychological being and community belonging in the project’s
methodological framework.
 Websites are designed so that people with disabilities can access
them
• links to social belonging, community belonging, and practical
becoming in the project’s methodological framework.

Transportation
Public transportation, taxis and GO trains are equally accessible for
people with disabilities as for people without disabilities
• Links to psychological being, and community belonging in the
project’s methodological framework
Design of Public Spaces
People with disabilities are able to get to stores, community centres
and other public facilities
• This indicator links to social belonging and community belonging in
the project’s methodological framework

Learnings from this Process
 People with disabilities are interested in participating in consultations
related to accessibility want to see that the ADO is taking the impact of
the AODA seriously.
 The most effective recruitment occurred through organizations
representing or serving the population
 The survey provided an excellent sense of what was important for people
with disabilities and supported reaching a large number of people
 The face-to-face consultation augmented this understanding plus
provided input into acceptable wording.
 A large number of people with disabilities have access to the technology
required to participate in on-line surveys
 The steps taken to accommodate people with disabilities, as part of the
consultation process, was noted and appreciated by participants.
 It is still challenging to find facilities that are fully accessible
 When conducting consultations, it is important to set the parameters in a
way that keeps the discussion focused, but does not disrespect the
importance of the other issues being raised.

Research Questions
 To what extent have there been changes in
quality of life as it relates to the five AODA
Standards for people with disabilities, older
adults and their families?
 What impact do the Accessibility Standards have
on key areas of daily living areas for people with
disabilities, older adults and their families?

Suggested Monitoring Method
On-line survey
Interviews
Use of existing data

Monitoring Cycle
 Baseline (2013/14)
• Recruit monitoring participants
• Survey administered and analyzed
• Interviews conducted and analyzed
• Existing data analyzed
• Baseline report produced
 Interim year (2014/15)
• Contact with participants
 Second monitoring cycle
(2015/16)
• Second report produced
 Interim year (2016/17)
• Contact with participants
 Third monitoring cycle
(2017/18)
• Review indicators to ensure
continued relevancy
• Third report produced

Strengths
 Builds on the consultation for developing the
indicators – people expressed interest in being
involved on an ongoing basis
 Covers all of the selected indicators
 Cost-effective
 Multiple lines of inquiry
 Able to measure change over time
 Able to measure statistical significance of change

Limitations
 Participants dropping off from the survey
• Annual contact will help mitigate this
• A small token of appreciation can encourage
ongoing participation
 Self-selecting so cannot be generalized to the
total population
• With a large enough sample size and tri-angulation
with other lines of inquiry we can speak to trends

Questions?
Accessibility Directorate of Ontario
Charene Gillies: Charene.Gillies@ontario.ca
Cathexis Consulting
Martha McGuire: martha@cathexisconsulting.ca
Melissa McGuire: melissa@cathexisconsulting.ca

Developing Performance Measures Through a Consultative Process

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Developing Performance Measures Through a Consultative Process

Similaire à Developing Performance Measures Through a Consultative Process (20)

Plus de CesToronto

Plus de CesToronto (20)

Dernier

Dernier (20)

Developing Performance Measures Through a Consultative Process