CANNEXUS 2010 presentation: Measuring effectiveness of career interventions: presentation examines how we can measure effectiveness of a career intervention using a variety of methods, both objective and subjective. results, challenges and future directions are explored.

1. HOW DO YOU KNOW IT
WORKS?
Measuring the Effectiveness of
Career Counselling
Interventions
Joan McCurdy Myers
Felicity Morgan

2. HOW DO YOU KNOW IT WORKS?
• UTMississauga Career Centre
• Career Counselling Interventions and
Evaluation
• Ace Your Career Case: content and
evaluation
• Results
• Challenges of integrating practice and
research
• Next steps

3. UTMississauga CAREER
CENTRE
• 10,500 students, mainly undergraduate
• Broad range of academic programs
• Large proportion of commuter students, living at home, GTA
• Services include career counselling, employment advising,
workshops, panels, employer events, job postings
• Staffing: 15 non-student staff
– 7(5.4 fte) staff in Career Development Unit
– 4 Employer Relations & Marketing
– 3 admin
– 18 student staff
• Partnerships with academic programmes & student groups: key to
tailoring for student needs

4. CAREER COUNSELLING
INTERVENTIONS
• Individual, workshops & events
• Some in class, some not
• Workshops & events:
– Evaluate using post session evaluations,
learning objectives, follow up surveys
– Students report high levels of satisfaction with
sessions and meeting learning objectives
• Standard: 75% rate 4/5 on 5 point scale

5. EVALUATION OF
INTERVENTIONS
• In 2006, decided to go beyond post-session
satisfaction/feedback & surveys & attempt to
measure what the students were truly learning
– Our own questions
– Learning outcomes focus at UT
– Student Research Assistant position
– Funding: through the Enhancing the Student
Experience Fund for two projects; Ace Your Career
Case and Intro Psych student research credit for
Career Development workshop

6. ACE YOUR CAREER CASE
• 2nd and 3rd year Management and
Commerce students
• Goal: career goal clarity: through self
assessment, gathering of information and
interaction with industry professionals
• Held annually in January for three years;
facilitated by career counsellor and job
coach consultant

7. ACE YOUR CAREER CASE
• 2 half days;
Day One - self assessment through career
counselling exercises; career research
methods and sources
Day Two – business introduction development
and practice; lunch and interaction with
employers; reflection on what was learned
and next steps
• Co-sponsored with Management
Department, funded by Provost’s Office

8. LEARNING OUTCOMES
• Identify clear career goal, based on self
knowledge and industry knowledge
• Locate and use career research
resources
• Develop, practice and deliver business
introduction
• Understand benefits and opportunities
available in employer events
• Participate more effectively in recruitment

9. METHODS OF EVALUATION
• Post workshop evaluation
• Follow up survey
• Video-taped business introduction (years 2
& 3)
• Students completed Career Decision
Profile pre Day One and post Day Two

10. CAREER DECISION PROFILE
• Developed by Prof. L. Jones (emeritus), North
Carolina State, President Career Key Inc.
• Measures:
– Decidedness
– Comfort
– Self Clarity
– Knowledge about Occupations & Training
– Decisiveness
– Career Choice Importance

11. WHY THE CDP?
• Previously used it with at risk students;
gave them clarity and purpose
• Scales break down the components of
career planning, separates them out
• Questions are process oriented, not final
answer oriented; fits with our approach
• Can be used as a counselling tool
• Had support from the author

12. CDP SCALES
• Decidedness: how decided are you about
your choice of career?
• Comfort: how comfortable are you with
where you are in the process of making
this choice?
• Self Clarity: how clear is your
understanding of your abilities, interests
and personality and how they fit with
careers

13. SELF CLARITY SAMPLE
QUESTIONS
• I wish I knew which occupations best fit my
personality
• I need to have a clearer idea of what my
interests are
• I need to have a clearer idea of my abilities, my
major strengths and weaknesses.
• 8 point scale; strongly disagree to strongly agree

14. CDP SCALES
• Knowledge: how well informed you
believe you are about occupations and
educational programs that fit your interests
• Decisiveness: how capable you believe
you are in making decisions without
unnecessary delay
• Career Choice Importance: importance of
choosing a career at this time

15. HOW EFFECTIVE WAS ACE?
• Results from post workshop evals
• Results from follow up survey
• Results from video-taped business intro
• Results from CDP

16. RESULTS
• Post-workshop Evaluation
– 100% recommended to a friend every year
– Met learning outcomes criteria re: clarity of career
goal, strengthened career plan, more effective
participation in recruitment,
– Did not meet learning outcomes criteria for:
• The strength of their self marketing strategies
• Satisfaction (wanted more time with industry professionals)
– Sometimes met: resources and how to use
opportunities

17. RESULTS
• Follow Up Survey (one month post)
– Overall, students felt all measures increased
significantly from pre-ACE to post-ACE
• they understood themselves and the work world
better
• they were more prepared for the recruitment
season
• their career plan was more thorough and clear
• their ability to self market had improved

18. RESULTS
• Video-taped Business Intro
– Yr 2- 45% scored in 7-10 range, average 6.35
– Yr 3-66% scored in 7-10 range, average 7.4
– Identifying 2 key strengths was a challenge

19. RESULTS
• CDP
– Overall, scores moved in positive directions
• Decidedness yielded mixed results
– Students significantly (<.05) increased their
scores in knowledge 2 of 3 years (year 1
<.10)
– Year 3 saw the most significant changes:
comfort, self clarity, knowledge
• Decidedness (significant but decreased)
• Decisiveness (approaching significant <.10)
– Importance of Career Decision

20. RESULTS-for our student
programming
• Great opportunity to experiment with new
teaching/learning approaches
– Business introduction (improved our teaching; used same
assessment in subsequent sessions)
– Positive response of industry reps and students to reps
encouraged us to incorporate format in more of our
activities
– Sharpened our focus on how to help students learn to
reflect-incorporated direct reflection opportunities in a
range of programming
– Style of greater student engagement on the spot also now
used in Fall Management Conference with very positive
results

21. CONCLUSIONS: How do you
know it works?
• Take multiple perspectives to see what works
– Students’ perceptions, satisfaction, learning
outcomes, feedback
– “objective” measures e.g. CDP, behavioural
measurement e.g. business intros, 3rd party
evaluations, rubrics
• Student participation in services, recruitment activities
• Student “success” e.g. # interviews at recruitment
– Develop database for comparison e.g. UTM CDP
• Use results to improve programming

22. INTEGRATING PRACTICE &
RESEARCH
• Proper evaluation without too much
evaluation
• Need more time & expertise to plan,
analyze, understand & use results
• Difficulty of balancing service
provision with need for more
structured assessment
• Speaking the language of the
academic e.g. statistical results

23. INTEGRATING PRACTICE &
RESEARCH
• Managing partnerships with academics is
challenging
• Gratifying to be able to document & report
changes
• Increased our skills and knowledge as
professionals; identified new learning goals
• Benefits other students through improved
programming

24. NEXT STEPS
• Using CDP in other interventions; assists us to
assess where our students are at the beginning
and improve our interventions to increase positive
outcomes
• Follow up with participants
• Encouraged us to go beyond satisfaction
measures; now “see” more opportunities for
broader range of assessment/evaluation
• Increasing our expertise as a Centre & division
• Looking for more partnerships

25. QUESTIONS?
Joan McCurdy-Myers
joan.mccurdy.myers@utoronto.ca
Felicity Morgan
felicity.morgan@utoronto.ca