Effect of language on the assessment of communication skills using a standardized measurement instrument in OSCE

1. A retrospective cohort study
RUBRICS USED TO ASSESS COMMUNICATION SKILLS OF U
The effect of language on the assessment of
communication skills using a standardized measurement
instrument in OSCEs
Winny Setyonugroho1; Maureen Kelly2; Kieran M Kennedy3; Eva Flynn2; Rosemary Geoghegan4; Ruth Murphy5; John J Morrison6;
Peter Hayes2; Thomas Kropmans7
OTTAWA 2016 PAPER UNDERGRADUATE ASSESSMENT

2. Without a standard measure, the comparison of Communication
Skills (CS) outcomes between different settings of an OSCE (i.e.
between students, modules, years and institutions) is very
challenging if not impossible.
Aim
• We examine the CS type – section 1, section 2, section 3 of the
MAAS-Global – that affect students' CS performance in OSCEs
• taking into consideration whether English is their first (L1) or
second (L2) language.

3. Method
• Retrospective cohort study evaluated the CS components of all OSCE rubrics
(i.e. assessment forms) used by 4 different disciplines in 4 yrs ME
• Three different cohorts of undergraduate medical students.
• A Two-way ANOVA was used for the comparison of the mean of MAAS-
Global score (MG Score) for each group of students and each of the
academic terms.
• G- and D- studies of the Generalizability Theory (GT) were used to calculate
the reliability of each OSCE.
• Significance levels were set at .05.

4. Summary of L1 and L2 students' country of origin.
Academic Year n of students n of students
1MB L1 403 (72%) L2 157 (28%)
Australia 1 Denmark 1
Canada 40 Finland 1
UK 6 France 1
Ireland 331 South Korea 1
Nigeria 1 Malaysia 151
Singapore 15 Norway 1
USA 9 Sweden 1
2MB L1 364 (79.5%) L2 94 (20.5%)
Australia 1 France 1
Canada 34 South Korea 1
UK 7 Malaysia 90
Ireland 300 Norway 1
Nigeria 1 Sweden 1
Singapore 12
Trinidad and
Tobago
1
USA 8
Result

5. 3MB L1 320 (81.6%) L2 72 (18.4%)
Australia 1 Bangladesh 1
Canada 26 Indonesia 1
Ireland 268 Iraq 1
Singapore 11 Malaysia 67
Trinidad and
Tobago
1 Poland 2
USA 6
UK 6
South Africa 1
4MB L1 299 (78.7%) L2 81 (21.3%)
Canada 15 Indonesia 1
Gibraltar 1 India 1
Ireland 261 Iraq 1
Singapore 8 Malaysia 74
Trinidad and
Tobago
1 Pakistan 1
USA 3 Poland 2
UK 9 Sudan 1
South Africa 1
Result

6. Result
• 1MB, 2MB, 3MB and 4MB OSCEs reliability ranged from G= 0.28
to G= 0.79 with a median of G=0.62.
• Rubrics calibration for the Disciplines of Medicine, General
Practice, Psychiatry, Obstetrics & Gynaecology, and Paediatrics
were 0.69, 0.83, 0.99, 0.45, and 0.79 respectively.
• Only in the Year 4 OSCEs did L1 students perform significantly
better than L2 students.
• This was the case for stations designed by General Practice (F(1,
368) = 21.46, p < .001),Psychiatry (F(1, 368) = 72.94, p < .001)
and Paediatric stations (F(1, 363) = 8.72, p < .003).
• No significant differences were found for stations designed by
Obstetrics & Gynecology.
INMED 2015

7. Academic year Academic Terms
MG
Proportion
Maas-Global ( in percent)
Section 1 Section 2 Section 3
Year 1 Communication 67%
Maas-Global ( in percent)
20% 23% 60%
Year 2 Communication 70%
Maas-Global ( in percent)
15% 18% 64%
Year 3 Communication 65%
Maas-Global ( in percent)
6% 8% 87%**
Year 4 Communication 65%
Maas-Global ( in percent)
14% 29% 70%
Result

8. Result
Introducing a gold standard Maas Global
• G-coefficient between OSCEs and Station is satisfactory >0.70 SEM 3%
• No statistical sign. difference section 1, 2 and 3 items of the MG score for yrs 1 and 2
• Statistical significant increase in section 3 items of the MG items in yr 3
• L1 and L2 don’t differ in yrs 1 and 2
• L1 and L2 differ statistically significant in yrs 3 and 4 (clinical years)
• L2 student significantly lower mean difference of 5% compared to L1

9. Discussion 1
• Single most important aspect of assessment is the measurement instrument
• A pre-requisite for excellent assessment is the validity and reliability (i.e.
reproducibility) of the assessment forms
• In most OSCE stations a combination of both CS and other clinical skills are
assessed in a combined fashion
• In the situation where CS instruments were not standardized, a calibration of
station scoring rubrics had to be executed in order for the results to become
comparable between years of study, participating OSCE contributors and
different groups of students (L1 and L2)

10. Discussion 2
• Information gathering is similar between L1 and L2 students in the
years 1, 2 and 3
• When it comes to analyzing the relevant information, clinical decision
making and treatment planning, the L2 students require more WM to
perform in what is for them a more stressful (non-native) environment
• The L2 students, in regarding their WM processes, need to first encode
the language, accents, unfamiliar terms and cues before they can focus
on their actual clinical task
• It should be noted that language is not only verbal communication, but
also includes terms, cues and health beliefs that are specific for the
native language of the L1 and the individual (standardized) patient.

11. Conclusion
• Calibration of OSCE scoring rubrics using the MAAS-Global
standard provides detailed information about communication
skills in an OSCE.
• Working memory theory helps to understand the differences
between L1 and L2 students’ OSCE CS scores, as they progress
through the medical curriculum and are assessed in increasingly
complex OSCE scenarios.
• Future theory driven research using Working Memory should
examine if providing additional time in the OSCE to L2 students
would result in improved OSCE CS performance.

12. Universal adoption of a standardized Communication Skills
measurement instrument (Maas-G) would allow comparison of
student competence across institutions and OSCE settings.
Take home message

13. Thank You