3. Taken from Vanessa Burch 2007 http://publishing.eur.nl/ir/repub/asset/10152 We used to think of format as sufficient enough an indicator of validity
4. Taken from Vanessa Burch 2007 http://publishing.eur.nl/ir/repub/asset/10152 In truth, tables such as this one are a very good guide but ….
5. MCQs and EMQs What can you measure with these formats? There are no guarantees!
6. MCQs testing core knowledge in 5 th year medical students Suppose we send faculty this request for material for a forthcoming test
7. What is the most common symptom in Conn’s syndrome? What is a tourniquet? Are blue eyes>brown? Which muscle is responsible for lateral rotation and extension of the hip joint and whose contraction supports the extended knee? Name 5 Tx for angina Are fractures>suicides? And suppose these are the items we receive in response What do you make of them?
8. TEST PURPOSE: GRADUATION Utility theory (van der Vleuten 1996) Utility = Validity x Reliability x Acceptability x Educational impact x Cost Suppose the purpose of the test we are planning is to graduate students - if we use utility theory to roughly calculate how useful our test is – how does it fare?
9. What is the most common symptom in Conn’s syndrome? What is a tourniquet? Are blue eyes>brown? Which muscle is responsible for lateral rotation and extension of the hip joint and whose contraction supports the extended knee? Name 5 Tx for angina Are fractures>suicides? … it doesn’t fare very well! This is partly because: we didn’t share the test purpose, the level we wanted to test at, and perhaps most important of all - we didn’t train our item writers!
10. Validity Face validity – given test purpose, are the items seemingly of value Content validity – do the test items approximate the intended content array Convergent validity – do results correlate with measures of the same construct Divergent validity – do results lack correlation with measures of other constructs Consequential validity – intended future consequences of the decision Understanding validity is key – getting item content right means you can be more assured of what you are really measuring
11. Achievement decisions “ Content is more important than format” Lambert Schuwirth Manchester Medical Education Conference, March 2009 low pass MCQ OSCE SAQ miniCEX 360 Feedback Report Diagnosis 30% History taking 35% Medicine 40% Surgery 35% Paediatrics 45% O&G 40% Psychiatry 43% Sciences 50% Research 55% Why not look across tests for information about performance?
12. Achievement decisions “ Reliability via sufficient sampling, measured across tests” Lambert Schuwirth, Manchester Medical Education Conference, March 2009 MCQ OSCE SAQ miniCEX 360 Reliability report Diagnosis α =0.70 History taking α =0.75 Medicine α =0.78 Surgery α =0.72 Paediatrics α =0.71 O&G α =0.79 Psychiatry α =0.82 Sciences α =0.70 Research α =0.90 Multiple sources of information, accrued over time for the best measurement of score reliability
13. Achievement decisions “ Assessment is neither formative nor summative. Assessment is continuous, and supportive” Lambert Schuwirth, Manchester Medical Education Conference, March 2009 OSCE MCQ miniCEX 360 SAQ Feedback Report Diagnosis 30% History taking 35% Medicine 40% Surgery 35% Paediatrics 45% O&G 40% Psychiatry 43% Sciences 50% Research 55% low pass Feedback Report Diagnosis 40% History taking 45% Medicine 50% Surgery 55% Paediatrics 55% O&G 60% Psychiatry 53% Sciences 50% Research 55% pass Piece the evidence together and discuss with students in a supportive fashion to create better learning
15. Which muscle is responsible for lateral rotation and extension of the hip joint and whose contraction supports the extended knee? A 65 year old man presents to his General Practitioner complaining of difficulty getting up out of his chair. On examination he has difficulty rising out of the chair and straightening his trunk. He has no difficulty in straightening his leg. What muscle is most likely to have been injured? What level of Bloom’s taxonomy?
16. What is the most common tumour cell? How does Myasthenia Gravis most commonly present? What is the most common symptom in Conn’s syndrome? Which of the following is a direct thrombin inhibitor? What about these, what level of Bloom’s taxonomy?
17. What about these, what level of Bloom’s taxonomy? What is the most appropriate treatment? What nerve is most likely injured? What is the most appropriate interpretation of acid base status? To whom should this now be reported?
18. Well, with these questions, you need a scenario – they work to test “application: using knowledge in new situations” What is the most appropriate treatment? What nerve is most likely injured? What is the most appropriate interpretation of acid base status? To whom should this now be reported?
19.
20. Stem Lead in Options Here and now scenario or problem - a typical permutation of a key theme A question which requires use of a key skill, e.g. “diagnosis” and which probes the stem scenario in a relevant way Five options – one most likely
21. Here, the scenario is ok but could be made more authentic – do patients ever phrase a presenting complaint in this way? Stem A 28 year old man complains of recurrent dull pains in the occipital region. The pain is present most days, and has been occurring for several months. He gets partial relief from paracetamol. There are no other associated symptoms. Examination shows some mild local tenderness in the region of the pain. Lead in What is the most likely diagnosis? A Brain metastases B Cluster headache C Migraine D Subarachnoid haemorrhage E Tension headache Written by a UMAP contributor at Sheffield Medical School
22. Stem A 28 year old man complains of recurrent dull pains in the occipital region. The pain is present most days, and has been occurring for several months. He gets partial relief from paracetamol. There are no other associated symptoms. Examination shows some mild local tenderness in the region of the pain. Lead in What is the most likely diagnosis? A Brain metastases B Cluster headache C Migraine D Subarachnoid haemorrhage E Tension headache Written by a UMAP contributor at Sheffield Medical School Make scenarios as realistic as possible and where test results are returned, include all as data, not in summarised form.
23. Stem A 28 year old man complains of recurrent dull pains in the occipital region. The pain is present most days, and has been occurring for several months. He gets partial relief from paracetamol. There are no other associated symptoms. Examination shows some mild local tenderness in the region of the pain. Lead in What is the most likely diagnosis? A Brain metastases B Cluster headache C Migraine D Subarachnoid haemorrhage E Tension headache Written by a UMAP contributor at Sheffield Medical School The question is measuring diagnostic skill – when creating questions identify the lead in question first, and then select a fitting, typical scenario for context
24. Stem A 28 year old man complains of recurrent dull pains in the occipital region. The pain is present most days, and has been occurring for several months. He gets partial relief from paracetamol. There are no other associated symptoms. Examination shows some mild local tenderness in the region of the pain. Lead in What is the most likely diagnosis? A Brain metastases B Cluster headache C Migraine D Subarachnoid haemorrhage E Tension headache Written by a UMAP contributor at Sheffield Medical School Feedback openly justifies why the correct answer is the most likely option, for reviewers as well as students Do not write your questions alone! Share them with others, and justify your answers Feedback This is because the location and nature of the pain occurring over a prolonged period of time without associated symptoms, such as nausea or vomiting is characteristic of tension headache.
25. Stem A 28 year old man complains of recurrent dull pains in the occipital region. The pain is present most days, and has been occurring for several months. He gets partial relief from paracetamol. There are no other associated symptoms. Examination shows some mild local tenderness in the region of the pain. Lead in What is the most likely diagnosis? A Brain metastases B Cluster headache C Migraine D Subarachnoid haemorrhage E Tension headache Written by a UMAP contributor at Sheffield Medical School You might choose to explain why each of the incorrect answers is wrong, or just give a general tutorial Feedback This is because the location and nature of the pain occurring over a prolonged period of time without associated symptoms, such as nausea or vomiting is characteristic of tension headache.
26. The stem scenario Avoid construct irrelevant variance (CIV) Also known as “content specificity” The stem content should not overtake the construct Choose typical scenarios e.g. the “consequences” that a graduation decision leads to Ensure that across a test you have sufficient questions targeting the skills areas, e.g. “diagnosis”, that you have deemed important this way you are more likely to arrive at a reliable set of scores
27. Stem A 28 year old man complains of recurrent dull pains in the occipital region. The pain is present most days, and has been occurring for several months. He gets partial relief from paracetamol. There are no other associated symptoms. Examination shows some mild local tenderness in the region of the pain. Lead in What is the most likely diagnosis ? A Brain metastases B Cluster headache C Migraine D Subarachnoid haemorrhage E Tension headache Written by a UMAP contributor at Sheffield Medical School Test for ambiguity! Ambiguous questions will derail the best students!......
28. Stem A 28 year old man complains of recurrent dull pains in the occipital region. The pain is present most days, and has been occurring for several months. He gets partial relief from paracetamol. There are no other associated symptoms. Examination shows some mild local tenderness in the region of the pain. Lead in What is the most likely diagnosis ? Written by a UMAP contributor at Sheffield Medical School … Could you answer the question without needing to see the answer options?
29. Stem A 28 year old man complains of recurrent dull pains in the occipital region. The pain is present most days, and has been occurring for several months. He gets partial relief from paracetamol. There are no other associated symptoms. Examination shows some mild local tenderness in the region of the pain. Lead in What is the most likely diagnosis ? Written by a UMAP contributor at Sheffield Medical School If you need to see the answer options to find out what you are being asked to do, the question needs work to become clearer.
30. Stem A 28 year old man complains of recurrent dull pains in the occipital region. The pain is present most days, and has been occurring for several months. He gets partial relief from paracetamol. There are no other associated symptoms. Examination shows some mild local tenderness in the region of the pain. Lead in What is the most likely diagnosis ? Written by a UMAP contributor at Sheffield Medical School If you need to see the answer options to find out what you are being asked to do, the question needs work to become clearer. A Brain metastases B Cluster headache C Migraine D Subarachnoid haemorrhage E Tension headache
31. The type of content needed to ensure the question tests application of knowledge: Data heavy! More description, more data, everything typical, no idiosyncrasies.....
32. The type of content needed to ensure the question tests application of knowledge: Data heavy! More description, more data, everything typical, no idiosyncrasies.....
33. The type of content needed to ensure the question tests application of knowledge: Data heavy! More description, more data, everything typical, no idiosyncrasies..... If your reaction to this is “SURELY THAT MAKES THE TASK TOO EASY…”
34. The type of content needed to ensure the question tests application of knowledge: Data heavy! More description, more data, everything typical, no idiosyncrasies..... … Then remember – as an expert you chunk information together very FAST
35. The type of content needed to ensure the question tests application of knowledge: Data heavy! More description, more data, everything typical, no idiosyncrasies..... If the summarised and the full data is put in front of you, your reaction time in either case is the same….
36. The type of content needed to ensure the question tests application of knowledge: Data heavy! More description, more data, everything typical, no idiosyncrasies..... … Not so for students who have not yet mastered this “pattern recognition” skill
37. What is the most likely renal abnormality in children with nephrotic syndrome and normal renal function? A B *C* D E Facility 1 0 99 0 0 94 8 1 90 1 0 A 2-year-old boy has a 1-week history of oedema. Blood pressure is 100/60 mm Hg, and there is generalized oedema and ascites. Serum concentrations are: creatinine 0.4 mg/dL, albumin 1.4 g/dL, and cholesterol 569 mg/dL. Urinalysis shows 4+ protein and no blood. 0 0 98 2 0 88 5 2 82 8 1 A 2-year-old black child developed swelling of his eyes and ankles over the past week. Blood pressure is 100/60 mm Hg, pulse 110/min, and respirations 28/min. In addition to swelling of his eyes and 2+ pitting edema of his ankles, he has abdominal distension with a positive fluid wave. Serum concentrations are: creatinine 0.4 mg/dL, albumin 1.4 g/dL, and cholesterol 569 mg/dL. Urinalysis shows 4+ protein and no blood. 0 1 98 1 0 84 10 9 66 10 5 From Susan M. Case & David B. Swanson: Constructing Written Test Questions, 2002 h ttp://www.nbme.org/about/itemwriting.asp. From Susan M. Case & David B. Swanson: Constructing Written Test Questions, 2002 h ttp://www.nbme.org/about/itemwriting.asp. Look at COLUMN C – showing top and bottom quintile results
38. What is the most likely renal abnormality in children with nephrotic syndrome and normal renal function? A B *C* D E Facility 1 0 99 0 0 94 8 1 90 1 0 A 2-year-old boy has a 1-week history of oedema. Blood pressure is 100/60 mm Hg, and there is generalized oedema and ascites. Serum concentrations are: creatinine 0.4 mg/dL, albumin 1.4 g/dL, and cholesterol 569 mg/dL. Urinalysis shows 4+ protein and no blood. 0 0 98 2 0 88 5 2 82 8 1 A 2-year-old black child developed swelling of his eyes and ankles over the past week. Blood pressure is 100/60 mm Hg, pulse 110/min, and respirations 28/min. In addition to swelling of his eyes and 2+ pitting edema of his ankles, he has abdominal distension with a positive fluid wave. Serum concentrations are: creatinine 0.4 mg/dL, albumin 1.4 g/dL, and cholesterol 569 mg/dL. Urinalysis shows 4+ protein and no blood. 0 1 98 1 0 84 10 9 66 10 5 From Susan M. Case & David B. Swanson: Constructing Written Test Questions, 2002 h ttp://www.nbme.org/about/itemwriting.asp. From Susan M. Case & David B. Swanson: Constructing Written Test Questions, 2002 h ttp://www.nbme.org/about/itemwriting.asp. Item 1 tests factual knowledge – top and bottom quintiles answer correctly most of the time 1 2 3
39. What is the most likely renal abnormality in children with nephrotic syndrome and normal renal function? A B *C* D E Facility 1 0 99 0 0 94 8 1 90 1 0 A 2-year-old boy has a 1-week history of oedema. Blood pressure is 100/60 mm Hg, and there is generalized oedema and ascites. Serum concentrations are: creatinine 0.4 mg/dL, albumin 1.4 g/dL, and cholesterol 569 mg/dL. Urinalysis shows 4+ protein and no blood. 0 0 98 2 0 88 5 2 82 8 1 A 2-year-old black child developed swelling of his eyes and ankles over the past week. Blood pressure is 100/60 mm Hg, pulse 110/min, and respirations 28/min. In addition to swelling of his eyes and 2+ pitting edema of his ankles, he has abdominal distension with a positive fluid wave. Serum concentrations are: creatinine 0.4 mg/dL, albumin 1.4 g/dL, and cholesterol 569 mg/dL. Urinalysis shows 4+ protein and no blood. 0 1 98 1 0 84 10 9 66 10 5 From Susan M. Case & David B. Swanson: Constructing Written Test Questions, 2002 h ttp://www.nbme.org/about/itemwriting.asp. From Susan M. Case & David B. Swanson: Constructing Written Test Questions, 2002 h ttp://www.nbme.org/about/itemwriting.asp. But items 2 and 3 test application – look at the drop off for the bottom quintile: 82 becomes 66 1 2 3
40. What is the most likely renal abnormality in children with nephrotic syndrome and normal renal function? A B *C* D E Facility 1 0 99 0 0 94 8 1 90 1 0 A 2-year-old boy has a 1-week history of oedema. Blood pressure is 100/60 mm Hg, and there is generalized oedema and ascites. Serum concentrations are: creatinine 0.4 mg/dL, albumin 1.4 g/dL, and cholesterol 569 mg/dL. Urinalysis shows 4+ protein and no blood. 0 0 98 2 0 88 5 2 82 8 1 A 2-year-old black child developed swelling of his eyes and ankles over the past week. Blood pressure is 100/60 mm Hg, pulse 110/min, and respirations 28/min. In addition to swelling of his eyes and 2+ pitting edema of his ankles, he has abdominal distension with a positive fluid wave. Serum concentrations are: creatinine 0.4 mg/dL, albumin 1.4 g/dL, and cholesterol 569 mg/dL. Urinalysis shows 4+ protein and no blood. 0 1 98 1 0 84 10 9 66 10 5 From Susan M. Case & David B. Swanson: Constructing Written Test Questions, 2002 h ttp://www.nbme.org/about/itemwriting.asp. From Susan M. Case & David B. Swanson: Constructing Written Test Questions, 2002 h ttp://www.nbme.org/about/itemwriting.asp. In item 3 (still not a perfect item) there is more information to synthesise together compared to 2 1 2 3
41.
42. Key points for good applied MCQ writing No medical terms in presenting complaints - I have a pain in my occipital region No summaries of test or examination findings - Use data and full descriptions Not just one test result or one examination finding – All of them
43. Theme: Breast disease Focus: Diagnosis A 73 year old woman presents with a fixed hard mass in the upper outer quadrant of the right breast. There is an associated lymphadenopathy. A mammogram reveals a spiculated density corresponding to the clinical abnormality. What is the most likely diagnosis? A. Breast cancer in pregnancy B. Gynaecomastia C. Lactational abscess D. Post menopausal breast cancer E. Pre-pubertal breast bud hypertrophy AVOID THIS! The answers should all be feasible on the key parameters in the stem! Here everyone answers correctly! Some because they know the topic, others because of the cue! Flaw no. 1
44. Theme: Urinary urgency Focus: Investigations A 35 year old woman presents with a 12 month history of gradually worsening urinary frequency, urgency and urge incontinence. She has to get up once or twice in the night to pass urine. There is no dysuria. She has two children aged two and five years. Her periods are regular and monthly What is the investigation most likely to lead to a diagnosis? A. Cystometry B. Intravenous pyelogram C. Micturating cystogram D. Ultrasound examination of bladder E. Urodynamics AVOID THIS! The answers be mutually exclusive! Cystometry is part of Urodynamics, yet the author said Cystometry was correct! Here noone answers correctly! Apart from those who guess! Flaw no. 2
45. Theme: Psychiatric complaints Focus: Treatment (Case) What is the most appropriate therapy? A. Amitriptyline and chlordiazepoxide 50mg BD B. Amitriptyline BD or chlordiazepoxide C. Chlordiazepoxide TD and Family therapy D. Family therapy without chlordiazepoxide E. Chlorpromazine 50mg AVOID THIS! The answers have added difficulty to the question – irrelevant difficulty! Here you have made sure that those who have the knowledge you are trying to measure can’t answer your question! Flaw no. 3
46. Theme: Cardiovascular system Focus: Anatomical features What is the best way to a man’s heart? A Through his aorta B Through his stomach C Through his chest D Via cardiac surgery E The same as the way to a woman’s Adapted from Susan M. Case & David B. Swanson: Constructing Written Test Questions, 2002 h ttp://www.nbme.org/about/itemwriting.asp. AVOID THIS! This is why people should write questions and quality assure them together. Subjectivity undermines good question writing. Insist on consensus agreement for every item! Problem 1 Flaw no. 4
