Contenu connexe
Similaire à ONLINE EDUCA 2011 (18)
ONLINE EDUCA 2011
- 1. New Research Shows How to Use
Quizzes and Tests to Slow the
Forgetting Curve
Educa Online, December 1st 2011
John Kleeman, Questionmark Chairman
www.questionmark.com
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
- 2. Would you like to
reduce learner
forgetting curves?
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 2
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
- 3. Retrieval helps you learn
Francis Bacon, 1620
If you read anything
over twenty times, you
will not learn it by heart
so easily as if you were
to read it only ten, trying
to repeat it between
whiles, and when
memory failed, looking
at the book.
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 3
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
- 4. Learning
Receive Working Long term
information memory memory
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 4
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
- 5. Learning useful if can retrieve
• Question Research shows
Cue • Other stimulus retrieval practice
helps future retrieval
• Retrieve from long
term memory Successful retrieval
Search
modifies memory
making it more
• Act or respond retrievable in future
Act
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 5
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
- 6. Experimental methodology
Quiz / retrieval practice
Re-study group group
Study Study
Re-study Quiz
Measure Measure
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 6
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
- 7. Roediger & Karpicke experiment 2006
120 students divided into groups randomly
Given reading comprehension passages used in
TOEFL
Some students
Study for 14 mins
Some students
Study for 7 mins
Recall material for 7 mins
Compared results after 5 mins and 1 week
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 7
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
- 8. How much learners retained
90%
80%
70%
60%
50%
Restudy
40%
Quiz
30%
20%
10%
0%
After 5 mins After 1 week
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 8
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
- 9. Marsh & others 2007. Experiments
with multiple choice questions
60%
50%
40%
30% No quiz
Quiz
20%
10%
0%
SAT II Bloom Definition Bloom Application
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 9
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
- 10. Roediger & Butler. Feedback on
multiple choice questions
Retention after one week
60%
50%
40%
30%
20%
10%
0%
No quiz Quiz, no feedback Quiz, immediate Quiz, delayed
feedback feedback
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 10
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
- 11. Karpicke & Blunt experiment 2011
80 students in 4 groups, studying a science text
Study
Repeated Study
Concept Mapping (mind maps)
Retrieval Practice
Surveyed learners on expectation of retention
Tested a week later to see how much retained
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 11
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
- 12. How much learners retained
Actual retention
Learner expectation of
80%
retention 70%
80%
70% 60%
60% 50%
50% 40%
40% 30%
30% 20%
20% 10%
10%
0% 0%
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 12
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
- 13. Larson experiment 2009
Quizzes reduce forgetting curve
In real-life, not in the laboratory
Medical residents (doctors in training)
20 doctors in each group
Quiz group Restudy group
Day 1 Study Study
Day 1 Quiz Re-study
2 weeks later Quiz Re-study
2 weeks later Quiz Re-study
6 months later Final exam Final exam
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
- 14. Part of the results
Quiz group Re-study group
62%
48%
44%
36%
19%
Initial Quiz Quiz 6 months
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 14
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
- 15. Retrieval practice key findings
Retrieval practice helps retention of learning
Applies to facts
And also concepts
Retrieval practice includes
Recall
Formative quizzes/tests
Self-assessment
Explain or read back of material
Timing of retrieval
Works best when spaced out
Strongest effect when retrieval is difficult
Feedback very helpful
Repetition of retrieval
Research suggests 5-7 spaced repetitions helpful
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 15
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
- 16. Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 16 Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
- 17. Computer quizzes are effective
retrieval practice because …
Learners do not realize they will forget
Quizzes encourage/force retrieval practice
Quizzes give corrective feedback
And also:
Randomize so easy to repeat
Mobile devices
Accessible
Measure that practice has happened
Computer assessments also have many “indirect“
benefits
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 17
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
- 18. Indirect benefits of post-learning
assessments
Direct to further learning
Correct misconceptions
Feed back to improve the course
Compare to pre-test
Measure achievement
Encourage study
Engage students
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 18
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
- 19. If you are a practitioner …
Give quizzes after learning sessions
Focus questions on important things to remember
Make quizzes effortful (delay & difficulty)
Question types:
Recall (e.g. short answer) best
Multiple choice work too
Include feedback
Usually the correct answer
Slight delay if possible
Encourage retrieval (don’t allow peek ahead)
Spacing helps significantly
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 19
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
- 20. If you are a manager …
How easy is it for learners to take quizzes?
How easy is it for your instructors to quiz
learners?
Could you make it easier?
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 20
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
- 21. Potential objections
“No time”
“We teach creativity not facts”
“How sure are you of the research?”
“Our learners don’t like to be tested”
“What about privacy?”
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 21
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
- 22. In a Perfect World…
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 22
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
- 23. In a Perfect World…
we would remember everything we learn!
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 23
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
- 24. In the Actual World…
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 24
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
- 25. In the Actual World…
we forget some of it!
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 25
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
- 26. Questions through Learning Process
Reduce
Forgetting
Create
Intrigue
Knowledge/Skills
Strengthen
Memory
Questions
Questions
Questions
Questions
Questions
Time
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 26
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
- 27. References
Who Title Journal Date Description URL
Roegiger & Test-Enhanced Learning Psychological 2006 Widely cited paper Here
Karpicke Science showing benefits of
testing vs studying
Marsh, The memorial consequences Psychonomic 2007 Shows benefits of multiple Here
Roediger, of multiple-choice testing Bulletin & Review choice questions
Bjork&Bjork
Larsen, Test-enhanced learning in Medical Education 2009 Shows experiment on Here
Butler & medical education. medical education and
Roediger retrieval practice
Fazio, Right/wrong feedback: Memory 2010 Research showing just Here
Huelser, Consequences for learning telling students right or
Johnson & wrong without correct
Marsh answer unhelpful
Roediger & The critical role of retrieval Trends in Cognitive 2010 8 page review article on Here
Butler practice in long-term retention Sciences retrieval and retention
Karpicke & Retrieval Practice Produces Science Magazine 2011 Clear paper comparing Select from here
Blunt More Learning than Elaborative retrieval with study and (non-
Studying with Concept Mapping mind maps commercial)
Kleeman Recent cognitive psychology International CAA 2011 Some more detail from Here
research shows strongly that Conference me
quizzes help retain learning.
What does this mean1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Copyright ©
for CAA?
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
- 28. Some questions for you:
What is the best way to remember this session for
the future?
What question types seem to give best effect?
What is the best timing for feedback?
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 28
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
- 29. Your questions
Contact me at john at questionmark.com or
Twitter: @johnkleeman
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 29
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Notes de l'éditeur
- In the actual world…