This webinar provides practical information on how to use published research findings and make contact with cognitive scientists in order to improve K-12 and university students’ learning from digital online resources, like Khan Academy videos or interactive mathematics exercises. The webinar focuses on how students’ motivation and grades have been increased by helping them believe they can take charge of their learning and become smarter, and how students can be supported in reflective thinking and seeking deep understanding, when questions and prompts for students to explain are inserted in videos and interactive exercises

1. Improving Online
Educational Resources
using Cognitive Science
Joseph Jay Williams
Office of Online Learning
Stanford University
September 2013

2. Improving Online Educational Resources using Cognitive Science
(& Online Collaborations between Scientists and Educators)
Joseph Jay Williams
josephjaywilliams@stanford.edu
www.josephjaywilliams.com/education
Lytics Lab, Office of Online Learning, Graduate School of Education
Stanford University
Slides at tiny.cc/inacolwebinar
Slides & Discussion at tiny.cc/inacolwebinar 2

3. What’s different about Online Educational Resources?
• A lot!
• Align Scientists & Educators
– Practical Improvements
– Scientific Research
• Refine Resources through Repeated Improvement
• Facilitate Collaboration
• Like Wikipedia?
3
Content Exercise

4. Overview
• I. Applying cognitive science to online learning
– I-a: How to increase motivation?
– I-b: Increase learning without changing materials?
– I-c: Teaching Learning Strategies
– I-d: Lessons for Growth Mindset + Learning Strategies
• II. Online collaborations between Scientists &
Educators
– II-a: Learning, Education & Research Network (LEARN)
– II-b: R.E.P.E.A.T. criteria for Research-Practice friendly resources
4

5. I. Reviewing & Synthesizing Cognitive Science Research
• Williams, J.J. (2013). tiny.cc/improveonlinelearningImproving Learning in MOOCs
by Applying Cognitive Science. Paper to be presented at the MOOCshop Workshop,
International Conference on Artificial Intelligence in Education, Memphis, TN.
• Harvey, A. G., Lee, J., Williams, J. J., Hollon, S., Walker, M., Thompson, M., & Smith,
R. (in press). Improving outcome of psychosocial treatments by enhancing memory
and learning. Perspectives in Psychological Science.
• www.josephjaywilliams.com/education
• Willingham, D. T. Why Don't Students Like School. Jossey-Bass (2010)
• Willingham, D. T. (2012). When can you trust the experts: How to tell good science
from bad in education.
5

6. I-a: How to increase motivation?
• Many ways to increase motivation…
• Change students’ implicit beliefs about whether
intelligence is fixed or malleable (Dweck, 2011; Yeager & Walton,
2012)
6

7. Implicit beliefs about Intelligence
• On a scale from 1 to 10, how much do you agree
that?
• Your intelligence is something very basic about you that you
can’t change very much.
• No matter how much intelligence you have, you can always
change it quite a bit.
• Fixed Mindset vs. Growth Mindset (Dweck, 2006)
– Self-fulfilling prophecy
– Avoid uncomfortable challenges vs. Pursue Learning Opportunities
– Avoid asking questions vs. Examining your mistakes
7

8. Research to link Science and Practice
• Experiment: Using Motivation Research to boost
students’ learning of Math online on Khan Academy
• Approach:
• 1. Quantify outcomes in real-world resource
• Math exercises at www.khanacademy.org
• 2. Synthesize recommendations from research
• Believing intelligence is malleable increases motivation
• 3. Embed experiment to evaluate
• Upcoming slides…
8

9. 1. Learning from Math exercises on www.KhanAcademy.org
• Typical Exercise
9
1. Number of
Problems
Completed
2. Percent
Correct

10. 2. Synthesize Scientific Findings
• Williams, J.J. (2013)Improving Learning in MOOCs by Applying
Cognitive Science. Paper presented at the MOOCshop
Workshop, International Conference on Artificial Intelligence
in Education, Memphis, TN.
• www.josephjaywilliams.com/education
• Teaching Growth Mindset of intelligence
• Research at Stanford by Dweck (2008) & Yeager &
Walton (2011)
10

11. 3. Add motivational messages
11
Practice-as-usual
Remember, the more you practice the smarter you become!
Growth Mindset Message

12. 3. Embedded in vivo Experiment
12
• Growth Mindset
Message
• "Remember, the more you practice
the smarter you become.”,
• "Mistakes help you learn. Think hard
to learn from them.”
• Practice-as-usual
• Benefit of Growth Mindset Message?
Jascha Sohl-Dickstein

13. Slides & Discussion at tiny.cc/inacolwebinar ‹#›

14. Practice-as-usualGrowth Mindset MessagePositive Message
3. Add Positive messages
14
Some of these problems are hard. Do your best!

15. Does any positive message work?
15
• Growth Mindset
Message
• "Remember, the more you practice
the smarter you become.”,
• "Mistakes help you learn. Think hard
to learn from them.”
• Positive Message
• "Some of these problems are
hard. Just do your best."
• "This might be a tough problem,
but we know you can do it.”
• Practice-as-usual

16. Number of Problems Attempted
Practice-as-usual
Growth Mindset
Message
Positive Message
Results to Analyze
16
Accuracy

17. Research to link Science & Practice
1. Outcomes in
Online Resources
2.
Recommendations
from Research
3. Experiments
Khan Academy
Math Exercises
Malleability of
Intelligence
Ongoing
Experiments
? ? ?
17

18. I-b: Increase learning without changing materials?
• Promote reflection with Socratic questions
• Questions before: Problem-Based Learning
• Questions during: Prompt for explanations
• Questions after: Use assessments for instruction
18

19. 19
Add
object
to bucket
Integrate
webpage
with internet
Content Exercise
Learning: Add vs. Integrate
Slides & Discussion at tiny.cc/inacolwebinar

20. Before: Start with Questions & Problems
• Problem Based Learning (Hmelo-Silver, 2006; Needham &
Begg, 1998; Schwartz, 1998)
20
How do you…?
Is it possible to…?

21. During: Request explanations
• Fonseca & Chi, 2011
• Renkl, 1997
• McNamara, 2004
• Rittle-Johnson, 2006
• Williams & Lombrozo, 2010
21
Why…?
How?
What are you thinking?
What next?

22. Explanation and Learning
• Does explaining…
– Provide a General boost to Learning Engagement
– Selectively guide learners to look for patterns
• The Subsumptive Constraints Theory:
Interpret target of why-explanation in terms of a broader
generalization (Williams & Lombrozo, 2010)
• Discover general patterns (Williams & Lombrozo, 2010, Cognitive Science)
• Use pre-existing knowledge (Williams & Lombrozo, 2013, Cog. Psych.)
• May mistakenly overgeneralize by ignoring specific
examples(Williams et al, 2013, JEP: General)
22

23. After: Use Assessments as Instructional Tools
• “Testing Effect” & Ten Benefits of Testing (Roediger et al, 2011)
23
Right away: Study+Study ~= Study+Test
Day/Week later: Study+Study < Study+Test
Learners think: Study+Study > Study+Test

24. After: Achieve Mixing Effect with assessments
• Mixing Effect (Rohrer, 2009)
24

25. Applying Cognitive Science
• Learning: Add vs. Integrate Knowledge
(Instructionism vs. Constructivism)
• Questions before: Problem-Based Learning
• Questions during: Prompt for explanations
• Questions after: Use assessments for instruction
• Further resources:
www.josephjaywilliams.com/education
25

26. I-c: Teaching Learning Strategies
• Spend a lesson teaching a concept vs. general
strategy?
• Online: Iteratively refine excellent lessons
• Online: Repeatedly reinforce habits & educational
behaviors
• Teach “What? Why? How?” self-
questioning/explanation strategies
26

27. Add “Socratic” prompts to explain & reflect
Clickable link. + Prompts embedded into hints.
Click here to learn about the What? Why? How? strategy
27

28. Embedded Prompts between Hint/Solution Steps
28

29. Self-questioning strategy: What? Why? How?
29

30. I-d: Lessons for Growth Mindset + Learning Strategies
• Learning Assistant: tiny.cc/learningassistant
• Short 5 minute videos
• Guided prompts
• Growth Mindset
• Self-Questioning strategy
• Experiment: How do lessons influence grades?
30

31. II. Collaborations between Scientists & Educators
• How to gain best of both worlds?
• Ask Questions, “Crowdsource” answers
• The Internet removes barriers to communication &
connects people
31

32. II-a: Learning, Education And Research Network (LEARN)
• www.learnnetwork.net
• Creating a virtual community:
• Educators, Researchers, Developers
• Mailing List, & Member list
• Discussion Forum – any teacher can ask a question,
so can any researcher
• Wiki of resources
• Collaborate in improving online educational
resources
32

33. II-b: R.E.P.E.A.T. Criteria: Research on Online Resources
• Can you collaborate to help your students & improve online
resources?
• Yes, if Online Educational Resources are:
– Realistic
– Experimental
– Product
– Evaluated
– Accessible
– Theoretically motivated
• REPEAT – iteratively improve through revision & collaboration
• http://edlab.tc.columbia.edu/
• Organizations like iNACOL!
33

34. Review
• I. Applying cognitive science to online learning
– I-a: How to increase motivation?
– I-b: Increase learning without changing materials?
– I-c: Teaching Learning Strategies
– I-d: Lessons for Growth Mindset + Learning Strategies
• II. Online collaborations between Scientists &
Educators
– II-a: Learning, Education & Research Network (LEARN)
– II-b: R.E.P.E.A.T. criteria for Research-Practice friendly resources
• Send advice & questions!
josephjaywilliams@stanford.edu
34

35. Contact Info
• Joseph Jay Williams
• josephjaywilliams@stanford.edu
• www.josephjaywilliams.com/education
• Slides at http://tiny.cc/inacolwebinar