A second discussion of Just-in-Time Teaching focusing more on practical matters of application and structure.

1. Name
School
Department
JUST IN TIME TEACHING
MAKING IT SHINE
(PART 2)
@ UNC CETL
NOVEMBER 13TH, 2015
DR. JEFF LOATS
DEPARTMENT OF PHYSICS

2. OVERVIEW
1. Recap of JiTT with an example
2. Best tools for JiTT
3. The “sales pitch”
4. Closing the loop
5. Writing good questions
(With apologies for repetition…)

3. THE EVIDENCE STANDARD
Teachers can feel bombarded…
I strive to be a scholarly teacher …
• Apply the rigor we bring to our academic
disciplines to the discipline of teaching.
• Choose teaching methods that are strongly
informed by the best empirical evidence
available.
Contrast teaching your subject with treating a
medical condition like diabetes

4. In your teaching do you have a method for holding
students accountable for preparing for class?
Previous anonymous poll results (compiled):
~17% → I don’t, but I ask/threaten really well
~50% → I use a paper method (quiz, journal…)
~11% → I use a digital method (clickers, etc.)
~17% → I have some other method
(N ~ 211)
A)
B)
C)
E)


5. PHYSICS EDUCATION REVOLUTION
Eric Mazur, Physicist at Harvard:

6. “ALL SIMILARLY (IN)EFFECTIVE…”

7. 29%
31%
20%
14%
5%
0%
10%
20%
30%
40%
50%
60%
0-20% 20-40% 40-60% 60-80% 80-100%
Consider a typical day in your class. What fraction
of students did their preparatory work before
coming to class?
Previous anonymous poll results (compiled):
N = 232

8. JUST-IN-TIME TEACHING
Online pre-class assignments
called WarmUps
First half - Students
• Conceptual questions, answered in sentences
• Graded on thoughtful effort
Second half - Instructor
• Responses are read “just in time”
• Instructor modifies that day’s plan accordingly.
• Aggregate and individual (anonymous) responses
are displayed in class.
Learne
r
Teacher

9. JUST-IN-TIME TEACHING
A different student role:
• Actively prepare for class
(not just reading/watching)
• Actively engage in class
• Compare your progress & plan accordingly
A different instructor role:
• Actively prepare for class with you
(not just going over last year’s notes )
• Modify class accordingly
• Create interactive engagement opportunities
Learne
r
Teacher

10. WARM-UP: TIPTOES
You can probably stand on your tiptoes…
However, if you stand with your toes against a
wall, you can’t. Try to explain why it is possible
in one case, but not in the other...
From last night:
~50% → Need to move your body forward, but
can’t
~33% → The wall makes balancing harder
~17% → Stumped!

11. WARM-UP: TIPTOES
“Because one looses balance much easier when you
stand against a wall or door”
“When we try to keep balance on tiptoes, our body
tends to lean forward. When our toes are against a
wall or door, the wall or door leaves no space for our
body to lean forward.”
“The wall limits where I can position my body, so if
my toes are against the wall, then I am restricted in
my attempt to adjust my center of gravity forward to
be over my new (smaller) area of contact between
my feet and the floor.”

12. MAZUR AFTER 1 YEAR

13. ELSEWHERE?
My Physics I
Summer
2015 (45,26)
Gain: 0.51

14. WARM-UP: JITT QUESTIONS
Either from your experience at the first workshop, or
from reviewing the information at the link above,
what is your biggest question/concern about Just-in-
Time Teaching?
But first… think about what a teacher/facilitator
“should” do with responses to a question like this.

15. WARM-UP: JITT QUESTIONS
“Is JiTT successful only in face-to-face learning
environments or can it be applicable to hybrid or
fully online courses?”
“Whether anyone in my discipline has done this
(or if I have to write all my warm-up questions
from scratch)”

16. WARM-UP: JITT QUESTIONS
“Time management, how can I be organized in a
way that I can have all the responses posted
before each class and plan the lesson
accordingly.”
“My concern is that the workload might be too
much for a teacher to review the pre-class
assignments if he or she is teaching a big ESL
class, say altogether 200 students.”

17. WARM-UP: JITT QUESTIONS
“I'm just not sure about some logistics, esp.
building in places where I can quickly modify my
lesson plans to address student understanding.”
“the lack of control in the classroom, not being
able to answer enough questions that are needing
answered”

18. THE JITT FEEDBACK LOOP
Student responses:
• Graded on thoughtful effort
• Sampled and categorized for display
• Quoted anonymously
Closing the loop:
• Respond to some students digitally
• Class time shifts to active engagement.

19. JUST-IN-TIME TEACHING
A different student role:
• Actively prepare for class
(not just reading/watching)
• Actively engage in class
• Compare your progress & plan accordingly
A different instructor role:
• Actively prepare for class with you
(not just going over last year’s notes )
• Modify class accordingly
• Create interactive engagement opportunities
Learne
r
Teacher

20. Which topic would you like to spend our
remaining time on?
A) Evidence for effectiveness
B) Best tools for JiTT
C) Getting student “buy-in”
D) Writing good questions
20

21. Evidence For Effectiveness

22. STUDIED EFFECTIVENESS
Used at hundreds of institutions
Dozens of studies/articles, in many disciplines:
Bio, Art Hist., Econ., Math, Psych., Chem., etc.
–Increase in content knowledge
–Improved student preparation for class
–Improved use of out-of-class time
–Increased attendance & engagement in class
–Improvement in affective measures

23. OUR STUDIES
Collected data from 18 courses in 5 disciplines
• Upper- and lower-division courses
• Larger and smaller courses
• General studies, required major courses,
elective courses
Variations in implementation of JiTT
• Weekly vs. biweekly
• 5-20% of final course grade

24. DATA FROM OUR COURSES
Overall Response Rates:
• Average response rates varied from
46% - 74% over the 15-week semester.
• Taking the mean of the 18 different class
averages we get 59% response rates.
Research: Burchfield & Sappington (2000) found
that ~33% of students were prepared for class.

25. 0
20
40
60
80
100
1 2 3 4 5 6 7 8 9 10 11 12 13 14
PSY2850: Smith, N = 33
PSY2410: Rocheleau, N = 36 WMS 390Z: Sgoutas, N = 18
0
20
40
60
80
100
1 2 3 4 5 7 8 9 10 11 14 15 16 17 18 19 22 23 24 25 26 27
PHY2010: Loats, N = 78
0%
20%
40%
60%
80%
100%
1 2 3 4 5 6 7 8 9 10 11 12 13
PSY1001: Smith, N = 22
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
0%
20%
40%
60%
80%
100%
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
PHY2020: Loats, N = 73
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

26. JITT VS. FINAL GRADE
CORRELATIONS
College Physics I, Fall 2013
0
20
40
60
80
100
0 10 20 30 40 50 60 70 80 90 100
CumulativeScore(withoutwarm-ups)
WarmUp Score
WarmUps vs. Cumulative Score
Correlation r = 0.71

27. CORRELATIONS AND EFFECT SIZE
Effect sizes for most pedagogical techniques are
small to medium (see, e.g., Hattie, 2009)
JiTT correlations with remaining course totals
across 18 classes. Range from 𝑟 = 0.38 − 0.84
with an average of 0.60.
Per Cohen (1988),
• 𝑟 = 0.30 is medium effect,
• 𝑟 = 0.50 is large effect

28. PROGRESSIVE EXAMS
CORRELATIONS?
College Physics I:
Important disclosure: This was not a hypothesis we were
testing, it appeared as we analyzed the data. Could be
0.18
0.33
0.43
0.54
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
Mini Exam
(week 4)
Exam 1
(week 7)
Exam 2
(week 11)
Final Exam
(week 16)
NoneWeakStrongModerate
Correlations between Total WarmUp Score
and Sequence of Exams

29. Mean on 1-5 scale
Preparation for class 4.06
Engagement during
class 3.93
STUDENT SURVEY RESULTS
9% 10%
81%
10%
18%
73%
10%
22%
68%
0%
20%
40%
60%
80%
100%
Harmful Neutral Helpful
How did WarmUps affect your...
Preparation Engagement Learning
N = 781

30. STUDENT SURVEY QUOTES
Physics:
“Initially, it was hard for me to get used to the
warm-ups. It seemed like along with the
homework assignments there was a lot of things
to do. Eventually I got used to it and ultimately
the warmups really helped me to learn the
material and stay caught up with the class.”
“If it weren't for warm ups, the amount of time I
spent reading the book would have dropped by
75%”

31. LIMITATIONS
Could be alternative explanations/confounds that
are uncontrolled in these analyses; e.g.,
–Academic motivation
–Quality of answer
–Conscientiousness
–Interest in course
But the repeated pattern across a variety of
courses gives some credibility.

32. Best Tools for JiTT

33. WHAT TOOLS TO USE?
The crucial part:
Daily reading, grading & using responses
• Automatic full credit for any response
• View all responses to a question together
• Grade responses on the same page with
minimal clicks
Wishlist:
Easy (quick!) individual feedback

34. SMALL ASIDE: TEXT EXPANDER
34
Every professor should have this!
You define a snippet like “ttyl” which instantly
gets replaced by “Talk to you later!”
Windows:
– Texter, PhraseExpres
(FREE, some advanced features, some flaws)
– Breevey ($40, worth it if you hit problems)
– AutoHotKey (free advanced automation tool)
Mac:
– TypeIt4Me, TextExpander, Typinator
(All cost $20-$30. Generally worth it!)

35. WHAT TOOLS TO USE?
• CMS/LMS (Blackboard, D2L, Moodle, etc.)
Ready to use, tools… imperfect  awful
• Free service from JiTTDL.org.
Designed just for JiTT. Additional website, not
very “shiny” by 2015 standards.
• Students email responses
Easy… also overwhelming and awful
• Blogging tools (WordPress)?
• New tools (TopHat? Learning Catalytics?)

36. Getting Student Buy-In
(a.k.a. “the sales pitch”)

37. OVERARCHING MESSAGE
Communicating with your students (humans)
• Message (explicit statements)
• Attitude (subtext, body language, etc.)
Consistent subtext:
"I am here to help you learn, and I have thought
about your learning trajectory carefully."
Consistent attitude:
I am comfortable and relaxed about my part of
this partnership.

38. THE SALES PITCH
13,000 hours in invisible contract indoctrination
Mindfulness in what we say and what we do.
Day 1 – Keep justifications short. Emphasize
purpose over mechanics.
Day 2 – Discuss their first experience & response
rates. Remind them about structure & purpose,
but mostly show them.
Day 3 – Return to “different roles” for both.
Demonstrate value, be consistent

39. STUDENTS: BUSY-WORK
DETECTORS
K-12 represents more than 13,000 hours of class
Students are experts at detecting what really
matters to an instructor:
• What does the instructor do with class time?
• What does the instructor talk about?
• Does the instructor push against the usual
“invisible contract” of the classroom?

40. DEMONSTRATE VALUE & BE
CONSISTENT
Demonstrating that you value JiTT
• Thank them for giving you insight
• Bring at least one “difficult/interesting” item from
WarmUp to class each day.
• Give non-verbal cues that you value discussing
WarmUps as much as other course components.
Be consistent with:
• Assignment releases and due dates/times
• Follow-up in class
• Summative assessments (e.g., exam questions) that
build on WarmUp questions.

41. Writing Good Questions

42. WARMUP QUESTIONS
• Every-day language
• Occasional simple comprehension question
• Mostly “higher level” questions
• Any question is better than none (don’t be precious)
Connections to evidence:
–Pre-class work reduces working memory load
during class.
–Multimodal practice (not learning styles):
JiTT brings reading, writing and discussion as
modes of practice.

43. METACOGNITION
Two questions in every WarmUp:
First:“What aspect of the material did you find
the most difficult or interesting.”
Last: “How much time did you spend on the pre-
class work for tomorrow?”
Connections to evidence:
–Forced practice at metacognition:
Students regularly evaluate their own
interaction with the material.

44. FEATURES OF A GOOD QUESTION
44
What would a “good” response look like?
– A paragraph? (too long)
– One word? (too short)
Make sure the reading is needed to respond (but a
sentence straight out of the book shouldn’t work).
Make sure a beginner can take a crack at the question
Be concrete:
– “Explain in 2-3 sentences.”
– “Give two brief examples.”
– “Explain how you got your estimate.”
“Game out” their responses a bit.

45. WRITE A QUESTION AND SHARE...
45
Consider an intro. course in your discipline.
Consider a topic you discuss early in that course.
Write one question… shoot for “higher level.”
Good starting words: apply, analyze, evaluate,
sketch, use, compare, estimate, etc.
Take 3 minutes… then trade questions with your
neighbor and do your best to answer theirs.

46. Summary Slides

47. A POSSIBLE PLAN
Choose one course you will teach next term.
A. Write two questions for each class meeting:
1. One lower-level (maybe multi-choice?).
One higher-level (sentences).
2. Give yourself 10 minutes to write each one
B. Write a standard (1st) metacognitive question
C. Discuss one question at the top of class, and
one in the middle. Use the metacognitive
responses as break points or highlights.

48. MY SUMMARY
JiTT may be among the easiest research-based
instructional strategies that you can consistently
integrate into your teaching.
From an evidence-based perspective, JiTT
addresses often-neglected areas.
Be prepared to find that students know less than
we might hope. (Perhaps freeing?)

49. YOUR SUMMARY
If you want to implement JiTT, what is your next
concrete action?
Email: jeff.loats@gmail.com
Twitter: @JeffLoats
Slides: www.slideshare.net/JeffLoats

50. JITT REFERENCES & RESOURCES
Simkins, Scott and Maier, Mark (Eds.) (2010) Just inTimeTeaching: Across the Disciplines, Across the
Academy, Stylus Publishing.
Gregor M. Novak, Andrew Gavrini, Wolfgang Christian, Evelyn Patterson (1999) Just-in-Time
Teaching: Blending Active Learning with WebTechnology. Prentice Hall. Upper Saddle River NJ.
K. A. Marrs, and G. Novak. (2004). Just-in-Time Teaching in Biology: Creating an Active Learner
Classroom Using the Internet. Cell Biology Education, v. 3, p. 49-61.
Jay R. Howard (2004). Just-in-Time Teaching in Sociology or How I Convinced My Students to
Actually Read the Assignment. Teaching Sociology,Vol. 32 (No. 4 ). pp. 385-390. Published by:
American Sociological Association
Stable URL: http://www.jstor.org/stable/3649666
S. Linneman, T. Plake (2006). Searching for the Difference: A ControlledTest of Just-in-Time
Teaching for Large-Enrollment Introductory Geology Courses. Journal of Geoscience Education, Vol.
54 (No. 1)
Stable URL:http://www.nagt.org/nagt/jge/abstracts/jan06.html#v54p18
ON-DEMAND SLIDES

53. ASIDE: LEARNING STYLES
“I think that many teachers teach in a way that
makes sense to them, according to their learning
style […]”
Best current evidence: Learning styles don’t exist
References:
• “The Myth of Learning Styles”
by Cedar Riener and Daniel Willingham
• YouTube: Learning Styles Don’t Exist
• Scholarly review: “Learning styles: Concepts
and evidence”, Pashler et al, 2008