Keynote presented at the Improving University Teaching Conference (http://www.iutconference.com/) Vancouver July 2014

1. A pilot study in a large enrolment
introductory science course
Expert-guided crowdsourced
learning content
Simon Bates
simon.bates@ubc.ca
@simonpbates
bit.ly/batestalks

3. ~2010
~2006
2013

4. Context - flipped classrooms
CC BY-NC 2.0 https://flic.kr/p/f3ynHx
Derek Bruff: Class time reconsidered
http://prezi.com/donq036eunko/class-time-reconsidered/

5. Weekly rhythm for our 1A class
Poh,
M.Z.,
Swenson,
N.C.,
Picard,
R.W.,
"A
Wearable
Sensor
for
Unobtrusive,
Long-­‐term
Assessment
of
Electrodermal
AcEvity,"
IEEE
TransacEons
on
Biomedical
Engineering,
vol.57,
no.5,
pp.1243-­‐1252,
May
2010.
doi:
10.1109/TBME.2009.2038487

6. Weekly rhythm for our 1A class

7. Weekly rhythm for our 1A class

8. Learning gains on PI

9. Learning gains on PI

10. 1
2
3
A ball initially at rest is
thrown upwards, comes
back down & is caught
!
Which of the following is a
plausible graph of the
acceleration of the ball
with time?

13. Freeman et al PNAS www.pnas.org/cgi/doi/10.1073/pnas.1319030111
Wieman commentary PNAS
http://www.pnas.org/cgi/doi/10.1073/pnas.1407304111
“This meta-analysis makes a powerful case that any college or university that is teaching its
STEM courses by traditional lectures is providing an inferior education to its students”

14. ~2010
~2006
2013

15. a"web&based"MCQ"repository"created"by"students"

16. Ins$tu$ons((signing(up(per(year:(
!
2009:! ! !22(
2010:! ! !66(
2011:! ! !204(
2012:! ! !266(
2013!(Jan,Jun):! !214(
Growing(content(repository:(
!
Courses:! !2,500(
Logins/month:! !75,000(
Ques<ons:! !600,000(
Answers:! !12,000,000(
Answers
(20,000,000)
Questions
(1,000,000)

17. Student'ownership'over'
learning'resource'
Student'familiarity'with'
social'so7ware'
Leveraging'student'
energy'and'crea9vity'

19. Badges' Points' Leaderboards'

20. Typical implementation
Minimum participation requirements for each
of two assessment exercises (PW1, PW2)
Write 1 Answer 5 Rate / comment 3
5% course credit
Physics 101, Energy & Waves
Winter Semester: 3 sections, 791 students

21. Scaffolding
TOCCLT'.'
-
QestIsct
How 1o...
q/t4osTer
Submit ond onswer questions
on topics in lhe torget region,
just obove lhe physics
you hove olreody mostered.
Dislroclers
) l*"*"ò
rNrrltng
Ih s region corrk:ins llte
plrysics knowledgo ond colrco¡ls
you connol leorn yel
becouse the louncalions
crrc nol in plocc
PHYSICS TOPICS IN YOUR TARG T REGION
buo'5ot't1 'Den¡¡lu
B¿:ovvì 1 J
F:rcq - YOUR CHOSEN TOPIC
à..
auC'q^ x@
bôo'osro
pn ro, d logroo,
o 9tS
COMMON MISCONCEPÎIONS AND ERRORS
(Sec hHÞ:/jÞhy¡.udofo¡.cdu/CJP/trêconcoÞllo¡t.pdt
tor o llil ol common mhconcepllons)
¿K N(,qht %cce Açxs no exi¡þ
, c.rrrçrJ On oblec'-t 5 õÞYJrnr cìü?À
I in rtuiò. ü
lrh'brot4ont eprç,e .I¡1enÀs crq
den-s,{^ c>Ç c.,þ¡ec , cìo+ .*¿n:r
oÇ Çtu,8,
fr- btrrqon! $crCe c¿cs Àoc¡:'cr
fr¿lt{e- r *hcrn ú?
Trr¿e. $ììfifflsz hrrr: ¿lÕoolr-g lnnS
(**= tcpo þ/rnl
{otr-* qrÇ bq.:. 0" Ooz
Physics knowledge
ond conccpluol underslonding
you lrove olrcody
conslrucleC ¡n your heod
¡,<'t.lerJ rt)^,¿z! Þ.rrp r- rr't¡ ,l?
Chcck lhol yovr on3Íêr ¡t rêo!ànoblê ðnd potrlble
'f<,,cc
Photo by Seth Casteel
http://www.littlefriendsphoto.com
Permission to use agreed
Introduced in tutorials
!
Extensive scaffolding
exercises
!
Revisited in subsequent
tutorials
!
Tutorials delivered by 24 TAs

22. Engagement with PeerWise
Number Multiplier Number Multiplier
Questions 1105 [1.7] (1.4) 998 [1.6] (1.3)
Answers 11393 [17.2] (14.4) 11807 [18.7] (15.0)
Comments 4901 [7.4] (6.2) 5509 [8.7] (5.0)
PW 1 PW 2

23. Engagement with PeerWise

24. Engagement with PeerWise
PW1 PW2
Contributed and met minimum requirements
Contributed but did not meet mininum requirements
Did not participate
0.84 0.80
Total: 791

26. Question/Explanation Quality
Bloom’s Taxonomy of levels in the cognitive domain
Score Level Description
1 Remember Factual knowledge, trivial plugging in of numbers
2 Understand Basic understanding of content
3 Apply Implement, calculate / determine. Typically one-stage problem
4 Analyze Typical multi-step problem; requires identification of strategy
Evaluate Compare &assess various option possibilities; often conceptual
Synthesize
Ideas and topics from disparate course sections combined. Significantly
challenging problem.
Text

27. 1 2 3 4 1 2 3 4
0
20
40
60
Numberofquestions
Assessment 1 Assessment 2
Bloom's Taxonomy: Question Quality
Textp>0.05, NS
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
50%
1 2 3 4 5 6
Taxonomic Category
PercentageofSubmittedQuestions
First semester N = 350
Second semester N = 252
Bates et al,
PRST PER, in press 2014
http://arxiv.org/abs/1308.2202

28. Question/Explanation Quality
Description of explanation quality
Score Level Description
0 Missing
No explanation provided or explanation incoherent/
irrelevant
1 Inadequate Wrong reasoning and/or answer; trivial or flippant
2 Minimal
Correct answer but with insufficient explanation/
justification/ Some aspects may be unclear/incorrect/
confused.
3 Good Clear and detailed exposition of correct method & ans
4 Excellent
Thorough description of relevant physics and solution
strategy. Plausibility of all answers considered. Beyond
normal expectation for a correct solution

29. 0 1 2 3 4 0 1 2 3 4
0
20
40
60
Numberofquestions
Assessment 1 Assessment 2
Explanation Quality
p=0.02

30. ~2010
~2006
2013

33. PHYS101:
Energy and Waves

34. Logistics
!
!
Cohort split into 4 groups
!
Each week one group tasked with creating LOs
!
Each submission counts for 2.5% of final grade
!
Repeat cycle twice per Semester
!
Students can submit >2 LOs & receive grade for best 2
!
Short survey on submission
!
Students encourage to apply CC licenses

35. Results: engagement
0 100 200 300
LO 1
LO 2
LO 3
LO 4
LO 5
LO 6
LO 7
LO 8
Number of students
Assigned
Optional

36. 0 10 20 30 40 50 60 70 80 90 100
0
100
200
300
Grade (%)
Numberofstudents LO Grade Distribution

37. 0 10 20 30 40 50 60 70 80 90 100
0
100
200
300
Grade (%)
Numberofstudents LO Grade Distribution
These students
didn’t even try!
0 10 20 30 40 50 60 70 80 90 100
0
100
200
300
Grade (%)
Numberofstudents LO Grade Distribution

38. Which students aren’t submitting LOs?
Non-participants are from all walks of life!
0 10 20 30 40 50 60 70 80 90 100
0
5
10
15
20
25
Final Exam (%)
Numberofstudents
Final Exam marks of students
that did not participate in LOs

39. Results: types of Los
PPT
DOC
Prezi
Ted-Ed
PDF
Image
Other

40. Results: time on task
0 100 200 300 400
Less than 0.5h
0.5 to 1 h
1 to 2h
2 to 3h
3 to 4h
4 to 5h
More than 5h
Number of students

41. Results: self reported change in
understanding
0 200 400 600 800
None
Little
Moderate
Good
Excellent
Number of students
0200400600800
Number of students

42. PHYS101:
Energy and Waves
3. Successes
http://youtu.be/BObyt_NJYrE

43. Sample 2 - Standing Wave in a bowl

44. Sample 3 - Colour Loss Underwater

45. Student generated exam content
3. Successes

49. simon.bates@ubc.ca @simonpbates bit.ly/batestalks
Relevant publications:
!
Scaffolding student engagement via online peer learning - European Journal of Physics 35 (4), 045002 (2014)
!
Student-Generated Content: Enhancing learning through sharing multiple-choice questions. International Journal of Science Education, 1-15 (2014).
!
Assessing the quality of a student-generated question repository - Phys Rev ST PER (2014) in press
!
Student-generated assessment - Education in Chemistry (2013) 13 1

50. Image credits
Crowd http://www.flickr.com/photos/18378655@N00/613445810/