Keynote given at the 2015 Western Conference on Science Education (WCSE2015), London Ontario, July 2015.

1. WCSE 2015
Faculty & Students
as collaborators, co-creators and makers
Simon.Bates@ubc.ca
@simonpbates
bit.ly/batestalks

2. Photo: Wikipedia

3. Edwin Thompson Jaynes
1922-1998
Photo: Wikipedia

4. Jaynes, E. T., 1993, `A Backward Look to the Future, ' in Physics and Probability, W. T. Grandy, Jr. and P. W. Milonni, Cambridge Univ. Press,

5. “But it required a few years before I
perceived what a science teacher's job
really is.
The goal should be, not to implant in
the student's mind every fact that the
teacher knows now; but rather to
implant a way of thinking that will
enable the student, in the future, to
learn in one year what the teacher
learned in two years.
Only in that way can we continue to
advance from one generation to the
next.”
Jaynes, E. T., 1993, `A Backward Look to the Future, ' in Physics and Probability, W. T. Grandy, Jr. and P. W. Milonni, Cambridge Univ. Press,

6. Overview
How things are changing; technology as an example
driver
Implications for instructors; the anatomy for future
success
The case for students as co-creators, and four examples of
how to enact that

7. CC BY-NC-SA 2.0 https://flic.kr/p/2ZdABF

8. Technology - scale and pace
Slide credit: Eric Grimson (MIT)

9. Technology - scale and pace
Slide credit: Eric Grimson (MIT)

10. Technology - reach and unbundling

11. Technology - reach and unbundling

12. Technology - disruption
Graph extracted from http://vikparuchuri.com/blog/on-the-automated-scoring-of-essays/

13. Technology - implications
Changing the
what, where, when, how,
from whom and with whom
of many aspects of life,
…and learning is included

14. Technology - implications
So what are the
we need to embrace, develop and refine?
skills, values and habits

18. Students as collaborators, co-creators
Four examples (of increasing complexity to implement)
1. “Things I wish I’d known”
2. The most important course you can take
3. Undergraduate learning assistants in the classroom
4. Student-generated assessment and content

19. “Things I wish I’d known”

20. “Things I wish I’d known”

21. The most important course you will take

22. The most important course you will take

23. The most important course you will take

24. Undergraduate learning assistants in the classroom

25. Undergraduate learning assistants in the classroom
S. Pollock, 2007 PERC Proc. 951, p.172

26. Undergraduate learning assistants in the classroom
S. Pollock, 2007 PERC Proc. 951, p.172

27. Undergraduate learning assistants in the classroom
S. Pollock, 2007 PERC Proc. 951, p.172

28. Students as producers of assessment content for learning
bit.ly/batestalks

29. Students as producers of assessment content for learning
bit.ly/batestalks

30. Selected results and analysis
Engagement - how do students use the system?
Benefits - what is the impact on learning?
Question quality - how good is what students produce?
Relevant publications:
Scaffolding student engagement via online peer learning - European Journal of Physics 35 (4),
045002 (2014)
Student-Generated Content: Enhancing learning through sharing multiple-choice questions.
International Journal of Science Education, 1-15 (2014).
Assessing the quality of a student-generated question repository - Phys Rev ST PER (2014)
10, 020105
Student-generated assessment - Education in Chemistry (2013) 13 1

31. Quality of student authored content
Bloom’s Taxonomy of levels in the cognitive domain
Score Level Description
1 Remember Factual knowledge, trivial plugging in of numbers
2 Understand Basic understanding of content
3 Apply Implement, calculate / determine. Typically one-stage problem
4 Analyze Typical multi-step problem; requires identification of strategy
Evaluate Compare &assess various option possibilities; often conceptual
Synthesize
Ideas and topics from disparate course sections combined. Significantly
challenging problem.
Text

32. Question quality
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
50%
1 2 3 4 5 6
Taxonomic Category
PercentageofSubmittedQuestions
First semester N = 350
Second semester N = 252

33. Explanation quality
0 Missing
No explanation provided or explanation incoherent/
irrelevant
1 Inadequate Wrong reasoning and/or answer; trivial or flippant
2 Minimal
Correct answer but with insufficient explanation/
justification/ Some aspects may be unclear/incorrect/
confused.
3 Good Clear and detailed exposition of correct method & answe
4 Excellent
Thorough description of relevant physics and solution
strategy. Plausibility of all answers considered. Beyond
normal expectation for a correct solution

34. 0 1 2 3 4 0 1 2 3 4
0
20
40
60
Numberofquestions
Assessment 1 Assessment 2
Explanation Quality

35. Question quality summary (UoE 2011)
2 successive years of the same course (N=150, 350)
‘High quality’ questions: 78%, 79%
Over 90% (most likely) correct, and majority of those wrong were
identified by students.
69% (2010) and 55% (2011) rated 3 or 4 for explanations
Only 2% (2010) and 4% (2011) rated 1/ 6 for taxonomic level.

36. That’s not common
Bottomley & Denny Biochem and Mol Biol Educ. 39(5)
352-361 (2011)
107 Year 2 biochem students
56 / 35 / 9 % of questions in lowest 3 levels.
Momsen et al CBE-Life Sci Educ 9, 436-440 (2010)
“9,713 assessment items submitted by 50 instructors in the
United States reported that 93% of the questions asked on
examinations in introductory biology courses were at the
lowest two levels of the revised Bloom’s taxonomy”

37. Students as producers of assessment content for learning
Why not short answer Qs?
Why not …. anything?

38. Students as producers of assessment content for learning
Why not short answer Qs?
Why not …. anything?

39. Students as producers of assessment content for learning
Why not short answer Qs?
Why not …. anything?
Test Kitchen:
Adaptive Comparative
Judgement

40. Students as producers of assessment content for learning
Why not short answer Qs?
Why not …. anything?
LEARNING OBJECTS
Test Kitchen:
Adaptive Comparative
Judgement

41. PHYS101:
Energy and Waves

42. PHYS101:
Energy and Waves

43. PHYS101:
Energy and Waves

44. PHYS101:
Energy and Waves

45. PHYS101:
Energy and Waves

46. Implementation logistics
Cohort split into 4 groups
Each week one group tasked with creating LOs
Each submission counts for 2.5% of final grade
Repeat cycle twice per Semester
Students can submit >2 LOs & receive grade for best 2
Short survey on submission
Students encouraged to apply CC licenses

47. Results: engagement
0 100 200 300
LO 1
LO 2
LO 3
LO 4
LO 5
LO 6
LO 7
LO 8
Number of students
Assigned
Optional

48. Results: time on task
0 100 200 300 400
Less than 0.5h
0.5 to 1 h
1 to 2h
2 to 3h
3 to 4h
4 to 5h
More than 5h
Number of students

49. Results: self-reported change in understanding
0 200 400 600 800
None
Little
Moderate
Good
Excellent
Number of students
0200400600800
Number of students
before creating it after creating it
How much did you understand the topic your LO was based on

50. PHYS101:
Energy and Waves
3. Successes
http://youtu.be/BObyt_NJYrE

51. Sample 2 - Standing Wave in a bowl

52. Sample 2 - Standing Wave in a bowl

53. Sample 3 - Colour Loss Underwater

54. Student generated exam content
before creating it after creating it
How much did you understand the topic your LO was based on

55. Resources for use in class
http://blogs.ubc.ca/phys101

56. Not quite the whole story
• Despite these outstanding examples, many students
didn’t like the assessment
• difficulty level vs other assessed components of the
course
• credit weighting
• Students dropped these assessments more than other
coursework
• Strange ‘phase transition’ for LO vs exam grades

59. One final thought….
why this all works