The second keynote I delivered at #eas14 continued the theme of Brookfield's lenses of reflection, but this time focussed on Assessment Analytics. The two examples identified were from the LIFTUPP project at Liverpool's Dental School, as well as a pilot from Liverpool's Medical School in using data from Turnitin and Grademark to analyse staff and student performance in written assignments.

1. VIEWING SUMMATIVE ASSESSMENT
THROUGH DIFFERENT LENSES
Peter Reed
eAssessment Scotland Keynote Presentation

2. CONTINUING THE THEME…
SELF
REFLECTION
STUDENTS LITERATURE
PEERS
PROGRAMME SYSTEM(S)
ANALYTICS

3. DATA CAPTURE

4. ANALYTICS @ LIVERPOOL
LIFTUPP TURNITIN

5. Curriculum Review – mapping to GDC domains
LIFTUPP
What should a Dentist be able to do upon qualification?

7. DEVELOPMENTAL INDICATORS
WORKING BELOW
EXPECTED LEVEL
WORKING AT
EXPECTED LEVEL
WORKING ABOVE
EXPECTED LEVEL
1 2 3 4 5 6

8. STUDENT VIEW
Particular domain

9. STUDENT VIEW
Skills within domain & development indicator

10. STUDENT VIEW
Areas of the mouth & procedures

11. STUDENT VIEW
Materials used

12. STUDENT VIEW

13. STAFF VIEW

14. GENERATED DATA

16. TURNITIN / GRADEMARK
Pilot in School of Medicine SSMs
• Tutor name included in submission title;
• SSMs marked against 7 objectives (intro, methodology, etc);
• Objectives marked as Unsatisfactory (1), Fair (2), Good (3), Excellent (4);
• Total possible marks was 7 * 4 = 28;
• 6 Markers & 36 Student assignments;

17. TURNITIN / GRADEMARK
120
100
80
60
40
20
0
Unsatisfactory Fair Good Excellent
Count
Count of marks awarded

18. TURNITIN / GRADEMARK
25
20
15
10
5
0
Frequency of Grades Awarded for Each Objective
Ob 1 Ob 2 Ob 3 Ob 4 Ob 5 Ob 6 Ob 7
Unsatisfactory
Fair
Good
Excellent

19. TURNITIN / GRADEMARK
140
120
100
80
60
40
20
0
Number of QM Comments Made by Objective
Objective 1 Objective 2 Objective 3 Objective 4 Objective 5 Objective 6 Objective 7

20. TURNITIN / GRADEMARK
30
25
20
15
10
5
0
Mean Points awarded by marker
Marker 1 Marker 2 Marker 3 Marker 4 Marker 5 Marker 6

21. TURNITIN / GRADEMARK
4.5
4
3.5
3
2.5
2
1.5
1
0.5
0
Mean Points Awarded per Objective, by Marker
1 2 3 4 5 6 7
Marker 1
Marker 2
Marker 3
Marker 4
Marker 5
Marker 6

22. QUESTIONS
Is anybody doing anything with this information?
Should we?
Share with Staff? With Students?
Ethical considerations?
Hawthorne effect?
Predictive analytics?
Email Interviews - Do you think the data available from the
system is useful and why?

23. EMAIL INTERVIEWS
“Yes, no Hawthorne for me, I didn't know marks were being
assessed in that way. Helps reassure students and externals that we
are monitoring for consistency.”
“I think it very useful as it will make it easy to identify conveners who
mark outside normal parameters and identify who is giving feedback
to the students and who isn't.”

24. EMAIL INTERVIEWS
“I’m not quite sure what I think about this. Obviously we’re all
interested in what our own performance is (although I’m not sure we
should be told!). Major inconsistencies between markers or between
objectives would be worrying, but I don’t think we have a solid basis
for expecting any particular result – to put it crudely, would ‘all
excellent’ be a good thing (great students and course-management)
or bad (slack marking)?”

25. ASSESSMENT ANALYTICS
For the Greater Good or Big Brother?