1.
The University of Sydney Page 1
Co-developing
bespoke, enterprise-scale
analytics systems with
teaching staff
@dannydotliu
danny.liu@mq.edu.au
danny.liu@sydney.edu.au

2.
The University of Sydney Page 2
“We found that mature foundations for LA implementations were
identified in institutions that adopted a rapid innovation cycle
whereby small scale projects are initiated and outcomes quickly
assessed within short time frames. The successful projects in this
cycle are then further promoted for scale-ability and mainstream
adoption. In the context of LA, this small-scale seeded approach
appeared more effective in terms of organisational acceptance
and adoption than a whole of institution model attempting to roll
out a single encompassing program.”
Learning Analytics in Australia: Office for Learning & Teaching 2016

3.
The University of Sydney Page 3
1. Actionable information from and in Moodle
2. Open standards for storing and analysing LMS data
3. Customisable web-based analytics engine

4.
Actionable information from and in Moodle
THE MOODLE ENGAGEMENT ANALYTICS PLUGIN (MEAP)
3 February 2016

5.
Learning analytics in Moodle
MUCH PROMISE BUT LITTLE DELIVERY?
https://moodle.org/plugins/block_gismo
Mazza et al. (2012)
https://moodle.org/plugins/browse.php?list=set&id=20
Log viewer
Statistics report
GISMO
MOCLog
Engagement Analytics

6.
Enhancing an existing plugin
6
MOODLE ENGAGEMENT ANALYTICS PLUGIN (MEAP)
• Originally developed by Phillip Dawson, Adam Olley, and Ashley Holman
Moodle
Report
Logins Forums Assessments
Parameters
Traffic
lights
BYO
Moodle Engagement
Analytics Plugin
Indicators
Action

7.
Involving staff
7
ACADEMICS AND STUDENT SUPPORT
Staff expectations around an
early alert system
Prototyping and development
User testing
Piloting
Feedback and further development

8.
MEAP
8
CAPABILITIES AND APPLICATIONS

9.
Stakeholder outcomes
9
PERSONALISED, DATA-DRIVEN INTERVENTIONS
• 3400+ personalised emails sent, average ~46% opened
0
50
100
150
200
250
300
350
01Aug
08Aug
15Aug
22Aug
29Aug
05Sep
12Sep
19Sep
26Sep
03Oct
Numberofstudents
0
20
40
60
80
100
AreyouhavingproblemsaccessingiLearn?
ACCG250-Ifyousnoozeyoulose!
AreyouhavingtroubleloggingintoiLearn?
AreyouhavingtroubleloggingintoiLearninACCG100?
MECH201:EngineeringDynamics-iLearnActivity
ENGG150quizfeedback
ECON110iLearnLogin
AHIS140MythintheAncientWorld
AHIS140MythintheAncientWorld
AreyouhavingtroubleloggingintoECON111
AHIS140Session2Participation
AHIS140MythintheAncientWorld
Quiz1
Quiz1Result-Afin253
Ilearn-Afin253
ACST101-Quiz1
ECON110TutorialSubmissions
ECON111-Tutorialscontributetoyourfinalgrade
ECON111-Tutorialscontributetoyourfinalgrade
MECH202Progressthusfar
MECH203Progressthusfar
ISYS114
ACST101-Lastdaytowithdrawwithoutfailchargeis…
ISYS114Performance
ISYS114weeklytutorialsubmissions
ISYS114AttendanceandPerformance
ISYS114WorkshopAttendanceandPerformance
ISYS114Workshopattendance
Tutorialfeedback
Tutorialparticipation
TutorialparticipationforEUL101SocietiesofEurope
ACST101-Mondayisthelastdaytowithdrawwithout…
ACST101-Mondayisthelastdaytowithdrawwithout…
ACST101-Mondayisthelastdaytowithdrawwithout…
Itsnottoolatetogethelp!
ACST101-Mondayisthelastdaytowithdrawwithout…
ECON111-Pleasecheckyourtutorialrecords
ECON111-pleasecheckyourtutorialrecords
Percentageopened
1 hour 1 day 1 weekEmail opened within:

10.
Stakeholder outcomes
10
PERSONALISED, DATA-DRIVEN INTERVENTIONS
• Who: unit
convenors and
student support
staff
• What: census,
updates, reminders
• Why: predominantly
for at-risk students
• How: logins,
assessment
submissions, grades,
attendance
I was surprised someone cared/was actually monitoring, kind
of a weird, I don't know totalitarian/'people are watching you'
feeling? But in this situation I was happy.
He gave me specific advice and encouraged me and it made
me feel much better.
The email basically kicked me into gear and I completed all my
assessments post-email to a high level.
Very useful. I wouldn't have been able to do such a large scale
analysis and identify so many students without MEAP. I
wouldn't have been able to send them such tailored, structured
and consistent messages.

11.
Next steps
11
ON THE ROADMAP
• At Macquarie
• Production and wider trial ~S1 2016
or S2 2016
• For the community
• Source code release and collaboration
• New Moodle LA spec
• Research & development
• Further evaluating impact
• Machine learning for determining
parameters
https://docs.moodle.org/dev/Learning_Analytics_Specification

12.
Open standards for
storing and analysing LMS data
THE MACQUARIE OPEN ANALYTICS TOOLKIT (MOAT)
3 February 2016

13.
Our approach
13
CONNECTING USERS WITH DATA THROUGH ANALYTICS
Macquarie
Open
Analytics
Toolkit
Data
Users
Analytics
LMS
Video
Classrooms
Mobile
Business
systems
External
courses
Understand students
Identify and predict

14.
Bringing data together
14
NUANCES OF LEARNING DATA
LMS
Video
Classrooms
Mobile
Business
systems
External
courses
Learning
Record
Store
(LRS)
Custom analytics engine

15.
Prototype analyses
15
LEARNING PATHS

16.
Next steps
16
ON THE ROADMAP
• LRS to production
• Working with xAPI community
• Open sourcing analytics tools

17.
The University of Sydney Page 17
Customisable web-based
analytics engine
The Student Relationship Engagement System
(SRES)
Standing out from a Crowd
SumAll CC BY-NC-ND 2.0
https://flic.kr/p/kYbv4C

18.
The University of Sydney Page 18
The contexts of learning analytics
Common barriers to adoption
– Policy and ethical
challenges
– Culture of resistance to
change
– Vendor solutions
– Data accuracy
– One-size-fits-all
Pressing institutional needs
– ~$7 million/year lost to
attrition
– Larger class sizes
– More disconnected students
– Feedback very generalised

19.
The University of Sydney Page 19
The Student Relationship Engagement System
Attendance
Interim
grades
LMS
metrics
Third party
tools
Other data
as needed Student
Relationship
Engagement
System

20.
The University of Sydney Page 20
Personalising connections with students
– Instructors have control
– Flexible
– Targeted and personalised
– Multi-channel
– Benefits
– Highly customisable
– Efficient – key data in one
place, operating at scale
– Connect staff and all
students (not just at-risk)
Student
Relationship
Engagement
System

21.
The University of Sydney Page 21
Co-evolution of the SRES
– Organic adoption by academics
– Co-evolution of capabilities
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
2011 2012 2013 2014
Attrited FA PS CR DI HD
0
5000
10000
15000
20000
25000
0
10
20
30
40
50
60
70
2012 2013 2014 2015
Numberofstudents
Numberofunitsorschools
Number of units Number of schools Number of students
Pilot
EWS
introduced
EWS
integrated
Machine
learning
Student outcomes*
Large first-year science unit
SRES

22.
The University of Sydney Page 22
Data-driven pedagogy and curriculum by stealth?
– Co-evolving data
capabilities and
competencies
Student
Relationship
Engagement
System

23.
The University of Sydney Page 23
Next steps
– At Sydney
– Facilitate wider roll-out
– Further developments – ML, student view, sub-messages, etc
– Research & evaluation
– At Sydney and with the community
– Redevelopment and open source

24.
The University of Sydney Page 24
1. MEAP Actionable information from and in Moodle
Chris Froissard, Deborah Richards, Amara Atif
2. MOAT Open standards for storing and analysing LMS data
James Hamilton, Ed Moore, Yvonne Breyer et al.
3. SRES Customisable web-based analytics engine
Charlotte Taylor, Adam Bridgeman, Kathryn Bartimote-Aufflick et al.
Your
Institution?
@dannydotliu
danny.liu@mq.edu.au
danny.liu@sydney.edu.au