Workshop run at the European Conference for e-Learning 2015 (ECEL 2015) at the University of Hertfordshire, UK. The workshop included an introduction of both learning analytics and learning design, as well as an exploration of how these could be employed in MOOCs. Some of the group work was focused on the Agincourt MOOC run by the University of Southampton on the FutureLearn platform.

1. Learning design and learning analytics:
building the links with MOOCs
European Conference for e-Learning 2015

2. Rebecca Ferguson
• The Open University (200,000 learners)
• Informal learning:
• YouTube (30 million views)
FutureLearn (2.5 million learners)
OpenLearn (39.3 million visits)
• Making use of big data for
45 years
• Learning analytics research / events
• LACE project – learning analytics
community exchange
2
Lead on MOOC evaluation at The Open University, UK
http://www.laceproject.eu/

3. Workshop overview
14.00 Introduction
Linking learning analytics, learning design and MOOCs
14.30 Initial group work
Learning outcomes in MOOCs
15.00 Coffee break and discussion
15.10 Group work
Learning design in MOOCs
15.30 Discussion
15.40 Group work
Learning analytics in MOOCs
15.40 Final plenary
16.00 Workshop end
3
You can view and download a version of these slides at
http://www.slideshare.net/R3beccaF

4. Learning analytics
What are they, and how are they used?
4

5. What are learning analytics?
High-level figures
Brief overviews for internal and external
reports
Academic analytics
Figures on retention and success, for the
institution to assess performance
Learning analytics
Use of big data to provide actionable
intelligence for learners and educators
5

6. Educators use analytics to
• Monitor the learning process
• Explore student data
• Identify problems
• Discover patterns
• Find early indicators for success
• Find early indicators for poor marks or drop-out
• Assess usefulness of learning materials
• Increase awareness, reflect and self reflect
• Increase understanding of learning environments
• Intervene, supervise, advise and assist
• Improve teaching, resources and the environment
6
Dyckhoff, A L, Lukarov, V, Muslim, A, Chatti, M A, & Schroeder, U. (2013).
Supporting Action Research with Learning Analytics. Paper presented at LAK13.

7. Learners use analytics to
• Monitor their own activities and interactions
• Monitor the learning process
• Compare their activity with that of others
• Increase awareness, reflect and self reflect
• Improve discussion participation
• Improve learning behaviour
• Improve performance
• Become better learners
• Learn!
7
Dyckhoff, A L, Lukarov, V, Muslim, A, Chatti, M A, & Schroeder, U. (2013).
Supporting Action Research with Learning Analytics. Paper presented at LAK13.

8. Analytics example: UK schools
8
• Aligned with
clear aims
• Huge and
sustained effort
• Agreed proxies
for learning
• Clear and
standardised
visualisation
• Driving
behaviour at
every level
BUT
• Stressed, unhappy learners
• Analytics with little value for learners or educators
• Omission of key areas, such as collaboration

9. Analytics example: Course Signals
Developed at Purdue University
9
Arnold, K E, & Pistilli, M (2012). Course Signals at Purdue: Using Learning Analytics
To Increase Student Success. Paper presented at LAK12, Vancouver, Canada.

10. Analytics example: SNAPP
Network analysis
10

11. Analytics example: iSpot
Heading
11

12. Making the links
between learning
analytics and
learning design
12

13. Learning design in MOOCs
● Puts the learning journey at the heart of the design process
● Provides a set of tools and information to support a learner-
activity based approach
● Helps to show the costs and performance outcomes of
design decisions
● Enables the sharing of best practice
● Helps MOOC designers to choose and integrate a coherent
range of media, technologies and pedagogies
● Enables a consistent and structured approach
to review and analytics
13
Mor, Y, Ferguson, R, & Wasson, B. (2015). Editorial: learning design, teacher inquiry into student learning
and learning analytics: a call for action. British Journal of Educational Technology, 46(2), 221-229.

14. MOOC learning design tools
• MOOC design template
• MOOC planner
• MOOC map
• Journey planner
14

15. Design template analytics
15
Learning outcome How this is assessed
1. Be able to define an ecosystem.
2. Have joined the iSpot community
and obtained identifications for
animals, plants or fungi.
1. Multiple choice. Week 1, question 5
2. Self report.
Analytics
1. How many attempted that question? How many got it right
1st / 2nd / 3rd time? How many followed the link back to resources?
2. Access to iSpot data. Use of MOOC hashtag. Persistence over time. Ethical
implications of tracking off-site.
Short description of course and learning outcomes

16. MOOC planner
• Delivered
• Reflection
• Collaboration
• Conversation
• Networking
• Browsing
• Assessment
16
Blocking out types of learning activity
Conole, Gráinne. (2010). Learning design – making practice explicit.
Paper presented at ConnectEd, Sydney, Australia. http://cloudworks.ac.uk/cloud/view/4001

17. MOOC planner analytics
Delivered Content (reading, watching, listening and observing)
Analytics: amount of content viewed, dwell time
Reflection (thinking, considering and reflecting)
Analytics: returns to the same material, reflection exercises completed,
quality of reflection
Collaboration (constructing, collaborating, defining and engaging)
Analytics: collaboration exercises completed, quality of collaboration
Conversation (debating, arguing, questioning, discussing…)
Analytics: number and length of contributions, quality of discussion
Browsing (exploring, searching, finding and discovering)
Analytics: Number of click-throughs to external links, number of visits,
number of resources
Assessment (answering, presenting, demonstrating, critiquing…)
Analytics: Assessments completed, scores, dwell time on hints,
persistence in answering questions
17

18. MOOC map analytics
● How long did you expect learners
to spend on these key elements?
● How long did learners actually
spend on the key elements
● How many missed out these
elements?
● How many jumped ahead to
these elements?
● Which types of element are
consistently (un)popular?
● How many left the MOOC at
these points?
18
The MOOC map identifies key elements of the course
0
100
200
300
400
500
600
Assimilative
InformationHandling
Productive
Experiential
Adaptive
Communicative
Assessment
Organisation
Minutes

19. MOOC journey planner analytics
19
Relationships between tools, resources, activities & narrative
A framework for data collection

20. Analytics to solve problems
Analytics could filter discussions or group learners
20
You have been actively engaged in the
discussions, which is excellent, thank you,
but with more than 23,000 participants it
means that our responses and comments
risk getting lost.
This will be primary school material for
some of you and exactly the opposite for
others. It is just not possible to tailor the
material to each of you […]
Introduction to
Forensic Science:
University of Strathclyde

21. Start with the pedagogy
• How do people learn?
• How can we use data to facilitate that process in our MOOC?
• Which elements are learners struggling with?
• Which sections engage them the most?
• What prompts them to ask questions?
• Are they finding assessment challenging?
• What misconceptions have learners shown?
• Are there any accessibility issues?
• How can analytics be used to obtain desired learning outcomes?
21

22. Learning analytics and design
Learning design – helping to identify useful analytics
● What do learners need to know in order to –
network, collaborate, browse or reflect?
● What do educators need to know to support them?
Learning design – helping to identify gaps in the data
● What data do we need to collect?
Learning design – helping to identify gaps in our toolkit
● Which design elements can we look at easily?
● Which ones still pose problems?
Learning design – helping to frame & focus analytics questions
● What did they learn?… in relation to learning outcomes
● Were they social?... when they were collaborating
● Did they share links?... when encouraged to browse
● Did they return to steps?... when encouraged to reflect
22
Making the links

23. Workshop activity
23

24. Learning outcomes
24
Make a note of the learning outcomes in these two videos
Promotional video for The University of Southampton’s FutureLearn MOOC: Agincourt 1415: Myth and Reality
You can currently access this without registering at futurelearn.com/courses/agincourt

25. Learning outcomes
By the end of the course, learners should know:
●Something about medieval weaponry from the period
●How important the long bow was at the battle of Agincourt
●How important gunpowder weaponry was at the time
●Why the battle still means so much today
●How historians operate
●What the Hundred Years War was
●Whether Henry V had a legitimate claim to the French crown
●Why Henry launched the campaign when he did
●How the soldiers were recruited, transported and organised
●What is meant by the ‘Southampton plot’
●Whether their ancestors served on the campaign
25
FutureLearn MOOC from University of Southampton

26. Reflecting, discussing
26
Shakespeare
is to blame
Being French, we were taught
of course about the battle of
Agincourt although not in the
same way as English students
have been! :)
it's the
difference
between myth
and reality.
I do Medieval re-creation in
the SCA and my persona is
that of a historian who lived
only a hundred years after the
Battle.
For me it's like being
interested in ancient sport
reviews, as if the hundred
years war was like the six
nations tournament of
medieval times!
I am an archer
My 17th great grandfather
served as a man at arms in
the retinue of the Duke of
Gloucester at Agincourt.
Agincourt was
presented to us
as the epitome of
britishness

27. My ancestor was
easily found
Exploring, searching, finding
27
Screenshots from The University of Southampton’s FutureLearn MOOC: Agincourt 1415: Myth and Reality
You can currently access this without registering at futurelearn.com/courses/agincourt
I think my family
would have been
on the other side!

28. Debating, arguing
28
Screenshots from The University of Southampton’s FutureLearn MOOC: Agincourt 1415: Myth and Reality
You can currently access this without registering at futurelearn.com/courses/agincourt

29. Thinking, considering, reflecting
29
Screenshots from The University of Southampton’s FutureLearn MOOC: Agincourt 1415: Myth and Reality
You can currently access this without registering at futurelearn.com/courses/agincourt

30. Why the battle still means so much
Delivered Content (reading, watching, listening and observing)
Ask learners to read a recent article on the subject
Reflection (thinking, considering and reflecting)
Ask ‘What reasons are given for the importance of Agincourt?’, and ask
them to watch a video of the recent commemoration event
Collaboration (constructing, collaborating, defining and engaging)
Ask learners to put together their own reasons for considering that the
battle is important
Conversation (debating, arguing, questioning, discussing…)
Ask learners to discuss an audio recording about the battle
Browsing (exploring, searching, finding and discovering)
Point learners to some key resources and ask them to develop a list of
reasons for the importance of the battle
Assessment (answering, presenting, demonstrating, critiquing…)
Ask learner to post a list of the five main reasons for the battle’s
importance, and then comment on another learner’s list 30

31. Group activity
15 minutes
https://www.futurelearn.com/
Take the learning outcomes of the
Agincourt MOOC and briefly outline a
different type of learning activity for each one
(delivered, reflection, conversation,
collaboration, browsing or assessment).
31

32. Types of learning behaviour
Delivered Content
reading, watching, listening and observing
Reflection
thinking, considering and reflecting
Collaboration
constructing, collaborating, defining and engaging
Conversation
debating, arguing, questioning, discussing…
Browsing
exploring, searching, finding and discovering
Assessment
answering, presenting, demonstrating, critiquing…
32

33. Example analytics questions
Read a recent article
Is the article engaging learners? How long did they spend on it? How
many skipped it?
Reflect on a video of a commemoration event
Are learners engaging in behaviours that indicate they are reflecting? Did
they return to the video? Did they pause the video?
Put together reasons why the battle is important
Does this work as a discusssion activity? How many engaged? How
detailed were the comments? Were they key words in the discussion the
same as key words in the previous video?
Develop a list of reasons for the importance of the battle
What misunderstandings do I need to address? What reasons did they
identify? Did they miss the importance of any reasons?
Post a list of five reasons, and then comment
What type of general feedback can I provide as a model for this activity in
future? What were common mistakes? What did they generally get right?
33

34. Group activity
15 minutes
https://www.futurelearn.com/
For each of your learning activities,
identify questions you would like answered in order
to improve learning and teaching on the MOOC
What types of data would help you
to answer these questions?
34