Workshop Learner Enhanced Technology: Can activity analytics support understanding engagement a measurable process? Delivered 20th Jan at HETL SOTE 2015, Utah Valley University.
1. Learner Enhanced
Technology
Can activity analytics support understanding engagement a measurable process?
James Ballard – HETL SOTE 2015, Utah Valley University
jameslballard
JamesBallard2
@jameslballard
2. My Research
Questions
What can online
learner activity
tell us about
engagement?
How can this be
used to inform
learning design?
“As we start a new school year, Mr. Smith, I just want you
to know that I’m an Abstract-Sequential learner and trust
that you’ll conduct yourself accordingly!”
4. Icebreaker Activity
Should engagement constitute a formal assessment of
teaching?
5 mins – introduce yourselves and give initial reactions to this question
9. Outcomes
JISC Transforming
Curriculum Delivery
Retention: 62% to 92%
Achievement: 69% to 73%
Anecdotal evidence that students felt more
supported
JISC Distributed Learning
Environments
Student satisfaction for course organisation
up 1.8%
Student satisfaction for appropriate
resource selection up 3.5%
Guardian University Award for Student
Experience 2013
EUNIS Elite Award for Excellence 2014
LeSoCo College 2008-2010 MMU 2010-2011
10. PERFORMATIVITY AND POWER
Constant supervision and correction
constrains learners through the need
to achieve conformity
HUMANISM AND DEMOCRATIC EDUCATION
Reflection
Ballard (2014)
Problem
solving
Human
expression
Lived
experience
Can we design learning
systems that promote more of
the right than the left?
12. Involvement Interaction Intimacy Influence
Engagement as Metric
What to track
• Site visits
• Time spent
• Pages viewed
• Search
keywords
• Navigation
paths
• Site logins
• Contributed
comments to
blogs
• Quantity /
frequency of
written
reviews, blog
comments,
forum posts
• Sentiment
tracking on 3rd
party sites
• Sentiment
tracking of
internal sites
• Opinions
expressed in
customer calls
• Net Promoter
(NP) score
• Service
satisfaction
ratings
• Brand affinity
• Content
forwarded to
friends
How to track
• Web Analytics • eCommerce
• Social Media
• Surveys
• Brand
monitoring
• Service calls
• Surveys
• Brand
monitoring
Haven & Vittel (2008)
Haven (2007)
16. Activity 1 – Instrument
Formation
What might we measure?
17. Conole’s Task Types
Assimilative
• attending and understanding content;
Information Handling
• gathering and classifying resource or manipulating data;
Adaptive
• use of modelling or simulation software;
Communicative
• dialogic activities (e.g. pair dialogues or group based discussion);
Productive
• construction of an artefact such as a written essay, new chemical compound or a sculpture;
Experiential
• practising skills in a particular context or undertaking an investigation
18. Extracted Logs Sample
Date Course Tool Action
03/01/2014 10:03 CIH L3 Cert in Housing
Practice Programme
Course View
03/01/2014 10:03 CIH L3 Cert in Housing
Practice Programme
Course View section
03/01/2014 10:03 CIH L3 Cert in Housing
Practice Programme
Forum View discussions
03/01/2014 10:03 CIH L3 Cert in Housing
Practice Programme
Forum Add post
03/01/2014 10:04 CIH L3 Cert in Housing
Practice Programme
Forum View discussion
09/01/2014 08:51 CIH L3 Cert in Housing
Practice Programme
Assignment View
09/01/2014 08:52 CIH L3 Cert in Housing
Practice Programme
Assignment View confirmation form
09/01/2014 08:53 CIH L3 Cert in Housing
Practice Programme
Assignment Submit
19. Case Study Activity 1
• Extracted log data is
provided for a learner;
• For each log decide which of
Conole’s task types this
represents - keep a tally in
the worksheet;
• A log may be none of the
types and excluded but
cannot more than one type;
• (Optional) Complete the pie
and/or column charts
Activity 1 Discussion
• What type of tasks is this
learner undertaking?
• How might this be used to
inform learning design?
• What other classifications
might you use?
In Small Groups: 20 minutes
20. Activity 2 – Practical
Application
Creating an Engagement Profile
21. Involvement Interaction Intimacy Influence
Appropriating Engagement Metrics
What to track
• Site visits
• Time spent
• Pages viewed
• Search
keywords
• Navigation
paths
• Site logins
• Contributions
to class
• Quantity and
frequency of
content
• Type of
content
accessed
• Learning
dispositions
• Discourse
tracking Social
network
analysis
• Evaluation
surveys
• Links to
outside world
• Extra-
curricular
contributions
• Practice based
research and
reflection
How to track
• System Logs
• Web Analytics
• System Logs
• Learning
Design
• Surveys
• Content
analysis
• Student
Development
Model
22. Case Study Activity 2
• Data for two students on are
provided – focus on student 1.
• Use the engagement metric to
make notes under each phase
of the relevant profile in the
worksheet;
• Rate each phase from low to
high at the bottom;
• (Optional) Make notes on
extra data you would like to
have for each phase
Activity 2 Discussion
• What can online learner
activity tell us about
engagement?
• If we extend the data we
collect can this be used to
design for engagement?
• What other types of data
would be useful for
engagement?
In Small Groups: 35 minutes
23. INVOLVEMENT INTERACTION INTIMACY INFLUENCE
• Above average
activity on site, but
below in course.
• Regular access
throughout year
• Primarily course
access
• Accessed
throughout day
• Interacts with peers
via forums
• Submitted
assignments
• Apparent focus on
low commitment
activities
• Potential strategic
learning strategy
• Forum posts
directed to or in
response to teacher
• Low peer-to-peer
posts
• No shared links
High Low-Medium Low-Medium Low
Student Profile 1: Jo March
24. INVOLVEMENT INTERACTION INTIMACY INFLUENCE
• Anomalously
high activity
profile
• Intermittent
access
• Interacts
predominantly
with course
materials
• No assignment
submissions
• Potential surface
learning strategy
• Peer interactions
not present
• No shared links
Low Low Low Low
Student Profile 2: Oliver Twist
25. Learner Enhanced Technology
Understanding engagement as a
measurable process
Thank You
James Ballard
jameslballard@outlook.com
JamesBallard2
@jameslballard
jameslballard