Aggregate judgements threaten to undermine the pedagogical intent of personalised learning. This resource outlines some of the threats to personalised learning, as well as six practical steps for overcoming them.

1. Junaid Mubeen Counter the Average
www.fjmubeen.com
1
How to resist aggregate judgements in personalised learning
COUNTER
THE AVERAGE

2. Junaid Mubeen Counter the Average
www.fjmubeen.com
2
A thought experiment
The way we evaluate an event depends on whether
it is isolated, or part of a larger collection of events.
We will flip a fair coin. If it lands heads, you win $200.
Tails, you lose $100. Would you take the bet? What if we
flipped the coin 100 times, and you were to receive or
pay the net difference?
Thought experiment (Paul Samuelson)
Most people resist the original bet because they cannot
tolerate the 50% chance of losing $100. But they are
prepared to take the second bet, where the losses are
overcome by the wins.
Results
In Education, we often base instruction on work works
‘overall’, even when individual students lose out. Our
approach to personalised learning must resist this impulse
and protect the learning potential of every student.
Reflections

3. Junaid Mubeen Counter the Average
www.fjmubeen.com
3
Averages are popular
because they succinctly
capture overall trends in
data…
Key concept:
…but students are too
complex to be averaged
out. We must stop teaching
to the mythical ‘average
student’.
Threats to
personalised
learning…and how
to overcome them.
Coming up:
Personalised
Learning
The antidote to ‘averagarian’
approaches to education.

4. Junaid Mubeen Counter the Average
www.fjmubeen.com
4
Threats to
Personalised
Learning:
SOME EXAMPLES
Machine learning algorithms that power
personalised learning products use averaging
techniques to make recommendations for one
student based on the behaviours of others.
Machine learning
Product providers issue generic usage guidelines for
implementation based on what works ‘overall’, often
ignoring the needs of each learning environment.
Usage guidelines
Efficacy studies evaluate the impact of personalised
learning based on the ‘overall’ effect on students’
learning.
Efficacy studies
Data dashboards quantify students’ achievement based
on their overall progress, disguising their variation
across learning strands.
Learning analytics

5. Junaid Mubeen Counter the Average
www.fjmubeen.com
5
Six ways to counter the
average
05
02
03
04
06
01
Probe the
context
Dig down to
the individual
Retain
human
judgement
Express
uncertainty
Embrace the
outliers
Iterate fast and
often
5
www.fjmubeen.com

6. Junaid Mubeen Counter the Average
www.fjmubeen.com
6
01
01 03 050402
Dig down to the individual
Make student-level
reporting the
foundation of your
reporting.
Show student-by-
student breakdowns in
class reports so that
teachers don’t have to
settle for averages.
Keep data privacy
requirements at the
forefront of interface
design.
Show the jaggedness in
each student’s profile by
assessing them across
multiple strands.
Make it easy for users to
drill down to student-level
reports from wherever they
are in the reporting
hierarchy.
Group averages, while useful for analysis, may disguise individual student behaviours.

7. Junaid Mubeen Counter the Average
www.fjmubeen.com
7
Machines do not always know best.
Embed hard-
coded expert
human
judgement in
your
algorithms.
Evaluate your
product against
human
performance.
Allow educators
to override
automated
decisions.
02Retain human judgement

8. Junaid Mubeen Counter the Average
www.fjmubeen.com
8
03Probe the context
8Junaid Mubeen Counter the Average
Do not rely solely on
analytics to convey
students’ learning.
Connect with users
on the ground to
understand why the
data looks as it does. Never assume a trend
will transfer from one
context to another -
always account for
local nuances.
There is a human story behind every data point.

9. Junaid Mubeen Counter the Average
www.fjmubeen.com
9
04Express uncertainty
Averages hide more information than they reveal, often disguising
important individual behaviours.
Use standard
deviations and
other measures
of spread.
Make your
model
assumptions
clear.
Plot time series
data to
illustrate
variation over
time.
Consider
alternatives
measures of
central tendency,
e.g. median.
Visualise
uncertainty,
e.g. error
bands.
01 03 05
0402

10. Junaid Mubeen Counter the Average
www.fjmubeen.com
05Iterate fast and often
Lean principles are designed to continuously
monitor and act on learning insights.
Record analytics that are easy to access,
easier to understand, and easiest to act
upon.
Leave room for small, regular tweaks to
product and implementation.
Minimise the length of your iteration
cycles.
Build
MeasureLearn
10

11. Junaid Mubeen Counter the Average
www.fjmubeen.com
11
06Embrace the outliers
In the paradigm of personalised learning, every
student is an outlier.
01
Study the outlier
students in each
distribution and
seek to understand
their behaviours.
02
Never assume the
distribution is a
bell curve without
checking.
03
Consider
alternative
distributions such
as Power Law,
which allow for
more outlier
behaviours.

12. Junaid Mubeen Counter the Average
www.fjmubeen.com
12
Further
reading
› Risk and Uncertainty: A Fallacy
of Large Numbers
› The End of Average (Todd Rose)
› Visualizing uncertainty in data
› When 'personalised learning’
forgets to be ‘personalised’
› The algorithmic paradox of
personalised learning
› The Lean Product Management
manifesto
Image
sources
› https://www.tibco.com/blog/
2013/04/21/why-is-your-
company-settling-for-just-
average/
› https://blog.gradescope.com/
the-average-student-does-not-
exist-bc885a818145
› https://www.krcs.co.uk/news/
personalised-learning-tk
› https://www.fastweb.com/
uploads/article_photo/photo/
2034785/learn-to-read-on-a-
college-level.jpg
› https://en.wikipedia.org/wiki/
Power_law

13. Junaid Mubeen Counter the Average
www.fjmubeen.com
13
Want more blogs and resources like this?
Visit www.fjmubeen.com and subscribe
to the fortnightly newsletter.
Stay in
touch
fjmubeen@gmail.com