Formation and Learning Analytics?

Formation and
Learning Analytics?
Simon Buckingham Shum
Knowledge Media Institute, The Open University
Visiting Fellow, Grad. School of Education, University of Bristol
From August: Connected Intelligence Centre, University of Technology Sydney
http://simon.buckinghamshum.net
http://twitter.com/sbskmi
Formation by Design (FxD) Symposium
Georgetown University, Washington, June 30 – July 2, 2014
https://futures.georgetown.edu/formation
1
Emergence

Learning

It’s out of the labs and into products: every learning
tool now has an “analytics dashboard” (a Google image search)
2

https://twitter.com/Wiswijzer2/status/414055472451575808
3

https://twitter.com/Wiswijzer2/status/414055472451575808
“Note:
check
the
huge

diﬀerence
between

knowing
and

measuring…”

4

5
c.f.
“Data
is
the
new
oil”
Scaleable

Precise

Quan?ﬁable

Reprocessable

For Morozov, analytics is
where technological
solutionism hits education:
6
“This flight from thinking
and the urge to replace
human judgments with
timeless truths produced by
algorithms is the underlying
driving force of
solutionism.”

If something like this is the pedagogical architecture
for (trans)formational learning…
7
Structuring
Knowledge
Performing
Generating
Learning
Power
Purposing

If something like this is the pedagogical architecture
for (trans)formational learning…
8
Structuring
Knowledge
Performing
Generating
Learning
Power
Purposing
…what roles can computational
learning analytics play?
The
worldview
clash

is
what
makes
this

fascina?ng!

For discussion on how analytics shape
educational worldviews
9http://people.kmi.open.ac.uk/sbs/2014/06/edmedia2014-keynote

thought
experiment:

could
we
imagine…

liminal
space

analy7cs?

10

liminal space
= opportunity for transformation
or withdrawal
11

Liminal space = opportunity for transformation, or withdrawal
12
So while we initially strive to make our students
feel comfortable on campus, we then must help them balance
their desire for security with the need to take risks and explore new
ideas and possibilities. Rather than attempting to resolve the tension,
a college should help students find their place on this
precarious threshold, in the liminal space
between the familiar and strange, the old and the new.
Johansson & Felten, 2014

Liminal space = opportunity for transformation, or withdrawal
13
“It is when you have left the tried and true, but have not yet
been able to replace it with anything else. [...] It is when you are
between your old comfort zone and any possible new answer. If you
are not trained in how to hold anxiety, how to live
with ambiguity, how to entrust and wait, you will
run... anything to flee this terrible cloud of unknowing.”
Richard Rohr O.F.M. on the spirituality of liminal space

meaningful
behavioural
proxies?

some
clues…

14

interpersonal

engagement
and

social
capital

analy7cs

15

Social Network Analysis (SNAPP)
16
What’s going on
in these discussion forums?

Social Network Analysis to gain insight into peer-peer
and peer-mentor dynamics (SNAPP tool)
http://www.slideshare.net/aneeshabakharia/snapp-20minute-presentation
Bakharia A and Dawson S. (2011) SNAPP: a bird's-eye view of temporal participant interaction. Proceedings of the 1st International Conference
on Learning Analytics and Knowledge. Banff, Alberta, Canada: ACM, 168-173.

Social Network Analysis (SNAPP)
18
http://www.slideshare.net/aneeshabakharia/snapp-20minute-presentation
2 learners connect
otherwise separate
clusters
tutor only engaging
with active students,
ignoring disengaged
ones on the edge

Visualizing and filtering online social ties: by topic and type of tie
Schreurs B., Teplovs C., Ferguson R., De Laat M. and Buckingham Shum S. (2013). Visualizing Social Learning Ties by Type and Topic: Rationale and Concept
Demonstrator. Proc. 3rd Int. Conf. on Learning Analytics & Knowledge. Leuven, BE: ACM, 33-37. Open Access Eprint: http://oro.open.ac.uk/36891

Visualizing and filtering online social ties: by topic and type of tie

disposi7onal

learning
analy7cs

24

Dispositional
Learning
Analytics
Workshop
25
http://learningemergence.net/events/lasi-dla-wkshp
http://learningemergence.net/2014/03/01/
assessing-learning-dispositions-academic-mindsets

Observation
informal and formal
Self-Diagnostic
informal and formal
Behavioural
Analytics
Assessing Learning Dispositions/Mindsets
Future sweetspot...
multiple lenses to
provoke self-reflection
http://people.kmi.open.ac.uk/sbs/2014/02/assessing-learning-dispositions-academic-mindsets

Self-report through reflective blogging
9-10 yr old EnquiryBloggers • Bushfield School, Wolverton, UK
EnquiryBlogger Wordpress Multisite plugins
http://learningemergence.net/tools/enquiryblogger
27

Masters level EnquiryBloggers
Graduate School of Education, University of Bristol
EnquiryBlogger: blogging for Learning Power & Authentic Enquiry
http://learningemergence.net/2012/06/20/enquiryblogger-for-learning-power-authentic-enquiry
28

29
EnquiryBlogger teacher’s
dashboard – direct navigation to
learners’ blogs from the visual
analytic

http://learningemergence.net/2014/03/01/assessing-learning-dispositions-academic-mindsets
Dispositional profile from behavioural traces, to
complement self-report?
Shaofu Huang: Prototyping Learning Power Modelling in SocialLearn
http://www.open.ac.uk/blogs/SocialLearnResearch/2012/06/20/social-learning-analytics-symposium
Simon Knight: http://people.kmi.open.ac.uk/knight/2014/02/knowledge-in-search
Social network patterns,
teamwork effectiveness
and initiation of
relationships
Questioning, arguing
and search behaviours
reveal intrinsic curiosity
and epistemic
commitments
Tagging/sharing/
blogging/social patterns
reveal how you see
connections between
ideas
Behavioural and somatic
traces associated with
perseverance, grit,
tenacity; overcoming
panic/stress when
stretched

narra7ve/

discourse

analy7cs

32
The
Storymaking
Project:
PhD
by
Joanna
Kwiat

hLp://projects.kmi.open.ac.uk/storymaking

1st International Workshop on
Discourse-Centric Learning Analytics
analytics that look
beneath the surface, and
quantify linguistic proxies
for ‘deeper learning’
Beyond number / size / frequency of posts; ‘hottest thread’
http://www.glennsasscer.com/wordpress/wp-content/uploads/2011/10/iceberg.jpg
http://solaresearch.org/events/lak/lak13/dcla13

Why are stories so important to formation?
34
“Fundamentally different to logical argument, the
story positively demands those things that the other
either denies or restricts: emotion, agency,
character, perspective, and so on. To understand a
story on some level is to become engaged in its
telling and this will normally be achieved by
allowing oneself to empathise or identify with the
story’s characters or its narrator.”

Discourse analytics on webinar textchat
Ferguson, R. and Buckingham Shum, S., Learning analytics to identify exploratory dialogue within synchronous text chat. In: 1st International Conference
on Learning Analytics and Knowledge (Banff, Canada, 2011). ACM, pp. 99-103. Open Access Eprint http://oro.open.ac.uk/28955
Can we spot the
quality learning
conversations in a
2.5 hr webinar?

-60
-40
-20
0
20
40
60
80
9:28
9:32
9:36
9:40
9:41
9:46
9:50
9:53
9:56
10:00
10:05
10:07
10:07
10:09
10:13
10:17
10:23
10:27
10:31
10:35
10:40
10:45
10:52
10:55
11:04
11:08
11:11
11:17
11:20
11:24
11:26
11:28
11:31
11:32
11:35
11:36
11:38
11:39
11:41
11:44
11:46
11:48
11:52
11:54
12:00
12:03
12:04
12:05
Average Exploratory
Sheffield, UK not as sunny
as yesterday - still warm
Greetings from Hong Kong
Morning from Wiltshire,
sunny here!
See you!
bye for now!
bye, and thank you
Bye all for now
Given a 2.5 hour webinar, where in the live
textchat were the most effective learning
conversations?
Not at the start and end of a webinar…
Ferguson, R., Wei, Z., He, Y. and Buckingham Shum, S., An Evaluation of Learning Analytics to Identify Exploratory Dialogue in Online Discussions. In: Proc.
3rd International Conference on Learning Analytics & Knowledge (Leuven, BE, 8-12 April, 2013). ACM. http://oro.open.ac.uk/36664

-60
-40
-20
0
20
40
60
80
9:28
9:32
9:36
9:40
9:41
9:46
9:50
9:53
9:56
10:00
10:05
10:07
10:07
10:09
10:13
10:17
10:23
10:27
10:31
10:35
10:40
10:45
10:52
10:55
11:04
11:08
11:11
11:17
11:20
11:24
11:26
11:28
11:31
11:32
11:35
11:36
11:38
11:39
11:41
11:44
11:46
11:48
11:52
11:54
12:00
12:03
12:04
12:05
Average Exploratory
…but if we zoom in on a peak…

-100
0
100
9:28
9:40
9:50
10:00
10:07
10:17
10:31
10:45
11:04
11:17
11:26
11:32
11:38
11:44
11:52
12:03
Averag
Classified as
“exploratory
talk”
(more
substantive
for learning)
“non-
exploratory”
…language is used in a manner more
akin to “Exploratory Talk” (Neil
Mercer)

Rhetorical discourse analytics
41
OPEN QUESTION:
“… little is known …”
“… role … has been elusive”
“Current data is insufficient …”
CONTRASTING IDEAS:
“… unorthodox view resolves …”
“In contrast with previous
hypotheses ...”
“... inconsistent with past
findings ...”
SURPRISE:
“We have recently observed ... surprisingly”
“We have identified ... unusual”
“The recent discovery ... suggests intriguing
roles”
http://technologies.kmi.open.ac.uk/cohere/2012/01/09/cohere-plus-automated-rhetorical-annotation
De Liddo, A., Sándor, Á. and Buckingham Shum, S., Contested Collective Intelligence: Rationale, Technologies, and a Human-Machine Annotation
Study. Computer Supported Cooperative Work, 21, 4-5, (2012), 417-448. http://oro.open.ac.uk/31052

42
Human analyst

43
Human analyst Computational analyst
http://technologies.kmi.open.ac.uk/cohere/2012/01/09/cohere-plus-automated-rhetorical-annotation
De Liddo, A., Sándor, Á. and Buckingham Shum, S., Contested Collective Intelligence: Rationale, Technologies, and a Human-Machine Annotation
Study. Computer Supported Cooperative Work, 21, 4-5, (2012), 417-448. http://oro.open.ac.uk/31052

44
Duygu Simsek’s PhD: http://people.kmi.open.ac.uk/simsek/research/
Glimpses of analytics capable of
detecting higher order thinking.
But humans will always read
differently to machines
Can we correlate this with
“academic writing”, and can such
analytics be used as formative
feedback on drafts?

45
Simsek D, Buckingham Shum S, Sándor Á, De Liddo A and Ferguson R. (2013) XIP Dashboard: http://oro.open.ac.uk/37391
CONTRAST
SUMMARY &
CONTRIBUTION

Evidence Hub: semantic storytelling for
students, practitioners and researchers
Systems Learning & Leadership Evidence Hub: http://sysll.evidence-hub.net
A wizard guides the user through
the submission of a structured
story:
•  What’s the Issue?
•  What claim are you making/
addressing?
•  What kind of evidence
supports/challenges this?
•  Link it to papers/data
•  Index it against the NIC’s core
themes

Evidence Hub:
Argument Maps
The wizard then generates a
structured Knowledge Tree
showing evidence-based
claims (and disagreements)

http://learningemergence.net/2013/07/17/deed-elli-ai-ci-systemic-school-learning
An alternative graphical
network rendering can be
generated to show the
current Argument Map of
supporting and challenging
evidence around an Issue
Evidence Hub:
Argument Maps

Evidence Hub: professional development
http://learningemergence.net/2013/07/17/deed-elli-ai-ci-systemic-school-learning
Issue

Poten?al

Solu?on

Suppor?ng

Evidence

(prac??oner

story)

epistemic
cogni7on

analy7cs

50

De Liddo, A., Buckingham Shum, S., Quinto, I., Bachler, M. and Cannavacciuolo, L. (2011). Discourse-centric learning analytics. 1st Int. Conf.
Learning Analytics & Knowledge (Banff, 27 Mar-1 Apr). ACM: New York. Eprint: http://oro.open.ac.uk/25829
Argumentation analytics (in Debate Hub & Cohere apps)
what epistemic contributions are learners making in the network?
51
Rebecca is playing the
role of broker,
connecting different
peers’ contributions in
meaningful ways We now have the basis
for recommending that
you engage with
people NOT like you…

Generates novel
visualizations to summarise
community arguments on a
given issue
http://consider.it
Structured Deliberation/Argument Maps

Shifts in epistemic commitments?
(Simon Knight, KMi PhD research)
What is it to ‘know’ when we search? http://sjgknight.com/finding-knowledge/2014/02/knowledge-in-search
Danish exams permit Net: http://sjgknight.com/finding-knowledge/2013/07/danish-use-of-internet-in-exams-epistemology-pedagogy-assessment
Epistemic networks for epistemic commitments: http://oro.open.ac.uk/39254
Learning sciences research into
Epistemic Commitments/
Cognition combined with
‘researching search’
methodologies.

Dimensions of Epistemic Belief
Certainty The degree to which knowledge is conceived as stable or changing, ranging from
absolute to tentative and evolving knowledge
Simplicity The degree to which knowledge is conceived as compartmentalised or
interrelated, ranging from knowledge as made up of discrete and simple facts to
knowledge as complex and comprising interrelated concepts
Source The relationship between knower and known, ranging from the belief that
knowledge resides outside the self and is transmitted, to the belief that it is
constructed by the self
Justification What makes a sufficient knowledge claim, ranging from the belief in observation
or authority as sources, to the belief in the use of rules of inquiry and evaluation
of expertise
Knight, Simon; Buckingham Shum, Simon and Littleton, Karen (2014). Epistemology, assessment, pedagogy: where
learning meets analytics in the middle space. Journal of Learning Analytics (In press). http://oro.open.ac.uk/39226

Behavioural Proxies of Epistemic Belief?
Certainty
Frequency and density of positive and negative polarity links (e.g. supports; builds on, is consistent with, versus challenges; refutes;
is inconsistent with).
Presence of stability markers – (e.g. current sources, bibliographic references, geographical spread).
Frequency and density of meta-discourse markers in the text of nodes, which moderate assertions in scholarly ways (e.g. in contrast to; remains
an unresolved issue)
Simplicity
Standard network analytics describing the topography of the structure
Semantic analysis of node and link types describing the variety of types used by a learner, the balance in link polarity (positive/neutral/negative),
and the balance of propositional (causes; refutes; solves) versus analogical thinking (is a metaphor of; is analogous to).
Semantic analysis of tags on nodes and links
Encapsulation of node clusters or sub-networks within a broader category expressing a higher order construct
The use of different argumentation schemes, which reflect different sources of authority (argument from expertise; argument by analogy;
argument from precedent)
In collaborative maps, the social network view showing who is connected to whom, but what kinds of links
Source
Presence within the text of nodes of ‘I think’ or other restatements of fact
Challenges to claims through negative link types and meta-discourse markers in node text
Few additional nodes made other than those created as quotations.
Justification
Judgements of relevance, and supporting or explanatory notes (‘this evidences/ explains x’). Ties to method ‘ideas’.
Use of well known Argumentation Schemes, especially evidence and arguments that grounded in the attribution of authority to people (‘experts’)
or publications based on their status (‘valuing peer review’)
Rhetorical constructions in node text which show a critical use of sources

analytics for wider
educational outcomes
Deakin Crick, R., Barr, S., Green, H. and Pedder, D. (2013). Evaluating the Wider Outcomes of Schools: Complex Systems Modelling. Centre for
Systems Learning & Leadership, Graduate School of Education, University of Bristol, UK. http://learningemergence.net/library/reports/

Providing a more complete analytic on the
health of the educational community
57
“Test grades are just ONE of a dozen indicators in
our vision of a healthy educational community”
• life-long/life-wide learners • citizenship
• self-confidence • teamwork • emotional
intelligence • “wellbeing”…

Hierarchical Process Modelling
(Univ. Bristol Perimeta tool)
58
Seeing a learning community as a complex adaptive system requires
the voices of learners, teachers, leaders, parents and ‘external’
stakeholders

The core mission and values of
a network of school academies

60
Hierarchical Process
Modelling transforms
qualitative and quantitative
inputs into a multi-level
visual analytic

‘Italian Flag’ visual analytic
61
Degree of green / white / red reflects current
(un)certainty over availability of evidence:
supporting / unknown / challenging

analytics as mirrors
to provoke
(trans?)formative
reflection

Your most
recent mood
comment:
“Great, at
last I have
found all the
resources that
I have been
looking for,
thanks to
Steve and
Ellen.
In your last discussion with your mentor, you decided
to work on your resilience by taking on more
learning challenges
Your ELLI Spider
shows that you
have made a start
on working on
your resilience,
and that you are
also beginning to
work on your
creativity, which
you identified as
another area to
work on.
1 2 3
45
Envisioning a social learning analytics dashboard
Ferguson R and Buckingham Shum S. (2012) Social Learning Analytics: Five Approaches. Proc. 2nd International Conference on Learning Analytics &
Knowledge. Vancouver, 29 Apr-2 May: ACM: New York, 23-33. DOI: http://dx.doi.org/10.1145/2330601.2330616 Eprint: http://oro.open.ac.uk/32910
63

liminal
space
analy7cs?

Use of computational techniques to reflect back
to learners apparent shifts in • how they engage
in a community of enquiry/practice • how they
narrate their learning journeys • how they
negotiate meaning • how their worldview seems
to have shifted • how they respond cognitively
(even somatically) when challenged.

towards a networked
improvement community
(NIC) for FxD?
hLp://learningemergence.net/2014/05/29/learning-‐analy?cs-‐plus-‐nics-‐for-‐systemic-‐edu-‐improvement

Defining features of a NIC
‘B’ + ‘C’ Level improvement
‘A’
—
what

prac??oners
do
in
a

given
ins?tu?on/
context

‘B’
—
trying
to

improve
local

prac?ce

‘C’
—
the
network

of
‘B’
people

sharing
insight
and

coordina?ng

innova?on

Based
on
Doug
Engelbart’s
forma?ve
work:

hLp://www.dougengelbart.org/about/nics.html

Defining features of an Educational-NIC
a
sociotechnical
infrastructure
and
modus
operandi
for
coordinated
aLack
on

educa?onal
challenges

an
inten?onal
community

facilitated
by
a
coordina?ng
hub

using
agreed
representa?ons
of
the
system
and
improvement
strategies

in
order
to
orchestrate
ac?on

in
pursuit
of
measurable
goals

by
sharing
data,
insights
and
arguments

in
order
to
build
evidence-‐based
claims

Bryk
et
al,
Carnegie
Founda?on
for
the
Advancement
of
Teaching:

hLp://www.carnegiefounda?on.org/sites/default/ﬁles/bryk-‐gomez_building-‐nics-‐educa?on.pdf

A NIC is in essence a “Collaboratory”
“a collaboratory is more than an elaborate collection of ICT; it is
a new networked organizational form that also
includes social processes; collaboration
techniques; formal and informal
communication; and agreement on norms,
principles, values, and rules” (Cogburn, 2003, p. 86).
Cogburn, D. L. (2003). HCI in the so-called developing world: what’s in it for everyone, Interactions, 10(2), 80-87, New York: ACM Press.
http://dx.doi.org/10.1145/637848.637866

Building blocks of an Educational-NIC
NIC Data
Hub
Collective
Sensemaking
Coordinating
Maps
(of the system,
problem, and
strategies)
Coordinating
Maps
(of the system,
problem, and
strategies)
Coordinating
Maps
(of the system,
problem, and
strategies)
Human + Machine
Analytics
Educational
ContextEducational
Context

NIC Data
Hub
Collective
Sensemaking
Human + Machine
Analytics
Educational
ContextEducational
Context
Coordinating
Maps
(of the system,
problem, and
strategies)
Coordinating
Maps
(of the system,
problem, and
strategies)
Coordinating
Maps
(of the system,
problem, and
strategies)

Shared Conceptual Maps:
of system / problem / strategies
http://www.carnegiefoundation.org/elibrary/getting-ideas-action-
building-networked-improvement-communities-in-education
http://www.carnegiealphalabs.org/persistence/

for knowledge navigation
People / Organizations / Projects / Claims / Evidence

for knowledge navigation
People / Organizations / Projects / Claims / Evidence
Evidence Hub for Research by Children & Young People: http://rcyp.evidence-hub.net

NIC Data
Hub
Human + Machine
Analytics
Educational
ContextEducational
Context
Coordinating
Maps
(of the system,
problem, and
strategies)
Coordinating
Maps
(of the system,
problem, and
strategies)
Coordinating
Maps
(of the system,
problem, and
strategies)
Collective
Sensemaking

Impact Map
http://chaos.open.ac.uk/hypothesis-list/
Impact Maps were inspired
by the Evidence Hub, and
are organised around core
hypotheses that the NIC is
investigating.

Impact Map
http://oermap.org/hypothesis/578/hypothesis-a-performance/

Impact Map
http://oermap.org/hypothesis/578/hypothesis-a-performance/
Impact Maps use Sankey flow diagrams to
show how findings from different contexts
support or challenge each of the NIC’s
core hypotheses

Formation and Learning Analytics?

Recommandé

Recommandé

Contenu connexe

Similaire à Formation and Learning Analytics?

Similaire à Formation and Learning Analytics? (20)

Plus de Simon Buckingham Shum

Plus de Simon Buckingham Shum (20)

Dernier

Dernier (20)

Formation and Learning Analytics?