Keynote, Learning Analytics Summer Institute, Ann Arbor, June 2017

1. Learning Analytics vs Cognitive Automation:
Rationale, Examples & Organisational Strategy
Simon Buckingham Shum
Connected Intelligence Centre • University of Technology Sydney
utscic.edu.au
Keynote, Learning Analytics Summer Institute, Ann Arbor, June 2017
@sbuckshum • Simon.BuckinghamShum.net

2. Acknowledgements to CIC colleagues whose work I’ll be sharing…

3. 3
UTS learning strategy, supported by
analytics innovation + impact
implications for learning analytics
+ some examples
the machines are coming…
cognitive automation

4. San Francisco, Fall Joint Computer Conference — Dec. 9th 1968

7. we need better tools to tackle
“humanity’s complex,
urgent problems”

8. 8
dougengelbart.org

9. US change index of work tasks 1960-2009
9
Frank Levy and Richard J. Murnane, Dancing with Robots: Human
Skills for Computerized Work (Washington, DC: Third Way, 2013), Fig.3

10. Routine manual and
cognitive work is
being automated
Only non-routine
manual and
cognitive work will
be done by people
(What counts as
‘non-routine’ will
reduce as AI
improves) 10

11. Jason Furman, Chairman of the Council of Economic Advisers
https://artificialintelligencenow.com/schedule/conference/presentation/time-different-opportunities-and-challenges-artifi

12. learning
12
analytics?
what are the implications for

13. utscic.e
du.au
As analytics aggregate lower level data
& A.I. gradually automates routine cognitive work…

14. utscic.e
du.au
Humans must move to the higher ground…
• Train data scientists to combine algorithmic intelligence with
creative intelligence, and ethical mindsets
• Deploy all the Educational & Data Science expertise we have to
cultivate the higher order graduate qualities
As analytics aggregate lower level data
& A.I. gradually automates routine cognitive work…

15. utscic.e
du.au
Humans must move to the higher ground…
• Train data scientists to combine algorithmic intelligence with
creative intelligence, and ethical mindsets
• Deploy all the Educational & Data Science expertise we have to
cultivate the higher order graduate qualities
As analytics aggregate lower level data
& A.I. gradually automates routine cognitive work…
Cultivate those qualities that are distinctively human
and devise practical, authentic ways to evidence them

16. The endless cycle…
16
What we take to be
“distinctively human” has
always been in transition, but
now at unprecedented pace

17. 17
At its best, education – esp. Higher Education –
should provide the conditions for an
integrated
possibly even transformational
experience

18. So we’re talking about…
Providing authentic learning contexts,
and assessments that build lifelong
learning capabilities
Focusing not solely on course
completion, but on whether students
have a sense of calling
Cultivating curiosity, agency, resilience,
an appetite for risk, a thirst for
different perspectives
Requiring evidence that academic
knowledge is being integrated with
personal experience, identity,
increasingly professional disposition

19. Towards analytics for a holistic higher education
Randy Bass
Georgetown University
Reinvent the University
for the Whole Person
http://reinvent.net/series/reinvent-the-university

20. Harnessing ‘disintegrative’ forces in higher education
in the service of integrative learning for holistic devpt.?
Randall Bass & Bret Eynon (2017).
From Unbundling to Rebundling: Design
Principles for Transforming Institutions in the
New Digital Ecosystem. Change: The Magazine
of Higher Learning, 49, (2), pp. 8-17. DOI:
http://dx.doi.org/10.1080/00091383.2017.1286211
Based on:
https://www.aacu.org/publications-
research/publications/open-and-integrative-
designing-liberal-education-new-digital
An interest in
narrative, e-portfolios
+ analytics that
empower learners to
reflect on their journey

21. disciplinary knowledge
+ transferable skills
+ open learning dispositions
/ innovation mindsets
21

22. “Knowledge of methods alone will not suffice:
there must be the desire, the will,
to employ them.
This desire is an affair of personal disposition.”
John Dewey
Dewey, J. How We Think: A Restatement of the Relation of Reflective Thinking to the Educative Process. Heath and Co, Boston, 1933
Knowledge, Skills & Dispositions

23. “One of the key issues emerging from these findings
was the learner’s orientation towards the
unknown, uncertainty and ambiguity, and
their tendency to either retreat from it or
move into it. The former effectively precludes deep
learning, and the latter is the beginning point for it.”
Ruth Deakin Crick & Chris Goldspink
Deakin Crick R. and Goldspink G. (2014) Learning Dispositions, Self-theories and Student Engagement,
British Journal of Educational Studies, 62,1,1-17. DOI: http://dx.doi.org/10.1080/00071005.2014.904038
Knowledge, Skills & Dispositions

24. Institute for the Future
http://www.iftf.org/futureworkskills

25. utscic.edu.au
Institute for the Future
http://www.iftf.org/futureworkskills

26. learning
26
analytics?
what are the implications for

27. 27
learning
in the wild

28. Analytics beyond the LMS: xAPI-based aggregation
28
?
?

29. Connected Learning Analytics Toolkit
K. Kitto, A. Bakharia, M. Lupton, D. Mallet, J. Banks, P. Bruza, A. Pardo, S. Buckingham Shum, S. Dawson, D. Gašević, G. Siemens, & G. Lynch, (2016). The connected
learning analytics toolkit. Proceedings of the Sixth International Conference on Learning Analytics & Knowledge (LAK '16). ACM, New York, NY, USA, 548-549.
https://doi.org/10.1145/2883851.2883881
Student dashboard Educator dashboard

30. 30
learningembodied

31. Sensing co-located teamwork
Martinez-Maldonado, R., Power, T., Hayes, C., Abdipranoto, A., Vo, T., Axisa, C., and Buckingham-Shum, S. (2017) Analytics Meet Patient Manikins: Challenges in an
Authentic Small-Group Healthcare Simulation Classroom. International Conference on Learning Analytics and Knowledge, LAK 2017, 90-94.

32. Tracking nurses’ movement around the patient
Martinez-Maldonado, R., Pechenizkiy, M., Power, T., Buckingham-Shum, S., Hayes, C. and Axisa, C. (2017) Modelling Embodied Mobility Teamwork Strategies in a
Simulation-Based Healthcare Classroom. International Conference on User Modelling, Adaptation and Personalization, UMAP 2017,

33. Movement heatmaps –> sequence analysis
Martinez-Maldonado, R., Pechenizkiy, M., Power, T., Buckingham-Shum, S., Hayes, C. and Axisa, C. (2017) Modelling Embodied Mobility Teamwork Strategies in a
Simulation-Based Healthcare Classroom. International Conference on User Modelling, Adaptation and Personalization, UMAP 2017,

34. 34
analytics
dispositional

35. Grade (x) vs Student Effort (y)
from teacher observational assessments
over the semester
35
Nagy, R. (2016). Tracking and visualizing student effort: Evolution of a practical analytics tool for staff and
student engagement. Journal of Learning Analytics, 3 (2), 165–193. http://dx.doi.org/10.18608/jla.2016.32.8
Seminar: https://utscic.edu.au/events/niccic-redlands-school-8-june-2016

36. 36
Assessing learning dispositions:
Crick Learning for Resilient Agency survey (CLARA)
Deakin Crick, R., Huang, S., Ahmed Shafi, A. and Goldspink, C. (2015). Developing Resilient Agency in Learning: The Internal
Structure of Learning Power. British Journal of Educational Studies: 62, (2), 121-160. http://dx.doi.org/10.1080/00071005.2015.1006574
https://utscic.edu.au/tools/clara • http://clara.learningemergence.com

37. 37
Structural Equation
Model underpinning
CLARA
Deakin Crick, R., Huang, S., Ahmed Shafi, A. and
Goldspink, C. (2015). Developing Resilient Agency in
Learning: The Internal Structure of Learning Power.
British Journal of Educational Studies: 62, (2), 121-
160. http://dx.doi.org/10.1080/00071005.2015.1006574
Knowing how constructs drive each other
not only underpins the survey’s internal
integrity, but could provide the empirical
basis for the automated analysis of
behavioural traces, as proxies for these
constructs

38. Immediate visual analytic generated by CLARA
Feedback to Stimulate Self-Directed Change
A framework for reflection and coaching
Deakin Crick, R., Huang, S., Ahmed Shafi, A. and
Goldspink, C. (2015). Developing Resilient Agency in
Learning: The Internal Structure of Learning Power.
British Journal of Educational Studies: 62, (2), 121-
160. http://dx.doi.org/10.1080/00071005.2015.1006574
40 item survey

39. Scaling CLARA in UTS
Barratt-See, G., Cheng, M., Deakin Crick, R. & Buckingham Shum, S. (2017). Assessing Resilient Agency with CLARA: Empirical Findings from Piloting a Visual Analytics
Tool at UTS. Proceedings UniSTARS 2017: University Students, Transitions, Achievement, Retention & Success. (Adelaide, 1-4 July, 2017). https://utscic.edu.au/tools/clara
Senior student
mentors
introduce 1st Years
to CLARA through
fictional personas,
based on
educator-
specified
archetypes

40. Scaling CLARA in UTS
Barratt-See, G., Cheng, M., Deakin Crick, R. & Buckingham Shum, S. (2017).
Assessing Resilient Agency with CLARA: Empirical Findings from Piloting a Visual
Analytics Tool at UTS. Proceedings UniSTARS 2017: University Students,
Transitions, Achievement, Retention & Success. (Adelaide, 1-4 July, 2017).
https://utscic.edu.au/tools/clara
n=876 n=957
n=548 n=602
§ For the 921 students with post-profile
data, there were significant positive
changes on all 8 dimensions.
§ We can also derive through cluster
analysis significantly different cohort
profiles, inviting reflection and possibly
intervention for those who might be
judged at risk

41. Behavioural proxies for
“Conscientiousness”?
Shute, V. J. and M. Ventura (2013). Stealth Assessment: Measuring
and supporting learning in video games. Cambridge, MA, MIT Press.
Figure 5 from report to The John D. and Catherine T. MacArthur
Foundation Reports on Digital Media and Learning
http://myweb.fsu.edu/vshute/pdf/Stealth_Assessment.pdf

42. Behavioural proxies for
“Conscientiousness”?
Shute, V. J. and M. Ventura (2013). Stealth Assessment: Measuring
and supporting learning in video games. Cambridge, MA, MIT Press.
Figure 5 from report to The John D. and Catherine T. MacArthur
Foundation Reports on Digital Media and Learning
http://myweb.fsu.edu/vshute/pdf/Stealth_Assessment.pdf

43. Behavioural proxies for
“Conscientiousness”?
Shute, V. J. and M. Ventura (2013). Stealth Assessment: Measuring
and supporting learning in video games. Cambridge, MA, MIT Press.
Figure 5 from report to The John D. and Catherine T. MacArthur
Foundation Reports on Digital Media and Learning
http://myweb.fsu.edu/vshute/pdf/Stealth_Assessment.pdf

44. Behavioural proxies for
“Conscientiousness”?
Shute, V. J. and M. Ventura (2013). Stealth Assessment: Measuring
and supporting learning in video games. Cambridge, MA, MIT Press.
Figure 5 from report to The John D. and Catherine T. MacArthur
Foundation Reports on Digital Media and Learning
http://myweb.fsu.edu/vshute/pdf/Stealth_Assessment.pdf

45. reflective writing
45
as a window
mind
on the

46. Automated feedback on reflective writing
§ Reflection is critical to the integration of academic + experiential knowledge
§ This is where you disclose what you’re uncertain about, and how you’ve
changed, in the first person
§ Scholarship clarifies the hallmarks of deeper reflective writing:
Gibson, A., Aitken, A., Sándor, Á., Buckingham Shum, S., Tsingos-Lucas, C. and Knight, S. (2017). Reflective Writing Analytics for Actionable Feedback.
Proc. LAK17: 7th Int. Conf. on Learning Analytics & Knowledge, March 13-17, 2017, Vancouver. (ACM Press). https://doi.org/10.1145/3027385.3027436 Best Paper Award

47. Automated feedback on reflective writing

48. For more examples see:
Learning Analytics for 21st Century
Competencies. (Eds.) Buckingham Shum
S. & Deakin Crick, R. (2016). Journal of
Learning Analytics (Special Section), 3,
(2), pp. 6-212.
http://dx.doi.org/10.18608/jla.2016.32.2

49. Caution! A Learning Analytic approach makes
educational claims (implicitly or explicitly)
49
epistemology
pedagogyassessment
Knight, S., Buckingham Shum, S. and Littleton, K. (2014). Epistemology, Assessment, Pedagogy: Where Learning Meets Analytics in the Middle Space.
Journal of Learning Analytics, 1, (2), pp.23-47. http://epress.lib.uts.edu.au/journals/index.php/JLA/article/download/3538/4156
Knight, S. and Buckingham Shum, S. (2017). Theory & Learning Analytics. Handbook of Learning Analytics. https://solaresearch.org/hla-17
the
middle
space of
learning analytics
In all of the examples shown,
the analytics approach makes
an impact when integrated
coherently into the learning
design, which is in turn driven
by this triangle

50. 50
these kinds of analytics go far beyond
current products: they require
innovation

51. 51
But universities also seek
impact

52. 52
notes on UTS
organisational
strategy

53. 2011
2011
Envisioning “the Data
Intensive University”
UTS-wide Forum to
consider the profound
implications of analytics
“To make UTS a world
leading university of
technology, where staff and
students understand data
[…] and can develop
analytical tools”
UTS-WIDE CONSULTATIONS & STRATEGIC ALIGNMENT

54. UTS-WIDE CONSULTATIONS & STRATEGIC ALIGNMENT
54
2011
2011
Envisioning “the Data
Intensive University”
DIU UTS-wide Forum
2012
Connected Intelligence
Working Party
UTS-wide consultations
and interviews
2013
Connected Intelligence
Strategy
Privacy & Ethics Forum
Plans for a Masters Degree
2014
Director Appointed
Launch of the Centre
UTS-wide engagement
Collaboration Proposals
Invited & Projects Initiated
2015
Masters Degree Launches
Analytics Pilots in Faculties
CIC Staffing Grows
First Pilot Evaluation Data
2016
Growing number of UTS &
Industry Partnerships
Analytical Tools Maturing
Academic Board

55. Faculties
&
Institutes
Student
Support
Units
CIC
Business
Units
DEVELOP & DEPLOY ANALYTICS ACROSS UTS
2017 http://utscic.edu.au

56. UTS FACULTY OF TRANSDISCIPLINARY INNOVATION
https://www.uts.edu.au/future-students/transdisciplinary-innovation

57. UTS INNOVATION & CREATIVE INTELLIGENCE STRATEGY
BACHELORS IN CI & INNOVATION
http://www.uts.edu.au/partners-and-community/initiatives/creative-intelligence • http://www.uts.edu.au/future-students/creative-intelligence-and-innovation

58. Architecting for
Innovation + Impact?

59. Architecting for
Innovation + Impact?
VC
DVC
Operations
IT
BI
Analytics
LMS
Analytics
IT SERVICES MODEL:
Analytics based in BI or LMS team

60. Architecting for
Innovation + Impact?
VC
DVC
Research
Faculty
School
Centre
Learning
Analytics
researchers
FACULTY RESEARCH MODEL:
Analytics based in a centre/school

61. Architecting for
Innovation + Impact? VC
DVC Education
CIC
HYBRID INNOVATION/SERVICES MODEL:
Analytics based in a non-faculty centre
reporting to DVC (Education), staffed by
academics + professional admin team

62. Architecting for
Innovation + Impact?
VC
DVC
Research
Faculty
School
Centre
Academics
DVC
Education
CIC
DVC
Operations
IT
BI
Analytics
LMS
Analytics
HYBRID INNOVATION/SERVICES MODEL:
Analytics based in a non-faculty centre
reporting to DVC (Education), staffed by
academics + admin team

63. Architecting for
Innovation + Impact?
Board Room
VC/DVCs/Deans/Directors
Common Room
Academic staff
Server Room
IT Division
CIC Skillset: strong interpersonal skills
+
Education, Learning Design, Interface
Design, Programming, Web Development,
Text Analytics, Machine Learning,
Statistics, Visualisation, Decision-Support,
Sensemaking, Creativity & Risk,
Participatory Design

64. ADVANTAGES THAT THIS ORG STRUCTURE BRINGS
Operating within the DVC’s
Education & Students Office
enables close coupling with student
services and teaching innovation
Baseline funding provides invaluable
stability for planning projects and
staff
Reporting directly to a DVC, and
talking directly to other operational
directors, gets stuff done
Operating outside a faculty
provides agility for decision-
making, and helpful neutrality

65. 65
lesson 1
“LA+LD”
learning analytics must be aligned with
the learning design, or adds little value

66. 66
lesson 2
“IA not AI”
a key role for analytics is less to judge,
and more to provoke productive reflection

67. What we take to be “distinctively human”
has always been in transition, but now at
unprecedented pace
The task of Learning Analytics may
increasingly be to make persistent and
visible what until now has remained
ephemeral and invisible
LA+LD designs this as formative feedback
to provoke reflection, insight, creativity
and deeper learning