Keynote Address: Team-based Learning Collaborative Asia Pacific Community (TBLC-APC) Symposium (“Impact of emerging technologies on learning strategies”) 8-9 February 2024, Sydney https://tbl.sydney.edu.au

1. The Generative AI System Shock:
and some thoughts on Collective Intelligence and Teamwork Analytics
Simon Buckingham Shum
Professor of Learning Informatics
Director, Connected Intelligence Centre
University of Technology Sydney
https://Simon.BuckinghamShum.net
https://www.linkedin.com/in/simon
UTS CRICOS 00099F
Keynote Address: Team-based Learning Collaborative Asia Pacific Community (TBLC-APC) Symposium
“Impact of emerging technologies on learning strategies” 8-9 February 2024, Sydney

2. 2
Visualizing my
communities
LinkedIn Maps (2013)
2024 add…
AI in Education
AI Ethics
2013:

3. Provocation bubbles…

4. Sept. 2022…
“Simon, a student of mine has been playing with something called GPT3…”
“It generates implicit premises for any argument you give it…”

5. Nov. 2022
Launch of ChatGPT

6. Ethical
Engagement
Effective
Engagement
GenerativeAI@UTS
E3: Effective Ethical Engagement
Generative
AI
https://lx.uts.edu.au/blog/2023/02/07/generative-ai-uts-engagement

7. Supporting staff (and HE sector with open access resources)
5 principles for
the effective,
ethical use of
GenAI
Guidelines to help
you determine what
is appropriate in
your context.
AI-resilience
diagnostic for
assessment
tasks
Help you check if
changes required
and mitigation
ideas.
Assignment
redesign drop-in
sessions
Get help to revise
your assessments,
especially those
that require
significant change.
Extra support
from LX Lab
Extra support hours
and staff if you have
a question or can’t
make a drop-in
session.
Guidance on
detection and
academic
misconduct
Know what to do if
you’re concerned
about a breach of
academic integrity.
https://lx.uts.edu.au/blog/tag/ai

8. 5 student-centred principles to translate
“effective, ethical engagement” into practice
Students understand the
significance of GenAI for
society, careers, and studies
1
Students understand
legitimate use of GenAI
in their studies
2
Students are equipped to
engage critically and ethically
with GenAI
3
Students experience GenAI’s
strengths and limitations as
aids to learning
4
Students are assessed on
what they need to know
in an AI world
5

9. Jan-Mar 2023…
What can
ChatGPT / Bing Chat
do?
Myriad exciting
demos
+
a few
evaluations

10. Jan-Mar 2023…
What can
ChatGPT / Bing Chat
do?
Turn that into a
student activity
Myriad exciting
demos
+
a few
evaluations
Myriad creative
ideas for
integrating
ChatGPT into
student tasks

11. Jan-Mar 2023…
What can
ChatGPT / Bing Chat
do?
Turn that into a
student activity
Pilot + Evaluate
Myriad exciting
demos
+
a few
evaluations
Myriad creative
ideas for
integrating
ChatGPT into
student tasks
Stories emerging
from the field
+
research papers
in press

12. Jan-Mar 2023…
What can
ChatGPT / Bing Chat
do?
Myriad exciting
demos
+
a few
evaluations
• Yes they can
• Better than students
• An AI classifier was better than educators at
distinguishing ChatGPT from student writing
https://doi.org/10.1016/j.caeai.2023.100140

13. A deeper dive example:
argument analysis
Buckingham Shum, S. (2024). Generative AI for Critical Analysis: Practical Tools, Cognitive Offloading and Human
Agency. 1st International Workshop on Generative AI for Learning Analytics: 14th International Learning Analytics and
Knowledge Conference (LAK’24), March 18-22, 2024, Kyoto, Japan
Blog https://simon.buckinghamshum.net/2023/05/conversational-genai-for-argument-analysis

14. Jan-Mar 2023…
What can
ChatGPT / Bing Chat
do?
Myriad exciting
demos
+
a few
evaluations
1. Analyse this rebuttal to the open
letter on pausing AI development
2. Identify the argument structure
3. Visualise this as an Argument Map

15. How to support the analysis of these arguments?
https://futureoflife.org/open-letter/pause-giant-ai-experiments/ https://www.dair-institute.org/blog/letter-statement-March2023
challenges

16. Paste the GPT-generated code into the ArgDown web app à Argument Map

17. GPT-generated Argument Map
(green) Elements classified
by Argumentation Scheme
article à GPT analysis à code à visualization
Claim in the original letter
which is not contested
(green supporting premises)
(white) Claims
and Premises

18. Evaluating the Argument Map
X
i
X
X
Correct summary of authors
Hallucination
Fallacy of Fallacy of omission
X
Incorrect term
i
Ad hominem
i “commentary” from Bing
Details in this blog post:
Conversational GenAI for argument analysis

19. GenAI can now apply a conceptual
framework to a text and visualize this.
Does this open new possibilities for
TBL learning designs?

20. What’s happened in the last 3 months?
Turn that into a
student activity
Myriad creative
ideas for
integrating
ChatGPT into
student tasks
https://twitter.com/sharplm/status/1649300115583107072
Mike Sharples (Open U, UK)

21. What if… we mapped the evidence landscape of successes + failures?
Context 1 Context 2 Context 3 Context 4
Example: tracking effectiveness of Sharples’ ChatGPT roles across contexts
Pilot + Evaluate
Stories emerging
from the field
+
research papers
in press
What if… we mapped the evidence for GenAI agents to augment TBL?

22. What would a TBL-framed
matrix look like
— GenAI roles vs contexts?

23. Educators can now articulate what ChatGPT literacy looks like, the ability
range in their cohort, and how to better scaffold students
Pilot + Evaluate
Stories emerging
from the field
+
research papers
in press
ChatGPT literacy template:
Context: <your course>
Task: <student assignment>
Capacity to engage critically:
• The most able students…
• The least able students…

24. Student critical engagement with ChatGPT
What we’re learning at UTS
Context: Applied Natural Language Processing, Master of Data Science
and Innovation
Task: Write a critical summary + visual map of ethical issues in NLP
applications. Encouraged to use ChatGPT for a starter text or to improve
their writing. Reflect on their use of it for learning.
Capacity to engage critically:
• The most able students could engage in deep conversations with AI
using excellent prompts (and follow-up replies)
• Less able students used simple prompts to access content on the
topic, and did not have a deeper discussion with AI
Dr. Shibani Antonette
Lecturer
Transdisciplinary School

25. Student critical engagement with ChatGPT
What we’re learning at UTS
Context: Interaction Design / School of Computer Science
Task: Students use ChatGPT to develop user personas, scenarios and
ideate new design solutions, and reflect critically on it
Capacity to engage critically:
• The most able students could use ChatGPT effectively to get desired
outputs: rich scenarios vividly describing personas’ problem and future
scenarios. (Yet no critical reflection of what makes an AI-generated outcome an appropriate
or accurate response — related in part to the subjective nature of design practice)
• Less able students may still use ChatGPT to get good responses
— but with even less reflection.
• Clearer guidance needed on effective, critical, and responsible use.
More examples and in class activities should be offered
Dr. Baki Kocaballi
Senior Lecturer
Faculty of Engineering &
Information Technology

26. Student critical engagement with ChatGPT
What we’re learning at UTS
Dr. Anna Lidfors Lindqvist
Lecturer
Faculty of Engineering
& Information Technology
Context: Mechanical Design Fundamental Studio 1
Task: Student teams building a robot encouraged to use ChatGPT, and
reflect critically on it
Capacity to engage critically:
• The most able students use ChatGPT as a tool for ideation and
brainstorming • refining presentation slides or speeches • checking
calculations • seeking advice during component selection and
comparison.
• Less able students tend to rely solely on ChatGPT's calculations
without verifying accuracy • struggle to apply information in the
context of their project.
• Some chose not to use ChatGPT: too much effort to direct it to do
what they wanted it to achieve.

27. Student critical engagement with ChatGPT
What we’re learning at UTS
Context: Bachelor of Engineering (Civil Eng) – Soil Behaviour Subject
(Year 2/3) – Research Project – Autumn 2023
Task: Assessing ChatGPT output quantitively and qualitatively against
Finite Element Simulation using PLAXIS software for Soil-Structure
Interaction problems
Capacity to engage critically:
• The most engaged students formulated meaningful queries after trial
& error (often 4-7 trials) • distinguished between useful advice/
common misconceptions/errors • more proficient in maths/physics and
interpreting the data
• Least engaged students struggled to articulate their queries (too
broad a question) • took AI’s responses at face value with no critical
assessment or identify errors or misconceptions • struggled to
comprehend the significance of the data
A/Prof. Behzad Fatahi
Subject Coordinator
School of Civil &
Environmental Engineering

28. Do you know the spread of
student GenAI literacy in
your TBL context?

29. GenAI for the teaching team:
distilling learning outcomes
Buckingham Shum, S. (2024). Generative AI for Critical Analysis: Practical Tools, Cognitive Offloading and Human
Agency. 1st International Workshop on Generative AI for Learning Analytics: 14th International Learning Analytics and
Knowledge Conference (LAK’24), March 18-22, 2024, Kyoto, Japan
Blog Co-designing learning outcomes with the UTS-CILObot prototype

30. Universities are configuring their intranet AI
Secure, authenticated, private GenAI: approved part of the tech ecosystem
Course Intended Learning Outcome (CILO) à CILObot

31. UTS Course Handbook:
(Bach. Sport & Exercise Mgnt.)
26 CILOs
1. Lead and manage in sport, exercise, and health contexts
Original CILOs:
1.Lead, manage and inspire within the fields of sport, exercise and health
1.1 Demonstrate leadership to individuals, groups and organisations in the fields of sport, exercise and
health
1.2 Develop and sustain collaborative partnerships with industry and professionals
1.3 Apply contemporary management practices to enable effective outcomes
2. Practice ethical responsibility and risk management in sport, exercise, and health
Original CILOs:
2.Take personal, social and ethical responsibility for their contribution to sport, exercise and health
2.1 Recognise the importance of personal, social, ethical and legal accountability in sport, exercise and
health
2.2 Assess and manage safety and risk appropriate to the client and context
2.3 Provide services using resources appropriately to ensure sustainable and equitable access
3. Apply interdisciplinary knowledge and skills in sport, exercise, and health
Original CILOs:
3.Competently apply knowledge and skills within the sport, exercise and health professions
3.1 Apply knowledge and skills in key content areas, including anatomy, biomechanics, exercise
physiology, sports psychology, motor learning and exercise prescription
3.2 Integrate knowledge and skills from key content areas to develop evidence-based interventions that
meet the unique needs of clients
4. Engage in research and critical analysis to address issues in sport, exercise, and health
Original CILOs:
4.Engage in research and critical thinking to integrate diverse knowledge and develop creative, effective
and evidence-based solutions
4.1 Identify, access and critically evaluate appropriate information resources
4.2 Develop and apply evidence-based systems to address contemporary issues in sport, exercise and
health
4.3 Engage with current international perspectives in the sport, exercise and health professions
5. Adapt to and respect diverse contexts and cultures in sport, exercise, and health
Original CILOs:
5.Adapt to diverse industry contexts to enable optimal and sustainable sport, exercise and health
outcomes
5.1 Manage and adapt the environment to maximise outcomes for a range of clients and stakeholders
5.2 Develop individualised experiences that are socially and environmentally responsible and provide
sustainable health outcomes
5.3 Act with respect and sensitivity to culture
6.3 Recognise the diversity of Indigenous Australians and integrate this knowledge into practice
6. Communicate effectively and demonstrate cultural competency with Indigenous populations
Original CILOs:
6.Demonstrate the ability to communicate effectively and sensitively with diverse populations to enable
positive change
6.1 Utilise a range of communication strategies to promote sport, exercise and health for individuals and
groups with diverse needs
6.2 Effectively collaborate with a range of sport, exercise and health professionals to develop optimal
solutions
7.Demonstrate professional cultural competency which contributes to the health and wellbeing of
Indigenous Australians, inclusive of physical, social, emotional and spiritual wellness
7.1 Demonstrate respect and value for world view differences and in particular Australian Indigenous
ways of knowing, being and doing
7.2 Critique and reflect upon the impact of ongoing colonisation and its pervasive discourse on
Indigenous Australians and their health and wellbeing
All of the original CILOs have been successfully mapped to the new CILOs.
UTS Azure OpenAI GPT-4:
mapping to 6 CILOs
Automated
mapping

32. Universities are configuring their intranet AI
• prompt engineering informed by academics and learning designers
• system prompt incorporates Bloom’s Taxonomy, UTS Indigenous-CILOs,
grounded in a CILO design corpus
• the results for several programs in our Health faculty validated by
disciplinary experts
• agreeing on how to distill 20-30 CILOs into 6 would normally be a
minimum of 3 hours’ meeting between the Course Director and the
program’s lead academics
• CILObot generates a coherent first draft in about 30 seconds

33. Could “TBL bots” accelerate the
adoption of TBL among new
educators?
Or provide feedback on TBL learning
designs?

34. Collective Intelligence

35. https://www.youtube.com/watch?v=GuPfS-nE7Sk

36. Example CI tool:
Supermind Ideator
Steven R. Rick, Gianni Giacomelli, Haoran Wen, Robert J. Laubacher, Nancy Taubenslag, Jennifer L. Heyman, Max Sina
Knicker, Younes Jeddi, Hendrik Maier, Stephen Dwyer, Pranav Ragupathy, Thomas W. Malone (2023). Supermind Ideator: Exploring
generative AI to support creative problem-solving (2023). arXiv (3 Nov. 2023) https://doi.org/10.48550/arXiv.2311.01937

37. Double-diamond process
à Supermind Ideator software
+ “Supermind”
methodological moves

38. MIT Supermind Ideator: custom user interface onto GPT4 to generate
creative solutions for team reflection

39. Example: CI tool
SWARM
Van Gelder, T., De Rozario, R., & Sinnott, R. O. (2018). SWARM: Cultivating Evidence-Based Reasoning.
Computing in Science & Engineering, 20(6), 22-34. https://doi.org/10.1109/mcse.2018.2873860
Van Gelder, T., Kruger, A., Thomman, S., De Rozario, R., Silver, E., Saletta, M., Barnett, A., Sinnott, R. O.,
Jayaputera, G. T., & Burgman, M. (2020). Improving Analytic Reasoning via Crowdsourcing and
Structured Analytic Techniques. Journal of Cognitive Engineering and Decision Making, 14(3), 195-217.
https://doi.org/10.1177/1555343420926287

40. SWARM team interface for reasoning on intelligence problems
Problem Responses Chat

41. SWARM team interface for reasoning on intelligence problems
Rating Tool
Aggregate Ratings

42. SWARM evaluation study
“In this study, analyst teams on SWARM substantially outperformed those same
analysts using normal approaches.”
“…it is possible to get very good analytic reasoning from people recruited through
social media, assigned randomly into large groups, given very basic training, and
offered no payment or other extrinsic reward.”
“our finding that crowd-sourcing on SWARM produced substantially better reasoning
than analysts using (something like) their normal methods lends early support to the
[…] conjecture that substantial gains in analytic reasoning might be obtained by using
systems integrating crowdsourcing with appropriate Structured Analytic Techniques.”
“teams tend to perform better not just when they generate more draft Reports, but also
when those draft Reports are more different from each other”

43. Contested Collective
Intelligence
De Liddo, A., Sándor, Á., & Buckingham Shum, S. (2012). Contested Collective Intelligence: Rationale, Technologies, and a Human-Machine
Annotation Study. Computer Supported Cooperative Work, 21(4-5), 417-448. https://doi.org/http://dx.doi.org/doi:10.1007/s10606-011-9155-x
Blogs: https://simon.buckinghamshum.net/tag/argument-mapping
…because people disagree, and machines can help track where and why

44. Mapping a nuclear power debate
from a book

45. Thanks to Anna De Liddo for mapping this Mark Lynas blog debate

46. 58
A megacryometeor is a giant hailstone; A blue-
sky megacryometeor is one that falls out of a
clear blue sky. Map based on: Douglas, E.
(2007). Mystery of the monster hailstones. New
Scientist, 23 Dec. 2007.
Hypothesis Mapping
(Tim van Gelder, Austhink Consulting)

47. Example:
Deliberation Analytics
Klein, M. (2012) Enabling Large-Scale Deliberation Using Attention-Mediation Metrics. Computer-
Supported Collaborative Work, 21(4-5):449-473.
Buckingham Shum, S., De Liddo, A., & Klein, M. (2014). DCLA Meet CIDA: Collective Intelligence
Deliberation Analytics. 2nd Int. Workshop on Discourse-Centric Learning Analytics, at 4th Int. Conf. on
Learning Analytics & Knowledge, Indianapolis, March 24 2014.
Klein, M. (2015). The Catalyst Deliberation Analytics Server. MIT Technical Report.

48. Examples of deliberation analytics (EU Catalyst Project)

49. Examples of deliberation analytics (EU Catalyst Project)

50. Collocated teamwork analytics:
making team processes visible, searchable, comparable (CI)
https://TeamworkAnalytics.net
Vanessa Echeverria, Lixiang Yan, Linxuan Zhao, Sophie Abel, Riordan Alfredo, Samantha Dix, Hollie Jaggard, Rosie Wotherspoon, Abra Osborne, Simon Buckingham
Shum, Dragan Gašević, and Roberto Martinez-Maldonado. 2024. TeamSlides: a Multimodal Teamwork Analytics Dashboard for Teacher-guided Reflection in a
Physical Learning Space. 14th
Learning Analytics and Knowledge Conference (LAK ’24), March 18–22, 2024, Kyoto. https://doi.org/10.1145/3636555.3636857
Buckingham Shum, S., Echeverria, V. & Martinez-Maldonado, R. (2019). The Multimodal Matrix as a Quantitative Ethnography Methodology. In: Eagan B., Misfeldt, M.
& Siebert-Evenstone, A. (Eds.), Advances in Quantitative Ethnography. Communications in Computer and Information Science, Vol. 1112. Springer: Cham, pp.26-40.
DOI: https://doi.org/10.1007/978-3-030-33232-7_3

51. TeamSlides dashboard to support debriefing nursing students after a
simulation exercise

52. TeamSlides dashboard to support debriefing nursing students after a
simulation exercise

53. What “TBL analytics” would you love
to see as an educator?
What would you feed back to
students for reflection?

54. In conclusion…

55. We’re moving towards hybrid human+AI
teamwork and collective intelligence
GenAI
GenAI
Student
Student
What might this
mean for TBL?