The document discusses artificial intelligence as an effective classroom assistant. It begins by describing what an AI education (AIED) system is and its key components: a pedagogical model, student model, interface, and domain model. An example AIED system called Betty's Brain is presented that provides metacognitive and motivational feedback to students. Several meta-reviews are then summarized that evaluated the effectiveness of AIED/ITS systems compared to traditional classroom instruction and one-on-one tutoring, generally finding AIED systems to be as effective or more so for STEM topics. The degree of effectiveness varied based on factors like subject area and duration of use.

1. Artificial Intelligence as
an Effective Classroom
Assistant
Benedict du Boulay
University of Sussex
UK

2. Bloom (1984) and the 2-sigma
challenge
2-sigma

3. Contents
1. What is an AIED system
2. An example: not just about cognition
3. A range of meta-reviews since 2011

4. What is an AIED system?

5. AIED systems: Adaptivity
• Reason about Pedagogy
• What task to set next
• How best to react to the leaner’s cognitive, affective and
motivational state
• Reason about the domain of instruction
• Follow learner’s problem-solving step by step
• Reason about how much is being learned
• Focused feedback, help and support
• Adjust learning tasks dynamically
• Use interface to model domain
• Both system and learner can reason in the interface

6. AIED (ITS) system: 4 components
Pedagogical model
Student model
Interface
Domain model

7. Betty’s Brain (Leelawong & Biswas,
2008)

8. Johnson (2010) Tactical
Iraqi

9. Recent META-REVIEWS
Comparing AIED/ITSs to whole class teaching and one-
to-one tutoring

10. VanLehn (2011)
Various ITSs
No. of
Studies
Effects And . . .
88 “just as effective
as adult, one-to-
one tutoring for
increasing gains
in STEM topics”
“ITSs should be used to
replace homework,
seatwork and perhaps
other activities but not
to replace a whole
classroom experience”

11. Ma, Adesope, Nesbit, Liu (2014)
Various ITSs
No. of
Studies
Effects And . . .
107 Good effect sizes
against
teacher-led, large
group instruction;
textbooks;
non-ITSs.
No difference against
individualised human
tutoring or small group
instruction

12. Steenbergen-Hu & Cooper (2013)
Various College level ITSs
No. of
Studies
Effects But . . .
39 “Have demonstrated
their ability to
outperform many
instructional methods or
learning activities in
facilitating college-level
students’ learning of a
wide range of
subjects . . .
. . . although they are
not as effective as
human tutors”

13. Steenbergen-Hu & Cooper (2013)
ITSs K-12 Maths
No. of
Studies
Effects But . . .
34 Very small
positive effect
sizes < 0.1
Better for ordinary pupils
than low achievers;
Better if intervention lasted
< 1 year

14. Kulik & Fletcher (2015)
Various ITSs
No. of
Studies
Effects But . . .
50 “ITSs can be very
effective
instructional tools”
Degree of
improvement
depended on whether
standardised or local
tests used

15. Nesbit, Adesope, Liu (2014)
Computer Science ITSs
No. of
Studies
Effects But . . .
22 “A significant
advantage of ITS
over teacher-led
classroom
instruction and over
computer-based
instruction”
No system combined
procedural with
conceptual tutoring

16. Cognitive Algebra Tutor: Koedinger et
al.

17. Pane, McCaffrey & Karam (2014)
Cognitive Tutor Algebra
No. of
Schools
Comparison Effects But . . .
73 high
74 middle
Across 7
USA
states
Post-tests
using
traditional
teaching
First year
not
significant.
Second
year high
schools
significant.
Effect size
0.20
No direct
monitoring of
how the system
actually used

18. -0.19

19. Effect
Size
0.47

20. Conclusions
• Described AIED systems
• Offered an example concerning Motivation and
Metacognition with partially successful evaluation
• Summarised recent meta-reviews
• Generally positive about ITSs vs most forms of
traditional teaching for STEM subjects
• including vs CAI systems
• Blended learning model favoured
• But degree of overlap in coverage between meta-
analyses not checked

21. An Example: not just about
cognition

22. Alison Hull - SQL-Tutor* - 2014
• Based on SQL-Tutor (Mitrovic)
• +Metacognitive & Motivational help and
feedback
• Invites learner to assess his or her confidence
• Tracks self-efficacy and its accuracy
• Provides metacognitive and motivational
feedback at start/end of session and at end of
problem
• Refers back to similar previous situations

25. Accuracy
of self-
efficacy
Performance
High Mid Low
Pessimistic “Even though you have
not found the problems
easy, you really have
made good progress in
successfully completing
problems . . .”
. . . Effort Poor: “Think about
focussing on problems with
complexity rating of 1 or 2 at
the beginning of the next
session to firmly establish
your core knowledge before
expanding on it . . .”
Effort Good: . . .
Realistic . . . . . . . . .
Optimistic “You have made really
good progress this
session, but have the
problems challenged
you enough? Have a
think whether you are
ready to try more difficult
. . . Effort Poor: “You have not
made much progress this
session, but that doesn't
mean the next session has to
be the same . . .”
Effort Good: . . .

26. Accuracy of
self-efficacy
Performance
Middle
Realistic You have made steady progress this session
Your performance matched your confidence this session.
How confident do you feel about answering SQL queries?
Please indicate below.

28. Evaluation
• Northumbria University: one semester.
• Basic SQL-Tutor vs enhanced SQL-Tutor* (n=39)
[+ vs no ITS ]
• Log files, assessment scores, questionnaire.

29. Results
1. Does providing M&M
feedback lead to any
measurable learning gains?
1. Does providing M&M
feedback lead to any
measurable gains in learner
focus?
2. Do the learners perceive any
benefit from using an ITS to
aid their learning?
3. What are the effects of
including M&M feedback
based on the learner’s past
states and experiences?
11 Yes study group move
slightly up class rankings;
both groups solved same
number of problems, but
Control group took more
attempts. Both groups better
than no Tutor
11 Control group bit more
persistent on unsolved
problems study group
made bit better use of the
system
11 Yes for both groups, study
group slightly more positive
11 Some positive trends
study group students
typically not follow up
references to past similar