Using technology to provide feedback on learning strategies

EksoBionicsﬂickr.com
Using technology
to provide
feedback on
learning strategies
Abelardo Pardo (@abelardopardo)
Faculty of Engineering and IT
slideshare.net/abelardo_pardo
Digital Learning Week 
University of South Australia 
16 March 2016

Abelardo Pardo Using technology to provide feedback on learning strategies 2
MichaelCoghlanﬂickr.com
About 
Feedback Feedback 
Higher Ed
Tech to scale 
feedback Examples

About 
Feedback

Abelardo Pardo Using technology to provide feedback on learning strategies
"information provided by an agent (teacher, peer,
book, parent, self, experience) regarding aspects
of one’s performance or understanding”
4
Hattie, J., & Timperley, H. (2007). The Power of Feedback. Review of Educational Research, 77(1), 81-112. doi:
10.3102/003465430298487
FarukAteşﬂickr.com

Innovations designed to strengthen the frequent
feedback that students receive about their
learning yield substantial learning gains
5
Black, P., & Wiliam, D. (1998). Assessment and Classroom Learning. Assessment in Education: Principles, Policy & Practice, 5(1),
7-74. doi:10.1080/0969595980050102
GarethChristopherﬂickr.com

If you could choose one…
• More than 500 meta-analyses
of student achievements
• 100 factors with potential
influence
• Feedback in top five
• (74 meta-analyses) Most
effective form: video, audio,
computer-assisted
instructional feedback, and/or
related goals
6
Hattie, J. A. (1999). Influences on student learning. Inaugural professorial address, University of Auckland, New Zealand

Feedback is inherent in and a
prime determiner of processes that
constitute self regulated learning
Galﬂickr.com
Butler, D. L., & Winne, P. H. (1995). Feedback and Self-Regulated Learning: A Theoretical Synthesis. Review of Educational
Research, 65(3), 245-281

Feedback models
1. Task Level (understanding,
performance)
2. Process Level (what to do to
understand, perform)
3. Self-regulation level (detecting
and directing effort)
4. Self level (personal evaluation
and affect)
8
Hattie, J., & Timperley, H. (2007). The Power of Feedback. Review of Educational Research, 77(1), 81-112. doi:
10.3102/003465430298487
Orkomedixﬂickr.com

What is the recipe?
1. encourage contact between student
and faculty
2. develops reciprocity and cooperation
among students
3. uses active learning techniques
4. gives prompt feedback
5. emphasizes time on task
6. communicates high expectations
7. respects diverse talents and ways of
learning
9
Chickering, A. W., & Gamson, Z. F. (1987). Seven principles for good practice in undergraduate education. American Association
for Higher Education Bulletin, 39, 3-7.  
Chickering, A. W., & Ehrmann, S. C. (1996). Implementing the seven principles: Technology as lever. American Association for
Higher Education Bulletin, 49, 3-6.

Hounsell, D. (2007). Toward more sustainable feedback to students. In D. Boud & N. Falchikov (Eds.), Rethinking Assessment in
Higher Education: Learning for the Longer Term. London and New York: Routledge.
Perceived as an administrative chore  
instead of a pedagogical necessity
MarcinWicharyﬂickr.com

Hounsell, D. (2008). The Trouble with Feedback. New Challenges, Emerging Strategies. Interchange(2).
Feedback has a Cinderella status
Bitslammerﬂickr.com

NicolasNovaﬂickr.com
Feedback is inherently problematic
12
Higgins, R., Hartley, P., & Skelton, A. (2010). Getting the Message Across: The problem of communicating assessment feedback.
Teaching in Higher Education, 6(2), 269-274. doi:10.1080/13562510120045230
Feedback as communication

1) Construct an understanding
2) Use it to regulate
13
Nicol, D. J., & Macfarlane‐Dick, D. (2006). Formative assessment and self‐regulated learning: a model and seven principles of good
feedback practice. Studies in Higher Education, 31(2), 199-218. doi:10.1080/03075070600572090
JamesButlerﬂickr.com

About 
Higher Ed

JoelSageﬂickr.com
“sustainability of feedback is under threat… pervasive
student concerns about the provision of feedback in
an era of mass education"
Hounsell, D. (2007). Toward more sustainable feedback to students. In D. Boud & N. Falchikov (Eds.), Rethinking Assessment in Higher
Education: Learning for the Longer Term. London and New York: Routledge.

AllenLaiﬂickr.com
Hounsell, D. (2007). Toward more sustainable feedback to students. In D. Boud & N. Falchikov (Eds.), Rethinking Assessment in Higher
Education: Learning for the Longer Term. London and New York: Routledge.
Downward spiral: Student disenchantment mounts
when feedback is uninformative and  
as a consequence, less feedback is provided
Solution: provide high-value feedback, rethink the role of
student, enhance relation between guidance and feedback

Feedback is delicate. It must be positive. High
fragility of conﬁdence felt by many students
17
Ferguson, P. (2011). Student perceptions of quality feedback in teacher education. Assessment & Evaluation in Higher Education,
36(1), 51-62. doi:10.1080/02602930903197883
Doranﬂickr.com

The feedback question gets systematically
lower values in student surveys
18
Krause, K.-L., Hartley, R., James, R., & McInnis, C. (2005). The First Year Experience in Australian Universities: Findings from a
decade of National Studies. University of Melbourne: Centre for the Study of Higher Education.
Eleafﬂickr.com

• Ambiguity or opacity (What do you
mean?)
• Negativity
19
Huxham, M. (2007). Fast and effective feedback: are model answers the answer? Assessment & Evaluation in Higher Education,
32(6), 601-611. doi:10.1080/02602930601116946
• Lateness
• Uncertainty about criteria and
contexts
Loonyhikerﬂickr.com

PostmodernSleazeﬂickr.com
Boud, D., & Molloy, E. (2013). Rethinking models of feedback for learning: the challenge of design. Assessment & Evaluation in
Higher Education, 38(6), 698-712. doi:10.1080/02602938.2012.691462
Conventional feedback: Mark 1
Impractical due to resources
Reserved for important concepts only
Could be extended by technology if 
careful selection is applied
Mark 2
From mechanistic to responsive
Curriculum-wide view
Foster self-regulation, feedback 
seeking skills, calibrated  
judgement, etc.

Active Learning Works

2 Hours
2 Hours
3 Hours
Pardo, A. & Mirriahi N. (in press). Design, Deployment and Evaluation of a Flipped Learning First Year Engineering Course. In C. Reidsema,
Kavanagh, R. Hadgraft & N. Smith (Eds.), Flipping the Classroom: Practice and Practices. Singapore: Springer.

Students are less likely to
engage in pre-class activities if
they are not interactive, do not
provide formative feedback,
and not coherently linked with
the face-to-face activities
O'Flaherty, J., & Phillips, C. (2015). The use of ﬂipped classrooms in higher education: A scoping review. The Internet and Higher
Education, 25, 85-95. doi:10.1016/j.iheduc.2015.02.002
DanKlimkeﬂickr.com

About 
Higher Ed
Tech to scale 
feedback

Instructive or process feedback is not common in
learning analytics
25
Gasevic, D., Dawson, S., & Siemens, G. (2015). Let's not forget: Learning analytics are about learning. TechTrends, 59(1), 64-75.
ChefCookeﬂickr.com

HungarianSnowflickr.com
Intelligence Amplification
Hierarchies of regulators to achieve amplification
We measure intelligence as the capacity to make
the appropriate selection
Ashby, W. R. (1957). An Introduction to Cybernetics. London: Chapman & Hall Ltd.

Baker, R. S. (2016). Stupid Tutoring Systems, Intelligent Humans. International Journal of Artificial Intelligence in Education. doi:
10.1007/s40593-016-0105-0
Rienties, B., Boroowa, A., Cross, S., Kubiak, C., Mayles, K., & Murphy, S. (2016). Analytics4Action Evaluation Framework: A
Review of Evidence-Based Learning Analytics Interventions at the Open University UK. Journal of Interactive Media in
Education, 2016(1). doi:10.5334/jime.394
Change in focus, from artificial intelligence to 
intelligence amplification
Tutors are informed about students struggling but it is up to
them to decide if/when to act

BrandonMartin-Andersonﬂickr.com
• Can data guide the provision of effective feedback
• Can this provision be done at scale?

JoshRepennekerFlickr.com
Can technology and
data be used to
change the economic
model of feedback,
achieve scale and
maintain effectiveness?

Simplistic View Emerging
“We are really shifting
the role of an educator
to someone who is
more of a data-
enabled detective”
Max Ventilla  
AltSchool founder 
(emphasis added)
30
Mead, Rebecca (2016). Learn Different. Silicon Valley disrupts education. The New Yorker, Annals of Technology, March 7 Issue

About 
Higher Ed
Tech to scale 
feedback Examples

• First year engineering
course
• Computer Systems
• Flipped Blended
Learning
• ~300 students
• Preparation activities
with formative feedback
and 
summative assessment
• 42% satisﬁed with
feedback (meh)

2 Hours
2 Hours
3 Hours
Pardo, A. & Mirriahi N. (in press). Design, Deployment and Evaluation of a Flipped Learning First Year Engineering Course. In C. Reidsema,
Kavanagh, R. Hadgraft & N. Smith (Eds.), Flipping the Classroom: Practice and Practices. Singapore: Springer.
2014 Edition

1. realtime feedback
2. convey engagement
3. compare with rest of
cohort
4. reset weekly
5. one click away from notes
Approach 1 (2014)
SeanDreilingerﬂickr.com

Tracking

Technology
Technology
Week N

Weekly real time feedback

No statistically signiﬁcant difference in the rating of feedback (2013 edition,
M=3.25, SD=0.97; 2014 edition, M=3.35, SD=1.03); t(389.78) = -0.97, p <0.17

1. encourage contact
between student and
faculty (forum)
2. uses active learning
techniques
3. gives prompt feedback
4. emphasizes time on task
5. kind, specific, helpful
Approach 2 (2015)Ben Manson flickr.com
BenMansonflickr.com

Week N
Technology
Human

Pardo, A., Mirriahi, N., Martínez-Maldonado, R., Jovanović, J., Dawson, S., & Gasevic, D. (2016). Generating Actionable Predictive
Models of Academic Performance. Paper presented at the International Conference on Learning Analytics and Knowledge,
Edinburgh.
1. Group students
2. Each group has an
estimated midterm score
3. Groups are determined by
conditions in engagement
measures
4. Can I act on this information?
Student Clustering based on Engagement

You should take a more
careful look at how symbols
are encoded in the video.
Would you be able to encode/
decode UAL symbols without
looking at the video?
Good initial work. However,
did you understand the trick
to handle encoding with a
variable number of bits?
Would you be able to provide
an example?
Good work. Would you be
able to come up with your
own machine language and
your encoding scheme?
Remember that it has to be
unambiguous.
Thorough work with the task
about machine language
encoding. Give it a quick
review before the midterm.
Q1 Q2 Q3 Q4
Instructor

Algorithm

Automatic
Email

1. Data collected weekly
2. Email sent at end of week
3. 4 weeks before the midterm
nateOneﬂickr.com

Signiﬁcant difference in the rating of feedback (2014 edition, M=3.35, SD=1.03;
2015 edition, M=3.82, SD=0.90); t(389.78) = -4.88, p <10-6 
Effect size (Cohen’s d) = 0.49. Medium positive effect

Signiﬁcant difference in the midterm score (2014 edition, M=12.80, SD=4.79;
2015 edition, M=13.83, SD=4.89); t(684.5) = -2.86, p < 0.002 
Effect size (Cohen’s d) = 0.21. Small positive effect

MatthewandHeaderﬂickr.com
Feedback: high impact/risk
Higher Ed is in dire need for feedback revamp
Hybrid methods: human intelligence ampliﬁcation
Data supported approaches allow effective scaling

Using technology to provide feedback on learning strategies

Recommandé

Recommandé

Contenu connexe

Plus de Abelardo Pardo

Plus de Abelardo Pardo (20)

Dernier

Dernier (20)

Using technology to provide feedback on learning strategies