Tuesday 22 November 2022

1. An overview of current QEI
team-led projects at the OU
INSTITUTIONAL RESEARCH FROM IET
Simon Cross
Senior Lecturer
Associate Director (QEI)

2. I. Introduction
2
● Quality Enhancement and Innovation (QEI) team based in Institute of
Educational Technology (IET) at the OU.
● One of three operational areas in IET (others: Research, Curriculum).
● Supports university in strategic use, evaluation and development of
educational technologies in teaching and learning.
● Contributions from a core of 7 academics and 4 academic-related staff
plus other academic staff.
● Commitment to external dissemination and publication.
● Areas of interest: 1. Assessment and feedback
2. Learning Analytics
3. The Changing Distance Learner
4. Accessibility

3. Working across the OU
3

4. II. Analytics
4
IET Qualification Pathways Model
Flexible multi-model database
including graph technology to enable
analysis of 100,000’s learner
pathways through qualifications
(overcoming limitations of existing
institutional approaches) and dealing
with problem of fragmented cohorts.
● Data includes all UG and PG
study since 2012.
● Insight into study complexity,
retention, progression, pass-
rates, study gaps, study intensity
and experience
● Supported qualification re-
design in faculties by providing
snapshot and cohort following.
Text Analytics
Automated text analytics of the
open comment responses given by
students in their end of module
survey.
• High performing students talk
more about content and
feedback whereas low talk about
help, problems and guidance.
• Difference with ethnicity.
Early Alert Indicators Dashboard
Predictive analytics using OU
Analyse (OUA) and Student
Probabilities (SPM) predict risk of
students failing their studies and
submission of assignments.
• Uses traffic light system to flag
risk to tutors (Associate
Lecturers).
• Positive impact on retention and
continues to be rolled-out
0
100
200

5. I. Assessment design
5
Student Experience of Feedback,
Assessment and Revision (SEFAR)
Student survey (n=572) comprising
over 100 items Including course
assessment, feedback, revision,
exams/EMAs, assessment networks,
assessment literacies.
● Insight into assessment as a
learning activity and a social
practice
● Identified differences in
experience of Black, Asian and
White students
● Significant variation in views
and interaction between
dimensions
● Support for greater adoption of
a dialogic feedback approach.
Online Remote Exams Experience
Survey
Two pan-university student surveys
(n=1193) and interviews (n=30)
covering around 50 modules (2021
and 2022) to support OU ambition to
move exams online
• 74%/79% distance learners would
prefer online exams to F2F exams
• More than half agreed with
suggested restrictions
• Great actionable insight
• Little difference between online
and F2F revision and exam
experience although indications of
difference by age, anxiety and
exam ‘space’
Digital Badges for Teacher
Professional Development in India
Field trial with 220 schools in
Assam working with TISS and
state government.
• Framework for scaling digital
badges for TPD.
• Over 90% of participants
positive about approach
• Factor analysis helps clarify
range of teacher motivations
• Builds on pilot of 360-degree
video for TPD and TESS-India
project

6. III. Designing for the changing learner
6
Visual Module Maps
Review of how modules have used
visual maps to convey module
teaching and interviews with
students about preferences.
● Report offering design advice
Learning Design of XR activities
Strategic project led by Fridolin
Wild to create a mobile-based
design environment for production
of XR learning activities.
• QEI supporting design and
development of alternative XR-
enabled learning activities for
pilot modules.
Skills For Prosperity Kenya (FCDO)
Working with majority Kenyan HEI’s
to support professional
development in digital and online
teaching (inc. use JISC framework).
Learning (Educational) Gain
Staff and Student interviews and
data analysis to understand what
they understand by concept of
Learning Gain.
Optionality and Student Choice
Reviewing how OU modules offer
students optionality in assessment,
learning activity choice (designed
paths) or differentiated or targeted
materials.
Enrichment Programme for FE
Colleges (Wales)
Leading evaluation of programme.

7. IV. Accessibility
7
Assistants to the disclosure and
management of information about
needs and support (ADMINS)
A virtual assistant to support students
disclosing disabilities was developed
through a grant from Microsoft.
Students talk to ‘Taylor’ through
either text or speech and AI models
classify what the student says about
their disabilities and needs.
● Trial of 134 students, 65%
preferred Taylor to the existing
form-based process.
● Embedded in OU systems on 2
November.
● Foundation of a new OU Societal
Challenges project
Securing Greater Accessibility
Network (SeGA)
Co-ordination of the university’s
SeGA network plus support and
advise for making learning and
assessment adjustments.
Accessible by Design
Seeking to make efficiency and
design improvements by
identifying opportunities to
consider accessibility in the
design phase (rather than having
to make adjustments later. It will
propose an operationally-
relevant definition of accessible
by design with examples of good
practice and integrate with OU’s
Foundations (innovating design)
Programme.

8. QEI Team
8
Simon Cross
Thea Herodotou
Tim Coughlan
Chris Edwards
Thomas Ullmann
Maria Aristeidou
Fereshte Goshtasbpour
Ben Tidman
Andrew Brasher
Emily Coughlan
Felipe Tessarolo
Working with many other colleagues
in IET

9. Simon Cross
Simon.j.cross@open.ac.uk
Senior Lecturer
Associate Director (QEI)

10. 10
Assessment
Cross, S., Wolfenden, F., Charania, A., Adinolfi, L., Sen, S. & Sarkar, D. (2022). Supporting practice-based teacher professional learning and
assessment at scale in the Global South. In: INTED2022 Proceedings: 16th International Technology, Education and Development Conference,
Valencia, Spain. ORO: http://oro.open.ac.uk/82525/
Edwards, C., Whitelock, D., Cross, S. & Wild, F. (2022). Developing automated ways to give essay writing feedback to students. In: INTED2022
Proceedings: 16th International Technology, Education and Development Conference, IATED Academy, Valencia, Spain, pp. 5446–5450. ORO:
http://oro.open.ac.uk/82372/
Cross, S., Wolfenden, F. & Adinolfi, L. (2022). Taking in the complete picture: framing the use of 360-degree video for teacher education practice
and research. Teaching and Teacher Education, 111, article no. 103597.
Cross, S. & Brasher, A. (2022). How well does OU assessment practice take into account students’ personal learning objectives and goals:
insight for action from the SEFAR survey.
Edwards, C., Whitelock, D., Cross, S., & Wild, F. (2022). Towards autonomous, confident academic writing: supporting students in developing
their writing through a stand alone online tool. AHE2022.
Cross, S., Charania, A., Wolfenden, F., Adinolfi, L., Sen, S. & Sarkar, D. (2022). Taking forward TPD@Scale in the Global South: what are we
learning from open digital badge implementations in India? Available via ORO: https://oro.open.ac.uk/82525/
Edwards, C. & Whitelock, D. (2022). Was our pilot study successful? An approach when data are sparse? In: EDEN 2022 Proceedings (in press).
Cross, S., & Brasher, A., (2021). Differences in assessment experience between Black, Asian and Minority Ethnic (BAME) students and White
students: An initial review using IET’s SEFAR2 dataset. Available via Scholarship Exchange.
Whitelock, D., Edwards, C. & Okada, A. (2020). Can e-Authentication raise the confidence of both students and teachers in Qualifications
Granted through the e-Assessment Process? Journal of Learning for Development - JL4D, 7(1) pp. 46–60.

11. 11
Analytics
Boroowa, A. & Herodotou, C. (2022). Learning analytics in open and distance higher education: The case of the Open University UK. In (Eds) Paul
Prinsloo, Sharon Slade, Mahammad Khalil: Learning Analytics in Open and Distributed Learning: Potential and Challenges. Springer Nature; Rienties, B., &
Herodotou, C. (2022).
Ullmann, T. & Rienties, B. (2021). ‘Using Text Analytics to understand open-ended student comments at scale: Insights from four case studies’, in
Shah, M., Richardson, J.T.E., Pabel, A., and Oliver, B. (eds) Assessing and Enhancing Student Experience in Higher Education. Cham: Springer International
Publishing, pp. 211–233. doi:10.1007/978-3-030-80889-1_9.
Ullmann, T.D. (2021a). Exploring differences in the student experience of ethnic minorities: insights from text analytics of student comments. QEI
report. Milton Keynes, UK: Institute of Educational Technology, The Open University. Available on the Scholarship Exchange.
Ullmann, T.D. (2021b). Supporting student academic skills development - an evaluation of an English for Academic Purposes pilot: quantitative
evaluation. QEI report. Milton Keynes, UK: Institute of Educational Technology, The Open University. Available on the Scholarship Exchange.
Ullmann, T.D., Hidalgo, R., Van Ameijde, J. & Murphy, Y. (2021). SEaM Text Analytics for the Analytics for Action programme: A first small-scale
exploration. QEI report. Milton Keynes, UK: Institute of Educational Technology, The Open University. Available on the Scholarship Exchange.
Herodotou, C., Maguire, C., Mcdowell, N. D., Hlosta, M., & Boroowa, A. (2021). The engagement of university teachers with predictive learning
analytics. Computers & Education, 104285.
Rets, I., Herodotou, C., Bayer, V., Hlosta, M., & Rienties, B. (2021). Exploring critical factors of the perceived usefulness of a learning analytics
dashboard for distance university students. International Journal of Educational Technology in Higher Education, 18, article 46.
Edwards, C. & Gaved, M. (2020a). The student study experience – analysing student study choices. In: 29th EDEN (European Distance Education
Network) Annual Conference 2020 - Human and Artificial Intelligence for the Society of the Future, 21-24 Jun 2020, Politehnica University, Timisoara,
Romania. ORO: http://oro.open.ac.uk/70779/.
Edwards, C. & Gaved, M. (2020b). Understanding student experience: a pathways model. In: L@S ‘20: Proceedings of the Seventh ACM Conference on
Learning @ Scale, 12-14 Aug 2020, Virtual event, USA, pp. 265–268. ORO: http://oro.open.ac.uk/70886/.

12. 12
Changing learner
Aristeidou, M., Cross, S. & Rossade, K. (2022). Online remote exams: Open University students’ views around the nuances and
variants of online exams.
Cross, S., Aristeidou, M., Rossade, K., Woods, C. & Brasher, A., (2022). The impact of online exams on the quality of distance learners’
exam and exam revision experience: perspectives from The Open University UK. At I:HE 2022, Athens, Greece, October 2022.
Brasher, A. (2022). Understanding student needs and potential for moving towards standardisation in Module Maps. Available on
the Scholarship Exchange.
Ismail, N., Aristeidou, M. & Cross, S. (2021). Interview insight into the impact of the Covid-19 pandemic on OU students: results
and analysis from thirty student interviews. Available soon on Scholarship Exchange.
Aristeidou, M. & Cross, S. (2021). The impact of Covid-19 disruption on the learning activities of OU students. Available on the
Scholarship Exchange.
Gaved, M., Brasher, A. & Edwards, C. (2019). Open University students’ perceptions of terms used in the National Student Survey
2019. Available on the Scholarship Exchange.
Herodotou, C., Sharples, M. & Scanlon, E. (2020). Can technology empower the public to think scientifically? The case of nQuire.
In: International Conference of the Learning Sciences (ICLS 2020), 19-23 Jun 2020, Nashville, TN, USA.
Cross, S., Sharples, M., Healing, G. & Ellis, J. (2019). Distance learners’ use of Handheld Technologies: Mobile Learning Activity,
changing study habits, and the ‘place’ of anywhere learning. International Review of Research in Open and Distributed Learning,
20(2) pp. 223–241.
Cross, S. (2019). Using VLEs to offer higher education students choice and differentiation in learning activities: Micro-pathway
Learning Design implementation and opportunities. In: European Conference on e-Learning, ACPI.

13. 13
Accessibility
Coughlan, T. & Lister, K. (2022). Creating stories of learning, for learning: Exploring the potential of elf-Narrative in
education with ‘Our Journey’. In 14th ACM conference on Creativity and Cognition (pp. 526-531).
Measuring Accessibility Practices and Perceptions at the Open University (MAPP), Findings of the 2021 staff survey.
Lister, K., Coughlan, T., Kenny, I., Tudor, R. & Iniesto, F. (2021). Taylor, The Disability Disclosure Virtual Assistant: A case
study of participatory research with disabled students. Education Sciences.
Coughlan, T., Lister, K. & Lucassen, M. (2021). Representing the Unseen with “Our Journey”: a platform to capture
affective experiences and support emotional awareness in University-Level study. Journal of Formative Design in
Learning, 5 (1) pp. 39-52.
Seale, J., Colwell, C., Coughlan, T., Heiman, T., Kaspi-Tsahor, D. & Olenik-Shemesh, D. (2021). “Dreaming in colour’:
disabled higher education students’ perspectives on improving design practices that would enable them to benefit from
their use of technologies”. Education and Information Technologies, 26 (2) pp. 1687-1719.
Lister, K., Coughlan, T., Iniesto, F., Freear, N. & Devine, P. (2020). Accessible Conversational User Interfaces:
Considerations for design. In: Proceedings of Web for All 2020.
Rets, I., Coughlan, T., Stickler, U. & Astruc, L. (2020). Accessibility of Open Educational Resources: how well are they
suited for English learners?. Open Learning: The Journal of Open, Distance and e-Learning.