2018-06-25 Paper presentation at International ICLS 2018 in London on Quality of MOOCs towards QRF by Christian M. Stracke and Esther Tan from OUNL

1. The Quality of Open Online
Learning and Education:
Towards a Quality Reference
Framework for MOOCs
Christian M. Stracke & Esther Tan

2. Open University of the Netherlands
Global cooperation: ECNU & KNOU
Global initiative ICORE for OR & OE
International WLS / LINQ Conference
eLC European Institute
ICDE Chair in OER
Dr. Christian M. Stracke:
Open Learning & Education, Innovations,
Policies, Quality & Competences, Impact

3. Open University of the Netherlands
The Quality of MOOCs
Let’s Learn to Learn
Seamless Learning
Dr. Esther Tan
Technology-Enhanced Learning,
Innovations in & out Classroom

4. Presentation Outline
1. The Quality of MOOCs: What is the problem?
2. A Conceptual Framework towards MOOC QRF:
Theoretical & Methodological Approach
3. Findings from MOOC surveys:
Learners, Designers & Facilitators
4. Findings from MOOC OEQ:
Learners, Designers & Facilitators

5. Any discussion on the quality of MOOCs should consider the goals
of both, the MOOC learner and the provider. (Hayes, 2015)
Critical questions to be addressed:
 Who are the MOOC users, and why?
 What makes a good MOOC from the learners’ experiences
with MOOCs?
 What are the best design principles and best practices
as indicators of quality?
 Are these quality indicators also MOOC domain specific and/ or
MOOC type specific?
Problem Statement

6. A Research Framework for MOOQ
“When one designs any course, one has to have some learner
cohort in mind.” (Macleod et al., 2015, p. 9)
Main research goals:
1. Establish a research framework for the subsequent analysis of
MOOC design patterns and best practices
2. Develop a Quality Reference Framework (QRF) for MOOCs
Theoretical Framework

7. Global MOOC
Quality Survey

8. Quality Reference Framework with
criteria & checklist for MOOC design
Our main goal is the collaboration with all
to improve Open Education & MOOCs
www.MOOC-quality.eu

9.  Subjects of Investigation:
Learners, Designers, Facilitators & Providers
 Mixed Methods Approach:
Quantitative & qualitative data
1. Global MOOC Quality Survey,
2. Open-Ended questions (OEQ) &
3. Semi-structured interviews
Methodological Framework

10. Instruments of Measure
MOOC
Learners
MOOC
Designers
MOOC
Facilitators
MOOC
Providers
TOTAL
Global
MOOC
Quality
Survey
166
(69 qns.)
68
(89 qns.)
33
(58 qns.)
- 267
Open-
ended
Questions
118
(4 qns.)
42
(4 qns.)
27
(4 qns.)
- 185
Semi-
structured
interviews
-
15
(15 qns.)
15
(10 qns.)
15
(13 qns.)
45

11. MOOC Survey Constructs
Constructs Learners Designers Facilitators
Experience with MOOC X X X
Learning Objectives X X X
Duration and Structure X X
Learning Resources X X X
Accessibility and Inclusion X X
Learning Progress X
Learning Environment X X
Learning Assessment X X X
Learning Certification X X
Design Process X
Pedagogical Decisions X
Learning Support
 Feedback & Facilitation
 Interaction & Collaboration
X X X

12. Global MOOC
Quality Survey
(GMQS)

13. Global MOOC Quality Survey

14. Demographic Profile
Age range of all survey participants (learners, designers & facilitators) by gender

15. Demographic Profile
Educational level of all survey participants (learners, designers & facilitators) by gender

16. n VB B N G VG
Learning
experience
166 4 4 13 75 70
Learning Experience (Learners)

17. n VB B N G VG
Design
experience
68 1 2 13 33 19
Design Experience (Designers)

18. Interaction from Learners‘ Perspective
n N/A SD D N A SA
LF 146 20 5 13 48 37 23
LL 146 15 3 17 34 51 26
LR 146 9 2 8 25 61 41
GG 146 37 4 15 50 24 16
Note:
LF: Interaction between learners and facilitators
LL: Interaction among learners
LR: Interaction between learners and learning resources
GG: Interaction among teams and groups

19. n R2 M2 p
LF by
learners
125 .094 9.382 .000***
LL by
learners
130 .101 10.818 .000***
LR by
learners
136 .112 12.286 .000***
GG by
learners
108 .045 4.131 .026*
Bivariate Correlations between LLR4 and LLE4

20. Interaction from Designers‘ Perspective
n N/A SD D N A SA
LF 52 2 1 5 11 24 9
LL 52 1 1 3 11 19 17
LR 52 3 1 0 4 22 22
GG 52 8 2 10 14 13 5
Note:
LF: Interaction between learners and facilitators
LL: Interaction among learners
LR: Interaction between learners and learning resources
GG: Interaction among teams and groups

21. n R2 M2 p
LF by
designers
49 .003 0.109 .703
LL by
designers
50 .043 1.595 .143
LR by
designers
48 .046 1.537 .138
GG by
designers
43 .001 0.038 .821
Bivariate Correlations between DLR4 and DDE4

22. Open-ended
Questions
(OEQ)

23. MOOC Open-ended Questions (OEQ)
MOOC Learners
Q1. What were the three main strengths of the MOOC?
Q2. What were the three main weaknesses of the MOOC?
Q3. What was missing in the MOOC?
Q4. Looking ahead into the development of this type of learning
experiences, what could be improved in future MOOCs?
MOOC Designers
Q1. Which were the main design decisions that you made during
the development of the MOOC that later proved to be successful?
Q2. Which were the three biggest difficulties that you faced when
designing the MOOC?
Q3. Which design decisions did not pay off as you expected?
Q4. Looking ahead into the development of this type of learning
design experiences, what methods and tools could be helpful to
improve the design of future MOOCs?

24. No of respondents across the six domains
Domain Learners Designers
Social Sciences, Humanities and Law 24 9
Education and Lifelong Learning 19 12
Computing and Informatics 18 5
Science, Math and Engineering 16 5
Nature, Environment and Health 21 6
Business, Management and Economics 20 5
Total 118 42

25. Open-ended Questions (MOOC Learners)
Q1. What were the three main strengths of the MOOC? 217 units of analysis
0
10
20
30
40
50
60
70
80
90
Curriculum
Design &
Delivery
Instructional
Design &
Technology
Assessment &
Evaluation
Facilitation &
Feedback
Interaction &
Collaboration
NumberofComments
Social Sciences, Humanities and Law
Education and Lifelong Learning
Computing and Informatics
Science, Maths and Engineering
Nature, Environment and Health
Business, Management and
Economics

26. Open-ended Questions (MOOC Learners)
Q2. What were the three main weaknesses of the MOOC? 123 units of analysis
0
5
10
15
20
25
30
35
Curriculum Design Instructional
Design &
Technology
Assessment &
Evaluation
Faciliation &
Feedback
Interaction &
Collaboration
NumberofComments
Social Sciences, Humanities and Law
Education and Lifelong Learning
Computing and Informatics
Science, Maths and Engineering
Nature, Environment and Health
Business, Management and Economics

27. Open-ended Questions (MOOC Learners)
Three main strengths and weaknesses of the MOOC
Strengths Weaknesses
Curriculum Design & Delivery
• Good choice & quality of content
• Good teachers, presenters & tutors
• LO aligns with content
• Weak choice & quality of content
• Short duration
Instructional Design & Technology
• Good integration of IT & media
• Support self-regulation & individual
learning paths
• Poor use of IT technological tools
• Resources lack variety & quality
Interaction & Collaboration
• Encourage local group discussion &
activities
• Foster interaction with field experts
• Lack interaction: learner-tutor
• No support for community building

28. Open-ended Questions (MOOC Designers)
Q1. Which were the main design decisions that you made during the
development of the MOOC that later proved to be successful?
59 units of analysis
1
3
3
42
1
1
3
3
6
5
4
0
4
8
12
16
20
24
Assessibility&
Inclusion
Assessment &
Evaluation
Interaction &
Collaboration
Certification&
Accreditation
Curriculum
Design& Delivery
Expertise &
Manpower
Feedback &
Facilitation
Instructional
Design&
Technology
No.ofComments
Social Sciences, Humanities and Law
Education and Lifelong Learning
Computer and Informatics
Science, Maths and Engineering
Nature, Environment and Health
Business, Management and Economics

29. Open-ended Questions (MOOC Designers)
Q2. Which were the three biggest difficulties that you faced when designing the
MOOC? 74 units of analysis
3 3
1
3
3
2
8
1
2
2
1
1
1
2
3
5
4
4
2
5
3
0
4
8
12
16
20
24
Assessment &
Evaluation
Curriculum
Design&
Delivery
Expertise &
Manpower
Feedback &
Facilitation
Institutional
Support &
Funding
Instructional
Design&
Technology
Interaction &
Collaboration
Open Access,
Copyrights &
Licensing
No.ofComments
Social Sciences, Humanities and Law
Education and Lifelong Learning
Computer and Informatics
Science, Maths and Engineering
Nature, Environment and Health
Business, Management and
Economics

30. Open-ended Questions (MOOC Designers)
Successful Decisions Biggest Challenges
Curriculum Design & Delivery
• Content delivery format
• Content structure & LOs
• Weak choice & quality of content
• Short duration
Instructional Design & Technology
• Choice of learning activities
• Integration of IT & media
• Platform, software & production
decisions
Interaction & Collaboration Expertise & Manpower
• Creating community of learners
• Foster interaction between learner
& tutor/ facilitator
• Gap in content & instructional
design knowledge
• Coordination & collaboration, e.g.,
different experts & teaching staff
Three main successful decisions and three biggest challenges

31. Overview of QRF (work-in-progress)
Dimension 1: Phases Analysis, Design, Implementation, Realization, Evaluation
Dimension 2: Perspectives Pedagogical, Technological, and Strategic
Dimension 3: Roles Designer, Facilitator, and Provider

32. Kind Invitation: MOOQ Conference
on 13th of July 2018 in Athens, Greece
The Quality Reference Framework (QRF) to be
discussed & improved in interactive sessions:
Registration is open and free!
www.conference.MOOC-quality.eu

33. First References for GMQS
Stracke, C. M., & Tan, E. (2018). The Quality of Open Online Learning and
Education: Towards a Quality Reference Framework for MOOCs.
Proceedings 13th International Conference on the Learning Sciences (ICLS).
(accepted, in print)
Stracke, C. M. et al. (2018). Gap between MOOC designers' and MOOC learners‘
perspectives on interaction and experiences in MOOCs: Findings from the
Global MOOC Quality Survey. Proceedings 18th IEEE International
Conference on Advanced Learning Technologies (ICALT).
(accepted, in print)
Stracke, C. M. et al. (2017). The Quality of Open Online Education: Towards
a Reference Framework for MOOCs. In Proceedings of 2017 IEEE Global
Engineering Education Conference (EDUCON) (pp. 1712-1715).
IEEE Xplore. DOI: 10.1109/EDUCON.2017.7943080
To be continued …

34. christian.stracke@ou.nl
esther.tan@ou.nl
@ChrMStracke
@Taneste
www.opening-up.education
Let us cooperate!

35. Thank you!
Your questions?
To be continued…

36. Open CC License for
sharing & re-using slides
This work is free to share under the creative commons licence:
"Attribution – Noncommercial – Share Alike 3.0"
You can copy, distribute and transmit the work under the following conditions:
1. Attribution –
2. Noncommercial –
3. Share Alike
Licence: Attribution-Noncommercial-Share Alike
Some rights reserved, see: http://creativecommons.org/licenses/by-nc-sa/3.0/