2018-08-23 Paper Presentation at EARLI SIG 6-7 Conference in Bonn on The Quality Reference Framework for MOOCs by Christian M. Stracke and Esther Tan from OUNL

1. Towards a Quality
Reference Framework
for MOOCs
@ EARLI SIG 7 in Bonn, 2018-08-23
by Christian M. Stracke & Esther Tan
Open University of the Netherlands

2. Open CC License for
sharing & re-using slides
This work is free to share under the creative commons licence:
"Attribution – Noncommercial – Share Alike 3.0"
You can copy, distribute and transmit the work under the following conditions:
1. Attribution –
2. Noncommercial –
3. Share Alike
Licence: Attribution-Noncommercial-Share Alike
Some rights reserved, see: http://creativecommons.org/licenses/by-nc-sa/3.0/

3. Open University of the Netherlands
Global cooperation: ECNU & KNOU
Global initiative ICORE for OR & OE
International WLS / LINQ Conference
eLC European Institute
ICDE Chair in OER
Dr. Christian M. Stracke:
Open Learning & Education, Innovations,
Policies, Quality & Competences, Impact

4. Open University of the Netherlands
The Quality of MOOCs
Let’s Learn to Learn
Seamless Learning
Dr. Esther Tan
Technology-Enhanced Learning,
Innovations in & out Classroom

5. MOOQ and the
Quality of MOOCs:
What are we
talking about?

6. Two core factors:
1. Globalisation &
2. Worldwide Internet
The Digital Age
Photo: https://ec.europa.eu/digital-agenda/en/digital-life/education

7. Global Competitions and societal changes
Close the gaps
& open new
opportunities
Challenges:
Learn to Learn
Internationalization
Figure: http://www.weforum.org/issues/global-competitiveness

8. We need
Change
in Education!

9. To improve
Quality
in Education!

10. Open Education (OE):
Innovations for changing
& opening up education
to improve the quality

11. Goal 4:
Inclusive and quality education
Sustainable dev goals
www.sustainabledevelopmentgoals.org

12. Goal 4:
Inclusive and quality education
Sustainable dev goals

13. What is
Quality
Education?

14. What is
Quality?

15. A simple
experiment ...

16. A simple
experiment ...

17. Quality
cannot be defined
... except through adaptation
to your situation and context!

18. Quality Standard with Reference Process Model:
Standard:ISO/IEC 40180
Communication
concept
Needs Analysis
Conception /
Design
Development /
Production
Implementation
Framework
Analysis
Learning Process
/ Realization
Initiation
Media realization
Technical
realization
Design realization
Content realizationAnalysis of the
external context
Testing of learning
resources
Learning objectives
Stakeholder
identification
Concept for
contents
Definition of
objectives
Demand analysis
Time and budget
planning
Environment
analysis
Activities
Organization of use
Activation of
learning resources
Adaptation of
learning resourcesDidactical
concept/ methods
Organizational
concept
Roles and activities
Technical concept
Concept for media
and interaction
design
Media Concept
Technical
infrastructure
Review of
competencies levels
Concept for tests
and evaluation
Analysis of staff
resources
Evaluation /
Optimization
Planning
Realization
Analysis
Optimization /
improvement
Analysis of the
institutional and
organizational
context
Analysis of target
groups
Concept for
maintenance
Administration
Maintenance

19. Example of required adaptation:
Standard:ISO/IEC 40180
Communication
concept
Needs Analysis
Conception /
Design
Development /
Production
Implementation
Framework
Analysis
Learning Process
/ Realization
Initiation
Media realization
Technical realization
Design realization
Content realizationAnalysis of the
external context
Testing of learning
resources
Learning objectives
Stakeholder
identification
Concept for contents
Definition of
objectives
Demand analysis
Time and budget
planning
Environment
analysis
Activities
Organization of use
Activation of
learning resources
Adaptation of
learning resources
Didactical
concept/ methods
Organizational
concept
Roles and activities
Technical concept
Concept for media
and interaction design
Media Concept
Technical
infrastructure
Review of
competencies levels
Concept for tests
and evaluation
Analysis of staff
resources
Evaluation /
Optimization
Planning
Realization
Analysis
Optimization /
improvement
Analysis of the
institutional and
organizational
context
Analysis of target
groups
Concept for
maintenance
Administration
Maintenance

20. MOOQ and the
Quality of MOOCs:
What are we
talking about?

21. What are
MOOCs?

22. Source: https://www.flickr.com/photos/catspyjamasnz/
Source: https://www.flickr.com/photos/gforsythe/
What are MOOCs?
Massive (?) Open (?)
Online (?) Courses (?)

23. Source: https://www.flickr.com/photos/-ed/
And the Quality
of MOOCs?
Many issues
are leading
to high
drop-out
rates (>90%)

24. What is the Future
of MOOCs?
MOOCs are dead!
(Stracke, 2017 - and
long live the MOOCs!)
MOOCs are the
educational
buzzword of 2012
(Daniel, 2012)

25. What is the Future
of MOOCs?
The fact is:
Number of MOOCs, MOOC learners
and MOOC providers are growing
until today
(Class Central,
2018)

26. What is
?

27. MOOQ for the quality of MOOCs:
“We will make MOOCs better”
Quality Reference Framework with
indicators for design & comparison
www.MOOC-quality.eu
Frameworks: MOOQ

28. Coordinator & QRF
development lead
ENS
UAb
NQIS
HOU
OUNL
OUNL
HOU
NQIS
UAb
ENS
MOOC development
& QRF contributor
Standardization &
QRF contributor
Research & QRF
contributor
Research & QRF
contributor

29. Mixed Method
Research with many
sources & externals
RESEARCH
QRF DESIGN
PILOTING
STANDARD
STEP 4
STEP 3
2 STEP
1 STEP
Development and
refinement of QRF
with practitioners &
experts worldwide
Development of two
MOOCs for testing
the QRF
Proposal for QRF as
European CEN &
international ISO
standard

30. Quality Reference Framework with
criteria & checklist for MOOC design
Our main goal is the collaboration with all
to improve Open Education & MOOCs
www.MOOC-quality.eu

31.  Subjects of Investigation:
Learners, Designers, Facilitators & Providers
 Mixed Methods Approach:
Quantitative & qualitative data
1. Global MOOC Quality Survey,
2. Open-Ended questions (OEQ) &
3. Semi-structured interviews
Methodological Framework

32. Instruments of Measure
MOOC
Learners
MOOC
Designers
MOOC
Facilitators
MOOC
Providers
TOTAL
Global
MOOC
Quality
Survey
166
(69 qns.)
68
(89 qns.)
33
(58 qns.)
- 267
Open-
ended
Questions
118
(4 qns.)
42
(4 qns.)
27
(4 qns.)
- 185
Semi-
structured
interviews
-
12
(15 qns.)
12
(10 qns.)
12
(13 qns.)
36

33. MOOC Survey Constructs
Constructs Learners Designers Facilitators
Experience with MOOC X X X
Learning Objectives X X X
Duration and Structure X X
Learning Resources X X X
Accessibility and Inclusion X X
Learning Progress X
Learning Environment X X
Learning Assessment X X X
Learning Certification X X
Design Process X
Pedagogical Decisions X
Learning Support
 Feedback & Facilitation
 Interaction & Collaboration
X X X

34. Global MOOC
Quality Survey
(GMQS)

35. Global MOOC Quality Survey

36. Demographic Profile
Age range of all survey participants (learners, designers & facilitators) by gender

37. Demographic Profile
Educational level of all survey participants (learners, designers & facilitators) by gender

38. n VB B N G VG
Learning
experience
166 4 4 13 75 70
Learning Experience (Learners)

39. n VB B N G VG
Design
experience
68 1 2 13 33 19
Design Experience (Designers)

40. Interaction from Learners‘ Perspective
n N/A SD D N A SA
LF 146 20 5 13 48 37 23
LL 146 15 3 17 34 51 26
LR 146 9 2 8 25 61 41
GG 146 37 4 15 50 24 16
Note:
LF: Interaction between learners and facilitators
LL: Interaction among learners
LR: Interaction between learners and learning resources
GG: Interaction among teams and groups

41. n R2 M2 p
LF by
learners
125 .094 9.382 .000***
LL by
learners
130 .101 10.818 .000***
LR by
learners
136 .112 12.286 .000***
GG by
learners
108 .045 4.131 .026*
Bivariate Correlations between LLR4 and LLE4

42. Interaction from Designers‘ Perspective
n N/A SD D N A SA
LF 52 2 1 5 11 24 9
LL 52 1 1 3 11 19 17
LR 52 3 1 0 4 22 22
GG 52 8 2 10 14 13 5
Note:
LF: Interaction between learners and facilitators
LL: Interaction among learners
LR: Interaction between learners and learning resources
GG: Interaction among teams and groups

43. n R2 M2 p
LF by
designers
49 .003 0.109 .703
LL by
designers
50 .043 1.595 .143
LR by
designers
48 .046 1.537 .138
GG by
designers
43 .001 0.038 .821
Bivariate Correlations between DLR4 and DDE4

44. Open-ended
Questions
(Learners)

45. No of respondents across the six domains
Domain Learners Designers Facilitators
Social Sciences, Humanities &
Law
24 9 2
Education & Lifelong Learning 19 12 18
Computing & Informatics 18 5 2
Science, Math & Engineering 16 5 1
Nature, Environment & Health 21 6 1
Business, Management &
Economics
20 5 3
Total 118 42 27

46. MOOC Open-ended Questions (OEQ)
MOOC Learners
Q1. What were the three main strengths of the MOOC?
Q2. What were the three main weaknesses of the MOOC?
Q3. What was missing in the MOOC?
Q4. Looking ahead into the development of this type of
learning experiences, what could be improved in
future MOOCs?

47. Open-ended Questions (MOOC Learners)
Q1. What were the three main strengths of the MOOC?
0
10
20
30
40
50
60
70
80
90
Curriculum
Design &
Delivery
Instructional
Design &
Technology
Assessment &
Evaluation
Facilitation &
Feedback
Interaction &
Collaboration
NumberofComments
Social Sciences, Humanities
and Law
Education and Lifelong
Learning
Computing and Informatics
Science, Maths and
Engineering
Nature, Environment and
Health
Business, Management and
Economics

48. Open-ended Questions (MOOC Learners)
Q2. What were the three main weaknesses of the MOOC?
0
5
10
15
20
25
30
35
Curriculum Design Instructional
Design &
Technology
Assessment &
Evaluation
Faciliation &
Feedback
Interaction &
Collaboration
NumberofComments
Social Sciences, Humanities and Law
Education and Lifelong Learning
Computing and Informatics
Science, Maths and Engineering
Nature, Environment and Health
Business, Management and Economics

49. Open-ended Questions (MOOC Learners)
Three main strengths and weaknesses of the MOOC
Strengths Weaknesses
Curriculum Design & Delivery
• Good choice & quality of content
• Good teachers, presenters & tutors
• LO aligns with content
• Weak choice & quality of content
• Short duration
Instructional Design & Technology
• Good integration of IT & media
• Support self-regulation & individual
learning paths
• Poor use of IT technological tools
• Resources lack variety & quality
Interaction & Collaboration
• Encourage local group discussion &
activities
• Foster interaction with field experts
• Lack interaction: learner-tutor
• No support for community building

50. Open-ended
Questions
(Designers)

51. MOOC Open-ended Questions (OEQ)
MOOC Designers
Q1. Which were the main design decisions that you made
during the development of the MOOC that later proved to
be successful?
Q2. Which were the three biggest difficulties that you faced
when designing the MOOC?
Q3. Which design decisions did not pay off as you expected?
Q4. Looking ahead into the development of this type of learning
design experiences, what methods and tools could be
helpful to improve the design of future MOOCs?

52. Open-ended Questions (MOOC Designers)
Q1. Which were the main design decisions that you made during the
development of the MOOC that later proved to be successful?
1
3
3
42
1
1
3
3
6
5
4
0
4
8
12
16
20
24
Assessibility&
Inclusion
Assessment &
Evaluation
Interaction &
Collaboration
Certification&
Accreditation
Curriculum
Design& Delivery
Expertise &
Manpower
Feedback &
Facilitation
Instructional
Design&
Technology
No.ofComments
Social Sciences, Humanities and Law
Education and Lifelong Learning
Computer and Informatics
Science, Maths and Engineering
Nature, Environment and Health
Business, Management and Economics

53. Open-ended Questions (MOOC Designers)
Q2. Which were the three biggest difficulties that you faced when designing the
MOOC?
3 3
1
3
3
2
8
1
2
2
1
1
1
2
3
5
4
4
2
5
3
0
4
8
12
16
20
24
Assessment &
Evaluation
Curriculum
Design&
Delivery
Expertise &
Manpower
Feedback &
Facilitation
Institutional
Support &
Funding
Instructional
Design&
Technology
Interaction &
Collaboration
Open Access,
Copyrights &
Licensing
No.ofComments
Social Sciences, Humanities and Law
Education and Lifelong Learning
Computer and Informatics
Science, Maths and Engineering
Nature, Environment and Health
Business, Management and
Economics

54. Open-ended Questions (MOOC Designers)
Successful Decisions Biggest Challenges
Curriculum Design & Delivery
• Content delivery format
• Content structure & LOs
• Weak choice & quality of content
• Short duration
Instructional Design & Technology
• Choice of learning activities
• Integration of IT & media
• Platform, software & production
decisions
Interaction & Collaboration Expertise & Manpower
• Creating community of learners
• Foster interaction between learner
& tutor/ facilitator
• Gap in content & instructional
design knowledge
• Coordination & collaboration, e.g.,
different experts & teaching staff
Three main successful decisions and three biggest challenges

55. Quality Reference
Framework (QRF)
for MOOCs

56. The Quality Reference Framework
Dimension 1: Phases Analysis, Design, Implementation, Realization, Evaluation
Dimension 2: Perspectives Pedagogical, Technological, and Strategic
Dimension 3: Roles Designer, Facilitator, and Provider

57. Quality Reference
Framework (QRF)
for MOOCs

58. Quality Reference
Framework (QRF)
for MOOCs

59. Analysis of the
needs and
demands for
the MOOC
DESIGN REALIZATION
ANALYSIS
IMPLEMENTATION
EVALUATION
Design
decisions on all
aspects of the
MOOC
Development of
the MOOC
following the
design
Running of the
MOOC with all
learning and
facilitation
processes
Parallel ongoing
evaluation of all
other four
phases
The Quality Reference Framework

60. MOOQ project:
www.MOOC-quality.eu
Online community for the
Quality Reference Framework:
www.MOOC-quality.net

61. First References for GMQS
Stracke, C. M., et al. (2018). Gap between MOOC designers' and MOOC
learners' perspectives on interaction and experiences in MOOCs: Findings
from the Global MOOC Quality Survey. In M. Chang, N.-S. Chen, R. Huang,
Kinshuk, K. Moudgalya, S. Murthy, & D. G. Sampson (Eds.), Proceedings
18th IEEE International Conference on Advanced Learning Technologies
(ICALT) (pp. 1-5). IEEE: Computer Society. DOI 10.1109/ICALT.2018.0000
Stracke, C. M., & Tan, E. (2018). The Quality of Open Online Learning and
Education: Towards a Quality Reference Framework for MOOCs. In J. Kay, &
R. Luckin (Eds.), Rethinking learning in the digital age. Making the Learning
Sciences Count: The International Conference of the Learning Sciences
(ICLS) 2018 (pp. 1029-1032). London: ISLS.
Stracke, C. M. et al. (2017). The Quality of Open Online Education: Towards
a Reference Framework for MOOCs. In Proceedings of 2017 IEEE Global
Engineering Education Conference (EDUCON) (pp. 1712-1715).
IEEE Xplore. DOI: 10.1109/EDUCON.2017.7943080
To be continued …

62. christian.stracke@ou.nl
esther.tan@ou.nl
@ChrMStracke
@Taneste
www.opening-up.education
Let’s collaborate!

63. Thank you!
Questions?