A deluge of empirical research became available on MOOCs in 2013-2015 and this research is available in disparate sources. This paper addresses a number of gaps in the scholarly understanding of MOOCs and presents a comprehensive picture of the literature by examining the geographic distribution, publication outlets, citations, data collection and analysis methods, and research strands of empirical research focusing on MOOCs during this time period. Results demonstrate that: more than 80% of this literature is published by individuals whose home institutions are in North America and Europe; a select few papers are widely cited while nearly half of the papers are cited zero times; and researchers have favored a quantitative if not positivist approach to the conduct of MOOC research, preferring the collection of data via surveys and automated methods. While some interpretive research was conducted on MOOCs in this time period, it was often basic and only a handful of studies were informed by methods traditionally associated with qualitative research (e.g., interviews, observations, focus groups). Analysis shows that there is limited research reported on instructor-related topics, and that even though researchers have attempted to identify and classify learners into various groupings, very little research examines the experiences of learner subpopulations.

1. A Systematic Analysis And Synthesis of
the Empirical MOOC Literature
Published in 2013-2015
George Veletsianos - Royal Roads University, Peter Shepherdson - University of Zurich, Laura Pasquini -
University of North Texas, & Rich McCue - UVic
The 9th Collaboration for Online Higher Education & Research (COHERE) Conference October 2015, Victoria & Halifax, Canada
Published paper: Veletsianos, G. & Shepherdson, P. ( 2016). A Systematic Analysis And Synthesis of the Empirical
MOOC Literature Published in 2013-2015. The International Review of Research in Open and Distributed Learning,
17(2). Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/2448/3655

2. Rich McCue
University of Victoria
George Veletsianos, Canada Research Chair in Innovative
Learning & Technology, Royal Roads University
Peter Shepherdson
University of Zurich
Laura Pasquini
University of North Texas

3. • What is the state of MOOC
research (very briefly)?
• Where are there gaps & what gaps
can we fill?
• Research Methods.
• Did we find anything interesting?
Overview

4. Introduction
We wanted to address a number of gaps in the scholarly
understanding of MOOCs and present a comprehensive
picture of the MOOC literature by examining:
geographic distribution,
publication outlets,
citations,
data collection and analysis methods, and
research strands of empirical research focusing on MOOCs.

5. Review of the Literature

6. cMOOCs and xMOOCs?
● cMOOCs are described as being “based on principles of connectivism, openness, and participatory teaching” (Jacoby, 2014, p.
76), “through a dynamic network of connections afforded by online technology” (Ebben & Murphy, 2014, p. 333).
● xMOOCs resemble “traditional teacher-directed course[s], yet automated, massive, and online” (Kennedy, 2014, p. 8).
● Early MOOCs tended to follow the cMOOC model, whereas more recently the number of xMOOCs delivered has been growing
rapidly.

7. Expected impacts of MOOCs on education
The size and openness of MOOCs have the potential to
disrupt conventional thinking about the role, value, and cost
of higher education.
The discourse around MOOCs could diminish of the
authority and importance of the educational leader.
MOOCs may threaten the business models of universities.

8. Demographics of MOOC users
• Vast majority male, 20-40 yrs old.
• College degree or higher.
• More than 1/2 of learners from
outside the United States.
• Majority of participants are from
North America and Europe, with a
small minority being from Asia,
Southeast Asia, or Africa.

9. Challenges for MOOC’s
Completion rates in MOOCs are less than 10%. Why?
A lack of incentive, many courses free.
Insufficient prior knowledge (e.g., lack of math or language skills).
Failure to understand the content and having no one to turn to for help,
Lack of time due to having other priorities and commitments.
90%

10. Research Questions:
RQ1: How is MOOC research geographically distributed?
RQ2: Is MOOC research usually published in journal or
conference proceedings? In which journals and conference
proceedings is MOOC research currently being published?
RQ3: Which empirical MOOC studies are cited the most?
RQ4: What data collection methods and data analysis
methods are used in empirical studies of MOOCs?
RQ5: What are the research strands of empirical MOOC
research?

11. Research Methods

12. Literature discovery searches were conducted using the [NEXT] keywords “MOOC” or “Massive Open Online Course.” To be
included articles had to be:
(1) empirical,
(2) published in a peer-reviewed journal, in conference proceedings, or in Educause Review,
(3) published or was available online as in press between January 2013 and January 2015, and
(4) written in English.
All identified papers were examined by [NEXT] two researchers to ensure consensus that they fit the inclusion criteria.
Forwarding references: In [NEXT] Google scholar looked up the papers and the looked at papers that cited those papers to
discover new papers. Found 60 new papers.
Completeness Search: Examined references of 17 papers published in 2015 in our corpus to identify any papers that we missed.
They were more likely to reference literature published in 2013-2014 than the papers published in 2013 or 2014. 5 new papers
found.
[NEXT] The final number of published papers that constituted the corpus of this study was 183.
Data Collection
Search Keywords: “MOOC” or “Massive Open Online Course”
Published Papers: n=183
Authors: n=460

13. Limitations
1. Limited duration - 3
years.
2. Only English
language data.
3. Quality of research
not evaluated.
4. Some content
analyzed, but not all.

14. Results

15. RQ1: How is MOOC research geographically distributed?
Country Corpus
USA 50.2%
UK Kingdom 10%
Australia 7.7%
China 5.4%
Spain 4.8%
Canada 4.5%
Germany 2.2%
Switzerland 1.3%
Netherlands 1.1%
Other (29 countries) 12.8%
Author Affiliations

16. RQ2a: Is MOOC research usually published in
journals or conference proceedings?

17. RQ2b: In which journals and conference
proceedings is MOOC research currently being
published?
Outlet name # of papers Type
International Review of Research in Open and
Distributed Learning (IRRODL) 18 Journal
Proceedings of the first ACM conference on Learning
@ scale (L@S '14) 11 Conference
Journal of Online Learning and Teaching (JOLT) 7 Journal
Conference on Empirical Methods in Natural
Language Processing (EMNLP) 5 Conference
Distance Education 5 Journal
NIPS Workshop on Data Driven Education 5 Conference
Proceedings of the 2014 ASCILITE Conference 4 Conference
EDUCAUSE Review 4 Journal

18. RQ3: Which empirical MOOC studies are cited
the most?
Paper # of citations
- Studying learning in the worldwide classroom research into edX’s
first MOOC.
124
- Deconstructing disengagement: Analyzing learner subpopulations in
massive open online courses.
117
- MOOCs and the funnel of participation. 83
- Tuned Models of Peer Assessment in MOOCs. 73
- Automated Feedback Generation for Introductory Programming
Assignments.
56
- Retention and Intention in Massive Open Online Courses: In Depth. 41
- “Turn on, Tune in, Drop out”: Anticipating student dropouts in
Massive Open Online Courses.
35

19. RQ4: What data collection methods & analysis are
used in empirical studies of MOOCs?
Analytic method Frequency (%) of
total papers
Descriptive statistics 93.4
Correlational 52.5
Basic qualitative study 38.8
Experimental and quasi-experimental 25.7
Grounded Theory 7.6
Natural Language processing 7.6
Social Network Analysis 6.6
Ethnography 4.4
Phenomenology 2.2
Discourse analysis 1.0

20. RQ5: What are the research strands of empirical
MOOC research?
● [NEXT] 83.6% of the papers focused on student-related areas.
● [NEXT] Nearly half (46.4%) of the papers identified in the literature search had some focus on
topics relating to the design, creation, and implementation of MOOCs themselves. This included
research pertaining to methods of assessment, the description of unique learning environments,
the creation of MOOCs on specific topics, and the evaluation of course success.
● [NEXT] Only 8.2% of papers focused on topics related to instructors and teaching. Papers within
this theme largely focused on academics’ awareness, perspectives of, and experiences with
MOOCs.
● [NEXT] 10.9% of the corpus contained content pertaining to the context and impact of MOOCs.
This included research into perceptions of MOOCs, their usefulness as an educational medium,
and their economic impact .
● [NEXT] Other.
84%
46%
8%
11%

21. Discussion & Conclusions

22. Dependence on particular research methods
may restrict our understanding of MOOCs
Analysis suggests that researchers have favored a
quantitative, if not positivist approach to the conduct of
MOOC research.

23. Survey data and secondary data collected via automated
methods were also favored.
While some interpretive research was conducted in MOOCs
in this time period, it was often basic and only a handful of
studies were informed by methods traditionally associated
with qualitative research (e.g., interviews, observations,
focus groups).

24. Thus, even though results suggest that research on MOOCs focuses on student-related topics, learners’ voices are largely absent
in the literature.
Based on these results, we suggest that an expansion of the methodological approaches used in MOOC research is urgently
needed.

25. Understanding Learner Subpopulations
Completing
Auditing
Disengaging
Sampling
No-shows
Observers
Drop-ins
Latecomers
Drop-in latecomers
Non-engaged
Engaged

26. Understanding Learner Subpopulations
Little research examines the experiences of
different populations and how and why learning
experiences differ between groups.
Future research into this area for instance could
examine why some learners disengage, how the
learning experience of drop-ins differs from the
learning experience of those who are non-
engaged, and what interventions may scaffold
different types of learners.

27. The geography of MOOC research
Our geographical analysis of author affiliations showed that over half of the authors conducted their research in the USA.
Over 80% of authors were affiliated with institutions in North America or Europe.

28. Percentage of Research Published in USA
In contrast, according to a Scimago search of all citable documents published in 2013 across all disciplines only 19% came from
the USA, and it takes the top 20 countries (including at least seven countries from outside of North America and Europe) to account
for 80% of academic output.
As such, current MOOC research is simply not a reflection of geographical contributions to academic output in general.
0% 15% 30% 45% 60%
MOOC Resarch
All Other Research

29. Future Directions
We hope that this systematic analysis enables researchers to make better sense of the empirical literature on MOOCs and its
direction and limitations.
There are many possibilities for future research in this area. Future systematic reviews of the literature may focus on:
Synthesizing the knowledge on particular areas of interest, (completion and retention in MOOCs; learner motivations in
MOOCs).
Examining whether research methods used to understand MOOCs follow standard methods of inquiry.
Follow methods that take into advantage the digital nature of learning and teaching in this context.

30. Final Thoughts
Finally, we hope that our results highlight the need for a critical reflection on the part of researchers as to the methods they have
used to understand MOOCs to date and encourage a greater diversity in research methods.

31. Thank you!
Research available at: http://veletsianos/publications
This presentation:
http://slideshare.net/richmccue
Contact:
veletsianos@gmail.com
@veletsianos on twitter
@richmccue on twitter
Photo credit: Got Credit

32. Special Thanks to the Creators of the Creative
Commons Images used in this presentation:
• https://www.flickr.com/photos/47572798@N00/8397808475
• https://commons.wikimedia.org/wiki/File:Wet_winding_road.jpg
• https://pixabay.com/en/world-map-map-world-black-earth-297446/
• http://creativecommons.org/weblog/entry/26283
• https://en.wikipedia.org/wiki/Printing_press#/media/File:PrintMus_038.jpg
• https://commons.wikimedia.org/wiki/File:2011_Library_of_Congress_USA_5466788868_card_catalog.jpg
• https://upload.wikimedia.org/wikipedia/commons/7/72/Rhll_wire_rope.jpg
• https://upload.wikimedia.org/wikipedia/commons/0/0a/
Urval_av_de_bocker_som_har_vunnit_Nordiska_radets_litteraturpris_under_de_50_ar_som_priset_funnits_(3).jpg
• https://www.flickr.com/photos/mathplourde/8620174342/sizes/l/in/photostream/
• http://orig00.deviantart.net/34c5/f/2011/263/d/6/destruction_by_tbh_1138-d4af5vp.png
• https://www.flickr.com/photos/68751915@N05/6355351769
• https://upload.wikimedia.org/wikipedia/commons/1/10/ECLA_of_Bard_Seminar,_2012.jpg
• https://www.flickr.com/photos/carbonnyc/143186839
• https://pixabay.com/en/beer-quit-quitting-can-vintage-398742/
• https://pixabay.com/en/blueprint-ruler-architecture-964629/
• https://www.flickr.com/photos/keithroper/8139626676
• http://www.picserver.org/images/highway/phrases/results.jpg
• https://upload.wikimedia.org/wikipedia/commons/4/4b/Colossus.jpg
• http://www.apa.org/images/apastyle_logo.gif
• https://upload.wikimedia.org/wikipedia/commons/3/39/Student_in_Class_(3618969705).jpg
• https://upload.wikimedia.org/wikipedia/commons/e/e6/Avrocar_schematic_(high_resolution).jpg
• http://teachers.net/teachers.jpg
• https://pixabay.com/p-90781/?no_redirect
• https://pixabay.com/en/people-sitting-chairs-in-front-690810/
• https://upload.wikimedia.org/wikipedia/commons/1/17/1895-Dictionary-Phrenolog.png
• https://www.flickr.com/photos/safari_vacation/10362491406
• https://www.flickr.com/photos/16210667@N02/15617432129
• http://www.veletsianos.com/wp-content/uploads/2013/05/edtechmooc.jpg
• https://www.flickr.com/photos/intersectionconsulting/7537238368/
• https://pixabay.com/en/compass-antique-map-of-the-world-429772/
• http://asiimdesgraphic.deviantart.com/art/Mirror-Ball-Reflection-214407739