This presentation examines the role of digital badges – an emerging alternative microcredential – in a Massive Open Online Course (MOOC) called Math in Sports (Math) offered by the University of Notre Dame in 2016. Three badges were designed and awarded with an accompanying study. The study addressed four research questions about the impact of digital badges on MOOC participants, the profile of digital badge pursuers, the perceptions of digital badges, and the ability digital badges have to ensure that MOOCs serve their purpose. We can draw five major conclusions from this research: “MOOC participant” should only describe those who engage with MOOC content MOOC passers are interested in credentials that distinguish them from other course participants MOOC passers do not perceive the digital badge to be equivalent to the verified certificate The majority of MOOC passers are previously-educated, adult males from the United States, but digital badges may be used to diversify the population of MOOC completers MOOC participants need more information about digital badges to fully understand and appreciate their value. With these conclusions in mind, we offer three design recommendations for badges in MOOCs: create digital badges that recognize, validate, and assess distinct and specialized knowledge or skills gained in the course; distribute more information about the value of digital badges and how to use them; and connect digital badges more directly with social networking sites, particularly LinkedIn.

1. Examining the Role of
Digital Badges in a
University’s Massive Open
Online Course
Elizabeth Anthony & G. Alex Ambrose
University of Notre Dame
AAEEBL Midwest Conference 2016

2. MOOC, Digital Portfolio, & Badge
Research & Design Team
Kaneb Center for Teaching & Learning
• G. Alex Ambrose: (PI) & Digital Learning Scientist
• Elizabeth Anthony: Research Assistant
• Kevin Barry: Director of Center
• Chris Clark: Assistant Director of Center
• Kevin Abbott: Educational Technology Specialist (OIT)
• John Dillon: Post Doc Fellow /USAID Fellow
• Alison Lanski: ePortfolio Implementation Specialist
Office of Digital Learning (ODL)
• Elliott Visconsi, Chief Academic Digital Officer
• Laurie Kirkner, Interim Director and Senior Learning Designer
• Crystal DeJaegher: ODL Instructional Designer
• KC Frye, Multimedia Artist and Editor
• Sonia Howell, Digital Initiatives Postdoctoral Researcher and Course Producer
Office of Information Technologies
• Xiaojing Duan: Platform Engineer & Data Czar
• Patrick Miller, Sakai Lead Learning Management Customer IT Solutions
• Laura Gekeler, Sakai Learning Management Professional Customer IT Solutions
2
Corporate Sponsors & Partners

3. Anthony, Elizabeth, Ambrose, G. Alex (2016) “Examining
the Role of Digital Badges in a University’s Massive Open
Online Course” Association of Authentic, Experiential, and
Evidenced-based Learning (AAEEBL) Midwest Regional
Conference. Notre Dame, Indiana.
How to Cite this Presentation:

4. Notre Dame MOOC Badge Timeline
July 2015
Course: I "Heart" Stats:
Learning to Love Statistics
Issued: 151x I Heart Stats
(passed) badges via
Credly
Sept 2015
Course: Math in Sports
Issued 21x Math in
Sports, 44x All-Star, and
76x Coach (peer ranked)
badges
May 2016
Issued 4 x Tempietto
(ePortfolio & evidence-
based) Badges via edX &
Digication integration
5 different (Credly) Open Badges in 3 (edX) MOOCs issued to 231 global learners

5. Background & Context
Credential Options for Math in Sports: Certificates
• Issued by edX
• Criteria
• Earn at least a 70% on all
course assessments
• Honor code certificate
• Free
• No longer available as of 2016
• Verified certificate
• Cost: $50

6. Background & Context
Credential Options for Math in Sports: Digital Badges
• Issued by the University
of Notre Dame
• Criteria
• “Math”: Access all course
modules and earn at least
a 70% on all course
assessments
• ”All-Star”: Access all
course modules and earn
at least a 90% on all
course assessments
• “Coach”: earn “Math” or
“All-Star” + be in the top
20 discussion board
posters

7. Background & Context
Vocabulary Enrollee
Participant
Passer = Math
Badge-earner = Honor Code Certificate-earner
Verified Certificate Earner
“All-Star” Badge-earner
“Coach” Badge-earner
not to scale

8. Research Problem
Traditional students
Modern
learners• Attend formal institutions
of higher education
• Hold general degree
• Only a small subset of
learners today
• Seek specific knowledge
and skills from
nontraditional settings
• Need a way to validate and
communicate knowledge
and skills
15%
completion
rate in
MOOCs
?
MOOC
credentials

9. Research Questions
1. How do digital badges impact MOOC participants'
course completion and performance?
2. What is the profile of MOOC digital badge earners?
3. What are MOOC participants' perceptions of digital
badges?
4. Can digital badges help ND ensure that MOOCs serve
their purpose?

10. Methodology
Research Design & Instrumentation
• Methodology: Design-Based
Research
• Type & Emphasis:
Development, Use, and
Evaluation of Tool
• Digital Badges
• Methods: Mixed
• Evaluation, survey, expert
review
• Expert review by Dr. Daniel
Hickey and Dr. Steve Lonn

11. Methodology
Procedure
I Heart Stats
Preliminary
study
• One digital badge
• Post-course
survey
Design
badges
• Grounded in
literature
Offer badges
during course
• Bi-weekly
reminder email
Issue badges
after course
• Credly
• One week claim
period
Post-course
survey
• Sent to all
passers
• Response rate
Evaluation
and Analysis
• Survey data
• Course
demographic
information
• Digital tracks and
traces from
course and
Credly

12. Results
Completion and Performance
• 10,076 students enrolled in the
course
• 753 (7.5%) students participated
in the course by engaging with
the course material
• 170 (1.7%) students passed the
course by earning 70% or above
• 83 students earned the “Math in
Sports” Badge
• 87 students earned the “All-Star”
Badge
• 40% declared that the digital
badges motivated them to
complete all course modules and
score highly on course
assessments

13. • Typical badge-earner:
• From the United States (40.5%)
or Europe (30.85%)
• Male (84%)
• Aged 24-60 (84.1%)
• Has at least a bachelor’s degree
(83%)
• Exceptions:
• Far greater proportion of female
“Coach” badge-earners
• Greater proportion of “All-Star”
badge-earners from Asia
Results
Profile
0%
20%
40%
60%
80%
100%
"Math" "All-Star" "Coach"
Gender of Math Passers
Male Female
0
10
20
30
40
50
60
USA Europe North
America,
outside
of the US
Asia South
America
Australia Africa Other
Location of Math Passers
"Math" "All-Star" "Coach"

14. Results
Perceptions
1. Perceptions of
digital badge
and verified
certificate are
generally
comparable
2. Generally
positive
perceptions of
digital badges
3. Distinguish
between the
verified
certificate and
digital badges
1
1.5
2
2.5
3
3.5
4
4.5
5
I believe in the
value of the…
Other people
believe in the
value of the…
I understand
where and how
to use the…
The ________
fulfills the same
role as the
________
The criteria for
the
_______were
clearly defined
and easy to
understand
I was familiar
with ________
before taking
this course
LevelofAgreementonLikertScale
Math Passers’ Perceptions of Verified
Certificates and Digital Badges
Verified Certificate Digital Badge

15. 25.3%
50.6%
55%
0
20
40
60
80
100
"Math" "All-Star" "Coach"
Number of Earned Badges
Claimed and Not Claimed
Not Claimed Claimed
Results
Perceptions
• The claim rate for the “All-Star” and
“Coach” badges were more than double
that of the “Math” badge.
• Comments reveal that digital badges
often meant very little to the course
participants, with the exception of the
“Coach” Badge.
• Those who did not pursue or claim their
digital badge(s)overwhelmingly
indicated that they did not need or did
not understand the value of the badge.
• 40% of badge-earners plan to display
their badge on a social networking site.
“Nice to see a credential which
validates the fundamental/organic
effort of class participation that is the
root of online academic efforts and
their comparative value over time…”

16. Results
Purpose
1. Encourage
innovation in
education and
provide opportunities
for educational
research.
2. Expand access to
education.
3. Showcase the
University’s merit
worldwide.
1
1.5
2
2.5
3
3.5
4
4.5
5
Before After
Math Passers’ Perceptions of the
University Before and After the Course

17. Conclusions
1. The term “MOOC Participant” should only be used to describe
those who engage with the course.
2. MOOC passers are more interested in credentials that distinguish
them from other MOOC passers than credentials that verify their
completion.
3. MOOC passers do not perceive the digital badge to be equivalent
to the verified certificate and do not desire a credential with the
features of the certificate.
4. The majority of MOOC passers are formally educated adult males
from the United States, but digital badges may be tools that can be
used to engage a more diverse population of students in MOOCs.
5. MOOC participants need more information about digital badges in
order to fully understand and appreciate the value of digital
badges.

18. Design Recommendations
1. Create digital badges that recognize, validate, and assess
distinct and specialized knowledge or skills gained in the
course.
2. Distribute more information about the value of digital badges
and how to use them.
3. Connect digital badges more directly with social networking
sites, particularly LinkedIn.

19. Future Research
ePortfolio & Evidence-based badges in Notre
Dame’s architecture MOOC

20. Digication (ePortfolio)   Credly (Badge)
Integration
-Single Sign On Account Sync
-Import & Embed Digital Badges into ePortfolio
-Showcase Badge with Linking Evidence
edX (MOOC)   Digication (ePortfolio )
Integration
-Single Sign On Account Sync
-Course & Assignment Management
-Template & Rubric

21. Citation
Ambrose, G. Alex, Anthony, Elizabeth, DeJaegher, Duan, Xiaojing, Crystal, Dillon, John ( 2015) "Examining Digital Badge Impact
on Learners’ Profiles, Performance & Perceptions in a Massive Open Online Course" University of Notre Dame, Research &
Assessment for Learning Design Lab. https://drive.google.com/a/nd.edu/file/d/0B8AIPHhRUVuYS3NuSlRSLWdMOWM/view

22. Questions?
Elizabeth Anthony
eanthony@nd.edu
Alex Ambrose
gambrose@nd.edu

23. Links
News Announcement: ND Issues (& Studies) 5 different Open Badges
in 3 edX MOOCs issued to 231 global learners
edX and Credly Links
• Course: Math in Sports
• Badge: Math in Sports
• Badge: Math in Sports Coach
• Badge: Math in Sports-All Star