Transaction Management in Database Management System
Wsu guest lecture_2015_czlv_v3
1. Evaluating the ‘Student’ Experience in
Massive Open Online Courses (MOOCs)
Catherine Zhao & Lorenzo Vigentini
Learning and Teaching Unit, UNSW Australia
2. In this presentation -
• Background – Previous work on Student learning
Experience & UNSW MOOCs
• Methods - the surveys for evaluation
• Findings -
o Surveys - fit-for-purpose?
o Learners – what’s it like to experience a MOOC?
• Current work & Challenges
3. A Model of the Student Learning Experience
Worth of
Experience
Future
Past
Motivation
Experience
Perception
Engagement
Aspirations
Expectations
Personal
development
‘match’ or ‘fit’ with
context/LE
Direct future
benefits
Vigentini, L., and Zhao, C. Revisiting end-of-semester evaluation: the questions, the instrument and the
process.,” in AAIR : turning silver into gold, Melbourne, 2014.
6. Five known facts about MOOCs
Massive enrollments
Dramatic dropouts
Unconventional learners
(Relatively) limited design/delivery options
Engagement and the learner experience
The rise of ‘certificates’
7.
8. Background - Four MOOCs we look at
A B C D
Disciplines Engineering Education Science Medicine
Duration (weeks) 9 6 9 8
Target Group Engineers Teachers
High school
students/ teachers
General public
Course Design All-at-once All-at-once Sequential Semi-sequential
Course Delivery All-at-once Staggered Staggered Staggered
Videos 110 224 98 58
Quizzes 10 22 42 2
Assignments 7 3 2 2
Forums 54 105 63 64
Registrants (Active
learners*)
32928 (59.96%) 28864 (62.59%) 22761 (46.67%) 13185 (35.44%)
*Active learner: those who appear at least once during the course
MOOCs – A = engineering, B = education, C = physics, D = medicine
9. Background- What & When Surveys
Evaluation Surveys Deployment A B C D
Demographic survey
Prior to course
commencement
Pre-course survey OR
First course activity
In-video surveys (IVS)
During
the
course
Quick evaluation (Video)
Quick evaluation (Module)
Quick Question survey
Post-course survey End of course
Present in MOOC
MOOCs – A = engineering, B = education, C = physics, D = medicine
10. Methods - Overview of the Pre-Course survey questions
Common A B C D Course-specific A B C D
Reasons/Purpose of Joining Background-Gender, Ethnicity
Prior Professional
Experience/Knowledge
English Proficiency
Intended Effort Prior MOOC Experience
Familiarisation with Topics
Geographic Location
Academic Qualification
Contributing Elements of
‘Teaching Online’
Industry of Employment
Defining ‘Completion'
Preventing Factors for Course
Completion
Learning Preference
Expected Outcomes
(course related)
Achievement Goal Oriented
Present in the course
MOOCs – A = engineering, B = education, C = physics, D = medicine
11. Common A B C D Course-specific A B C D
Course Satisfaction
Proportion of Completed Course
Activities
Course Features Evaluating Peer Assessment
Actual Effort
(Probing) Self-Regulated Learning
Desired Topics
Satisfaction on Personal Development
(Intend to) Apply Course Topics
Considering Credit-Bearing Courses
Methods- Overview of the Post-course survey questions
MOOCs – A = engineering, B = education, C = physics, D = medicine
12. Methods– Pre- & Post-Course Survey Sample Sizes
A B C D
# Active Learners 19708 18024 10576 4673
Required sample size* 2141 2119 1957 1587
Pre- & Post- Course survey
Pre-course survey sample 2684 4490 264 401
response rate 13.62% 24.91% 2.50% 8.58%
Margin of error 1.76% 1.28% 5.96% 4.68%
Post-course survey sample 638 313 126 71
response rate 3.24% 1.74% 1.19% 1.52%
Margin of error 3.82% 5.49% 8.68% 11.54%
Coursera Demographic Survey
# of registrants 32928 28864 22487 13185
Demographic Survey Responses 2774 2873 2720 1466
Margin of error 1.20% 2.20% 1.52% 2.41%
*Required sample size is calculated under the condition of CL = 95%, and margin of error = 2%
13. Methods – Patterns of participation in Pre-& Post-
Course surveys
Counts of responses in the four MOOCs’ pre-course surveys (logarithmic scale) over 9 weeks.
1
10
100
1000
10000
W1 W2 W3 W4 W5 W6 W7 W8 W9
Countofresponses A
Pre Post
1
10
100
1000
10000
W1 W2 W3 W4 W5 W6 W7 W8 W9
B
Pre Post
1
10
100
1000
10000
W1 W2 W3 W4 W5 W6 W7 W8 W9
Countofresponses
C
Pre Post
1
10
100
1000
10000
W1 W2 W3 W4 W5 W6 W7 W8 W9
D
Pre Post
14. Methods- Collection tools
Course A IVS (In-video survey) Course C Quick Evaluation
MOOCs – A = engineering, B = education, C = physics, D = medicine
15. Research Questions
1) Are the tools used robust enough to characterize the
learners’ experience in MOOC?
2) Are there substantial differences in the modes of
gathering feedback?
3) Is it possible to identify key elements of the learners’
MOOC experiences?
4) What is the relevance of ‘teaching’ questions for the
online experience?
16. Results – Are the surveys robust?
• Accuracy
• A B C D
Required sample size (CI=95%,margin of error 2%) 2141 2119 1957 1587
Pre-course survey sample 2684 4490 264 401
Margin of error 1.76% 1.28% 5.96% 4.68%
Post-course survey sample 638 313 126 71
Margin of error 3.82% 5.49% 8.68% 11.54%
17. Results – Are the surveys robust? (Cont’d)
A B C D
Motivation Block 0.78 0.86 0.65
Course experience Block 0.90 0.94 0.89 0.92
IVS items 0.82
Confidence 0.84
Satisfaction of personal gain 0.91
Satisfaction of course materials and
personal gain
0.92
(probing) self-regulated learning skills 0.90
• Reliability – Cronbach’s alpha
MOOCs – A = engineering, B = education, C = physics, D = medicine
18. Results – Are the surveys robust? (Cont’d)
• For the motivation block that is <.7
(D) Reason items
Cronbach's Alpha if
Item Deleted
I am taking the course out of general interest, curiosity, or
enjoyment.
.722
I am interested in taking a course from this particular
institution.
.616
I am interested in learning from the expert researchers
and clinicians delivering the course.
.636
I want to connect with other students interested in this
topic.
.589
The course supports my current academic program. .565
The course is aligned with my current job responsibilities
or company's line-of-business.
.588
The skills gained from this course may be useful for
obtaining a new job.
.571
MOOCs – A = engineering, B = education, C = physics, D = medicine
19. Results – Comparison of Evaluation Approaches
• The reasons for adopting different approaches
1. Technical constraints
2. Accommodating various requirements put
forward by academic leads
3. Test different tools/ways to evaluate
• Two key sets of approaches we are comparing -
1. First course activity in A & B vs. Pre-course
survey in C & D
2. In-Video Surveys (IVS) vs. Quick Evaluation (QE)
MOOCs – A = engineering, B = education, C = physics, D = medicine
20. Results – Effectiveness of Pre-course survey
• ‘Embedded’ (= first course activity) is more effective
than ‘stand-alone’
• Higher response rate
• measures how much learners want to engage
A B C D
13.62% 24.91% 2.50% 8.58%
MOOCs – A = engineering, B = education, C = physics, D = medicine
21. 1
10
100
1000
10000
100000
W1 W2 W3 W4 W5 W6 W7 W8 W9
A - IVS
INTSE Views of
lecture videos
INTSE # Participation
in IVQs
1
10
100
1000
10000
100000
W1 W2 W3 W4 W5 W6 W7 W8 W9
C - Quick Evaluation P2P Views of lecture
videos
P2P # Participation in
Quick Evaluation
Results – In-Video-Surveys vs. Quick Evaluation
Inside the video is more effective than outside the video.
MOOCs – A = engineering, B = education, C = physics, D = medicine
22. Our learners are…
A B C D
Ave. Age (SD)
34.37
(11.37)
39.91
(12.39)
35.9
(15)
37.35
(13.84)
Gender ratio
(M/F)
4/1 1/1.2 2.6/1 1/1.2
Full-time
Employed
57% 63% 59% 59%
Full-time
Studying
41% 30% 39% 38%
Bachelors and
Above
74% 82% 63% 64%
Prior Experience 41.51% 57.20% - 79.25%
MOOCs – A = engineering, B = education, C = physics, D = medicine
23. Results – motivations I
A C D
M(SD) M(SD) M(SD)
For personal growth and enrichment 3.23 (.76) 2.10(.72)
General interest in topic 3.14 (.74) 2.23(.72) 3.23(.92)
Relevant to job 2.63 (1.01) 1.6(.76) 2.71(1.38)
For fun and challenge 2.58 (.96) 1.42(.63)
Earn a certificate/statement of
accomplishment
2.42 (1.03) 1.55 (.70)
Course offered by prestigious
university/professor
2.17 (1.03) 1.91(.77) 3.06(.97)
For career change 2.07 (1.06) 1.32(.59) 1.92(1.41)
Experience an online course 1.81 (.93) 1.21(.51)
Relevant to school or degree program 1.75 (1.00) 1.74(.83) 2.56(1.37)
To improve my English skills 1.66 (.95) 1.52(.71)
Relevant to academic research 1.65 (.98) 2.37(.86)
Meet new people 1.52 (.76) 1.65(.70) 1.96(1.06)
Take with colleagues/friends 1.32 (.68) 1.93(.75)
1=Not at all important, 2 = somewhat important, 3 = very important, 4 = extremely important
24. Results – Motivations II
Reasons
Item loading
A1
C2
D3
I II III VI I II III VI I II
Experience an online course1,2 0.7 0.7
Meet new people1,2,3 0.7 0.8 0.6
To improve my English skills1,2 0.7 0.2 0.6
Take with colleagues/friends1,2 0.7 0.6
Course offered by prestigious
University/Professor1,2,3
0.6 0.8 0.8
For fun and challenge1,2 0.8 0.7
Personal growth&enrichment1,2 0.8 0.8
General interest in topic1,2,3 0.7 0.8 0.5
Relevant to academic research1,2 0.9 0.8
Relevant to academic study1,2,3 0.9 0.8 0.8
Relevant to job1,2,3 0.8 0.7 0.8
Earn a certificate1,2 0.6
For career change1,2,3 0.5 0.8 0.7
Items with loading lower than 0.5 are removed. Superscripts 1, 2, 3 represent occurrence in A, C, & D respective.
MOOCs – A = engineering, B = education, C = physics, D = medicine
26. Results – Course B Specifics (Cont’d)
2.20%
2.77%
3.72%
3.77%
8.76%
14.17%
30.43%
34.17%
I want to develop my knowledge and
understanding of a specific concept
I want to exchange ideas and learn from
colleagues
I am interested in looking at the course design
I am curious about the topic
I want to complete the course and obtain
recognition and certification for my work
I want to use my knowledge to develop a personal
online teaching strategy
I want to develop my knowledge and
understanding of the overall topic
I want to develop an online learning design that I
can use in my own teaching
(N = 4083)
• Overview of Learners’ Intents
27. Results – Course B Specifics (Cont’d)
• Intent is not sig. correlated to learners’ behaviours
• Data Mining (K-NN, Naïve Bayes) using learners’ characteristics (incl.
behaviours) indicate 30% predictive accuracy for performance class (no
grade, pass, distinction).
32. Results – Engagement with Forum
A B
C D
MOOCs – A = engineering, B = education, C = physics, D = medicine
33.
34. Results – Engagement with Quizzes
A B
C D
MOOCs – A = engineering, B = education, C = physics, D = medicine
35. Results – Engagement with Videos
A B
C D
MOOCs – A = engineering, B = education, C = physics, D = medicine
36. Results – Engagement III
Comparing paying & non-
paying learners
F t df
Sig.
2-tails
Mean
Diff’
CI=95% Interval
of diff'
Lower Upper
Q1 I find this lecture useful.
33.66 -4.9 9092 <0.05 -0.11 -0.16 -0.07
Q2 I understand the content
of this lecture. 24.81 -3.89 9045 <0.05 -0.09 -0.13 -0.04
Q3 I’d like to explore other
modules of this course.
43.62 -5.16 8896 <0.05 -0.11 -0.15 -0.07
Results of F-test suggest unequal variance hence the t-test results assumed unequal variance are used.
(Using Course A as an example – in-video survey items)
MOOCs – A = engineering, B = education, C = physics, D = medicine
37. Results – Learners’ Course Experience
Course experience items (mapped on new CATEI)
A B C
I II I II I II
Overall, I am satisfied with the quality of this course. 0.9 0.7 0.6
The material of the course was presented in an engaging manner. 0.9 0.8 0.8
Overall, this course met my expectations. 0.9 0.8 0.6
Examples, illustrations or real-world cases were used effectively to
explain things.
0.9 0.8 0.7
The course encouraged my interest in this field of study. 0.8 0.8
The goals and requirements of the course were made clear to me. 0.8 0.8 0.7
Overall I have improved the knowledge/skills I will need. 0.8 0.8 0.8
Quizzes helped me to evaluate my progress effectively. 0.6 0.6 0.8
Interacting in the forums helped me to clarify things I did not
understand.
0.9 0.9 0.6
Items with loading lower than 0.5 are removed.
MOOCs – A = engineering, B = education, C = physics, D = medicine
38. Course experience Items
D
I II
Depth of content 0.8
Course materials 0.8
Lecture videos 0.8
Clarity of explanations 0.8
Course content 0.8
Knowledgeability 0.8
Instructor/teaching staff 0.8
Overall satisfaction 0.7
In-video quizzes 0.7
Presentation skills 0.5
Assessments 0.9
Discussion forums 0.6
Extraction Method: Principal Component Analysis.
Rotation Method: Varimax with Kaiser Normalization.
a. Rotation converged in 3 iterations.
Results – Learners’ Course Experience (Cont’d)
MOOCs – A = engineering, B = education, C = physics, D = medicine
39. Results – the Worth of Course Experience
R2 F Sig.
A 0.17 1.22 0.26
C 0.79 0.72 0.72
D 0.39 1.08 0.38
• Motivation cannot predict satisfaction
LinearRegression(motivations) -> ‘How satisfied am I’
MOOCs – A = engineering, B = education, C = physics, D = medicine
40. Aspects
(Question Block) A coefficient Sig'
Course
Experience
Overall, this course met my expectations 0.54 0.00
The material of the course was presented in an engaging manner 0.19 0.00
The course encouraged my interest in its field of study 0.11 0.00
Examples, illustrations or real-world cases were used effectively to explain
things
0.08 0.01
The goals and requirements of the course were made clear to me 0.07 0.02
Aspects
(Question Block B coefficient Sig'
Course
Experience
Overall this course met my expectations 0.57 0.00
Satisfaction Provided an opportunity for fun and challenge 0.10 0.03
What can predict satisfaction
‘How satisfied am I’ <= LinearRegression(multiple items)
Results – the Worth of Course Experience (Cont’d)
41. Results – the Worth of Course Experience (Cont’d)
Aspects
(Question Block)
C coefficient Sig'
Overall I have improved the knowledge/skills I will need 0.63 0.00
Course Experience The course encouraged my interest in this field of study 0.61 0.01
The material of the course was presented in an engaging manner 0.32 0.02
The course encouraged my interest in this field of study 0.36 0.02
The experimental tasks were useful to apply the theory 0.30 0.02
LinearRegression(multiple items) ->‘How satisfied am I’
Aspects
(Question Block)
D coefficient Sig'
Self-regulation
I am confident that I could deal efficiently with unexpected events. 0.87 0.00
I can remain calm when facing difficulties because I can rely on my
coping abilities. 0.62 0.01
I can solve most problems if I invest the necessary effort. 0.35 0.02
I can usually handle whatever comes my way 0.32 0.05
Course Experience
Discussion forums 0.42 0.01
In-video quizzes 0.45 0.03
Depth of content 0.67 0.03
I found the material interesting. 0.31 0.03
Motivation
I am taking the course out of general interest, curiosity, or
enjoyment.
0.28 0.03
42. Conclusions
1. HBR paper*: two types of learners – ‘career
builders’ and ‘education seekers’.
2. Motivation weakly related to engagement or
satisfaction
3. Some relations between behaviors (engagement)
and performance
4. ‘Embedded’ evaluation tools are more effective
5. Mapping of ‘online’ learning experience
* Zhenghao, C., Alcorn, B., Christensen, G., Eriksson, N., Koller, D., & Emanuel, E. J. (2015). Who’s Benefiting
from MOOCs, and Why. Retrieved October 29, 2015, from https://hbr.org/2015/09/whos-benefiting-from-
moocs-and-why
43. Challenges
What are the characteristics of MOOCs that scaffolds
learning?
The intricacy of the combination of the factors which
contribute to the worth of course experience.
The matter of Agency.
Student experience is too broad, includes campus life and you may get questions
Services are supporting it, but focus is on learning experience
Be prepared
Bridging slide for everyone
Give them context
Tell them what A, B, C..
Explain multiple iterations
3x intse
2x ltto
2x p2p
1x Pmed
HL next in line
CATEI Is eval of teaching for management, so we test some questions
Say its relatively small
Call it how keen they are to do stuff in the mooc instead.
NEED to run an ANOVA on N responses
Factors/levels: Credit (y/n) x course (4) x timing (1st week/later)
Correlated, NOT related suggesting low/weak association
Separate things!
Otherwise you have no idea what it is
Question, what’s the best way of presenting/delivering content?
Left is bigger pic of activity for the course: patterns of access for the course as a whole: distinct access people added up for class
% of activity per row
timeline for each module with spikes each week (traffic) another representation including everything for a module, then you break it down
BOTTOM is goal setting activity done all at the start
Right is splitting by intents
activity by intents over time (they are the same)
Need to explain a bit
Learners more likely to participate at the beginning regardless of intents
Too many dimensions
Example of how ‘teaching’ drives activity
Whenever you have an announcement spikes of activity follow (LTTO)
Breaking down by activity type over multiple courses.
Very dangerous
No performance difference in free
Scrap this, otherwise you need the figure for 4 courses
We will edit later… weeks
This is important, highlight mapping to the new CATEI
Where do you say that it is course B?
explain
Deceiving
Motivation predicts satisfaction not the other way
Why highlight NS?
What is C? large R not sure about result
Same here
These are sig, but no highlight
Relatively large
Same swap of vars
large
Harvard business review paper
Is it supported? Then explain
2 explain more
3. Give more examples