The 10th class session agenda covered various topics related to survey data analysis in instructional design, including: types of surveys used by educators; student satisfaction surveys; pre-test/post-test analyses; competency/aptitude profiles; best practices and pitfalls of surveys; ethical standards; and common survey analysis tools. The class included breakout activities for students to discuss surveys and their use, as well as a review of qualitative and quantitative data analysis techniques.

1. onlinetechnology
EDUC5103G
in education
10th Class Session Agenda:
• Check-In
• Types of Surveys Most Used by Educators
• Break-Out 1: Student Satisfaction Surveys
• Data Analysis in Instructional Design
• Break
• Pre-Test / Post-Test Analyses
• Competency / Aptitude Profiles
• Break-Out 2: Best Practices & Pitfalls
• Ethical Standards
• Common Survey Analysis Tools
• Check-Out
• To Do List and Course Wrap-Up

2. Welcome to
Class!

3. We acknowledge the lands and people of the
Mississaugas of Scugog Island First Nation which is
covered under the Williams Treaties. Ontario Tech
University is situated on the Traditional Territory of
the Mississaugas, a branch of the greater
Anishinaabeg Nation which includes Algonquin,
Ojibway, Odawa and Pottawatomi.

4. Cape Breton University’s Department of
Education recognizes that Cape Breton Island is in
Mi’kma’ki, the ancestral and unceded territory of the
Mi’kmaq People. This territory is covered by the “Treaties
of Peace and Friendship” which Mi’kmaq and Wolastoqiyik
(Maliseet) people first signed with the British Crown in
1725. The treaties did not deal with surrender of lands and
resources but in fact recognized Mi’kmaq and
Wolastoqiyik (Maliseet) title and established the rules for
what was to be an ongoing relationship between nations.

5. checkin

6. surveyresults
in instructional design
Three types of survey data most likely to be used by
educators and instructional designers:
1. Student Satisfaction Surveys
2. Pre-Test / Post-Test Results
3. Competency / Aptitude Profiles

7. flippedclass
student satisfaction surveys
activity
Student Surveys: Using Student Voice to Improve
Teaching and Learning
https://youtu.be/cP-8OGwQRPI
6 Critical Steps in Conducting Regular Assessments of Student
Satisfaction
http://blogem.ruffalonl.com/2013/09/12/critical-steps-conducting-
regular-assessment-college-student-satisfaction/
Noel-Levitz: Using Satisfaction Data for Retention
https://youtu.be/p9Nm-OE_-t0

8. openingmindmup

9. openingmindmup

10. Group Instructions:
Watch the video / read the blog post (can be opened directly from MindMup)
Choose a group recorder and reporter (can be the same person)
As a group, use MindMup to:
• Create a summary of the key points
• How are learner satisfaction surveys being used at your institution?
o Institutional Level?
o Teacher / Classroom Level?
o Course Level?
• How would you use the video (blog post), and the key points, to inform your own practice as an
educator / instructional designer?
• Describe how you would use the video (blog post), and the key points, to frame and plan a professional
development session for teachers / trainers at your institution
• Report your summaries to the class
flippedclass
student satisfaction surveys
activity

11. mindmuplinks
Group 1:
https://drive.google.com/file/d/1E2Yf7V5mXFgmBuCI7
TZQ3vYKk1iSFmS4/view?usp=sharing
Group 2:
https://drive.google.com/file/d/1dDMcO9mvYPmL6b8v
ydggaJ7gKTwdbj4I/view?usp=sharing
Group 3:
https://drive.google.com/file/d/1pDnEEieS1G8Av1doDt
dHTu5KVCJqNVHq/view?usp=sharing

12. surveyresults
in instructional design
February 1st
March 1st

13. February 1st
March 1st

14. surveyresults
in instructional design
URGENT:
Things that NEED to be
Addressed NOW
NICE:
Things that CAN be Addressed
NOW
FUTURE:
Things that CANNOT be
Addressed NOW
NOT POSSIBLE:
Things that CANNOT, SHOULD
NOT, or DO NOT NEED to be
changed
 More clarity on
instructions / due
dates, etc
 Add video intros
and tips for
specific
assignments and
activities
 Offer learners
opportunity to
change groups and
/ or topics for
presentations
 Add a video on
using the Zotero
Tool
 Adding
gamification or
games based
learning to the
topics list
 Add a course
calendar to the
website
Utilize features of
Blackboard?
 Adding wearables,
AR and VR to topics
list
 Adding data
analytics to topics
list
 Utilize features of
Blackboard?

15. surveyresults
qualitative analysis

17. http://dfreelon.org/utils/recalfront/recal2/

18. breaktime
Please return by 7:45 pm ET

19. surveyresults
cause & effect
Correlation ≠ Causation

20. surveyresults
pre-test / post-test analysis

21. surveyresults
pre-test / post-test analysis
Maturation Effect

22. surveyresults
pre-test / post-test analysis
Comparison to a Control Group
Necessary to demonstrate a causal relationship
Experimental Design and Survey Data

23. surveyresults
pre-test / post-test analysis
What are the chances
that these changes
were random?
Less than 5%
= Statistically Significant

24. surveyresults
pre-test / post-test analysis
(mTSES2 – mTSES1) – (TSES2 – TSES1) = Net Change(Intervention Effect)
Pre-Test:
Self-Efficacy with mLearning
Post-Test:
Self-Efficacy with mLearning
Pre-Test:
Self-Efficacy with Teaching
Post-Test:
Self-Efficacy with Teaching
Setting a control within the same group is
possible – with careful instrument design
(Power, 2015)

25. surveyresults
pre-test / post-test analysis
0.19
0.31
0.01
0.57
0.68
0.11
0.57 0.68
0.10
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Student Engagement Instructional Strategies Classroom Management
Overall Trends in Self-Efficacy Scores
(Power, 2015)
TSES mTSES Net Change
(mTSES2 – mTSES1) – (TSES2 – TSES1) = Net Change(Intervention Effect)

26. surveyresults
competency profiles
Visual Auditory
Reading &
Writing
Kinesthetic
Learning Styles Preferences
IQ

27. 0.19
0.31
0.01
0.57
0.68
0.11
0.57 0.68
0.10
0
0.2
0.4
0.6
0.8
Student Engagement Instructional
Strategies
Classroom
Management
Overall Trends in Self-Efficacy
Scores
(Power, 2015)
TSES mTSES Net Change
0.51
0.64
0.49
0.64 0.62
0.45
0.12
-0.01
-0.04
-0.10
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
Student Engagement Instructional
Strategies
Classroom
Management
Overall Trends in Self-Efficacy
Scores
(ID4ML)
TSES (ID4ML) mTSES (ID4ML) Net Change (ID4ML)
More info:
https://youtu.be/JN4K8XevefM

28. flippedclass
best practices & pitfalls
activity
7 tips for good survey questions
https://youtu.be/Iq_fhTuY1hw
Confirmation Bias
https://youtu.be/B_YkdMwEO5U
How Ice Cream Kills! Correlation vs. Causation
https://youtu.be/VMUQSMFGBDo

29. Group Instructions:
Watch the Video (can be opened directly from MindMup)
Choose a group recorder and reporter (can be the same person)
As a group, use MindMup to:
• Create a summary of the key points
• How would you use the video, and the key points, to inform your own
practice as an educator / instructional designer?
• Describe how you would use the video, and the key points, to frame and plan
a professional development session for teachers / trainers at your institution
• Report your summaries to the class
flippedclass
best practices & pitfalls
activity

30. mindmuplinks
Group 1:
https://drive.google.com/file/d/1-
HUq1PTn0cBMLDCGKqzta13TADq-RdrJ/view?usp=sharing
Group 2:
https://drive.google.com/file/d/1XhQUXY9lV64ocHwJKFdJlt4KRj
PPCNYO/view?usp=sharing
Group 3:
https://drive.google.com/file/d/12UZc-RdrRZQMdWhhUua-
ZGx7gSYS2A5F/view?usp=sharing

31. https://youtu.be/9pCfq5QH_v4
surveyresults
research ethics

32. Canadian Tri-Council Policy Statement
(TPS-2)
http://www.pre.ethics.gc.ca/eng/policy-
politique/initiatives/tcps2-eptc2/Default/
UOIT Research Services
https://research.uoit.ca/faculty/ethics-
safety-research-conduct/index.php

33. surveyresults
common tools
Quantitative Tools: Statistics Package for the Social Sciences (SPSS)
http://www-01.ibm.com/software/analytics/spss/

34. surveyresults
common tools
Qualitative Tools: NVivo

35. surveyresults
common tools
Quantitative & Qualitative Tools: Excel (Spreadsheet Applications)

36. moreresources
Videos:
Confirmation Bias - https://youtu.be/B_YkdMwEO5U
How Ice Cream Kills! Correlation vs. Causation - https://youtu.be/VMUQSMFGBDo
Research Ethics - https://youtu.be/9pCfq5QH_v4
Six Critical Steps in Conducting Student Satisfaction Assessments - https://youtu.be/d14AZE61_EE
7 tips for good survey questions - https://youtu.be/Iq_fhTuY1hw
Types of Survey Sampling Techniques - https://youtu.be/rASK8PpqakM
Using the mTSES to Evaluate and Optimize mLearning Professional Development -
https://youtu.be/JN4K8XevefM
Ways to Guide Strategic Planning with Student Satisfaction Survey Results - https://youtu.be/ns7ekUDVe1Q
Ways to Impact Student Retention with Satisfaction Survey Results - https://youtu.be/HBtuYScGEcQ
Ways to Use Student Satisfaction Results for Accreditation - https://youtu.be/9aECPrIDaDw

37. moreresources
Other Resources:
Freelon, D. (2011). ReCal2: Reliability for 2 coders. Retrieved from http://dfreelon.org/utils/recalfront/recal2/
Government of Canada (2015). Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans
(TPS-2 2014). Available from http://www.pre.ethics.gc.ca/eng/policy-politique/initiatives/tcps2-eptc2/Default/
IMB (n.d.). SPSS Software. Available from http://www-01.ibm.com/software/analytics/spss/
Power, R., Cristol, D., Gimbert, B., Bartoletti, R, & Kilgore, W. (in press). Using the mTSES to evaluate and
optimize mLearning professional development. The International Review of Research in Open and Distributed
Learning.
Power, R. (2015). A framework for promoting teacher self-efficacy with mobile reusable learning objects
(Doctoral dissertation, Athabasca University). Available from http://hdl.handle.net/10791/63
QSR International (n.d.). Nvivo: The #1 software for qualitative data analysis. Available from
http://www.qsrinternational.com/product
University of Ontario Institute of Technology (2016). Ethics, Safety and Research Conduct. Available from
https://research.uoit.ca/faculty/ethics-safety-research-conduct/index.php

38. checkout

39. wrappingup
To Do List:
• Submit Assignment 4 Self-Evaluation
by 11:59 pm ET SUNDAY, NOV
28TH!
• Submit All Remaining Assignment 3:
Learning Reflections by 11:59 pm ET
SUNDAY, DEC 5TH!
• Have a great break!