A powerful feature of online instruction is the ability to embed assessment throughout and capture data on how students are learning. The TAACCCT program calls on institutions to use such data to “continuously assess the effectiveness of their strategies in order to improve their program… and build evidence about effective practice.” In this interactive session, we will share the OLI approach to data driven continuous improvement. Together we will discuss strategies for using learning data to refine course materials by examining examples from our project. We will also present an overview of the learning data and tools available to co-development and Platform+ participants.

1. Data Driven Continuous
Improvement
John Rinderle @JohnRinderle
Norman Bier @NormanBier

2. Outcomes for Today
By the end of this session you will be able to…
• Explain the value of a data driven approach.
• Implement design strategies that facilitate
meaningful data capture and use.
• Avoid commonly pitfalls in the use of
learning data.
• Share strategies for data driven improvement
with your project team.
• Explain the data collected and analysis tools
available to co-development and platform+.

3. Infinite Points of Light

4. Infinite Points of Light

5. Infinite Points of Light

6. Infinite Points of Light

7. Infinite Proliferation
The 4 R’s
Reuse
Redistribute
Revise
Remix

8. Infinite Proliferation
The 4 R’s NOT:
Reuse Recreate
Redistribute
Revise
Remix Add:
Evaluate

9. The problems of variety
• Quality is highly variable
• Much duplication of effort
• Difficult to choose appropriately
• Hard to evaluate
• Impossible to improve
• Hard to scale success up

10. Effectiveness
Can be hit or miss

11. Effectiveness
Demonstrably support students in meeting
articulated, measurable learning outcomes in a
given set of contexts

12. How do we design for effectiveness?

13. The Course Design Triangle
Objectives
Descriptions of what students
should be able to do at the
end of the course
Assessments
Tasks that provide
feedback on students’
knowledge and skills
Instructional Activities
Contexts and activities that foster
13
students’ active engagement in
learning

14. Why Focus on Objectives?
1. They communicate our intentions clearly to students
and to colleagues.
2. They provide a framework for selecting and
organizing course content.
3. They guide in decisions about assessment and
evaluation methods.
4. They provide a framework for selecting appropriate
teaching and learning activities.
5. They give students information for directing their
learning efforts and monitoring their own progress.
14
Based on A.H. Miller (1987), Course Design for University Lecturers. New York: Nichols Publishing.
Also see, C.I. Davidson & S. A. Ambrose (1994), The New Professor’s Handbook: A Guide to Teaching and Research in Engineering and Sciences.
Bolton, MA: Anker Publishing Company Inc.

15. Why a “learner-centered”
approach?
Learning results from what the student does and
thinks and only from what the student does
and thinks. The teacher can advance learning
only by influencing what the student does to
learn (Herb Simon, 2001).
It’s not teaching that causes learning. Attempts
by the learner to perform cause learning,
dependent upon the quality of feedback and
opportunities to use it (Grant Wiggins, 1993).

16. Learning Dashboard

17. Let’s Analyze Some Examples
Checklist: Is the objective…?
• Student centered (i.e., student should be able to…)
• Broken down into component skills (grain size)
• Phrased with an action verb
• Measurable
Here are some samples:
• Understand the U.S. stock market
• Recognize logical flaws in a written argument
• Appreciate the historical context of the 1940’s
17
• Apply Newton’s Second Law appropriately

18. Scenario #1: Early Enrollment vs.
Performance
Success by Enrollment Date
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Days Before Semester Start

19. Data needs to be actionable!
19

20. Scenario #2: Psychology Pilot

21. “liking” ≠ learning
21

22. Types of Learning Data
Program/Degree
Demographic
Contextual
What gets captured
Behavioral
Interaction Semantic Actions
Raw

23. Learning Curve Analysis
DataShop: Pittsburgh Science of Learning Center

24. Other Learning Curves
learnig
DataShop: Pittsburgh Science of Learning Center

25. 1.2
1
0.8
0.6
0.4
Activites 1st Try Correct
0.2 Activities Eventually Correct
Assessment Correct
0

26. Other Metrics
Engaged Didn’t Engage
College Pass Fail Pass Fail
CFA 87% 13% 36% 64%
CIT 98% 2% 81% 19%
CMU 100% 0% 67% 33%
HSS 98% 1% 58% 42%
MCS 96% 4% 75% 25%
SCS 99% 1% 83% 17%
TSB 98% 2% 69% 31%
Total 96% 4% 67% 33%

27. A Virtuous Cycle
Educational
Technology Data
& Practice
Theory

28. Share Alike and Share Data