Slides by Barry Dahl, Excellence in e-Education for ITC eLearning 2012 Conference in Long Beach

1. “Quality”
in Online Education

2. Barry Dahl dot com

4. Quality
Definition?
high grade; superiority; excellence

5. Quality
in Manufacturing
Definition?
What does ISO say? International
Organization for Standardization

6. ISO 8402:1986
• This ISO standard defines quality
as “the totality of features and
characteristics of a product or
service that bears its ability to
satisfy stated or implied needs.”
Does that actually
mean anything?

7. Another Definition
• In manufacturing, a measure of
excellence or a state of being free from
defects, deficiencies, and significant
variations, brought about by the strict
and consistent adherence to measurable
and verifiable standards to achieve
uniformity of output that satisfies
specific customer or user requirements.
Just like education, right?

8. Quality
in Education
Definition?
High standards, consistently applied,
with efficiency or “value for money”

9. Quality
in Education
Definition?
Or, what D. Green said in 1994
What is quality in higher education?
Buckingham, U.K.: SRHE and The Open University Press.

10. What is quality in higher education?
• …quality is a relative concept, ....different
interest groups or ‘stakeholders’ in higher
education have different priorities and their
focus of attention may be different. The best
that can be achieved is to define as clearly as
possible the criteria that each stakeholder
uses when judging quality, and for these
competing views to be taken into account
when assessments of quality are undertaken
(p. 17).

11. Quality
in Online Ed
Definition?

12. E-Learning Quality =
Quality Matters™
Right?

13. Through the use of rubrics and standards
related to the quality of online courses
(i.e. Quality Matters™), we are
sufficiently addressing the questions
about e-learning quality
60% 1.Real ity
40% 2.Myth

14. Quality Matters is Sufficient
Learning
• Um, no, it isn’t!!
• Quality Matters
looks at the quality
of course design.
Quality
• That’s good, but it’s Concerns
only one leg holding
up the stool. Teaching Design

15. 3 Major Components of e-Quality
Learning Level
Is High
Learning Assessment
Teaching Level Course Design
Is High Meets Standards

16. Possible Reasons?
Learning Level
Is Low
Learning Assessment
Teaching Level Course Design
Is High Meets Standards

17. Is This a Problem?
Learning Level
Is High
Learning Assessment
Teaching Level Course Design
Is High Below Standard

18. Independent Study, Perhaps?
Learning Level
Is High
Learning Assessment
Teaching Level Course Design
Is Low Meets Standards

19. Beautiful, Just Beautiful
Learning Level
Is Low
Learning Assessment
Teaching Level Course Design
Is Low Meets Standards

20. Maybe, but unlikely
Learning Level
Is Low
Learning Assessment
Teaching Level Course Design
Is High Below Standard

21. Triple Ick – Just Start Over?
Learning Level
Is Low
Learning Assessment
Teaching Level Course Design
Is Low Below Standard

22. Quality
We know it when
we see it

23. Expectations about Completion

24. Weekly News Articles

25. CCRC Reports: Washington & Virginia
http://ccrc.tc.columbia.edu/Publication.asp?UID=872

26. CCRC Reports: Washington & Virginia
Washington State community and technical colleges in the fall of 2004.
Students were tracked for nearly five years, until the spring of 2009
“students were more likely to fail or withdraw from
online courses than from face-to-face courses”
“were slightly but significantly less likely to attain an
educational award or transfer to a four-year institution”
http://ccrc.tc.columbia.edu/Publication.asp?UID=872

27. Say What?
• “slightly but significantly less likely”
• Translation … “I’m really, really sure
that there’s a very small difference.”

28. CCRC Makes Big Headlines
http://californiawatch.org/dailyreport/online-community-
college-students-more-likely-fail-withdraw-11581

29. CCRC Makes Big Headlines
http://www.insidehighered.com/news/2011/07/19/study_finds_higher_dr
opout_rates_for_community_college_students_who_take_online_courses

30. What are Reasonable
Expectations for Completion
Rates for Online Learners?
And how do you know?
Is completion a
measure of quality?

31. Online Student Satisfaction
Is satisfaction a
measure of quality?

32. Option 1 - PSOL

33. Why PSOL?

34. PSOL Basics
There are 72 questions that comprise the PSOL.
Average completion time is 15 minutes.
26 Priorities statements (can add 10)
7 Information sources about school/program
11 Factors to enroll in the program
3 Overall satisfaction questions
14 Demographics questions (can add 1)
NOTE: questions are answered on a 7-point Likert scale, where 7 is high.
34

35. The Importance of Importance

36. That’s all good,
but…….

37. Are online students, as a
group, more satisfied than
the on-campus students?

38. NOTE: SSI is the Noel-Levitz Survey for on-ground learners, PSOL is the Noel-
38
Levitz survey for online learners.

39. NOTE: SSI is the Noel-Levitz Survey for on-ground learners, PSOL is the Noel-
39
Levitz survey for online learners.

40. What drives
their level of
satisfaction?

41. How Well Do You Know Your Students?
What do They Expect from
Online Learning?

42. 2009 PSOL – Summary Statement
So far, how has your college experience met your expectations?
7=Much better than expected
Overall score
6=Quite a bit better than I expected 5.0
5=Better than I expected
4=About what I expected
3=Worse than I expected
2=Quite a bit worse than I expected
1=Much worse than expected
0% 5% 10% 15% 20% 25% 30%

43. But seriously…

44. You Already Have Expectations
• The questions are …
– How clearly have they been communicated?
– How easy are they for the following to find?
•Students?
•Faculty? <= We’ll focus here
•Staff and Administration?

45. A Common Scenario
• “What do you mean you expect
interaction in online classes?”
• “I’ve never heard that before.”
This came from a 10-year veteran of
online teaching at the college.

46. Course Design Rubrics Can Help
V. LEARNER INTERACTION
General Review Standard: The effective design of instructor-learner interaction and
meaningful learner cooperation is essential to learner motivation, intellectual commitment,
and personal development. (From LSC Course Design Rubric)
Specific Review Standards Points
V.1 The course design provides learning activities to
foster instructor-student, content-student, and if 3
appropriate, student-student interaction.
V.2 The student requirements for course interaction
are clearly articulated. 3
V.3 Clear standards are set for instructor response
and availability (turn-around time for email, grades 2
posted, etc.)

47. Course Design Rubrics Can Help
• But they only go so far.
• Did the interactions actually occur?
• Did the feedback actually happen in
a timely manner?
• The design rubric is only part of the
answer.

48. Do You Evaluate Teaching Quality?
• If not, why not?
• If so, what do you base the
evaluation on?
• How clear are the expectations of
the factors upon which they will be
evaluated?

49. Expectations for (of) Faculty
Links to these pages at:
http://xlents.com

50. St. Petersburg College
• Examples of posted expectations for faculty:
• Respond to email within 24 hours, 5 of the 7
days a week at the instructor’s discretion.
• Exam marked within 7 days of due date.
• Assignment marked within 7 days of due date
• Routine updates to all students every 7 days.

51. Lawrence Tech University
• Examples of posted expectations for faculty:
• Check the discussion forum daily and be sure
to post responses to student contributions at
least four times weekly.

52. Penn State
• Examples of posted expectations for faculty:
• The instructor is asked to grade and submit
to students all digitally formatted
assignments and exams within two business
days of receipt.

53. Colorado CC Online
• Examples of posted expectations for faculty:
• As mandated by our NCA accreditation,
CCCOnline courses cannot be self-paced.
Therefore, CCCOnline requires faculty to
create a Course Schedule that provides unit,
week, and specific activity dates.
• Instructor responds individually to all student
introductory posts.

54. Lake Superior
• Examples of posted expectations for faculty:
• Course Outlines: There is only one official course
outline for each course title offered at LSC. This
means the following:
– Delivery method is not generally considered during the
curriculum approval process.
– Each class syllabus should include the official course
outcomes regardless of the delivery method employed.
– Each class syllabus should include the official course
description regardless of the delivery method employed.

55. College expectations of online faculty
• Example - Final Exams:
• Sample of possible topics: – Online faculty are
expected to give final
– Course design exams (or due dates for
other types of final
– Interaction w/students projects) during the final
– Online office hours exam period as
established and posted by
– Feedback/response time the college.
– Final exams – The final exam period
typically consists of four
– Proctored exams weekdays and may or may
– Due dates/times not include a weekend.
This schedule is posted on
– Sick/Personal leave the college website and
otherwise makes the
information available to
all faculty.

56. Comprehensive
Faculty Evaluations

57. Evaluation Components
1. Self Evaluation
2. Professional Development Plan
3. Course Evaluations (by students)
4. Course Observation (by admin)

58. Course Evaluations

59. Questions without Answers
• Optional or required?
• Formative or Summative?
• Anonymous?
• Incentives?
• Short or long? (Brief or comprehensive?)
• Comparable to on-campus evals?

60. Optional or Required?
• Better practice: Required to opt-out if
not interested/willing.
• Through conditional release – force
students to view the evaluation form –
with ability to opt-out if desired.
• Good practice: Purely optional (expect
low response rates)

61. Formative or Summative?
• Formative eval – mid-term assessment
– Students provide feedback to instructor
• Summative eval – end-of-term
– Students provide feedback to faculty & admin
• Better practice: Use both types

62. Anonymous?
• Difficult for formative evals to be
anonymous.
• Important for students to believe that
summative evals are anonymous – but
many still won’t believe it.
• Good practice: Do not use the course
shell to give the eval if it is intended to
be anonymous.

63. Incentives?
• Good practice: impress upon students
that their participation is an important
part of the quality improvement process
• Major issue: any kind of tangible
incentive eliminates the possibility of
true anonymity, or even the appearance
thereof.

64. Short or Long?
• An evaluation form that is longer than
10-15 questions will usually result in low
response rates.
• Good practice: 10 important questions.
• If students are allowed to opt-out, they
are more likely to opt-in if they see that
the survey is very short and their time
invested will be very small.

65. Comparable to On-Campus Evals?
• How is this valuable?
• What is your purpose for asking the
same questions?
• Good practice: gather data that are
valuable to the online teaching and
learning experience – whether or not
they are comparable to on-campus data.

66. A GPA-type Rating Scale
• 4 – Strongly agree
• 3 – Agree
• 2 – Neutral
• 1 – Disagree
• 0 – Strongly disagree
• N/A – Choose not to answer, Don’t
know, or Not applicable

67. Sample Statements
• The instructor has created a course
layout that is easy to navigate.
• This course challenged me intellectually.
• The instructor provided feedback in a
timely manner.
• The feedback received on my
coursework was helpful.

68. More Sample Statements
• There was agreement between the
posted course objectives (in syllabus)
and what was taught in the course.
• Policies for determining grades in this
course were clearly explained.
• Overall, this instructor has created a
valuable learning experience for
students.

69. Eval Statements Tied to Rubric?
• Instructor presence?
• Expectations for interaction?
• Effective use of media?
• Positive learning environment?
• Clear written communication?

70. Eval Statements Tied to Other?
• PSOL Data?
–Student assignments are clearly
defined in the syllabus
–The frequency of student and
instructor interactions is adequate.
–Faculty are responsive to student
needs.
• NSSE or CCSSE?

71. Classroom Observation
An Administrator’s
Right?

73. Basic Expectations – 1A

74. Basic Expectations – 1A

75. Basic Expectations – 1A

76. Basic Expectations – 1C

77. Basic Expectations – 1D

78. Basic Expectations – 1D

79. Learning Experiences – 2A

80. Learning Experiences – 2B

81. Activity & Interactivity – 3E

82. Evaluations, Assessments – 4B

83. Where is the Value in eLearning?
• Why is XXCC engaged in eLearning?
• What are the benefits to the school?
• What are the driving factors?

84. To Make a Difference

85. “Quality”
in Online Education
Discussion
Time