1. Informational Webinar:
Quality Online Learning and Teaching
(QOLT)
Brett Christie, Ph.D., CSU Academic Technology Services
Li Wang, Ph.D., CSU Northridge
Dora Preminger, Ph.D., CSU Northridge

2. BEST USE OF THE WEBINAR
• If you haven’t done so already, enter your
name, campus, and role in the chat box.
• QOLT will be described in as much detail with as much clarity
as possible. Visuals are included and all information is available
at the QOLT site.
• Feel free to enter comments/questions into the chat box. We
will screen those and address all possible.
• Should questions arise after the session, contact Brett Christie
(qolt@cdl.edu).
• Session is being recorded for review or referral at QOLT site.

4. AGENDA
• Introductions & Acknowledgments (5 minutes)
• Brett Christie, CSU Faculty Development Liaison, QOLT Program Director
• Li Wang, CSUN Instructional Designer, Quality Matters, Research Colleague, QOLT Associate
• Kathy Fernandes & John Whitmer, CSU LMSS leadership
• Overview of QOLT (10 minutes)
• How we got here
• Purpose; Brief review of 2011-2012 pilot
• 2012-2013 Project timeline
• A closer look at the QOLT instrument and process (20 minutes)
• Instructor Self-review
• Student Report
• Campus Coordinator review/verification role
• Discussion (20 minutes)
• Closure and feedback (5 minutes)

5. QOLT PURPOSE
• Create a useful evaluation tool that can help faculty
(re)develop quality hybrid/online courses.
• Identify exemplary practices for teaching and learning
through hybrid/online courses.
• Inform faculty development activities and programs
related to hybrid/online teaching.
• Recognize faculty, programs, and campuses that are
creating quality online courses. And, share!

6. MODELS EXAMINED
• Rubric for Online Instruction: Designed to assist development and
evaluation of online courses while promoting dialog about the nature of
student learning.
• Quality Matters: Faculty-centered, peer-review process designed to certify
quality of hybrid/online courses.
• Quality Online Course Initiative: Course rubric and evaluation system
developed in Illinois to improve online accountability.
• Course Review Toolkit:An instrument and a system, developed at CSU
Northridge, for informing and assessing Face-to-Face, Hybrid, and Online,
teaching and learning.
• Online Course Evaluation Program: Developed by the Monterey Institute for
Technology and Education, OCEP is a criteria-based evaluation tool to
assess and compare the quality of online courses.
• eLearning Best Practices Rubric @ Sacramento State
• Various research and publications on effective teaching, learning, design,
assessment, syllabi, etc.

7. X

8. X
X

9. RESULTING QOLT EVALUATION INSTRUMENT
1. Course Overview and Introduction (8)
2. Assessment and Evaluation of Student Learning (6)
3. Instructional Materials and Resources Utilized (6)
4. Student Interaction and Community (8)
5. Facilitation and Instruction (8)
6. Technology for Teaching and Learning (6)
7. Learner Support and Resources (3)
8. Accessibility and Universal Design (4)
9. Course Summary and Wrap-up (3)

10. • Participants should not enter into lightly
• Participation time estimated at 1+ hour
• Course evaluation, not simple survey
• Process is about analysis and reflection
• Pay-off is informed course redesign
• Validation of teaching effectiveness effort

11. QOLT SCORING PER OBJECTIVE
There is also an open-ended box per section for descriptive.

15. Section 1. Course Overview and Introduction
Instructor gives a thorough description of the
course, as well as introducing students to the
course protocol and expectations.

17. 1.B. ‘COURSE DESCRIPTION’ BASELINE
§ There should be specific meaning to the course,
section, instructor, and students, not just the
minimal course description that appears in the
catalog.
"Directed conversation in Spanish for elementary-level
students. Includes individual and class assignments in
laboratory. May be repeated for credit. Admission by
consent of instructor."

18. 1.B. ‘COURSE DESCRIPTION’ EXEMPLARY

21. 1.3 ‘INSTRUCTOR INFORMATION’ EXEMPLARY

23. 1.D. ‘ONLINE ETIQUETTE’ SAMPLE
§ Do not dominate any discussion.
§ Do not use offensive language.
§ Never make fun of someone’s ability to read or write.
§ Use simple English.
§ Use correct spelling and grammar.
§ Share tips with other students.
§ Keep an “open-mind” and be willing to express even your minority
opinion.
§ Be aware of the University’s Academic Honesty Policy.
§ Think before you push the “Send” button.
§ Do not hesitate to ask for feedback.
§ When in doubt, always check with your instructor for clarification.
Mintu-Wimsatt, A. (2010). Netiquette: Make it part of your syllabus. Journal of Online Learning and
Teaching, 6(1),

25. 1.F. ‘TECHNICAL COMPETENCIES’ SAMPLE
§ Connect to the Web using a web browser.
§ Navigate around the Web and use search engines.
§ Send and receive e-mail with attachments.
§ Basic word processing, including cutting and pasting.
§ Open, save, and manage files.
§ Organize folders and files (create, name, rename, move).
§ Compress files and folders.
*Depends on course level and proportion of online mode

26. =
20

27. SECTION 1, RUBRIC:
“COURSE OVERVIEW AND INTRODUCTION”

28. SECTION 2:
ASSESSMENT OF STUDENT LEARNING
Student Evaluation and Assessment refers to the
process used to gather evidence of the
achievement of the Student Learning
Objectives/Outcomes (SLOs).
We strongly recommend that instructors contact
the Office of Academic Assessment (or similar) for
assistance and information about this section.

29. Objectives Example
2.a All Student Learning Objectives/Outcomes (SLOs) are If the mandated course level objectives are not measurable, then
specific, well-defined, and measureable. module level objectives should be measurable and support course
level objectives.
2.b The grading policy is clearly stated for the course and Instructor provides late submission policy and scale, weights of
individual assignments. respective assignments, and the corresponding letter grade if scores
are accumulated at the end.
2.c The learning activities (including the assignments and Instructors explain how learning activities such as assignments,
ungraded activities) promote the achievement of the SLOs. discussions contribute to the achievement of the stated SLOs.
2.d The assessment instruments selected are sequenced, There are multiple ways to demonstrate mastery-e.g., project, paper,
varied, and appropriate to the student work being assessed. tests. One is built upon the other tool.
2.e Throughout the semester Instructor provides multiple Activities may include but not limited to blogs for reflection, peer
opportunities to give feedback on students learning (strengths review, practice test and draft of term paper, module summary.
and weaknesses) and to “self-check” students learning/progress.
2.f Throughout the semester, instructor provides multiple Instructor may consider the use of surveys, discussion forums, or
opportunities to solicit feedback from their students about their item analyses to collect feedback or attitudinal data (that goes
learning and on the course for the improvement of the course. beyond student learning outcomes) on the effectiveness or difficulty
of the resources and activities (e.g., “Muddiest Point”), or item
analysis of test questions in order to improve the course in the future.

30. Objectives Example
2.a All Student Learning Objectives/Outcomes (SLOs) are If the mandated course level objectives are not measurable, then
specific, well-defined, and measureable. module level objectives should be measurable and support course
level objectives.
2.b The grading policy is clearly stated for the course and Instructor provides late submission policy and scale, weights of
individual assignments. respective assignments, and the corresponding letter grade if
scores are accumulated at the end.
2.c The learning activities (including the assignments and Instructors explain how learning activities such as assignments,
ungraded activities) promote the achievement of the SLOs. discussions contribute to the achievement of the stated SLOs.
2.d The assessment instruments selected are sequenced, There are multiple ways to demonstrate mastery-e.g., project,
varied, and appropriate to the student work being assessed. paper, tests. One is built upon the other tool.
2.e Throughout the semester Instructor provides multiple Activities may include but not limited to blogs for reflection, peer
opportunities to give feedback on students learning (strengths review, practice test and draft of term paper, module summary.
and weaknesses) and to “self-check” students learning/
progress.
2.f Throughout the semester, instructor provides multiple Instructor may consider the use of surveys, discussion forums, or
opportunities to solicit feedback from their students about item analyses to collect feedback or attitudinal data (that goes
their learning and on the course for the improvement of the beyond student learning outcomes) on the effectiveness or
course. difficulty of the resources and activities (e.g., “Muddiest Point”), or
item analysis of test questions in order to improve the course in
the future.

31. Graphically Representing SLOs (hidden)

32. Graphically Representing SLOs
(expanded)

34. Sample Online Assignment Rubrics
• Online Discussions
• Student/Group Wikis
• Student Blogs
• Twitter
• PowerPoint or Enhanced PodCast
• ePortfolio
• Video Production
http://tinyurl.com/7vbvag2

36. Section 3.
Instructional Materials and
Resources Utilized
The instructor has carefully selected a variety
of materials and material formats to represent
course content and enable students to
meet relevant learning outcomes.

37. Objectives Example
3.a Instructor provides students with adequate Instructor includes instruction in the syllabus or elsewhere in the course as
time and notice to acquire course materials. to acquire course materials including textbooks, and other types of
external resources.
3.b Syllabus lists whether textbooks and Instructor separates the materials and labels them as either required or
materials are required or recommended. recommended.
3.c Instructor articulates the purpose of all For required and recommended materials, there are brief statements as to
materials as to how they are related to the the value/purpose in meeting student learning objectives/outcome(s). If
course and module learning objectives. external links/websites are used, the links should be self-evident or a short
description of the specific link needs to be provided instead of posting a
general link for students to explore.
3.d When possible, instructor provides s Course materials include both the Open Educational Resources (e.g.
options in terms of how students acquire MERLOT) and external materials
course materials, including Open Educational
Resources (e.g., MERLOT).
3.e There is a variety of instructional material Materials types include PowerPoint, videos, text. Multiple perspectives
types and perspectives, while not overly relying refer to different opinions from scholars in the field.
on one content type such as text.
3.f All resources and materials used in the These resources and materials include text, images, tables, videos, audio,
course are appropriately cited. and website.

38. Section 4.
Student Interaction and Community
(Course Design)
Addresses how the instructors provide opportunities
for students to interact with the content,
peers, instructors and the LMS and promote
students to become active learners and
build the online community.

40. Section 5. Facilitation and Instruction
(Course Delivery)
Instructor facilitates the course and communicates
with students frequently and engages
them to be active learners.
Instructor actions reinforce the development of a
sense of community among course participants.

42. Section 6.
Technology for Teaching and Learning
Instructor utilizes technology to effectively deliver
course content, engage students in learning
activities (individual, student-to-student, instructor-
to-student) and for students to express themselves
or demonstrate learning.

44. Section 7.
Learner Support and Resources
Learner Support and Resources refers to
program, academic, and/or technical
resources available to learners.

46. SECTION 8
ACCESSIBILITY AND UNIVERSAL DESIGN
The course utilizes principles of accessibility and
universal design that are critical to some learners,
as well as offering benefits to all learners.
We strongly recommend that instructors contact
the Universal Design Center (or similar) for
assistance and information about this section.

48. Students with Disabilities
Upon identifying themselves to the instructor and
the university, students with disabilities will
receive reasonable accommodation for learning
and evaluation. For more information, contact
Services to Students with Disabilities in Building,
Room (x4232).

49. Universal Design Statement
“As your instructor, I feel I have a responsibility to do
everything within reason to actively support a wide range
of learning styles and abilities. As such, I have applied
the principles of Universal Design for Learning to this
course. Feel free to discuss your progress in the course
with me at any time. In addition, if you require an
accommodation, submit your verified accommodations
form to me during the first two weeks of the course.”

50. Section 9
Course Summary and Wrap-up
The course gives students an opportunity to
summarize the semester, and establish the
connection with other courses, and prepares
students to start the next phase of their journey.

52. Questions Regarding Faculty
Instrument and Process?

53. Gathering Student Ratings
• Online evaluations due December 15, 2012
• Minimal demographic information
• Confidential (coordinator and instructor only)
• Ratings submitted directly to QOLT Central
• Process will be streamlined via Likert items
o Optional open-ended at end of each section
• Results used toward determining most
exemplary per campus and across system.

54. Purpose of Student Ratings
1. Identify any gaps between instructor’s
perception and actual student experience.
2. Make changes to courses based on evidence
3. Use the data to receive administrative support
to teach online or hybrid

56. Student
ratings
from
the
Quality
Online
Learning
and
Teaching
(QOLT)
survey
Dora
Preminger
Dept.
Physics
&
Astronomy
•
Li
Wang
Faculty
Technology
Center
•
California
State
University
Northridge

57. The
course
Physics
220A
–
Classical
Mechanics
Calculus-­‐based
(Engineers)
Why
use
the
QOLT
survey?
• New
hybrid
format
Is
the
course
well-­‐designed?
(Survey
makes
QOLT
rubric
• Is
it
viable?
concrete)
Perceptions
of
students
• Improve
for
future?
Weak/missing
elements?

58. Sample
Results:
Instructional
Materials
and
Resources
The
instructor
uses
a
variety
of
All
materials
are
meaningful
to
me
as
instruc9onal
material
types
and
presents
to
how
they
are
related
to
helping
me
mul9ple
perspec9ves,
while
not
overly
achieve
the
course
learning
goals
and
relying
on
one
content
type
such
as
text.
objec9ves.
Some
Student
comments:
• There
are
way
too
many
course
materials
for
this
course...more
so
than
the
non-­‐hybrid
counterpart
which
would
already
have
been
borderline
too
expensive.
• The
Ac9ve
physics
assignments
are
very
confusing.
I
was
not
able
to
understand
how
to
properly
use
it
9ll
the
instructor
told
me
that
should
use
the
different
browser
to
use
it
properly.

59. Sample
Results:
Students
Interaction
and
Community
At
the
beginning
of
the
course,
GeBng
I
find
the
naviga9on
throughout
the
online
to
know
other
course
par9cipants
components
of
the
course
is
reasonable,
gave
me
a
sense
of
belonging
in
the
intui9ve
and
straighEorward.
course.
The
learning
ac9vi9es
encourage
me
to
interact
with
the
instructor,
my
peers,
and
the
course
content
frequently.

60. Sample
Results:
Overall
Satisfaction
Please
rate
your
overall
course
Would
you
recommend
this
course
learning
experience.
to
your
friend?
Least………………….Most
sa9sfied
No……………………….Yes,
definitely!

61. Summary
Survey
results:
• Nice
breakdown
of
the
different
elements
of
the
course
–
helps
me
see
how
I
am
fiBng
them
together
(or
not!)
• Easy,
quick
view
of
statistics
• Hear
student
voices
• View
of
overall
student
experience
• Pinpoint
areas
for
course
improvement

62. X
X

63. AFTER JANUARY 20
• Each Campus Coordinator is given their data
set within 1 week
• They take 3 weeks to review the self-evaluations and
respective student rating
• With permission, may use sysadmin course access
• They rank-order Top 5 and submit with their comments to
QOLT Review Panel via survey
• QOLT Review Panel begins with #1 from each campus as
CSU “Top 15.” Reviews #2s next to see if any elevate.
Followed by #3s, etc. Top 20 result.
• Review panel "triangulates" between faculty, student, and
coordinator data.

64. QOLT REVIEW PANEL (2011-2012)

65. Recognition Process, After March 15
• All faculty who submit a full self-evaluation receive a
letter of participation from Campus Coordinator
(draft provided to Campus Coordinator)
• Top 5 designated from each campus receive an
additional letter of recognition from CSU ATS.
• Top 20 systemwide receive additional letter of
recognition from CSU ATS and are featured at
website and via QOLT dissemination (e.g., ITL
newsletter; spring webinar; CATS)

66. Featuring Your QOLT Faculty

67. Further information on QOLT, CFP, and Evaluation Instruments
qolt.ecatalst.org
Contact Brett Christie
qolt@cdl.edu
707.486.6519