Quality Matters approach to assessing online student

1. Assessing Learner
Progress
Jim Marteney

2. “Teaching
without learning
is just talking.”

3. “Teaching
without learning
is just talking.”

4. “I’d really enjoy
teaching if it
wasn’t for
grading.”

7. What do you want your
students to learn?

9. How do you know your
students have learned?

10. Traditional Testing?

11. How do you evaluate
your students?

14. What is
Quality
Matters?
A set of quality standards for
online course design
A faculty-centered peer
review process for online
courses

15. What is
Quality
Matters?
A tool used by instructional
design staff their work with
faculty
A professional development
opportunity for both faculty
and staff

16. Faculty-driven
Peer Reviewed
Review Process that is

17. Collaborative Continuous
Collegial
Centered

18. Quality Matters Rubric

19. Rubric Standards
Course Overview and Introduction
Learning Objectives
Assessment and Measurement
Instructional Materials
Learner Interaction and Engagement
Course Technology
Learner Support
Accessibility

20. Rubric Standards
Course Overview and Introduction
Learning Objectives
Assessment and Measurement
Instructional Materials
Learner Interaction and Engagement
Course Technology
Learner Support
Accessibility

21. General Standard 3
Assessment and Measurement
“Assessments are integral to the learning
process and are designed to evaluate learner
progress in achieving the stated learning
objectives or mastering the competencies.”

22. General Standard 3
Assessment and Measurement

23. The assessments measure the stated
learning objectives or competencies.
The course grading policy is stated
clearly.
Specific and descriptive criteria are
provided for the evaluation of learners’
work and are tied to the course grading
policy.
The assessment instruments selected
are sequenced, varied, and suited to the
learner work being assessed.
The course provides learners with multiple
opportunities to track their learning progress.
General Standard 3

24. General Standard 3
Assessment and Measurement

25. General Standard 3
The Key:
“Critical course
components
reinforcer one
another”
The assessments measure the stated
learning objectives or competencies.
Alignment

26. General Standard 3
The Key:
with your activities?
The assessments measure the stated
learning objectives or competencies.
Alignment
Do your assignments line up
with your objectives?
with your modules?

27. General Standard 3
“Course assessments are
consistent with the course and
module learning objectives or
competencies.”
The assessments measure the stated
learning objectives or competencies.
“From types of assessments chosen,
it is clear that learners can
successfully complete the
assessments if they have met the
unit’s learning objectives.”

28. General Standard 3
The assessments measure the stated
learning objectives or competencies.

29. General Standard 3
What are your objectives?
The assessments measure the stated
learning objectives or competencies.
To know if you have arrived, you
need to know where you’re going.

30. Creating Course Objectives
Should we add
our
module
objectives?
Do we even have
module objectives?
We have
course objectives

31. Creating Course Objectives
Standard 2
The module/unit learning objectives
describe outcomes that are measurable
and consistent with the course-level
objectives
All learning objectives are stated
clearly and written from the student’s
perspective
Instructions to students on how to
meet the learning objectives are
adequate and stated clearly.

32. Creating Course Objectives
Create a Stem
Add a
Verb
Determine the Outcome
After completing the lesson, the student will be able to . . .
After this unit, the student will have . . .
By completing the activities, the student will . . .
At the conclusion of the course/unit/study the student will . .
re-tell in his/her own words _____.
summarize the plot of _____.
identify states of matter . . .
create a timeline of events . . .
identify the causes and effects of . . .
create Venn Diagrams which compare and contrast . . .
recall information about the reading . . .
After this unit, the student will be able to create
a timeline of events leading up to the Civil War.

33. General Standard 3
The course grading policy is stated
clearly.
“A clear, written statement fully
explains how the course grades
are calculated.”
“The relationship(s) between points,
percentages, weights, and letter grades
are explained.”
“The instructor’s policy on late
submissions is clearly stated.”

34. General Standard 3
The course grading policy is stated
clearly.
Where can you post your grading policy?
Syllabus
Introduction letter
Modules
Quiz
Discussion
Announcement

35. General Standard 3
The Key:
Specific and descriptive criteria are
provided for the evaluation of learners’
work and are tied to the course grading
policy.
Rubrics

36. General Standard 3
Heidi Andrade
“A document that articulates
the expectations for an
assignment by listing the
criteria, or what counts, and
describing levels of quality from
excellent to poor.”

37. General Standard 3

39. General Standard 3

40. General Standard 3
Let’s not be this guy
Let’s create a Rubric

41. General Standard 3
Goal 1
Goal 2
Goal 3
Goal 4
Level 1 Level 2 Level 3

42. General Standard 3
Followed
format: 5
Missed 2
aspects: 3
No relation to
format: 0
Used 5
sources: 5
Used 3-4
sources: 3
Used 0-2
sources: 0
No mistakes: 5
1-5 mistakes:
3
6 or more
mistakes: 0
Format
Sources
Grammar
Stapled
Hot Medium Cold
Used staples:
2
Not use
staples: 0

43. General Standard 3
My Rubric For Discussions

44. General Standard 3
The Key:
The assessment instruments selected
are sequenced, varied, and suited to the
learner work being assessed.
Variety that works with online learning

45. General Standard 3
Are Traditional Tests
The Best Evaluative
Methods?
The assessment instruments selected
are sequenced, varied, and suited to the
learner work being assessed.

46. General Standard 3
The assessment instruments selected
are sequenced, varied, and suited to the
learner work being assessed.
“Multiple assessment strategies are used in both
the online and face-to-face settings, including
alternative assessments that require learners to
apply what they learn and to think critically.”

47. General Standard 3
The assessment instruments selected
are sequenced, varied, and suited to the
learner work being assessed.
http://cft.vanderbilt.edu/guides-sub-pages/writing-
good-multiple-choice-test-questions/
Vanderbilt University
Center for Teaching
Writing a Multiple Choice Test

48. General Standard 3
The assessment instruments selected
are sequenced, varied, and suited to the
learner work being assessed.
Writing a Multiple Choice Test
A multiple choice item consists of a problem, known
as the stem, and a list of suggested solutions, known
as alternatives. The alternatives consist of one correct
or best alternative, which is the answer, and incorrect
or inferior alternatives, known as distractors.

49. General Standard 3
The assessment instruments selected
are sequenced, varied, and suited to the
learner work being assessed.
The stem should be meaningful by itself and should
present a definite problem. A stem that presents a
definite problem allows a focus on the learning
outcome.
Writing a Multiple Choice Test

51. The assessment instruments selected
are sequenced, varied, and appropriate to
the learner work being assessed.
General Standard 3
“In traditional assessments, such as those that use
multiple choice, true-false, or matching, learners
are asked to select a response from different
options, and tests often are self-scoring.”

52. The assessment instruments selected
are sequenced, varied, and appropriate to
the learner work being assessed.
General Standard 3
“In alternative assessments, also commonly called
performance or authentic assessments, learners
are asked to develop an answer in response to a
prompt or stimulus, which is graded by the
instructor. Such assessments may include
interviews, journals, portfolios, observations,
demonstrations, performance tasks, and exhibits.”

53. The assessment instruments selected
are sequenced, varied, and appropriate to
the learner work being assessed.
General Standard 3
“Assessments are varied in order to provide
multiple ways for learners to demonstrate mastery,
and to accommodate diverse learners.”

54. The assessment instruments selected
are sequenced, varied, and appropriate to
the learner work being assessed.
General Standard 3
“The assessments are sequenced so as to promote
the learning process and to build on previously
mastered knowledge and skills gained in this
course and prerequisite courses. Assessments are
paced to give learners adequate time to achieve
mastery and complete the work in a thoughtful
manner.”
Scaffolding

55. Using Online Technologies
Beyond Traditional Tests
What do You Use to
Evaluate Your Learners?
What have you ever
thought of trying?

56. Using Online Technologies
Beyond Traditional Tests
Discussions: more than Participation
Peer to Peer Evaluation
Journaling
Individual Projects
Group Projects
Sharing Homework with peers vs. just the
instructor

57. Using Online Technologies
Beyond Traditional Tests
CATS in tests and quizzes
??
Feedback from testing
Creating content like videos/powerpoints
Moderating their own discussion

58. The course provides learners with multiple
opportunities to track their own learning
progress.
General Standard 3
“Learning is more effective if learners receive
frequent, substantive, and timely feedback. The
feedback may come from the instructor directly,
from assignments and assessments that have
feedback built into them, or even from other
learners.”

59. The course provides learners with multiple
opportunities to track their own learning
progress.
General Standard 3
“Look for examples of self-check quizzes and
activities, as well as other types of practice
opportunities that provide timely feedback. Such
assignments may be voluntary and/or allow multiple
attempts.”

60. The assessments measure the stated
learning objectives or competencies.
The course grading policy is stated
clearly.
Specific and descriptive criteria are
provided for the evaluation of learners’
work and are tied to the course grading
policy.
The assessment instruments selected
are sequenced, varied, and suited to the
learner work being assessed.
The course provides learners with multiple
opportunities to track their learning progress.
General Standard 3

61. Thank you
Now Go Forth and Assess