comparison of rubrics qualities to the qualities of dogs

1. Good Rubric, Bad Rubric
What’s the Difference?
Kathleen Norris, MFA, EdD
Plymouth State University

2. Good, or Emerging?
Rubrics are a phenomenon of the profession of
education.
They have been embraced by some educators,
tolerated by others, and loathed by some.
Some educators are afraid of them, or don’t
understand them.
In many ways, rubrics are just like dogs…

3. Which ones are good?

4. Good Dogs
 Do what you want them to do.
 Are pleasing to look at, are not too big or too
small for their breed or their bones; they fit
their frame.
 Are connected to you, are reliable.
 Everybody experiences them as good dogs;
there is agreement on their “goodness.”
 No matter how good a dog might be, a
person who’s afraid of dogs is afraid of all
dogs and will stay as far away as possible.

5. Good Dogs

6. Good Rubrics
 Do what you want them to do, assess what
you want to assess.
 Are pleasing to look at, are easy on the
eyes, are clear to everyone, are not too big
or small for their purpose.
 Are reliable, valid, fair, and completely
connected to what you are assessing.
 Everybody understands the same thing when
they read one, but they can still be scary.

7. Emerging Dog

8. Emerging Dog
 Does things that he wants to do, not
necessarily what I want him to do; might
wander off on his own.
 Could be in a bit better shape, is not quite
the right weight for his frame.
 Is connected sometimes, but is not
completely reliable.
 May seem “good” to some people but not so
good, or even scary, to others, especially
when he jumps up without warning.

9. Emerging Rubrics
 Assess something, but not necessarily what
you want them to assess.
 Could be clearer, easier on the eyes; might
need to be smaller or larger.
 Are not completely connected to the
standards, the performance (task), or both,
so are not reliable, valid, or fair.
 Not everyone understands them the same
way and they can be very scary.

10. Student Voices; What’s Worst
 "I'm kind of getting tired of rubrics because
we even have a rubric at lunch which is
rediculess [sic].”
 "Some rubrics don't fully cover all the
situations.”
 "Most of the time it's the language. It can be
hard to understand, especially the CAS2
(district test) rubric.”
Reforming Middle Schools and School Systems: What students say about rubrics.
http://www.middleweb.com/CSLB2CubRub.html

11. Authentic Assessment
According to Grant Wiggins,
“Assessment is authentic when we
directly examine student
performance on worthy intellectual
tasks.”

12. Traditional Assessment
Again from Grant Wiggins:
“Traditional assessment…relies on
indirect or proxy 'items'--efficient,
simplistic substitutes from which we
think valid inferences can be made
about the student's performance….”

13. Authentic Assessment
“A move toward more authentic tasks and
outcomes thus improves teaching and
learning: students have greater clarity about
their obligations (and are asked to master
more engaging tasks), and teachers can
come to believe that assessment results are
both meaningful and useful for improving
instruction.”
Wiggins, Grant (1990). The case for authentic assessment. Practical Assessment, Research &
Evaluation, 2(2). Retrieved January 22, 2007 from http://PAREonline.net/getvn.asp?
v=2&n=2.

14. Good Rubrics
 Assess the performance of a student in
the completion of a task.
 Complement, but do not completely
replace, traditional assessments.
 Communicate expectations clearly.
 Communicate the level of success a
student is having.
 Inform future instruction.

15. Rubric Development
 Questions that need to be answered,
and should be answered
collaboratively:
 What should the students know and be
able to do? (These are the standards.)
 What products indicate that students
have met these standards? (Describe the
authentic task(s) or project(s) that will be
scored.)

16. Rubric Development
 What does “perfect” performance on these tasks,
or a “perfect” project look like, and what does an
unsuccessful project look like? (These are the
top and lowest score criteria for judgment.)
 To communicate the expectations for the task,
and, in the end, how well students performed, a
rubric is developed that contains
 the elements of the task or project that meet the
established standards,
 the criteria for the evaluation of each element, and
 ultimately the student’s “score” on each element.

17. Good Rubric Development
 Collaboration is essential in this work.
 Define the standards to be met.
 Describe the task(s) or project(s) to be
completed.
 Describe the criteria for judging the task or
project on each required element.
 Set the scale from “perfect” to unsuccessful
on each required element.

18. Sample Writing Rubric
Essential
Elements
0 1 2 3
Purpose
Lacks purpose Unclear purpose Limited
expression of
purpose
Purpose is clearly
expressed
Development
Inappropriate
response to
thesis, or lacks
thesis; no topic
development
Undeveloped or
vague thesis,
theme, or topic
Rudimentary
development of
thesis, theme, or
topic; limited in
depth or clarity
Good development
of thesis, theme, or
topic; conclusion is
more than a
summary
Details
Virtually no
relevant details
Few relevant
details
Details lack
elaboration;
important details
omitted
Details are
adequate &
relevant
Organization
Lacks
organization
Unclear
organizational
strategy
Organizational
strategy includes
transitions
Logical
progression of
ideas

19. Designing the Rubric
 Creating a table with scores across the top
and criteria down the side and language
within each box is easy, but you might be
creating an “emerging dog” instead of your
desired, and good, “dog.”
 The language in each of the boxes is the key
to a “good rubric.”
 Rubrics that are “emerging” can be
rehabilitated and become “good” ones!

20. Language Within a Rubric
 Each “box” will have a description of
what that score value means relative to
the criteria.
 The descriptions will be written in
language that is appropriate for the
grade level of the students and that
avoids any jargon or language that
would be unclear to parents.

21. Language Within a Rubric
 The language will be agreed upon by the
developers of the rubric, at a minimum. Best
practice would be to have at least a sample
of the students and parents involved in
revising a draft of the rubric to raise any
questions of clarity about standards, criteria
or the task, and to assist in revising the
language before it becomes the “final” draft.

22. Testing a Rubric
 The “final draft” of a new rubric should
be “tested” by having a sample of tasks
provided to readers for scoring on the
rubric.
 Ideally, teachers, students and a
sample of parents would be involved in
this, but at a minimum the teachers
would have collaborated on this rubric.

23. Validity and Reliability
 The rubric can be considered valid if the
different readers assign the same score
values for each sample,
 and it can be considered reliable if a set of
“perfect” projects are all given “perfect”
scores by the various readers, and a set of
unacceptable projects also get unacceptable
scores by the readers, etc.

24. Fairness
 The task, or project, assigned must be
designed as an authentic assessment
of how well students have achieved the
standards.
 The students should be given the
rubric with the assignment, and should
be taught how to use the rubric to
create their best work.

25. Fairness
 The rubric must be applied equally to
all tasks or projects; the identity of the
student should not affect the scoring of
the project.
 The connection of the rubric to a final
“grade” on the assignment must be
made clear from the beginning.

26. FAQ’s
 Choose to run scores from left to right, or
right to left? How big should the matrix be?
 We naturally read English documents from left to
right. Some suggest starting on the left with the
“perfect” score column so students see that first,
but this can feel awkward to traditional learners.
 The matrix should contain no more than 6-7
score points and no fewer than 3. Five seems to
be enough for clarity for most rubrics. You can
put +/- signs in between the columns if you need
finer discrimination in scoring.
 The number of criteria should be the minimum
necessary to assess the project or task.

27. FAQ’s
 Can a good rubric be created by a person all on their
own? What about all the “rubric maker” software
programs?
 The best rubrics are developed and tested collaboratively.
 If there are predefined standards and established criteria,
an individual can create a good rubric. There are many
free online rubric creating programs as well as websites
and books with sample rubrics that can be adjusted. See
the full page list of these websites on the handout.
 The best rubrics are tested for validity and reliability and
this requires collaboration. Authentic assessment in
general will be used more effectively to inform instruction
when teachers collaborate on the testing of rubrics.

28. FAQ’s
 If you don’t have a collection of samples to use to
test the rubric, how can you be sure that it’s reliable
and valid?
 You really can’t be sure.
 If you have a “new” task or project then the rubric is being
“tested” with it as the students complete the project or
task. The “untested” rubric will have to be handed out with
the assignment, but even at that stage it can be improved
based on student feedback.
 The particular challenge with reliability and validity comes
with getting inter-rater agreement on the scoring and you
need to have samples and more than one reader to
establish the “goodness” of the rubric.

29. FAQ’s
 Should the points given “add up” to a grade, or be averaged for
a grade?
 This really depends on the task or project and what the grade is
meant to represent. If the task itself is getting a grade, then the
rubric can be scored to produce a grade. However, many tasks
are part of a larger graded unit, so there might be a rubric created
for the group of assignments that all work together to produce a
grade.
 A writing assignment, for example, can be scored on a rubric for
the quality of the writing, but there may be a number of elements
to the written piece that must be included in order for the paper to
be successful at meeting content standards that are outside of
actual writing standards.

30. Rubrics
 They are an integral part of the authentic
assessment of worthy intellectual tasks and
projects.
 They can be used effectively to
communicate with students and parents
about what’s expected and how well a
student is meeting those expectations.
 They can inform instruction more effectively
than traditional assessments.

31. Good Rubrics
 They can be time consuming to produce, but
valid, reliable and fair rubrics will last for as
long as the standards are assessed by a
task or project. The criteria are based on the
standards, so they can be adjusted, if
necessary, for different authentic
assignments, without being rewritten.
 They need to be introduced systematically
for the educators, students and parents who
may be “afraid” of them. Don’t let them “jump
up.”

32. Student Voices; What’s Best
 "In the end they help make things fair so that a
student can't complain about his grade, and it also
gives the teacher back up for the grade he/she gave.
A student can also use it for them if they feel they
have been unfairly graded.”
 "I think rubrics can help teachers grade better than
just thinking of a grade in their heads.”
 "I like the way they word the scores and also how they
show you to get that score. The guidelines are right
down to the point and are precise to what you need to
get that score.
Transforming Middle Schools and School Systems: What students say about rubrics.
http://www.middleweb.com/CSLB2CubRub.html

33. Thank You
As you implement best practices in
Curriculum, Instruction and
Assessment, you’ll find that the wise
use of good rubrics will contribute
greatly to the improvement of
instruction and measurable gains in
student learning. Take the time you
need to build good rubrics!