1. Assessment Professional Learning
Module 5:
Making Consistent Judgements
1

2. Assessment FOR learning:
occurs when teachers use Assessment AS learning:
inferences about student occurs when students reflect
progress to inform on and monitor their progress
their teaching. to inform their future
learning goals.
Assessment OF learning:
occurs when teachers use evidence
of student learning to make
judgements on student achievement
against goals and standards.
2

3. As educators it is imperative that we make high
quality judgements about student learning.
What does that mean we should do?
3

4. EXAMPLE
“ How can you possibly award prizes when
everyone missed the target?” said Alice.
“Well,” said the Queen, “Some missed by more
than others and we have a fine normal
distribution of misses, which means we can
forget about the target.”
Lewis Carroll, quoted in Rowntree, 1987, pp 181-2

5. Quality and trustworthiness in
evidence of learning requires both:
• valid (fair, accurate, appropriate) and
• consistent (reliable)
assessment judgements.
5

6. Valid
Everyday synonyms:
- truthful
For valid assessment
- appropriate fairness,
- justified accuracy and
- convincing appropriateness
- accurate matter.
- legally acceptable
- fair
6

7. Validity is a matter of
fairness, appropriateness and accuracy
• Are the tasks constructed, presented and conducted so that
all students have an equal chance of demonstrating their
learning?
• Do the tasks represent all the valued learning you want the
students to have undertaken? Do they assess a sample of all
the important concepts and objectives?
• Are the assessment tasks probing the students thinking to the
depth you want - or are they able to “get away with” a
superficial understanding?
7

8. Validity is a matter of
fairness, appropriateness and accuracy
• Are the assessment tasks monitoring what you think they are
(and not some intervening prerequisite skill, or conceptual
understanding)?
In particular, do the assessment tasks require specific
knowledge or skills which some students may not have, and
which have not been explicitly taught?
(e.g. how to: read, draw a graph or ‘google’?)
8

9. Assessment is valid if it …
• assesses appropriate content and objectives
• provides information which is useful for some
valuable purpose (for/as/of)
• is assessed with sufficient accuracy
• is fair to all students.
9

10. Consistency is a matter of reliability
How confident can you be that judgements that you have
made are not significantly affected by chance factors such as:
• how the student was feeling on the day?
• how the assessor was feeling on the day?
• who the assessor was?
• luck in being assessed on some things and not others?
• luck in that the mode of assessment suited the
student particularly well (or didn’t)?
10

11. Examples of everyday use of the term “reliability”
 A set of kitchen scales is reliable if ...
 A clock is reliable if …
 A train is reliable if …
 A friend is reliable if …
 An assessment task is reliable if … (what?)
11

12. Definition of consistency
The consistency of an assessment is the reliability with which it
assesses whatever it assesses.
It is the reproducibility that is the focus.
12

13. The assessor matters in making consistent judgements.
We are human and our judgements may be influenced by many
factors besides the actual standard of work.
• INTER-rater reliability:
if another assessor judged the work, would the student be
awarded the same result?
• INTRA-rater reliability:
if the same assessor judged the work on another day would the
result be the same?
13

14. Consistent assessment
An assessment is consistent if the scores that students get are
reliable:
• from one occasion to the next
• from one form of assessment to another
• from one assessor to another.
14

15. The difference between consistency and validity
Consistency (or reliability) is a technical question. Will work
be judged in the same way every time, regardless of who
assesses it, or how, or when?
Validity has both a technical element (does the assessment
accurately judge what it says it judges?) and a philosophical
one (is the assessment appropriate and fair?).
Validity is an indication of the value and correctness of what
we do.
15

16. Consistency, alone, is not
sufficient.
Assessment must also be valid
(fair, appropriate and
accurate).
16

17. Each step in the assessment process involves:
• the possibility of error
• value-driven decisions
• assessment processes that may disadvantage
some types of students.
Use collaborative planning processes, transparency of
assessment tasks, criteria, rubrics and variety to maximise
valid and consistent judgements.
17

18. Strategies and Protocols
For valid and consistent judgements of student learning
progression, we need to put in place strategies such as:
• common assessment tasks planned collaboratively
• shared design processes for assessment tasks and for
rubrics
• comparing work with exemplars (e.g. assessment maps)
• cross-marking of sample assessment tasks
• moderation and consistency protocols.
18

19. Critical stages of the Moderation Process
1. Developing a common understanding of what is to be
assessed and the purpose of the assessment
2. Drafting the assessment task requirements
3. Drafting the criteria- or marking scheme
4. Sample assessing
5. Final assessing.
19

20. The final assessing stage of moderation may involve:
• assessors meeting together
• sharing of assessed work samples
• discussion of queries
• cross-marking each other’s work samples
• discussion of borderline or special circumstance cases.
20

21. When deciding HOW to make more valid and consistent
judgements “efficiency” issues must also be considered:
• time efficient (yours and your students!)
• learning efficient (maximises learning)
• teaching efficient (reduces wastage)
• cost efficient
??? other factors
21

22. Making consistent judgements is important.
Assessment
FOR learning
Assessment AS
Learning
Assessment OF
learning
But consistency is not enough -
we have to make valid judgements as well.
22