1. Criterion- based assessment
forms for prevocational
trainees

2. In NSW, in research conducted in 2010 (and reported at the last Forum), we
reviewed prevocational assessment forms completed by prevocational
trainees and found that, as currently used by trainees and supervisors, the
assessment forms may under‐report trainee underperformance, do not
discriminate strongly between different levels of performance of trainees or
the training system, and do not provide trainees with enough specific
feedback to guide their professional development.
The results of that study, which looked at over 3000 assessment forms,
were subsequently published in the MJA. I have distributed some copies of
that paper, and also copies of the red end‐term assessment form we
evaluated then, and a new set of forms that we are going to talk about
today. Please share copies with your neighbours if necessary.
1

3. Our aim is to find better methods of assessment for prevocational trainees.
Among our underlying assumptions, we think that these are the purposes
of assessment.
1. Formative: to guide the trainee’s professional development.
2. Safety: to identify safety concerns and underperformance issues.
3. Summative: to document satisfactory performance
(certification/registration/progression).
4. Systemic: to identify strengths and weaknesses of the education and
training process.
These are four big purposes, and it is quite likely that a range of assessment
methods is required to meet all of them. However, in NSW, and I think in
other jurisdictions as well, assessment is conducted through a mid‐term
formative appraisal and an end‐term summative assessment — two forms,
filled out at two interviews between the term supervisor and the trainee.
2

4. This graph shows the main result from our first study. It shows the overall
performance ratings by trainee self‐assessment and supervisor assessment
at mid‐term and end‐term. You can see that the lowest rating, 1, below
expected level, is virtually unused. Trainees never use it and supervisors use
it about one in a thousand times. The borderline rating, 2, is rarely used –
about one in a hundred times. Trainees rate themselves highly and
supervisors rate them even more highly – you can see that at end‐term,
supervisors rate more trainees as performing above expected level than at
expected level.
This is the result for the overall performance rating, but a virtually identical
graph could be drawn for the 18 specific competency ratings that are also
assessed. In summary, At end term, 98.5% of trainees were rated by
themselves and by their supervisors as performing at or above the
expected level on every item. This starts in term 1 as an intern and
continues through to term 5 as a resident. The problem with this rosy
picture is that we suspect from other evidence that it is not realistic. A
small proportion of trainees at any one time are underperforming on some
aspects of their job, but this is not picked up in assessment ratings.
3

5. However, these forms provide plenty of room for written comments
intended to give trainees specific advice about their strengths and
weaknesses. Potentially, this could be where the real guidance is.
Unfortunately,
3

6. We found that supervisors typically did not write much on the forms, and
that what was written, while encouraging in a general sense, lacked
specificity.
Coupled with reports from JMOs that supervisors often do not provide
much verbal feedback at assessment sessions, this made us fear that the
assessment process was sometimes not giving trainees the feedback they
need. Feedback is generally agreed to be the most important outcome of
assessment as far as trainees are concerned.
4

7. In relation to the four purposes of assessment, we concluded that the
assessment forms as they were actually used in the field were not really
working.
This is likely to be for several reasons, and we ended our MJA article by
saying that “the apparent weakness of the current process may be a
measure of the time and engagement that trainees and supervisors bring to
written assessment. There is always a risk that filling in forms will be seen
as a mere formality. Future research on assessment methods that are
sustainable in the clinical environment and supported by trainees and
supervisors would be valuable. Changes in the workplace to facilitate
adequate time for feedback and assessment may be required, and
supervisors may need training in assessment methods.”
So, there’s more to this than forms. Nonetheless, we thought it might be
worthwhile to try to come up with something better.
5

8. Our hypothesis is that at least part of the problem with the current form
relates to the 4‐point rating scale, “Below expected level”, “Borderline”, “At
expected level”, “Above expected level”. This scale is relativistic and may be
difficult to apply or interpret.
Performing “at expected level” does not indicate a specific level of
competence in any of the clinical skills assessed on the form, as the forms
are used with trainees from their first term as an intern to their last term as
a resident.
Different supervisors are likely to have different expectations, and if a
trainee is told that they are below expected level, this does not in itself tell
them anything about what they should be doing.
6

9. In consultation with DPETs, the Prevocational Training Council and the JMO
Forum, we redesigned the assessment forms to promote criterion‐based
assessment. That is, the assessment items provide explicit descriptions of
the behaviours that constitute inadequate, not‐quite‐adequate, adequate
and excellent performance. In this way, even a “tick‐box” approach to
assessment provides some feedback to trainees on their specific
performance and the requirements for excellence. The criterion‐based
approach is also closer to an explicit judgement of a trainee’s specific level
of competence for each of the assessable items on the form.
This is not an original approach; it has been used in other situations and by
other people. We tried to write descriptive criteria appropriate to the
assessment of prevocational trainees.
You have copies of our experimental forms in front of you to look at. On this
slide, we are looking at the overall performance summative rating item,
which has been set up to express results in terms of the trainee’s readiness
to progress to the next stage of training.
7

10. The overall assessment is based on assessment of thirteen specific items,
listed here, each with its specific description of performance criteria.
8

11. Here is one of the simpler assessment items. It is our hope that both the
trainee and the supervisor will be able to make a clear choice between each
of these levels, that it will be clear what a particular rating means, and that
the rating description itself will give the trainee some idea about what
behaviour is considered excellent (which in this case, for instance, includes
the ability to admit mistakes).
9

12. Here is one of the more complex assessment items. For some assessment
items, we thought it was necessary to provide a choice of descriptors. So, in
this case, poor communications could be for various reasons and the
assessment form allows for the relevant reasons to be highlighted.
10

13. OK, so we are now in the initial stages of testing these new forms to see if
they should be changed further and whether they have any potential for
general use.
We asked Directors of Prevocational Education and Training (DPETs) at sites
across NSW to try the new forms with a small group of trainees and
supervisors in terms 4 and 5. Most of them kindly agreed to help, but many
wanted to start in term 5, and we have not collected much data yet.
We provided a one page guide to the new forms, briefly explaining the
rationale.
We asked the trainees and supervisors to fill in a very simple evaluation
form, which asked:
“Which forms would you prefer to use in the future?” (the old ones
or the new ones).
And “Why?”
11

14. As I said, we have not collected much data
yet, but if the trend so far continues, we
might have some difficult decisions ahead of
us ...
11

15. ... Because our users are quite divided in their responses. When you look at the reasons
given by respondents, there are clearly two camps.
12

16. The reasons given by supervisors and trainees were similar
Old forms more succinct, more concise (this is true – there are far more
words on the new forms)
Old forms less tedious (I think that’s important — there is certainly enough
tedium in the world)
New forms wordier but add no extra information (I find this an amazing
comment, but I guess it means that the extra information provided on the
new forms had no value to the trainee who made this comment)
No need for extra description on new forms as supervisor can add detail in
comments if required (this supervisor comment made me think “If only!”)
New forms use more paper
And
Putting self‐assessment and supervisor assessment on separate forms
makes comparison more difficult (on our test forms, we did put self‐
13

17. assessment and supervisor assessment on separate documents, thereby
doubling the paper from 4 to 8 pages, but it would have been possible to put
both on one. Would this have changed perceptions of the size of the form?)
Supervisor is familiar with the old form (this trainee comment isn’t much of
an objection, is it? Given 12 months, everyone could be familiar with the
new form)
Harder to see where to tick (another unfamiliarity comment – but it is true
that we designed the new form partly in the hope that people would no
longer be able to tick straight down one column without stopping to think,
so this objection could represent an attitude to assessment that we really
don’t want to encourage)
13

18. The reasons given by supervisors and trainees were similar
New forms more detailed and comprehensive
New forms give better options for accurate assessment
New forms more helpful to identify issues
New forms better for feedback, generate more specific discussion, give
clearer advice
New forms more explicit about what is being assessed
Those in favour of the new forms were very much in agreement about why
they were better. Some of these people also added the comment that they
were harder to fill in than the old forms, but they thought the extra effort
was worthwhile.
The reasons given in favour of the new forms are gratifying to us, the
designers, because they mirror the hopes we had for the form. If only this
group was the clear majority, we’d be home and hosed! We’ll see what
happens when we get more numbers in.
14

19. What the taste test has told us so far, I think, is that the devil is in the detail.
There is plenty of room for further tinkering with the design of the form and
the exact wording of the criteria. Could we reduce the number of questions
and still be assessing the important components of trainee performance?
Could our criteria statements be briefer? Should self‐assessment be part of
summative assessment? Could the summative assessment be made simpler
than the formative appraisal? And what are the other things we should be
doing to improve assessment processes? We are working on the form
because we had evidence that the previous form was not working, but that
doesn’t mean that this is the only thing we should be doing.
Other things we are thinking about include the possibility of assessing
particular competencies in particular terms, such as assessing aseptic
technique in the surgical term or prescribing in the medical term. We are
thinking about more training for supervisors in how to do assessment, and
whether other team members should be contributing more to assessment.
14

20. To conclude
There isn’t a clear preference in the user group for the new form.
But in fact this kind of taste test is not the critical test. What we really want
to see with this assessment form is some evidence that assessments are
more realistic, and that the guidance they provide to trainees is more
useful. Evaluating the quality of guidance will be difficult, but there is one
measure of the realism of the new form that will be fairly easy to evaluate.
With the old form, both interns and residents tend to receive the same
ratings all the time in every term. With the new form, because of the way
the criteria are described, we would expect the intern cohort to have lower
ratings than the resident cohort, and we would expect trainees to show
improving ratings as they progress from term to term. This is something
easily testable, although we will need much more data than we have
collected so far.
I think now might be a good time for questions and comments, except to
say that CETI is happy to share this form with other jurisdictions and to
15

21. work with other jurisdictions on research in improving assessment. It is a
difficult area, but very important, because assessment doesn’t just measure
trainees. It also measures the effectiveness of our training programs.
15

22. In NSW, in research conducted in 2010 (and reported at the last Forum), we
reviewed prevocational assessment forms completed by prevocational
trainees and found that, as currently used by trainees and supervisors, the
assessment forms may under‐report trainee underperformance, do not
discriminate strongly between different levels of performance of trainees or
the training system, and do not provide trainees with enough specific
feedback to guide their professional development.
The results of that study, which looked at over 3000 assessment forms,
were subsequently published in the MJA. I have distributed some copies of
that paper, and also copies of the red end‐term assessment form we
evaluated then, and a new set of forms that we are going to talk about
today. Please share copies with your neighbours if necessary.
16

23. I might add that we now know from John Collins’ review of the Foundation
Programme in the UK that fixing assessment is not simply a matter of
adding extra assessments.
This slide quotes data presented by John Collins in a talk at CETI recently. In
the UK, he told us, “Workplace‐based assessment has been changed to
providing opportunities for supervised learning events rather than acting as
a component of overall assessment.” That is, less formal written assessment
and more opportunities for feedback.
Anyway, in NSW there was never much prospect that we were going to
introduce such an ambitious scheme of multiple workplace‐based
assessments for prevocational trainees. Our 800‐plus term supervisors
probably would have stormed the Bastille. But, given that the current forms
are not working, we thought it worthwhile to try to come up with
something better.
So now I want to tell you about our small experiment with a new kind of
form.
17