ISSOTL12

1. ISSOTL12 Hamilton, Canada
Doing institutional change ‘the fourth way’:
a whole institutional change programme for assessment and feedback
Professor Shân Wareing
Dean of Learning and Teaching Development
University of the Arts London
E: s.wareing@arts.ac.uk
From 5th November 2012
Pro Vice Chancellor Learning and Teaching
Buckinghamshire New University
E: shan.wareing@bucks.ac.uk
Follow me on twitter @shanwareing
Trigger:
If participation prevails – if what matters most is left unreified – then there may
not be enough material to anchor the specificities of coordination and to uncover
diverging assumptions.
Wenger 1998 p65
Synopsis
This paper explores the model of quality assurance and professional development
articulated in Hargreaves and Shirley’s (2009) The Fourth Way through its application to a
case study of institutional changes to assessment, using the model as an explanation of the
initial hostility of staff to the changes and of their ultimate success.
Case Study
The case study relates to an intervention to improve assessment practices at a large visual
and creative arts university in London, where clarity around assessment standards and
expectations, consistency of grading, usefulness of feedback, and turn-around time, were of
concern.
1

2. The University of the Arts London’s overall curriculum was loosely defined in a number of
significant fields, and assessment in particular was based on an assumption of apparently
tacitly shared standards. A national Quality Review in 2007 questioned the adequacy of
assessment policy and processes. The National Student Survey results showed students did
not perceive assessment to fair nor the criteria on which they were assessed to be clear
beforehand. With up to 50% of teaching undertaken by hourly paid staff, and a lack of
social spaces for staff, it seemed unlikely that the presumed shared standards based on
interaction amongst staff could be established satisfactorily. There was also research
suggesting that students from under-represented backgrounds found it harder to deduce
tacit standards, with adverse impact on their achievement. UAL had a gap between the
achievement of white students and Black and Minority Ethnic students which is 10% higher
than the national average gap. Anecdotally there were stories of very minimal and
unhelpful feedback being given to students.
The initiative consists of eight explicit standard criteria, a standard feedback form which has
been developed into an online form integrated with the student record system, and a fixed
matrix indicating key identifiers of achievement at different grades. There are
undergraduate and taught postgraduate versions of each. Courses can adapt the feedback
form by indicating any criteria that are not applicable, and by customising the explanatory
text appearing below each criterion to make it more relevant to their students.
Achievement is indicated using radio buttons against each of the criteria, and feedback can
be provided against each criterion either using standard text from the matrix or writing
unique text. All markers are required to write free text ‘feed forwards’ to students. The
grade is not the computational outcome of the achievement against each criterion. The
matrix in indicative only, but cannot be adapted: it has to appear in its standard form in all
course handbooks.
The purpose of the initiative was to introduce explicit reified standards into assessment, as
the basis for marker moderation discussions and for explaining assessment to students; to
prompt more effective feedback on student work; to speed up the process of providing such
feedback for markers; and to underpin course design. Feedback from staff suggests that the
development has at least partly met all these goals.
The new criteria and feedback forms were introduced first in September 2009, and for final
year undergraduate students and students on taught postgraduate courses in September
2010. There was an average increase of 7% in UAL’s NSS scores for assessment and
feedback in 2011, and of 7%-10% for the assessment and feedback questions in the
Postgraduate Taught Experience Survey. A further increase of 2% occurred in the 2012 NSS.
Additionally, there was an increase in 2010/11 in the proportion of 1st/2:1 degrees
awarded, up c. 2% to 66% on the trend of proceeding years, when it has been in a range
62%-64%. Student feedback obtained through focus groups was positive from the outset
about the criteria.
A qualitative evaluation of the initiative found that staff with a more pedagogically
developed understanding of assessment found the new criteria and forms at least as good
or better than their previous practices, but staff with a more superficial understanding of
assessment found the criteria and forms constraining.
2

3. Analysis
The literature on assessment in art and design emphasises connoisseurship, and tacitly
established shared standards, built up through community interactions (e.g. Orr 2006).
However, this approach does not account for power imbalances between staff and students,
nor the possibility of prejudice and bias in marking (Sabri 2011) nor for the logistical
difficulties of developing shared standards amongst hourly paid staff in the absence of
community spaces such as staff rooms.
An ideal intervention to improve assessment might have involved an extensive professional
development programme, supporting all staff in acquiring a personal and scholarly
understanding of assessment, to underpin personal and locally crafted assessment and
marking practices. However, barriers to this approach included lack of resource to provide
extensive professional development, including paying hourly paid staff to attend, logistical
and motivational difficulties to ensuring sufficient participation by staff, and the delay in
realising the benefits of the change programme to students.
It was therefore decided to impose a universal assessment system, designed to address
some of the known shortfalls in the existing approach. The new system was pedagogically
robust and underpinned by the scholarship of learning and teaching (particularly drawing on
Bloom 1956, Kolb 1984, Perry 1970 and Biggs 2003, and taking account of the disciplinary
needs of the visual arts).
Many academic staff initially saw the imposition of the assessment changes as
compromising their autonomy, a managerial challenge to their professional judgement, and
a blow to the educational value of their teaching via an inflexible, inappropriate set of
standards. However, student feedback was positive, and scores for students’ perception of
assessment and feedback in national surveys rose. Over two years of implementation, staff
feedback has improved dramatically: “The assessment tool ... speeded up the process of
assessment and allowed ... staff to give better informed, critical and useful feedback to
students.”
Hargreaves and Shirley’s analysis of education over 60 years charts a shift from
unquestioned professional autonomy, through a period of where transparency and
students’ rights resulted in imposed and deprofessionalising standards. The most effective
leadership and management processes balance support and accountability, freedom and
consistency, the empowerment of staff and the empowerment of students. Reactions to
the assessment change programme, and its ultimate success, can be explained through the
lens of this model.
3

4. P48, Fig 3.1 What to retain and what to abandon
Retain Abandon
First Way Inspiration, innovation and autonomy Inconsistency and professional license
Second Way Urgency, consistency and all-inclusive Cut-throat competition and excessive
equity standardisation
Third Way Balance and inclusiveness, public Persistent autocracy, imposed targets,
involvement, financial reinvestment, obsession with data, effervescent
better evidence, and professional interactions
networks
Fourth Way “A democratic and professional path to improvement that builds from the bottom,
steers from the top, and provides support and pressure from the sides” p107
P51 ‘you can’t just adopt the end product of something that too others years to develop...
no two places are alike...nevertheless they can be cross-pollinated with other successful
reform initiatives and movements’
Hargreaves, A. and Shirley, D. (2009) The Fourth Way: The inspiring future for Educational
Change, London: Sage.
Conclusion
1) It's clearer in hindsight than it was at the time
2) Assessment problems were exacerbated by the disciplinary culture, by headline debates
about good assessment in HE, & by logistical problems such as c. 50%-75% of the
teaching undertaken by fractional and hourly paid staff, no staff rooms, reduced budgets
for staff development, and no time for lunch hours or coffee breaks
3) Half the effort was on the solution, the other half was building the capacity to create and
implement the solution. That's why good practice isn't directly transferable
4) Change management in mass higher education must enable professional autonomy
within clear structures and systems, introduced collegiately and supportively.
5) Leadership has a moral imperative, which is the source of its momentum
4

5. Links
The Marking matrix, information about its development, and formal evaluation:
http://www.arts.ac.uk/assessment/markingcriteria/index.html
A formal evaluation of the marking criteria introduction:
http://www.arts.ac.uk/assessment/assets/doc/Report-Evaluation_of_UAL_Marking_Criteria.pdf
Screengrabs of the online assessment tool, developed from staff feedback after the marking
criteria and standard feedback sheet were introduced:
https://picasaweb.google.com/114823604275354290558/OATScreengrabs181012?authuser=0&authkey=Gv1sRgCIf74Y2vw4-
LywE&feat=directlink
Evidence of improvement in students’ perception of assessment:
http://www.arts.ac.uk/assessment/assets/doc/UAL-NSS-assessment-results-2008-12.pdf
References
Archer, L. (2007) ‘Widening Participation, Social Class and Ethnicity: Issues and
Considerations’, Conference Presentation at University of the Arts London 2 nd Annual
Learning and Teaching Conference 21st-22nd May 2007.
Blair, B. (2006). "'At the end of a huge crit in the summer, it was "crap" - I had worked really
hard but all she said was "fine" and I was gutted.'" Art, Design and Communication in
Higher Education 5(2): 83-95.
Cowdroy, R. and Williams, A. (2006) ‘Assessing creativity in the creative arts’ Art, Design &
Communication in Higher Education 5 (2) pp97-117
Elton, L. (2006) ‘Assessing creativity in an unhelpful climate’ Art, Design & Communication in
Higher Education 5 (2) pp119-130
Gawande, A. (2010) The Checklist Manifesto. London: Profile Books
Hargreaves, A. and Shirley, D. (2009) The Fourth Way: The inspiring future for Educational
Change, London: Sage.
Orr, S. (2006) ‘Assessment Practices in Art and Design’ Art, Design & Communication in
Higher Education 5 (2) pp79-81
Orr, S. (2010). 'We kind of try to merge our own experience with the objectivity of the criteria':
the role of connoisseurship and tacit practice in undergraduate fine art assessment. Art,
Design and Communication in Higher Education 9(1): 5-19
Sabri, D. (2011) An Evaluation of marking criteria at the University of the Arts, London, London:
UAL
Sadler, D. Royce (1983) Evaluation and the Improvement of Academic Learning, Journal of
Higher Education, 54: 60-79
Sadler, D.Royce. (1989) Formative assessment and the design of instructional systems,
Instructional Science 18: 119-144
Sadler, R. D. (2009). Indeterminacy in the use of pre-set criteria for assessment and grading.
Assessment and Evaluation in Higher Education 34(2): 159-179.
Sadler, R. D. (2010) Transforming holistic assessment and grading into a vehicle for complex
learning. In Assessment, Learning and Judgement in Higher Education. ed G. Joughin.
London: Springer
Wenger, E. (1998) Communities of Practice. Cambridge: Cambridge University Press
5