General Principles of Intellectual Property: Concepts of Intellectual Proper...
Principles as discourse: A Blueprint for transformational change in assessment
1. Principles as discourse: A blueprint for
transformational change in assessment
Emeritus Professor of Higher Education
University of Strathclyde, Scotland
Visiting Professor, University of Ulster
Adjunct Professor, University of Swinburne, Australia
Expert Consultant to JISC: Assessment and Feedback Programme
Sheffield Hallam University: 15 September 2015
2. Barriers to enhancing the quality of teaching,
learning and assessment institution-wide
Principles-led change
REAP project (University of Strathclyde)
Viewpoints project (University of Ulster)
Discourse-centred change – learning from REAP
and Viewpoints
Value at Sheffield Hallam University
Plan for session
3. Barriers to transformational educational
change institution-wide
1. Lack of shared educational frame of reference to
guide innovations in practice
2. Difficulty defining good educational practice
3. Accessibility of educational research
4. Disciplinary differences in teaching and learning
5. Weak links between local practices and educational
policies, strategies and procedures
6. Challenge of getting multi-stakeholder buy-in
7. Cultures and micro-cultures in institution
See Nicol & Draper (2009)
4. Shifting paradigm for assessment & feedback
Teacher-centred approach
Assessment of learning
Experts make judgements
Focus on leaning outcomes
Transmission of criteria
Individual assessment tasks
Teacher as feedback source
Quality of feedback message
Feedback as monologue
Teacher-feedback reviews
Externally provided feedback
Teacher responsibility
Delivery of feedback
Learning-centred approach
Assessment for learning
Students (learn) to judge
Focus on process and outcome
Co-construction of criteria
Collaborative tasks
Multiple sources – peers, others
Quality of feedback interaction
Feedback as dialogue
Student feedback-reviews
Internally generated feedback
Shared responsibility
Use of feedback
5. The challenge
How do we embed new thinking in policies and
in educational practices across a whole
institution in ways that are informed and
enhancing and not constraining?
6. Approaches to quality enhancement
1. Improve teachers’ skills – workshops, courses
2. Introduce new teaching method institution-wide (e.g.
problem based learning)
3. Facilitate changes in teachers’ conceptions
(reflection)
4. Institutional plans to define and support change
5. Disciplinary focus – foster scholarly discussion
amongst colleagues
6. Action research – teachers investigate things of
interest to them
Amundsen and Wilson (2012)
7. Re-engineering Assessment Practices
(REAP) project (www.reap.ac.uk)
Scottish Funding Council (£1m): 2005-2007
Enhance learning quality, evidence teaching
efficiencies (technology) and embed changes.
Strathclyde, Glasgow and Glasgow Caledonian
Large 1st year classes (160-900 students)
Range of disciplines (19 modules ~6000 students)
Many technologies: online tests, simulations, discussion
boards, e-portfolios, e-voting, peer/feedback software,
VLE, online-offline
Assessment for learner self-regulation
Set of assessment and feedback principles
8. Original 7 principles of good
assessment and feedback
Good assessment and feedback should:
1. Clarify what good performance is (goals, criteria,
standards).
2. Facilitate the development of reflection and self-
assessment in learning
3. Deliver high quality feedback to students: that enables
them to self-correct
4. Encourage peer and student-teacher dialogue around
learning
5. Encourage positive motivational beliefs & self esteem
through assessment
6. Provide opportunities to act on feedback
7. Provide information to teachers that can be used to help
shape their teaching (making learning visible)
Nicol and Macfarlane-Dick (2006)
9. Thinking behind REAP
The vision – Assessment and Feedback should support
the development of learner self-regulation
A set of assessment & feedback principles drawn from
research – to operationalise this vision
Principles seen as translation device – to make the
research easily accessible
A common frame of reference across disciplines
Examples of implementation in different disciplines
10. Good assessment and feedback practice should
1. Clarify what good performance is (goals, criteria,
standards).
Provide students with list of criteria before task.
Get students to write out criteria in own words
Groups rank samples of work according to criteria
Students derive criteria from exemplars before a task
Students peer review others’ work (with/without criteria)
Groups create problems (e.g. MCQs) for others to solve
Groups create criteria for an assignment
Students must develop own ‘concept of quality’
11. Good assessment and feedback should:
2. Facilitate the development of reflection and self-
assessment (and peer assessment) in learning.
Provide an abstract with an essay (reflection)
Students identify what is strong and weak when they
hand in an assignment
Students request the type of feedback they want
Provide written explanation of concepts underpinning a
set of problems they are working on (deep reflection)
Or evaluate the elegance of different solution pathways
Implement peer review: e.g. students’ comment on
each other’s work (see Nicol, 2015: Nicol et al, 2014)
then review their own work
Give students practice in making evaluative judgements
12. Good assessment and feedback practice should:
3. Delver high quality information to students: that helps
them to self-correct
Students request feedback they wish (cover sheet)
Feedback on processes and skills – maximise transfer
Teacher provides ‘feed forward’ rather than feedback
Feedback on students’ self-assessments and/or peer
reviews
Don’t give feedback – point to resources where
answer/issue can be elaborated
Ask students to review peer work but insert a sample
of your work to be reviewed as well
Calibrate students’ ability to make evaluative judgements
(see Hattie and Timperley, 2007)
13. Good assessment and feedback should:
4. Encourage teacher-student and peer dialogue around
learning
Discussions of feedback in tutorials or scheduled
feedback events (e.g. Bring and discuss)
Collaborative assignments
Electronic voting methods: polling and peer discussion
Students reviewing each other’s work
Wrap dialogue around any or all assessment processes –
before, during and after (Nicol,2010)
Attenuates teachers’ voice and strengthens students’
voice (shifts responsibility towards students)
14. Good assessment and feedback should:
5. Encourage positive motivational beliefs and self-esteem
Encourage climate of respect and accountability
Emphasise mistakes are part of learning
Focus students on learning rather than marks
Sequence tasks for progressive level of difficulty
Align formative and summative tasks
Use reader-responsive feedback (non-evaluative)
Use real life (authentic) learning and group tasks
Give learners choice in topic, methods, criteria
Implement other principles (group working etc)
About self-esteem and giving students a sense of control over their
learning: balancing structure with increasing responsibility
15. Good assessment and feedback should:
6. Provide opportunities to act on (respond to) feedback
Align your feedback to goals/criteria
Provide feedback as action points
Linked assignments so feedback can be used
Reward actual use of feedback in a new task (Gunn,
2010)
Get students to respond to teacher feedback – say what
it means
Get groups to discuss feedback and create action plan
Get students to say how used feedback when submit
next assignment [proforma]
Ensures feedback is processed and leads to knowledge building. Key
principle if your goal is to enhance NSS results
16. Good feedbacks:
7. Provides information to teachers that helps them shape
their teaching
Requested feedback
Just-in-time teaching –online tests before lecture
delivery
Electronic voting methods allow dynamic adaptation
One-minute papers
Discussion boards
18. Implementation
Local redesigns
19 module redesigns - principles guided implementations
Success: learning gains in exams (11 out of 19 modules)
improved quality w/o extra costs, high levels of student
satisfaction, efficiency gains.
Institutional developments
Sharing of good practice using principles as reference
Dep. Principal T&L set up working group: new A&F policy
(Strathclyde) grounded in principles agreed by Senate
Principles embedded in QA procedures
Many departmental/university initiatives referencing
REAP and using the principles
Feedback as dialogue campaign for students (see leaflet)
Widespread use of principles, nationally/internationally
19.
20.
21. In summary: learning from REAP
Principles-based approach has great potential
People talked about and used the principles – an
emerging discourse
Widespread take-up of principles in UK and,
surprisingly, internationally – translation/pragmatic
Nicol & Draper (2009) talked about the principles
acting as a kind of rhetorical resource – their format
and use seemed to persuade many people to come on
board.
22. Question
How to take things forward so that the ideas spread
more rapidly and deeply across the whole institution?
23. The Viewpoints project
Built on the REAP assessment and feedback principles
and examples
Funded by Jisc UK under its Curriculum Design and
Delivery programme (2008-12)
Took the REAP principles and examples turned them
into artefacts.
Tools to be used in face-to-face course redesign
workshops
28. Working round table participants storyboard a sequence of
teaching and learning activities while engaging in dialogue
using the assessment and feedback cards as prompts.
Each participant has a set of cards
The main phases of the process......are
Course design: the workshop process
29. Activity 1: Agree objective/challenge
The team agree the challenge/objective for their
session and write it at the top of the module
worksheet.
EG. encourage greater use of feedback, enhance
learner engagement.
30. Activity 2 – Select principles
Examine the front of the ‘Assessment and
Feedback’ cards and initially choose those that
might help you address your objective(s).
31. Activity 3: Map principles to timeline
Place the cards on the timeline, where relevant,
considering the student perspective.
At the induction phase, during the first few
weeks
Create a conceptual map
Note: cards can be repeated on the timeline.
32. Activity 4: Consider specific
implementation ideas
Turn over the most important card(s) and identify
any ideas that might help your group address
your challenge/objective(s).
Adapt or formulate new examples
Record by ticking examples and/or writing post-it
notes with examples
33. Activity 5 - Tailoring a solution
Discuss how your selected ideas could be used
in practice to address your challenge/objective(s)
As a group write a brief plan and make notes on
the worksheet using post-its or markers in the
‘your plan’ area.
34. Activity 6: Record Outputs & share
Record outputs (e.g. Take photo of design
plan)
Share outputs with others
37. Participants’ experience of workshop
Sociability of a board game –tactile, principles put in ‘the
hands of the users’
Sharing, problem-solving, solution focused, exploratory,
creative
Structured by timeline – storyboarding the learning
Discussion shaped/facilitated by principles (research-
informed) – adapted to their discipline
Participants construct, reconstruct, co-construct
meanings of principles again and again
Learning-focused rather than content-focused
38. Evaluation of Viewpoints
Workshop process highly successful - ideas had spread
beyond the workshop and across the institution.
As in REAP, principles established in policy, embedded in
quality procedures, used to support revalidation, taken
up by lifelong learning unit, used by Students’ Union in
leaflet etc. Departments ran workshops themselves.
Seeded new conversations
Like REAP, the emergence of a new discourse about
assessment and feedback across the institution
Model of change extrapolated – ‘principles-based
discourse’ model
39. Key criteria in constructing principles
1. Embody compelling vision – principles a way of operationalising that
vision
2. Informed by best available research – each captures a poweful
idea/aspiration from that research
3. Succinct (6-13 words) and written in plain English – accessible,
immediate impact, face validity
4. Action-oriented (verb-phrase): call to action
5. Tight-loose in formulation: applicable to any disciplinary context,
users construct/interpret
6. Complemented by other texts that enrich meaning – disciplinary
examples, research arguments, solutions to problems
7. Relevant to wider educational agendas/concerns
8. Number of principles important
See: http://www.reap.ac.uk/TheoryPractice/Principles.aspx
40. What educational principles are NOT
Procedures (e.g. ensure faster turnaround time for
assignments)
Vague statements with no obvious action (e.g. use every
opportunity to develop students as learners)
Things that are worthwhile but are taken for granted in
assessment (e.g. ensure assessment is valid or reliable or
fair)
Contentious issues (e.g. encourage competition in
learning)
Ideas with no clear educational basis (e.g. maximise use
of available classroom/learning space)
Efficiency directives (e.g. reduce time spent marking)
41. Changing the discourse: research
Grant and Marshak (2011)
A different way of thinking about and planning for change
Discourse plays a central role in the construction of
social reality
It shapes how people think about things and how they
act: and how people think and act shapes their
discourse
Changing the existing and dominant discourse will
support and lead to organisational change
Grant and Marshak (2011) identify criteria for
planning and evaluating change in discourse
Ref: Grant and Marshak (2011) Towards a discourse centred understanding of
organisational change, Journal of Applied Behavioural Science, 47(2), 204-235
42. Discourse is not just about conversations but also about
written texts, official documents, emails, memos,
stories, narratives, metaphors, slogans etc.
(Grant and Marshak, 2011)
Definition
43. New focus in organisational development
The central task of the change agent in an organization
is to identify what the prevailing discourses are, how
they serve to maintain the status quo and how
alternative discourses supportive of an intended
change might be established and maintained.
(Grant and Marshak, 2011)
44. Changing discourse: Grant & Marshak criteria
1. Establish the discourse for the intended change: principles
2. Facilitate change in discourse at multiple levels – projects focused
on interpersonal discourse but influenced organisational discourse
3. Conversations are the key discursive practice in seeding change –
focal point for REAP/Viewpoints conversations of disciplinary groups
4. Influence the discourse of those with power – DVPs put principles in
policy, also in validation and review procedures etc.
5. Harness alternative discourses to support change – assessment
problems, technology applications, graduate attributes all relevant
6. Keep the discourse on message: embed in documentation,
disseminate successes using principles
7. Maintain openness/self-reflective stance – advice to change agents
Ref: Grant and Marshak (2011) Towards a discourse centred understanding of
organisational change, Journal of Applied Behavioural Science, 47(2), 204-235
45. JISC-funded projects: 2011-2014
Other ways of spreading the principles-based discourse
Queens’ University Belfast: principles to baseline current
practice and envision future (Appreciative Enquiry) – planned
conversations about strategy, values and mission
Bath-Spa and Winchester – Student Fellows facilitated
conversations with academics about principles
Exeter and Hertfordshire – principles used in toolkits to support
decisions about technology application – discourse of technology
and discourse of principles brought together
Exeter – student employability agenda linked to assessment
principles, seen as a way of realising it
46. Way of addressing change in complex organisations
Focuses on meaning-making and the social construction
of reality
Gives a single focus for all educational change
activities
About back-stage processes as much as front-stage
events
Identifies discourse as a driver for change not just as a
symptom
There are already discourses about assessment and
feedback but are these research-informed?
Why important?
47. Addressing the barriers to change
Lack of shared frame of reference to guide
innovations: the big idea and principles
Difficulty defining good educational practice
Isolation of academics from educational research:
research synthesised into key ideas for action
Disciplinary differences – tight-loose structure of
principles
Weak links between local practices and educational
policies and strategies – single set of educational
ideas, tying each course design to policy framework
Challenge of getting multi-stakeholder buy-in
Cultures and micro-cultures
49. Further resources
Nicol, D (2009) Transforming assessment and feedback: enhancing
integration and empowerment in the first year. Scottish
Enhancement themes publication http://tinyurl.com/ca7dygx
Nicol, D. (2012) Transformational change in teaching and learning:
recasting the educational discourse, Evaluation of the Viewpoints
project at the University of Ulster. Funded by JISC UK. July 22nd
[also provides a detailed description of workshop with variations]
http://www.reap.ac.uk/TheoryPractice/Principles.aspx
Viewpoints Resources: provides presentation template, workshop plan,
principles cards, timeline etc that can be downloaded.
http://wiki.ulster.ac.uk/display/VPR/Home
Grant and Marshak (2011) Towards a discourse centred understanding of
organisational change, Journal of Applied Behavioural Science, 47(2),
204-235
See also other resources on REAP website at www.reap.ac.uk especially
page at www.reap.ac.uk/TheoryPractice/Principles.aspx
50. References
Nicol, D (2012) Transformational change in teaching and learning: recasting
the educational discourse, Evaluation of the Viewpoints project at the
University of Ulster. Funded by JISC UK. July 22nd [This document shows how
the principles developed in Nicol, 2009 were turned into a toolkit and used to
spread a new discourse across a whole HE institution.]
Nicol and Draper (2009) A blueprint for transformational organisational change
in higher education: REAP as a case study available at:
http://www.reap.ac.uk/TheoryPractice/Principles.aspx
Nicol, D. (2009) Transforming assessment and feedback: enhancing integration
and empowerment in the first year. Published by the Quality Assurance
Agency for Higher Education. [This document provides a list of 12 principles,
the research rationale for each, an explanation , some examples of how they
could be implemented in different disciplinary contexts]
Nicol, D, (2009), Assessment for Learner Self-regulation: Enhancing
achievement in the first year using learning technologies, Assessment and
Evaluation in Higher Education, 34(3),335-352
Nicol, D, J. & Macfarlane-Dick (2006), Formative assessment and self-
regulated learning: A model and seven principles of good feedback practice,
Studies in Higher Education, 31(2), 199-218.