The document discusses the nominal group technique (NGT) consensus development method. NGT is a structured group process that allows a group to generate ideas and prioritize solutions. It involves individuals privately generating ideas which are then shared anonymously and discussed before private voting to rank items. The document provides examples of NGT being used to evaluate aspects of healthcare education programs, such as modules and feedback. Benefits identified are that it prevents domination, ensures equal participation, increases productivity, and provides immediate prioritized results.
Transaction Management in Database Management System
Evaluating Learning with NGT
1. The Use of the Nominal Group
Technique Consensus Development
Method for the Evaluation of
Learning, Teaching & Assessment
Heather Gray ProfD MSc FHEA
Brenda Bain MPhil FHEA
Sivaramkumar Shanmugam PhD MSc FHEA
2. Nominal Group Technique (NGT)
Consensus development method
(Van de Ven & Delbecq, 1972)
Facilitates groups in ideas generation,
decision-making & priority setting
3. Competences required of
physiotherapy students during
first-year placements
(Steward, 2001)
Occupational health nurses’
perceptions of their education &
training needs
(Bamford & Warder, 2001)
Factors affecting quality & capacity
in clinical education in radiography
(Williams et al, 2006)
Evaluation of a special study module
in literature & medicine
(Lancaster et al, 2002)
Evaluation of the teaching &
assessment of mental health nurses’
inter-personal skills
(Perry & Linsley, 2006)
Learners’ perceptions of a medical
education course’s strengths and
weaknesses
(Dobbie et al, 2004)
NGT Examples in Healthcare Education
4. Sample NGT Questions from Physiotherapy
What were the most beneficial aspects of the
Vocational Rehabilitation module?
What issues do international students on the
MSc Physiotherapy programme have with
written coursework feedback?
How could your self-directed learning be
improved in the Neurorehabilitation module?
5. Stages of NGT Group Meeting
1. Private
generation of
responses
2. Master list
compilation
3. Item
clarification
4. Merging of
similar items
5. Silent private
ranking of top
5 items
6. Ranked
results shared
with the group
6. Benefits of NGT for Participants
Prevents dominant
members
controlling
Ensures
relatively equal
participation
Increases
group
productivity
Mitigates status
of group
members
Protects
participants’
views
7. Benefits of NGT for Group Facilitator
Superior to
focus
groups
Prioritised
list
generated
Immediate
results
available
Results can be
compared to
other groups
Qualitative &
quantitative
results
Cost
effective
8. Most Beneficial Aspects of Vocational
Rehabilitation Module
Rank
Position
Topic Median
Rank
1 Coverage of the Systems Approach 4.0
2
How to carry out a workplace assessment
(theory)
3.5
3
Conducting a real-life workplace assessment &
presenting findings
3.0
4 Provision of workbook 3.0
5 Role play of work-focussed interview 2.0
9. Issues with Written Assessment Feedback
for International MSc Students
Rank
Position
Topic Median
Rank
1 Lack of clarity of feedback provided 5.0
2 Late timing of feedback 4.0
3 Confusion regarding coursework presentation 2.0
4 Lack of clarity in assignment instructions 1.0
10. Suggestions for Improving Self-Directed Learning
in Neurorehabilitation Module
Rank
Position
Topic Median
Rank
1 To provide a framework for analysing video-clips 4.5
2 To provide answers for video-clip case studies 4.0
3 Access to PowerPoint slides prior to classes 2.5
4 To make directed reading more specific 2.0
5
Provide access to tutorial PowerPoint slides
immediately after class
1.0
11. My Top 5 Reasons for Using NGT
1. User guides available
2. Time efficient
3. Solutions focussed
4. Transparent results
5. Students & staff like it!
12. Bedtime Reading
Delbecq, A. 1975. Group techniques for program planning: a guide to
nominal group and Delphi processes. Glenview Ill., Scott Foresman.
Dunham, R.B. 1998, Nominal group technique: A users' guide [online].
Available at: http://www.peoplemix.com/documents/general/ngt.pdf
Fox, W.M. 1993, "The improved nominal group technique (INGT)", Journal of
Management Development, Vol. 8, no. 1, pp. 20-27.
Varga-Atkins , V., Bunyan, N., Fewtrell, R., McIsaac, J. 2011 The NGT– a
practical guide for Facilitators, Liverpool University. Available at:
http://www.slideshare.net/tundeva/the-nominal-group-technique-a-
practical-guide-for-facilitators
13. The Use of the Nominal Group
Technique Consensus Development
Method for the Evaluation of
Learning, Teaching & Assessment
Heather Gray ProfD MSc FHEA
Brenda Bain MPhil FHEA
Sivaramkumar Shanmugam PhD MSc FHEA
Notes de l'éditeur
Good morning, my presentation today will outline the use of the nominal group technique as a method for evaluating learning, teaching and assessment. My plan is to provide an overview of the NGT, its uses and advantages, and supplement this with examples from the physiotherapy programme at Glasgow Caledonian University in Scotland. So, the nominal group technique uses structured face to face group sessions for the purposes of achieving consensus and action planning on a specific topic. It was developed in the late 1960s by Van de Ven and Delbecq, and it akin to having a face to face Delphi.The NGT originally arose in management science as a superior form of group brainstorming, and can be used to facilitate groups in the generation & prioritisation of ideas.The term ‘nominal group’ means that the session is only nominally in name a group, whilst it is relying on individual input in a group environment; the interactions between individuals during the sessions are actively discouraged (O‟Neil and Jackson 1983).
Curriculum development & teaching and learning evaluation Qualitative feel to the titlesEvaluation/ perceptions/ factors
What was good about the assessment process for this module?What solutions do you have for how your self-directed learning could be improved? What suggestions do you have to improve the Mental Health module?
NGT is highly structured and follows a step-by-step process that is outlined in detail by the original authors (Delbecq & Van de Ven, 1975) and revised further by Fox (1993), who describes the Improved NGT (INGT). When using NGT, two or more voting rounds are strongly recommended, as this increases the likelihood of resulting in some convergence of individual judgements,
NGT – a highly structured approach, which manipulates an essentially qualitative method of data collection to mimic a quantitative approach, and prevents dominant people from controlling the group. It also protects participants’ views from direct castigation or dismissal by other group members (Wellings, et al., 2000 in Aspinal, et al., 2006). Greater contribution from participantsImproved communication structure of groupsIncreased individual & group productivityNominal groups superior to focus groups (Van de Ven & Delbecq, 1974; Herbert & Yost, 1979)NGT originally arose in management science (Delbecq, Van de Ven & Gustafson, 1975) as a superior form of group brainstorming; superior in the sense of allowing and encouraging more people to contribute to a group's output (given the tendency of some participants to be silent or overridden in the presence of more articulate and/or powerful members), and also of "organising or structuring the group's interaction for efficiently reaching a joint solution to a problem" (Penley, 1978). Penley further suggests that NGT improves the communication structure of groups, leading to greater individual and group productivity.
Benefits for the facilitator: The results are immediately available at the end of the session. Cost-effectiveness – there is no need for transcription and analysis. Benefits for the stakeholders to whom findings are reported: Nominal group sessions are said to generate more ideas in terms of volume than other face-to-face group approaches, e.g. focus groups (Delbecq 1975; O‟Neil and Jackson 1983) Results immediately offer action planning in the form of a prioritised list. Results can easily be compared with one another if sessions are run with different kinds of stakeholder groups (in volume or with different kinds of groups e.g. with students, staff and other subject experts). Combines qualitative and quantitative methods, so can be attractive to stakeholders who quantitative results.
From 23 items suggestedItems that at least 50% of the group ranked – use median rank
Research question: Explore the perceived issues/problems of written summative coursework feedback for international students currently undergoing masters in Physiotherapy. NGT was chosen as opposed to focus groups because international students may not be very forthcoming and open in a focus group and NGT limits participant’s interaction to the minimum while maximising their input to the specific question being explored. The key issue identified by consensus is interpretation of the marking criteria as it is very vague and generic, and the interpretation made by both academic staff and students are variable. Other issues identified (ranks in bracket) through NGT are • Late feedback from the lecturers, not able to implement before the next coursework (11)• Different types of assignment - feedback is not transferable to other types of assignment (3)• Coursework presentation - markers don’t understand what we were saying, so marked low, not even related to the marking criteria (4)• Clarity of feedback- not able to understand what is expected (10)• Handwritten feedback, difficult to understand (5)• Vague feedback - not very specific on how to improve (2)• Instructions regarding assignment are not clear (4)
From 25 original suggestions
The Nominal Group Technique seems best suited to single topic evaluations or when items for action need to be identified (e.g. typically answering the question „How could this programme be improved?‟), and less suited to researching general experiences, or when a complex topic needs exploring (e.g. a typical research question could be „What are your experiences with XY teaching approach?‟) Typically provides a greater sense of closure than can be obtained through group discussion
Chapple, M., & Murphy, R. (1996). The Nominal Group Technique: extending the evaluation of students’ teaching and learning experiences. Assessment & Evaluation in Higher Education, 21(2), 147-160. Davis, D., Rhodes, R., & Baker, A. (1998). Curriculum revision: Reaching faculty consensus through the nominal group technique . Journal of Nursing Education, 37(7), 326-328. O’Neil, M. J., & Jackson, L. (1983). Nominal Group Technique: A process for initiating curriculum development in higher education. Studies in Higher Education, 8, 129-138.