Ines Paland Riedmuller, Claudia Pop: Assessing oral proficiency

Eaquals International Conference, Lisbon, 21 – 23 April 2016
Assessing Oral Proficiency
Ines Paland-Riedmüller,
Deutsch-Uni Online
Claudia Pop,
TestDaF-Institut,
g.a.s.t. e.V. Germany
www.eaquals.org

Content of the Workshop
 Specifications of speaking
 The basics of assessment: Purposes,
methods, formats
 The subjectivity of assessment
 Quality management tools
 What is viable for your institution?

articulating
(phonetic)
phrasing /
formulating
(linguistic)
planning &
organising
(cognitive)
Specifications of speaking
Complexity
Pronun-
ciation
Time

The basics of assessment: Purposes
• Summative assessment
• used at end of a certain unit of the learning process
• ”assessment of learning”
• Formative assessment
• used continuously, during the learning process
• ”assessment for learning”

The basics of assessment: Purposes
• Reference group orientated assessment
• comparing the test outcome with the group
outcome
• ranking, e.g. course
• Criteria orientated assessment
• comparing the outcome directly with the aspects
intended to be measured

The basics of assessment: Methods
• Holistic/global scoring
• assessing the performance as a whole
• one general mark
• Analytic/profile scoring
• decomposing the performance according to
several aspects of interest
• several marks

The basics of assessment: Formats
 Direct assessment
 face-to-face interview
 one or more candidates
 one or more raters
 Semi-direct assessment
 recorded tasks
 recorded performances

The basics of assessment: Formats
situation face to face, media transmitted
real communication, simulated communication
partners interlocutor, assessor, co-candiate(s)
real person, avatar
activities production, interaction,
spontaneous, reflected
number of
partners
monologue, dialogue, multilogue
input
material
aural, written, visual, graphical
rating simultaneous, posterior
human, (semi-) automated

Which aspects actually influence
our
decision making?
An example

Is the grey
surface of object
X in shape and
size identical
with the grey
surface of object
A, object B or
object C?
X

X
Is the grey
surface of object
X in shape and
size identical
with the grey
surface of object
A, object B or
object C?


X
Is the grey
surface of object
X in shape and
size identical
with the grey
surface of object
A, object B or
object C?

X

Is the grey
surface of object
X in shape and
size identical
with the grey
surface of object
A, object B or
object C?

We have built-in perceptions of
geometrical forms.
How much stronger might this effect be
in assessing learners´ proficiency?

The subjectivity of assessment
• Rudolf Weiss, 1965
• main question: How large is the variability in
teachers´ assessment?
• rating of one student´s essay by 92 teachers
independently, criteria:
• correctness
• style
• content
• general mark

Mark Correctness Style Content
1 7% 24% 26%
2 28% 41% 47%
3 39% 19% 20%
4 22% 14% 7%
5 4% 2% 0%
Weiss, 1965

-10%
0%
10%
20%
30%
40%
50%
60%
1 2 3 4 5
10%
45%
35%
10%
0
General mark
Weiss, 1965

Replication of the Weiss-study: Birkel & Birkel, 2002
0
10
20
30
40
50
60
1 2 to 3+ 3 to 4+ 4 to 5+ 5
1
18
51
16
2
Birkel & Birkel, 2002

Assessment “errors”:
• halo effect
• contamination effect
• serial effect
• central/ extreme tendency
• personal dispositions

Serious reliability problem
– what can we do?
 Standardise!

Quality management tools
 trainings
 calibration material
 monitoring system
 rating grids

Tools: Trainings
• Face-to-Face-Interview
• standardise the procedure
• discuss appropriate behaviour
• analyse recorded material
• practise: Role play

Tools: Trainings
• Assessment
• recollect the goal (construct)
• introduce criteria
• operationalise the criteria
• assess in groups and discuss the
results
• raise error awareness
• discuss problems

Tools: Calibration material
• Guidelines
• manual, script, interlocutor frame
• best practice example of a face to face
interview
• Bench marks
• typical performances for each level
• calibrated assessment (expert group)
• framework of expectation

Tools: Monitoring
• two or more raters for the same
performance
• (peer) observation
• video analysis
 constructive feedback

Tools: Rating grids
• “dare” to modify the CEFR according
to your context and aim of
assessment
• e.g. ALTE-Materials for the Guidance
of Test Item Writers (module 4)

Descriptors should be
• positive
• short
• accurate/ precise
• independent of each other

Tools: Rating grids
• “dare” to modify the CEFR according
to your context and aim of
assessment
• e.g. ALTE-Materials for the Guidance
of Test Item Writers (module 4)
• evaluate and modify your grid before
introducing it

Feasible?
• What is already done in your
institution?
• Which additional tools do you think
might be suitable?

Thank you
claudia.pop@testdaf.de
paland@deutsch-uni.com

Ines Paland Riedmuller, Claudia Pop: Assessing oral proficiency

Recommandé

Recommandé

Contenu connexe

Similaire à Ines Paland Riedmuller, Claudia Pop: Assessing oral proficiency

Similaire à Ines Paland Riedmuller, Claudia Pop: Assessing oral proficiency (20)

Plus de eaquals

Plus de eaquals (20)

Dernier

Dernier (20)

Ines Paland Riedmuller, Claudia Pop: Assessing oral proficiency