This document discusses issues related to testing writing ability. It emphasizes the importance of ensuring tests are representative, valid, and reliably scored. Representative tasks should sample all possible content domains. Multiple tasks increase validity by providing more opportunities to demonstrate skills. Scoring should be either holistic, considering overall impression, or analytic, separating different skill aspects. Proper scorer training and calibration are needed to reliably rate performance. The goals are to accurately measure writing as an independent skill while maintaining practicality and meaningfulness.

1. TESTING WRITING Carolina Salazar
Testing and Evaluation

2. THE TESTING PROBLEM
REPRESENTATIVE VALIDLY
SAMPLED
VALIDLY AND
RELIABLY
SCORED
REPRESENTATIVE

3. REPRESENTATIVE TASKS

4. SPECIFICATION OF ALL THE POSSIBLE CONTENT
Operations Types of
text Adressees Length of
texts
Topics Dialect Style
Expressing,
directing,
describing…
Form, letter,
fax, note…
Boss,
friends,
family…
1 page,
150
words…
Not
specialist,
relevant…
Standard
Formal,
informal…

5. REPRESENTATIVE SAMPLE OF THE SPECIFIED
CONTENT
Try to include a representative sample of tasks
MORE
TASKS
MORE
VALIDITY

6. EXAMPLES

7. CCSE (CAMBRIDGE CERTIFICATES IN
COMMUNICATIVE SKILLS IN ENGLISH)
FIRST STAGE: READING
TOPIC: Working in a Summer Camp for
Children in America

8. TASK 1
Write a letter to American Summer Camps.
Find out about
 Dates
 Hours of work
 Type of accomodation
Ask for the application form

9. TASK 2
Fill in the application form.

10. TASK 3
You are now working in the ASCFC. You
write a postcard to an English-speaking
friend
On your postcard tell your friend:
 Where you are
 Why you are there
 Two things you like about the summer camp

11. TASK 4
You have arranged to go out tonight
with two other Helpers at the ASCFC in
Florida. You have to change your plans
suddenly, and cannot meet them. Leave
them a note.
 Apologize and explain why you cannot meet
them
 Suggest a different day to go out

12. ANALYSIS
Representative
sample of
tasks?
Content
validity?
Practical?

13. THE TESTING PROBLEM
REPRESENTATIVE VALIDLY
SAMPLED
VALIDLY AND
RELIABLY
SCORED
VALIDLY
SAMPLED

14. VALID SAMPLE OF WRITING ABILITY

15. SET AS MANY SEPARATE TASKS AS FEASIBLE
People’s performance  Consistent
Offer them more opportunities to start
Desirable Practical

16. TEST ONLY WRITING, NOTHING ELSE
NO YES

17. TEST ONLY WRITING, NOTHING ELSE
READING
ILLUSTRATIONS
SERIES OF PICTURES
GRAPHICS – CHARTS
NOTES

18. THE TESTING PROBLEM
REPRESENTATIVE VALIDLY
SAMPLED
VALIDLY AND
RELIABLY
SCORED
VALIDLY AND
RELIABLY
SCORED

19. VALID AND RELIABLE SCORING

20. ENSURE VALID AND RELIABLE SCORING
Set tasks which can be reliably scored
Set as many tasks as posible
Restrict candidates
Give no choice of tasks
Ensure long enough samples
Create appropriate scales for scoring

21. HOLISTIC SCORING
“Impressionistic”
Overall impression
Purpose of the test
PRO
Very rapid
CON
Scorer reliability?
http://www.ets.org/toefl/pbt/scores/writing_score_guide/

22. ANALYTIC SCORING
http://www.ode.state.or.us/wma/teachlearn/testing/scoring/gui
des/2011-12/wriscorguide_eng_no-dates.pdf
Separate score – separate aspects
PROS
Uneven development of subskills
Scorers consider details of performance
More reliable
CONS
Time consuming
Too specific (The whole is greater than the
sum of its parts)

23. HOLISTIC OR ANALYTIC?
PURPOSE CIRCUMSTANCES
ACCURACY RESOURCES
DECISIONMULTIPLE SCORING

24. MORE CONSIDERATIONS ON SCORING

25. SOME CONSIDERATIONS
Calibrate the scale
 Collect samples of performance
 Cover the range of scales
Select and train scorers
 Native speakers
 Sensitive to language
 Experienced
 Training (3 stages)
Follow acceptable scoring procedures
 Score each task independently
 Several scorers
 Identify discrepancies
Give feedback
 Useful – positive
 Decide the content during callibration

26. CONCLUSIONS

27. CONCLUSIONS
Pay attention to validity and reliability
Give importance to writing as an independent skill
Always be meaningful and enhance real communication

28. SOURCES
http://www.ode.state.or.us/wma/teachlearn/testing/scoring/guides/2011-
12/wriscorguide_eng_no-dates.pdf
http://www.ets.org/toefl/pbt/scores/writing_score_guide/