Reading 2 - test specification for writing test - vstep

1. Test specifications for Writing test - VSTEP
1. General information:
 Time: 60 minutes
 General description of the test: the test which is composed of 2 parts aims at
assessing students’ written communicative competence. It targets at the test takers
at level 3 to level 5 CEFR-VN levels.
 Task demands and response format:
o Input: in written form, with the topic written in the target language at level
3 of VSTEP. The topics are familar.
o Expected output: a letter serving different purposes of communication (task
1) and an essay giving information, expressing and defending opinions
(task 2).
 Description of each task:
o Task 1: Time limit: 20 minutes. Length: 120 words at least. Weighting:
10/30. Test takers write a(n) letter/ email in response to a(n) given
letter/email, or test takers write a(n) letter/email serving different
communication purposes. This task tests written interaction skills.
o Task 2: Time limit: 40 minutes. Length: 250 words at least. Weighting:
20/30. Test takers write an essay on a given topic, using his/her experience
and knowledge to support his/her arguments. This task tests written
production skills.
 Number of items: 2
 Score reporting plan: A raw score out of 30 is converted to a score in 10-point
scale

2. Task demands and response
format
Time
recomme
nd-ation
Intended operations Task types Instructions
Task 1 Input:
 Part of a(n) letter/email about
an event (80-90 words)
related to a familiar, everyday
situation. or
 A familiar, everyday situation
(80-90 words) that requires
correspondence
Expected output:
 A (reply) letter/email (at least
120 words)
Content knowledge:
familiar matters regularly
encountered in work, school,
leisure; topics which are
familiar, or of personal interest.
20
minutes
 Can convey information
and ideas on abstract as
well as concrete topics
(e.g work, study or films,
music), check information
and ask about or explain
problems with reasonable
precision.
 Can write personal letters
describing experiences,
feelings and events in
some detail.
 a(n) letter/
email in
response to
a(n) given
letter/
email
 a(n) letter/
email asking
for
information,
describing
experiences,
feelings and
events.
Test takers are asked to write
a(n) letter/ email in response to
a(n) given letter/email or to
give/ask for information.

3. Task 2 Input:
 A given text/an extract from a
book/an article on a given
topic (50-60 words). The
topics may be related to
Vietnamese context.
Output: (at least 250 words)
 An essay evaluating/giving
opinions
 An essay explaining
advantages/disadvantages
 An essay explaining
causes/effects
 An essay explainin
problems/solutions
Content knowledge:
Education, health,
entertainment, transportation,
history, art, etc
40
minutes
 Can write an essay
supporting arguments,
highlight main ideas and
supporting them with
relevant details.
 Can write an essay to
develop an idea, giving
reasons for arguments,
explaining the advantages
and disadvantages.
An essay Test takers are asked to write an
essay on a given topic, using
his/her experience and
knowledge to support his/her
arguments.

4. 2. Sample instructionand items:
Task 1:
You should spend about 20 minutes on this task.
You have invited an American businessman, Mr. Noah Watson, to give a talk at your English
club monthly meeting. Read part of the letter from him below.
I would be glad to come and give a talk to your English club. In order to make a good
preparation, I would like some information about my audience. I could talk about either my
experience when starting up my company or some general rules when doing business. Which
one would be more interesting?
Also, I am staying in a hotel in the city centre. Could you tell me where the meeting is and
how to get there?
I look forward to meeting you soon.
Write a letter responding to Mr. Noah Watson. You should write at least 120 words. You are
not allowed to include your name. Your response will be evaluated in terms of Task fulfilment,
Organization, Vocabulary and Grammar.
Task 2:
You should spend about 40 minutes on this task.
Read the following extract from a book chapter in 1 minute: Read the following extract from a
book chapter in 1 minute:
Now, write an essay in 40 minutes to discuss what measures in your opinion are more effective
in controlling the crime rate.
Give reasons for your answer and include any relevant examples from your own knowledge or
experience. Your response will be evaluated in terms of Task fulfilment, Organization,
Vocabulary and Grammar.
You should write at least 250 words.
There are fewer violent crimes in some countries in recent years, but the overall number of crimes is still
very high. People have different opinions about the best way to control the crime rate. Some suggest
preventing people from an/ any illegal action before they do it while others prefer punishing criminals
more heavily.
Adapted from Academic Listening Encounters - Life in Society, Chapter 8, Page 113-114

5. Part 2: Guidelines for the design of effective writing prompts
2.1. General principles indesigning effective writing prompts
2.1.1. Clearly define the intended writing skills to be tested by the item.
Decide which of the following intended writing skill lends itself better to be assessed by an
essay question.
Example A
You should spend 40 minutes on this task. Read the following text from a book about tourism.
Tourism has become one of the fastest growing industries in the world. Millions of
people today are travelling farther and farther, not only in the summer but all year round.
Some people argue that the development of tourism has had negative effects on local
communities; others think that its influences are positive.
Express your opinion on the issue. Include reasons and relevant examples to support your
answer.
Example B
You should spend 40 minutes on this task. Read the following text from a book about tourism.
Tourism has become one of the fastest growing industries in the world. Millions of
people today are travelling farther and farther, not only in the summer but all year round.
Some people argue that the development of tourism has had negative effects on local
communities; others think that its influences are positive.
Write an essay to discuss the effects of tourism on the tourist destinations. Include reasons and
relevant examples to support your answer.
Example B is more useful for guiding the development of an essay question.
Specific intended writing skills are crucial to designing effective writing questions. If the
expected outcome to be assessed lacks clarity and specificity, the writing question meant to
assess students’ performance of a skill will likely assess something other than what is
intended. Example A is too general to provide clear guidance in answering the question. The
directive verb, “express” is vague and difficult to assess. It is especially unclear about what
form of the answer should be. Meanwhile, example B explicitly requires candidates “to write”
and the desired form of the answer is “an essay”.
In specifying the intended writing skills, item writers should clarify the performance that
students should be able to demonstrate as a result of what they have learned. Tested writing
skills meant to guide the development of test items will typically begin with a specific
directive verb. The statement will describe the observable behavior, action or outcome that
students should demonstrate.

6. 2.1.2. Clearly define and situate the task within a problem situation.
a. Define the task
With some writing questions, test takers can feel like they have an infinite supply of lead to
write a response on an indefinite number of pages about whatever they feel happy to write
about. This can happen when the writing question is vague or open to numerous
interpretations. Remember that effective writing questions: provide candidates with an
indication of the types of thinking and content to use in responding to the question.
A clearly defined task will specify the performance test takers should exhibit when responding
to the writing question. A clearly defined task is composed of a directive verb and the object of
that verb. For example, consider the following tasks:
Limiting the subject matter for a given question depends on different factors. Among other
things, how to restrict the task can depend on the hierarchy of knowledge related to the
question. The following example illustrates the process of limiting the scope of the task for a
given essay question.
Example of an Evolving Essay Question that Becomes More Focused
1. Less focused
essay question:
Evaluate the impact of the social networking sites.
2. More focused
essay question:
Evaluate the impact of the social networking sites on the young
generation.
Failure to establish adequate and effective limits for the response to the writing question
allows test takers to set their own boundaries for their response, meaning that they might
provide responses that are outside of the intended task, that are too long, or that only address a
part of the intended task. Test takers’ failure to answer within the intended limits of the essay
question can be ascribed to poor or ineffective wording of the task to some extent. Therefore,
item writers are responsible for writing questions in such a way that they provide test takers
with clear boundaries for responses. In short, item writers should avoid indeterminate
questions.
One way to avoid indeterminate questions is to stay away from vocabulary that is ambiguous.
Moreover, item writers should also avoid including vocabulary that is too advanced for the test
takers. The suggested language level for both tasks are B1 (CEFR) vocabulary which may
account for 80-90% of the words used in the prompts.
b. Create a problem situation

7. Once the writing task is clearly defined and the content that test takers are to use in
accomplishing the task has been delimited, the question is ready to be situated in a problem.
Both the tasks and the problems are key elements of writing questions.
The problem in writing questions includes the unsettled matter or undesirable state of affairs
that needs to be resolved. The purpose of the problem is to provide the students with a context
within which they can demonstrate the writing performance to be assessed.
Writing problem situations requires creativity and clarity. The circumstances should be clear in
the sense that even a minimum test taker of level 3 can identify the described event and its
related issues . At the same time, the problem situation should not be completely foreign to the
test takers’ experience.
Example A: Analyze the task and the problem of the following writing question.
You should spend 20 minutes on this task
You received an email from your English pen friend, Jane, to ask information about one of
your friends. Read part of her email below.
...I’ve just got an email from your friend, Mai. She is going to do a course in London t his
summer and wants to rent a room in my house. Could you tell me a bit about her
(work/study, personality, hobbies and interests) for me to see if she would fit in with my
family?
…
Write an email responding to Jane. You should write at least 120 words. Do NOT include your
name.
2.1.3. Present a reasonable task.
When defining the task for a writing question, item writers need to make sure that they
present a reasonable task to test takers. One of the challenges that item writers face in
composing essay questions is that because of their extensive experience and background
knowledge they may be tempted to demand unreasonable content expertise from the test
takers. Hence, item writers need to make sure that their test takers can “be expected to have
adequate material with which to answer the question” (Stalnaker, 1952, p.520). The writing
topics should be familiar and accessible to all candidates.
Identifying an audience for test takers to write their responses to may also be helpful. This
would make the task more authentic. A key element in writing communication is to clearly
identify and write to a specific audience.
2.1.4. Specify the relative point value, the approximate time limit and word limit in clear
directions.

8. Specifying the relative point value and the approximate time limit helps test takers allocate
their time in answering several writing questions in a writing test because the directions clarify
the relative merit of each question. Without such guidelines test takers may feel at a loss as to
how much time to spend on a question. When deciding how much time should be spent on a
question, the slower candidates can have a clearer plan for their response.
Also make sure that test takers can be realistically expected to provide an adequate answer in
the given and/or the suggested time.
2.1.5. Specify the assessment criteria.
The expected response should be described clearly in terms of the criteria that will be used to
judge it. For example, a letter-writing task will be scored without consideration of grammatical
accuracy, as long as the response is comprehensible, examinees should be told as much. On
the other hand, if an essay will be scored on the basis of grammatical accuracy, vocabulary
use, content and organization, those categories should be disclosed to the test takers in the
directions.
2.1.6. Improve the essay question through preview and review.
The following steps can help item writers improve an item before and after they hand it out to
test takers, making the question much more effective.
Preview (before handing out the essay question)
a. Predict responses.
Try to respond to the question from the perspective of a typical candidate. Evaluate whether
candidates have the content knowledge and the skills necessary to adequately respond to the
question. Repair any possible weaknesses of the essay question before handing out the exam.
b. Write a model answer.
Before using a question, write model answer(s) or at least an outline of major points that
should be included in an answer. Writing the model answer allows reflection on the clarity of
the essay question. Furthermore, the model answer(s) serve as a basis for grading responses.
Once the model answer has been written compare its alignment with the question and the
tested writing skills and make changes as needed to assure that the tested writing skills, the
essay question, and the model answer are aligned with each other.
c. Ask a knowledgeable colleague to critically review the question, the model answer, and the
tested writing skills for alignment.
Before using the essay question on a test, ask a person knowledgeable in the subject to
critically review the essay question, the model answer, and the the tested writing skills to

9. determine how well they are aligned with each other. Based the tested writing skills, revise the
question as needed. By having someone else look at the test the likelihood of creating effective
test items is increased. Try asking a colleague to evaluate the essay questions based on the
guidelines for constructing essay questions contained in this guideline.
Review (after receiving the responses)
After test takers complete the test, carefully review the range of answers given and the manner
in which test takers seem to have interpreted the question. Make revisions based on the
findings. Writing good questions is a process that requires time and practice. Carefully
studying the responses can help evaluate test takers' understanding of the question as well as
the effectiveness of the question.
2.2. Principles indesigning writing prompts for Task 1 and Task 2
Guidelines for Task 1
a. Components of an item
There are three parts in an item of Task 1: Instruction, Letter extract and Task requirement.
 Instruction suggests how much time candidates should spend on Task 1 and includes
one or two sentences in italics, which often introduce the name of the sender and
explain the relationship between the sender and the responder.
 Letter extract is part of an email or a letter placed in textbox.
 Task requirement explains the expected output from the candidates, the minimum
number of words in the target response, and how candidates’ answers are assessed. The
Task Requirement is virtually the same in all items in Task 1, except for the name of
the sender.
 Prompt includes both the instruction in italics and the letter extract.
b. Principles in writing Task 1 prompt
As Task 1 in the Writing Test aims at examining candidates’ ability in writing correspondence,
the expected response is in the form of an email or a letter. When designing questions for Task
1, item writers should select a familiar topic, one which is regularly encountered in work,
school or leisure, or of personal interest. The criterionis to make sure that even candidates
around Level 3 will not be jeopardized by a less familiar topic.
Furthermore, a letter extract needs to be written in a consistent style, using either formal or
informal language throughout. This will help to set the tone of the expected response. A letter

10. extract in Task 1 has approximately 80-90 words. All fully-developed responses in Task 1
should allow candidates to communicate three main ideas, which means the letter extracts
should raise three questions to be answered, or address three problems that need to be settled,
or a combination of these.
Another requirement is concerned with the complexity of language, including lexis and
sentence structures to be used in the instructions and letter extracts. Similar to the requirement
about topic familiarity, item writers should design with the level of candidates at the lowest
level (around Level 3) in mind. For all prospectus candidates to be able to comprehend the
content of a prompt, it is suggested that 80 to 90 percent of the lexis and syntax used
throughout the prompt corresponds to Linguistic competence Level 3 descriptors. Item writers,
therefore, can refer to Vocabulary English Profile (http://vocabulary.englishprofile.org/) or
Cambridge Dictionaries Online (http://dictionary.cambridge.org/) to check the level of a
certain vocabulary item.
To make it consistent across items in the item bank, a prompt needs to provide information
about: (i) the sender; (ii) the relationship between the sender and the responder; (iii) the topic
of the letter/email extract; and (iv) the context of the letter/email (why this letter/email is being
sent, what is expected from the responder). A prompt also needs to clearly indicate what
communication purpose(s) is expected in the response (such as giving direction, asking
for/giving advice, making enquiries, thanking, accepting/refusing invitation, …).
Guidelines for Task 2
a. Components of an item
Similarly to Task 1, there are also three parts in an item of Task 2.
 Instruction suggests how much time candidates should spend on Task 2 and introduces
the genres of the prompts.
 Prompt is often an extract from a book, part of a magazine article or journal article, etc.
Prompts are placed in a textbox.
 Task requirement indicates the expected response from the candidates and explains
what candidates are required to write about. The last paragraph in Task requirement
suggests how candidates can support their argument, the minimum number of words in
their responses, and lists criteriaagainst which candidates’ answers are assessed. The
last paragraph is the same in all items in Task 2.
b. Principles in writing Task 2 prompt
As Task 2 in the Writing Test aims to assess candidates’ ability of written production, the

11. target response is an essay. Topics in Task 2 include more general issues, such as education,
health, entertainment, transportation, etc.
A prompt in Task 2 should be written in academic language, with the limit of approximately
50 to 60 words. Following the prompt, Task requirement should have 40 to 50 words. Like
Task 1, items in Task 2 should use lexis and syntax that accommodate candidates around
Level 3.
Task requirement (an essay to evaluate/ to give opinion/ to explain advantages and
disadvantages/ to discuss problem and solutions …) has to be clearly indicated.
Prompts are often written in compound sentences and follow a consistent structure. The first
one or two sentence(s) in a prompt needs to introduce the topic and provides the background
for a certain issue. The next one or two sentences can serve either of the following functions:
 to list different opinions about the topic;
 to introduce a certain problem; or
 to introduce different aspects of the topic.
REFERENCES
Andrade, H. and Du, Y. (2005), “Student perspectives on rubric-referencedassessment”,
Practical Assessment, Research & Evaluation, Vol. 10 No. 5, available at:
http://PAREonline.net/getvn.asp?v¼10&n¼3 (accessed December 14, 2006).
Arter, J. and McTighe, J. (2001), Scoring Rubrics in the Classroom, Corwin, Thousand Oaks,
CA.
Boulet, J.R., Rebbecchi, T.A., Denton, E.C., Mckinley, D. and Whelan, G.P. (2004),
“Assessing the written communication skills of medical school graduates”, Advances in
Health Sciences Education, Vol. 9, pp. 47-60.
Bresciani, M.J., Zelna, C.L. and Anderson, J.A. (2004), Assessing Student Learning and
Development: A Handbook for Practitioners, National Association of Student Personnel
Administrators (NASPA), Washington, DC.
Brown, G., Bull, J. and Pendlebury, M. (1997), Assessing Student Learning in Higher
Education, Routledge, Oxon and New York, NY.
Callison, D. (2000), “Rubrics”, School Library Media Activities Monthly, Vol. 17 No. 2, pp.
4-36, 42.

12. Campbell, A. (2005), “Application of ICT and rubrics to the assessment process where
professional judgement is involved: the features of an e-marking tool”, Assessment
&Evaluation in Higher Education, Vol. 30 No. 5, pp. 529-37.
Carr, Nathan T. (2011). Designing and Analyzing Language Tests. Oxford: Oxford University
Press.
Cambridge ESOL (2011). Using the CEFR: Principles of good practice. Retrieved from
http://www.cambridgeenglish.org/images/126011-using-cefr-principles-of-good-practice.pdf
Council of Europe (1992). Transparency and coherence in language learning in Europe:
Objectives, evaluation, certification. Report on the R¨uschlikon Symposium. Strasbourg:
Council of Europe.
Fraser, L., Harich, K., Norby, J., Brzovic, K., Rizkallah, T. and Loewy, D. (2005), “Diagnostic
and value-added assessment of business writing”, Business Communication Quarterly, Vol. 68
No. 3, pp. 290-305, available at:
http://bcq.sagepub.com/cgi/content/abstract/68/3/290(accessed May 2, 2007).
Gopinath, C. (2004), “Exploring effects of criteria and multiple graders on case grading”,
Journal of Education for Business, Vol. 79 No. 6, pp. 317-22.
Little, D. (2011). The common European framework of reference for languages: A research
agenda. Language Teaching, 44(3), 381-393.
Reiner, C., Bothell, T., Sudweeks, R. & Wood, B. (2002). Preparing effective essay questions.
New York: New Forums Press.
Shaw, S. & Weir, C. (2008). Examining Writing. Cambridge: Cambridge University Press.