This document discusses establishing speaking level benchmarks in an intensive English program. The motivation was that new teachers had difficulty assessing whether students were ready to progress to the next speaking level. The process involved collecting oral interviews from students who had progressed, then experienced teachers rated the interviews. The interviews were compiled into DVD benchmarks for each level. This provided objective criteria for teachers to use to assess students. The results were that new teachers felt the benchmarks helped them set goals and assess students more accurately. Plans for the future include continuing to use the benchmarks and exploring additional assessment methods.

1. Establishing Speaking Level Benchmarks in an Intensive English Program Sherry Warren University of South Carolina sherrytatiana@gmail.com sherry_warren@epi.sc.edu

3. The Process

4. Preparation

5. Structure

6. Technology

7. The Result

9. Accommodation “It is a natural feature of authentic conversation that as part of the negotiation of meaning speakers adjust their language to features of their interlocutor.” -- Richards and Malvern, 2000 Gass and Varonis, 1985

10. What are your primary concerns when dealing with assessment in an intensive English program? Time. Accuracy. Simplicity.

11. How does your program assess speaking ability? Do your students progress through speaking levels? Do teachers decide whether to promote students? Are new teachers given adequate training for this? Are teachers expected to be able to holistically assess their students’ speaking proficiency? If so, are they given the tools they need to do so accurately?

13. 9 weeks per term

15. Describing levels of proficiency The Motivation Student Profile At the beginning of the term, SL6 students are able to • comprehend, most English spoken at normal speed; • understand main ideas, some details, and important content vocabulary when listening to the radio and watching TV/movies; • speak with pronunciation that is comprehensible without repetition and has acceptable stress, rhythm, and intonation; • speak somewhat confidently but not always accurately; • state and support opinions on abstract topics with some degree of proficiency; • easily participate in everyday conversations with Americans; • sustain conversations (speaking in paragraphs) on a wide range of topics but fluency is uneven.

17. Grammar scores on the Michigan Placement Test

18. Writing Sample

19. For Reading-Vocabulary class:

20. Reading and Vocabulary scores on the Michigan Placement Test

21. For Speaking-Listening class:

22. Teacher’s score from Oral Interview

25. Benefits:

26. more class time for instruction

27. The teacher knows the student’s strengths and weaknesses

28. EPI’s recent expansion

29. New teachers had difficulty knowing whether a student was ready to move up

30. Is the upper half of the class ready to move up?

31. Is the best student in the class ready to move up?

33. Benefits:

34. more class time for instruction

35. The teacher knows the student’s strengths and weaknesses very well, so they were able to assess the students the most accurately

36. EPI’s recent expansion

37. New teachers had difficulty knowing whether a student was ready to move up

38. Is the upper half of the class ready to move up?

39. Is the best student in the class ready to move up

40. Pressure from students because of criteria for matriculationTeachers needed an objective means of assessing their students at the end of term in order to effectively make recommendations about whether to promote them to the next Speaking-Listening level

42. Assessment Criteria The Process Displays some features of the level below Displays some features of the level above

43. Assessment Criteria The Process EXCELLENT—VERY GOOD: Has control of vowels and consonants. GOOD—AVERAGE: Consistently produces most vowels and consonants ADEQUATE—SURVIVAL: Can combines vowels and consonants, but has difficulty producing certain sounds in specific situations

44. The Process Assessment Criteria

45. The Process Obtaining the Oral Interviews Teachers suggested a student from their classes who they felt was ready to move to the next level An experienced teacher gave the interview; the class teacher served as a second rater The interview lasted around 15 minutes and was videotaped Inter-rater reliability was high (90%) In the one case where there was a >%10 discrepancy between raters’ scores, a third rater scored the interview

46. Interviews used for the Benchmarks Interviews used: Students who comfortably placed in the next level at the end of term, based on the oral interview scores from the taped interview and the students’ scores on the MPT A mix of nationalities At least two 15 minute interviews composed each benchmark The students’ full score profile was also presented on the DVD, along with their placement score for SL The Process

47. Additional Steps taken to Ensure Homogeneity Across Speaking Level Classes The Process Teacher Retreat Discussed and revised goals for each speaking level Viewed oral interviews and discussed criteria for promoting students Decided to change the weight of pronunciation in the assessment criteria (still in process)

49. Technology used for the Benchmarks The Process Mac programs: iMovie iDVD PC program: Roxio Creator 2010 DVD-free alternative: Upload files to a simple website for easy distribution

51. Interaction between students during a communicative class activity can be used for assessment

52. New teachers now have a way of setting goals for their own students when teaching a new level

53. Teachers are provided the means to objectively assess their students and make decisions about whether to promote themThe Result

55. New teachers had a better sense of what their students needed to achieve by the end of term

57. Questions to think about when composing your own benchmarks What is your program’s approach to teaching speaking? What speaking assessment descriptors do you already have? Are your speaking courses geared to helping students achieve communicative competence or is there another goal?

59. Dimensions of proficiency (fluency, grammar, etc.)

60. Questions to think about when composing your own benchmarks How can teachers use the benchmarks to assess their students without losing class time?

61. Questions? Comments?