Wasc arc conf final, iswi, april 8 2011.pptx

WASC ARC Conference
April 8, 2011, San Francisco, CA

Improving Student Writing and

Strengthening Writing Programs –
Technology and Techniques That Work
Ellen Junn, Associate Provost
Jennifer Ivie, Assistant Professor of Psychology
Kim Morin, Professor of Theatre Arts
William Covino, Provost
California State University, Fresno

California State University, Fresno
(est. 1911)

•23-campus CSU system

•20,932 students

•1,197 faculty

Copyright © April 2011 2

Fresno State Demographics


Fresno State Demographics
§ First generation
college: 68%

§ English proficiency:
63% freshmen require
English remediation


Multi-pronged Approach to
Teaching Writing on our Campus
¨ English Composition
¨ Upper Division Writing Exam
¡ Graduation Requirement
¡ Substitution of Designated “W” courses
¨ Writing Requirement for ALL General
Education Courses
ú Requires “Iterative” writing assignments


The Dilemma
¨ High percentage of remedial students.

¨ Perception that surface errors distract
from content.

¨ Effective writing instruction requires
innovative pedagogies.


INSTRUCTORS KEEP
ASKING…
“Where Is
the Time to
Respond To
All of Those
Essays?”

TECHNOLOGY AS ONE
SOLUTION?
•Professor Kim Morin

•“E-scholar” Program

•UpperDivision online
GE courses

•Faculty interest across
campus

A DISCOVERY…
Computer Essay
Scoring Programs

Several College-level
Essay Scoring and
Writing programs
available…

COMPUTER ESSAY SCORING
PROGRAMS

• ETS® Criterion®
• http://www.ets.org/criterion/higher_ed/about
• IEA Intelligent Essay Assessor
http://www.knowledge-technologies.com/prodIEA.shtml
• SAGrader
• https://www.sagrader.com/sgm/features
• Pearson MyWriting Lab
• http://www.mywritinglab.com/whatis.html

ETS CRITERION® SELECTED
ON A TRIAL BASIS…

First Trial (Spring 2009):
30 students – 1 essay
assignment
First Pilot (Fall 2009)
1 Faculty member
100 students- 5 essay
assignments

About ETS Criterion®
• Students write and revise essays
online
• Diagnostic Feedback / Holistic Score
within 20 seconds.
• Topics Library provides prompts.
• Faculty can create topics/prompts.
• Errors are highlighted but not
corrected.


ABOUT ETS CRITERION®
• No instructor Fees
• Simple Registration
• Students purchase Access codes through
Bookstore
• Approximately $11.00 per student
• One fee provides student use in all classes per
term/semester
• Technical Support from ETS®

ABOUT ETS CRITERION®
• Bilingual feedback available
• Spanish, Japanese, Simplified
Chinese, Korean, ELL

• Advanced levels available
• College–1st & 2nd year, TOEFL, GRE

• No prior essays required

ETS CRITERION® Online Tour

http://www.ets.org/Media/Products/Criterion/tour2/critloader.html

Initial Presentation to Faculty

“Computer Essay Scoring has had a
positive effect so far.
I spend more time assessing content, less
on grammatical errors.
Students spend more time revising.
However, it still misses errors and does not
grade for content.”


Early Student Comments
¡ “I like the instant feedback.”

¡ “I like that it goes into depth about why you
didn't get the max score or why you did well.”

¡ “It is easy to use - just copy and paste!”

¡ “I love the fact that I can revise my work for a
better score.”

¡ “It is very helpful and I can see my growth as a
writer.”


Facts About College Student Writing
FACT # 1:
Student improvements in writing increase with more
practice
¨ specific, constructive feedback from faculty
¨ opportunity for revisions.

IMPLICATION:
ü Students need to write or revise more frequently
ü Examine policy requiring “iterative” writing in GE or
W courses

FACT # 2:
If students do not continue to practice,
their writing performance may actually deteriorate.

IMPLICATION:
ü Examine all writing programs across campus
ü Identify key courses with writing requirement
ü Target specific faculty teaching those courses


FACT # 3:
Teaching students to write
effectively can be
time-consuming and labor-intensive.

IMPLICATION:
ü Identify effective technology-related tools
ü reduce faculty workload
ü provide specific, timely feedback to students.
ü Target part-time faculty who teach writing
intensive courses.


FACT # 4:
Writing instruction involves faculty who are NOT
trained as writing teachers.

IMPLICATION:
ü Implement Criterion training for faculty.
ü Offer Writing Across the Curriculum workshops.
ü Provide professional development funds for all
faculty during Year 1 Pilot.


FACT # 5:
Assessing & documenting student learning outcomes in
writing performance are key elements to success.

IMPLICATION:
ü Ongoing data collected for past 3 semesters
ü Share results with faculty learning community to
determine Best Practices.
ü Modify training based on feedback and assessment.


Next Came…

ISWI Pilot Launch

Spring 2010

ISWI
Improving Student Writing
Initiative
November, 2009 - Campus email announcement
calling for faculty participants
§ Criterion® Training:
§ Faculty ISWI Coordinator
§ Faculty Learning Community (FLC)
§ Writing Across the Curriculum Workshops
§ Assessment of Criterion®


Campus-Wide Initiative
By The Numbers...
Year 1 (2 Semesters)
Spring 2010 / Fall 2010

• 349 Classes Involved
• 173 Instructors Used Criterion®
• 134 Spring 2010
• 68 Fall 2010
• 5,920 Students (Spring 2010)
• 3,756 Students (Fall 2010)
Numbers may include duplicates


Campus-Wide Initiative
By The Numbers...
Year 2 (1 Semester)
Spring 2011

• 231 Classes Involved
• 93 Instructors Used Criterion®
• 5442 Students (Spring 2011)
• 44,080 Essays Submitted (by March 15)
Numbers may include duplicates


Faculty Participation- Year 2
Out of 93 instructors opting to use Criterion, only
20 received Professional Development funds as
an incentive in Year 2.


Assessments from Spring 2010..
1. Analyzed Criterion® automatically generated
data for trends.

2. CLA scores for students with & without
Criterion®

3. First and last papers submitted by students
on Criterion® scored by independent faculty
panel

4. Collected student and April 2011 surveys
Copyright © faculty 28

Automatically-Generated Data

¡ Students who used Criterion® for revision
with more than one submission, scored
better than those who did not.

¡ Students with a larger number of
assignments and more submissions on
Criterion® increased their holistic score on
average by approximately 1 level.


Collegiate Learning Assessment
(CLA)
¡ A significant difference was found
between Criterion® users and non-
users on the CLA performance task.

¡ No significant differences were found
between the two groups on the
analytic writing task.


Faculty Scoring Panel

First and last papers submitted by students
on Criterion® scored by independent
faculty panel.

¨ Overall, student writing improved in
classes that used ETS Criterion®.
¡ The greatest improvements were made when
instructors engaged students in substantial
discussions of writing and how to use Criterion®.


Faculty Feedback –
Positive Comments
When using Criterion®, most faculty felt that:
1 . Papers were easier to grade.

2 . The program improved their students’

writing skills.
3 . Students spent more time revising written

assignments.
4 . Creating assignments was easy.


Faculty Feedback - Challenges

Many faculty felt that Criterion®
1. Did not reduce their workload.
2 . Did not do as much as they had
hoped.
3 . Was not worth the cost to the
students.


Student Survey Response


Student Survey Highlights

What students liked about Criterion®:
1. It was available on-line 24 hours a day.

2. It allowed them to correct grammatical
or mechanical errors before turning in a
paper.

3. It gave immediate feedback.


What students found challenging:

1. Criterion® identified technical terms or
citations as errors.

2. The program identified errors but did
not correct them.

1. The program did not grade for content.


Student Survey Response

Freshmen/sophomores were
significantly more likely to agree:
- Criterion has helped me improve my
writing.
- Criterion should suggest less and correct
more.
- I was able to apply what I learned to other
writing assignments.


ESL Student Survey Response


English as a Second Language
Student Responses
ESL students were more likely to respond
that Criterion® helped them improve their
writing
úPreferred Criterion® feedback to instructor
feedback.
úSubmitted their documents more frequently.

- I like the kind of feedback Criterion® gives.
- I spent more time improving my writing with
Criterion®.
- I wish I could use Criterion® for other classes.


English as a Second Language
Student Response

http://www.youtube.com/watch?v=pP6feJotLVM

Positive Comments From Students

¨ “Awesome program. I find it really useful.”
¨ “I like that it is really convenient and I
found myself using the program for all my
classes.”
¨ “I started noticing themes in my writing
that could use improvement.”


Ongoing Research
¨ Studies with Control Groups
¨ Continue Collecting Data
¨ Determine Best Practices / Uses


Conclusion
So Far, Evidence suggests-
¨ Criterion® helps improve student writing
when combined with effective instructor
practice.
¨ Criterion® appears useful for editing
grammar and mechanics.
¨ Students who use Criterion® are more
engaged with the writing process.


Next Steps
¨ Analyze and refine assessment data.
¨ Provide online video tutorials.
¨ Expand Professional Development &
training.
¨ Establish ISWI committee
ú analyze writing instruction across campus.
¨ Recognize and thank participating faculty.


What Have We Learned?
¨ Students generally found Criterion® to be easy
to use and cost effective.
¨ ESL students responded more positively to
Criterion® feedback.
¨ Faculty found Criterion® helpful when used to
complement instruction.
¨ ISWI shows the high degree of faculty interest
in improving student writing across campus.


ISWI Broadly Endorsed & Supported By:
¨ Writing Competency Subcommittee (subcommittee of Senate’s GE Committee)
¨ Professional Development Subcommittee (subcommittee of Senate’s Personnel
Committee)
¨ Chair of the GE Committee
¨ Office of Undergraduate Studies
¨ Division of Graduate Studies
¨ Institutional Research & Assessment Planning Director leading faculty ISWI Assessment
Team
¨ Provost’s Office as implemented by the Associate Provost through the Center for the
Scholarly Advancement of Learning and Teaching (CSALT) & Technology Innovations for
Learning and Teaching (TILT)


¨ William Covino, Provost ¨ Kim Morin, Professor
¨ wcovino@csufresno.edu ¨ Department of Theatre Arts
¨ 559-278-2636
¨ Artistic Director, Theatre for
Young Audiences; English/Drama
¨
Credential Advisor
¨ Ellen Junn, Associate Provost ¨ ISWI Faculty Coordinator
¨ Interim Director, Center for the ¨ kimm@csufresno.edu
Scholarly Advancement of Learning ¨ 559-278-4342
and Teaching (CSALT) ¨
¨ Interim Senior Academic ¨ Jennifer Ivie, Assistant Professor
Technology Officer (SATO) ¨ Department of Psychology
and Technology Innovations for ¨ Interim Director, Center for
Learning and Teaching (TILT) the Scholarly Advancement of
¨ ejunn@csufresno.edu Learning and Teaching (CSALT)
¨ 559-278-2636 Thank You!
jivie@csufresno.edu
¨

¨ 559-278-2842

Contact Information- ETS®
Arthur Ruzzano
ETS® Criterion® Western Regional Manager
¨ http://www.ets.org/criterion
Higher Education Assessment
Solutions
Phone: 310.944.4034
¨ Susan L. Yetman
Fax: 609.683.2040
¨ Criterion Account Manager
Email: aruzzano@ets.org
¨ Higher Education, ETS
¨ Rosedale Road MS 51-L Bill Wynne
¨ Princeton, NJ 08541 Product Manager - ETS
Proficiency Profile
¨ Direct line: 609.683.2675
ETS Programs and Services
¨ Toll free: 866.717.1915 Division
¨ Fax: 609.683.204 Telephone (609) 683-2006
0 E-mail wwynne@ets.org
¨ Email: syetman@ets.org Copyright © April 2011 48

Wasc arc conf final, iswi, april 8 2011.pptx

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Wasc arc conf final, iswi, april 8 2011.pptx

Similaire à Wasc arc conf final, iswi, april 8 2011.pptx (20)

Plus de California State University, Fresno

Plus de California State University, Fresno (20)

Dernier

Dernier (20)

Wasc arc conf final, iswi, april 8 2011.pptx