Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
Clear expectations and habits of
mind: A self-evaluation checklist for
student writing
Annette Marines, Research Support L...
Project Context
● Campus-wide assessment
○ Active program outcomes assessment
○ Focus: Written communications and informat...
Introduction
● What we were hearing and seeing
○ Students were not citing; were citing incorrectly; were not evaluating so...
Why it’s a problem?
● Expert versus novice
● Information deluge
● Communication
● Time management
Some Literature
● Assigning Inquiry: How Handouts for Research Assignments Guide Today's
College Students, Alison J. Head ...
Hypothesis
Students’ proficiencies in citation, attribution, and source evaluation can improve
with a checklist
Research Strategy
With full cooperation with the course instructor, the checklist intervention was
assigned in a senior se...
Method
Participants
● Upper division, undergraduate seniors
● Psychology majors
● Senior seminar course enrollees
Intervention : The Checklist
● We provided the course instructor with a PDF copy of the checklist
● The instructor:
○ Assi...
Sample
● No checklist n = 24 ; Checklist n = 27
● Unit of analysis: Draft Research Paper
● 100% of the research papers fro...
Measurement Instrument : Rubric
● Informed by AACU Value Rubrics and others
● 9 criteria
○ Each checklist item is matched ...
Data Collection
● 2 rubric norming sessions
○ Discussed discrepancies until we came to an agreement
● 2 readers, read thro...
Results
IRAPS ran the data analysis. The questions covered:
● Setting a baseline
● Comparing no checklist and checklist cl...
Key Decision
● The Developing & Proficient levels were combined
● The counts for Proficient were too low to analyze
In what areas do most students have developing or proficient skills and in
what areas do they need improvement?
(the basel...
LOW
0-59 %
D1*, B2**, E
In what areas did the checklist lead to improvement?
MEDIUM
60-79%
B1***, C
HIGH
80-100%
D2, A [ F...
LOW
0-59 %
D1, B2, E
In what areas do students still lack skills?
MEDIUM
60-79%
B1, C
A - Source Types
B&C - Bibliography
...
LOW
0-59 %
D1 Study 1 - SOC &
Bilingual
D1 Study 2* - SOC
Were there any differences related to background characteristics...
Results Sample Overview
● Both classes were comparable in terms of ethnicity and language status
○ Bilingual students incl...
Discussion
● The results support our hypothesis: Students proficiencies can and did
improve with the checklist
● Students ...
Evaluating Sources
MED-HIGH LOW
21-59%
BOTTOM LOW
0-20 %
E
A - Source Types
B&C - Bibliography
D - Attribution
E - Evaluat...
New text
❏ Have you included an evaluation of your sources?
Pay particular attention to the (1) the authoritativeness, per...
Why it’s a problem, again?
Expert versus novice, Information deluge, Communication, Time management
Limitations
● The chec...
Extending the Project
● We’re working with the same instructor with their lower-division, introduction
to the major course...
Photo Credits
● The University of California Santa Cruz, Doc Searls, (CC BY-SA 2.0), https://flic.kr/p/jmLrq
● Understand ...
Annette Marines
amarines@ucsc.edu
Aaron Zachmeier
azachmei@ucsc.edu
More information (checklists)
http://guides.library.uc...
Clear expectations and habits of mind: a self-evaluation checklist for student writing - Marines & Zachmeier
Clear expectations and habits of mind: a self-evaluation checklist for student writing - Marines & Zachmeier
Clear expectations and habits of mind: a self-evaluation checklist for student writing - Marines & Zachmeier
Prochain SlideShare
Chargement dans…5
×

Clear expectations and habits of mind: a self-evaluation checklist for student writing - Marines & Zachmeier

411 vues

Publié le

Presented at LILAC 2017

Publié dans : Formation
  • Soyez le premier à commenter

Clear expectations and habits of mind: a self-evaluation checklist for student writing - Marines & Zachmeier

  1. 1. Clear expectations and habits of mind: A self-evaluation checklist for student writing Annette Marines, Research Support Librarian Aaron Zachmeier, Instructional Designer University of California, Santa Cruz LILAC 2017 Swansea University
  2. 2. Project Context ● Campus-wide assessment ○ Active program outcomes assessment ○ Focus: Written communications and information literacy ● Teaching support efforts ○ Liaison Librarian: Collections and Information Literacy Instruction ○ Instructional Designer: Online Education ○ Assessment Assistant Director: Assessment Support ● This Research Team ○ Liaison Librarian ○ Instructional Designer ○ Institutional Research, Assessment, and Policy Studies (IRAPS)
  3. 3. Introduction ● What we were hearing and seeing ○ Students were not citing; were citing incorrectly; were not evaluating sources ○ Assignment handouts avoided guidance ● The intervention ○ Opportunity for Librarian and Instructional Designer to collaborate ○ Developed a checklist to use as a research paper cover sheet ● The research project ○ Teamed up with IRAPS and a Psychology professor ○ Assessed the checklist
  4. 4. Why it’s a problem? ● Expert versus novice ● Information deluge ● Communication ● Time management
  5. 5. Some Literature ● Assigning Inquiry: How Handouts for Research Assignments Guide Today's College Students, Alison J. Head & Michael B. Eisenberg, Project Information Literacy ● They can find it but they don't know what to do with it: describing the use of scholarly literature by undergraduate students, Stephanie Rosenblatt ● Checklist manifesto: How to get things right, Atul Gawande ● Teaching Research Processes, William B. Badke
  6. 6. Hypothesis Students’ proficiencies in citation, attribution, and source evaluation can improve with a checklist
  7. 7. Research Strategy With full cooperation with the course instructor, the checklist intervention was assigned in a senior seminar course. Students were required to turn in the checklist as a cover sheet to the draft of a research paper A rubric addressing learning outcomes of the checklist was used to evaluate the draft research papers of two classes: one using no checklist and one using the checklist
  8. 8. Method Participants ● Upper division, undergraduate seniors ● Psychology majors ● Senior seminar course enrollees
  9. 9. Intervention : The Checklist ● We provided the course instructor with a PDF copy of the checklist ● The instructor: ○ Assigned the checklist along with the draft research paper ○ And made it available in the LMS ○ No additional instruction was provided ● Students handed in the checklist with the paper ● The intervention was handled exclusively by the course instructor
  10. 10. Sample ● No checklist n = 24 ; Checklist n = 27 ● Unit of analysis: Draft Research Paper ● 100% of the research papers from each class were included ○ Started by reading 50% (n=12 and n=14). IRAPS asked us to read all the papers to increase significance ● Classes were taught three year apart by the same instructor
  11. 11. Measurement Instrument : Rubric ● Informed by AACU Value Rubrics and others ● 9 criteria ○ Each checklist item is matched to a rubric criterion, Ex. Evaluation separated into 3 domains to capture ■ Makes rubric more precise in evaluating information literacy and critical thinking ● 4 levels ○ Unacceptable (no signs), Beginning, Developing, Proficient (mastered)
  12. 12. Data Collection ● 2 rubric norming sessions ○ Discussed discrepancies until we came to an agreement ● 2 readers, read through and scored papers ● Readers evaluated: ○ Citation style, formatting, consistency ○ Attribution ○ Evaluation of sources ● Blind review
  13. 13. Results IRAPS ran the data analysis. The questions covered: ● Setting a baseline ● Comparing no checklist and checklist classes ● Areas that remained low ● Effect of checklist on specific populations
  14. 14. Key Decision ● The Developing & Proficient levels were combined ● The counts for Proficient were too low to analyze
  15. 15. In what areas do most students have developing or proficient skills and in what areas do they need improvement? (the baseline) MEDIUM 60-79% C HIGH 80-100% D2, A [ F, D3 ] LOW 0-59 % B1, D1, B2, E A - Source Types B&C - Bibliography D - Attribution E - Evaluation F - Quoted Text
  16. 16. LOW 0-59 % D1*, B2**, E In what areas did the checklist lead to improvement? MEDIUM 60-79% B1***, C HIGH 80-100% D2, A [ F, D3 ] * Significance at p <.05 ** at p <.07 *** Not significant A - Source Types B&C - Bibliography D - Attribution E - Evaluation F - Quoted Text
  17. 17. LOW 0-59 % D1, B2, E In what areas do students still lack skills? MEDIUM 60-79% B1, C A - Source Types B&C - Bibliography D - Attribution E - Evaluation F - Quoted Text
  18. 18. LOW 0-59 % D1 Study 1 - SOC & Bilingual D1 Study 2* - SOC Were there any differences related to background characteristics (ethnicity, language, and transfer status)? MEDIUM 60-79% D2 Study 1 - SOC D1 Study 2* - Bilingual HIGH 80-100% D2 Study 2* - SOC * Significance differences Fisher’s exact test p < .08 Study 1 findings in blue Study 2 findings in purple SOC - Students of color D1 - attribution of facts D2 - attribution of ideas
  19. 19. Results Sample Overview ● Both classes were comparable in terms of ethnicity and language status ○ Bilingual students included those who at the age of 5 spoke English and another language or learned English later in school. ● Class 2 (checklist) had fewer transfer students than Class 1 (no checklist) ● There were more first generation students in Class 2 than in Class 1 ● There were greater number of women in both classes, with fewer men in Class 2
  20. 20. Discussion ● The results support our hypothesis: Students proficiencies can and did improve with the checklist ● Students improved in every area, with significant differences in 3 of these ● Clear guidelines and steps help students
  21. 21. Evaluating Sources MED-HIGH LOW 21-59% BOTTOM LOW 0-20 % E A - Source Types B&C - Bibliography D - Attribution E - Evaluation F - Quoted Text
  22. 22. New text ❏ Have you included an evaluation of your sources? Pay particular attention to the (1) the authoritativeness, perspective, and potential bias of the author; (2) credibility or validity of the findings; (3) relevance and purpose of the source itself. Evaluating Sources Old text ❏ Have you evaluated your sources? Pay particular attention to the (1) the authoritativeness, perspective, and potential bias of the author; (2) credibility or validity of the findings; (3) relevance and purpose of the source itself.
  23. 23. Why it’s a problem, again? Expert versus novice, Information deluge, Communication, Time management Limitations ● The checklist is not a teaching tool ● No evidence yet that repeated use will develop habits of mind Unexpected findings ● Closed gaps seen in ethnic and language populations ○ The checklist helped everyone, but helped these groups more
  24. 24. Extending the Project ● We’re working with the same instructor with their lower-division, introduction to the major course. ● We’re currently reading papers for a Senior Seminar course in the History Department Teaching Support ● We promote the checklist in our teaching support consultations ● Our new rubric experience has opened doors with assessment conversations, where I wouldn’t have brought it up before
  25. 25. Photo Credits ● The University of California Santa Cruz, Doc Searls, (CC BY-SA 2.0), https://flic.kr/p/jmLrq ● Understand context, Paul Downey, (CC BY 2.0), https://flic.kr/p/eTB94c ● Ginger man... NAILED IT!, Sharon Vicino, (CC BY-SA 2.0), https://flic.kr/p/quA8aW ● Rubik's Cubes, CC0 Public Domain, http://maxpixel.freegreatpicture.com/photo-1243091 ● College Students, CollegeDegrees360, (CC BY-SA 2.0), https://flic.kr/p/cEJAgJ ● cmyk color management linen tester, Markus Spiske, (CC0 1.0), https://flic.kr/p/RS7kbT ● Crunches, Everkinetic (http://everkinetic.com/), (CC BY-SA 3.0), https://commons.wikimedia.org/wiki/File%3ADecline-crunch-2.png & https://commons.wikimedia.org/wiki/File%3ADecline-crunch-2.png ● Crystal Wynwood Mural, Elvert Barnes, (CC BY-SA 2.0), https://flic.kr/p/pic2z7 ● Hooray Penguin, David Goehring, (CC BY 2.0), https://flic.kr/p/JZ4Dmn ● Students in Class, Tulane Public Relations, (CC BY 2.0), https://commons.wikimedia.org/wiki/File%3AStudent_in_Class_(3618969705).jpg ● The Questioning Roboto, Matthew Hutchinson, (CC BY 2.0), https://flic.kr/p/7VhPft
  26. 26. Annette Marines amarines@ucsc.edu Aaron Zachmeier azachmei@ucsc.edu More information (checklists) http://guides.library.ucsc.edu/checklists

×