Developing Reliable, Valid, and
Practical Assessments of Online
Research and Comprehension
(ORCAs)
Donald J. Leu, Ph.D
The...
Today





Introduce a performance based assessment
of online research and comprehension in
science.
Share the results o...
Institute of Educational Science - U.S. Department of Education
Principal Investigator
Donald J. Leu, The University of Co...
Our Focus Is The New Literacies of
Online Research and
Comprehension


Defines how we read online when we
conduct informa...
Locate
Can the student locate the correct email message in an
inbox on the first click?
Can the student use appropriate ke...
Evaluate
Can the student identify the author of the website?
Can the student evaluate the author's level of
expertise?
Can...
Synthesize
Can students provide a summary of one important element from
the first website?
Can students use their own word...
Communicate: Email (Wiki)
Does the student include the correct email address in an email
message?
Does the student include...
Two Formats:
Closed - Multiple Choice
Four Research Questions
TOPIC

TYPE OF RESEARCH COMMUNICATION TOOL
PROBLEM
FOR THE R...
Reliability and Validity Data





Final statistics are being calculated
These eight ORCAs were the best
performing fro...
ORCA-Closed
12
13
14
15
16
17
18
Reading to
Locate
Information
Reading to
Locate
Information
Online
Reading to Synthesize
Information Online
Reading to Critically Evaluate
Information Online
Reading to Critically Evaluate
Information Online
Reading to Critically Evaluate
Information Online
Reading to Critically Evaluate
Information Online
Reading to Critically Evaluate
Information Online
Reading to Critically Evaluate
Information Online
Reading to Critically Evaluate
Information Online
Reading to Critically Evaluate
Information Online
Reading to Critically Evaluate
Information Online
Reading to Critically Evaluate
Information Online
Reading to Critically Evaluate
Information Online
Reading and Writing to
Communicate Information Online:
Email Task
33
ORCA-Multiple Choice: An
Identical Sequence of Items
How Well Do Students Conduct Online
Research and Comprehend?
Students are only minimally prepared.
Average % of Correct
Re...
Skill
Reading to Locate Online Information

% Correct
52.9%

Locate the correct email message in an inbox or the
correct s...
Reading to Synthesize Online Information
64.6%
Summarize an important element from one website.
77.5%
Synthesize important...
A Second Study: Students in
Rich and Poor School Districts

Leu, D. J., Coiro, J., O’Byrne, W. I.,
Zawilinski, L., McVerry...
In the U.S., The Rich Get
Richer and the Poor Get Poorer
in OFFLINE READINGReading. 90/10 Income Achievement & BlackOfflin...
Our Study’s School District
Differences
West Town

East Town

Median Family Income

$119,338

$58,981

% of Families Below...
A Significant Achievement Gap
Existed in Offline Reading
(State Test: Reading)

282.6

215.1

t (237) = 14.34 p = .000


...
A Significant Achievement Gap Existed in
Online Research and Comprehension
Abilities

15.00

7.65

t (255) = 9.80, p = .00...
...Even When an ANCOVA Analysis
Was Conducted
Differences Controlled: Offline Reading + Prior
Knowledge

12.96

9.27

F (1...
A Third Study: Comparing Online
Research Skills in Two States
(n=1,129)
State 1

State 2
(laptops)

TOTAL

Locate (8)

4.5...
Adjusted Total Mean Scores
When Covariates of SES and
Prior Knowledge Controlled
State 1

Adjusted Means

State 2
(laptops...
A Practical Assessment Of
Online Research and
Comprehension
The Design of Our Assessments:
Practicality Concerns


Administration time?





Multiple Choice - 25 minutes
Closed -...
Invitations:


Please send me names, titles, emails of
any superintendent, principal, or middle
school teacher whom you t...
Developing Reliable, Valid, and
Practical Assessments of Online
Research and Comprehension
(ORCAs)
Donald J. Leu, Ph.D
The...
Prochain SlideShare
Chargement dans…5
×

Leu AASL 11-14-13

461 vues

Publié le

ORCAs: Online Research and Comprehension Assessments developed at the New Literacies Lab, University of Connecticut

Publié dans : Formation, Technologie
  • Soyez le premier à commenter

  • Soyez le premier à aimer ceci

Leu AASL 11-14-13

  1. 1. Developing Reliable, Valid, and Practical Assessments of Online Research and Comprehension (ORCAs) Donald J. Leu, Ph.D The New Literacies Research Lab The Neag School University of Connecticut
  2. 2. Today   Introduce a performance based assessment of online research and comprehension in science. Share the results of several studies of online research and comprehension.
  3. 3. Institute of Educational Science - U.S. Department of Education Principal Investigator Donald J. Leu, The University of Connecticut Co-Principal Investigators Jonna Kulikowich, The Pennsylvania State University Nell Sedransk, National Institute of Statistical Sciences Julie Coiro, University of Rhode Island Scientific Advisory Board P. David Pearson, The University of California, Berkeley Irwin Kirsch, Educational Testing Service Rand Spiro, Michigan State University Elizabeth Stage, Lawrence Hall of Science, Berkeley Glenn Kleimann, Friday Institute, NCSU 3
  4. 4. Our Focus Is The New Literacies of Online Research and Comprehension  Defines how we read online when we conduct informal and formal research to learn and develop new knowledge.     Locate Evaluate Synthesize Communicate Castek, 2008; Coiro & Dobler, 2007; Henry, 2007; Leu, Castek, Hartman, Coiro, Henry, Kulikowich, & Lyver, 2005; Leu, O’Byrne, Zawilinski, McVerry, & Everett-Cacopardo, 2009; Leu, Kinzer, Coiro, Castek, & Henry, 2013
  5. 5. Locate Can the student locate the correct email message in an inbox on the first click? Can the student use appropriate keywords in a search engine? Can the student locate the best site for a task from a set of search engine results on the first click? Can the student locate and communicate the correct website addresses from two different search tasks.
  6. 6. Evaluate Can the student identify the author of the website? Can the student evaluate the author's level of expertise? Can the student identify the author's point of view? Can the student evaluate the reliability of a website?
  7. 7. Synthesize Can students provide a summary of one important element from the first website? Can students use their own words to integrate one detail from each of the first two websites? Can students use their own words to integrate one detail from each of the second two websites? Can students use their own words to develop an argument after reading all four websites?
  8. 8. Communicate: Email (Wiki) Does the student include the correct email address in an email message? Does the student include an appropriate subject line in an email message? Does the student include an appropriate greeting in an email message to an important, unfamiliar person? Does the student compose and send a well-structured, short report of their research in an email with sources and appropriate argument structure, containing at least one relevant claim and at least two pieces of evidence.
  9. 9. Two Formats: Closed - Multiple Choice Four Research Questions TOPIC TYPE OF RESEARCH COMMUNICATION TOOL PROBLEM FOR THE REPORT How do energy drinks affect Learn more about heart health? email How can snacks be hearthealthy? Learn more about email Do cosmetic contact lenses harm your eyes? Investigate conflicting claims Wiki Do videogames harm your eyesight? Investigate conflicting claims Wiki
  10. 10. Reliability and Validity Data    Final statistics are being calculated These eight ORCAs were the best performing from 24 tested the previous year. Items revised based on the previous year’s data.
  11. 11. ORCA-Closed
  12. 12. 12
  13. 13. 13
  14. 14. 14
  15. 15. 15
  16. 16. 16
  17. 17. 17
  18. 18. 18
  19. 19. Reading to Locate Information
  20. 20. Reading to Locate Information Online
  21. 21. Reading to Synthesize Information Online
  22. 22. Reading to Critically Evaluate Information Online
  23. 23. Reading to Critically Evaluate Information Online
  24. 24. Reading to Critically Evaluate Information Online
  25. 25. Reading to Critically Evaluate Information Online
  26. 26. Reading to Critically Evaluate Information Online
  27. 27. Reading to Critically Evaluate Information Online
  28. 28. Reading to Critically Evaluate Information Online
  29. 29. Reading to Critically Evaluate Information Online
  30. 30. Reading to Critically Evaluate Information Online
  31. 31. Reading to Critically Evaluate Information Online
  32. 32. Reading to Critically Evaluate Information Online
  33. 33. Reading and Writing to Communicate Information Online: Email Task 33
  34. 34. ORCA-Multiple Choice: An Identical Sequence of Items
  35. 35. How Well Do Students Conduct Online Research and Comprehend? Students are only minimally prepared. Average % of Correct Responses Locate Skills 52.9% Evaluate Skills 54.3% Synthesis Skills 64.6% Communicate Skills 44.4% TOTAL 54%
  36. 36. Skill Reading to Locate Online Information % Correct 52.9% Locate the correct email message in an inbox or the correct section of a wiki. 60.5% Use appropriate key words in a search engine. 59.5% Locate the correct site in a set of search engine results. 56.0% Locate and share correct website addresses in two different search tasks. 33.0% Reading to Evaluate Online Information Identify the author of a website. Evaluate an author’s level of expertise. Identify an author’s point of view. Evaluate the reliability of a website. 54.3% 78.5% 47.5% 51.0% 39.5%
  37. 37. Reading to Synthesize Online Information 64.6% Summarize an important element from one website. 77.5% Synthesize important elements from two websites. 63.5% Synthesize important elements from a second set of two 57.0% websites. Synthesize important elements from websites to develop an 60.5% argument. Writing to Communicate Online Information 44.4% Include the correct address in email and make a wiki entry in the correct location. 41.5% Include an appropriate subject line in email and an appropriate heading in a wiki. 53.0% Include an appropriate greeting in email and use descriptive voice in a wiki. 49.5% Compose a short report of research, including sources, in email and wiki. 33.0%
  38. 38. A Second Study: Students in Rich and Poor School Districts Leu, D. J., Coiro, J., O’Byrne, W. I., Zawilinski, L., McVerry, J. G., Cacopardo, H., Kennedy, C., & Forzani, E. (2011). Online reading comprehension assessment (ORCA): A preliminary study of online reading comprehension ability in rich and poor school districts. Paper presented at the meeting of the Literacy Research Association. Jacksonville, FL.
  39. 39. In the U.S., The Rich Get Richer and the Poor Get Poorer in OFFLINE READINGReading. 90/10 Income Achievement & BlackOffline COMPREHENSION: Average Difference in S.D. Units on National Assessments White Gaps in Reading, 1943-2001 Cohorts *Adapted from: Reardon, S.F. (2011). The widening academic achievement gap between the rich and the poor: New evidence and possible explanations. In R. Murnane & G. Duncan (Eds.), Whither Opportunity? Rising Inequality and the Uncertain Life Chances of Low-Income Children. New York: Russell Sage Foundation Press. Reardon, S.F. (2011). The widening academic achievement gap between the rich and the poor: New evidence and possible explanations. In R. Murnane & G. Duncan (Eds.), Whither Opportunity? Rising Inequality and the Uncertain Life Chances of Low-Income Children. New York: Russell Sage Foundation Press.
  40. 40. Our Study’s School District Differences West Town East Town Median Family Income $119,338 $58,981 % of Families Below Poverty Line 2.1% 11.8% % of Students Eligible for 4% Free/Reduced Price Lunches 67%
  41. 41. A Significant Achievement Gap Existed in Offline Reading (State Test: Reading) 282.6 215.1 t (237) = 14.34 p = .000   West Town (Rich) Mean = 282.6 (SD = 41.54)  East Town (Poor) Mean = 215.1 (SD = 31.07) eta squared = .466 (large)
  42. 42. A Significant Achievement Gap Existed in Online Research and Comprehension Abilities 15.00 7.65 t (255) = 9.80, p = .000  West Town Mean = 15.00 (SD=5.69)  East Town Mean = 7.65 (SD=4.39)  eta squared = .319 (large)
  43. 43. ...Even When an ANCOVA Analysis Was Conducted Differences Controlled: Offline Reading + Prior Knowledge 12.96 9.27 F (1,234) = 15.84, p = .001  West Town (rich) adjusted mean = 12.96  partial eta squared = .063  East Town (poor) adjusted mean = 10.27  (medium)
  44. 44. A Third Study: Comparing Online Research Skills in Two States (n=1,129) State 1 State 2 (laptops) TOTAL Locate (8) 4.52 4.64 4.58 Evaluate (8) 3.61 3.32 3.47 Synthesize (8) 6.07 5.86 5.97 Communicate (8) 4.22 4.00 4.11 TOTAL (32) 18.42 17.81 18.13 State 1: 4th in Median Family Income/ Few Laptops State 2: 32nd in Median Family Income/1 to 1 Laptops
  45. 45. Adjusted Total Mean Scores When Covariates of SES and Prior Knowledge Controlled State 1 Adjusted Means State 2 (laptops) 17.56 19.08* *p < .05 F (1, 1021) = 14.854, p = .000 45
  46. 46. A Practical Assessment Of Online Research and Comprehension
  47. 47. The Design of Our Assessments: Practicality Concerns  Administration time?    Multiple Choice - 25 minutes Closed - 40 minutes Scoring?    Closed - Auto scoring and hand scoring Format?   Multiple Choice - Auto scoring Multiple Choice and Performance Based Reports?  Specific skill performance (16 skills)  Skill Area performance (4 Areas)  Total score 47
  48. 48. Invitations:  Please send me names, titles, emails of any superintendent, principal, or middle school teacher whom you think may be interested in viewing a video and completing a 15 minute survey. donald.leu@uconn.edu
  49. 49. Developing Reliable, Valid, and Practical Assessments of Online Research and Comprehension (ORCAs) Donald J. Leu, Ph.D The New Literacies Research Lab The Neag School University of Connecticut

×