Having the skills and strategies to read, learn from, and communicate with the Internet will play a central role in our students’ success in an information age. But how can we best measure these new literacies? This session explores some of the challenges associated with developing valid and reliable measures of the complex literacy strategies and dispositions required to search for, comprehend, and respond to information on the Internet. The presenter will first share task examples and student responses from several assessments developed to measure online reading comprehension and communication skills. Then, conversation will turn to a number of important issues to consider when developing online literacy assessments that are not only psychometrically sound, but also useful to both researchers and classroom teachers. Participants will have an opportunity to share their own thoughts about how we might rethink the ways in which we evaluate the skills, strategies, and dispositions associated with reading and learning online.

1. How Does The Internet Impact
Our Thinking About Effective
Literacy Assessment?
Julie Coiro, Ph.D.
University of Rhode Island
jcoiro@snet.net

2. Goals for today’s conversation
Understand the issues that cause us to think
differently about literacy assessment
View examples of assessments developed to
measure the new literacies of online reading
comprehension
Explore some of the concerns and challenges
related to measuring online literacy and learning
Dialogue & reflect on connections to issues in your
own teaching / learning context

3. The New Literacies
Research Team QuickTimeª and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
Supported by:
• Ray and Carole Neag
• The Carnegie Corporation of New York
• U.S. Department of Education
• The National Science Foundation
• North Central Educational Research Lab
• PBS
• The Annenberg Foundation
QuickTimeª and a • William and Flora Hewlett Foundation
TIFF (Uncompressed) decompressor
are needed to see this picture.
• Bill and Melinda Gates Foundation
• Australian Council of Educational
Research

4. A New Literacies Perspective
considers the Internet as this
generation’s defining technology
for information, communication,
and especially for learning.
As a result, several important
differences require our attention:

5. New skills and strategies are required
to comprehend and learn from
information on the Internet
You begin by identifying an important question
New ways of locating information
New reasons for critically evaluating the
information
New contexts for synthesizing information to
answer your questions
New ways of communicating the answers to others
Leu, Kinzer, Coiro, and Cammack (2004)

6. New dispositions (attitudes, beliefs,
mindsets) are also required
Initiative, confidence, self-direction, critical
stance, adaptability, emotional resilience,
persistence, flexibility, divergent and
convergent thinking, personal productivity,
leadership, social responsibility, teamwork,
curiosity, motivation, openness, …
(e.g., American Library Association; Partnership for 21st
Century; New Literacies Research Team;
Johnston, 2005; Tsai & Tsai, 2003;
Whipp & Chiarelli, 2004)

7. Processes may be even more
important than the products
(especially for informing instruction)
Product: I can’t find it.
Process…

8. Developmental differences appear
across each phase of the online
inquiry process (and are particular to tasks)
Lower Average Higher
Locating .com strategy Whole phrase Uses keywords
Communicating Copies the long Toggles back and Copy/pastes
address by hand forth to type address address with mouse
shortcut
Evaluating Struggles to locate Judges reliability Examines author’s
reliability “About Author” based on length of level of
page coverage integrity/expertise
Coiro, 2007
Do our assessments adequately capture ability and growth
across each of these stages and across a range of tasks
in ways that can inform instruction?

9. Other issues to consider when
assessing online literacies
Students often work collaboratively in
groups or seek help from others online, yet
we continue to measure reading performance
individually and without online assistance
New assessments are needed to evaluate group
collaboration and productivity as well as how
readers seek assistance from a globally networked
community

10. Other issues to consider when
assessing online literacies
Today’s authentic problems often require
interdisciplinary connections, yet we most often
measure reading performance outside of these
authentic problem-solving contexts
Valid new assessments of online reading comprehension
should situate reading tasks within a range of relevant
content area/interdisciplinary information challenges
Valid new assessments should also integrate the
information and communication tools used in the
workforce and in students’ daily lives (“real”
blogs, wikis, instant message, Flickr sites,
etc.)
QuickTimeª and a
TIFF (Uncompressed) decompressor
are needed to see this picture.

11. The ultimate challenge to consider
when assessing online literacies…
Online texts, tools, and learning environments will
continue to rapidly change…
We need to be prepared to continually reconsider,
change, and expand what it means to be a skilled
online reader.
Thus… our measures need to continually change and
expand…
Which makes it difficult to replicate the use of the
same measure over time (establishing reliability) or
the same tool/environment over time (ecological
validity)

12. To summarize, at least eight issues
impact our thinking about literacy
assessment in a digital age
New literacy skills and strategies
New dispositions
New processes to add to products
New developmental differences to capture
New group measures to add to individual measures
New authentic problems to solve across disciplines
New tools and technologies for school, work, and
daily life
Rapid and continual change

13. Your Turn..
Connecting with you
Are you noticing challenges to
assessment with and of technology
use in your discipline?
If so, can you share some examples?
Possible solutions?

14. So, what have we learned in the
past three years about designing
assessments to measure online
reading comprehension?

15. Assessments of Online
Reading Comprehension
ORCA-Instant Message
ORCA-Blog
ORCA Scenarios I and II
ORCA Iditarod
ORCA Iditarod-Revised
Survey of Internet Use and Online Reading
ORCA Growth-Curve Measures
Formative Assessments of Online Strategy Use
ISSUES: Environments; Items; Scoring Systems; Group or Individual;
Protocols for Administering & Conducting; “Plan B or C”; Managing
Student Data; Analysis; Making Teacher Friendly

16. ORCA Instant Message:
(2004) Homelessness and The Solar
System

17. Psychometric Properties
ORCA-Instant Message (Solar System)
N= 36 (Three ability levels; N=12 at each level)
Admin. individually (1 student, 2-3 researchers - very
“real-time” and time consuming!)
Content Validity Post-Test (4 tasks, 12 items)
Principal components analysis supported one composite
explaining 66% of the variance
Internal Consistency:
Cronbach’s Alpha=.91

18. ORCA-Blog: Human Body Systems (2005)

19. Psychometric Properties
ORCA-Blog (Human Body Systems)
N=89
Admin to group - post to blog simultaneously
Content Validity Post-Test (3 tasks, 10 items)
Principal components analysis supported one
composite explaining 59.2% of the variance
Internal Consistency:
Cronbach’s Alpha=.84

20. ORCA-Scenarios I and II (Coiro,
2006)Carbon Monoxide Poisoning, Asthma,
and the Respiratory System

22. Related Variables to Consider…
Internet Use
Prior Knowledge

23. Related Variables… Dispositions
Open-ended questions:
(a) How approach; (b) How respond; (c): Self-efficacy
• What is easiest for you about using the Internet for research?
• What is hardest for you about using the Internet for research?
• Can you think of a time when you had trouble finding
something using the Internet? How do you feel when this
happens? How long do you keep trying before you give up?
• What do you know about using the Internet effectively that
some kids your age might not know?

25. Psychometric Properties
ORCA-Scenarios I and II
N=120; Group administered (record with Camtasia)
ORCA-Scenario 1
Content Validity (3 tasks, 20 items)
• Principal components analysis supported one composite
explaining 51.7% of the variance
• Factor Loadings: Task 1(.617); Task 2(.755); Task 3(.775)
Internal Consistency: Cronbach’s Alpha=.92
Inter-Rater Reliablity: k=.96
ORCA-Scenario II
Content Validity (3 tasks, 20 items)
• Principal components analysis supported one composite
explaining 44.1% of the variance
• Factor Loadings: Task 1(.683); Task 2(.751); Task 3(.541)
Internal Consistency: Cronbach’s Alpha=.91
Inter-Rater Reliablity: k=.96

26. ORCA-Iditarod (2007)

27. ORCA-Iditarod Task 1
Formulate Question/Locate Task
(and Evaluate Accuracy in Task 2)

28. ORCA-Iditarod Task 4
Evaluate Bias/Perspective and Reliability
Big idea
Smaller idea
www.newliteracies.uconn.edu/iesproject

29. ORCA-Iditarod Task 5
Synthesize from Multiple Sources and…
www.newliteracies.uconn.edu/iesproject

30. ORCA-Iditarod Task 5
and… Communicate Your Opinion

31. Psychometric Properties
ORCA-Iditarod
Content Validity Post-Test (5 Tasks, 44 items)
Principal components analysis supported one
composite explaining 53% of the variance with
separate task loadings ranging from .613-.795
Task 1 (.659); Task 2 (.786); Task 3 (.795); Task 4
(.613); Task 5 (.766)
Internal Consistency: Cronbach’s Alpha=.88
Inter-rater reliability: Cohen’s Kappa = .94

32. ORCA-Iditarod Revised (2007-08)

36. Survey of Internet Use and Online Reading
(2006-2007)

37. Psychometric Properties
Survey of Internet Use and Online
Reading
Forced Response Items: (5)
P values from .31 to .86 indicating item difficulty was
sufficient
3 items “very good” and two items “good” for test
discrimination
Open-Ended Items: (4)
Internal Consistency: Cronbach’s Alpha=.69
Inter-rater reliability: Cohen’s Kappa = .87

38. ORCA Growth Curve Assessment
Develop a measure that reflects growth of online
reading comprehension skills and strategy use over
time
Decrease scoring and analysis time
Help directly inform instruction
8 minutes online searching (quick scoring scheme)
12 minutes multiple choice (quick scoring scheme)
Administered 5 times over the course of the year
and follows the same structure each time

39. ORCA Growth Curve Assessment

40. Formative Assessment of Students’
Emerging Knowledge of Internet
Strategies
(2006-2008)
• Create many opportunities for students to practice all
three dimensions of strategic knowledge.
• Be sure to measure all three dimensions when
determining how well your students read and learn on
the Internet.

41. Summary of Assessments of
Online Reading Comprehension
ORCA-Instant Message
ORCA-Blog
ORCA Scenarios I and II
ORCA Iditarod
ORCA Iditarod-Revised
Survey of Internet Use and Online Reading
ORCA Growth-Curve Measures
Formative Assessments of Online Strategy Use

42. Overall, what are we learning from
assessments of online reading?
Patterns of effective & ineffective online strategies
Generating questions/keywords: location and quality of keywords
Searching: skimming for relevancy and flexible strategy use
Critical Evaluation: surface level understanding versus application;
differences between prompted and unprompted applications
Synthesis: incorporating multiple aspects of what’s “best”
Communication: efficiency factor; varying complexities depending on
the tool/interface used
Important variables for online reading comprehension &
learning
Standardized reading scores: correlated with online reading?
Prior Knowledge: challenging our assumptions
Dispositions: possibly having an influence on online reading patterns
New literacies: critical new online skills & strategies not measured in
current assessments of reading comprehension

43. Real-time videos of
Online Reading Comprehension
http://www.newliteracies.uconn.edu/coirodissertation/
http://www.newliteracies.uconn.edu/reading.html
http://www.newliteracies.uconn.edu/iesproject/videos/

44. Some challenges with online
assessments…
Developing tasks when websites come and go
Multiple paths to the correct (or incorrect) answer
Implications of “electronic scaffolds”
Differences in search engines and results
Issues of capturing and watching process data
Using real-time video for each student
Time-consuming scoring
The nature of online reading scenarios prompts new
ways of thinking about missing data

45. Your Turn..
Connecting with you
What successes have you had
with learning how to capture,
measure, analyze, and apply
information about your students’
proficiency with learning with
technology?
Other questions…comments?

46. Thank you for the conversation!
For more information:
Julie Coiro
University of Rhode Island
Jcoiro@snet.net
http://www.uri.edu/hss/education/faculty/coiro.html
New Literacies Research Team
http://www.newliteracies.uconn.edu