The document provides information about PISA (Programme for International Student Assessment), an international study that assesses educational achievement and skills. It discusses why countries participate in PISA, the assessment framework and domains assessed. It notes new directions for PISA including assessing digital reading, which involves both text processing and navigation of digital texts. Sample PISA reading passages, tasks and item formats are presented to illustrate how PISA assesses reading comprehension.

1. PISAPISA
Reading FrameworkReading Framework
Comillas, Spain
September 2013
Juliette Mendelovits
September 2013
Leader, PISA reading framework and test development

2. Why do countries participate iny p p
international studies of educational
achievement?achievement?
• To collect national-level data when there is noTo collect national level data when there is no
other mechanism for doing so
• To compare themselves with other countries
• To be involved in – and help to shape - newp p
developments in education

3. PISA Survey Cycley y
2000 2003 2006 2009 2012 20152000 2003 2006 2009 2012 2015
Reading Literacy
Mathematical Literacy
Scientific Literacy
Questionnaire
Major domain cycle in yellow
In PISA, framework development takes place during
the major domain cycle.

4. What is an assessment
framework?
• An assessment framework is an explicit statement
and discussion about what an assessment intends to
measure.
• Its purposes are:
– To guide test development
– To give a common language to stakeholders for
di i f th bj tdiscussion of the subject
– To ensure continuity from one year or one grade
level to anotherlevel to another
– To communicate the purpose and features of the
assessment program to the publicassessment program to the public

5. Framework developmentp
begins with a definition of the
d idomain ...

6. PISA’s definition of reading
Reading literacy is understanding, using,
reflecting on and engaging with written textsreflecting on and engaging with written texts,
in order to achieve one’s goals, to develop
’one’s knowledge and potential, and to
participate in society.y

7. PISA uses four taskPISA uses four task
characteristics in the construction
f ll di lit t kof all reading literacy tasks
• Context
• Text format – text type
• AspectAspect
• Item format

8. Reported Subscales for PISA
Reading 2000 and 2009
Text format
– Continuous
– Non-continuous Aspects
– Mixed
– Multiple
Aspects
– Access and retrieve
I d– Multiple – Integrate and
interpret
– Reflect and evaluate

9. Text formatText format
• Continuous
I f ti
• Non-continuous
I f ti– Information
presented in
sentences and
– Information
presented in formats
other than prose forsentences and
paragraphs
• Argument
other than prose, for
example
• Tables
• Description
• Exposition
I i
• Graphs
• Diagrams
M• Instruction
• Narration
• Maps
• Forms
Mixed

10. Example of a
mixed textmixed text
A text thatA text that
comprises
continuous andcontinuous and
non-continuous
partsparts
Mixed

11. Text formatText format
• Continuous
I f ti
• Non-continuous
I f ti– Information
presented in
sentences and
– Information
presented in formats
other than prose forsentences and
paragraphs
• Argument
other than prose, for
example
• Tables
• Description
• Exposition
I i
• Graphs
• Diagrams
M
Stimulus
comprising• Instruction
• Narration
• Maps
• Forms
comprising
several texts of
any kind
Multiple
any kind
Multiple

12. Text formats
… further categorised by text types
• Continuous• Continuous
N ti
• Narration
• Exposition
• Non-continuous
p
• Description
• Mixed • Argumentation
• Instruction
• Multiple
• Instruction
• Transaction

13. Aspects
• Access and retrieve
• Integrate and interpret• Integrate and interpret
• Reflect and evaluate

14. Sample Text from PISA 2000 and 2009
MacondoMacondo
Dazzled by so many and such marvellous inventions, the people of
Macondo did not know where their amazement began They stayed upMacondo did not know where their amazement began. They stayed up
all night looking at the pale electric bulbs fed by the plant that Aureliano
Triste had brought back when the train made its second trip, and it took
time and effort for them to grow accustomed to its obsessive toom-time and effort for them to grow accustomed to its obsessive toom
toom. They became indignant over the living images that the
prosperous merchant Don Bruno Crespi projected in the theatre with
the lion-head ticket windows, for a character who had died and was,
buried in one film, and for whose misfortune tears of affliction had been
shed, would reappear alive and transformed into an Arab in the next
one. The audience, who paid two centavos apiece to share the
difficulties of the actors, would not tolerate that outlandish fraud and
they broke up the seats. The mayor, at the urging of Don Bruno Crespi,
explained by means of a proclamation that the cinema was a machine
of illusions that did not merit the emotional outburst of the audience.
With that discouraging explanation many felt that they had been the
victims of some new and showy gypsy business and they decided not
t t t th i id i th t th l d h d tto return to the movies, considering that they already had too many
troubles of their own to weep over the acted-out misfortunes of
imaginary beings.

15. Macondo
Sample access and retrieve item
Macondo
Dazzled by so many and such marvellous inventions, the people of
Macondo did not know where their amazement began They stayed upMacondo did not know where their amazement began. They stayed up
all night looking at the pale electric bulbs fed by the plant that Aureliano
Triste had brought back when the train made its second trip, and it took
time and effort for them to grow accustomed to its obsessivetime and effort for them to grow accustomed to its obsessive
toom-toom. They became indignant over the living images that the
prosperous merchant Don Bruno Crespi projected in the theatre with
the lion-head ticket windows, for a character who had died and was,
buried in one film, and for whose misfortune tears of affliction had been
shed, would reappear alive and transformed into an Arab in the next
one. The audience, who paid two centavos apiece to share the
difficulties of the actors, would not tolerate that outlandish fraud and
they broke up the seats. The mayor, at the urging of Don Bruno Crespi,
explained by means of a proclamation that the cinema was a machine
How much did it cost to go to the
cinema in Macondo?
of illusions that did not merit the emotional outburst of the audience.
With that discouraging explanation many felt that they had been the
victims of some new and showy gypsy business and they decided not
t t t th i id i th t th l d h d t
One centavo
Two centavos
Th tto return to the movies, considering that they already had too many
troubles of their own to weep over the acted-out misfortunes of
imaginary beings.
Three centavos
Four centavos

16. Macondo
Sample integrate and interpret item
Macondo
Dazzled by so many and such marvellous inventions, the people of
Macondo did not know where their amazement began They stayed upAt the end of the passage why didMacondo did not know where their amazement began. They stayed up
all night looking at the pale electric bulbs fed by the plant that Aureliano
Triste had brought back when the train made its second trip, and it took
time and effort for them to grow accustomed to its obsessive toom-
At the end of the passage, why did
the people of Macondo decide not
to return to the movies?time and effort for them to grow accustomed to its obsessive toom
toom. They became indignant over the living images that the
prosperous merchant Don Bruno Crespi projected in the theatre with
the lion-head ticket windows, for a character who had died and was,
buried in one film, and for whose misfortune tears of affliction had been
shed, would reappear alive and transformed into an Arab in the next
one. The audience, who paid two centavos apiece to share the
difficulties of the actors, would not tolerate that outlandish fraud and
they broke up the seats. The mayor, at the urging of Don Bruno Crespi,
explained by means of a proclamation that the cinema was a machine
of illusions that did not merit the emotional outburst of the audience.
With that discouraging explanation many felt that they had been the
victims of some new and showy gypsy business and they decided not
t t t th i id i th t th l d h d tto return to the movies, considering that they already had too many
troubles of their own to weep over the acted-out misfortunes of
imaginary beings.

17. Macondo
Sample reflect and evaluate item
Macondo
Dazzled by so many and such marvellous inventions, the people of
Macondo did not know where their amazement began They stayed upMacondo did not know where their amazement began. They stayed up
all night looking at the pale electric bulbs fed by the plant that Aureliano
Triste had brought back when the train made its second trip, and it took
time and effort for them to grow accustomed to its obsessive toom-time and effort for them to grow accustomed to its obsessive toom
toom. They became indignant over the living images that the
prosperous merchant Don Bruno Crespi projected in the theatre with
the lion-head ticket windows, for a character who had died and was
Do you agree with the final judgement of
the people of Macondo about the value of
the movies?
,
buried in one film, and for whose misfortune tears of affliction had been
shed, would reappear alive and transformed into an Arab in the next
one. The audience, who paid two centavos apiece to share the
the movies?
Explain your answer by comparing your
attitude to the movies with theirs.
difficulties of the actors, would not tolerate that outlandish fraud and
they broke up the seats. The mayor, at the urging of Don Bruno Crespi,
explained by means of a proclamation that the cinema was a machine
attitude to the movies with theirs.
of illusions that did not merit the emotional outburst of the audience.
With that discouraging explanation many felt that they had been the
victims of some new and showy gypsy business and they decided not
t t t th i id i th t th l d h d tto return to the movies, considering that they already had too many
troubles of their own to weep over the acted-out misfortunes of
imaginary beings.

18. Example of students’ answers to Macondo
reflect and evaluate itemreflect and evaluate item
Do you agree with the final judgement of the
people of Macondo about the value of thepeople of Macondo about the value of the
movies?
Explain your answer by comparing your attitude
to the movies with theirs
Yes, I agree, there’s enough suffering in the
world without making it up
to the movies with theirs.
world without making it up.
 No, people understand that when you go
to the movies what happens on the screento the movies what happens on the screen
isn’t real.
It d d If th fil i bbi h I f l likIt depends. If the film is rubbish, I feel like
walking out, but if it’s good you are caught up in it
and don’t mind if it’s not true.

19. Item formatItem format
About
• Selected response
Multiple choice
About
55%
How much did it cost to go to the cinema– Multiple choice
– Complex multiple choice
How much did it cost to go to the cinema
in Macondo?
One centavo
Two centavos
About
Two centavos
Three centavos
Four centavos
• Constructed response
Cl d t t d
45%
D ith th fi l j d t f th l f– Closed constructed response
– Short constructed response
Do you agree with the final judgement of the people of
Macondo about the value of the movies?
Explain your answer by comparing your attitude to the
movies with theirs
– Extended constructed response
movies with theirs.
___________________________________________
___________________________________________

20. Why do countries participate iny p p
international studies of educational
achievement?achievement?
• To collect national-level data whenTo collect national level data when
there is no other mechanism for doing
soso
• To compare themselves with otherp
countries
• To be involved in and help to shape• To be involved in – and help to shape -
new developments in education

21. PISA Survey Cycley y
2000 2003 2006 2009 2012 2015
Reading Literacy
Mathematical Literacy
Scientific Literacy
Questionnaire

22. PISA Survey Cycley y
2000 2003 2006 2009 2012 2015
Reading Literacy
Mathematical Literacy
Scientific Literacy
Questionnaire
Problem Solving

23. PISA Survey Cycley y
2000 2003 2006 2009 2012 2015
Reading Literacy
Mathematical Literacy
Scientific Literacy
Questionnaire
Problem Solving
Financial LiteracyFinancial Literacy

24. PISA Survey Cycley y
2000 2003 2006 2009 2012 2015
Reading Literacy
Mathematical Literacy
Scientific Literacy
Questionnaire
Problem Solving
Financial LiteracyFinancial Literacy
Computer-based Science Digital
reading
Three
domains
ALL
domains

25. NEW DIRECTIONS IN PISANEW DIRECTIONS IN PISA
READING:READING:
DIGITAL READING

26. What is digital reading?
Reading literacy is understanding, using,
reflecting on and engaging with writtenreflecting on and engaging with written
texts, in order to achieve one’s goals, to
develop one’s knowledge and potentialdevelop one s knowledge and potential,
and to participate in society.
(PISA definition of reading literac )(PISA definition of reading literacy)
• In the 21st century, proficiency in reading
means proficiency in both print and digital
media
• Digital reading is not just reading print
text on a computer screen

27. Analogies …Analogies …
Page of a goIn the print medium In the electronic medium
Page of a go
Page of a book Web page
A book A web site
Bibliography, references, “further
reading”
Links on a web site to other sites
A very very very big library The Internet

28. In PISA digital reading isIn PISA, digital reading is
conceptualised in terms of two
components
• Text processingText processing
• Navigation

29. Relationship between navigation and
text processing for digital reading taskstext processing for digital reading tasks

30. Example of a PISA digital reading task (1)g g ( )

31. Example of a PISA digital reading task (2)g g ( )

32. Example of a PISA digital reading task (3)g g ( )

33. DRA Sample ItemsDRA Sample Items
http://erasq.acer.edu.auhttp://erasq.acer.edu.au

34. Thank youy
For more information about PISA go tog
www.pisa.oecd.org