PISA 2009: Evaluating Systems to Improve Education
1. Programme for International Student Assessment
1
1
Students On Line
PISA 2009
Evaluating systems to improve education
PISA
International Student Assessment
OECD Programme for
Sophie Vayssettes
Indicators and Analysis Division, EDU, OECD
2. 2
2
PISA 2009 in brief
• Half a million students in 74 countries/economies
covering 87% of world economy
Students On Line
• 2-hour test of skills that go beyond reproducing
what is taught in schools
• Information on student background, attitudes and
PISA
International Student Assessment
OECD Programme for
their schools + data from parents, principals and
system leaders
• 19 countries participated in digital reading
assessment
3. 3
3
OECD’s computer-based assessments
• PISA 2009 digital reading assessment
19 countries
Students On Line
• PISA 2012
43 countries
• PISA 2015
Almost all countries
PISA
International Student Assessment
OECD Programme for
• PIAAC - international survey of adult skills
interviewing adults aged 16-65 years in their homes – 5 000 in
each participating country;
literacy and numeracy skills and ability to solve problems in
technology-rich environments;
results will be published in 2013
4. 4
4
Why PISA 2009 included a digital
reading assessment
• “Digital reading” means reading on screen, mostly
Students On Line
via the Internet
• Digital reading reading on line is an
PISA
International Student Assessment
OECD Programme for
increasingly important form of reading in the 21st
century
• Digital reading has different features to print
reading and makes new demands on readers
5. 5
5
Why PISA 2009 included a digital
reading assessment
• “Digital reading” means reading on screen, mostly
Students On Line
via the Internet
By March 2011, more than 2 billion people were using
the Internet worldwide
• Digital reading is an increasingly important form of
PISA
International Student Assessment
OECD Programme for
reading in the 21st century
• Digital reading has different features to print
reading and makes new demands on readers
6. 6
6
Why PISA 2009 included a digital
reading assessment
• “Digital reading” means reading on screen, mostly
Students On Line
via the Internet
• Digital reading reading on line is an
increasingly important form of reading in the 21st
century
PISA
International Student Assessment
OECD Programme for
taxpayers fill in online forms;
students search the web for information;
jobseekers look up ads on employment websites;
consumers order goods in online stores;
people build and maintain social communities on line.
• Digital reading has different features to print
reading and makes new demands on readers .
7. 7
7 Digital reading has different features to print reading
and makes new demands on readers
There are different kinds of texts
There are many mixed texts, including short pieces, diagrams, graphics,
fragments
Students On Line
There are many interactive texts, where readers can change, add to, or
create their own texts
The reader has more responsibility, in selecting and constructing what
they read
There is often no guide to the order in which a text should be read – The
PISA
International Student Assessment
OECD Programme for
reader can take many pathways
There is an almost infinite number of texts available
The extent of a text is abstract and unknown – the reader cannot see the
whole text
The reader has more responsibility, in evaluating the text
Traditional filters, like publishers and booksellers, are absent
The reader has to ask, “Is this text useful for me?” “Do I trust this text?”
8. 8
8
Digital reading features and demands –
How were these features implemented in PISA 2009?
There are different kinds of texts
Students On Line
The tasks are based on a range of typical digital text types and
formats
The reader has more responsibility, in selecting
and constructing what they read
PISA
International Student Assessment
OECD Programme for
Students often needed to navigate to find the relevant part of the
text
Digital reading texts included many different pages
The reader has more responsibility, in evaluating
the text
Some tasks asked students to evaluate the text
An example task
9. PISA Students On Line
OECD Programme for
International Student Assessment 9
9
Example of a PISA digital reading task (1)
10. PISA Students On Line
OECD Programme for
International Student Assessment 10
10
Example of a PISA digital reading task (2)
11. PISA Students On Line
OECD Programme for
International Student Assessment 11
11
Example of a PISA digital reading task (1)
12. PISA Students On Line
OECD Programme for
International Student Assessment 12
12
Example of a PISA digital reading task (3)
13. PISA Students On Line
OECD Programme for
International Student Assessment 13
13
Example of a PISA digital reading task (4)
14. PISA Students On Line
OECD Programme for
International Student Assessment 14
14
Example of a PISA digital reading task (5)
15. PISA Students On Line
OECD Programme for
International Student Assessment 15
15
Example of a PISA digital reading task (6)
16. PISA Students On Line
OECD Programme for
International Student Assessment 16
16
Example of a PISA digital reading task (7)
17. PISA Students On Line
OECD Programme for
International Student Assessment 17
17
Example of a PISA digital reading task (8)
18. PISA Students On Line
OECD Programme for
International Student Assessment 18
18
Example of a PISA digital reading task (9)
19. 19
19
Identifying key elements of digital reading
Students On Line
• text processing locating, interpreting
and evaluating words on the screen
• navigating from one screen to
PISA
International Student Assessment
OECD Programme for
another, to find relevant information
20. 20
20 Performance in digital reading
350 400 450 500 550 600
Korea 568
New Zealand 537
Australia 537
Students On Line
Japan 519
Hong Kong-China 515
Iceland 512
Sweden 510
Ireland 509
Belgium 507
Norway 500
PISA
International Student Assessment
OECD Programme for
OECD average 499
France 494
Macao-China 492
Denmark 489
Spain 475
Hungary 468
Poland 464
Austria 459
Chile 435
Colombia 368
21. PISA Students On Line
OECD Programme for
International Student Assessment 21
21
0
80
60
40
20
20
60
80
40
100
Korea
Japan
Australia
Percentage of students
Hong Kong-China
New Zealand
Macao-China
Ireland
Iceland
Sweden
Norway
Belgium
Denmark
France
OECD average - 16
Spain
Poland
Hungary
Austria
Chile
How proficient are students in digital reading?
Colombia
22. 22
22
Differences between boys and girls
compared with print reading
Gender difference in print reading performance (girls- boys)
50
Poland
Norway
Girls have a larger
Sweden New Zealand
advantage in print reading Iceland
Austria
Students On Line
40 France
Japan
Ireland
Hungary
OECD Average
Macao - China
Australia
Korea
Hong Kong -China
30
Belgium
Denmark
Spain
PISA
International Student Assessment
OECD Programme for
Chile
20
10 Girls have a larger advantage
Colombia
in digital reading
0
0 10 20 30 40 50
Gender difference in digital reading performance (girls-boys)
23. 23
23
Navigation
• Navigation is considered to be part of the
Students On Line
cognitive process of digital reading
• Tracking and analysing the sequences of pages
PISA
International Student Assessment
OECD Programme for
students visit provide insights into effective
navigation behaviours in digital reading
24. 24
24 Number of relevant pages visited and
Digital reading
score digital reading performance
600
Korea
Students On Line
550 New Zealand
Australia
Hong Kong Japan
Iceland
Ireland Sweden
500 Norway Belgium
France
y = 8.666x + 96.82 Macao Denmark
R² = 0.958 Hungary Poland
Spain
PISA
International Student Assessment
OECD Programme for
450 Austria
Chile
400 The index number of relevant pages
visited describes how many of the
pages judged to be relevant to a
Colombia task were accessed while the
student worked on that task
350
30 35 40 45 50 55
Number of relevant pages visited
25. 25
25
ICT questionnaire
• Students’ access to and use of computers and
Internet
Students On Line
• Their general attitudes towards and self-
confidence in using computers
• 29 OECD countries and 16 partner countries and
economies
PISA
International Student Assessment
OECD Programme for
• Not designed to assess the quality of ICT use at
school and the integration of ICT in pedagogy
• Focus on students’ use of ICT to access, manage
and present information
26. PISA Students On Line
OECD Programme for
International Student Assessment 26
100
0
10
20
30
40
50
60
70
80
90
Denmark
Finland
Iceland
Norway
Sweden Percentage of students
Switzerland
Korea
Germany
Australia
Austria
Canada
Belgium
2009
Portugal
Czech Republic
Ireland
2000
France
Italy
New Zealand
Israel
Poland
OECD average
- 27
Hungary
United States
Spain
between PISA 2000 and 2009
Greece
Japan
reported having a computer at home
Chile
26 Changes in percentage of students who
Mexico
27. PISA Students On Line
OECD Programme for
International Student Assessment 27
0.00
0.05
0.10
0.15
0.20
0.25
Austria
Mean Index
Norway
United States
New Zealand
Australia
Japan
Sweden
Hungary
Korea
Switzerland
Denmark
Belgium
2009
OECD average-25
Finland
Czech Republic
2000
Iceland
Ireland
Portugal
Italy
Poland
Germany
Mexico
between PISA 2000 and 2009
Israel
Greece
Spain
27 Changes in computers-per-student ratio
Chile
28. 28
28 Digital divide by country
However, some countries lag behind on digital use
100
Netherlands
Norway , Denmark
Australia
90
Sweden
Students On Line
Finland
Percentage of students who use a computer at school
Austria
New Zealand
Canada
80 Iceland
Slovak Republic Czech Republic
Switzerland
OECD
70 Hungary
PISA
International Student Assessment
OECD Programme for
Spain
Germany
Ireland Italy Belgium
Korea
Poland
60 Japan
Greece Slovenia
Chile
Estonia
Portugal
Turkey Israel
50
40
40 50 60 70 80 90 100
Percentage of students who use a computer at home
29. 29
29 Digital divide by socio-economic
background within countries
Within countries, there is a digital divide by socio-economic background of students
60
Disadvantaged students have less Panama
50
access to a computer at school
Students On Line
40
Percentage difference (top - bottom) at school
30
Disadvantaged students
have less access to a
20
computer at home
Thailand
Jordan
PISA
International Student Assessment
OECD Programme for
Serbia
10
Canada
Iceland New Zealand
Australia Qatar Russian Federation Turkey
Sweden
Finland Bulgaria
Korea Trinidad and Tobago
Norway Estonia Israel Lithuania Japan
Denmark Slovak Republic Uruguay
Hong-Kong China Czech Republic
0 Macao-China Ireland
Netherlands Belgium Chile
Austria Germany
Liechtenstein Slovenia
Switzerland Singapore Croatia
Poland Hungary Greece
-10
Italy Latvia
Portugal
-20
Disadvantaged students have more
-30
access to a computer at school
0 10 20 30 40 50 60 70
Percentage difference (top - bottom) at home 80
30. 30 ICT use at home for leisure and digital
30
reading performance
520
Play one-player games
Students On Line
510
Play collaborative online games
500 Use E-mail
Digital Reading Performance
490 Chat on line
PISA
International Student Assessment
OECD Programme for
Browse the Internet for fun
480
Download music, films, games
470 or software form the Internet
Publish and maintain a personal
460 page, weblog or blog
Participate in online
forums, virtual communities or
450 spaces
never or hardly ever Once or twice a monthOnce or twice a week
Everyday or almost everyday
31. 31
31 ICT use at home for schoolwork and
digital reading performance
520
Students On Line
Browse the Internet for schoolwork
510
500 Use e-mail to communicate with
Digital Reading Performance
other students about schoolwork
490
PISA
International Student Assessment
OECD Programme for
Use e-mail to communicate with
teachers about schoolwork
480
470
Download, upload or browse
material from your school's website
460
Check the school's website for
450 announcements
Never or hardly Once or twice a Once or twice a Everyday or
ever month week almost everyday
32. 32 ICT use at school and digital reading
32
performance
520
Chat on line
Students On Line
500
Use e-mail
480
Browse the Internet for schoolwork
Digital Reading performance
460 Download, upload or browse material
from the school's website
PISA
International Student Assessment
OECD Programme for
440 Post your work on the school's
website
420 Play simulations at school
400 Computer use at school for practice
and drilling
Doing individual homework on a
380 school computer
Group work and communication with
360 other students
Never or hardly ever Once or twice a Once or twice a Everyday or almost
month week everday
33. 33 After accounting for performance in print
33
reading
Digital reading performance Leisure at home
510
Play collaborative online
games
Students On Line
Browse the Internet for fun
500 Schoolwork at home
Browse the Internet for
schoolwork at home
PISA
International Student Assessment
OECD Programme for
Use e-mail for
communication with other
490 students about schoolwork
At school
Browse the Internet for
schoolwork at school
Practice and drilling
480
Never or hardly ever or twice a month or twiceEvery day or almost every day
Once Once a week
34. 34
34 Computer use and navigation
(students with above-average reading scores)
Index of the number of
relevant pages visited
7 Leisure at home
Play collaborative online games
Students On Line
Browse the Internet for fun
6
Schoolwork at home
Browse the Internet for
schoolwork at home
PISA
International Student Assessment
OECD Programme for
5
Use e-mail for communication
with other students about
schoolwork
4
At school
Browse the Internet for
schoolwork at school
Practice and drilling
3
Never or hardly ever Once or twice a month Once or twice a week Every day or almost
every day
35. 35
35 Some conclusions
• Identifying effective strategies to teach digital reading
skills is an important policy objective
Ability to critically evaluate the quality and credibility of texts,
integrate information from multiple texts and – crucially – navigate
Students On Line
effectively
• ICT use at home for leisure is – up to a point – positively
related to performance, navigation skills and self-
confidence in completing high-level ICT tasks
Parents and teachers need to encourage students to use
PISA
International Student Assessment
OECD Programme for
computers so that they can improve their navigation skills but also
provide guidance on balancing time spent using computers with
time for other activities
• Digital reading can be a lever to reduce the gender gap
The gender gap in digital reading is much smaller than in print
reading, and relates to differences in navigation skills between
boys and girls
Reading more and reading with enjoyment promotes better
reading, and better reading fosters stronger engagement .
36. 36
36 Some conclusions
• Access to computers has improved significantly, both at
home and at school, but important gaps remain between
countries and social groups
Students On Line
Strategies that promote wider access to ICT at school can help
minimise impact of social background on digital competency gaps,
with possible consequences for future employment opportunities
• Schools can do more to integrate ICT into learning
Opportunities for students to solve problems using ICT
Reading methods that improve students’ ability to distinguish
PISA
International Student Assessment
OECD Programme for
between relevant and irrelevant material, and to structure,
prioritise, distil and summarise texts
• ICT can…
Enable students to obtain more regular feedback on their learning
processes
Make students more active participants in learning processes in
classrooms and tailor these processes to individual students’
needs
Provide students with up-to-date access to the world’s current
research and thinking .
37. 37
37
THANK YOU
Students On Line
Find out more about PISA at…
OECD www.pisa.oecd.org
PISA
International Student Assessment
OECD Programme for
All national and international publications
The complete micro-level database
Email: Sophie.Vayssettes@OECD.org