1. LIB 620 Library Management 1
Fall 2011
Evaluating School Media Services

2. 2
Based in part on:
Preemptive Action:
The busy librarian’s guide to
program evaluation
A workshop developed by the
Pennsylvania School Librarians Association
Professional Development Committee
(No longer available online)
indicates the slide was originally from the above presentation

3. 3
Evaluate?
What do we mean by evaluate?
– verb (used with object), -at⋅ed, -at⋅ing.
1. to determine or set the value or amount of;
appraise: to evaluate property.
2. to judge or determine the
significance, worth, or quality of; assess: to
evaluate the results of an experiment.
3. Mathematics. to ascertain the numerical value
of (a function, relation, etc.).
• The American Heritage®
Dictionary of the English Language, Fourth Edition © 2006 by Houghton Mifflin
Company.

4. 4
http://www.evaluationtrust.org/evaluation/evaluate

5. 5
What about school library evaluation?
Evaluation of school library media
centers:
– When you examine something or
someone, you examine the subject and
make a judgment about the
quality, significance, or condition of
whatever is to be evaluated.
• Emanuel T. Prostano and Joyce S. Prostano,
The School Library Media Center 5th ed. (Englewood, CO:
Libraries Unlimited, 1999), 44.

6. 6
Purposes of evaluation
To find out what is right
To find out what is wrong
• Summary of definition in Blanche Woolls, The School
Library Media Manager 2nd ed. (Westport, CT: Libraries
Unlimited, 1999).

7. 7
What can you evaluate?
Everything!
– Collection – Reference
– Personnel service/collection:
– Library Layout • Interview skills
– Programs • Quality of the answers
provided
– Curriculum collaboration
• Satisfaction of your
patrons
– Students or teachers—or
other school staff

8. 8
What should you evaluate?
That depends . . .
– On your priorities
– On your economic needs and situation
– On the policies and procedures established by
your school, school district and/or state
– On how much time you have left over from
your other duties to devote to planning and
executing an evaluation project

9. 9
Why evaluate?
Because you have to
– Continuous assessment of services can be an
ongoing requirement for
accreditation/funding, etc.
Because you want to
– You want to find out how well you‟re doing
– You want to find out how well others think
you‟re doing
– You want to find ways to improve your
service

10. 10
Why Evaluate?
• Schools are being evaluated by student
academic achievement in
reading, writing and math.
• Recent studies show student
achievement correlates positively with
effective school library programs.

11. 11
Why Evaluate?
Studies show the positive correlation
with student achievement occurs when
there is an effective school library
program [and]:
– School librarians are full time and
– Librarians collaborate with teachers on
instructional planning.

12. 12
Why Evaluate?
Studies also show an effective
school library program includes
the following attributes:
– Quality collections;
– Increased hours of access beyond school day;
– Professional development for teachers and
librarians;
– Student access to technology; and
– Collaboration with other types of libraries.

13. 13
Why Evaluate? Because. . .
To improve, the librarian must have baseline
data about the program.
To be relevant, the librarian must know if the
library services and resources are aligned with
the school‟s goals for student achievement.
To build a case for better support, the
librarian needs data that demonstrates the
value of the program.

14. 14
What Can an Evaluation Do?
An evaluation enables you to:
– Determine success in attaining program
goals.
– Determine students‟ and teachers‟ needs
so they can be incorporated into the program.
– Provide a basis for resource allocation.
– Recognize strengths and accomplishments.
– Examine the impact of the program on
student learning.
• Nancy Everhart, Evaluating the School
Library Media Center, 1998)

15. 15
It makes sense—
 Focus data collecting and
evaluation on those components of a
school library program that the
research shows correlate with student
achievement.

16. 16
Types of Evaluation
 Formal
 Informal
 External
 Internal
 Formative
 During a program
 Summative
 After the program is completed

17. 17
Evaluation requires standards
How do you know if the media center is ―good‖/ ―not
good‖ or even ―good enough‖?
– “Good enough” for what? For whom?
Guidelines or rubrics or objectives for
collections, performance can be found in:

18. 18
Types of Data
Quantitative Qualitative
Number of OPAC Students‟ success rate in
locating appropriate
searches
resources in OPAC searches
Size of Collection supports the
collection curriculum
Number of Comprehensive and
lessons planned collaborative planning is in
with teachers place

19. 19
Standards require interpretation
Quantitative: relatively easy
– Library Media Staffing
– The school library media program is staffed as follows:
– Under 200: 1 full-time school media librarian (SML)
– 200-500: 1 full-time SML; .5 clerk
– 501-800: 1 full-time SML; 1 clerk
– 801-1200: 1 full-time SML; 1.5 clerk
– 1201-1600: 1.5 SML; 1.5 clerk
– 1601-2000: 2 full-time SML; 2 clerks
– 2001 and Up: 2.5 SML; 2 clerks
» Library Media Program Rubric August 2010

20. 20
Standards require interpretation
Qualitative: more difficult
What constitutes “sufficient funding”?

21. 21
Interpretation means operationalization
Operationalize:
– To define a concept in a way that can be
measured. In evaluation research, to translate
program inputs, outputs, objectives, and goals
into specific measurable variables.
• Program Evaluation Glossary
http://www.epa.gov/evaluate/glossary/o-esd.htm

22. 22
Interpretation means creating goals and objectives
The purpose and study goals should determine
the types of methods and measures you use to
conduct the evaluation
Objectives will define your standard of
excellence--the minimum level of appropriate
service for your particular clientele
– Jo Bell Whitlach, Evaluating Reference Services

23. 23
Methods of Evaluation
Obtrusive
– People are aware of the evaluation
• Self-evaluation
• Surveys
• Observation
Unobtrusive
– People are unaware of the evaluation
• Unobtrusive measures of physical facilities
• Use of proxies—“mystery patrons”

24. 24
Obtrusive reference evaluations in a school
library context
Questionnaires or interviews of students or
teachers
Numbers gathering:
– Reference question counts
– Numbers/types of reference books used
– Circulation statistics
Observation
– By external observer
– Self-observation: Journal

25. 25
For example. . .
Research finding:
Students whose school librarian plays an
instructional role tend to achieve higher than
average test scores. This is also dependent on
collaboration between school librarians and
teachers and the inclusion of the library
materials in the curriculum.

26. 26
Evaluation Model
1. Define the question.
2. Collect data--determine needed data and
method of collection.
3. Analyze the data.
4. Formulate recommendations.
5. Develop an action plan.

27. 27
Evaluation Model Step 1.
Define the Question
Research Finding The Question
Students whose school What data are needed
librarian plays an to determine whether
instructional role or not the school
tend to achieve higher librarian plays an
than average scores. instructional role in
the school?

28. 28
Look at the rubric

29. According to 29
Information Power (1998)--
Instructional role is determined by:
Collaboration between school librarians
and teachers that results in the inclusion of
library materials in the curriculum.

30. According to 30
Information Power--
Instructional role is defined by:
– Information literacy standards
– Collaborative planning
– Effective teaching
• Differentiated learning options
• Inquiry
• Assessment
– Student Engagement

31. 31
According to Empowering Learners (2009)
Guideline 1:
– The school library media program promotes
collaboration among members of the learning
community and encourages learners to be
independent, lifelong users and producers of
information.

32. 32
According to Empowering Learners
Guideline 3:
– The school library media program provides
instruction that addresses multiple
literacies, including information
literacy, media literacy, visual literacy, and
technology literacy.

33. 33
Evaluation Model Step 2.
Collect data
Determine the type needed.
– Quantitative/Qualitative
Determine the method of collection.
– Existing statistic
– Survey, questionnaire, focus
group, observation

34. 34
One method to collect data--
Use Information Power‟s Teaching and
Learning Rubric as a questionnaire.
Administer it to
– School Library staff,
– Administrative staff,
– Teaching staff.
Summarize each group; determine where
perceptions differ and agree.

35. 35
Data collected from the Teaching and Learning Rubric
are qualitative--
Example:
Principal puts librarian on weekly faculty
meeting agenda to encourage library use
and promote curriculum-oriented library
materials. Math teacher does not see an
easy connect and tunes out.
When completing the questionnaire--
School librarian & principal:  EXEMPLARY!
Math teacher: 0 NOTHING!

36. 36
Evaluation Model Step 3.
Analyze the data
How can the data be analyzed to determine
whether or not the librarian plays an effective
instructional role in the school?
– Identify areas of agreement and disagreement
among groups.
– Identify areas of strength and areas that need
improvement.
– Identify areas where more data is needed.

37. 37
Agreement Example--
Target Indicator: Information Literacy Standards
are integrated into content learning.
Library Staff Response
____Basic 100% Proficient ____ Exemplary
Administrative Response
____Basic 100% Proficient ____ Exemplary
Teaching Staff Response
12% Basic 88% Proficient ____ Exemplary

38. 38
Disagreement Example--
Target Indicator: Curriculum development is
modeled and promoted.
Library Staff Response
100% Basic _____ Proficient ____ Exemplary
Administrative Response
____Basic 100% Proficient ____ Exemplary
Teaching Staff Response
82%Basic 18% Proficient ____ Exemplary

39. 39
Strength Area Example--
Target Indicator: Collaborative planning is
modeled and promoted.
Library Staff Response
____Basic 100% Proficient ____ Exemplary
Administrative Response
____Basic 100% Proficient ____ Exemplary
Teaching Staff Response
____Basic 100% Proficient ____ Exemplary

40. 40
Weakness Area Example--
Target Indicator: Students are engaged in
reading, writing, speaking, viewing & listening
for enjoyment, enrichment, & understanding.
Library Staff Response
____Basic 100% Proficient ____ Exemplary
Administrative Response
100% Basic ____ Proficient ____ Exemplary
Teaching Staff Response
82% Basic 18% Proficient ____ Exemplary

41. 41
Contradictory Data Example—
Target Indicator:Effective teaching modeled &
promoted.
Library Basic ____ Prof. 100% Exemplary
Admin. ____ Basic ____ Prof. 100% Exemplary
Teacher Basic 60% Prof. 40% Exemplary
Target Indicator: Student achievement is
assessed.
Library 100% Basic Prof. Exemplary
Admin. 100% Basic ____ Prof. Exemplary
Teacher 90% Basic 10% Prof. Exemplary

42. 42
Back to Step 2
Collecting More Data – Focus Groups
1. How are student products and performances assessed
in units involving use of the library?
2. How do you determine the products and performances
to demonstrate mastery of content in units involving
use of the library?
3. What is the role of reflection in student work
involving use of the library?

43. 43
Evaluation Model Step 4.
Formulate recommendations
Role of an Advisory Committee of
stakeholders—
• Review the data, formulate recommendations.
• For example:
– Because of the strong correlation between the school
librarian‟s instructional role and student
achievement, the Advisory Committee establishes a
three year goal of reaching the Exemplary level in all
target indicators in the Teaching and Learning
Rubric

44. Example: 44
Recommendations for First Year
Standards for the 21st Century Learner
– Review the Standards for the
21st Century Learner to ensure they
align with content standards and
set student expectations for analysis,
evaluation and inquiry.
– Meet with teaching teams to solicit recommended
revisions.
– Convene a workshop for new teachers to review the
standards.
Adapted from

45. Example: 45
Recommendations for First Year
Collaborative Planning
– Establish regular common planning time for
teachers with the library media staff.
Curriculum Development
– Review school policies to remove any barriers that
prevent librarians from participating in building and
district curriculum sessions.

46. Example: 46
Recommendations for First Year
Reading, Writing, Speaking, Viewing
– Provide a series of author workshops and develop
plans with teachers for student
reading, writing, speaking, and viewing responses to
the author workshops.
– Involve Parents.
Effective Teaching
– Clarify conflicting feedback through focus groups
targeting use of assessment, differentiation, & inquiry
in instruction.
– Develop recommendations based on new data.

47. 47
Evaluation Model Step 5.
Develop an action plan
Library staff develop an action plan for
each recommendation.
The Advisory Committee of
Stakeholders reviews and approves the
action plans.

48. 48
Action Plan Example:
Target Indicator: Curriculum development is modeled and promoted
Objective Activity Documentation Participants New Completion
Resources Date
Review barriers Review School Council School None Sept. 1
preventing policies minutes Council & projected
librarians from admin.
participating in Secure Post schedule in Principal None Sept. 10
curriculum schedule lib. & faculty rm.
sessions.
Participate Meeting minutes Librarians None June 1
in mtgs. projected
Report Faculty mtg. Librarians None Monthly
back minutes
Written In professional Librarians None Within 5
material library days of mtg

49. 49
This example of an evaluation was…
• Type (which one?) • Data collection methods:
– Formal – Research
– Internal – Rubric-based
– Formative questionnaire
– Focus Group
• Data type:
– Quantitative
– Qualitative

50. 50
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
– “The basic premise . . . is that you can learn a great
deal . . . by looking at how things wear
(„erosion‟), how things are left in the building
(„traces‟) and how things are rearranged
(„adaptations for use‟).”
• Nancy Everhart, Evaluating the School Library
Media Center: Analysis Techniques and
Research Practices. Libraries Unlimited, 1998.

51. 51
Unobtrusive evaluation in schools 2
Use of proxies—―mystery patrons‖
– Technique used more often in academic libraries and
government documents reference services
• Half-right reference: the 55% rule
• 5-minute rule
– Peter Hernon and Charles R. McClure, “Unobtrusive
Reference Testing: The 55 Percent Rule” Library
Journal April 15, 1986, 37-41.
• “It‟s not true, and now we know why . . . the so-called “55%
rule” has never been tested against a truly representative
field sample.”
– John V. Richardson, Jr., “Reference Is Better Than We
Thought,” Library Journal April 15, 2002, 41-42.

52. 52
Importance of a sophisticated model
Richardson:
– “The reference service performance model [that led
to the 55% rule] was overly simplistic, samples were
way too small, and the test questions were not
representative of real-world reference questions.”
– “Students of reference service should learn about the
existence of multiple performance outcomes
(i.e., accuracy, utility, and satisfaction) and to
recognize that each outcome is driven by different
factors.”
• “Reference Is Better Than We Thought.”

53. 53
Characteristics of good models
Measures of reference service must be
– Valid
• They “accurately reflect the concept being studied.”
– Reliable
• They “are stable and dependable, and provide consistent
results with each repeated use.”
– Practical
• They “require that data be relatively easy to collect.”
– Useful
• They “provide information that can be used to
improve reference services.”
– Whitlach, Evaluating Reference Services.

54. 54
Problems of Evaluation in School Context
Often the school librarian is the only
one working in the library
An evaluation project can be time-
consuming to plan and to put into action

55. 55
Solutions to evaluation problems
Depends on your local situation
– Your resources:
• Time, money, available warm bodies
– You and your creativity
Make evaluation part of your routine
– As you plan/prepare your program(s), include
an evaluation component
• Appropriate, affordable, accessible, accountable