This document describes the process undertaken by a CUNY task force to develop an evaluation tool for e-book collections. The task force included representatives from various CUNY institutions. They created categories and questions to develop a weighted checklist to evaluate e-book packages. Task force members then used the checklist to evaluate two e-book vendor proposals, which facilitated in-depth discussions and helped the task force come to a clear recommendation. Based on their experience, the task force refined the checklist and shared the results and survey instrument with other CUNY librarians.

1. P T . 1 : D E V E L O P I N G A N E V A L U A T I O N T O O L
CUNY E-BOOK TASK
FORCE:

2. CITY UNIVERSITY OF NEW YORK
(CUNY)
• 11 senior colleges, 7 community colleges, Macaulay
Honors College, and 5 graduate and professional
schools—all located throughout the 5 boroughs.
• 269,000 degree-credit students; 270,000 certificate
and professional students.
• From certificate courses to Phd programs, CUNY
offers postsecondary learning to a remarkably
diverse population of students of all backgrounds
from 208 countries.

3. AN INTRODUCTION TO OUR LIBRARIES
7
Community
Colleges
5
Graduate
Center and
Professional
Schools
Office of Library Services
11
Senior
Colleges
Aleph catalog
SFX link resolver
EZ proxy
Cataloging
support
E-resources
Students
Expertise
Print collections

4. ELECTRONIC RESOURCES ADVISORY
COUNCIL (ERAC) OF CUNY
• 1 Representative from each campus library
• Usually the Electronic Resources Librarian
• Many consortium pricing arrangements are
discussed and decided upon at ERAC (some
funded by the University; others funded by
individual libraries)
• From this Committee, the University Dean of Libraries
put together a task force.

5. TASK FORCE CHARGE:
• Task Force 1: Develop the evaluative criteria we
should be using to determine which, if any, e-book
collection would best suit the needs of the
University.

6. TASK FORCE MEMBERS
• Task Force 1:
• Nancy Egan, Chair, John Jay
College
• Linda Dickinson, Hunter College
• John Drobnicki, York College
• Madeline Ford, Hostos CC
• Maria Kiriakova, John Jay
College
• Catherine Stern, LaGuardia CC
• Lisa Tappeiner, Hostos CC
• Mike Waldman, Baruch College
• Task Force 2:
• Angela Sidman, Central
Office, Co-Chair
• Nancy Egan, John Jay
College, Co-Chair
• Jane Fitzpatrick, Graduate
Center, CUNY
• Helen Georgas, Brooklyn
College
• Maria Kiriakova, John Jay
College
• Laroi Lawton, Bronx CC
• Linda Roccos, College of Staten
Island
• Catherine Stern, LaGuardia CC
• Lisa Tappeiner, Hostos CC
• Susan Vaughn, Brooklyn College
and Central Office

7. COMMITTEE MAKE-UP, TASK FORCE 1
62%
38%
Members by Institution Type
Senior College
Community
College
29%
22%21%
7%
7%
7%
7%
Members by Functional Area
acquisitions
electronic resources
collection
management
cataloging
media
public services
chief

8. TASK FORCE 1

9. TASK FORCE 1
INITIAL MEETING
• Fleshing out the mission
• Determining broad categories:
Ease of Procurement
Content
Technical Requirements
Licensing Terms
Collection Building
Cataloging and Catalog Maintenance
Compatibility with other Library Resources
Administrative Functions
User Experience
Product Support for Librarians
Library Culture

10. SHARING THE WORK
• Used categories to create outline
• Every category had at least one person.
• Shared articles, thoughts, glossary terms, etc. on
GoogleDocs
• Used GoogleDoc information to fill in the outline
• Second meeting & discussion to refine model

11. THE CHECKLIST
• URL: http://guides.lib.jjay.cuny.edu/ebooktaskforce
• Categories and subcategories
• Questions under each category and/or
subcategory
• An accompanying glossary and report

12. WEIGHTING SYSTEM
• Making the system work:
• Weighting system will be for this particular purpose
• Categories and subcategories are assigned a weight
• 1 is considered helpful, 5 is important, and 10 is very
important or vital to the needs of the University
community
• Questions are used to flesh out each category (or
subcategory) and determine how the vendor fares in
each category
• The vendor gets points—all or none in each
category—based on whether they’ve met the criteria

13. TASK FORCE CHARGE
• Task Force 2: Apply the model to two competing
vendors’ Ebook packages and make a
recommendation to the University Librarian.

14. P T . 2 : A P P L Y I N G T H E E V A L U A T I O N T O O L
CUNY E-BOOK TASK
FORCE:

15. TASK FORCE, PT. 2
• Goal:
• To apply the checklist
and evaluate the
two competing
ebook packages
• To evaluate the
effectiveness of the
checklist
• Method:
• 10 people on
committee
• Cross section of
functional areas and
institutions
represented
• Each person
evaluates two areas

16. COMMITTEE MAKE-UP, ROUND II
10%
50%
30%
10%
Members by institution type
Central Office
Senior College
Community
College
Graduate
Center
30%
20%20%
10%
10%
10%
Members by functional area
E-resources
Media
Collection
management
Public services
Acquisitions
Cataloging

17. EVALUATION IN ACTION
• Assigned 2 sections each for review
• Given a month to evaluate both ebook packages
• Base conclusions on:
• Own evaluation
• Information provided by the vendor
• Feedback from colleagues
• Regroup and discuss > Formally assign points

18. EVALUATING THE CHECK LIST
• Would the checklist help us evaluate each product
fully?
• Would the all or nothing points system prove
effective?
• Are partial points needed?
• Would discussion be productive?
• Would one package come out the clear winner?

19. WHAT WE FOUND: EBOOKS
• Both ebook packages were excellent
• Use of the checklist teased out differences between
them and allowed us to weigh our own values and
come to a clear decision

20. WHAT WE FOUND: CHECKLIST
Pros
• Elicited great discussion
• The list covered a
range of scenarios but
easy to pick only
applicable ones
• Provided a neutral
avenue for considering
products, bypassing
vendor prejudices
• The points worked!
Cons
• Written to cover any
scenario, so quite long
and time consuming to
apply
• For best results (and
most likely consensus)
multiple participants
are needed

21. OUTCOMES
• Based on evaluation, change made to checklist
• Wrote findings up in a report, shared with University
Dean, E-resources Advisory Committee, and on
internal listserv of 300+ librarians
• Allowed for harmonious decision-making
• Transparent process
• Participation and buy-in from all major groups
• Basis of feedback provided to vendors
• Shared survey instrument with other CUNY librarians
• Available via Support Site