Description of a systematic review and evidence based librarianship related to virtual reference services.

1. Evaluating E-Reference: An Evidence Based Approach Elaine Lasda Bergman and Irina I. Holden University at Albany Presentation for Reference Renaissance Denver, CO August 10, 2010

2. Overview What is Evidence Based Librarianship? Methods What constitutes “evidence?” Systematic reviews and analyses Systematic Review Process Research question Database Search Article Review Critical Appraisal Synthesize, analyze, discuss

3. Overview Results of our review Methods of determining user satisfaction Comparison of variables Range of results Conclusions, lessons learned About evidence based librarianship About research quality About user satisfaction with electronic reference

5. “The Retrieval of rigorous and reliable evidence to inform… decision making” (Booth and Brice, ix)

6. What is Evidence Based Librarianship (EBL)? History Gained traction in Medical fields in 1990’s and spread to social sciences after that Medical librarians were the first to bring this approach to LIS research Increasingly used in social sciences and information/library science Sources: Booth and Brice, ix.

7. Don’t we ALREADY use “evidence”? Evidence is “out there, somewhere” Disparate locations: many different journals, many different researchers Evidence is not summarized, readily available and synthesized No formal, systematized, concerted effort to quantify and understand if there is a pattern or just our general sense of things

8. Heirarchy of “Evidence” Source: http://ebp.lib.uic.edu/applied_health/?q=node/12

9. Systematic Reviews vs. Literature Reviews

10. Systematic Reviews: When Are They Useful? Too much information in disparate sources Too little information, hard to find all of the research Help achieve consensus on debatable issues Plan for new research Provide teaching/learning materials

12. Database Search

13. Review Results

14. Critical Appraisal

16. Our Research Questions 1. What is the level of satisfaction of patrons who utilize digital reference? 2. What are the measures researchers use to quantify user satisfaction and how do they compare?

17. Database Search LISTA (EBSCO platform): 123 articles retrieved LISA (CSA platform): 209 articles retrieved ERIC: no unique studies retrieved

18. Working with Results 279 Results after de-duplication Only format retrieved: journal articles Abstracts were reviewed applying inclusion and exclusion criteria

19. Inclusion/Exclusion Criteria Should be pre-determined at the beginning of the study Minimizes bias Allows outside verification of why studies were included/excluded

20. Sample Inclusion/Exclusion Criteria Inclusion Peer reviewed journals Articles comparing e-reference with face-to-face reference Articles on academic, public and special libraries Articles on e-mail, IM, and “chat” reference Exclusion Articles describing how to implement digital reference programs Articles discussing quantitative or demographic data only Reviews, editorials and commentary Non-English articles

21. Working with Results 93 articles were selected based on inclusion/exclusion criteria Full text was obtained and read by both authors independently to determine if at least one variable pertaining to user satisfaction was present; then the results were compared

22. Results of Full Text Review

23. Critical Appraisal Tools QUOROM (The Lancet, 1999, vol. 354, 1896-1900) Downs-Black scale (“Checklist for study quality”) CriSTAL (Critical Skills Training in Appraisal for Librarians (Andrew Booth)

24. Glynn’s Critical Appraisal Tool Population Data collection Study design Results

25. Critical Appraisal Process 24 articles were subjected to critical appraisal Each question from Glynn’s tool was answered (either yes, no, unclear or N/A) and the results were calculated 12 research papers selected and subjected to the systematic review

26. Analysis (Findings of Review) Settings and general characteristics: Multiple instruments in a single article 9 unique journals US based Methods and timing of data collection 7 paper surveys 3 pop up surveys 3 transcript analysis

27. Similar Variables in Surveys “Willingness to return” 11 surveys of all instruments (Nilsen) Staff person vs service “Have you used it before?” Ranged from 30%-69% (email) Positivity of experience 7 point, 4 point, 3 point scales 65% - 98.2% (email, small group) 14-417 respondents Staff quality 7 point, 4 point, 3 point scales 68% - 92.8% (14 respondents)

28. Analysis Other questions in obtrusive studies “Were you satisfied?” “Would you recommend to a colleague?” each only asked in only 1 of the studies

29. Analysis: Reason for variation: Nature of questions asked is contingent on context in which satisfaction was measured [correlate to guidelines, librarian behaviors, reference interviews, etc.]

30. Unobtrusive studies: Transcript Analysis 2 Basic Methods: Transcript analysis by person asking the question (proxy patron) (Schachaf and Horowitz, 2008, Sugimoto, 2008). 75% “complete”, 68% “mostly incomplete” Transcripts independently assessed for quality and coded (Marsteller and Mizzy, 2003, Schachaf and Horowitz, 2008) 3 point scale, “+ or –” scale 2.24 out of 3 (level of quality); 5 negatives/200 transactions Research question: Efficacy of third party assessors vs. user surveys

31. Lessons Learned Lessons about user satisfaction with electronic reference: Overall pattern of users being satisfied, regardless of methodology or questions asked Measurement of user satisfaction is contingent upon context Researchers most often try to connect user satisfaction to another variable, satisfaction the sole focus of only one article

32. Lessons Learned Lessons about library research Extensive amount of qualitative research makes performing systematic reviews challenging Inconsistency of methodologies used in original research makes the systematic review challenging, meta-analysis is more often than not impossible Common pitfalls in LIS research that affect the quality of the published article

33. Lessons Learned Benefits of undertaking a systematic review: Sharpens literature searching skills: benefits for both librarians and their patrons who need this kind of research Researcher gains the ability to critically appraise research The practice of librarianship is strengthened by basing decisions on a methodological assessment of evidence

34. Systematic Reviews and EBL:Impact on the Profession Formal gathering and synthesis of evidence may: Affirm our intuitive sense about the patterns in current research Refine, clarify and enhance a more robust understanding of a current problem in librarianship May, on occasion, provide surprising results!

35. Questions?http://www.slideshare.net/librarian68 Elaine M. Lasda Bergman ebergman@uamail.albany.edu Irina I. Holden iholden@uamail.albany.edu