This is a March 12, 2008 PowerPoint presentation I did at UCLA’s Graduate School of Education and Information Science on the results of my dissertation “Online Virtual Chat Library Reference Service: A Quantitative and Qualitative Analysis.” I was invited by Dr. John V. Richardson, my mentor, to speak in one of his graduate library classes. My Doctor of Education in Educational Technology is from Pepperdine University, class of 2007.

1. Online Virtual Chat LibraryOnline Virtual Chat Library
Reference Service:Reference Service:
A Quantitative and QualitativeA Quantitative and Qualitative
AnalysisAnalysis
Dave Harmeyer, M.L.S., Ed.D.Dave Harmeyer, M.L.S., Ed.D.
Director of Research & DevelopmentDirector of Research & Development
University LibrariesUniversity Libraries
Azusa Pacific UniversityAzusa Pacific University
March 12, 2008March 12, 2008
UCLA, GSE&ISUCLA, GSE&IS

2. OutlineOutline
1.1. Purpose of the StudyPurpose of the Study
2.2. Research QuestionsResearch Questions
3.3. MethodologyMethodology
4.4. Variables (I.V., D.V.)Variables (I.V., D.V.)
5.5. Significant FindingsSignificant Findings
6.6. ConclusionsConclusions
7.7. Questions & AnswersQuestions & Answers

3. Purpose of the StudyPurpose of the Study
 Virtual chat reference augments face-to-Virtual chat reference augments face-to-
face reference interviewface reference interview
 Library reference literature lacks research-Library reference literature lacks research-
based findings to back up recommendedbased findings to back up recommended
practicespractices
 This study fills the void with a theoreticalThis study fills the void with a theoretical
conceptual model based on an empiricalconceptual model based on an empirical
study of chat reference transactionsstudy of chat reference transactions

4. Research QuestionsResearch Questions
1. What measurable indicators are found for virtual1. What measurable indicators are found for virtual
chat reference transactions, looking exclusivelychat reference transactions, looking exclusively
at data created from the chat referenceat data created from the chat reference
transcripts?transcripts?
2. Do published reference interview guidelines from2. Do published reference interview guidelines from
RUSA, a set of other strategies and the nature ofRUSA, a set of other strategies and the nature of
the query contribute to an accurate answer?the query contribute to an accurate answer?
3. What conceptual model of best practices can be3. What conceptual model of best practices can be
suggested by an analysis of the data?suggested by an analysis of the data?

5. MethodologyMethodology
 Two-and-a-half years of archived academic library chatTwo-and-a-half years of archived academic library chat
transcripts using Krippendorff’s (2004) content analysistranscripts using Krippendorff’s (2004) content analysis
 333 random transcripts from 2,500333 random transcripts from 2,500
 Analyzing 16 independent variables and their relationshipAnalyzing 16 independent variables and their relationship
with one dependent variable of an accurate referencewith one dependent variable of an accurate reference
answeranswer
 Pearson correlations and ANOVA variance testsPearson correlations and ANOVA variance tests
 120 virtual librarians at 43 American institutions120 virtual librarians at 43 American institutions
 320 remote patrons accessing the service through one320 remote patrons accessing the service through one
Southern California undergraduate/masters universitySouthern California undergraduate/masters university

6. Methodology (cont.)Methodology (cont.)

7. VariablesVariables
 Research Question 1 answered:Research Question 1 answered:
 16 independent variables16 independent variables
 1 dependent variable - question accuracy1 dependent variable - question accuracy
 Influence on question accuracyInfluence on question accuracy
 Observed in content analysis of chatObserved in content analysis of chat
transcriptstranscripts
 Derived from RUSA guidelinesDerived from RUSA guidelines
 Derived from literature reviewDerived from literature review

8. Quantitative IVsQuantitative IVs
1. Librarian’s initial contact time (1. Librarian’s initial contact time (hold timehold time, in, in
seconds)seconds)
2. Total time of transaction (2. Total time of transaction (service timeservice time, in seconds), in seconds)
3. Longest time gap by librarian (in seconds)3. Longest time gap by librarian (in seconds)
4. Number of URLs co-browsed with the patron4. Number of URLs co-browsed with the patron
5. Keystrokes by librarian5. Keystrokes by librarian
6. Keystrokes by patron6. Keystrokes by patron
7. Keystrokes by both7. Keystrokes by both

9. Qualitative IVsQualitative IVs
1. The question’s difficulty (seven-point scale)1. The question’s difficulty (seven-point scale)
2. Response to a patron’s “are you there” statements (scored2. Response to a patron’s “are you there” statements (scored
as present, not present, not applicable or ambiguous whenas present, not present, not applicable or ambiguous when
coders disagreed)coders disagreed)
3. Librarian’s friendliness3. Librarian’s friendliness
4. Lack of jargon4. Lack of jargon
5. Use of open-ended questions5. Use of open-ended questions
6. Use of closed and/or clarifying questions6. Use of closed and/or clarifying questions
7. Librarian maintains objectivity7. Librarian maintains objectivity
8. Asking if the question was answered completely8. Asking if the question was answered completely
9. The type of question (seven categories: ready reference,9. The type of question (seven categories: ready reference,
research question, library technology, request for materials,research question, library technology, request for materials,
bibliographic verification, other and ambiguous forbibliographic verification, other and ambiguous for
disagreements among coders)disagreements among coders)

10. Dependent VariableDependent Variable
Coders Qualitative JudgmentsCoders Qualitative Judgments Service QualityService Quality
88 Librarian gave (or referred) patron to a singleLibrarian gave (or referred) patron to a single
source with an accurate answersource with an accurate answer ExcellentExcellent
77 Librarian gave (or referred) patron to more thanLibrarian gave (or referred) patron to more than
one source, one of which provided an accurateone source, one of which provided an accurate
answeranswer Very goodVery good
66 Librarian gave (or referred) patron to a singleLibrarian gave (or referred) patron to a single
source which does not lead directly to ansource which does not lead directly to an
accurate answer but did serve as a preliminaryaccurate answer but did serve as a preliminary
sourcesource GoodGood
55 Librarian gave (or referred) patron to moreLibrarian gave (or referred) patron to more
than one source, none of which leads directlythan one source, none of which leads directly
to an accurate answer but one which served asto an accurate answer but one which served as
a preliminary sourcea preliminary source SatisfactorySatisfactory

11. Dependent Variable (cont.)Dependent Variable (cont.)
Coders Qualitative JudgmentsCoders Qualitative Judgments Service QualityService Quality
44 No direct accurate answer given,No direct accurate answer given,
referred to another person or institutionreferred to another person or institution Fair / poorFair / poor
33 No accurate answer (or referral) givenNo accurate answer (or referral) given
(e.g., “I don’t know”)(e.g., “I don’t know”) FailureFailure
22 Librarian gave (or referred) patron to aLibrarian gave (or referred) patron to a
single source which did not answer thesingle source which did not answer the
questionquestion UnsatisfactoryUnsatisfactory
11 Librarian gave (or referred) patron toLibrarian gave (or referred) patron to
more than one source, none of whichmore than one source, none of which
answered the questionanswered the question Most unsatisfactoryMost unsatisfactory
(Richardson and Reyes, 1995)(Richardson and Reyes, 1995)

12. Significant FindingsSignificant Findings
SummarySummary
 Research Question 2 answered: yesResearch Question 2 answered: yes
 30 significant relationships (p < .05)30 significant relationships (p < .05)
 From 9 of 16 variablesFrom 9 of 16 variables
 5 found in RUSA guidelines5 found in RUSA guidelines
 4 found in other strategies or nature of4 found in other strategies or nature of
online chatonline chat

13. Significant FindingsSignificant Findings
Answer AccuracyAnswer Accuracy
Answer Accuracy as Judged by Coders (N=331)Answer Accuracy as Judged by Coders (N=331)
CriteriaCriteria Point FrequencyPoint Frequency %% Cum. %Cum. %
Accurate Answer (single source)Accurate Answer (single source)
ExcellentExcellent 8.08.0 8888 26.626.6 26.6 (1/4)26.6 (1/4)
Accurate Answer (mult. sources)Accurate Answer (mult. sources) 7.57.5 1818 5.45.4 32.032.0
Very goodVery good 7.07.0 6464 19.319.3 51.3 (1/2)51.3 (1/2)
Preliminary Source (single source) 6.5Preliminary Source (single source) 6.5 99 2.72.7 54.054.0
GoodGood 6.06.0 3333 10.010.0 64.0 (2/3)64.0 (2/3)
Preliminary Source (mult. sources)Preliminary Source (mult. sources) 5.55.5 77 2.12.1 66.166.1
SatisfactorySatisfactory 5.05.0 3535 10.610.6 76.7 (3/4)76.7 (3/4)
No Accurate Answer, referredNo Accurate Answer, referred 4.54.5 1111 3.33.3 80.080.0
Fair / poorFair / poor 4.04.0 5757 17.217.2 97.297.2
““I don’t know,” no referralI don’t know,” no referral 3.53.5 22 0.60.6 97.897.8
FailureFailure 3.03.0 33 0.90.9 98.798.7
Not Accurate (single source)Not Accurate (single source) 2.52.5 22 0.60.6 99.399.3
UnsatisfactoryUnsatisfactory 2.02.0 22 0.60.6 99.999.9
Not Accurate (multiple sources)Not Accurate (multiple sources) 1.51.5 00 0.00.0 99.999.9
Most unsatisfactoryMost unsatisfactory 1.01.0 00 0.00.0 99.999.9

14. Significant FindingsSignificant Findings
Best PracticesBest Practices
Research Question 3 answered: yesResearch Question 3 answered: yes
A Conceptual Model for Reference Chat AccuracyA Conceptual Model for Reference Chat Accuracy
minor plus est (less is more)minor plus est (less is more)
1. Keep time gaps between sending responses to1. Keep time gaps between sending responses to
patrons to no more than one-and-a-half minutespatrons to no more than one-and-a-half minutes
2. Maintain a total chat transaction time of eight2. Maintain a total chat transaction time of eight
minutes or lessminutes or less
3. Keep total keystrokes per transaction to within six3. Keep total keystrokes per transaction to within six
and-a-half lines of text (or 480 characters).and-a-half lines of text (or 480 characters).
4. Expect to type twice as many characters as the4. Expect to type twice as many characters as the
patronpatron

15. Significant FindingsSignificant Findings
Best Practices (cont.)Best Practices (cont.)
5. Be careful about beginning the question5. Be careful about beginning the question
negotiation segment of the reference interviewnegotiation segment of the reference interview
with an open question unless the nature of thewith an open question unless the nature of the
patron’s question explicitly calls for one.patron’s question explicitly calls for one.
6. Ask closed or clarifying questions when6. Ask closed or clarifying questions when
appropriateappropriate
7. At the end of the reference transaction, ask “Does7. At the end of the reference transaction, ask “Does
this completely answer your question?”this completely answer your question?”
8. Even moderately difficult questions decrease8. Even moderately difficult questions decrease
answer accuracy and not just the medium to highanswer accuracy and not just the medium to high
difficult questionsdifficult questions

16. Significant FindingsSignificant Findings
1. Gaps1. Gaps
 Keep time gaps between sendingKeep time gaps between sending
responses to patrons to not much more thanresponses to patrons to not much more than
one-and-a-half minutesone-and-a-half minutes
 Reinforces RUSA’sReinforces RUSA’s interestinterest guideline (2.6),guideline (2.6),
time away from the patron short, maintaintime away from the patron short, maintain
“word contact” (RUSA, June 2004)“word contact” (RUSA, June 2004)
 Anything nearing two minutes or higher isAnything nearing two minutes or higher is
likely to decrease answer accuracylikely to decrease answer accuracy

17. Significant FindingsSignificant Findings
1. Gaps1. Gaps
Longest Librarian GapLongest Librarian Gap
Quartiles (min.) Acc. Mean Sig. of Diff. (p)_____Quartiles (min.) Acc. Mean Sig. of Diff. (p)_____
1st 0 --1st 0 -- 1.851.85 6.686.68
2nd2nd 1.87 – 2.831.87 – 2.83 5.975.97 .016.016 (1st & 2nd)(1st & 2nd)
3rd 2.85 – 4.453rd 2.85 – 4.45 6.316.31 .403 (1st & 3rd, no sig.).403 (1st & 3rd, no sig.)
4th4th 4.47 --4.47 -- 6.036.03 .036.036 (1st & 4th)(1st & 4th)
Diff=.71
Diff=.65
Diff=.71

18. Significant Findings:Significant Findings:
2. Service Time2. Service Time
 Maintain a total chat transaction time ofMaintain a total chat transaction time of
eight minutes or lesseight minutes or less
 Average = 16.0 minutes (n = 331)Average = 16.0 minutes (n = 331)
 7 minutes more than Richardson’s (2002)7 minutes more than Richardson’s (2002)
8.9 minutes (n = 20,000)8.9 minutes (n = 20,000)
 However, similar to six f2f studies with meanHowever, similar to six f2f studies with mean
service time ranging from 10 to 20service time ranging from 10 to 20

19. Significant FindingsSignificant Findings
2. Service Time2. Service Time
Service Time of TransactionsService Time of Transactions
Quartiles (min.) Accuracy Mean Sig. of Diff. (p)Quartiles (min.) Accuracy Mean Sig. of Diff. (p)
1st 0 – 8.31st 0 – 8.3 6.826.82
2nd 8.32 – 13.082nd 8.32 – 13.08 6.026.02 .005 (1st & 2nd).005 (1st & 2nd)
3rd 13.1 – 20.753rd 13.1 – 20.75 6.046.04 .007 (1st & 3rd).007 (1st & 3rd)
4th 20.77 --4th 20.77 -- 6.126.12 .020 (1st & 4th).020 (1st & 4th)
Diff=.78
Diff=.70
Diff=.80

20. Significant Findings:Significant Findings:
3. Keystrokes3. Keystrokes
 Keep total keystrokes per transactionKeep total keystrokes per transaction
to within six and-a-half lines of text (orto within six and-a-half lines of text (or
480 characters)480 characters)
 Application to virtual software vendorsApplication to virtual software vendors
(add a timer)(add a timer)
 Anything over 15 lines of text willAnything over 15 lines of text will
decrease accuracydecrease accuracy

21. Significant FindingsSignificant Findings
3. Keystrokes3. Keystrokes
KeystrokesKeystrokes
Keystroke Quartiles Accuracy Mean Sig. of Diff. (p)Keystroke Quartiles Accuracy Mean Sig. of Diff. (p)
LibrarianLibrarian
1st 0 – 480 (6.5 lines)*1st 0 – 480 (6.5 lines)* 6.586.58
4th 1128 (15 lines) --4th 1128 (15 lines) -- 5.935.93 .041 (1st & 4th).041 (1st & 4th)
PatronPatron
1st 0 – 188 (2.5 lines)1st 0 – 188 (2.5 lines) 6.656.65
4th 545 (7.5) --4th 545 (7.5) -- 5.965.96 .023 (1st & 4th).023 (1st & 4th)
Both Librarian & PatronBoth Librarian & Patron
1st 0 – 690 (9 lines)1st 0 – 690 (9 lines) 6.636.63
4th 1668 (22.5 lines) --4th 1668 (22.5 lines) -- 5.995.99 .041 (1st & 4th).041 (1st & 4th)
*measured at 74 keystrokes per line of text*measured at 74 keystrokes per line of text

22. Significant FindingsSignificant Findings
4. Twice the Typing4. Twice the Typing
 Expect to type twice as many characters asExpect to type twice as many characters as
the patronthe patron
 Appeared across all four quartile segmentsAppeared across all four quartile segments
between librarian and patron.between librarian and patron.

23. Significant FindingsSignificant Findings
5. Open-ended Questions5. Open-ended Questions
 Be careful about beginning the question
negotiation segment of the reference interview
with an open question unless the nature of the
patron’s question explicitly calls for one
Frequency of Open-ended Questions
CategoryCategory FrequencyFrequency PercentPercent
PresentPresent 112112 3333
Absent (but should) 75Absent (but should) 75 22.522.5
Not ApplicableNot Applicable 7676 22.822.8
AmbiguousAmbiguous 6767 20.120.1

24. Significant FindingsSignificant Findings
5. Open-ended Questions5. Open-ended Questions
Open-ended QuestionsOpen-ended Questions
Category Accuracy Mean Sig. of Diff. (p)Category Accuracy Mean Sig. of Diff. (p)
Not ApplicableNot Applicable 6.726.72
PresentPresent 5.975.97 .008 (3 & 1).008 (3 & 1)

25. Significant Findings:Significant Findings:
6. Closed-ended Questions6. Closed-ended Questions
 Ask closed or clarifying questions when
appropriate
Frequency of Closed-ended and/or Clarifying QuestionsFrequency of Closed-ended and/or Clarifying Questions
CategoryCategory Frequency PercentFrequency Percent
PresentPresent 183183 55.055.0
AbsentAbsent 4848 14.414.4
Not ApplicableNot Applicable 5656 16.816.8
AmbiguousAmbiguous 4444 13.213.2

26. Significant FindingsSignificant Findings
6. Closed-ended Questions6. Closed-ended Questions
Closed and/or Clarifying QuestionsClosed and/or Clarifying Questions
CategoryCategory Accuracy Mean Sig. Of Diff. (p)Accuracy Mean Sig. Of Diff. (p)
3. Not Applicable3. Not Applicable 6.696.69
2. Absent2. Absent 5.915.91 .065 (3 & 2, not sig.).065 (3 & 2, not sig.)
Ambiguous filteredAmbiguous filtered
3. Not Applicable3. Not Applicable 6.696.69
2. Absent2. Absent 5.915.91 .040.040 (3 & 2)(3 & 2)

27. Significant FindingsSignificant Findings
7. Follow-up Question7. Follow-up Question
 At the end of the reference transaction, ask “Does
this completely answer your question?”
Frequency of the Librarian Asking If the QuestionFrequency of the Librarian Asking If the Question
Had Been Answered CompletelyHad Been Answered Completely
CategoryCategory FrequencyFrequency PercentPercent
PresentPresent 125125 37.537.5
AbsentAbsent 4242 12.612.6
Not ApplicableNot Applicable 108108 32.432.4
AmbiguousAmbiguous 5656 16.816.8

28. Significant FindingsSignificant Findings
8. Question Difficulty8. Question Difficulty
Question DifficultyQuestion Difficulty
Criteria Point Frequency %Criteria Point Frequency % Cum. %Cum. %
LowLow 1.01.0 5959 17.717.7 17.817.8
1.51.5 4141 12.312.3 30.230.2
2.02.0 6666 19.819.8 50.2 1/250.2 1/2
2.52.5 5555 16.516.5 66.866.8
3.03.0 2727 8.18.1 74.9 3/474.9 3/4
MediumMedium3.53.5 2626 7.87.8 82.882.8
4.04.0 1717 5.15.1 87.987.9
4.54.5 1818 5.45.4 93.493.4
5.05.0 66 1.81.8 95.295.2
5.55.5 44 1.21.2 96.496.4
6.06.0 77 6.6%6.6% 2.12.1 98.598.5
6.56.5 33 0.90.9 99.499.4
HighHigh 7.07.0 22 0.60.6 100.0100.0

29. Significant FindingsSignificant Findings
8. Question Difficulty8. Question Difficulty
Question Difficulty and Accuracy (reporting only significance)Question Difficulty and Accuracy (reporting only significance)
CriteriaCriteria Points Accuracy Mean Sig. of Diff. (p)Points Accuracy Mean Sig. of Diff. (p)
LowLow 1.01.0 7.247.24
2.02.0 6.366.36 .035 (1.0 & 2.0).035 (1.0 & 2.0)
2.52.5 5.945.94 .000 (1.0 & 2.5).000 (1.0 & 2.5)
MediumMedium 3.53.5 5.425.42 .000 (1 & 3.5).000 (1 & 3.5)
4.04.0 5.535.53 .001 (1.0 & 4.0).001 (1.0 & 4.0)
4.54.5 5.225.22 .000 (1.0 & 4.5).000 (1.0 & 4.5)
5.05.0 4.924.92 .009 (1.0 & 5.0).009 (1.0 & 5.0)
5.55.5 4.04.0 .001 (1.0 & 5.5).001 (1.0 & 5.5)
HighHigh 6.06.0 4.834.83 .006 (1.0 & 6.0).006 (1.0 & 6.0)
LowLow 1.51.5 6.956.95
2.52.5 5.945.94 .041 (1.5 & 2.5).041 (1.5 & 2.5)
MediumMedium 3.53.5 5.425.42 .002 (1.5 & 3.5).002 (1.5 & 3.5)
4.04.0 5.535.53 .032 (1.5 & 4.0).032 (1.5 & 4.0)
4.54.5 5.225.22 .001 (1.5 & 4.5).001 (1.5 & 4.5)
5.55.5 4.04.0 .005 (1.5 & 5.5).005 (1.5 & 5.5)
HighHigh 6.06.0 4.834.83 .038 (1.5 & 6.0).038 (1.5 & 6.0)

30. 5. Conclusions5. Conclusions
 Virtual reference lacks a statistically sound conceptualVirtual reference lacks a statistically sound conceptual
model to guide the library profession toward improving themodel to guide the library profession toward improving the
reference interview through empirical studies which informsreference interview through empirical studies which informs
best practices in professional training and assessment.best practices in professional training and assessment.
 This study addresses that knowledge void by its discoveryThis study addresses that knowledge void by its discovery
of several statistical relationships between nine behavioralof several statistical relationships between nine behavioral
factors and an acurate answer in the reference interview.factors and an acurate answer in the reference interview.
 It is hoped that the suggested eight-point rubric and otherIt is hoped that the suggested eight-point rubric and other
results of this project can be a catalyst for practicalresults of this project can be a catalyst for practical
application toward improving the practice of the globalapplication toward improving the practice of the global
community of professionals and stakeholders in the field ofcommunity of professionals and stakeholders in the field of
library and information studies.library and information studies.

31. 6. Questions6. Questions
&&
AnswersAnswers