-

1. Examining the Potential Role of Text-
Based Recommendations in Academic
Information Seeking
1
Keenious, Norway
2
Uppsala University Library, Sweden
Technological advancements allow for new
opportunities in the way research is
conducted. One such example is the
application of artificial intelligence systems to
explore academic literature.
This study aims to evaluate the effectiveness
and user experience of Keenious, a text-based
article recommendation tool, in comparison
to Web of Science, a conventional keyword-
based information retrieval system. The user
study involved 16 librarians from Uppsala
University Library.
The results revealed both similarities and
differences between the two approaches to
information retrieval. The librarians perceived
Keenious as a valuable addition to their
information-seeking process, however, they
also expressed the need for increased
transparency regarding the ranking of results.
Abstract
Poster Presentation
RLUK23 Virtual Conference 22-24 March 2023
Barkelind, M., Hartman, A., Jansson, C., Kotka, A., Aure, J. B. & Johansen, J. S.
1 1 1 1 2 2

2. What We Did
To compare WoS and Keenious, we used a test design inspired by a
similar study (3). 16 librarians from UU were assigned the task to identify
relevant research articles related to a given abstract (see the three
alternative abstracts on the next page).
Participants completed two separate discovery sessions based on the
same assigned abstract; one session with WoS and one with Keenious.
After each discovery session, the librarians reported the articles they
identified as relevant during the session and filled out a questionnaire
about their experience using the respective tool.
Background and Motivation
Academic libraries are likely to adopt varied applications of artificial
intelligence (AI) in the future, and knowledge discovery could be one of
the most profound areas of use (1). To ensure that the library and its
users' needs are met, it has been argued that librarians should play an
active role in the development of AI systems by collaborating closely
with technology actors (2).
With this approach in mind, Uppsala University Library (UU) and the
startup company Keenious investigated the potential role of AI-based
recommendations in the information-seeking process by comparing
the effectiveness and user experience of Web of Science (WoS) and
Keenious.
We were curious as to how librarians would perceive the relevance of
the information provided by keyword- versus a text-based research
tool. What are the benefits and disadvantages? How could AI-based
recommendations fit with traditional information retrieval?

3. Common
Introduction
Keenious
Group 1
Web of Science
Group 2
Keenious
Group 2
Web of Science
Group 1
What We Did
Abstracts
Test Design
Nippert-Eng, C. (1996). Calendars
and keys: The classification of
“home” and “work”. Sociological
forum, 11(3), 563-582.
https://doi.org/10.1007/BF02408393
.
Goto, Y., Hayasaka, S., & Nakamura,
Y. (2012). Bathing in hot water,
bathing in Japanese style hot
spring and drinking green tea may
contribute to the good health
status of Japanese. The Journal of
the Japanese Society of
Balneology, Climatology and
Physical Medicine, 75(4), 256-267.
https://doi.org/10.11390/onki.75.256
Murakami, E.T., Tornsen, M, &
Pollock, K. E. (2014). Expectations
for the Preparation of School
Principals in Three Jurisdictions:
Sweden, Ontario and Texas.
Canadian and International
Education, 43(1), Article 7.
Common
Discussion

4. What We Learned
Polarised Perception of Keyword-Based Searching
When asked about the ease of finding relevant articles for the topic,
the results of the questionnaire showed a close similarity in the
average scores for both tools, with WoS receiving slightly more
positive ratings (2.27) than Keenious (2.07).
The figure below showcases the responses to be more diverse for
WoS compared to Keenious. More participants reported "To a very
large extent" when using WoS, however, there was also a higher
portion of participants who reported "To a small extent" compared
to Keenious’ ratings. This suggests a more polarized viewpoint about
the ease of finding relevant articles when using WoS.
0
Not at all
1
To a little
extent
-
I don't
know
2
To some
extent
3
To a large
extent
4
To a very
large extent
"It was easy to find relevant articles for the topic"

5. Very Little Overlap Between Results
We also analyzed the number of articles the librarians identified as
relevant when using the respective tools. The figure below shows
that the total number of articles identified was similar across the
two tools*. Interestingly, only 4 (4.3%) articles appeared in both lists
- meaning that different articles were considered relevant
dependent on the tool that was used.
The difference in participants' article selection with WoS and
Keenious may have several explanations. For example, the initial
dataset is not the same. Secondly, the participants did not have a
genuine need for information, which in turn may have influenced the
selection. However, the findings raise the question as to whether
there are qualitative differences in the kinds of articles being
discovered through keyword-based searches and text-based
recommendations.
4 of 94 (4.3%) 46 (48.9%)
48 (51.1%)
A Venn diagram of the number of relevant articles identified with each tool.
The intersection illustrates the 4 articles that were identified with both tools.
*Two outliers were excluded from the analysis.
What We Learned

6. Positive Attitudes towards Combining The Tools
The absence of result overlap between the tools does not signal any
definitive advantage or disadvantage in itself. Therefore the ratings
on the question "Do you see advantages in combining the two
tools?" revealed an intriguing insight.
As the pie chart below illustrates, 5 of the participants answered "To
a very high extent", 4 answered "To a great extent", and 6 answered
"To a certain extent", indicating a generally positive view towards
combining the two tools. The positive attitude of librarians towards
combining the tools suggests that the dissimilarities between the
tools may prove beneficial to the overall information-gathering
process.
What We Learned
4 - To a very large extent
31%
I don't know
6%
2 - To some extent
38%
3 - To a large extent
25%
"Do you see advantages in combining the two
tools?"
Note. "Not at all" = 0%, "To a litte extent" = 0%

7. Discussion
Based on this study, it appears that the participating librarians perceived AI-
based recommendation tools such as Keenious as a useful complement to
traditional search tools such as Web of Science, rather than a replacement.
The main findings in our case study were that articles identified as relevant
during the discovery sessions varied greatly between the two tools, with only
4 of 94 articles appearing in both tools’ lists. This suggests that the two tools
offer different kinds of relevant articles. That finding was further supported by
the participant's positive attitudes toward combining the tools as part of the
information-seeking process.
The main objection to Keenious' recommendations was the lack of a clear
description of how searching and ranking work - librarians wanted more
transparency. The learnings from this case study led to a follow-up workshop
where Keenious and 21 librarians from UU sat together to further understand
how AI-based discovery tools can improve system transparency.
Curious to know more about the study?
Contact: jesper@keenious.com
References
1. Cox, A. (2023). How artificial intelligence might change academic library work:
Applying the competencies literature and the theory of the professions. Journal of
the Association for Information Science and Technology, 74(3), 367-380.
https://doi.org/10.1002/asi.24635
2. Gasparini, A., & Kautonen, H. (2022). Understanding artificial intelligence in
research libraries–extensive literature review. LIBER Quarterly: The Journal of the
Association of European Research Libraries, 32(1). https://doi.org/10.53377/lq.10934
3. Johansen, J.S., & Borlund, P. (2022). Academic information searching and
learning by use of Keenious and Google Scholar: a pilot study. In Proceedings of
CoLIS, the 11th. International Conference on Conceptions of Library and
Information Science, Oslo, Norway, May 29 - June 1, 2022. Information Research,
27(Special issue), paper colis2231. https://doi.org/10.47989/colis2231