Presentation for Museums and the Web 2018 (Vancouver) April 19, 2018 In a world in which a family historian can type her grandfather's name into Ancestry.com to start building a family tree, and a journalist can Google to download public domain images, where do the collections searches, online tools, and APIs that museums and archives provide fit in? This paper outlines strategies for better serving people who are looking for the knowledge and expertise within your collections and staff. At the National Archives and Records Administration of the United States, we undertook a significant user experience (UX) research project to better understand the online experiences of professional researchers, family historians, and history enthusiasts. Research methods included audits of existing user data (e.g., Google Analytics, survey data) as well as new user interviews, usability testing, a survey, and a landscape analysis. Key findings include the fact that researchers struggle to complete their tasks using existing online tools; people researching family history are particularly unsatisfied and in need of better support; and all audiences require just-in-time help and appropriate orientation to archival research. A major challenge highlighted by this research is how to meet user expectations for item-level records while providing access to digitized records at massive scale. Read full paper: https://mw18.mwconf.org/paper/serving-researchers-in-a-self-service-world/

1. Serving researchers
in a self-service world
Dana Allen-Greil
@danamuses
Museums and the Web - April 29, 2018

2. In the next 15 minutes
1. How and why the National Archives
is investing in UX research
2. Key findings that might apply to
similar organizations
3. Areas for further research and
collaboration
Paper: http://bit.ly/SelfServeResearch

3. About the National Archives
The National Archives (NARA) is
America’s record keeper. We hold
hundreds of miles of film and tape,
14 million still photographs, and
billions of letters, drawings, maps,
treaties, posters, and other
documents that tell the stories of
America’s history as a nation.
Workers unloading Veteran’s Bureau Records (June 1936)

4. The Problem.
● Digital products
developed over time
without a strong (or
unifying) user-centered
framework
● Low customer
satisfaction overall
● Lots of assumptions
about researchers
A frustrated driver sits through a traffic jam in Herald Square (May 1973)

5. Satisfaction scores by audience
Average
Public service gap
D

6. Most users are unable to accomplish their task
How might we we help more people
accomplish their tasks?
…(Yes and) how might we help people
better define research tasks so they can
be accomplished?
Did you accomplish what
you wanted to
do today on the site?

7. Core products
for researchers

8. Archives.gov

9. Catalog

10. Consistently declining customer satisfaction
Scores for the
Catalog fell 16
points in 7 months.
What happened?

11. 11
History Hub
Learn more about how
you can get involved:
bit.ly/joinhistoryhub

12. Why is this issue urgent now?
Our strategic plan dictates:
★ By FY 2024, NARA will digitize 500 million pages of records and make them
available online to the public through the National Archives Catalog.
★ By FY 2025, NARA will provide digital, next-generation finding aids to 95
percent of the holdings described in the National Archives Catalog.
★ By FY 2025, NARA will have 1 million records enhanced by citizen contributions
to the National Archives Catalog.

13. Why invest in UX?
❏ In the past, NARA designed and developed products without a user-centered
framework. These legacy products are now very difficult to change due to rigid
procurement and development processes.
❏ We must figure out not only how to provide access to a massive number of
archival resources but also how to do it in a way that is usable and useful at
scale.
❏ Our research is designed to help us use our limited resources most efficiently
and effectively to support successful user-centered products.

14. UX add to strategic plan!
Added “Build capabilities in design and user experience, to better understand user
needs.”

15. Research Goals
To better understand today's online marketplace for researchers and how we can
best serve their needs.
1. Assess the current level of audience engagement
2. Understand NARA’s high-value audiences
3. Identify new, aspirational audiences
4. Review peer organizations for best practices, features, and trends
5. Recommend ways to improve user experience on current digital platforms
6. Identify new opportunities to engage audiences

16. Research Methods
1
Usability testing/ observation
9 remote users were given task-based
questions to carry out on Archives.gov,
Catalog, and History Hub, and asked to
explain their choices
3
Comparative analysis
Reviewed 12 websites to compare
features, functionality, and content to
understand the wider online research
services landscape
5
Review of existing data
-Customer satisfaction survey
- Google Analytics data
- Search keyword metrics
- Social media analytics
-Textual analysis of Contact Us emails
2
User interviews
19 interviews (30-60 min) with
genealogists, in-person researchers,
online researchers, electronic records
experts, and citizen archivists
4
New survey of researchers
1,847 responses

17. Personas
archives.gov/digitalstrategy/personas

18. Audiences vs. Tasks
“Online researcher” is
a diverse category.
We should optimize
digital services to
meet the needs of
specific user tasks
rather than broad
audience categories.

19. Online Researchers Expect
Online Records First
● Only 25% of online researchers have
ever visited a NARA research facility.
● 86% said online research (as opposed
to in-person research) is important or
very important.
● Most researchers self-identified as
experienced/very experienced.
World War I poster (1943-1945)

21. Not just a “learning curve”
Returning visitors are no more satisfied than new
visitors.
➔ Suggests that no new knowledge is gained in
the first visit that makes subsequent visits easier
to execute.
➔ Solution can’t just be about “better orientation”
or more research guides (although just-in-time
support would help).
➔ Need to fix fundamental UX problems so our
digital services are usable and useful.

22. Thanks, Google.
Keyword topics and names of people provide relevant results.

24. People researching family history
are particularly unsatisfied
● At least 1/3 of our online researcher audience
● Only 24% of this audience was able to complete their
task
● Many visit Ancestry.com or similar sites first, where they
are accustomed to user-friendly features like person
name search and highlighted results
○ Popularity of DNA heritage tests is driving growth of this audience
● Most do not identify with the term “genealogist” or even

25. Orientation
Even experienced researchers need help orienting to your
collection:
● scope and size
○ What is an archive?
○ What’s in YOUR archive?
● how is it structured
● what is digitized and what isn’t
Recommended reading: Whitelaw, M. “Generous Interfaces for Digital Cultural Collections”

26. Wizards

27. Just-in-time help
Guidance (in a PPT file!) on a different
website is not helpful in the middle of a
search
How can we help people when they get
stuck?
Image: NYMag.com

28. How might AI help us support research?
1. Item-level expectations - Our systems and processes are structured around
descriptions of records groups and series. While we have millions of digitized
items, we don’t have good metadata to support their discoverability (e.g., by topic,
by person, by what is viewable in an image). How might AI (artificial intelligence)
help bridge this gap?
2. Finding needles in the haystack - How might machine learning make search
results more relevant and get smarter over time based on use? (This is especially
interesting for thinking about the tsunami of email records coming soon.)
3. Providing in-line support - Can chatbots provide useful customer service for
common researcher requests? How can we re-purpose knowledgebases like
History Hub to scale up personalized support?

29. Next steps
Front end
1. Consolidate products (distinction
between websites unclear to users)
2. Show digitized items first by default
(DUH!)
3. Build streamlined interfaces to guide
users in specific research tasks
4. Orient audiences who are
underprepared for archival research,
including a new visual overview of
holdings
5. Offer in-line help and support
Back end
1. Transform our procurement and
development to more agile
methodologies with UX work baked in
to the process
2. Leverage AI and machine learning to
support discoverability and tailored
support
3. Expand and improve API (including
faster performance, better image
delivery, and more coherent data
structure)

30. How can we better leverage our API?
To kick the tires on our API and uncover limitations (performance, data structure,
documentation), we have developed 6 prototypes:
1. Treemap/overview of holdings, a “generous interface” that communicates scope
and breadth
2. Facebook Messenger chatbot which encourages tagging of records
3. "When Am I?" game for engaging with records by time period (and adding tags)
4. Voice-activated Alexa skill which enables exploration of recipes
5. Drupal image selection and cropping tool (open source module)
6. Drupal CAPTCHA module that facilitates tagging records (open source module)

32. Facebook Messenger
Chatbot
How might we massively scale up
crowdsourcing of metadata?
(Our API is read/write to include
citizen-contributed tags,
transcriptions, and comments.)

37. Let’s talk!
Dana Allen-Greil
@danamuses
dana.allen-greil@nara.gov