Having issues with your intranet search experience? This session will give you the tools and techniques to address your search woes. You will walk away with a deeper understanding of the role your content plays in the process. You will learn how to use analytics to diagnose problems, establish potential solutions, and measure your progress. Finally, you will be exposed to some new technologies that can support you in delivering a search experience your users will love.

1. Strategies and tactics
to improve search
June 2019

2. Greg Nemeth
Technology and Digital Platforms Leader
Ernst & Young LLP
Cleveland, Ohio
5 April 2019 Strategies and tactics to improve searchPage 2
The views expressed by the presenter are their own and not necessarily those of Ernst & Young LLP or other members of the global EY organization.
These slides are for educational purposes only and are not intended to be relied upon as accounting, tax, or other professional advice. Please refer to your advisors for specific advice.

3. Organization of member firms
operating in more than 150 countries
• Assurance, tax, transaction and advisory services
• Wide, diverse business units and offerings
• Constantly growing and evolving
• Approximately 290,000 employees
About Ernst & Young LLP (EY)
13 June 2019 Strategies and tactics to improve searchPage 3

4. Goal of intranet search
Right insight Right person Right time
13 June 2019
Strategies and tactics to improve search
Page 4

5. • Bullets to be added during the session
• Bullets to be added during the session
What makes intranet search difficult?
13 June 2019 Strategies and tactics to improve searchPage 5

6. Model for improving search
13 June 2019 Strategies and tactics to improve searchPage 6
Content Identify,
tag, optimize and
index critical content
Measure
Measure and interpret
search activityUser
Understand
and clarify
user needs
Tune and verify
Tune for the optimal
experience
Improve
search

7. Session logistics
13 June 2019 Strategies and tactics to improve searchPage 7
Section one
Understand and clarify user needs
(20 minutes)
Section two
Identify, tag, optimize and index critical content
(20 minutes)
Break 10 minutes
Section three
Measure and interpret search activity
(20 minutes)
Section four
Tune and verify adjustments
(20 minutes)
Session wrap-up
Confirm key learnings
(10 minutes)

8. User: understand and
clarify user needs
13 June 2019 Strategies and tactics to improve searchPage 8

9. “The first step in exceeding
your customer’s
expectations is to know
those expectations.
Roy H. Williams
Author and marketing consultant
“
13 June 2019 Strategies and tactics to improve searchPage 9

10. How do you get to know your users?
User research
„ Surveys
„ Focus groups
„ Directed interviews
„ Ethnographic observation
„ Personas
„ User journeys
Search analytics
„ Top queries
„ Zero-hit queries
„ Abandoned queries
„ Search conversion/click-
through
„ Top search terms
„ Refinement rate
Critical to do both, as one provides context for the other.
13 June 2019 Strategies and tactics to improve searchPage 10

11. User research
13 June 2019 Strategies and tactics to improve searchPage 11
Surveys Quantitative results to
identify problem areas or
trends
„ Beginning of an effort to establish a baseline
„ After changes have been made to determine improvement
„ Most useful when brief with targeted questions
Focus groups Qualitative results to
generate an explanation or
prepare a for a quantitative
study
„ Can be used as the lead-up or follow-up to a survey
„ Frequently successful when exploring a direction and when
paired with a quantitative study
„ Need to be mindful of group think and user intent vs. actual
behavior
Directed
interviews
Qualitative results most
useful with audience
segmentation
„ Extremely helpful as a follow-up to a survey
„ Excellent for comparing and contrasting behavior
„ Provides the foundation for personas
Ethnographic
observation
Qualitative results with rich
contextual information
„ Helpful in the early stages of an effort to confirm the relevant
audience, tasks, goals and context
„ Great for gaining first-hand, spontaneous insight frequently
missed through other methods
„ Need to be mindful of various forms of bias (sample,
research, scope, etc.)
The method you use depends on the question you are trying to answer …
Method Output When to use

12. User research
13 June 2019 Strategies and tactics to improve searchPage 12
Personas Mix of qualitative and
quantitative data specific to
different user types
„ Helpful when making specific decisions about the search
experience
„ Requires proper segmentation and thorough data collection
to complete/update
User journeys Mix of qualitative and
quantitative data specific to
the context, task flows and
pain points for specific user
types
„ Extremely helpful in depicting the user’s relationship with an
existing experience
„ Helpful in depicting actual behavior vs. user intent to identify
specific pain points
„ Great at answering “what if” questions or formulating
strategic recommendations
The method you use depends on the question you are trying to answer …
Method Output When to use

13. Search analytics
13 June 2019 Strategies and tactics to improve searchPage 13
Top queries „ Determines what users are seeking
„ Should be continually monitored to provide clarity
„ Critical to examine patterns over time
Zero-hit queries „ Depicts areas of interest not being met
„ Should be continually monitored to confirm gaps are
being addressed
Abandoned queries „ May determine gaps in the search experience
„ Establish a baseline and actively monitor for trends
What are users
looking for?
Are they finding it?
Type Notables

14. Search analytics
13 June 2019 Strategies and tactics to improve searchPage 14
Search
conversion/click-
through
„ Approximation of search success
„ Can be used as a proxy for “the ground
truth”
Top search terms „ Confirms the specific areas of interest
„ Provides clues about the vocabulary of the
user
„ May change over time and/or be seasonal
Refinement rate „ Measures use of refiners
„ May illustrate the conceptual path users may
be taking to locate content
Activity Notables
What are users
looking for?
Are they finding it?

15. Group exercise
13 June 2019 Strategies and tactics to improve searchPage 15
Scenario:
User survey indicates > 40% are dissatisfied with search,
yet the number of queries and user sessions per month are
consistently high
Potential conclusion and next step?

16. Group exercise
13 June 2019 Strategies and tactics to improve searchPage 16
34%
8%
3%19%
21%
15%
Search log analysis
Service delivery
Account management
Legal
Administration
Tools
Career and benefits
Scenario:
Based on the search log analysis depicted, what can you
conclude? How would you respond to complaints from legal
about their content not being accessible via search?
Potential conclusion and next step?

17. Content: identify,
tag, optimize and
index critical content
13 June 2019 Strategies and tactics to improve searchPage 17

18. There are two key aspects to improving content for search
13 June 2019 Strategies and tactics to improve searchPage 18
1
The search
index
2
Metadata and
search engine
optimization
(SEO)
„ Content we think we have but don’t
„ Content we have, but don’t make available
„ Content we have, but don’t need
„ Content we know we don’t have, but users want
„ Content with no or inaccurate metadata
„ Content with poor titles and file names
„ Content with no abstracts

19. 1. Review your top queries, zero-hit queries and
abandoned queries
2. Conduct user research
3. Have users (or content owners) confirm the
content that should be returned
4. Identify new sources of content to index
5. Invest in a content review and archive process
How do you improve your search index?
13 June 2019 Strategies and tactics to improve searchPage 19
Rinse and repeat — this process never stops!

20. What to do with content you can’t index?
13 June 2019 Strategies and tactics to improve searchPage 20
Best bets/promoted results
• Derived from zero-hit queries
and keyword analysis
• Should be used sparingly
• Need to be aware of bias from
consumer experiences

21. Growing your search index may lead to long-term consequences
Increases the amount of “noise”
May “push” relevant results down
on the result page
Old publication dates create
credibility challenges
13 June 2019 Strategies and tactics to improve searchPage 21
Content review and archive
Start by creating a content review
group
Add metadata to support the
process (last review date, shelf life,
expiry date)
Consider automating the process to
ensure consistent execution
Measure and report progress
Solution:

22. 13 June 2019 Strategies and tactics to improve searchPage 22
Metadata – Changes this …

23. 13 June 2019 Strategies and tactics to improve searchPage 23
Into this

24. „ Given special attention by the search engine
„ Drives search refiners
„ Visible aspects significantly impact the optics of
the results
„ Should reflect the vocabulary of your users
„ Be wary of “keyword stuffing”
Metadata
13 June 2019 Strategies and tactics to improve searchPage 24

25. Search engine optimization (SEO)
13 June 2019 Strategies and tactics to improve searchPage 25
Keywords: identify a list
of keywords describing
the essence of the content
and rank them
Title: create a descriptive
title that includes the
most important keywords
Consider creating formal SEO guidelines and training your content creators.
Document properties:
verify document
properties
► Ensure a proper title,
author, keywords
► Be wary of documents
with values that have
been propagated from
other documents
File name: validate the file name
► Syntax (some search engines
have trouble with names
containing no spaces or with
special characters) — “2018
Digital Transformation Proposal”
vs.
“2018_Digital_Transformation_Pr
oposal”
► Descriptive (following the
approach used for the title) —
“Americas Oil and Gas Financial
Trends” vs. “Trends PowerPoint”
1
2
3 4

26. Group exercise
13 June 2019 Strategies and tactics to improve searchPage 26
The first EY Global Capital Confidence Barometer was launched 10 years
ago in the wake of the global financial crisis (GFC). At that time, the
survey results reflected the great unease across the markets. We again
find ourselves in a period of uncertainty. Though nothing like the
magnitude we experienced a decade ago, we don’t need to look far to
find negative headlines and speculation about an economic downturn.
However, the one major difference between now and then is how the C-
suite views uncertainty. Ten years ago, boardrooms were paralyzed by
uncertainty. Today they are motivated by uncertainty. The boardroom of
2019 is concentrating on proactively managing risks and seizing the
upside opportunities of disruption.
What keywords would you use for this article?

27. Group exercise
13 June 2019 Strategies and tactics to improve searchPage 27
Scenario:
When researching why your expertise search returns multiple,
incorrect results for a name-based search, you discover the
document properties of CV files have been incorrectly
propagated.
How could you correct the records already impacted? How do
you try to avoid this in the future?

28. Measure: measure
and interpret search
activity
13 June 2019 Strategies and tactics to improve searchPage 28

29. Measurement basics
„ Precision is the percentage of retrieved
results that are relevant
„ Recall is the percentage of all relevant
results that the search system actually
retrieves
13 June 2019 Strategies and tactics to improve searchPage 29
Precision =
+
Recall =
+
100
100 + 400
100
100 + 100
You are searching for a new SUV
from a total population of 1,000
cars. Assume there are 200 SUV
entries. However, your search
returns 500 results of which 100
are SUVs.
Precision: = 20%
Recall: = 50%
Example:

30. Interpreting precision and recall
13 June 2019 Strategies and tactics to improve searchPage 30
Low recall
and low
precision
High recall
and low
precision
True
negatives
False
negative
(FN)
False
positive
(FP)
True
positives
(TP)
True
negatives
FN FP
True
positives
Balanced, but an
insufficient set of
relevant results
Retrieves more of the
ideal content, but
with too many
irrelevant results

31. Interpreting precision and recall
13 June 2019 Strategies and tactics to improve searchPage 31
Low recall
and high
precision
High recall
and high
precision
True
negatives
FN FP
True
positives
True
negatives
FN FPTrue
positives
Relevant results, but
missing key content
Ideal experience!

32. Is there something better?
Mean average precision (MAP):
Method of evaluation that considers
whether all of the relevant items return
high in the result list by first averaging
the precision at each position and then
calculating the mean value across
multiple query results
13 June 2019 Strategies and tactics to improve searchPage 32
MAP =
∑ ( )
Your search results are as follows:
1, 0, 0, 1, 1, 1
1. Precision at 1 will be 1/1 = 1
2. Precision at 2 will be 0/2 = 0
3. Precision at 3 will be 0/3 = 0
4. Precision at 4 will be 2/4 = 0.5
5. Precision at 5 will be 3/5 = 0.6
6. Precision at 6 will be 4/6 = 0.66
Average precision will be
1+.5+.6+.66/4 = 0.69
Example:

33. Ground truth: the confirmed outcome or standard
against which you are measuring search relevance
„ Assemble subject-matter resources (SMRs)
relevant to your top queries
„ Identify and routinely confirm the “best” content
for a given search query
Measuring search requires the “ground truth”
13 June 2019 Strategies and tactics to improve searchPage 33
Do you find these results relevant?

34. Group exercise
13 June 2019 Strategies and tactics to improve searchPage 34
Scenario:
When analyzing the results of your top query analysis, you
discover that, while your result set is relevant, a significant
portion of your ideal content is not being returned.
Which precision/recall scenario is this? What would you do to
correct the situation?

35. Group exercise
13 June 2019 Strategies and tactics to improve searchPage 35
Scenario:
When analyzing the results of your top query analysis, you
find the following results:
Query set A: 1, 1, 0, 0, 0, 1, 1, 0, 1, 0
Query set B: 0, 1, 1, 1, 1, 1, 0, 1, 0, 0
Query set C: 0, 0, 1, 1, 1, 0, 1, 1, 0, 1
What is the average precision for each query set? What is the
mean average precision?

36. Tune and verify:
tune for the optimal
experience
13 June 2019 Strategies and tactics to improve searchPage 36

37. Tuning process
1.Identify queries
2.Measure relevance
3.Make an individual tuning adjustment
4.Verify and implement
13 June 2019 Strategies and tactics to improve searchPage 37
Make small rather
than large changes
Iterate to avoid
making things worse

38. Content fields
Content sources
The query
The content itself
The algorithm
What can be “tuned?”
13 June 2019 Strategies and tactics to improve searchPage 38
Complexity and
risk increase as
you progress
down this list.

39. Tuning content fields: “field boosting”
You can tune search to boost content that
matches the search within certain priority fields.
Examples:
Document title
Content type
Date
13 June 2019 Strategies and tactics to improve searchPage 39
Review your top
queries and top
query terms to
identify potential
fields to consider
boosting.

40. Tuning content sources
You can tune search to boost content that is
stored in certain priority sources.
Examples:
Policy database
Code repository
Service request application
13 June 2019 Strategies and tactics to improve searchPage 40
Consider any
authoritative
sources that match
your search
analytics

41. Tuning the query: “query cooking”
You can tune search to improve the query in
order to translate the intent of the user into
something optimized for search.
Examples:
Match user preferences
Highlight hard-to-locate content
Facilitate partial name searches
13 June 2019 Strategies and tactics to improve searchPage 41
Artificial intelligence
and machine
learning may
provide powerful
improvements to
this space

42. Tuning the content itself: content enrichment
You can tune search by improving the
content in the index.
Examples:
Associated supporting content: expand the profile
for expertise location with data from a project
database or CV repository
Entity extraction: the process of identifying and
classifying key elements from content into pre-
defined categories or metadata
13 June 2019 Strategies and tactics to improve searchPage 42
Entity extraction is
another tuning
approach that can
be greatly aided by
new technologies to
better disambiguate
meaning and
understand context.

43. Tuning the algorithm
Many search engines allow you to select a
specific algorithm to match the desired
search experience. Some even allow you to
modify the algorithm itself.
Examples:
SharePoint search: “default with two linear
stages,” “search model with boosted minspan,”
“people search name social distance,” etc.
Elasticsearch: “Lucene’s Practical Scoring
Function,” “BM25,” etc.
13 June 2019 Strategies and tactics to improve searchPage 43
Great care should be
taken if you make
changes to an out-
of-the-box
algorithm.

44. Group exercise
13 June 2019 Strategies and tactics to improve searchPage 44
Scenario:
You have a large amount of content that is largely user-
generated with a poor application of metadata.
What tuning method(s) would you use to improve your
search experience?

45. Group exercise
13 June 2019 Strategies and tactics to improve searchPage 45
Scenario:
You have good content from an authoritative source that is
consistently not being returned by your search engine.
What tuning method(s) would you use to improve your search
experience?

46. Key learnings
• Start with your user
• Get to know your search logs
• A content review and archive process is key
• Measuring search requires “the ground truth”
• Focus on small changes and iterate
5 April 2019 Strategies and tactics to improve searchPage 46 13 June 2019 Strategies and tactics to improve searchPage 46

47. EY | Assurance | Tax | Transactions | Advisory
About EY
EY is a global leader in assurance, tax, transaction and
advisory services. The insights and quality services we
deliver help build trust and confidence in the capital markets
and in economies the world over. We develop outstanding
leaders who team to deliver on our promises to all of our
stakeholders. In so doing, we play a critical role in building a
better working world for our people, for our clients and for
our communities.
EY refers to the global organization, and may refer to one or
more, of the member firms of Ernst & Young Global Limited,
each of which is a separate legal entity. Ernst & Young
Global Limited, a UK company limited by guarantee, does not
provide services to clients. Information about how EY
collects and uses personal data and a description of the
rights individuals have under data protection legislation are
available via ey.com/privacy. For more information about
our organization, please visit ey.com.
© 2019 Ernst & Young, LLP.
All Rights Reserved.
Ernst & Young LLP is a client-serving member firm of
Ernst & Young Global Limited operating in the US.
1903-3082875
ED None
ey.com