Julian Harty, Sr. Sales Engineer, Splunk reviews the internals of how a Splunk search is performed, use of job inspector, search log, and gives a review of where and when to use certain commands.

1. Copyright
©
2014
Splunk
Inc.
Julian
Harty
SE,
Splunk>
Search
Op@miza@on
in
500
easy
steps

2. Disclaimer
2
During
the
course
of
this
presenta@on,
we
may
make
forward
looking
statements
regarding
future
events
or
the
expected
performance
of
the
company.
We
cau@on
you
that
such
statements
reflect
our
current
expecta@ons
and
es@mates
based
on
factors
currently
known
to
us
and
that
actual
events
or
results
could
differ
materially.
For
important
factors
that
may
cause
actual
results
to
differ
from
those
contained
in
our
forward-­‐looking
statements,
please
review
our
filings
with
the
SEC.
The
forward-­‐looking
statements
made
in
the
this
presenta@on
are
being
made
as
of
the
@me
and
date
of
its
live
presenta@on.
If
reviewed
aSer
its
live
presenta@on,
this
presenta@on
may
not
contain
current
or
accurate
informa@on.
We
do
not
assume
any
obliga@on
to
update
any
forward
looking
statements
we
may
make.
In
addi@on,
any
informa@on
about
our
roadmap
outlines
our
general
product
direc@on
and
is
subject
to
change
at
any
@me
without
no@ce.
It
is
for
informa@onal
purposes
only
and
shall
not,
be
incorporated
into
any
contract
or
other
commitment.
Splunk
undertakes
no
obliga@on
either
to
develop
the
features
or
func@onality
described
or
to
include
any
such
feature
or
func@onality
in
a
future
release.

3. Am
I
in
the
right
Session…
and
Who
is
this
guy?
3
Goal
of
Presenta:on:
Search
Op:miza:on
• How
the
hell
do
I
speed
this
search
up?
Background
of
your
Presenter:
Julian
Harty
• Splunker
for
2+
Years
-­‐
Variety
of
installa@ons
from
10GB
to
100TB’s+
• Ex-­‐Oracle/MySQL
DBA
(Recovering)
• Contact
info
julian@splunk.com

4. Background
–
Great
to
Not
So
Great
Growth
without
op@miza@on
=
subop@mal
performance
-­‐>
our
goal:
gejng
great
performance
at
scale
4
• More
Data
• More
Users
• New
Searches
• Even
More
Data
• Even
More
Users
• Even
More
Searches…
Op@miza@on
Steps

5. Challenge
–
Why
so
slow?
The
maturity
of
a
Splunk
deployment
5
Question?
Is your environment tuned
correctly?
Question?
Has your deployment
been architected
correctly?
Question?
Are your searches optimized?
Solution:
Architecting And
Designing Your Splunk
Deployment
- Simeon Yep
Solution:
Jiffy Lube Quick Tune Up For
Your Splunk Environment
– Sean Delaney
Solution:
Welcome to this session!!!

6. Agenda:
Objec@ves
of
this
Session
6
• The
Basics:
• Common
pinalls
-­‐
Best
prac@ces
and
what
not
to
do
• Take
away:
Basic
steps
to
a
beoer
search
• Beyond
the
Basics:
• Search
Architecture
and
Workflow
• Detailed
Search
review
–
using
Job
inspector
search
examples
• Take
away:
Job
Inspector
Cheat-­‐Sheet
• Q&A

7. Iden@fying
Poorly
Performing
Searches

8. End
User
Enquiries
8

9. SOS
–
Expensive
Searches
Search
Ac@vity,
Usage
Paoerns
-­‐>
SOS
–>
Search
-­‐>
Search
Detail
Ac@vity
-­‐>
Expensive
Searches
9

10. For
Splunk
6.2
Users
–
_Introspec@on
Index
10

11. Search
Tuning
–
The
Basics

12. The
Basics:
Common
Search
Behavior
12
>
be=selec@ve
AND
be=specific
|
…
Narrow
@me
range
>
foo
bar
>
host=web
sourcetype=access*
Use
Summary
Indexing
Use
Report
Accel
or
Summary
Indexing
Use
Fast/Smart
Mode
where
Possible
Bad
Behavior
Good
Behavior
Performance
Improvement
Comment
index=xyz
10-­‐50%
Index
and
default
fields
source=www
-­‐24h@h
365x
30x
Limit
Time
Range
>
foo
bar
30%
Combine
Searches
Fast/Smart
20-­‐50%
Fast
Mode
A
AND
C
AND
D
AND
E
5-­‐50%
Avoid
NOTS
Data
Models
and
Report
Accelera@on
Summary
Indexing
All
Time
Searches
>*
>
foo
|
search
bar
Verbose
Mode
Use
Intelligently
Use
Sparingly
1000%
1000%
Searches
over
large
datasets
Searches
over
long
periods
A
NOT
B

13. The
Basics:
Common
Op@miza@on
Mistakes
13
• Summary
indexing
is
Awesome!
– Ini@al
reac@on
-­‐
Summarize
EVERYTHING!!!
ê Summarizing
too
much
data
negates
the
point
• Report
Accelerate
=
Turbo
buoon
– Ini@al
reac@on
-­‐
Report
Accelerate
EVERYTHING!!!
ê Too
many
searches
=
skipped
search
issues
• Data
Models
are
the
answer!
– Ini@al
reac@on
–
everything
can
be
included!
ê Convoluted
data
models
can
increase
workload

14. OK,
But
How
can
you
enforce
these
recommenda@ons?

15. How
do
you
enforce
Best
Prac@ces?
15
Architect
Perspec:ve:
• User
educa@on
–
Best
Prac@ces
for
Users
Admin
Perspec:ve:
Restric@ng
User
Controls:
Pulling
in
the
reins
• Restric@ng
Role
Capabili@es
• Limit
index
• Limit
search
terms
• Limit
search
@me
range
• Limi@ng
Power
user
role
• Restrict
Number
of
RT+
Concurrent
Searches

16. How
do
you
enforce
Best
Prac@ces?
16
Admin
Perspec:ve:
• Time
range
defaults
(ui-­‐prefs.conf)
• Time
range
Web
dropdown
op@ons
(Times.conf)

17. OK
Now
More
advanced
Op@miza@on:
Lets
start
with
-­‐
the
skinny
on
How
Search
Works…

18. How
Search
Works
–
Physical
Perspec@ve
18
db_lt_et_4
db_lt_et_2
db_lt_et_1
db_lt_et_3
.tsidx
Sources.data
SourceTypes.data
Hosts.data
.gz
.gz
.gz
.gz
.gz
.gz
.gz
.gz
db_1290057665_1289504696_1
history
_internal
main

19. How
Search
Works
-­‐
Logically
Search
Query
Structure
Parse,
Fetch,
Summarize,
Display
19
Index=mydata
|
eval
loc=long+lat+alt
|
stats
count
retrieve
events
filter/transform/map

20. Splunk
Distributed
Search
20
4
Steps
to
a
Splunk
Search:
Parse,
Fetch,
Summarize,
Display
" StreamingCommand:
Applies
a
transforma@on
to
search
results
as
they
travel
through
the
processing
pipeline.
Eval
rex
where…
" Repor:ngCommand:
Processes
search
results
and
generates
a
repor@ng
data
structure.
Examples:
stats,
top,
and
@mechart…

21. Types
of
Searches
21
• Dense
– Low
cardinality
– Example:
sourcetype=access
method=GET
• Sparse
– High
cardinality
– Example:
sourcetype=access
method=GET
ac@on=purchase
• Super
Sparse
(or
Needle
in
a
Haystack)
– Very
high
cardinality
– Example:
sourcetype=cisco:asa
ac@on=denied
src=10.2.3.11
• Rare
– Use
Case:
user
behavior
tracking
– Example:
sourcetype=magicsource
|
rare
Dense
Super
Sparse
Sparse

22. Dense
Searches
(>10%
matching
results)
(scanCount
vs
eventCount
in
Job
Inspector)
22
Challenge:
• CPU
and
I/O-­‐bound
– Ini@al
spike
in
CPU
due
to
decompression
of
raw
events.
– Retrieval
rate:
50K
events
per
second
per
server
Solu:on:
• Divide
and
conquer
– Distribute
search
to
an
indexing
cluster
– Parallel
compute
and
merge
results
• Report
Accelera@on
or
use
of
Summaries
–
divide
and
Conquer
– Report
on
summarized
data
vs.
raw
data
>
sourcetype=access_combined
method=GET

23. Sparse
Searches
23
Challenge:
• CPU-­‐bound
– Dominant
cost
is
uncompressing
*.gz
raw
data
files
– Some@mes
need
to
read
far
into
a
file
to
retrieve
a
few
events
Solu:on:
• Avoid
cherry
picking
– Be
selec@ve
about
exclusions
(avoid
“NOT foo”
or
“field!=value”)
– Leverage
indexed
fields
• Filter
using
whole
terms
– Instead
of
> sourcetype=access_combined clientip=192.168.11.*!
– Use
> sourcetype=access_combined clientip=TERM(192.168.11.2)!
>
sourcetype=access_combined
status=404

24. Super
Sparse
Searches
24
• “Needle
in
Haystack”
• Very
I/O
intensive
• May
take
up
to
2
Seconds
to
parse
each
bucket
>
sourcetype=access_combined
status=404
10.2.1

25. Rare
Term
Searches
25
• Bloom
Filters*
– Bloom
filters
stored
in
each
bucket
– 50-­‐buckets
processed
per
second
– I/Os
reduced
as
buckets
are
excluded
from
100-­‐200
to
just
a
few
– 50-­‐100x
faster
than
Super
Sparse
searches
on
conven@onal
storage,
>1000x
faster
on
SSD
(Due
to
random
reads)
>
sourcetype=access_combined
sessionID=1234
*
A
Bloom
filter
is
a
data
structure
designed
to
tell
you
whether
or
not
an
element
is
present
in
a
set

26. How
can
I
determine
if
my
search
is
Dense
or
Sparse?
Use
Job
Inspector…
26
Component
Descrip:on
scanCount
The
number
of
events
that
are
scanned
or
read
off
disk.
eventCount
Number
of
events
that
are
returned
to
base
search
• For
dense
searches
scanCount
~=
eventCount.
• For
sparse
searches,
scanCount
>>
eventCount.
>
sourcetype=access_combined
status=404
81.11.191.113

27. Job
Inspector
Review

28. Measuring
Search
Using
the
Splunk
Search
Inspector
28
Copyright*©*2011,*Splunk*Inc.* Listen*to
*
Using*the*Search*Inspector*
3*
Timings*from*distributed*
Remote*timeline*
Timings*from*the*search*
command.*
Timings
from
distributed
peers
Timings
from
the
search
command
*
Using*the*Search*Inspector*
Timings*from*distributed*p
Remote*timeline*
Timings*from*the*search*
command.*
Key
Metrics:
• Comple@on
Time
• Number
of
Events
Scanned
• Search
SID
Job
Inspector

29. Job
Inspector
Walkthrough
–
Search
Command
29
Rawdata:
Improving
I/O
and
CPU
load
KV:
Are
field
extrac@ons
efficient
Lookups:
Used
appropriately
Autolookups
causing
issues
Typer:
Inefficient
Evenoypes
Alias:
Cascading
alias

30. Reading
Job
Inspector
-­‐
Search.Index
30
Search.index
=
Time
to
parse
and
read
the
tsidx
files
to
determine
where
to
read
in
rawdata
How
do
you
op:mize
this?
• Improving
I/O

31. Reading
Job
Inspector
-­‐
search.rawdata
31
Search.rawdata
=
Time
to
read
actual
events
from
rawdata
files
How
do
you
op:mize
this?
• Filtering
as
much
as
possible
• Add
Peers
• Alloca@ng
more
CPU,
improving
I/O

32. Reading
Job
Inspector
-­‐
search.kv
32
Search.KV=
Time
taken
to
apply
field
extrac@ons
to
events
How
do
you
op:mize
this?
Regex
op@miza@ons
• Avoid
greedy
operators
.*?
• Use
of
Anchors
^
$
• Non
Capturing
groups
for
repeats

33. Reading
Job
Inspector
-­‐
search.lookups
33
Search.lookups
=
Time
to
apply
lookups
to
search
How
do
you
op:mize
this?
• Use
Appropriately
(at
end
of
search)
• Autolookups
maybe
causing
issues

34. Reading
Job
Inspector
-­‐
search.typer
and
tags
34
Search.typer
=
Time
to
apply
event
types
to
the
search
How
do
you
op:mize
this?
• Use
Appropriately
• Removed
unused
tags
and
evenoypes

35. Job
Inspector
Walkthrough
–
Distributed
Search
35
Dispatch.createProviderQueue
Time
to
establish
connec@on
with
peers
Dispatch.fetch
Time
spent
wai@ng
to
fetch
events
Dispatch.evaluate
The
@me
spent
parsing
the
search
and
sejng
up
the
data
structures
needed
to
run
the
search.
How
do
you
op:mize
this?
• Improving
Peer
conduc@vity
• Improve
Bundle
replica@on
• Faster
storage

36. Job
Inspector
Walkthrough
–
Distributed
Search
36
Dispatch.stream.remote
Time
to
retrieve
events
from
each
remove
peer
Issue:
1. Unequal
Indexer
performance
• Either
Hardware
mismatch
• Uneven
distribu@on
of
indexes
2. AutoLB
issues

37. Job
Inspector
Conclusions:
Search
Command
Summary
37
Component
Descrip:on
index
look
in
tsidx
files
for
where
to
read
in
rawdata
rawdata
read
actual
events
from
rawdata
files
kv
apply
fields
to
the
events
filter
filter
out
events
that
don’t
match
(e.g.,
fields,
phrases)
alias
rename
fields
according
to
props.conf
lookups
create
new
fields
based
on
exis@ng
field
values
typer
assign
evenoypes
to
events
tags
assign
tags
to
events

38. Job
Inspector
Conclusion:
Distributed
Search
Summary
38
Metric
Descrip:on
Area
to
review
createProvider
Queue
The
@me
to
connect
to
all
search
peers.
Peer
conduc@vity
fetch
The
@me
spent
wai@ng
for
or
fetching
events
from
search
peers.
Faster
Storage
stream.remote
The
@me
spent
execu@ng
the
remote
search
in
a
distributed
search
environment,
aggregated
across
all
peers.
evaluate
The
@me
spent
parsing
the
search
and
sejng
up
the
data
structures
needed
to
run
the
search.
Possible
bundle
issues

39. Addi@onal
Key
Logfiles
related
to
search
39
Search
log:
" Stored
in
$SPLUNK_HOME/var/run/splunk/dispatch/
" Detailed
analysis
of
every
step
taken
by
the
search
" Search
‘stack
trace’

40. What
is
the
best
search
command
to
use?

41. Stats
vs
Transac@on
41
Search
Goal:
compute
sta@s@cs
on
the
dura@on
of
web
session
(JSESSIONID=unique
iden@fier):
>
|
stats
range(_@me)
as
dura@on
by
JSESSIONID
|
chart
count
by
dura@on
span=log2
>
sourcetype=access_combined
|
transac@on
JSESSIONID
|
chart
count
by
dura@on
span=log2
Not
so
Great:
Much
BeUer:

42. Dedup
vs
Latest
42
Search
Goal:
Return
latest
cart
ac@on
for
each
web
site
customer
>
sourcetype=access*
|
stats
latest(clien@p)
by
ac@on
>
sourcetype=access*|
dedup
clien@p
sortby
-­‐
_@me
|table
clien@p,
ac@on
Not
so
Great:
Much
BeUer:
Note:
dedup
can't
be
used
with
report
accelera@on

43. Joins
and
Subsearches
43
Search
Goal:
Return
latest
JESSIONID
across
two
sourcetypes
>
(sourcetype="access_combined")
OR
(sourcetype="applogs")
|
stats
latest(*)
as
*
by
JSESSIONID
>
sourcetype="access_combined"
|
join
type="inner"
JSESSIONID
[search
sourcetype="applogs"
|
dedup
JSESSIONID
|
table
JSESSIONID,
clien@p,
othervalue]
Not
so
Great:
Much
BeUer:

44. Wrap-­‐up

45. In
Closing…
45
1. Implemen@ng
Architecture
best
prac@ces
for
performance
at
scale
• With
search
behavior
in
mind…
2. Implemen@ng
User
Onboarding
Best
Prac@ces
• Basic
op@miza@on
steps
3. Periodic
Performance
Review
• Applying
accelera@on
technologies
where
appropriate
• Removing
unused
searches
4. Review
addi@onal
sides
for
• Search
flow
detail
• Op@mizing
Splunk
Web

46. And
By
the
way…
46
Other
Sessions
to
look
out
for:
• How
to
Actually
Use
Splunk
Data
Models
-­‐
David
Clawson
Presented
on
Tuesday
–
Check
out
the
session
notes
• Jiffy
Lube
Tune-­‐Up
for
your
Splunk
Deployment
-­‐
Sean
Delaney
Presented
on
Tuesday
–
Check
out
the
session
notes
• ArchitecCng
and
Sizing
your
Splunk
Environment
-­‐
Simeon
Yep
2:15-­‐3:15
Today
• Splunk
Search
AcceleraCon
Technologies
–
Gerald
Kanapathy
10:30-­‐11:30
Tomorrow
My
Contact
informa:on:
julian@splunk.com
@julian_Harty

47. THANK
YOU

48. Take
Away:
Basic
Steps
to
a
beoer
search
48
• Avoid
use
of
*
where
ever
possible.
• Avoid
the
use
of
All
Time.
• Avoid
subsearches
searches.
• Incorporate
the
use
default
fields
(source,
sourcetype,
host)
as
well
as
specific
indexes
to
every
search
(where
possible).
• Use
Fast
or
Smart
mode
where
possible
avoid
‘Verbose’
mode.
• Use
Report
Accelera@on
Sparingly
(and
Strategically)
on
reports
on
large
datasets.
• Use
Summary
Indexing
when
building
reports
over
@me
spans
beyond
target
index
reten@on.
• Use
Job
Inspector
and
Search
inspector
to
get
more
info
(hold
on
for
more
details!!!)

49. A
few
notes
on
how
to
op@mize
Splunk
Web
49
|
fields
Change
Segmenta@on
Use
Fast
Mode
Collapse
Timeline

50. Search
flow
–
Local
and
Distributed
50
Key
Files:
• Info
• Status
• Results
• Preview
Key
Flow:
1. Find
which
Bundle
to
use
2. Find
Buckets
to
use
(@me
range)
3. LISPY
TSIDX
search
4. Process
+
Summarizes
Events
hop://wiki.splunk.com/Community:HowDistSearchWorks