This document provides an agenda and overview for a beginner technical workshop on Splunk. The agenda includes introductions to getting started with Splunk, searching, alerts, dashboards, deployment and integration, and a question and answer session. It also provides background on Splunk's capabilities for searching machine data in real-time, monitoring systems proactively, and gaining operational visibility and real-time business insights. Demo sections are included to illustrate key Splunk functions.

1. Copyright*©*2013*Splunk*Inc.*
Technical*Workshop*
Beginner*
Ma?hias*Maier,*CISSP*
System*Engineer*

2. Agenda*
• GeGng*Started*with*Splunk*
• Search*
• Alert*
• Dashboard*
• Deployment*and*IntegraKon*
• Community*
• Help*&*QuesKons*
2*

3. Make*machine*data*accessible,*usable**
and*valuable*to*everyone.**
3*

4. “TwoQthirds*of*all*IT*spending*is*just*to*
sustain*the*business,*not*to*grow*or*
transform*the*business”*
4*
Source:(Gartner,(IT(Metrics:(IT(Spending(and(Staffing(Report,(2010(

5. 5*
Energy*
Manufacturing*
Shipping* RFID*
Web*
Services*
Developers*
App*
Support*
Telecoms*
Networking*
Desktops*
Servers*
Security*
Data*
Warehouse*
Storage*
Messaging*
Shopping*
Basket*
Clickstream*
GPS/Cellular*
Virtual*
Physical*
Clou
d*
Online*
Services*
*Machine*Data*Is*Pervasive*
AddiKonal*Sources* Core*IT* CustomerQfacing*IT*

6. Energy'
Manufacturing'
Shipping' RFID'
Web'
Services'
Developers'
App'
Support'
Telecoms'
Networking'
Desktops'
Servers'
Security'
Data'
Warehouse'
Storage'
Messaging'
Shopping'
Basket'
Clickstream'
GPS/Cellular'
Virtual'
Physical'
Cloud'
Online'
Services'
*Machine*Data*Is*Pervasive*
6*
AddiKonal*Sources* Core*IT* CustomerQfacing*IT*
" Machine*data*volumes,*sources*and*types*exploding*
" 80Q95%*of*an*organizaKon’s*data*is*unstructured*
" If*stored,*in*silos*throughout*the*organizaKon**
" New*technologies*adding*to*data*explosion*
(mobile*devices,*sensors,*GPS,*virtualizaKon,*cloud)*
" Market*trends*makes*this*data*valuable*to*the*business*
" Contains*a*categorical*record*of*acKvity*and*behavior*

7. 7*
What*It’s*Like*In*The*Trenches*
Service'Desk'
Applica/on'
Support'
Systems'
Administrator'
Applica/on'
Developer'
Applica/on'
Developer'
Database'
Administrator'
Log*call.*The*
console*says*
everything*is*
green.*
Java*monitoring*
tools*don’t*
show*anything*
either.*Call*the*
developer.*
Stop*working*on*
new*code*to*
troubleshoot.*
Need*producKon*
logs!*
Stop*what*
they’re*doing*to*
idenKfy*and*
gather*
producKon*logs*
for*developer.*
Manual*
invesKgaKon*
establishes*not*
applicaKon*
problem.**
*
DBA*analyzes*
audit*logs*which*
points*to*bad*
query.*
Escalate.' Escalate.' Escalate.' Respond.' Escalate.' Now'what?'

8. And*When*The*Business*Needs*Intelligence...*
8*
I'need'a'new'
report'
We’ll*need*to*
change*the**
schema*for*that*
I'need'to'see'it'in'
real'/me'
Our*system*is*
batch.*We’ll*need*
a*new*system*
What’s'the'trend'
over'the'last'
year?'
Sorry.*We*only*
keep*7*days*of**
data*online*

9. **
Collect,*index*and*harness*your*machine*data*
to*idenKfy*problems,*pa?erns,*risks*and*
opportuniKes*and*drive*be?er*decisions**
for*IT*and*the*business.*
9*

10. Harnessing*Your*Machine*Data*
10*
Search*and*
InvesKgate*
Reac/ve'
IT'silo'chaos'
InvesKgate,*find*and*fix*problems*dramaKcally**
faster*across*your*organizaKon*
Proac/ve'

11. Harnessing*Your*Machine*Data*
11*
Search*and*
InvesKgate*
ProacKve*
Monitoring*
Reac/ve'
AutomaKcally*monitor*to*idenKfy*issues,*
problems*and*a?acks*before*they**
impact*your*customers*and*services*
IT'silo'chaos'
Proac/ve'

12. Harnessing*Your*Machine*Data*
12*
Search*and*
InvesKgate*
ProacKve*
Monitoring*
OperaKonal*
Visibility*
Reac/ve'
Gain*endQtoQend*visibility*to*track*and*deliver*on*
*IT*KPIs*and*make*be?erQinformed*IT*decisions*
IT'silo'chaos'
Proac/ve'

13. Harnessing*Your*Machine*Data*
13*
Search*and*
InvesKgate*
ProacKve*
Monitoring*
OperaKonal*
Visibility*
RealQKme**
Business*Insights*
Proac/ve'
Reac/ve'
IT'silo'chaos'
Gain*realQKme*insight*from*operaKonal*data*to*
*make*be?erQinformed*business*decisions*

14. GeGng*Started*With*
Splunk*

15. Install*Splunk*
Start*Splunk*
• WIN:*Program*FilesSplunkbinsplunk.exe'start'
• Other:*/opt/splunk/bin/splunk'start'
www.splunk.com/download*
• 32*or*64*bit?*
• Indexer*or*Universal*
Forwarder?*
Splunk*Home*
• WIN:*Program*FilesSplunk*
• Other:*/opt/splunk*(ApplicaKons/splunk)*

16. Splunk*Licenses*
Free*Download*Limits*Indexing*to*500MB/day*
• Enterprise*Trial*License*expires*arer*60*days*
• Reverts*to*Free*License*
Features*Disabled*in*Free*License*
• MulKple*user*accounts*and*roleQbased*access*controls*
• Distributed*search*
• Forwarding*to*nonQSplunk*Instances*
• Deployment*management*
• Scheduled*saved*searches*and*alerKng*
• Summary*indexing*
Other*License*Types*
• Enterprise,*Forwarder,*Trial*

17. 17*
Splunk*Web*Basics*
Browser*Support*
• Firefox*3.6,*10.x*and*latest*
• Internet*Explorer*6,*7,*8*and*9*
• Safari*(latest)*
• Chrome*(latest)*
Default*on*install*is*h?p://localhost:8000*
*
Index*some*data*
• Add*data*
• GeGng*Started*App*
• Install*an*App*(Splunk*for*Windows,**NIX)*

18. 18*
Splunk*Web*Basics*
Splunk*Apps*
• Splunk*Home*Q>*Find*more*apps*
• Apps*create*different*contexts*for*your*data*out*of*sets*of*views,*
dashboards,*and*configuraKons*
• You*can*create*your*own!*
Search*is*an*App*
• Summary*will*show*everything*you*have*indexed**
• Updated*in*realQKme*
• Click*on*any*source,*sourcetype,*or*host*to*look*at*events*

19. Search*Basics*

20. Searching*
20*
Search*>***
Select*Time*Range*
• Historical,*custom,*or*realQKme*
Using*the*Kmeline*
• Click*events*and*zoom*in*and*out*
• Click*and*drag*over*events*for*a*specific*range*

21. 21*
Everything*is*searchable*
Everything*is*searchable*
• **wildcard*supported**
• Search*terms*are*case*
insensiKve*
• Booleans*AND,*OR,*NOT**
– Booleans*must*be*
uppercase*
– Implied*AND*between*
terms*
– Use*()*for*complex*
searches*
• Quote*phrases**
fail*&
fail*&nfs&
error&OR&404&
error&OR&failed&OR&(sourcetype=access_*(500&OR&503))&
login&failure &

22. Search*Assistant*
22*
Contextual*Help*
• advanced*type*ahead*
History*
• search*
• commands*
Search*Reference*
• short/long*
descripKon*
• examples*
suggests search terms
and displays count
updates as you type
shows examples and help
toggle off / on

23. Searches*can*be*managed*as*
asynchronous*processes*
Jobs*can*be**
• Scheduled*
• Moved*to*background*tasks*
• Paused,*stopped,*resumed,*finalized*
• Managed*
• Archived*
• Cancelled*
Job*Management*
send to background
pause finalize
cancel

24. 24*
Tagging*and*Event*Typing*
Even?ypes*for*more*humanQreadable*reports*
• categorize*and*make*sense*of*mountains*of*data*
• punctuaKon*helps*find*events*with*similar*pa?erns*
Search*>*even?ype=failed_login*instead*of*
Search*>*“failed*login”*OR*“FAILED*LOGIN”*OR*“AuthenKcaKon*failure”*OR*“Failed*to*
authenKcate*user”*
Tags*are*labels*
• apply*adhoc*knowledge*
• create*logical*divisions*or*groups*
• tag*hosts,*sources,*fields,*even*even?ypes*
Search*>*tag=web_servers*instead*of*
Search*>*host=“apache1.splunk.com”*OR*host=“apache2.splunk.com”*OR*
host=“apache3.splunk.com”*

25. Search*Commands*
25*
Search*>*error*|*head*1*
Search*results*are*“piped”*to*the*command*
Commands*for:*
• ManipulaKng*fields*
• FormaGng*
• Handling*results*
• ReporKng*

26. Over*100*Commands!*
26*
splunk.com > Documentation > Search Reference
abstract accum addcoltotals addinfo addtotals af analyzefields anomalies
anomalousvalue append appendcols ar associate audit autoregress bin bucket
chart cluster collect common contingency convert correlate counttable crawl
ctable dbinspect dedup delete delta diff discretize erex eval eventcount eventstats
excerpt extract file fillnull folderize format gentimes head highlight iconify input
inputcsv inputlookup iplocation join kmeans kv kvform loadjob localize localop
lookup macro makecontinuous makemv maketable map metadata multikv
mvcombine mvexpand nomv outlier outlierfilter outputcsv outputlookup
outputtext overlap rangemap rare regex relevancy rename replace reverse run
savedsearch savedsplunk script scrub selfjoin sendemail set sichart sirare sistats
sitimechart sitop slc stash strcat streamstats sumindex summaryindex tail test
timechart top transaction transam trendline typeahead typelearner typer uniq
untable xmlkv xmlunescape xpath xyseries
h?p://www.splunk.com/base/DocumentaKon/latest/SearchReference/SearchCheatsheet*

27. Date*
Name*
Title*
DEMO*

28. Field*ExtracKon*

29. Fields*
29*
Default*fields*
• host,*source,*sourctype,*linecount,*punct*etc.*
• View*on*ler*panel*in*search*results*or*all*in*field*picker*
Where*do*fields*come*from?*
• PreQdefined*by*sourcetypes*
• AutomaKcally*extracted*keyQvalue*pairs*
• User*defined*

30. Extract*Fields*
30*
InteracKve*Field*Extractor*
• generate*PCRE*
• editable*regex*
• preview/save*
props.conf!
!
[mysourcetype]!
REPORT-myclass = myFields!
!
transforms.conf!
!
[myFields]!
REGEX = ^(w+)s!
FORMAT = myFieldLabel::$1 !
ConfiguraKon*File*
• manual*field*extracKon*
• delimQbased*extracKons*
Rex*Search*Command*
!... | rex field=_raw "From: (?<from>.*) To: (?
<to>.*)"!

31. Date*
Name*
Title*
*
Technical*
Workshops*
GeGng*Started*User*Training*
DEMO*

32. Saved*Search*&*Alert*Basics*

33. Saved*Searches*and*AlerKng*
33*
Find*Something*InteresKng?*
*
*
*
OR*
*

34. AlerKng*
34*
Searches*run*on*a*schedule*and*fire*an*alert**
• Example:*Run*a*search*for*“Failed*password”*every*15*min*
over*the*last*15*min*and*alert*if*the*number*of*events*is*
greater*than*10*
Searches*are*running*in*realQKme*and*fire*an*alert*
• Example:*Run*a*search*for*“Failed*password*
user=john.doe”*in*a*1*minute*window*and*alert*if*an*event*
is*found*
*

35. AlerKng*AcKons*
35*
• Send*email*
• RSS*
• Execute*a*script*
• Track*in*Alert*Manager*

36. Report*&*Dashboard*

37. ReporKng*
37*
Build*reports*from*the*results*of*any*search*
*
Select*type*of*report*(Values*over*Kme,*Top*Values,*Rare*Values)**
and*on*which*fields*to*report*or*perform*staKsKcs* Choose*the*type*of*chart*(line,*area,*column,*etc)*and**
other*formaGng*opKons*

38. ReporKng*Examples*
38*
• Use*wizard*or*reporKng*commands*(Kmechart,*top,*etc)*
• Build*realQKme*reports*with*realQKme*searches*
• Save*reports*for*use*on*dashboards*

39. Dashboards*
39*
Create*dashboards*from*search*results*

40. Dashboard*Examples*
40*

41. Splunk*Manager*
41*
Now*Manage*All*of*that*Cool*Stuff*You*Just*Created*(and*more!)*
• Permissions*
• Saved*Searches/Reports*
• Custom*Views*
• Distributed*Splunk*
• Deployment*Server*
• License*Usage….*

42. Date*
Name*
Title*
*
Technical*
Workshops*
GeGng*Started*User*Training*
DEMO*

43. Deployment*and*
IntegraKon*

44. CollecKon*
Indexing*
Search*
Core*FuncKons*
Access*
Controls*
Stats/*
AnalyKcs*
Alerts* Dashboards*Reports*
Apps*and*SoluKons*
ApplicaKon*
Monitoring*
SDK'User'Interface' APIs'
IT*
OperaKons*
Security* Compliance*
Business*
AnalyKcs*
Web*
Intelligence*
44*

45. Splunk*Has*Four*Primary*FuncKons*
45*
• Searching*and*ReporKng*(Search*Head)*
*
• Indexing*and*Search*Services*(Indexer)*
*
• Local*and*Distributed*Management*(Deployment*Server)*
*
• Data*CollecKon*and*Forwarding*(Forwarder)*
A*Splunk*install*can*be*one*or*all*roles…**

46. GeGng*Data*Into*Splunk*
Agent*and*AgentQless*Approach*for*Flexibility*
46*
perf*
shell*
code*
Mounted'File'Systems'
hostnamemount+
syslog'
TCP/UDP+
WMI'
Event+Logs+Performance'
Ac/ve''
Directory'
syslog'compa/ble'hosts'
and'network'devices'
Unix,'Linux'and'Windows'hosts'
Windows'hosts' Custom'apps'and'scripted'API'connec/ons'
Local'File'Monitoring'
log+files,+config+files+
dumps+and+trace+files+
Windows'Inputs'
Event+Logs+
performance+counters+
registry+monitoring+
AcAve+Directory+monitoring+
virtual'
host'
Windows'hosts'
Scripted'Inputs'
shell+scripts+custom+
parsers+batch+loading+
'
AgentNless'Data'Input' Splunk'Forwarder'

47. Understanding*the*Universal*Forwarder*
47*
Forward*data*without*negaKvely*impacKng*producKon*performance.*
Scripts*
Universal*Forwarder*Deployment*
Logs* ConfiguraKons*Messages* Metrics*
Central*Deployment*Management*
Monitor*files,*changes*and*the*system*registry;*capture*metrics*and*status.*
Universal'Forwarder' Regular'(Heavy)'Forwarder'
Monitor*All*
Supported*
Inputs*
* *
*
RouKng,*
Filtering,*
Cloning*
* *
Scripted*
Inputs*
*
*
*
*
Splunk*Web* *
Python*
Libraries*
*
Event*Based*
RouKng*
*

48. Horizontal*Scaling*
48*
Load*balanced*search*and*indexing*for*massive,*linear*scale*out.*
Forwarder***
Auto*Load*
Balancing*
Distributed*Search*

49. MulKple*Datacenters*
49*
Headquarters*
London* Hong*Kong* Tokyo* New*York*
Distributed*Search*
Index*and*store*locally.*Distribute*searches*to*datacenters,*networks*&*geographies.*

50. High*Availability,*On*Commodity*Hardware*
50*
Splunk*Universal*
Forwarder*Pool*
Constant*
UpKme*
" As*Splunk*collects*data,*it*keeps*mulKple*
idenKcal*copies*
" Data*integrity*and*resilience*without*a*SAN*
" If*indexer*fails,*incoming*data*conKnues*to*
get*indexed*
" Indexed*data*conKnues*to*be*searchable*

51. High*Availability*
51*
Combine*auto*load*balancing*and*cloning*for*HA*at*every*Splunk*Ker.*
*Clone*Group*1*:*Complete*Dataset*
Data*Cloning*&**
Auto*Load*Balancing*
Distributed*Search* Distributed*Search*
Clone*Group*2*:*Complete*Dataset**
*
Shared*Storage*

52. Service*Desk*
Event*Console*
SIEM*
Send*Data*to*Other*Systems*
52*
Route*raw*data*in*real*Kme*or*send*alerts*based*on*searches.*

53. Integrate*External*Data*
53*
LDAP,*AD* Watch**
Lists*
CRM/
ERP*
CMDB*
Correlate*IP*addresses*with*locaKons,*accounts*with*regions*
Extend*search*with*lookups*to*external*data*sources.*

54. Integrate*Users*and*Roles**
54*
Problem*InvesKgaKon* Problem*InvesKgaKon* Problem*InvesKgaKon*
Save*
Searches*
Share*
Searches*
LDAP,*AD**
Users*and*Groups*
Splunk*Flexible*Roles*
Manage*
Users*
Manage*
Indexes*
CapabiliKes*&*Filters*
NOT*
tag=PCI*
App=ERP*
…*
Map*LDAP*&*AD*groups*to*flexible*Splunk*roles.*Define*any*search*as*a*filter.*
Integrate*authenKcaKon*with*LDAP*and*AcKve*Directory.*

55. Centralized*Licensing*Management*
55*
Problem*InvesKgaKon*
Groups,*Stacks,*and*Pools*for*Enterprise*Deployments*

56. Deployment*Monitoring*
56*
Keep*Tabs*On*Your*Splunk*Enterprise*Deployment*
Forwarders*Indexers*Sourcetypes*Licenses*

57. Support*and*
Community*

58. Support*Through*the*Splunk*Community*
58*
Browse and share Apps
from Splunk, Partners and
the Community
splunkbase.splunk.com
Splunkbase*
Community-driven
knowledge exchange
and Q&A
answers.splunk.com
5 tracks, more than 40
sessions, the smartest
Splunk users together
conf.splunk.com

59. Where*to*Go*for*Help*
59*
• DocumentaKon*
– h?p://www.splunk.com/base/DocumentaKon*
• Technical*Support**
– h?p://www.splunk.com/support*
• Videos*
– h?p://www.splunk.com/videos*
• EducaKon*
– h?p://www.splunk.com/goto/educaKon*
• Professional*Services*
*

60. Thank*You*