An evening focused discussing the [often] missing layers of event collection within Splunk deployments. We'll cover the ins and outs of traditional syslog collection and also explore how the Splunk HTTP Event Collector can be used to similar effect.

1. © 2018 SPLUNK INC.© 2018 SPLUNK INC.
Splunk User Group Edinburgh

2. © 2018 SPLUNK INC.
Harry McLaren
● Alumnus of Edinburgh Napier (Mentor)
● Senior Security Consultant at ECS
● Leader of the Splunk User Group Edinburgh

3. © 2018 SPLUNK INC.
Introduction to ECS
Splunk Partner - UK
– Type: Security / IT Operations / Managed Services (SOC / Splunk)
– Awards: Splunk Revolution Award & Splunk Partner of the Year

4. © 2018 SPLUNK INC.
Agenda
• Housekeeping: Event Overview & House Rules
• Syslog Collection with rsyslog
• HTTP Event Collection with Splunk
• User Group Update & Request for Speakers

5. © 2018 SPLUNK INC.
Splunk [Official] User Group
“The overall goal is to create an authentic, ongoing
user group experience for our users, where
they contribute and get involved”
● Technical Discussions
● Sharing Environment
● Build Trust
● No Sales!

6. © 2018 SPLUNK INC.
Syslog Collection
with rsyslog
Graeme Curtis

7. Syslog - Analysis and Collection
Graeme Curtis
Head of Research & Development, ECS Security

8. agenda
• syslog overview
• different syslog flavours
• modular syslog configuration
• debugging your configuration
• packaging as a Splunk app
• logfile weeding
• architecture

9. what is syslog?
• a shared message logging service originally created on BSD in the 1980’s
• logging follows a standard format
<34>Oct 11 22:14:15 MYMACHINE su: 'su root' failed for gac on /dev/pts/8
priority timestamp hostname tag message
RFC316
4<34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - - - 'su root' failed for gac on /dev/pts/8
RFC542
4
priority
version
timestamp hostname processname
pid
msgid
bom
message
• actually, many vendors do exactly as they please and often ignore
the syslog standards

10. the right way to ingest syslog
• simply open a couple of listening ports TCP/UDP514
• choose a unique port per device type
• collect syslog sources on dedicated syslog servers
recommendation:
1. always implement standalone syslog servers as part of your Splunk Infrastructure
2. ensure that the configuration of the syslog server is a responsibility of the Splunk team

11. syslog server options
• choose a *nix operating system
• two common options for syslog servers
recommendation:
rsyslog is my preferred choice as it doesn’t implement a dual license model
‘advanced’ features of syslog-ng are only available via a commercial license

12. creating a modular rsyslog config
• rsyslog configurations can be broken down into 4 major components:
1. globals –
defining global variables such as queue / message size
loading modules such as udp and tcp sockets
2. inputs –
describes message input types
3. rules –
determines what action to take when a message is received
4. templates –
sets the output format of the message

13. creating a modular rsyslog config
• rsyslog will allow the use of include files and hence we can readily
modularise each of the components
/etc/rsyslog.conf
/etc/rsyslog.d/splunk.conf
/etc/rsyslog.d/splunk-global
/etc/rsyslog.d/splunk-inputs
/etc/rsyslog.d/splunk-rules
/etc/rsyslog.d/splunk-templates
Let’s have a look at how we structure our filesystem and what goes
where…

14. testing your configuration 1
whenever you onboard a syslog source, always keep a representative
example of events in the ./splunk-tests directory

15. testing your configuration 2
remember we mentioned vendors sending non-standard syslog…
Nov 2 15:04:45 10.91.254.15 devname=SGPDC22-F3-NFW01 devid=FGT1KD3916801084 vd=root
date=2017-11-02 time=15:04:42 logid=0100100032002 type=event subtype=system level=alert
vd=root logdesc="Admin login failed" sn=0 user="c" ui=console method=console srcip=0.0.0.0
dstip=0.0.0.0 action=login status=failed reason="name_invalid" msg="Administrator c login
failed from console because of invalid user name"

16. testing your configuration 3
consider the rules…
Nov 2 15:04:45 10.91.254.15 devname=SGPDC22-F3-NFW01 devid=FGT1KD3916801084 vd=root date=2017-11-02 time=15:04:42
logid=0100100032002 type=event subtype=system level=alert vd=root logdesc="Admin login failed" sn=0 user="c" ui=console
method=console srcip=0.0.0.0 dstip=0.0.0.0 action=login status=failed reason="name_invalid" msg="Administrator c login failed from
console because of invalid user name"
if $msg contains 'devid=FGHA' then {
action(type="omfile" DynaFile="syslog_514_fortigateha" ...)
stop
}
if $msg contains 'devid=FG' then {
action(type="omfile" DynaFile="syslog_514_fortigate” ...)
stop
}

17. testing your configuration 3
consider the templates…
Nov 2 15:04:45 10.91.254.15 devname=SGPDC22-F3-NFW01 devid=FGT1KD3916801084 vd=root date=2017-11-02 time=15:04:42
logid=0100100032002 type=event subtype=system level=alert vd=root logdesc="Admin login failed" sn=0 user="c" ui=console
method=console srcip=0.0.0.0 dstip=0.0.0.0 action=login status=failed reason="name_invalid" msg="Administrator c login failed from
console because of invalid user name"
if $msg contains 'devid=FGHA' then {
action(type="omfile" DynaFile="syslog_514_fortigateha" ...)
stop
}
if $msg contains 'devid=FG' then {
action(type="omfile" DynaFile="syslog_514_fortigate" ...)
stop
}
template (name="syslog_514_fortigate” … string=”…/fortigate/%msg:R,ERE,1,DFLT:devname=([^ ]+)--end%/%fromhost-ip%.log")
template (name="syslog_514_fortigateha” … string=”…/fortigate/%msg:R,ERE,1,DFLT:[ ]vd=([^ ]+)--end%/%fromhost-ip%.log")
Some template content snipped for ease of reading

18. testing your configuration 4
debugging is configured in our standard policy via a global variable…
# Set debug status to either true or false
set $/debug = "false";

19. testing your configuration 5
this will redirect message flow from the syslog_514 ruleset to the debug
ruleset …
# Set debug status to either true or false
set $/debug = "false";
ruleset(name="syslog_514") {
$RulesetCreateMainQueue on # Create ruleset specific main queue for performance benefit
if $/debug == 'true' then {
call debug
stop
}

20. testing your configuration 6
add a couple of variables to allow you to track what’s happening in the
config…
# Set debug status to either true or false
set $/debug = "false";
ruleset(name="syslog_514") {
$RulesetCreateMainQueue on # Create ruleset specific main queue for performance benefit
if $/debug == 'true' then {
call debug
stop
}
if $msg contains 'devid=FG' then {
set $!debugrule="fortigate_rule_002";
set $!debugtemplate="fortigate";
action(type="omfile" file="/var/splunk-syslog/debug/debug.log" template="debug")
stop
}

21. testing your configuration 7
finally, we output the message as it’s been parsed by rsyslog into JSON
…
ruleset(name="syslog_514") {
$RulesetCreateMainQueue on # Create ruleset specific main queue for performance benefit
if $/debug == 'true' then {
call debug
stop
}
if $msg contains 'devid=FG' then {
set $!debugrule="fortigate_rule_002";
set $!debugtemplate="fortigate";
action(type="omfile" file="/var/splunk-syslog/debug/debug.log" template="debug")
stop
}
template (name="debug" type="list") {
property(name="jsonmesg")
constant(value="n")
}

22. logfile weeding
remember to clean-up any locally stored logfiles…
- can use logrotate
- or alternatively, use your templates to create a folder structure
containing datetime values and schedule a simply deletion script.

23. typical recommended architecture
• consider using a hardware load
balancer
• what’s with the universal forwarders?

24. © 2018 SPLUNK INC.
HTTP Event Collection
(HEC) with Splunk
Harry McLaren

25. © 2018 SPLUNK INC.
HTTP Event Collector (HEC)
▶ A token-based JSON API for
events/metrics.
▶ Send events directly from anywhere
(servers, mobile devices, IOT, cloud).
▶ Easy to configure / works out of the box.
▶ Easy to secure using tokens.
▶ Highly performant, scalable and available.

26. © 2018 SPLUNK INC.
How To Use
▶ Enable HTTP Event Collector (Splunk Enterprise)
▶ Create a unique token
▶ Send events to Splunk using the token
• Use HTTP(S) Directly
• Create a POST request and set the auth header
• POST JSON in our event format to the collector
• Use logging libraries
• Support for .NET, Java and JavaScript loggers

27. © 2018 SPLUNK INC.
Sending Data with: JSON Event Example
curl -k -H "Authorization: Splunk 12345678-1234-
1234-1234-1234567890AB" https://http-inputs-
mysplunkcloud.example.com:8088/services/collector/
event -d '{"sourcetype": "mysourcetype", "event":
"http auth ftw!"}'

28. © 2018 SPLUNK INC.
Sending Data with: JSON Metric Example
curl -k https://localhost:8088/services/collector
-H "Authorization: Splunk b0221cd8-c4b4-465a-
9a3c-273e3a75aa29" -d '{"time":
1486683865.000,"event":"metric","source":"disk","h
ost":"host_99","fields":{"region":"us-west-
1","datacenter":"us-west-
1a","rack":"63","os":"Ubuntu16.10","arch":"x64","t
eam":"LON","service":"6","service_version":"0","se
rvice_environment":"test","path":"/dev/sda1","fsty
pe":"ext3","_value":1099511627776,"metric_name":"t
otal"}}'

29. © 2018 SPLUNK INC.
Sending Data with: Raw Event Example
curl https://http-
input.splunkcloud.com/services/collector/raw -H
"X-Splunk-Request-Channel: FE0ECFAD-13D5-401B-
847D-77833BD77131" -H "Authorization: Splunk
BD274822-96AA-4DA6-90EC-18940FB2414C" -d 'Jun 10
12:17:50 payroll proftpd[15997]: payroll.acme.com
(172.16.0.16[172.16.0.16]) - USER Administrator:
no such user found from 172.16.0.1 [172.16.0.16]
to 10.1.1.1:21' -v

30. © 2018 SPLUNK INC.
Tips
▶ Create tokens per app, department, component, service. etc. Not per user or
device especially if you are talking about a large number (> 10000).
▶ Consider partitioning tokens to different indexes. This will speed up searches
and make it easy to archive.
▶ Consider delegating token management using DevOps / Engineering.
▶ Explicitly set allowed indexes on the token. If not set, the token can send data
to any index.
▶ Use HTTP over HTTPS when you can. You can get about a 30% performance
gain.
▶ Ask your developers to batch events, it greatly improves throughput.

31. © 2018 SPLUNK INC.
AWS Cloud Based Example
Use AWS Lambda with HTTP Event Collector

32. © 2018 SPLUNK INC.
Distributed Deployment
HTTP Event Collector can scale to meet many of your needs!
• Build into splunkd, nothing special to install
• Run directly on the indexer
• Or run on a dedicated collector instance (heavy forwarder) and forward to an
indexer
• Uses Deployment Server to to sync tokens across the Collector instances

33. © 2018 SPLUNK INC.
Scale & High Availability: Scenario 1

34. © 2018 SPLUNK INC.
Scale & High Availability: Scenario 2

35. © 2018 SPLUNK INC.
Scale & High Availability: Scenario 3

36. © 2018 SPLUNK INC.
Mid-Size Deployment

37. © 2018 SPLUNK INC.
Shake IT?
http://splunk.com/shake

38. © 2018 SPLUNK INC.
▶ Introduction to Splunk HTTP Event Collector
▶ Set up and use HTTP Event Collector
▶ HTTP Event Collector Walkthrough
▶ Use AWS Lambda with HTTP Event Collector
Resources

39. © 2018 SPLUNK INC.
User Group Update &
Request for Speakers
Harry McLaren

40. © 2018 SPLUNK INC.
Technical
Discussion
Community
Driven
Open to All
Experience
Levels
Varied
Topics
Splunk User Group, Edinburgh

41. © 2018 SPLUNK INC.
Request for Speakers & Topics
Experts Novices

42. © 2018 SPLUNK INC.
Get Involved!
● Splunk User Group Edinburgh
– https://usergroups.splunk.com/group/splunk-user-group-edinburgh.html
– https://www.linkedin.com/groups/12013212
● Splunk’s Slack Group
– Register via http://splunk-usergroups.signup.team/
– Channel: #edinburgh
● Present & Share at the User Group?
Connect:
‣ Harry McLaren | harry.mclaren@ecs.co.uk | @cyberharibu | harrymclaren.co.uk
‣ ECS | enquiries@ecs.co.uk | @ECS_IT | ecs.co.uk

43. © 2018 SPLUNK INC.© 2018 SPLUNK INC.
Thank You