1.
SHOWDOWN:
Threat Stack
vs.
RedHat AuditD
By Jen Andre, Co-Founder Threat Stack

2.
We like magic.

3.
But since magic isn’t real, we have to come up with the next
best thing to make our agent the best in its class.

4.
Savvy operations and security
people that use our service are
blown away by the types of
information the Threat Stack agent
can collect, correlate, and analyze
from Linux servers.

5.
“I’ve tried to do this with (Red Hat) audits with little to no success
…how do you guys do it?” they say.

6.
The Linux audit subsystem is a
very powerful way to collect
information about system calls
and other security-relevant
activity.

7.
The best part…
No kernel module is required
to enable this detailed level of auditing
since it’s built right into Linux!

8.
You can write a log anytime a particular system call
happens, whether that be unlink or getpid.

9.
And since the auditing operates
at such a low level, the granularity
of information is incredibly useful.

10.
Traditionally, people have used the user land
daemon ‘auditd’ built by some good Red
Hat folks to collect and consume this data.

11.
However, there are a couple of problems with
traditional open source auditd and auditd libraries
that we’ve had to deal with ourselves…

12.
…especially when trying to run it on performance
sensitive systems and make sense of the obtuse
data that traditional auditd spits out.

13.
To that effect, we’ve written a custom audit listener
from the ground up for the Threat Stack agent
(tsauditd)

14.
This is what makes Threat Stack’s
audit listener special…

15.
1. Performance Enhancements
Many people have tried to use traditional Red Hat audits
in production to do very detailed auditing of user,
process, and network syscall activity, but have failed
due to the performance impact.

16.
We’ve ensured that our agent is
responsible with resource utilization
through our unique parsing model.

17.
While benchmarking a web server,
we saw auditd consume 120% of
the CPU!
Threat Stack’s agent CPU
consumption
was only 10%!

18.
2. Output Enhancements
This Linux audit system will output many different
lines across disparate events into syscall, which
you then have to correlate later via your ingestion
engine or your log management system…

19.
The key-value format is also
cumbersome to parse,
and values are often
encoded into hex randomly…

20.
We’ve decided that all related events should
be grouped together and have conveniently
parsed everything correctly for you.

21.
We then transformed that into a
JSON output format that is much
simpler to read and parse.

22.
3. Network tooling (“src/dst port”)
Tracking network connections across multiple
hosts can be a manual and painful process when
trying to connect across boxes.

23.
To make it easier, our agent adds metadata
to network connection events to
determine where the connection is
originating from and where it is going.

24.
Our backend then correlates these network events to
determine the originating process and potential user activity
that caused that network event, so long as the agent lives on
both the source and destination server.

25.
This is especially useful for
tracking SSH sessions
across your environment and debugging
what servers are speaking to one another and why.

26.
4. User Activity Auditing
Digging around the server logs to see where a user on
your system went is not an easy job.

27.
You’d need to manually find the agent and session that a user
connected to, yet all the kernel gives us is a nasty hex
encoding string representing the connection address in the
traditional auditd logs!

28.
On top of that, most of the information
logged by auditd is not really relevant, and
hard for the human eye to parse.

29.
To correct that…
We’ve designed Threat Stack to keep storage of events,
activity, and commands associated with a logged in user,
and automatically reconstruct this information into a
clean, compact, and readable timeline.

30.
Stay tuned for more engineering feats we are
accomplishing at Threat Stack…

31.
To learn more about Threat Stack
or to sign up, visit
threatstack.com