2. Who
am
I?
• Co-‐Founder
/
CSO
at
Signal
Sciences
• Built
and
led
the
Etsy
Security
Team
• Prior
to
that,
offensive
research
and
penetra%on
tes%ng
@
iSEC
Partners
3. About
this
talk
Real
world
approaches
to
web
applica%on
security
challenges
4. About
this
talk
Specifically,
techniques
that
are
simple
and
effec*ve
13.
Being
able
to
deploy
quick
is
our
#1
security
feature
14. Compared
to
We’ll
rush
that
security
fix.
It
will
go
out
…
in
about
6
weeks.
-‐
Former
vendor
at
Etsy
15. What
it
boils
down
to
(spoiler
alert)
• Make
things
safe
by
default
• Detect
risky
func%onality
/
Focus
your
efforts
• Automate
as
much
as
you
can
• Know
when
the
house
is
burning
down
17.
How
have
the
tradi%onal
defenses
for
XSS
worked
out?
19. Safe
by
default
• Problems?
– OZen
done
on
a
per-‐input
basis
• Easy
to
miss
an
input
or
output
– May
use
defenses
in
wrong
context
• Input
valida%on
paern
may
block
full
HTML
injec%on,
but
not
injec%ng
inside
JS
– May
put
defenses
on
the
client
side
in
JS
– Etc
…
These
problems
miss
the
point
20. Safe
by
default
• The
real
problem
is
that
it’s
hard
to
find
where
protec%ons
have
been
missed
• How
can
we
change
our
approach
to
make
it
simpler?
23. Safe
by
default
Encode
dangerous
HTML
characters
to
HTML
en%%es
at
the
very
start
of
your
framework
To
repeat…
Before
input
reaches
main
applica%on
code
24. Safe
by
default
On
the
surface
this
doesn’t
seem
like
much
of
a
change
25. Safe
by
default
Except,
we’ve
just
made
lots
of
XSS
problems
grep-‐able
26.
27. Safe
by
default
Now
we
look
for
a
small
number
of
paerns:
• HTML
en%ty
decoding
func%ons
or
explicit
string
replacements
• Data
in
formats
that
won’t
be
sani%zed
– Ex:
Base64
encoded,
double
URL
encoded,
etc
• Code
that
opts
out
of
plaeorm
protec%ons
28. Safe
by
default
Fundamentally
shiZs
us:
From:
“Where
is
my
app
missing
protec%ons?”
(hard)
To:
“Where
is
it
made
deliberately
unsafe?”
(easy)
29. Safe
by
default
Obviously
not
a
panacea
– DOM
based
XSS
– Javascript:
URLs
– Can
be
a
pain
during
interna%onaliza%on
efforts
31. Focus
your
efforts
• Con%nuous
deployment
means
code
ships
fast
• Things
will
go
out
the
door
before
security
team
knows
about
them
• How
can
we
detect
high
risk
func%onality?
32. Detect
risky
func%onality
• Know
when
sensi%ve
por%ons
of
the
codebase
have
been
modified
• Build
automa%c
change
aler%ng
on
the
codebase
– Iden%fy
sensi%ve
por%ons
of
the
codebase
– Create
automa%c
aler%ng
on
modifica%ons
33. Detect
risky
func%onality
• Doesn’t
have
to
be
complex
to
be
effec%ve
• Approach:
– sha1sum
sensi%ve
plaeorm
level
files
– Unit
tests
alert
if
hash
of
the
file
changes
– No%fies
security
team
on
changes,
drives
code
review
34. Detect
risky
func%onality
• At
the
plaeorm
level,
watching
for
changes
to
site-‐wide
sensi%ve
func%onality
– CSRF
defenses
– Session
management
– Encryp%on
wrappers
– Login/Authen%ca%on
– Etc
35. Detect
risky
func%onality
• At
the
feature
level,
watching
for
changes
to
specific
sensi%ve
methods
• Iden%fying
these
methods
is
part
of
ini%al
code
review/pen
test
of
new
features
36. Detect
risky
func%onality
• Watch
for
dangerous
func%ons
• Usual
candidates:
– File
system
opera%ons
– Process
execu%on/control
– Encryp%on
/
Hashing
– Etc
37. Detect
risky
func%onality
• Unit
tests
watch
codebase
for
dangerous
func%ons
– Split
into
separate
high
risk/low
risk
lists
• Alerts
are
emailed
to
the
appsec
team,
drive
code
reviews
38. Detect
risky
func%onality
• Find
out
about
unused
but
reachable
pages
• Any
files
s%ll
reachable
but
barely
requested
are
probably
old
or
“temporary”
code
– aka
a
goldmine
of
vulnerabili%es
39. Detect
risky
func%onality
1. Walk
DocumentRoot,
build
list
of
files
2. Compare
each
file
against
access
log
3. Alert
on
any
files
accessed
<
X
%mes
in
last
30
days
Iden%fied
files
are
worth
a
manual
review,
can
likely
be
removed
en%rely
40. Detect
risky
func%onality
• Monitor
applica%on
traffic
• Purpose
is
twofold:
– Detec%ng
risky
func%onality
that
was
missed
by
earlier
processes
– Groundwork
for
aack
detec%on
and
verifica%on
41. Detect
risky
func%onality
• Regex
incoming
requests
at
the
framework
– Sounds
like
performance
nightmare,
shockingly
isn’t
• Look
for
HTML/JS
in
request
– This
creates
a
huge
number
of
false
posi%ves
• That’s
by
design,
we
refine
the
search
later
42. Detect
risky
func%onality
• We
deliberately
want
to
cast
a
wide
net
to
see
HTML
entering
the
applica%on
• From
there,
build
a
baseline
of
HTML
– Entering
the
applica%on
in
aggregate
– Received
by
specific
endpoints
43. Detect
risky
func%onality
What
to
watch
for:
– Did
a
new
endpoint
suddenly
show
up?
• A
new
risky
feature
might’ve
just
shipped
– Did
the
amount
of
traffic
containing
HTML
just
significantly
go
up?
• Worth
inves%ga%ng
46. Automate
as
much
as
you
can
• Automate
finding
simple
issues
to
free
up
resources
for
more
complex
tasks
• Use
aacker
traffic
to
automa%cally
drive
tes%ng
• We
call
it
A<ack
Driven
Tes@ng
47. Automate
as
much
as
you
can
• Some
cases
where
this
is
useful:
– Applica%on
faults
– Reflected
XSS
– SQLi
48. Automate
as
much
as
you
can
• Applica%on
faults
(HTTP
5xx
errors)
• As
an
aacker,
these
are
one
of
the
first
signs
of
weakness
in
an
app
– As
a
defender,
pay
aen%on
to
them!
49. Automate
as
much
as
you
can
• Just
watching
for
5xx
errors
results
in
a
lot
of
ephemeral
issues
that
don’t
reproduce
• Instead:
– Grab
last
X
hours
worth
of
5xx
errors
from
access
logs
– Replay
the
original
request
– Alert
on
any
requests
which
s%ll
return
a
5xx
50. Automate
as
much
as
you
can
• Cron
this
script
to
run
every
few
hours
• If
a
request
s%ll
triggers
an
applica%on
fault
hours
later,
it’s
worth
inves%ga%ng
51. Automate
as
much
as
you
can
• Similar
methodology
for
verifying
reflected
XSS
• For
reflected
XSS
we:
– Iden%fy
requests
containing
basic
XSS
payloads
– Replay
the
request
– Alert
if
the
XSS
payload
executed
52. Automate
as
much
as
you
can
• Basic
payloads
commonly
used
in
tes%ng
for
XSS:
– alert()
– document.write()
– unescape()
– String.fromCharCode()
– etc
53. Automate
as
much
as
you
can
We
created
a
tool
to
use
NodeJS
as
a
headless
browser
for
verifica%on
54. Automate
as
much
as
you
can
Test
webserver
1.
Fetch
URL
containing
poten%al
XSS
55. Automate
as
much
as
you
can
Test
webserver
2.
Page
contents
returned
to
a
temp
buffer,
not
interpreted
yet
56. Automate
as
much
as
you
can
Test
webserver
3.
Inject
our
instrumented
JS
into
page
contents
+
Our
JS
Page
contents
57. Automate
as
much
as
you
can
Test
webserver
4.
Combina%on
of
instrumented
JS
+
page
contents
interpreted
+
Our
JS
Page
contents
58. Automate
as
much
as
you
can
Test
webserver
5.
If
instrumented
JS
is
executed,
alert
appsec
team
for
review
59. Automate
as
much
as
you
can
• Sample
instrumented
JS:
(function() {
var proxiedAlert = window.alert;
window.alert = function() {
location="XSSDETECTED";
};
})();
60. Automate
as
much
as
you
can
• Open
sourced
NodeJS
tool
– hps://github.com/zanelackey/projects
• Combine
this
approach
with
driving
a
browser
via
Wa%r/Selenium
– Make
sure
to
use
all
major
browsers
66. Know
when
the
house
is
burning
down
• Methodology:
– Instrument
applica%on
to
collect
data
points
– Fire
them
off
to
an
aggrega%on
backend
– Build
individual
graphs
– Combine
groups
of
graphs
into
dashboards
• We’ve
open
sourced
our
instrumenta%on
library
– hps://github.com/etsy/statsd
71. Know
when
the
house
is
burning
down
Now
we
can
visually
spot
aacks
72. Know
when
the
house
is
burning
down
But
who’s
watching
at
4AM?
73. Know
when
the
house
is
burning
down
• In
addi%on
to
data
visualiza%ons,
we
need
automa%c
aler%ng
• Look
at
the
raw
data
to
see
if
it
exceeds
certain
thresholds
• Works
well
for
graphs
like
this…
77. Know
when
the
house
is
burning
down
• We
need
to
smooth
out
graphs
that
follow
usage
paerns
• Use
exponen%al
smoothing
formulas
like
Holt-‐
Winters
• Math
is
hard,
let’s
look
at
screenshots!
79. Know
when
the
house
is
burning
down
• Now
that
we’ve
smoothed
out
the
graphs…
• Use
the
same
approach
as
before:
– Grab
the
raw
data
– Look
for
values
above/below
a
set
threshold
– Alert
80. Know
when
the
house
is
burning
down
Have
the
ability
to
quickly/easily
correlate
events
81. Know
when
the
house
is
burning
down
• Global
Request
IDs
<?php
global
$request_uuid;
apache_note(’request_uuid',
$request_uuid);
82. Know
when
the
house
is
burning
down
[01/Aug/2012:16:37:41
+0000]
"GET
/members/twokb/payments
HTTP/1.1"
200
"hps://XXX/members/twokb"
"Mozilla/5.0
(Windows
NT
6.1;
WOW64)
AppleWebKit/536.11
(KHTML,
like
Gecko)
Chrome/
20.0.1132.57
Safari/536.11"
MF9JqDVpY93VOMreyvI2UC24wRjT
[Wed
Aug
01
16:37:41
2012]
[MF9JqDVpY93VOMreyvI2UC24wRjT]
[info]
[XXX]
[kbarry]
about
to
call
shop_get_data
for
shop:
[5971709]
[Wed
Aug
01
16:37:41
2012]
[MF9JqDVpY93VOMreyvI2UC24wRjT]
[info]
[XXX_audit]
[kbarry]
ac%on="view_payments"
staff="kbarry"
user_id="5597626"
sec%on="payment_info"
83. Know
when
the
house
is
burning
down
Alert
on
events
that
(should)
never
happen
84. Know
when
the
house
is
burning
down
Successful
aacks
don’t
happen
in
a
vacuum!
They
generate
signals
85. Know
when
the
house
is
burning
down
1. Iden%fy
the
signals
associated
with
a
vulnerability
class
2. Alert
when
a
signal
occurs
3. Fix
the
iden%fied
weaknesses
86. Know
when
the
house
is
burning
down
Two
examples:
SQLi
and
code
execu%on
87. Know
when
the
house
is
burning
down
• The
road
to
exploited
SQLi
is
liered
with
broken
queries
1. Watch
the
logs
for
SQL
syntax
errors
2. Alert
when
they
appear
3. Fix
the
lack
of
valida%on
allowing
the
error
88. Know
when
the
house
is
burning
down
• Further
along
the
aack
process,
a
SQLi
aack
looks
like…
your
database
• Sensi%ve
DB
table
names
shouldn’t
be
showing
up
in
requests
– Alert
if
they
do!
• aka
the
“Two
hours
un%l
the
db
is
up
on
pastebin”
alert
89. Know
when
the
house
is
burning
down
A
funny
story
about
a
code
execu%on
vuln…
90. Know
when
the
house
is
burning
down
• preg_replace()
in
PHP
has
an
interes%ng
modifier
“e
(PREG_REPLACE_EVAL)
If
this
modifier
is
set,
preg_replace()
does
normal
subs%tu%on
of
backreferences
in
the
replacement
string,
evaluates
it
as
PHP
code,
and
uses
the
result
for
replacing
the
search
string.
“
91. Know
when
the
house
is
burning
down
• preg_replace()
in
PHP
has
an
interes%ng
modifier
“e
(PREG_REPLACE_EVAL)
If
this
modifier
is
set,
preg_replace()
does
normal
subs%tu%on
of
backreferences
in
the
replacement
string,
evaluates
it
as
PHP
code,
and
uses
the
result
for
replacing
the
search
string.”
92. Know
when
the
house
is
burning
down
What
do
the
signals
for
this
look
like?
100. References
/
Thanks
• DevOpsSec:
hp://www.slideshare.net/nickgsuperstar/
devopssec-‐apply-‐devops-‐principles-‐to-‐security
• Special
Thanks:
– Nick
Galbreath,
Dan
Kaminsky,
Marcus
Barczak