There is a serious misalignment of interests between Application Security vulnerability assessment vendors and their customers. Vendors are incentivized to report everything they possible can, even issues that rarely matter. On the other hand, customers just want the vulnerability reports that are likely to get them hacked. Every finding beyond that is a waste of time, money, and energy, which is precisely what’s happening every day.

1. Every. Little. Bit.
JEREMIAH GROSSMAN
ALL THESE VULNERABILITIES, RARELY MATTER
FOUNDER & CEO
U.S. BANK STRENGTH IN SECURITY (OCT 10, 2018)
@jeremiahg
https://www.jeremiahgrossman.com/
https://bitdiscovery.com/

2. BIO
▸20 years in InfoSec / AppSec
▸Professional Hacker
▸Founder of WhiteHat Security
▸Black Belt in Brazilian Jiu-Jitsu
WHO I AM…

3. THE PROBLEM I’M WORKING ON
YOU CAN’T SECURE WHAT YOU DON’T KNOW YOU OWN
Strange as it sounds, the vast majority of organizations with
more than a handful of websites do not know what they are,
what they do, or who is responsible for them. If a company
doesn't know what websites they own, they have little hope
of protecting their most important business assets.
An asset inventory is recommended by
every expert and ever industry standard.
ASSET INVENTORY

4. A complete portfolio of your company's websites.
Instantly created.
Automatically updated.

5. VULNERABILITY ASSESSMENT INDUSTRY
MISALIGNMENT OF INTERESTS
▸ Vendors are incentivized to report everything
they possible can, even issues that rarely matter.
▸ Customers just want the vulnerability reports that
are likely to get them hacked.
Every finding beyond that is a waste
of time, money, and energy.

6. VULNERABILITY LIKELIHOOD
(1 OR MORE)
70%!
56%!
47%!
29%! 26%! 24%!
16%! 15%! 11%! 11%! 8%! 6%! 6%! 6%! 5%!
0%!
10%!
20%!
30%!
40%!
50%!
60%!
70%!
80%!
90%!
100%!
I
n
s
u
f
fi
c
i
e
n
t
T
r
a
n
s
p
o
r
t
L
a
y
e
r
I
n
f
o
r
m
a
t
i
o
n
L
e
a
k
a
g
e
!
C
r
o
s
s
S
i
t
e
S
c
r
i
p
t
i
n
g
!
B
r
u
t
e
F
o
r
c
e
!
C
o
n
t
e
n
t
S
p
o
o
fi
n
g
!
C
r
o
s
s
S
i
t
e
R
e
q
u
e
s
t
F
o
r
g
e
r
y
!
U
R
L
R
e
d
i
r
e
c
t
o
r
A
b
u
s
e
!
P
r
e
d
i
c
t
a
b
l
e
R
e
s
o
u
r
c
e
L
o
c
a
t
i
o
n
!
S
e
s
s
i
o
n
F
i
x
a
t
i
o
n
!
I
n
s
u
f
fi
c
i
e
n
t
A
u
t
h
o
r
i
z
a
t
i
o
n
!
D
i
r
e
c
t
o
r
y
I
n
d
e
x
i
n
g
!
A
b
u
s
e
o
f
F
u
n
c
t
i
o
n
a
l
i
t
y
!
S
Q
L
I
n
j
e
c
t
i
o
n
!
I
n
s
u
f
fi
c
i
e
n
t
P
a
s
s
w
o
r
d
R
e
c
o
v
e
r
y
!
F
i
n
g
e
r
p
r
i
n
t
i
n
g
!
WHITEHAT’S WEBSITE SECURITY STATISTICS REPORT 2015

7. TOP 10 VULNERABILITY CATEGORIES
BY PROGRAMMING LANGUAGE
VERACODE: STATE OF SOFTWARE SECURITY REPORT VOL 6, FALL 2015

8. TRUSTWAVE GLOBAL SECURITY REPORT (2016)

9. 1,642,339,233
NETCRAFT: SEP 2018 WEB SERVER SURVEY

10. AVERAGE TIME-TO-FIX
(DAYS)
WHITEHAT’S WEBSITE SECURITY STATISTICS REPORT 2015
73!
97! 99! 108! 111!
130! 132! 136!
158! 160!
191! 192!
227!
0!
50!
100!
150!
200!
250!
T
r
a
n
s
p
o
r
t
a
t
i
o
n
!
A
r
t
s
&
E
n
t
e
r
t
a
i
n
m
e
n
t
!
A
c
c
o
m
m
o
d
a
t
i
o
n
!
P
r
o
f
e
s
s
i
o
n
a
l
&
S
c
i
e
n
t
i
fi
c
!
P
u
b
l
i
c
A
d
m
i
n
i
s
t
r
a
t
i
o
n
!
O
t
h
e
r
S
e
r
v
i
c
e
s
!
I
n
f
o
r
m
a
t
i
o
n
!
E
d
u
c
a
t
i
o
n
a
l
S
e
r
v
i
c
e
s
!
H
e
a
l
t
h
C
a
r
e
&
S
o
c
i
a
l
!
F
i
n
a
n
c
e
&
I
n
s
u
r
a
n
c
e
!
M
a
n
u
f
a
c
t
u
r
i
n
g
!
U
t
i
l
i
t
i
e
s
!
R
e
t
a
i
l
T
r
a
d
e
!

11. WINDOWS OF EXPOSURE
WHITEHAT’S WEBSITE SECURITY STATISTICS REPORT 2015
60%!
38%!
52%!
39%!
9%!
11%!
11%!
14%!
10%!
14%!
12%!
11%!
11%!
16%!
11%!
18%!
11%!
22%!
14%!
17%!
Retail Trade!
Information!
Health Care &!
Social Assistance!
Finance &!
Insurance!
Always Vulnerable!
Frequently Vulnerable (271-364 days a year)!
Regularly Vulnerable (151-270 days a year)!
Occasionally Vulnerable (31-150 days a year)!
Rarely Vulnerable (30 days or less a year)!

12. VERACODE: STATE OF SOFTWARE SECURITY REPORT VOL 6, FALL 2015
REMEDIATION RATES

13. WHY ALL THOSE ‘SERIOUS’
WEBSITE VULNERABILITIES
ARE NOT EXPLOITED?

14. PLAUSIBLE THEORIES
1.These ‘vulnerabilities’ are not really vulnerabilities in the
directly exploitable sense.
2.The vulnerabilities are too difficult for the majority of
attackers to find and exploit.
3.The vulnerabilities are only exploitable by insiders.
4.There aren’t enough attackers to exploit all or even most of
the vulnerabilities.
5.There are more attractive targets or exploit vectors for
attackers to focus on.
6.They are being exploited, but no one knows it (yet).

15. 9 OUT OF 10 TIMES, THE VENDOR WHO PRODUCES THE BEST
RESULTS IN TERMS OF HIGH-SEVERITY VULNERABILITIES WITH
LOW FALSE-POSITIVES WILL WIN THE DEAL. AS SUCH, EVERY
VENDOR IS HEAVILY INCENTIVIZED TO IDENTIFY AS MANY
VULNERABILITIES AS THEY CAN TO DEMONSTRATE THEIR SKILL
AND OVERALL VALUE.
Top vulnerability assessment vendors invest millions upon millions
of dollars each year in R&D to improve their scanning technology
and assessment methodology to uncover every possible issue.
WINNING A SALES BAKE-OFF

16. WHEN IT COMES TO DYNAMIC APPLICATION SECURITY
TESTING (DAST), SPECIFICALLY TESTING IN PRODUCTION,
THE WHOLE POINT IS TO FIND AND FIX VULNERABILITIES
BEFORE AN ATTACKER WILL FIND AND EXPLOIT THEM.
WHY DO WE DO DAST?
Technically, exploiting just 1 vulnerability for the
attacker to succeed.

17. IF ATTACKERS REALLY AREN’T FINDING, EXPLOITING, OR
EVEN CARING ABOUT THESE VULNERABILITIES AS WE CAN
INFER FROM THE SUPPLIED DATA — THE VALUE IN
DISCOVERING THEM, OR EVEN LOOKING, IN THE FIRST
PLACE BECOMES QUESTIONABLE.
If so, then all those vulnerabilities that DAST is finding rarely
matter much and we’re collectively wasting precious time and
resources focusing on them.
WHERE ARE ALL THE BREACH THAT COULD OR SHOULD BE HAPPENING?

18. THE PRIMARY PURPOSE OF STATIC APPLICATION SECURITY
TESTING (SAST) IS TO FIND VULNERABILITIES DURING THE
SOFTWARE DEVELOPMENT PROCESS BEFORE THEY LAND
IN PRODUCTION WHERE THEY’LL EVENTUALLY BE FOUND
BY DAST AND/OR EXPLOITED BY ATTACKERS.
WHY DO WE DO SAST?
What’s the overlap between SAST and DAST?

19. VULNERABILITY OVERLAP
BETWEEN THE ADVERSARY, DAST, AND SAST
VULNS
SAST
FINDS
Conceptually, SAST helps find them those issues earlier.
But, does it really? 5-15% of the vulnerabilities reported by
SAST are found by DAST.
VULNS
DAST
FINDS
VULNS
ADVERSARY
FINDS

20. THIS IS ALSO WHY CYBER-INSURANCE FIRMS FEEL
COMFORTABLE WRITING POLICIES ALL DAY LONG,
EVEN IF THEY KNOW FULL WELL THEIR CLIENTS ARE
TECHNICALLY RIDDLED WITH VULNERABILITIES,
BECAUSE STATISTICALLY THEY KNOW THOSE ISSUES
ARE UNLIKELY TO BE EXPLOITED OR LEAD TO CLAIMS.
WHAT THE CYBER-INSURANCE CARRIERS ALREADY KNOW
Exploitation of a vulnerability does not automatically result in a
‘breach,’ which does not necessarily equate to a ‘material
business loss,’ and loss is the only thing the business or their
insurance carrier truly cares about.

21. LESSONS LEARNED
▸We’re wasting huge amounts of time, money, and energy finding
and fixing vulnerabilities that rarely matter.
▸We need a better way to prioritize and justify remediation, or not,
of the vulnerabilities we already know exist and should care about.
▸We must more efficiently invest our resources in the application
security testing process.
LOOKING FORWARD

22. RISK MODELING
▸ Assumptions: SQL Injection vulnerability in a non-authenticated
portion of the application. A 50% likelihood of being exploited
over a year period. If exploitation results in a material breach, the
expected loss is $1,000,000 for incident handling and clean up.
▸$1,000,000 (expected loss) x 0.5 (probability of breach) =
$500,000 (risk)
▸If the vulnerability costs less than $500,000 to fix, then that’s the
reasonable choice. If remediation costs more than $500,000, then
leave it as is.
PROBABILITY (OF BREACH) X LOSS (EXPECTED) = RISK

23. RISK MODELING
▸$500,000 (expected loss) x 1% (probability of breach) = $5,000 (risk)
▸If vulnerability remediation costs less than $5,000, it makes sense to
fix it. If more, or far more, then one could argue it makes business
sense not to.
THE OTHER EXTREME

24. IF YOUR POSITION IS RECOMMENDING THAT THE
BUSINESS SHOULD FIX EACH AND EVERY
VULNERABILITY IMMEDIATELY REGARDLESS OF THE
COST, THEN YOU’RE REALLY NOT ON THE SIDE OF THE
BUSINESS AND YOU WILL CONTINUE BEING IGNORED.
PLEASE, DON’T BE THAT GUY

25. MODERN VULNERABILITY REMEDIATION DECISION-MAKING
This light is green, because in most places
where we put this light it makes sense to be
green, but we're not taking into account
anything about the current street’s situation,
location or traffic patterns.
Should you trust that light has your best interest
at heart? No.
Should you obey it anyway? Yes. Because once
you install something like that you end up
having to follow it, no matter how stupid it is.

26. REMEDIATION ALTERNATIVES
▸Web Application Firewalls (WAF)
▸Run-Time Application Security Protection (RASP)
ANYTHING TO LOWER THE COST AND DIFFICULT OF FIXING VULNERABILITIES

27. THE EDGE OF KNOWLEDGE
▸Matrix must take into account each vulnerability class,
assigns a likelihood of actual exploitation using whatever
available data, and contain an expected loss range.
▸Take into account the authentication status of the
vulnerability, mitigating controls, the industry, resident
data volume and type, insider vs external threat actor, etc.
INNOVATION IN VULNERABILITY REMEDIATION DECISION-MAKING

28. IF WE HAD A BETTER VULNERABILITY REMEDIATION DECISION-MAKING
▸We’ll know what types of vulnerabilities we care about
in terms of actual business risk and financial loss.
▸Investment can be prioritized to only look for those
and ignore all the other worthless junk.
▸Bulky vulnerability assessment reports would likely
dramatically decrease in size and increase in value.
SOLUTION TO THE LACK OF EFFICIENCY IN THE APPLICATION SECURITY TESTING PROCESS.

30. THANK YOU.
Jeremiah Grossman
@jeremiahg
https://www.facebook.com/jeremiahgrossman
https://www.linkedin.com/in/grossmanjeremiah
https://www.jeremiahgrossman.com/
http://blog.jeremiahgrossman.com/
https://bitdiscovery.com/