Measuring the effectiveness of any security activity is widely discussed – security leaders debate the topic with a religious fervor rivaling that of any other hot button issue. Virtually every organization has some sort of application security training effort, but data on training effectiveness remains scarce. Last year our research team delivered the first-ever survey that captured developer awareness of secure coding concepts and the impact of formal application security training on a developer’s ability to write secure code. We learned that most software developer were aware of certain application security concepts, yet when asked how to write more secure code, they faired poorly.
This year’s 600-developer survey provides more quantitative data on what software developers understand about application security, both concepts and practices. It dives most deeply into awareness of defensive coding practices, which most developers largely did not grasp in the 2013 survey. It also is separates respondents by roles, so we can better understand how architects, developers, and QA staff grasp key application security concepts and put them to work. It better captures how software developers learn in general, so one can tailor any security training effort to how software developers, in practice, actually learn. This information will provide data to application security managers responsible for corporate security training that should allow them them to make more fact-based decisions about security training.
AppSec Survey 2.0 Fine-Tuning an AppSec Training Program Based on Data
1. AppSec USA 2014
Denver, Colorado
AppSec Survey 2.0: Fine-Tuning an
AppSec Training Program Based on
Data
John B. Dickson, CISSP
@johnbdickson
September 18, 2014
2. Introduction
John
B.
Dickson,
CISSP
• Application Security Enthusiast
• Ex-AF Guy & ISSA Distinguished Fellow
• Serial Entrepreneur & MBA Type
• Dad
3. When Not Thinking about AppSec…
I am Snake Hunting on a Ranch in South Texas
5. • Background
• Premise
• AppSec
Study
1.0
Results
–
What
We
Learned
• Approach
and
Survey
ParKcipants
• Key
Results
• What
We
Can
Put
To
Work
• Conclusions
and
QuesKons
&
Answers
Overview
6. AppSec Study 1.0 Results
• Things
we
Knew
Last
Year
• Key
Findings
of
Last
Year’s
Study
• AddiKonal
Stuff
We
Learned
Along
the
Way
• Development
training
is
hard
• Results
are
rarely
measured
for
ROI
• Training
is
typically
part
of
any
AppSec
program
7. AppSec Study 1.0 Results
• Things
we
Knew
Last
Year
• Key
Findings
of
Last
Year’s
Study
• AddiKonal
Stuff
We
Learned
Long
the
Way
• 25%
retenKon
aXer
training
• QA
did
worse
than
architects
and
soXware
developers
• Respondents
answered
basic
awareness
quesKons
but
not
coding
pracKces
8. • Things
we
Knew
Last
Year
• Key
Findings
of
Last
Year’s
Study
• AddiConal
Stuff
We
Learned
Long
the
Way
• SoXware
developers
learn
differently
than
companies
teach
• IncenKves
ma[er
• Surveys
are
hard!
AppSec Study 1.0 Results
9. Overview of 2014 “2.0” Study
• 600
respondents
• Represents
mulKple
industries
• Asked
the
same
applicaKon
security
quesKons
as
2013
survey
• Expanded
to
include
training
method
quesKons
• No
“before”
and
“aXer”
analysis
• No
classroom
training
opportuniKes
• Used
more
social
media
• Data
collecKon
ongoing
10. Approach and Survey Participants
Sample
QuesCons
QuesKons
that
tested
basic
knowledge
of
applicaKon
security:
• ApplicaKon
security
is
best
defined
as…
• Threat
Modeling
is…
• Input
ValidaKon
is…
11. Approach and Survey Participants
Sample
QuesCons
QuesKons
that
tested
understanding
of
defensive
coding:
• Marking
a
cookie
as
“secure”
will…
• Which
of
the
following
will
help
protect
against
XSS…
• Which
of
the
following
is
NOT
an
example
of
good
session
policy…
12. Approach and Survey Participants
Delivery
Means
• Direct
Delivery
of
Customized
Links
via
E-‐mail
• Survey
Monkey
paid
• Social
Media
– Facebook
– Linkedin
Targets
• SoXware
Developers
• Architects
• Quality
Assurance
13. Demographic Questions Asked
• What
is
your
primary
job
funcKon?
• What
is
your
company's
size?
• How
many
years
of
soXware
development
experience
do
you
have?
• How
much
previous
applicaKon
security
training
have
you
received?
14. 2014 Study Demographics
How
many
years
of
soMware
development
experience
do
you
Less
than
a
Year
18%
1-‐2
Years
9%
2-‐4
Years
10%
4-‐7
Years
13%
More
than
12
7-‐12
Years
16%
Years
34%
have?
15. 2014 Study Demographics
What
is
your
primary
job
Other
35%
Quality
Assurance
6%
SoXware
Developer
53%
funcCon?
Architect
6%
16. 2014 Study Demographics
What
is
your
company
size?
8%
8%
29%
8%
10%
37%
1-‐24
Employees
25-‐99
Employees
100-‐499
Employees
500-‐2499
Employees
2500-‐9999
Employees
10,000
or
more
Employees
17. 2014 Study Demographics
How
much
previous
applicaCon
security
training
experience
have
None
31%
Less
than
a
Day
19%
More
than
3
At
least
1
day,
but
less
than
2
days
17%
At
least
2
days,
but
less
than
3
days
8%
days
25%
you
received?
18. Key Survey Results
• Data
shows
soXware
developers
posiKvely
answer
quesKons
about
applicaKon
security
56%
of
the
Kme
• 2013
Denim
Group
study
results:
58%
• 2014
Aspect
Study:
60%
19. Change Implementation
Did
your
organizaCon
implement
any
SDLC
or
process
improvement
steps
to
formalize
concepts
learned
in
training?
Yes
33%
No
25%
I
don't
know
42%
20. Types of Training Received
Types
of
Training
Received
0
50
100
150
200
250
Other
Wri[en
Materials
1-‐on-‐1
Coaching
Webinars
or
Videos
Websites
Crowdsourcing
Sites
Developer
E-‐mail
Lists
or
RSS
feeds
Social
Learning
Plaqorms
Social
Media
e-‐Learning,
CBT
Instructor-‐Led
PresentaKons
21. E-Learning & Instructor-Led Training
Types
of
Training
Received
0
100
200
300
Other
Wri[en
Materials
1-‐on-‐1
Coaching
Webinars
or
Videos
Websites
Crowdsourcing
Sites
E-‐Learning
&
Instructor-‐led
Training
are
SKll
the
Primary
ApplicaKon
Security
Training
Approach
Developer
E-‐mail
Lists
or
RSS
feeds
Social
Learning
Plaqorms
Social
Media
e-‐Learning,
CBT
Instructor-‐Led
PresentaKons
22. Perceived Effectiveness of Training
0
50
100
150
200
250
300
350
400
450
500
Wri[en
Materials
1-‐on-‐1
Coaching
Webinars
or
Videos
Websites
Crowdsourcing
Sites
Developer
E-‐mail
Lists
or
RSS
feeds
Social
Learning
Plaqorms
Social
Media
e-‐Learning,
CBT
Instructor-‐Led
PresentaKons
1:
Not
EffecKve
2:
Somewhat
EffecKve
3:
Very
EffecKve
23. Question Types
Respondents
Fared
Far
Worse
on
QuesKons
Involving
Secure
Coding
PracKces
versus
ApplicaKon
Security
Awareness
QuesKons
41%
59%
0%
10%
20%
30%
40%
50%
60%
70%
Awareness
QuesKons
PrescripKve
QuesKons
%
of
QuesKons
Answered
Correctly
24. Pass Rate by Job Function
Quality
Assurance
respondents
Fared
50%
worse
than
soXware
developers
and
architects
Average
Pass
Rate
25%
20%
15%
10%
5%
0%
Other
SoXware
Developer
Quality
Assurance
Architect
70%
or
more
quesKons
answered
correctly
25. Pass Rate by Previous Training
The
Pass
Rate
More
Than
Doubled
for
Respondents
Who
Had
More
Than
Three
Days
ApplicaKon
Security
Training
Average
Pass
Rate
30%
25%
20%
15%
10%
5%
0%
Less
than
a
Day
or
None
At
least
1
day,
but
less
than
3
days
More
than
3
days
70%
or
more
correct
26. Pass Rate by Job Function: Security
Respondents
that
worked
for
security
organizaKons
or
vendors
DID
fare
well
compared
to
other
respondents
Average
Pass
Rate
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Security-‐Related
Everyone
Else
70%
or
more
quesKons
answered
correctly
27. What we Can Put to Work
• Refresher
training
is
criCcal
• Even
with
3+
days
of
appsec
training,
most
respondents
did
not
have
a
“passing”
grade
of
70%
• Like
any
other
training
topic,
leX
unreinforced,
what
learned
will
be
forgo[en
over
Kme
• ParKcularly
given
the
lack
of
SDLC
changes
• Likely
an
area
for
addiKonal
study
for
2015
appsec
training
study
28. What we Can Put to Work
• Training
without
SDLC
changes
likely
will
produce
the
same
results
• 33%
of
the
respondents
said
their
organizaKon
implemented
some
security
SDLC
improvements
• 67%
either
answered
“no”
or
“don’t
know”
• OrganizaKons
cannot
rely
exclusively
on
developers
retenKon
and
iniKaKve
to
produce
long-‐term
decline
in
applicaKon
vulnerabiliKes
29. What we Can Put to Work
• Augment
QA
with
Focused
AppSec
Training
• QA
has
consistently
responded
poorly
relaKve
to
developers
and
architects
• Many
organizaKon
put
their
most
junior
developers
in
QA
to
start
• QA
is
where
appsec
“lives”
in
many
organizaKons
• OrganizaKons
might
considering
“doubling
down”
on
appsec
training
for
QA
staff
to
compensate
for
this
fact
30. What we Can Put to Work
• IncenCves
Ma`er
When
Working
with
Developers
• We
used
incenKves
throughout
the
study
to
collect
responses
-‐
#Success!
• SoXware
developers
have
infinite
reasons
to
ignore
engagement
by
the
AppSec
team
• Rewards
help
nudge
soXware
developers
31. What we Can Put to Work
• Training
programs
must
be
tailored
to
be
effecCve
• Formal
programs
like
classroom
training
and
e-‐
Learning
are
sKll
the
bread
and
bu[er
of
appsec
training
programs
• ConsumpKon
rates
of
e-‐Learning
sKll
abysmal
without
incenKves
or
internal
markeKng
• Add
newer
ways
of
learning
to
reinforce
certain
key
points
and
to
serve
AppSec
corner
cases
• Leverage
current
events
to
reinforce
other
key
points
32. Conclusions
• Data
shows
soXware
developers
posiKvely
answer
quesKons
about
applicaKon
security
56%
of
the
Kme
• Data-‐driven
applicaKon
security
programs
will
likely
be
more
successful
and
chart
improvement
• SophisKcated
security
managers
use
incenKves
and
tailor
programs
to
improve
appsec
IQ
33. Questions and Answers
White
Paper?
MenCon
it
on
Twi`er
John
B.
Dickson,
CISSP
@johnbdickson
#appsecstudy