An overview of our group's work on teaching people not to fall for phishing attacks, using simulated phish. The summary is that simulated phish work surprisingly well, in terms of learning and retention.

1. Jason Hong
Carnegie Mellon University
Wombat Security Technologies
Teaching Johnny
Not to Fall for Phish

2. What are Effective Ways of Teaching
People not to Fall for Phish?

3. PhishGuru Embedded Training
• Use embedded training to teach people how
to avoid phishing in regular use of email
– People get simulated phishing email from good guys
– Teach how to protect self in engaging format
– Applies learning science for training
• Motivating users – “teachable moment”
• Started as research at CMU, product
by Wombat Security Technologies

4. Subject: Revision to Your Amazon.com InformationSubject: Revision to Your Amazon.com Information

5. Subject: Revision to Your Amazon.com InformationSubject: Revision to Your Amazon.com Information
Please login and enter your informationPlease login and enter your information
http://www.amazon.com/exec/obidos/sign-in.htmlhttp://www.amazon.com/exec/obidos/sign-in.html

7. Tells people why they are
seeing this message, uses
engaging character
Tells people why they are
seeing this message, uses
engaging character

8. Tells a story about what
happened and what the
risks are
Tells a story about what
happened and what the
risks are

9. Gives concrete examples of
how to protect oneself
Gives concrete examples of
how to protect oneself

10. Explains how criminals conduct
phishing attacks
Explains how criminals conduct
phishing attacks

12. Series of User Studies
Studies Results
Lab study I • Security notices are ineffective
• Users educated with PhishGuru made better decisions
Lab study II • Users in embedded condition retain and transfer
knowledge more effectively than other conditions even
after 7 days
Real-world
study I
• PhishGuru is effective in training people in the real world
• Trained participants retained knowledge after
7 days of training
Real-world
study II
• People trained with PhishGuru were less likely to click
on phishing links than those not trained
• People retained their training for 28 days
• Two training messages are better than one
• PhishGuru training does not make people less likely
to click on legitimate links

13. First lab study results
• Are security notices
effective?
– Ineffective for training
• Is embedded training
effective?
– Embedded training
condition made better
decisions than those
sent security notices
Kumaraguru, P., Rhee, Y., Acquisti, A., Cranor, L. F., Hong, J., and Nunge, E. Protecting people from
phishing: the design and evaluation of an embedded training email system. CHI ’07, pp. 905-
914.

14. Second lab study results
• Can people retain what they learned?
– Users educated with PhishGuru
retained knowledge after seven days
• Do people have to fall for phish?
– Users trained with embedded
did better than users trained
with non-embedded
Kumaraguru, P., Rhee, Y., Sheng, S., Hasan, S., Acquisti, A., Cranor, L. F., and Hong, J.
Getting users to pay attention to anti-phishing education: Evaluation of retention and
transfer. e-Crime Researchers Summit, Anti-Phishing Working Group (2007).

15. Real world study: Portuguese ISP
• Does PhishGuru training extend to real world?
– Did reduce rate of falling for phishing
– Trained participants retained knowledge
after 7 days of training
– Don’t have to train
all people in organization
Kumaraguru, P., Sheng, S., Acquisti, A., Cranor, L. F., and Hong, J. Lessons from a real
world evaluation of anti-phishing training. e-Crime Researchers Summit, 2008

16. Real world study: CMU
• Replicate previous study at larger scale
• Investigate retention after 1 week, 2 weeks,
and 4 weeks
• Compare effectiveness of 2 training
messages vs 1 training message
• Examine demographics and phishing
P. Kumaraguru, J. Cranshaw, A. Acquisti, L. Cranor, J. Hong, M. A. Blair, and T. Pham.
School of Phish: A Real-World Evaluation of Anti-Phishing Training. 2009. SOUPS 2009.

17. Study design
• Sent email to all CMU students, faculty and
staff to recruit participants (opt-in)
• 515 participants in three conditions
– Control / One training message / Two messages
• Emails sent over 28 day period
– 7 simulated spear-phishing messages
– 3 legitimate (cyber security scavenger hunt)
• Campus help desks and all spoofed
departments notified before messages sent

18. Our Simulated Spear Phish
URL is not hiddenURL is not hidden
Plain text email
without graphics
Plain text email
without graphics

19. Our Simulated Phishing Website
http://andrewwebmail.org/password/change.htm?ID=9009http://andrewwebmail.org/password/change.htm?ID=9009

20. Our Simulated Phishing Website
http://andrewwebmail.org/password/thankyou.html?ID=9009http://andrewwebmail.org/password/thankyou.html?ID=9009

21. Effect of PhishGuru Training
Condition N % who
clicked on
Day 0
% who
clicked on
Day 28
Control 172 52.3 44.2
Trained 343 48.4 24.5

22. Results conditioned on participants
who clicked on day 0
TestTest

23. Results conditioned on participants
who clicked on day 0
Trained
participants
less likely to
fall for phish
Trained
participants
less likely to
fall for phish
Trained
participants
remember
what they
learned 28
days later
Trained
participants
remember
what they
learned 28
days later
Test +
train
Test +
train
TestsTests TestsTests

24. Results conditioned on participants
who clicked on day 0 and day 14
Two-train participants less likely
than one-train participants to
click on days 16 and 21
Two-train participants less likely
than one-train participants to
click on days 16 and 21

25. Results conditioned on participants
who clicked on day 0 and day 14
Two-train participants less likely
than one-train participants to
click on days 16 and 21
Two-train participants less likely
than one-train participants to
click on days 16 and 21
Two-train participants less likely
than one-train participants to
provide information on day 28
Two-train participants less likely
than one-train participants to
provide information on day 28

26. Does PhishGuru Affect Clicking
on Legitimate Emails?
Condition N Day 0 Day 7 Day 28
Clicked % Clicked % Clicked
%
Control 90 50.0 41.1 38.9
One-train 89 39.3 42.7 32.3
Two-train 77 48.1 44.2 35.1
For Cyber Security Scavenger Hunt
No difference between the three
conditions on days 7 and 28
For Cyber Security Scavenger Hunt
No difference between the three
conditions on days 7 and 28

27. Students Most Vulnerable
• Students significantly more likely to fall for
phish than staff before training
• No significant differences based on
student year, department, or gender
• 18-25 age group most vulnerable
Age group Day 0 Day 28
18-25 62% 36%
26-35 48% 16%
36-45 33% 18%
45 and older 43% 10%

28. Most Participants Liked PhishGuru,
Wanted More
• 280 post study responses
• 80% recommended that CMU continue
PhishGuru training
– “I really liked the idea of sending CMU students fake
phishing emails and then saying to them, essentially,
HEY! You could've just gotten scammed! You should
be more careful - here's how....”
– “I think the idea of using something fun, like a
cartoon, to teach people about a serious subject is
awesome!”

29. Summary
• People trained with PhishGuru far less likely
to click on phishing links than not trained
• People retained training for 28 days
• Two training messages better than one
• PhishGuru training does not make people
less likely to click on legitimate links

30. For More Information
• Forthcoming SOUPS 2009 paper
• White paper on Wombat Security web site
• PhishGuru commercialized
by Wombat Security

31. Acknowledgments
• Supporting Trust Decisions group
• CyLab Usable Privacy and Security Lab
• CMU’s Information Security Office
• APWG
• Supported by National
Science Foundation,
Army Research Office,
CyLab, ISP in Portugal

33. Study schedule
Day of the
study
Control One training
message
Two training
messages
Day 0 Test and real Train and real Train and
real
Day 2 Test
Day 7 Test and real
Day 14 Test Test Train
Day 16 Test
Day 21 Test
Day 28 Test and real
Day 35 Post-study survey

34. Why is Teaching People Hard?

35. Why is Teaching People Hard?
• Problems
– Existing materials good, but could be better
• Not many opportunities for testing skills
– Most people don’t proactively look for
security training materials
– “Security notice” emails tend to be ignored
• Too much to read
• People don’t consider them relevant

36. Legitimate emails
No difference between the three
conditions on day 0, 7, and 28
No difference between the three
conditions on day 0, 7, and 28
No difference within the three
conditions for the three emails
No difference within the three
conditions for the three emails
Condition N Day 0 Day 7 Day 28
Clicked % Clicked % Clicked
%
Control 90 50.0 41.1 38.9
One-train 89 39.3 42.7 32.3
Two-train 77 48.1 44.2 35.1