"Using the CGC's Fully Automated Vulnerability Detection Tools in Security Evaluation and Its Effectiveness - Are Tools Good for Hackers Good for Security Evaluators? -" @ CODE BLUE 2016, Tokyo, Japan (October 20, 2016)

1. Inhyuk Seo(inhack), Jisoo Park(J.Sus), Seungjoo Kim
SANE(Security Analysis aNd Evaluation) Lab
Korea University(高麗大學校)
Using the CGC’s fully automated
vulnerability detection tools in
security evaluation and its
effectiveness
Are tools good for hackers good for security evaluators?

2. Contents
• Who are we?
• Introduction
• Security Engineering, the Way to Information Assurance
• High-Assurance, the Key of CPS
• Tools for Security Testing & Evaluation
- Tools for Design Assurance / Tools for Code Assurance
• Demo (Design / Code)
• Conclusion
• Acknowledgement
• Q&A
• Reference
2 / 76

3. Who are we?
Inhyuk Seo (徐寅赫)
E-mail : jisoo8881@korea.ac.kr
Jisoo Park received his B.S (2015) in Computer Science Engineering from Dongguk University in
Korea. He worked at antivirus company Ahnlab as S/W QA trainee for 6 month. Also he
completed high-quality information security education course “Best of the Best” hosted by
KITRI(Korea Information Technology Research Institute). Now, He is a M.S course student at CIST
SANE Lab, Korea University and interested in Common Criteria, Security Engineering(Especially
Threat modeling).
Jisoo Park (朴志洙)
E-mail : inhack@korea.ac.kr
My name is Inhyuk Seo(Nick: inhack). I graduated B.S. in Computer Science and Engineering at
Hanyang University(ERICA) in 2015. Now I’m a researcher and M.S. of SANE(Security Analaysis aNd
Evaluation) Lab at Korea University. In 2012, I completed high-quality information security
education course “the Best of the Best(BoB)” hosted by KITRI(Korea Information Technology
Research Institute) and participated in many projects related with vulnerability analysis. I’m
interested in Programming Language, Software Testing, Machine Learning, Artificial Intelligence.
3 / 76

4. Seungjoo Gabriel Kim (金昇柱)
E-mail: skim71@korea.ac.kr
Homepage : www.kimlab.net
Facebook, Twitter : @skim71
Prof. Seungjoo Gabriel Kim received his B.S, M.S and Ph.D. from Sungkyunkwan University(SKKU)
of Korea, in 1994, 1996, and 1999, respectively. Prior to joining the faculty at Korea University (KU)
in 2011, he served as Assistant & Associate Professor at SKKU for 7 years. Before that, he served
as Director of the Cryptographic Technology Team and the (CC-based) IT Security Evaluation Team
of the Korea Internet & Security Agency(KISA) for 5 years. He is currently a Professor in the
Graduate School of Information Security Technologies(CIST). Also, He is a Founder and Advisory
director of hacker group, HARU and an international security & hacking conference, SECUINSIDE.
Prof. Seungjoo Gabriel Kim’s research interests are mainly on cryptography, Cyber Physical Security,
IoT Security, and HCI Security. He is a corresponding author.
Who are we?
4 / 76

5. Intro
Level of trust that it really does!
Assurance
The User’s degree of trust
in that information
Information
Assurance
5 / 76

6. Intro
Rise of the Information Assurance
Gulf War has often
been called the first
information war.
“The harbinger of IA”
1991
U.S. DoD Directive
5-3600.1 :
The first standardized
definition of IA
1996
Information Security
(INFOSEC) Era
1980 ~
“The communication network that supported Operation Desert Storm was the largest joint
theater system ever established. It was built in record time and maintained a phenomenal 98
percent availability rate. At the height of the operation, the system supported 700,000
telephone calls and 152,000 messages per day. More than 30,000 radio frequencies were
managed to provide the necessary connectivity and to ensure minimum interference.”
Debra S. Herrmann, “Security Engineering and Information Assurance”
6 / 76

7. Intro
Information Assurance
“Measures that protect and defend information and information systems by
ensuring their availability, integrity, authentication, confidentiality, and non-
repudiation. This includes providing for restoration of information systems by
incorporating protection, detection, and reaction capabilities.”
DoD Directive 8500.01E
Rise of the Information Assurance
Gulf War has often
been called the first
information war.
“The harbinger of IA”
1991
U.S. DoD Directive
5-3600.1 :
The first standardized
definition of IA
1996
Information Security
(INFOSEC) Era
1980 ~
7 / 76

8. What are the differences between
Information Security and Information Assurance?
Intro
8 / 76

9. Intro
Information Security (情報保護) Information Assurance (情報保證)
Dates Since 1980s Since 1998
Subject of protection Information and Information system Business as a whole
Goal Confidentiality, Integrity, Availability
Confidentiality, Integrity, Availability, Non-
repudiation, Accountability, Auditability,
Transparency, Cost-effectiveness, Efficiency
Type of information Primarily electronic All types
Approach
Domination of the technical approach, initial
attempts to consider soft aspects
All-encompassing multi-disciplinary
systematic approach
Security Mechanism
Primary focus is on technical security
mechanism; initial consideration of
organizational and human-oriented
mechanism
All available
(technical, organizational, human-oriented,
legal)
Role within a business
Supporting system, often inducing some
restrictions on business
An integral aspect of business, business
enabler
Flow of security
decision
Bottom-Top Top-Bottom
9 / 76

10. Intro
Information Security (情報保護) Information Assurance (情報保證)
Dates Since 1980s Since 1998
Subject of protection Information and Information system Business as a whole
Goal Confidentiality, Integrity, Availability
Confidentiality, Integrity, Availability, Non-
repudiation, Accountability, Auditability,
Transparency, Cost-effectiveness, Efficiency
Type of information Primarily electronic All types
Approach
Domination of the technical approach, initial
attempts to consider soft aspects
All-encompassing multi-disciplinary
systematic approach
Security Mechanism
Primary focus is on technical security
mechanism; initial consideration of
organizational and human-oriented
mechanism
All available
(technical, organizational, human-oriented,
legal)
Role within a business
Supporting system, often inducing some
restrictions on business
An integral aspect of business, business
enabler
Flow of security
decision
Bottom-Top Top-Bottom
Protecting information and information
systems from unauthorized access, use,
disclosure, disruption, modification, or
destruction
Validating that the information is
authentic, trustworthy, and
accessible
10 / 76

11. Security Engineering,
the Way to Information Assurance

12. What is Information Assurance’s Goal?
Security Engineering
12 / 76

13. Security Engineering
Goal of Information Assurance
Trustworthiness
(Dependability)
The ability of the system
to deliver services when
requested
Availability
The ability of the system
to deliver services as
specified
Reliability
The ability of the system
to protect itself against
accidental or deliberate
intrusion
The ability of the system
to operate without
catastrophic failure
Safety Security
Reflect the extent of the user’s confidence that
it will operate as users expects that it will not ‘fail’ in normal use
13 / 76

14. Security Engineering
Goal of Information Assurance
Safety Security
Availability Reliability
Each Element is not independent.
They interact with each other
<For example>
Adding State-of-the-art technology
(Availability, Reliability) to
Smart Devices may affect Security
features of the devices
Must considered all of them during whole System Life cycle
(From Requirements to Operation)
14 / 76

15. Domain Reliability Security Safety
Financial System Medium High No
DB of Medical Records Medium Medium Medium
Air Traffic Control System Medium High High
Automobile High Medium High
Defcon 23 – Charlie Miller & Chris Valasek “Remote
Exploitation of an Unaltered Passenger Vehicle”
It was ‘Low’ at first,
Security Engineering
Goal of Information Assurance
15 / 76

16. How can we achieve Information Assurance?
Security Engineering
16 / 76

17. How can we achieve Information Assurance?
Security Engineering
Security Engineering
17 / 76

18. Security Engineering is about building systems to remain dependable in the
face of malice, error and mischance. As a discipline, it focuses on the tools,
needed to design, implement and test complete systems and to adapt
existing systems as their environment evolves.
– Ross Anderson, Computer Laboratory in University of Cambridge -
What is Security Engineering?
Security Engineering
18 / 76

19. Requirement
(Policy)
Assurance
Mechanisms
Requirements(Policy) Assurance
Design Assurance
Implementation Assurance
Operational Assurance
Assurance needed at all stage of
System life cycle
Ultimate Goal of Security Engineering
Security Engineering
What is Security Engineering?
19 / 76

20. Requirements Design Implementation Release Maintenance
System Engineering Life Cycle Process (ISO/IEC/IEEE 15288 : 2015)
• Business or
Mission Analysis
• Stakeholder Needs and
Requirements Definitions
• System Requirements
Definition
• Architecture
Definition
• Design Definition • System Analysis • Implementation • Integration
• Verification • Transition • Validation • Operation
• Maintenance • Disposal
Security Engineering
What is Security Engineering?
Security Engineering throughout the Life Cycle
(ISO, Common Criteria, C&A, CMVP.. etc)
20 / 76

21. Case Study : Microsoft Security Development Life Cycle
Security Engineering
21 / 76

22. Case Study : Microsoft Security Development Life Cycle
Does it really work?
34
3
187
SQL Server 2000 SQL Server 2005 Competing
commercial DB
Total Vulnerabilities Disclosed
36 Month after Release
46%
reductio
n
119
66
400
242
157
Windows
XP
Windows
Vista
OS A OS B OS C
Total Vulnerabilities
Disclosed On year after Release
46%
reduction
After SDLBefore SDL After SDLBefore SDL
91%
reduction
Analysis by Jeff Jones(Microsoft technet security blogWindows Vista One year Vulnerability Report, Microsoft Security Blog 23 Jan 2008
Security Engineering
22 / 76

23. High-Assurance, the Key of CPS

24. High Assurance, the Key of CPS
What is “High-Assurance”(High-level of Trust)?
High-Assurance means that it can be mathematically
proven that the system works precisely as intended and
designed.
and High-Assurance development means that
there are clear and compelling evidences in each
development phase.
24 / 76

25. What is “CPS”?
Cyber Physical Systems(CPS) are co-
engineered interacting network of
physical and computational
components. CPS will provide the
foundation of our critical infrastructure,
form the basis of emerging and future
smart services, and improve our quality
of life in many areas.
Internet of Things Cyber Physical System
AssuranceSecurity VS
High Assurance, the Key of CPS
25 / 76

26. Where “High-Assurance” needed
Information
Assurance
Security
Engineering
Critical Infrastructure
Finance
Aviation
Government
Medical
Automotive
Railway
Energy
.
.
High-
Assurance
Apply & Guarantee
High Assurance, the Key of CPS
26 / 76

27. ISO/IEC 29128 and ISO/IEC 15408 have “Reliability” and “Security”
ISO 26262, DO-254 : Mainly focusing on “Safety” and “Reliability”
Standard / Regulation Assurance Level
ISO 26262 ASIL A ASIL B ASIL C ASIL D
DO-254 DAL E DAL D DAL C DAL B DAL A
ISO/IEC 29128 PAL 1 PAL 2 PAL 3 PAL 4
ISO/IEC 15408 EAL1 EAL 2 EAL 3 EAL 4 EAL 5 EAL 6 EAL 7
HighLow
High Assurance, the Key of CPS
27 / 76

28. ISO/IEC 29128 and ISO/IEC 15408 have “Reliability” and “Security”
ISO 26262, DO-254 : Mainly focusing on “Safety” and “Reliability”
Standard / Regulation Assurance Level
ISO/IEC 29128 PAL 1 PAL 2 PAL 3 PAL 4
ISO/IEC 15408 EAL1 EAL 2 EAL 3 EAL 4 EAL 5 EAL 6 EAL 7
HighLow
High Assurance, the Key of CPS
28 / 76

29. Example : ISO/IEC 29128
Verification of Cryptographic Protocol
Protocol
Assurance Level
PAL1 PAL2 PAL3 PAL4
Protocol
Specification Semiformal
description of
protocol specification
Formal description
of protocol
specification
Formal description of protocol specification
in a tool-specific specification language,
whose semantics is mathematically defined
Adversarial Model
Security Property
Self-assessment
evidence
Informal argument or
mathematically formal
paper-and-pencil
proof that the
cryptographic
protocol satisfies the
given objectives and
properties with
respect to the
adversarial model
Tool-aided bounded
verification that the
specification of the
cryptographic
protocol satisfies the
given objectives and
properties with
respect to the
adversarial model
Tool-aided
unbounded
verification that the
specification of the
cryptographic
protocol satisfies the
given objectives and
properties with
respect to the
adversarial model
Tool-aided
unbounded
verification that the
specification of the
cryptographic
protocol in its
adversarial model
achieves and satisfies
its objectives and
properties.
High Assurance, the Key of CPS
29 / 76

30. Example : Common Criteria ISO/IEC 15408
Evaluation criteria for IT security
Evaluation
Assurance Level
Description
EAL 7 Formally verified design and tested
EAL 6 Semiformally verified design and tested
EAL 5 Semiformally designed and tested
EAL 4 Methodically designed, tested, and reviewed
EAL 3 Methodically tested and checked
EAL 2 Structurally tested
EAL 1 Functionally tested
Gerwin Klein, Operating System Verification – An Overview
High Assurance, the Key of CPS
30 / 76

31. Example : Common Criteria ISO/IEC 15408
Corresponding assurance levels in ISO/IEC 29128
High Assurance, the Key of CPS
31 / 76

32. How to Get it?
• Measurable & Mathematically provable
à Formal Verification
• By using Automated Tools
High Assurance, the Key of CPS
32 / 76

33. How to Get it?
Established in March 2012, as a Research Association, which headquarters is located in Tagajo City of
Miyagi Prefecture. CSSC’s testbed is composed of 9-types of simulated plants and it is capable to organize
cybersecurity hands-on exercises which simulate cyber attack
Control System Security Center (CSSC)
Major operation plans – System security verification
High Assurance, the Key of CPS
(http://www.css-center.or.jp/pdf/cssc-activity_e.pdf)
33 / 76

34. How to Get it?
“The goal of the HACMS program is to create technology for the construction of high-assurance cyber-
physical systems, where high assurance is defined to mean functionally correct and satisfying
appropriate safety and security properties.”
Dr. Raymond Richards, Information Innovation Office
Program Manager of HACMS
High-Assurance Cyber Military System (HACMS)
High Assurance, the Key of CPS
(http://www.darpa.mil/program/high-assurance-cyber-military-systems)
34 / 76

35. Automated Tools for
Security Testing & Evaluation

36. Tools for Security Testing & Evaluation
Automation Tools for Hacker & Bug Hunters
• Automation Vulnerability Detection Tools developed by
hacker/bug hunter are only for the purpose of finding 0-day
(Unknown Vulnerability) easily.
Automation Tools for Evaluation
Ultimate goal of Security testing & evaluation
There are no mistakes in security testing process and
Guarantee objective analysis reports or evaluation results
Independent from evaluator’s capability or expertise. So anyone
who uses the same tools should be able to make same results.
36 / 76

37. What should we consider when we choose
Automated security testing tools in evaluation?
Tools for Security Testing & Evaluation
37 / 76

38. Assessment Features for Automated Tools
User-Friendly Effectiveness Scalability
Tools for Security Testing & Evaluation
38 / 76

39. Tools for
Design Assurance

40. Tools for Design Assurance
Assessment items to choose Automated Tools for
Design Assurance
(1) User-Friendly
• Usability
• Analysis Report
• Requirement to Evaluator (Expertise, Background Knowledge)
(2) Effectiveness
• Automation Level
• Model Description Method
• Licensing & Cost
(3) Scalability
• Supported Platforms
40 / 76

41. Cryptographic Protocol
Model Checking
Theorem Proving Based
• NRL
• FDR
• SCYTHER
• ProVerif • AVISPA(TA4SP)
• CryptoVerif • EBMC
…….
• Isabelle/HOL
• BPW
• Game-based Security Proof
• VAMPIRE • …….
Tools for Design Assurance
41 / 76

42. Tools for Design Assurance
Cryptographic Protocol (Model Checking)
• The Maude NRL Protocol Analyzer (Maude-NPA)
Assessment Items Description
Usability GUI(Graphic User Interface)
Analysis Report O
Requirement to Evaluator Protocol Design & Modeling Ability
Automation Level Interactive
Model Description Method
Maude-PSL (Maude Protocol Specification
Language)
Licensing & Cost Non-Commercial (University of Illinois)
Supported Platform Mac OS X
42 / 76

43. Cryptographic Protocol (Model Checking)
• FDR(Failure-Divergence-Refinement)
Assessment Items Description
Usability GUI
Analysis Report O
Requirement to Evaluator Protocol Design & Modeling Ability
Automation Level Interactive
Model Description Method Formal Language (CSP)
Licensing & Cost Non-Commercial (University of Oxford)
Supported Platform Linux / Mac OS X
Tools for Design Assurance
43 / 76

44. Cryptographic Protocol (Model Checking)
• Scyther
Assessment Items Description
Usability GUI
Analysis Report O
Requirement to Evaluator Protocol Design & Modeling Ability
Automation Level Interactive
Model Description Method SPDL (Standard Page Description Language)
Licensing & Cost Non-Commercial (University of Oxford)
Supported Platform Linux / Windows / Mac OS X
Tools for Design Assurance
44 / 76

45. Cryptographic Protocol (Model Checking)
• ProVerif
Assessment Items Description
Usability CLI (but Easy to Use)
Analysis Report O
Requirement to Evaluator Protocol Design & Modeling Ability
Automation Level Interactive
Model Description Method PV Script (ProVerif Script)
Licensing & Cost Non-Commercial (PROSECCO)
Supported Platform Linux / Windows / Mac OS X
Tools for Design Assurance
45 / 76

46. Cryptographic Protocol (Theorem Proving)
• Isabelle/HOL(Higher-Order Logic)
Assessment Items Description
Usability GUI, IDE(Integrated Development Environment)
Analysis Report O
Requirement to Evaluator Protocol Design & Modeling Ability
Automation Level Interactive
Model Description Method Functional & Logic Language (HOL)
Licensing & Cost Non-Commercial (University of Cambridge)
Supported Platform Linux / Windows / Mac OS X
Tools for Design Assurance
46 / 76

47. Tools for
Code Assurance

48. Tools for Code Assurance
Assessment Items to choose Automated Tools for Code
Assurance
(1) User-Friendly
• Usability
• Analysis Report
• Requirement to Evaluator (Expertise, Background Knowledge)
(2) Effectiveness
• Automation Level
• Analysis Method
• Detectable Vulnerability Type
• Code Coverage
• Licensing & Cost
(3) Scalability
• Supported Languages
• Supported Platforms
48 / 76

49. CGC(Cyber Grand Challenge) Finalist
• Mayhem CRS (ForAllSecure)
• Xandra (TECHx)
• Mechanical Phish (Shellphish)
• Rebeus (Deep Red)
• Crspy (Disekt)
• Galactic (Codejitsu)
• Jima (CSDS)
Tools for Code Assurance
49 / 76

50. CGC (Cyber Grand Challenge)
• CRS (Cyber Reasoning System)
• Fully Automated Security Testing for Software
(no human intervention!)
Generate
Input
(Random, Mutation,
Model-Based, … )
Input
Generation
Software
Analysis
&
Excavate
Vulnerability
Vulnerability
Scanning
Crash is
Exploitable?
Crash
Anaylsis
Generate
Exploit Code
Automatically
Exploit
Generation
Patched
Binary
Automatic
Patching
Tools for Code Assurance
50 / 76

51. Fortify SCA
Assessment Items Description
Usability GUI(Graphic User Interface), Easy to Use
Analysis Report XML Report
Requirement to Evaluator X
Automation Level Fully Automated
Analysis Method Static / Source Code Analyzer
Detectable Vulnerability Type Hundreds of Vulnerability
Code Coverage High Code Coverage
Licensing & Cost Commercial (HP Enterprise)
Supported Languages
Java, .NET, C/C++, JSP, PL/SQL, TSQL, Javascript/Ajax,
PHP, ASP, VB6, COBOL
Supported Platforms Windows, Linux, Solaris, Mac OS X
Tools for Code Assurance
51 / 76

52. CodeSonar
Assessment Items Description
Usability GUI, Easy to use
Analysis Report HTML, XML, CSV Report
Requirement to Evaluator X
Automation Level Fully Automated
Analysis Method Static / Source Code Analyzer / Binary Anaylzer
Detectable Vulnerability Type Hundreds of Vulnerability
Code Coverage High Code Coverage
Licensing & Cost Commercial (Grammatech)
Supported Languages C, C++, Java
Supported Platforms Windows, Linux, Solaris
Tools for Code Assurance
52 / 76

53. CheckMarx SAST
Assessment Items Description
Usability GUI, Easy to Use (Just throw the source code!)
Analysis Report Dashboard Report (PDF, RTF, CSV, XML)
Requirement to Evaluator X
Automation Level Fully Automated
Analysis Method Static / Source Code Analyzer
Detectable Vulnerability Type Hundreds of Vulnerability
Code Coverage High Code Coverage
Licensing & Cost Commercial (CheckMarx)
Supported Languages
Java , Javascript , PHP , C# , VB.NET , VB6 , ASP.NET ,
C/C++ , Apex , Ruby , Perl , Objective-C , Python ,
Groovy , HTML5 , Swift , APEX , J2SE , J2EE
Supported Platforms Android , iOS , Windows
Tools for Code Assurance
53 / 76

54. KLEE
Assessment Items. Description
Usability CLI
Analysis Report X
Requirement to Evaluator O
Automation Level Interactive
Analysis Method Dynamic / Concolic Execution
Detectable Vulnerability Type Memory Corruption
Code Coverage High Code Coverage
Licensing & Cost Non-Commercial (Researched by Stanford University)
Supported Languages C, C++, Objective C
Supported Platforms Linux
Tools for Code Assurance
54 / 76

55. Mayhem (Research Paper Ver.)
Assessment Items Description
Usability CLI, Write Input Specification
Analysis Report
O
(Exploit Type, Input Source, Symbolic Input Size,
Precondition, Adivsory ,Exploit Generation Time)
Requirement to Evaluator O
Automation Level Interactive
Analysis Method Dynamic / Concolic Execution
Detectable Vulnerability Type Memory Corruption
Code Coverage High Code Coverage
Licensing & Cost Non-Commercial (Carnegie Mellon University)
Supported Languages Raw Binary Code
Supported Platforms Linux, Windows
Tools for Code Assurance
55 / 76

56. SAGE
Assessment Items Description
Usability Unknown
Analysis Report Unknown
Requirement to Evaluator O
Automation Level Interactive
Analysis Method Dynamic / Whitebox Fuzz Testing
Detectable Vulnerability Type Hundreds of Vulnerability
Code Coverage Limited Code Coverage
Licensing & Cost Restriced-Commercial (Microsoft)
Supported Languages Raw Binary Code
Supported Platforms Windows
Tools for Code Assurance
56 / 76

57. AFL (American Fuzzy Lop)
Assessment Items Description
Usability
CLI(Command Line Interface)
Install & Setup process is a little complexed.
But provide colorful user interface and statistics.
Analysis Report Crash/Vulnerability Type by Address Sanitizer
Requirement to Evaluator O (Crash Analysis, Exploit Generation, Patching)
Automation Level Interactive
Analysis Method Dynamic / Guided Fuzz Testing
Detectable Vulnerability Type Memory Corruption
Code Coverage High Code Coverage (More time, More Coverage)
Licensing & Cost Open Source (Michael Zalewski)
Supported Languages C, C++, Objective C
Supported Platforms
Linux, *BSD, Solaris, Mac OS X
On Linux, Only Binary(Blackbox) Testing Possible
Tools for Code Assurance
57 / 76

58. IoTcube
Assessment Items Description
Usability Easy to Use (Web Interface, Drag & Drop)
Analysis Report O
Requirement to Evaluator X
Automation Level Fully Automated
Analysis Method
Source Code Analysis (Code Clone Detection)
Binary Fuzz Testing
Network Vulnrability Testing (TLS)
Detectable Vulnerability Type Hundreds of Vulnerability
Code Coverage High Code Coverage
Licensing & Cost Non-Commercial (CSSA, cssa.korea.ac.kr, iotcube.net)
Supported Languages C/C++, Raw Binary Code
Supported Platforms Linux, Windows, Mac OS X
Tools for Code Assurance
58 / 76

59. Mechanical Phish (Shellphish CRS)
Assessment Items Description
Usability
CLI, Install & Setup process is a little complexed but
Easy to Use
Analysis Report -
Requirement to Evaluator
X (Vulnerability Excavation, Crash Analysis, Exploit
Generation, Patch)
Automation Level Fully Automated
Analysis Method
Dynamic, Concolic Execution, Guided Fuzz Testing,
Automatic Exploit Generation, Automatic Patching
Detectable Vulnerability Type Memory Corruption
Code Coverage High Code Coverage
Licensing & Cost Non-Commercial (Shellphish)
Supported Languages Raw Binary Code
Supported Platforms Linux-Like Platforms(Custom by CGC), Intel x86
Tools for Code Assurance
59 / 76

60. Tools for Code Assurance
Automation
Level
Analysis
Report
Analysis Method Target Type Usability
Static Dynamic Binary Source
Fortify SCA
Sparrow
CodeSonar
CheckMarx
KLEE
Mayhem
AFL
IoTcube
Mechanical
Phish
Springfield
(MS - SAGE)
60 / 76

61. Demo (Design / Code)

62. Demo
Widely used Security Protocol
TLS is Standard based on SSL (3.0)
Protect the Transport Layer -> Privacy & Data Integrity
So, We apply design and code assurance to openSSL
62 / 76

63. Demo (Design)
Simplified TLS Key Transport Protocol
- Protocols for Authentication and Key Establishment
1. A -> B : 𝑁"
2. B -> A : 𝑁#
3. A -> B : 𝐸# 𝑃𝑀𝐾 , 𝑆𝑖𝑔"(𝑀𝑒𝑠𝑠_𝑆𝑒𝑞1), {𝑀𝑒𝑠𝑠_𝑆𝑒𝑞4}𝐾"#
4. B -> A : {𝑀𝑒𝑠𝑠_𝑆𝑒𝑞7}𝐾"#
A : Client
B : Server
N : Nonce
𝐸# : RSA Public Key Encryption
Sig : Digital Signature Algorithm (Private Key)
PMK : Pre-Master Secret
𝐾"# : Session Key ( 𝑀𝐴𝐶:;<(𝑁", 𝑁#) )
𝑀𝑒𝑠𝑠_𝑆𝑒𝑞1 : H(𝑁", 𝑁#, 𝐸# 𝑃𝑀𝐾 )
𝑀𝑒𝑠𝑠_𝑆𝑒𝑞4 : H(𝑁", 𝑁#, 𝐸# 𝑃𝑀𝐾 , 𝑀𝑒𝑠𝑠_𝑆𝑒𝑞1)
𝑀𝑒𝑠𝑠_𝑆𝑒𝑞7 : H(𝑁", 𝑁#, 𝐸# 𝑃𝑀𝐾 , 𝑀𝑒𝑠𝑠_𝑆𝑒𝑞1 , 𝑀𝑒𝑠𝑠_𝑆𝑒𝑞4)
63 / 76

64. Demo (Design)
Simplified TLS Key Transport Protocol
- Protocols for Authentication and Key Establishment
1. A -> B : 𝑁"
2. B -> A : 𝑁#
3. A -> B : 𝐸# 𝑃𝑀𝐾 , 𝑆𝑖𝑔"(𝑀𝑒𝑠𝑠_𝑆𝑒𝑞1), {𝑀𝑒𝑠𝑠_𝑆𝑒𝑞4}𝐾"#
4. B -> A : {𝑀𝑒𝑠𝑠_𝑆𝑒𝑞7}𝐾"#
A : Client
B : Server
N : Nonce
𝐸# : RSA Public Key Encryption
Sig : Digital Signature Algorithm
PMK : Pre-Master Secret
𝐾"# : Session Key ( 𝑀𝐴𝐶:;<(𝑁", 𝑁#) )
𝑀𝑒𝑠𝑠_𝑆𝑒𝑞1 : H(𝑁", 𝑁#, 𝐸# 𝑃𝑀𝐾 )
𝑀𝑒𝑠𝑠_𝑆𝑒𝑞4 : H(𝑁", 𝑁#, 𝐸# 𝑃𝑀𝐾 , 𝑀𝑒𝑠𝑠_𝑆𝑒𝑞1)
𝑀𝑒𝑠𝑠_𝑆𝑒𝑞7 : H(𝑁", 𝑁#, 𝐸# 𝑃𝑀𝐾 , 𝑀𝑒𝑠𝑠_𝑆𝑒𝑞1 , 𝑀𝑒𝑠𝑠_𝑆𝑒𝑞4)
Verification by using Scyther
64 / 76

65. Demo (Code)
Shellphish’s Mechanical Phish
65 / 76

66. Demo (Code)
IoTcube (Whitebox)
- CSSA(Center for Software Security and Assurance)
66 / 76

67. Demo (Code)
AFL(American Fuzzy Lop) - lcamtuf
AFL to openSSL 1.0.1f
67 / 76

68. Conclusion

69. Conclusion
There are many kind of Vulnerability Detection Tools developed by hackers,
researchers.
In present, we use these tools for security testing and evaluation.
But there are some limits.
• Objectivity
• Coverage
Recently, many of hackers research and develop automation tools that can
find unknown vulnerability easily.
We can’t apply these tools to security evaluation immediately.
But if fully automated security testing techniques are developed and
we make an effort to apply it for evaluation continuously, achieving high-
assurance is not too far.
69 / 76

70. Acknowledgement
This work was supported by Institute for Information & communications
Technology Promotion(IITP) grant funded by the Korea government(MSIP)
(R7117-16-0161,Anomaly detection framework for autonomous vehicles)
70 / 76

71. Q&A

72. Reference

73. Reference
[1] Debra S. Herrmann, “A practical guide to Security Engineering and Information Assurance”
[2] Sommerville, “Software Engineering, 9ed. 11 & 12, Dependability and Security Specification”
[3] Charlie Miller, Chris Valasek, “Remote Exploitation of an Unaltered Passenger Vehicle”
[4] Ross Anderson, “Security Engineering”
[5] ISO/IEC/IEEE 15288 : 2015, “Systems and Software engineering-System life cycle process”,
[6] Joe Jarzombek, “Software & Supply Chain Assurance : A Historical Perspective of Community Collaboration”,
Homeland Security
[7] David Burke, Joe Hurd and Aaron Tomb, “High Assurance Software Development”, 2010
[8] Ron Ross, Michael McEilley and Janet Carrier Oren, “NIST SP 800-160 : Systems Security Engineering – Consideration
for a Multidisciplinary Approach in the Engineering of Trustworthy Secure Systems”, 2016
[9] Scott A.Lintelman, Krishna Sampigethaya, Mingyan Li, Radha Poovendran, Richard V. Robinson, “High Assurance
Aerospace CPS & Implications for the Automotive Industry”, 2015
[10] NIAP, “Common Criteria-Evaluation and Validation Scheme, Publication #3, Guidance to Validators version 3”, 2014
[11] ISO/IEC 27034-2, “Information technology – Security techniques – Application Security”, 2015
[12] Paul R. Croll, “ISO/IEC/IEEE 15026, Systems and Software Assurance”, 21st
Annual Systems and Software Technology
Conference, 2009
73 / 76

74. Reference
[13] EURO-MILS, “Secure European Virtualisation for Trustworthy Applications in Critical Domains, Used Formal Methods”,
2015
[14] Vijay D’Silva, Daniel Kroening, and Georg Weissenbacher, “A Survey of Automated Techniques for Formal Software
Verification”, 2008
[15] Daniel Potts, Rene Bourquin, Lesile Andresen, “Mathematically Verified Software Kernals: Rasing the Bar for High
Assurance Implementation
[16] Bernhard Beckert, Daniel Bruns, Sarah Grebing, “Mind the Gap : Formal Verification and the Common Criteria“, 2010
[17] Gerwin Klein, Kevin Elphinstone, Gernot Heiser, June Andronick, David Cock, Philip Derrin, Dhammika Elkaduwe, Kai
Engelhardt, Rafal Kolankski, Michel Norrich, Thomas Sewell, Harvey Tuch, Simon Winwood, “seL4 : Formal Verification of
an OS Kernel”, 2009
[18] Gerwin Klein, NICTA, “Operating System Verification – An Overview”, 2009
[19] Jesus Diaz, David Arroyo, Francisco B. Rodriguez, “A formal methodology for integral security design and verification
of network protocols”, 2012
[20] Yoshikazu Hanatanil, Miyako Ohkubo, Sinichiro Matsuo, Kazuo Sakiyama, and Kazuo Ohta, “A Study on
Computational Formal Verification for Practical Cryptographic Protocol: The Case of Synchronous RFID Authentication”,
2011
[21] Alexandre Melo Braga, Ricardo Hahab, “A Survey on Tools and Techniques for the Programming and Verification of
Secure Cryptographic Software”, 2015
74 / 76

75. Reference
[22] Shinichiro Matsuo, Kunihiko Miyazaki, Akira Otsuka, David Basin, “How to Evaluate the Security of Real-life
Cryptographic Protocol? The cases of ISO/IEC 29128 and CRYPTREC, 2010
[23] Bruno Blanchet, Ben Smyth, and Vincent Cheval, “ProVerif 1.94pl1: Automatic Cryptographic Protocol Verifier, User
Manual and Tutorial”, 2016
[24] Charles B. Weinstock, John B. Goodennough, “Toward an Assurance Case Practice for Medical Devices”, 2009
[25] CISCO, “Building Trustworthy Systems with Cisco Secure Development Lifecycle”, 2016
[26] Yannick Moy, Emmanuel Ledinot, Herve Delseny, Virginie Wiels, Benjamin Monte, “Testing or Formal Verification :
DC-178C Alternatives and Industrial Experience”, 2013
[27] Karen Scarfone, Murugiah Souppaya, Amanda Cody, Angela Orebaugh, “NIST SP 800-115, Technical Guide to
Information Security Testing and Assessment – Recommandations of the National Institue of Standards and Technology”,
2008
[28] Steve Lipner, Microsoft, “The Security Development Lifecycle”, 2010
[29] Michael Felderer, Ruth Breu, Matthias Buchler, “Security Testing : A Survey”, 2016
[30] Vijay D’Silva, Daniel Kroening, George Weissenbacher, “A Survey of Automated Techniques for Formal Software
Verification”
[31] John Rushby, Xidong Xu, Rangarajan and Thomas L. Weaver, “Understanding and Evaluating Assurance Case”, 2015
[32] David J.Rinehart, John C. Knight, Jonathan Rowanhill, “Current Practices in Constructing and Evaluating Assurance
Case with Application to Aviation”, 2015
[33] The Government of Japan, “Cybersecurity Strategy 2015”
75 / 76

76. Reference
[34] Yasu Taniwaki, Deputy Director-General National Information Security Center, “Cybersecurity Strategy in Japan”, 2014
[35] “The NRL Protocol Analyzer : An Overview”, 1994
[36] Bruno Blanchet, “Automatic Verification of security protocols : the tools ProVerif and CryptoVerif”, 2011
[37] Tobias Nipkow, “Programming and Proving in Isabelle/HOL”, 2016
[38] Assistant Secretary of the Navy Chief System Engineer, “Software Security Assessment Tools Review”, 2009
[39] S.Santiago, C.Talcott, S.Escobar, C.Meadows, J.Meseguer, “A Graphical User Interface for Maude-NPA”, 2009
[40] NIST, "Source Code Security Analyzers"
[41] Cadar, Cristian, "KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs",
2008
[42] Cha, Sang Kil, "Unleashing MAYHEM on Binary Code", 2012
[43] Giovanni Vigna, "Autonomous Hacking: The New Frontiers of Attack and Defense", 2016
[44] Antonio Bianchi, "A Dozen Years of Shellphish From DEFCON to the Cyber Grand Challenge", 2015
[45] Jonathan Salwan, "Triton: Concolic Execution Framework", 2016
[46] Godefroid, "SAGE: Whitebox Fuzzing for Security Testing", 2012
[47] Michael Zalewski, "American Fuzzy Lop (http://lcamtuf.coredump.cx/afl/)", 2015
[48] Vegard Nossum, Oracle, "Filesystem Fuzzing with American Fuzzy Lop", 2016
[49] Hongzhe Li, "CLORIFI: software vulnerability discovery using code clone verification", 2015
[50] Stephens, "Driller: Augmenting Fuzzing Through Selective Symbolic Execution", 2016
[51] John Rushby, “The Interpretation and Evaluation of Assurance Cases”, SRI International Technical Report, 2015
76 / 76