The talk covers several technologies and best practices to managing Security Vulnerabilities, which are told as interconnected stories. We will cover how the largest clouds in production came together through the Xen Project to develop an industry leading open source security process to manage software vulnerabilities effectively, how those vendors collaborated to stop cloud reboots through Live Patching and how security and CPU vendors collaborated to protect against 0-day vulnerabilities and advanced persistent threats using hardware assisted virtual machine introspection. Finally, we will also provide information how you can use tools such as CVE Details to assess how secure an open source technology is relative to another, such that you don't have to rely solely on security stories from the technology press. The talk will cover how these technologies work, the limitations and challenges which still remain and how they are used in practice using examples of Xen Project based products and installations. We will also cover how these technologies impact software vulnerability management processes and system administrators.

1. Presentations on www.slideshare.net/xen_com_mgr/presentations
Lars Kurth
Community Manager, Xen Project
Chairman, Xen Project Advisory Board
Director, Open Source, Citrix lars_kurth

5. www.xenproject.org
www.xenserver.org

6. Vulnerability
A weakness in the computational logic of software that, when exploited, results in a negative
impact to confidentiality, integrity, OR availability of that software.
Patch / Live Patch
A fix for a security vulnerability.
A live patch can be applied to a running system
Known vulnerabilities
with patches

7. Malware
Software designed to disrupt, damage, or gain authorized access to a computer system.
Exploit
Malware designed to take advantage of a vulnerability
This includes file-less/memory based attack techniques (around 14% at end of 2016)
0-Day Exploit
An exploit of an undisclosed/previously unknown vulnerability that is/has been exploited
Some malware relies on
social engineering: e.g.
phishing, trojans, adware, …

8. Exploit Framework (e.g. Metasploit, …)
A tool for developing and executing exploit code against a remote target machine
Rootkit
Software tools that enable unauthorized users to gain control of a computer system without
being detected, some for for a long period of time

9. A new way to protect against malware
Developed by Zentific, Citrix,
Bitdefender, Intel and others

10. VM3
Guest OS
App
VMn
Guest OS
App
VM2
Guest OS
App
Dom0
Dom0 Kernel
Drivers Agent(s) Agent(s) Agent(s)
Installed in-guest agents, e.g. anti-virus software,
VM disk & memory scanner, network monitor, etc.
Can be disabled by rootkits

11. Several
VM3 VMnVM2Dom0
Dom0 Kernel
Drivers
VM3
Guest OS
App
VMn
Guest OS
App
VM2
Guest OS
App
Security
Appliance
VM1
Introspection
Engine
Protected area
authentication mechanism to protect the IF
Uses HW extensions to monitor memory (e.g. Intel EPT)  Low Intrusion
Register rules with Xen to trap on and inspect suspicious activities
(e.g. execution of memory on the dynamic heap)

12. All malware need an attack technique to gain a foothold
Attack techniques exploit specific software bugs/vulnerability
Most exploits use one of a small set of attack techniques
Buffer Overflows, Heap Sprays, Code Injection, API Hooking, …
Because VMI protects against attack techniques
It can protect against entirely new malware
Verified to block these advanced attacks in real-time
APT28, Energetic Bear, DarkHotel, Epic Turla, Regin, ZeuS, Dyreza, EternalBlue
… solely by relying on VMI
WannaCry/EternalBlue blocked in real installations1
1 businessinsights.bitdefender.com/hypervisor-introspection-defeated-enternalblue-a-priori

13. Rootkits
Exploit 0-days in Operating Systems/System Software
Can disable agent based security solutions (mask their own existence)
VMI solutions operate from outside the VM
Thus, it cannot be disabled using traditional attack vectors
BUT:
VMI is not a replacement, for traditional security solutions
It is an extra tool that can be used to increase protection

14. Pratap Sankar @ Flickr
Documentation
wiki.xenproject.org/wiki/Virtual_Machine_Introspection
Products
Bitdefender HVI
XenServer
www.bitdefender.com
Protection & Remedial
Monitoring & Admin
Citrix Ready
Zentific Zazen
Xen & XenServer & …
www.zentific.com
Protection & Remedial
Monitoring & Admin
Forensics & Data gathering
Malware analysis
AIS Introvirt
XenServer
www.ainfosec.com

15. Pratap Sankar @ Flickr
https://www.youtube.com/watch?v=qpQPBvOniUU

16. Result of several community consultations

18. R: Vulnerability reported to security@xenproject.org
P: Vulnerability pre-disclosed on xen-security-issues@lists.xenproject.org
R P
Fixing Security Bugs:
Dedicated security team =
security experts from within
the Xen Project Community
Security Team:
Triage
Creation of fix/patches
Validation of fix/patches
Assignment of CVE
Issue description and risk analysis

19. AR P
Fix their systems/software:
Eligible Xen Project Users
are informed under embargo
of the vulnerability
Eligible Users = Pre-disclosure list members:
Product Companies, Open Source & Commercial Distros (e.g. Huawei, Debian)
Service/Cloud Providers (e.g. Alibaba)
Large Private Downstream (e.g. Google)
Allowed to share information via
xen-security-issues- discuss@lists.xenproject.org
R: Vulnerability reported to security@xenproject.org
P: Vulnerability pre-disclosed on xen-security-issues@lists.xenproject.org
A: Vulnerability announced on xen-announce@lists.xenproject.org & xenbits.xen.org/xsa

20. A XR P
General Publication:
Information about
vulnerability is made public
Everyone else:
Patches their systems either through
security updates from distros/products or
builds them from source.
Users of service/cloud providers will
not be impacted
R: Vulnerability reported to security@xenproject.org
P: Vulnerability pre-disclosed on xen-security-issues@lists.xenproject.org
A: Vulnerability announced on xen-announce@lists.xenproject.org & xenbits.xen.org/xsa

21. A XR P
R: Vulnerability reported to security@xenproject.org
P: Vulnerability pre-disclosed on xen-security-issues@lists.xenproject.org
A: Vulnerability announced on xen-announce@lists.xenproject.org & xenbits.xen.org/xsa
Product Vendors:
Create and test live patches
Product customers:
Apply live patches here
Service Providers:
Create, test and deploy live patches
Users of service/cloud providers will
not be impacted
Security Team:
Can a Livepatch can be created?
No? If possible, re-write fix/patches

22. A tale of close collaboration within
the Xen Project Community

23. 2015 2016 2017
4.84.7 P
Why did we develop Live Patching?
Cloud reboot affected AWS, Rackspace, IBM SoftLayer and others
Deploying security patches may require reboots; Inconveniences users
How did we fix this?
2015: Design with input from AWS, Alibaba, Citrix, Oracle and SUSE
2016: Xen 4.7 came with Live Patching for x86
2016: Xen 4.8 added extra x86 use-cases and ARM support
2017: XenServer 7.1 releases Live Patching in first commercial product
D
10% 1%

24. const char *xen_extra_version(void)
{
return XEN_EXTRAVERSION;
}
push %rbp
mov %rsp,%rbp
lea 0x16698b(%rip),%rax
leaveq
retq
const char *xen_extra_version(void)
{
return “Hello World”;
}
push %rbp
mov %rsp,%rbp
lea 0x29333b(%rip),%rax
leaveq
Retq
Replacing compiled functions with new code, encoded in an ELF file called
payload, while the hypervisor is running without impacting running guests.
Design: xenbits.xenproject.org/docs/unstable/misc/livepatch.html

25. The exact source tree used to
build the running Xen instance.
The .config from the original
build of Xen.
A build-id onto which the
livepatch will be applied.
A source patch.
livepatch-
build-
tools
The exact same
compilation toolchain
used to build the
running Xen.
Livepatch
payload

26. Supports stacking of different payloads; payloads depend on build-id
Functionality:
list: lists loaded and applied live patches
upload: load & verify a live patch
unload: unload a live patch
apply: apply a live patch
revert: un-apply a live patch Xen 4.8.1
XSA 213
XSA 214
XSA 215
Depends on
build-id of 4.8.1
Depends on
build-id of XSA 213
Depends on
build-id of XSA 214

27. Target
Dom0 &
Guest
Linux
Kernel
Hypervisor
Technology
Kernel
Live Patching
kPatch
(RedHat)
Xen
LivePatch
kSplice
(Oracle)
kGraft
(SUSE)
Function +
Data
✔
✔
✔ Xen 4.7
✔
✔
Inline f()
patching
✗
✗
✗ Future
✔
✗
Data
Structures
✗
✔ via hooks
✔
✔
✗
Xen 4.8
via hooks
XenServer
LivePatch
Integrates
different solutions
into a single user
experience
For Dom0
(CentOS)
For Xen

28. Source patches
+ other
build artifacts
Hot Fixes contain
Per valid patch level: a Xen or Dom0 Live Patch
Matching RPMs for most recent patch level
In case of a reboot or for Xen/Dom0 not capable of Live Patching
Extensive Verification and Validation:
The process of patching a live hypervisor or kernel
is not an easy task. What happens is a little bit like open
heart surgery. The patient is the hypervisor and/or Dom0
itself, and precision and care are needed to get things right.
One wrong move and it is game over.
Live
Patch
Live
Patch
Live
Patch
build
for each
patch level
package
Hot Fix
LPsRPMs
RPMRPMRPM
build
for most recent
patch level
Hot Fix
LPsRPMs
Publication
SigningValidation
Verification
Q&A
(livepatch-build
or kpatch-build)
(iso)

29. XAPI
Toolstack
Hot Fix
LPsRPM
downloadHot Fix
LPsRPM
Dom0
Dom0 Kernel
(CentOS)
Hypervisor
XenCenter
or xe
Initiates
host update
SysAdmin
Running System
instance that supports
live patching
Disk
updates
(such that
after reboot
the patches
are applied)
works out
correct LP
& updates
(using native
live patching
tools or APIs)

30. Pratap Sankar @ Flickr
xenbits.xenproject.org/people/larsk/LCC17 - Build LivePatch.mp4
xenbits.xenproject.org/people/larsk/LCC17 - Apply LivePatch.mov

32. Pratap Sankar @ Flickr
Xen Project LivePatch Specification & Status
xenbits.xenproject.org/docs/unstable/misc/livepatch.html
wiki.xenproject.org/wiki/LivePatch
Xen Project LivePatch Presentations & Videos
xenbits.xenproject.org/people/larsk/FOSDEM17-LivePatch.pdf (Short)
people/larsk/XPDS16-LivePatch.pdf (Long)
Xen Project LivePatch Videos
fosdem.org/2017/schedule/event/iaas_livepatxen/
XenServer
xenserver.org

33. Security Process
Number of Vulnerabilities
Media Coverage
Other Considerations

34. A XR P
Responsible Disclosure: fix critical systems/software before publication
R: Vulnerability reported to security@...
P: Vulnerability pre-disclosed to eligible users
A: Vulnerability announced publicly
F: Fix available
Full Disclosure, immediate (no-fix): public disclosure without a fix
A XR F
A XR
Full Disclosure, post-fix: public disclosure with a fix
F
F

35. 4) New members: http://seclists.org/oss-sec/2017/q2/638
5) http://www.openwall.com/lists/oss-security or devel list
6) https://wiki.qemu.org/index.php/SecurityProcess
Only handles x86 KVM bugs (no ARM or other bugs)
1) Is the CVE severity used to handle vulnerabilities differently?
2) Days embargoed (information is classified)
3) D = Distros/Products, S = Public Service, P = Private Downstream
4) http://oss-security.openwall.org/wiki/mailing-lists/distros
Responsible only
Days 2 Who? 3FOSS Project Bug Severity 1 Process Type
14-19 D 4
Linux Kernel via
OSS-security distros 4 ≥ Medium – Critical Responsible Disclosure
14-19 D 4
QEMU (KVM) via
QEMU Security
Process 6
≥ Medium – Critical
≤ Low
Responsible Disclosure
Full Disclosure, no-fix
3-5 D, S, POpenStack OSSA
OpenStack OSSN
≥ Medium – Critical
≤ Low
Responsible Disclosure
Full Disclosure, post-fix
Xen Hypervisor
Includes Linux & QEMU
vulnerabilities in supported Xen
configurations
Low – Critical Responsible Disclosure 14 D, S, P
14-19
Linux Kernel via
OSS-security distros 4
OSS-security 5
≥ Medium – Critical
≤ Low
Responsible Disclosure
Full Disclosure, no-fix

36. Impacts how long a user (aka you) is at risk
Is my distro/vendor on a pre-disclosure list?
A surprisingly large number of distros are not:
including a few on Linux.com’s Best Linux Distros of 2016 list
Impacts cloud / service providers
As a user, are security issues fixed before public disclosure?
Low Severity vulnerabilities can still be High Risk
Temporal and Environmental CVSS scores are not covered by CVE databases
(neither cvedetails.com or nvd.nist.gov)
Vulnerabilities can be chained together, making the combo High Severity
(e.g. Hot Potato used 3 old unpatched vulnerabilities to gain root access)

37. cvedetails.com (bit.do/guide-cvedetails)
Easy to use interface for vulnerability data
Data from several sources
Browsable by vendor, product, version, type, date…
Vulnerability statistics, trends, reports
BUT: rigid ➜ getting data outside pre-defined vendor/product categories is near-impossible
vulners.com (good guides on slideshare.net – search for vulners)
In many ways more accurate and flexible than cvedetails
type:cve AND (description:kvm OR description:qemu) AND published: [2012 TO *] ➜307
type:cve AND (description:xen) AND published: [2012 TO *] ➜ 245
…
Works best when used through its API (in particular if you want to visualize the data)

38. Vulnerability data from vulners.com
0
50
100
150
200
250
300
350
2012 2013 2014 2015 2016 2017
 Linux Kernel
 KVM w. QEMU
 Xen Project
*) Data up to Sept 4th, 2017
169
517
222
197
129
120
51
7
63
143
34
5
Legend: CVSS
Score Distribution
Low
Medium
High
Critical
*

39. Data covering September 2016 – September 2017: from vulners.com, mention.com and theregister.com
Clips covering
vulnerabilities
(US only)
% of Vulnerability
Stories on
The Register155
370 314
7283
0
1000
2000
3000
4000
5000
6000
7000
8000
Xen KVM QEMU Linux
0
50
100
150
200
250
300
350
400
450
Xen KVM QEMU Linux
Number of
Vulnerabilities
0%
5%
10%
15%
20%
25%
30%
35%
40%
Xen KVM QEMU Linux
33% of Xen stories
cover Vulnerabilities
2.5%
16%
6.5%

40. Does the Project Look for Vulnerabilities?
Approach to Quality and Testing: e.g. Fuzzing, Audits of components
Does the project award Bug Bounties (e.g. NetBSD)?
Do vendors supporting the project offer Bug Bounties?
Infrastructure related to Vulnerabilities
Transparency: How well are processes documented
Vulnerability Testing: XTF (in Xen)
Vulnerability Tooling: XSATool, XSAMatch (in Xen)

41. Picture by Lars Kurth

42. Xen Project:
Only Hypervisor with VMI
Protection from new classes of malware
Several security companies working with XenServer
Live Patching
Disruption free application of vulnerabilities
Used by several cloud providers
Used best in commercial products, e.g. XenServer
Industry Leading Vulnerability Process
Includes QEMU and Kernel XSAs
General:
Tools do assess project’s track records
Not an easy task
Harder for proprietary products due to lack of information
Picture by Lars Kurth

43. www.slideshare.net/xen_com_mgr/presentations
Picture by Lars Kurth