iperfTZ: Understanding Network Bottlenecks for TrustZone-based Applications

21st International Symposium on Stabilization, Safety, and
Security of Distributed Systems 2019
Pisa, Italy
iperfTZ: Understanding Network Bottlenecks for
TrustZone-based Applications
Christian Göttel, Pascal Felber, Valerio Schiavoni
University of Neuchâtel, Computer Science Department, Complex Systems
SSS’19 23.10.2019 | IIUN | Christian Göttel | christian.goettel@unine.ch
iperfTZ: Understanding Network Bottlenecks for TrustZone-based Applications 1 / 21

Introduction
What are Trusted Execution Environments?
Execution Environment (EE):
Sets of hardware and software components that can run
applications. Some examples are:
HypervisorOS
OS
OS
VM
OS OS
VM
CPU CPU CPU CPU
Trusted Execution Environment (TEE):
Any EE that satisﬁes sets of security requirements
?
Rack
!
Home
Lack of trust in cloud provider
Shield services from compromised
hosts
Regain control over code and data

Introduction
TEE Hardware and Specifications
MultiZone™
2003 201920112007 201520092005 2013 2017
ARM®
OMTP/WAC ATE: TR1
GSMA ATE: TR1
SME/SEVAMD
PSP Secure ProcessorAMD
SGXIntel®
GlobalPlatform Specifications / Technical Documents
Hex Five Security
OmniShield™Imagination
zACI/SSCIBM®
TrustZone®
HardwareSpecifications
HardwareSpecifications
AMD is a trademark of Advanced Micro Devices.

Introduction
TEE Software
2003 201920112007 201520092005 2013 2017
Trusted FoundationsTrusted Logic/Gemalto
Trustonic
G&D MobiCore
<t-base
ZirconGoogle
Kinibi™/Kinibi-M™
(<t-base derivative?)AMD
NVIDIA®
TLK
Android™ Trusty
Qualcomm®
QSEE
STM/Linaro™ OP-TEE
TrustKernel TSEE
ICRI-SC Open-TEE
Solacia/Hansol Secure securiTEE
Sierraware SierraTEE
Nagoya University TOPPERS-SafeG
Samsung®
Knox
Sequitur Labs CoreTEE™
LKTravis Geiselbrecht
Open Trust GroupGzOS
ARM® Trusted Firmware
AMDisatrademarkofAdvancedMicroDevices.
GemaltoisatrademarkofGemalto.
GoogleisaregisteredtrademarkofGoogleLLC.

Introduction
Arm TrustZone
EL0
EL1
EL2
EL3
Normal World Secure World
EL0
EL1
EL2
EL3
Secure Monitor
ARM TF Dispatcher
Hypervisor
OS Driver
App
GP Client API
Hypervisor
TOS TOS
GP API
TA
TrustZone Exception Levels
TrustZone is a set of Arm
security extensions
System on a chip (SoC)
manufacturers are free to
implement any TrustZone
subset
Exception Levels layer the
architecture into a privilege
hierarchy
Secure Monitor Call (SMC)
instruction to switch worlds

Introduction
GlobalPlatform
Trusted OSRich OS
TEE
Trusted space
Platform Hardware
User space
REE
Secure Element
Trusted Storage
Peripherals
Comm. Agent
Driver Driver
Comm. Agent
API
Client API
CA
TA
Core API
Socket API
GlobalPlatform System Architecture
Rich Execution Environment (REE)
Trusted Execution Environment (TEE)
GlobalPlatform Speciﬁcations:
CA := Client Application
TEE Client API
TEE Internal Core API
Trusted Storage API
Crypto. Operations API
Time API
Arithmetical API
TEE Sockets API
and many more APIs

Introduction
OP-TEE: Open Portable TEE
EL0
EL1
EL2
EL3
EL0
EL1
EL2
EL3
Secure Monitor Dispatcher
xen
other
Driver Driver
OP-TEE
tee-supplicant
libteeclibc
TA
libutee
OP-TEE
OS PTA
CA
Linux is running in the normal world, while OP-TEE is running in the
secure world.
Client application (CA), (pseudo) trusted application (PTA)

Motivating Scenario
ServerClient
CA
TEE REE
server
REE
Observing high throughput from client REE, . . .

Motivating Scenario
ServerClient
CA
TEE REE
server
REE
ServerClient
TA
TEE REE
server
REE
Observing high throughput from client REE, but low throughput from
client TEE

Motivating Scenario
ServerClient
iperfTZ
TEE REE REE
*
iperfTZ
*
Where * can be: iperf3 or netperf or nuttcp or . . .

Implementation
Architecture
ServerClient
User
CA
TA tee-supplicant
TEE REE
server
REE

Implementation
Threat Model
Client
User
CA
TA tee-supp
TEE REE Server
server
REE
SD / USBeMMC
Server
server
REE
Server
server
REENode
Networking
REE / TEE

Implementation
Threat Model
TA
Client
!
CA
tee-supp
TEE REE Server
server
REE
SD / USBeMMC
Server
server
REE
Server
server
REENode
Networking
REE / TEE

Evaluation
Setup
Comparison of evaluation platforms.
Device QEMU Raspberry
CPU Model Intel Xeon E3-1270 v6 Broadcom BCM2837
CPU Frequency 3.8 GHz 1.2 GHz
Memory Size 63 GiB DDR4 944 MiB LPDDR2
Memory data rate 2400 MT/s 800 MT/s
Network Bandwidth 1 Gbit/s 100 Mbit/s
TZMA
TZPC
TZASC
TrustZone
GIC (4B )

Evaluation
Shared Memory Benchmark
TEE Context
Whole
Partially
Temporarily
Shared Memory
Memory Object
Evaluate the three diﬀerent
types of shared memory
Static hash table
implementation of kazlib
Run operations on 1 KiB
chunks:
DEL: delete
GET: retrieve
PUT: insert
MIX20: 20% PUT
MIX50: 50% PUT

Evaluation
Shared Memory Results
0 2 4 6
0
5
10
15
20
Latency[s]
■■■■■■■■■■ ■ ■■
■
■
■
■■■■■■■■■■ ■ ■■
■
■
■
▲▲▲▲▲▲▲▲▲▲ ▲ ▲▲
▲
▲
▲
▲▲▲▲▲▲▲▲▲▲ ▲ ▲
▲
▲
▲
▲
◀◀◀◀◀◀◀◀◀ ◀ ◀ ◀◀
◀
◀
◀
◀◀◀◀◀◀◀◀◀ ◀ ◀ ◀ ◀ ◀
◀
◀
▼▼▼▼▼▼▼▼▼▼ ▼ ▼ ▼ ▼
▼
▼
▼▼▼▼▼▼▼▼▼▼ ▼ ▼ ▼▼
▼
▼
▶▶▶▶▶▶▶▶▶ ▶ ▶ ▶ ▶ ▶
▶
▶
▶▶▶▶▶▶▶▶▶ ▶ ▶ ▶ ▶ ▶
▶
▶
QEMU DEL■
QEMU GET■
QEMU PUT▲
QEMU MIX20▲
QEMU MIX50◀
RPi3 DEL◀
RPi3 GET▼
RPi3 PUT▼
RPi3 MIX20▶
RPi3 MIX50▶
Partially SHM
0 2 4 6
0
5
10
15
20
Throughput [kop/s]
Latency[s]
■■■■■■■■■■ ■ ■
■
■
■
■
■■■■■■■■■■ ■ ■
■
■
■
■
▲▲▲▲▲▲▲▲▲▲ ▲▲
▲
▲
▲
▲
▲▲▲▲▲▲▲▲▲▲ ▲ ▲▲
▲
▲
▲
◀◀◀◀◀◀◀◀◀ ◀ ◀ ◀◀
◀
◀
◀
◀◀◀◀◀◀◀◀◀ ◀ ◀ ◀ ◀ ◀
◀
◀
▼▼▼▼▼▼▼▼▼▼ ▼ ▼ ▼▼
▼
▼
▼▼▼▼▼▼▼▼▼▼ ▼ ▼ ▼▼
▼
▼
▶▶▶▶▶▶▶▶▶ ▶ ▶ ▶ ▶▶
▶
▶
▶▶▶▶▶▶▶▶▶ ▶ ▶ ▶ ▶▶
▶
▶
Whole SHM
0 2 4 6
0
5
10
15
20
25
■■■■■■■■■■ ■ ■■
■
■
■
■■■■■■■■■■ ■■■
■
■
■
▲▲▲▲▲▲▲▲▲▲ ▲▲▲
▲
▲
▲
▲▲▲▲▲▲▲▲▲▲ ▲▲▲
▲
▲
▲
◀◀◀◀◀◀◀◀◀ ◀ ◀ ◀◀
◀
◀
◀
◀◀◀◀◀◀◀◀◀ ◀ ◀ ◀ ◀ ◀◀
◀
▼▼▼▼▼▼▼▼▼▼ ▼ ▼ ▼▼
▼
▼
▼▼▼▼▼▼▼▼▼▼ ▼ ▼ ▼▼
▼
▼
▶▶▶▶▶▶▶▶▶ ▶ ▶ ▶ ▶▶
▶
▶
▶▶▶▶▶▶▶▶▶ ▶ ▶ ▶ ▶▶
▶
▶
Temporarily SHM
0 20 40 60 80
0
5
10
15
20
25
Throughput [kop/s]
■■■■■■■■■■■■■ ■ ■■ ■
■
■
■
■■■■■■■■■■■■■ ■ ■■■
■
■
■
▲▲▲▲▲▲▲▲▲▲▲▲▲▲ ▲▲▲
▲
▲
▲
▲▲▲▲▲▲▲▲▲▲▲▲▲▲ ▲▲▲
▲
▲
▲
◀◀◀◀◀◀◀◀◀◀◀◀◀ ◀ ◀◀◀
◀
◀
◀
◀◀◀◀◀◀◀◀◀◀◀◀◀ ◀ ◀ ◀ ◀ ◀◀
◀
▼▼▼▼▼▼▼▼▼▼▼▼▼▼ ▼ ▼ ▼ ▼
▼
▼
▼▼▼▼▼▼▼▼▼▼▼▼▼▼ ▼ ▼ ▼▼
▼
▼
▶▶▶▶▶▶▶▶▶▶▶▶▶ ▶ ▶ ▶ ▶ ▶
▶
▶
▶▶▶▶▶▶▶▶▶▶▶▶▶ ▶ ▶ ▶ ▶ ▶
▶
▶
Inside REE
12 to 14 × overhead on QEMU and 12 to 17 × on Raspberry Pi

Performance Evaluation
0 2 4 6
0
5
10
15
20
Latency[s]
■■■■■■■■■■ ■ ■■
■
■
■
■■■■■■■■■■ ■ ■■
■
■
■
▲▲▲▲▲▲▲▲▲▲ ▲ ▲▲
▲
▲
▲
▲▲▲▲▲▲▲▲▲▲ ▲ ▲
▲
▲
▲
▲
◀◀◀◀◀◀◀◀◀ ◀ ◀ ◀◀
◀
◀
◀
◀◀◀◀◀◀◀◀◀ ◀ ◀ ◀ ◀ ◀
◀
◀
▼▼▼▼▼▼▼▼▼▼ ▼ ▼ ▼ ▼
▼
▼
▼▼▼▼▼▼▼▼▼▼ ▼ ▼ ▼▼
▼
▼
▶▶▶▶▶▶▶▶▶ ▶ ▶ ▶ ▶ ▶
▶
▶
▶▶▶▶▶▶▶▶▶ ▶ ▶ ▶ ▶ ▶
▶
▶
QEMU DEL■
QEMU GET■
QEMU PUT▲
QEMU MIX20▲
QEMU MIX50◀
RPi3 DEL◀
RPi3 GET▼
RPi3 PUT▼
RPi3 MIX20▶
RPi3 MIX50▶
Partially SHM
0 2 4 6
0
5
10
15
20
Throughput [kop/s]
Latency[s]
■■■■■■■■■■ ■ ■
■
■
■
■
■■■■■■■■■■ ■ ■
■
■
■
■
▲▲▲▲▲▲▲▲▲▲ ▲▲
▲
▲
▲
▲
▲▲▲▲▲▲▲▲▲▲ ▲ ▲▲
▲
▲
▲
◀◀◀◀◀◀◀◀◀ ◀ ◀ ◀◀
◀
◀
◀
◀◀◀◀◀◀◀◀◀ ◀ ◀ ◀ ◀ ◀
◀
◀
▼▼▼▼▼▼▼▼▼▼ ▼ ▼ ▼▼
▼
▼
▼▼▼▼▼▼▼▼▼▼ ▼ ▼ ▼▼
▼
▼
▶▶▶▶▶▶▶▶▶ ▶ ▶ ▶ ▶▶
▶
▶
▶▶▶▶▶▶▶▶▶ ▶ ▶ ▶ ▶▶
▶
▶
Whole SHM
0 2 4 6
0
5
10
15
20
25
■■■■■■■■■■ ■ ■■
■
■
■
■■■■■■■■■■ ■■■
■
■
■
▲▲▲▲▲▲▲▲▲▲ ▲▲▲
▲
▲
▲
▲▲▲▲▲▲▲▲▲▲ ▲▲▲
▲
▲
▲
◀◀◀◀◀◀◀◀◀ ◀ ◀ ◀◀
◀
◀
◀
◀◀◀◀◀◀◀◀◀ ◀ ◀ ◀ ◀ ◀◀
◀
▼▼▼▼▼▼▼▼▼▼ ▼ ▼ ▼▼
▼
▼
▼▼▼▼▼▼▼▼▼▼ ▼ ▼ ▼▼
▼
▼
▶▶▶▶▶▶▶▶▶ ▶ ▶ ▶ ▶▶
▶
▶
▶▶▶▶▶▶▶▶▶ ▶ ▶ ▶ ▶▶
▶
▶
Temporarily SHM
0 20 40 60 80
0
5
10
15
20
25
Throughput [kop/s]
■■■■■■■■■■■■■ ■ ■■ ■
■
■
■
■■■■■■■■■■■■■ ■ ■■■
■
■
■
▲▲▲▲▲▲▲▲▲▲▲▲▲▲ ▲▲▲
▲
▲
▲
▲▲▲▲▲▲▲▲▲▲▲▲▲▲ ▲▲▲
▲
▲
▲
◀◀◀◀◀◀◀◀◀◀◀◀◀ ◀ ◀◀◀
◀
◀
◀
◀◀◀◀◀◀◀◀◀◀◀◀◀ ◀ ◀ ◀ ◀ ◀◀
◀
▼▼▼▼▼▼▼▼▼▼▼▼▼▼ ▼ ▼ ▼ ▼
▼
▼
▼▼▼▼▼▼▼▼▼▼▼▼▼▼ ▼ ▼ ▼▼
▼
▼
▶▶▶▶▶▶▶▶▶▶▶▶▶ ▶ ▶ ▶ ▶ ▶
▶
▶
▶▶▶▶▶▶▶▶▶▶▶▶▶ ▶ ▶ ▶ ▶ ▶
▶
▶
Inside REE

Evaluation
0 2 4 6
0
5
10
15
20
Latency[s]
■■■■■■■■■■ ■ ■■
■
■
■
■■■■■■■■■■ ■ ■■
■
■
■
▲▲▲▲▲▲▲▲▲▲ ▲ ▲▲
▲
▲
▲
▲▲▲▲▲▲▲▲▲▲ ▲ ▲
▲
▲
▲
▲
◀◀◀◀◀◀◀◀◀ ◀ ◀ ◀◀
◀
◀
◀
◀◀◀◀◀◀◀◀◀ ◀ ◀ ◀ ◀ ◀
◀
◀
▼▼▼▼▼▼▼▼▼▼ ▼ ▼ ▼ ▼
▼
▼
▼▼▼▼▼▼▼▼▼▼ ▼ ▼ ▼▼
▼
▼
▶▶▶▶▶▶▶▶▶ ▶ ▶ ▶ ▶ ▶
▶
▶
▶▶▶▶▶▶▶▶▶ ▶ ▶ ▶ ▶ ▶
▶
▶
QEMU DEL■
QEMU GET■
QEMU PUT▲
QEMU MIX20▲
QEMU MIX50◀
RPi3 DEL◀
RPi3 GET▼
RPi3 PUT▼
RPi3 MIX20▶
RPi3 MIX50▶
Partially SHM
0 2 4 6
0
5
10
15
20
Throughput [kop/s]
Latency[s]
■■■■■■■■■■ ■ ■
■
■
■
■
■■■■■■■■■■ ■ ■
■
■
■
■
▲▲▲▲▲▲▲▲▲▲ ▲▲
▲
▲
▲
▲
▲▲▲▲▲▲▲▲▲▲ ▲ ▲▲
▲
▲
▲
◀◀◀◀◀◀◀◀◀ ◀ ◀ ◀◀
◀
◀
◀
◀◀◀◀◀◀◀◀◀ ◀ ◀ ◀ ◀ ◀
◀
◀
▼▼▼▼▼▼▼▼▼▼ ▼ ▼ ▼▼
▼
▼
▼▼▼▼▼▼▼▼▼▼ ▼ ▼ ▼▼
▼
▼
▶▶▶▶▶▶▶▶▶ ▶ ▶ ▶ ▶▶
▶
▶
▶▶▶▶▶▶▶▶▶ ▶ ▶ ▶ ▶▶
▶
▶
Whole SHM
0 2 4 6
0
5
10
15
20
25
■■■■■■■■■■ ■ ■■
■
■
■
■■■■■■■■■■ ■■■
■
■
■
▲▲▲▲▲▲▲▲▲▲ ▲▲▲
▲
▲
▲
▲▲▲▲▲▲▲▲▲▲ ▲▲▲
▲
▲
▲
◀◀◀◀◀◀◀◀◀ ◀ ◀ ◀◀
◀
◀
◀
◀◀◀◀◀◀◀◀◀ ◀ ◀ ◀ ◀ ◀◀
◀
▼▼▼▼▼▼▼▼▼▼ ▼ ▼ ▼▼
▼
▼
▼▼▼▼▼▼▼▼▼▼ ▼ ▼ ▼▼
▼
▼
▶▶▶▶▶▶▶▶▶ ▶ ▶ ▶ ▶▶
▶
▶
▶▶▶▶▶▶▶▶▶ ▶ ▶ ▶ ▶▶
▶
▶
Temporarily SHM
0 20 40 60 80
0
5
10
15
20
25
Throughput [kop/s]
■■■■■■■■■■■■■ ■ ■■ ■
■
■
■
■■■■■■■■■■■■■ ■ ■■■
■
■
■
▲▲▲▲▲▲▲▲▲▲▲▲▲▲ ▲▲▲
▲
▲
▲
▲▲▲▲▲▲▲▲▲▲▲▲▲▲ ▲▲▲
▲
▲
▲
◀◀◀◀◀◀◀◀◀◀◀◀◀ ◀ ◀◀◀
◀
◀
◀
◀◀◀◀◀◀◀◀◀◀◀◀◀ ◀ ◀ ◀ ◀ ◀◀
◀
▼▼▼▼▼▼▼▼▼▼▼▼▼▼ ▼ ▼ ▼ ▼
▼
▼
▼▼▼▼▼▼▼▼▼▼▼▼▼▼ ▼ ▼ ▼▼
▼
▼
▶▶▶▶▶▶▶▶▶▶▶▶▶ ▶ ▶ ▶ ▶ ▶
▶
▶
▶▶▶▶▶▶▶▶▶▶▶▶▶ ▶ ▶ ▶ ▶ ▶
▶
▶
Inside REE

Evaluation
OP-TEE Interface Sockets
EL0
EL1
EL2
EL3
1
2
3
45
6
7
8
9
EL0
EL1
EL2
EL3
Shared memory
Socket
tee-supplicant
libc
Driver
OP-TEEnet
Driver
Secure Monitor
PTA
OP-TEE
OS
TA
libutee
OP-TEE Interface Socket
– Use Socket API
— System call
˜ Delegate request
™ SMC
š Map data and switch
› regular ioctl call
œ System call
Network driver
ž Send/receive data

Evaluation
OP-TEE Interface Socket Results
0 200 400 600
0
20
40
60
80
100
120
Throughput [Mbit/s]
Latency[s]
■■■■■■ ■ ■ ■ ■ ■
■
■
■
▲▲▲▲▲▲▲ ▲ ▲ ▲
▲
▲
iperf3■ iperfTZ▲
0
1000
2000
3000
Energy[J]
1 2 4 8 16 32 64 128 256 512 1024 2048
Bit rate [Mbit/s]
iperf3
iperfTZ
QEMU
0 20 40 60 80 100
0
10
20
30
40
50
Throughput [Mbit/s]
Latency[s]
■■■■ ■ ■ ■
■
■
■
▲▲▲▲ ▲ ▲ ▲ ▲
▲
▲iperf3■ iperfTZ▲
0
20
40
60
80
100
120
Energy[J]
1 2 4 8 16 32 64 128 256 512
Bit rate [Mbit/s]
iperf3
iperfTZ
Raspberry Pi
QEMU 33 % energy overhead, Raspberry Pi 11 % energy overhead.

Evaluation
Improvements
EL0
EL1
EL2
EL3
1
2
3
456
7
EL0
EL1
EL2
EL3
tee-supplicant
libc
Driver
OP-TEEnet
Driver
Secure Monitor
PTA
OP-TEE
OS
TA
libutee
Forward network requests directly
to the network driver
3
EL0
EL1
EL2
EL3
1
2
4
EL0
EL1
EL2
EL3
tee-supplicant
libc
Driver
OP-TEEnet
Driver
Secure Monitor
OP-TEE
OS
TA
libutee
net
Driver
Use a network driver in the
trusted OS

Conclusion Future Work
Conclusion:
Performance and energy evaluation of TEEs
Shared memory, interface sockets
Implemented a network performance tool for trusted
applications in Arm TrustZone
Highlight bottlenecks in current OP-TEE design
Future:
Explore additional TrustZone features with diﬀerent SoCs
Detailed breakdown of energy consumption
Comparison with Intel SGX and AMD SEV

Thank you
Thank you for your attention!
Code is available under https://github.com/ChrisG55/iperfTZ
The research leading to these results has
received funding from the European Union’s
Horizon 2020 research and innovation
programme under the LEGaTO Project
(legato-project.eu), grant agreement No 780681.

iperfTZ: Understanding Network Bottlenecks for TrustZone-based Applications

Recommandé

Recommandé

Contenu connexe

Similaire à iperfTZ: Understanding Network Bottlenecks for TrustZone-based Applications

Similaire à iperfTZ: Understanding Network Bottlenecks for TrustZone-based Applications (20)

Plus de LEGATO project

Plus de LEGATO project (20)

Dernier

Dernier (20)

iperfTZ: Understanding Network Bottlenecks for TrustZone-based Applications