2018 saw fundamental shifts in security boundaries which were previously taken for granted. A lot of work has been done in the past 2 years, and largely in secret under embargo, but there is plenty more work to be done to strengthen the existing mitigations and to try to recover some performance without reopening security holes. This talk will look at speculative execution sidechannels, the work which has already been done to mitigate the security holes, and future work which hopes to bring some improvements.

1. Speculative Sidechannels and Mitigations
Andrew Cooper
Citrix Hypervisor
Wednesday 10th July 2019
Andrew Cooper (Citrix Hypervisor) Speculative Sidechannels and Mitigations Wednesday 10th
July 2019 1 / 10

2. Speculative Vulnerabilities
Bounds Check Bypass, Spectre v1
Branch Target Injection, Spectre v2
Rogue Data Cache Load, Meltdown
Rogue System Register Read
Bounds Check Bypass Store, Spectre v1.1
Read-only Protection Bypass, Spectre v1.2
SpectreRSB
NetSpectre
LazyFPU
L1 Terminal Fault, Foreshadow
Microarchitectural {Load Port, Store Buffer, Fill Buffer, Uncacheable
Memory} Data Sampling, Rogue In-Flight Data Load, Write Transient
Forwarding, ZombieLoad
Not speculative: TLBleed, SPOILER
Andrew Cooper (Citrix Hypervisor) Speculative Sidechannels and Mitigations Wednesday 10th
July 2019 2 / 10

3. Speculation
What is speculation?
To perform a task, in the anticipation of it being needed.
e.g. Speculative Decode or Execution.
Why speculate?
Performance
Hide the latency of longer operations
Vulnerabilities: Two categories
Incorrect prediction ⇒ e.g. execute the wrong instructions
Deferred fault checking ⇒ e.g. execute past a fault
Shared CPU resources
Branch predictors, some shared, some statically partitioned
Internal data busses, some shared, some statically partitioned
L1D cache shared, coherent across the system
Andrew Cooper (Citrix Hypervisor) Speculative Sidechannels and Mitigations Wednesday 10th
July 2019 3 / 10

4. Bounds Check Bypass - Spectre v1
Attacker poisons/trains the conditional branch predictor.
Predicts conditional jumps ⇒ Jcc rel, JCXZ rel
CPU starts speculatively executing the wrong basic block.
Classic example is an OoB array read, and second dependent read.
Easily constructed in JIT code. Less common in compiled code.
Mitigations
Arrays ⇒ create data dependency for the index value.
Problematic with data-value speculation.
General ⇒ LFENCE after the Jcc instruction.
Usually need to fence both basic blocks.
Hardware fix is still an open problem.
It is context dependent which data values are safe to speculate on.
Some gadgets are very common and very useful to an attacker.
Type confusion ⇒ if ( is_pv(v) ) { ... } else { ... }
Arbitrary cache load ⇒ Single OoB array read.
Andrew Cooper (Citrix Hypervisor) Speculative Sidechannels and Mitigations Wednesday 10th
July 2019 4 / 10

5. Branch Target Injection - Spectre v2
Attacker poisons the Branch Target Buffer
Predicts indirect branches ⇒ CALL r/m, JMP r/m
Take control of function pointers, switch jump tables.
Redirect speculative execution to an attacker chosen location.
Software workaround ⇒ Retpoline
RET has dedicated predictions to match previous CALL’s
RSB - Return Stack Buffer (Intel), RAS - Return Address Stack (AMD)
Not safe on parts where an RSB underflow falls back to the BTB.
Microcode workaround ⇒ new functionality in MSRs
ABI designed for future hardware fix and existing microcode capabilities
IBRS - Prevent poisoning from influencing more privileged code.
Set on every entry, clear on exit to user/guest. Expensive and invasive.
STIBP - Prevent cross-thread BTB poisoning.
IBPB - Flush the BTB. Very expensive, used on vcpu context switch.
OS/Hypervisor needs to opt in to Enhanced IBRS on newer hardware.
Andrew Cooper (Citrix Hypervisor) Speculative Sidechannels and Mitigations Wednesday 10th
July 2019 5 / 10

6. Rogue Data Cache Load - Meltdown
Faults from page protections (U/S, R/W) deferred until retirement.
Attacker constructs a pointer into kernel space and reads from it.
TLB and L1 cache hits ⇒ data forwarded to dependant instructions.
Able to leak via cache timing attack, etc.
Attack pulled off from userspace, with unprivileged instructions!
To mitigate, must prevent one of the two hits.
Flush L1D cache on exit to user/guest, or
Flush TLB on exit to guest/user, or
Put all sensitive data in uncached memory.
Split user and kernel pagetables, switch on every context switch.
Expensive and invasive, but it does prevent TLB hits.
Inspired by earlier KAISER paper. KPTI/XPTI/KVAS.
Overhead can reduced with Process Context ID support.
PCID switches don’t flush the TLB.
Fixed in hardware which enumerates RDCL_NO.
Andrew Cooper (Citrix Hypervisor) Speculative Sidechannels and Mitigations Wednesday 10th
July 2019 6 / 10

7. Brief mentions
Spectre v1 variations
Out of bounds write ⇒ speculative stack smashing.
Speculatively clobbered return address does the attacker-helpful thing.
Combine with Meltdown ⇒ speculative update to read-only data.
Speculative Store Bypass
Better described as “Memory Access Misprediction”.
Load moved ahead of a dependent store. Executes with stale data.
Problematic when a stack slot is reused for a new object.
NetSpectre
Demonstrated Spectre-v1 timing attacks via network latency.
AVX frequency sidechannel. Speculative decode of an AVX instruction
drops the CPU frequency until the vector pipeline powers up.
LazyFPU
CPU speculates past #NM (Device Not Available) exception.
Leaks the previous tasks FPU registers.
EagerFPU is actually a performance win.
Andrew Cooper (Citrix Hypervisor) Speculative Sidechannels and Mitigations Wednesday 10th
July 2019 7 / 10

8. L1 Terminal Fault - Foreshadow
Terminal Fault is a pagewalk which has no valid translation.
Pagewalk speculatively looks up the next-level address.
Either the next PTE on the walk, or the target memory address.
Performs an L1D lookup before checking Present/RSVD bits.
Attacker constructs a not-present mapping and reads from it.
L1 cache hit ⇒ data forwarded to dependant instructions.
Bypasses all SMM, EPT and SGX protections!
Page Size bit is considered ⇒ speculative superpages.
Userspace can use mprotect() or know the paging-out algorithm.
Guest kernel can construct pagetables directly.
Mitigations totally different for native and virtualised cases.
Native (inc. Xen PV): Real L1D width generally larger than reported.
Invert all bits for not-present PTEs ⇒ L1D miss.
Virtualised (HVM): Hypervisor can’t control guest PTEs.
Disable HT. New microcode MSR to flush L1D on VMEntry.
Fixed in hardware which enumerates RDCL_NO.
Andrew Cooper (Citrix Hypervisor) Speculative Sidechannels and Mitigations Wednesday 10th
July 2019 8 / 10

9. Microarchitectural Data Sampling -
M{LP,SB,FB}DS/MDSUM/Fallout/RIDL/ZombieLoad
Faulting or assisting load forwards stale data from a buffer.
Attacker constructs a misaligned pointer which faults or sets an A/D
bit, and reads from it.
Store Buffer ⇒ likely the content of the most recent XSAVE.
Load/Fill Buffers ⇒ memory operands from other thread.
Includes data from uncached memory ⇒ No longer safe for secrets.
Mitigations
Legacy VERW instruction given new flushing side effect.
Use on return to user/guest path to flush uarch buffers.
Synchronised Scheduling (synchronise interrupt/exception/vmexit
handling to ensure you are never running code of two different privilege
levels concurrently), or disable HT.
Fixed in hardware which enumerates MDS_NO.
Fill Buffer subset already fixed with RDCL_NO.
Andrew Cooper (Citrix Hypervisor) Speculative Sidechannels and Mitigations Wednesday 10th
July 2019 9 / 10

10. Current and future work
Complete
Retpoline, IBRS, XPTI ⇒ XSA-254
SSBD ⇒ XSA-263
Eager FPU ⇒ XSA-267
PV-L1TF, no-SMT ⇒ XSA-273
MD-CLEAR ⇒ XSA-293
In development
Core-aware scheduling
MSR ARCH CAPS virtualisation for guests
Per-guest SSBD settings on AMD
“half-spectre-v1” hardening
Future
Enhanced IBRS for Intel
Removal of mappings in Xen
Andrew Cooper (Citrix Hypervisor) Speculative Sidechannels and Mitigations Wednesday 10th
July 2019 10 / 10

11. Questions
Andrew Cooper (Citrix Hypervisor) Speculative Sidechannels and Mitigations Wednesday 10th
July 2019 11 / 10