2. Agenda
Motivations
E
Types of Attacks
D
C
B
IOS architecture
A
9
Detection of Attacks
8
7
Challenges with IOS
6
5
4
Methods of overcoming some issues
3
2
IOS shellcode
1
0
Introducing the Black-Hat-O-Meter
3. Why Cisco?
This talk is Cisco centric
E 92% market share* for routers above $1,500
D
71% market share* enterprise switch market
C
B
This talk is access layer equipment centric
A
9
Small boxes, PowerPC based
8
7
What about Juniper?
6
5
From both attacker and forensics point of view, Juniper routers
4
are just FreeBSD
3
2
What about <someCheapHomeRouter>
1
0
From both attacker and forensics point of view, they are just
embedded Linux systems
*Source: stolen randomly
4. Who would hack routers?
E
D
C
ARP games
B
A blocked by
9 the switch.
8
BBI
7
6
(The Big Bad Internet)
5
Switch
4 „Behind“ the
separates
3 Firewall
2 Hosts
1
0
Neighbor
systems have
local firewalls.
5. Who would hack routers?
E
D
C
B
A
9
8
BBI
7
6
(The Big Bad Internet)
5
4
3
2
1 Separation broken (ARP tricks are transparent now)
0
Modification of any traffic
Hard to recognize from the host
There just is no Reverse-NAC.
6. Who would hack routers?
E
D
C
B
A
9
8
BBI
7
6
(The Big Bad Internet)
5
4
3
2
1
Control over the entire network
0
Impersonation of the network against
the Internet
7. And on a larger scale...
E
D
C
B
The Internet
A
9
8
7
6
5
(OK, maybe this is too large)
4
3
2
1
0
8. Inter-Network Security
Network Firewall,
IDS, IPS
E
D
C EIGRP 3
EIGRP 2
B
Ingress & Egress
A
9 Filtering,
8 anti-spoofing,
7 OSPF
route redistribution
6
5
4
3
2 Full Trust
1 EIGRP 1
within the EIGRP 4
0
autonomous
system
9. Network Security
Network security is hierarchical
E
Defending against your downstream is common
D
C
Defending against your upstream is rather hard
B
A
Defending against your peers is rare
9
8
Control anything in the hierarchy and you control
7
6
everything below
5
4
3
2
1
0
10. Hierarchical Compromises
(_x_)
E
D Local network
C compromise
EIGRP 3
EIGRP 2
B
A
9
8
7 OSPF
6
5
4
3
2
1 EIGRP 1
EIGRP 4
0
11. Hierarchical Compromises
(_x_)
E
D Just another
C EIGRP 3
router
EIGRP 2
B
A
9
8
7 OSPF
6
5
4
3
2
1 EIGRP 1
EIGRP 4
0
12. But we got <secureProtocol>
Secure protocols can guarantee that nobody
E
…modified the protocol messages
D
C
…spoofed the communication peer
B
A
…replayed the protocol messages
9
8
But if someone did exactly that, they cannot do
7
6
anything about it.
5
4
3 The choice is: Availability or Security
2
1
What would your boss / mom do?
0
13. But we got <secureProtocol>
(_x_)
E
D
If the user could
C EIGRP 3
EIGRP 2
B control the path his
A communication is
9
using, it would be
8
called „source routing“
7 OSPF
6 and there is a reason
5
this is no longer in use
4
anywhere in the
3
2 Internet: The user
1 would have power over
EIGRP 1
EIGRP 4
0
the network.
14. All this is by design
E
In IP networks
D
C
The network node makes the forwarding decisions
B
A
The leaf node cannot control the traffic flow
9
8
7
6
5
4
3
2
1
0
15. Attacker Motivation
Windows and UNIX become harder targets
E
IOS boxes are going to be around for some time
D
C
B We don’t see a new IOS for all the metal out there
A
9
IOS attack surface increases constantly
8
7
12.4 enterprise default feature set runs out of the box
6
5
a full Voice-XML IVM
4
3 New protocols constantly “invented”
2
1
Backdoored IOS images become popular
0
We need ways to detect and handle intrusions
16. What Type of Attacker?
Infrastructure is not attacked for a quick hack
Development of reliable IOS exploits costs too much for quick
E
hacks
D
C The chance of wasting a 0day exploit is too high
B
What an infrastructure attacker wants is a solid foothold
A
9 Gain access to the infrastructure any time in the future
8
Be able to shut down the network at any given time
7
6 Stay undetected
5
According to estimates by F-Secure, modern Rootkits for
4
3 Windows cost about 40.000 € in development
2
IOS exploit development begins to make commercial sense for
1
an organization with offensive capabilities (three letters of your
0
favorite UNICODE page)
17. Types of Attacks
Protocol based attacks
E
D
Functionality attacks
C
B
Binary exploitation
A
9
8
7
6
5
4
3
2
1
0
18. Protocol attacks
Injection of control protocol messages into the
network (routing protocol attacks)
E
D
C Attacker becomes part of the network’s internal
B
communication
A
9
Attacker influences how messages are forwarded
8
7
Typical examples include:
6
5
ARP poisoning
4
3
DNS poisoning
2
1
Interior routing protocol injections (OSPF, EIGRP)
0
Exterior routing subnet hijacking (BGP)
19. Functionality attacks
Configuration problems
E
Weak passwords (yes, they are still big)
D
C Weak SNMP communities
B
Posting your configuration on Internet forums
A
9
8 Access check vulnerabilities
7
6
Cisco’s HTTP level 16++ vulnerability
5
SNMPv3 HMAC verification vulnerability (2008!)
4
3
memcmp( MyHMAC, PackHMAC, PackHMAC_len );
2
1
Debianized SSH keys
0
Queuing bugs (Denial of Service)
20. Binary exploitation
Router service vulnerabilities:
E
D Phenoelit’s TFTP exploit
C
B
Phenoelit’s HTTP exploit
A
9
Andy Davis’ FTP exploit
8
7
6
Router protocol vulnerabilities:
5
4
Phenoelit’s OSPF exploit
3
2
1
Michael Lynn’s IPv6 exploit
0
21. Detection and Monitoring
SNMP
Polling mechanisms, rarely push messages (traps)
E
D
Syslog
C
B Free-form push messages
A
Configuration polling
9
8
Polling and correlation
7
6
Route monitoring and looking glasses
5
4 Real-time monitoring of route path changes
3
Traffic accounting
2
1
Not designed for security monitoring, but can yield
0
valuable information on who does what
22. Who detects what?
SNMP Syslog Config Route Traffic
polling monitoring accounting
E
D Poisioning Yes Yes - Yes Yes
C attacks
B
Interrior routing Yes Yes (rare) - Yes Yes
A
attacks
9
8 Exterrior routing Yes Yes - Yes Yes
7 attacks
6
5 Illegal access Yes Yes Maybe - -
4 due to config
3 issues
2
Access check - Yes Maybe - -
1
0 vulns
Binary exploits - - Maybe - -
(if stupid)
23. The Common Solution
Centrally log everything
E
The interesting information is in the debug messages.
D
C Too many, too slow
B
Who wades through the logs?
A
9
Messages keep changing over IOS releases
8
7
SNMP
6
5
Doesn’t contain the information you need to decide if
4
3 you are looking at a regular crash or an attack
2
1
Attempting to detect the exploitation while it
0
happens has proven to suck badly
24. But there is Crashinfo
If the exploit failed, you might get a crashinfo file
E
Not all IOS releases write crash-info files
D
C
Is there enough space on the flash: device?
B
A
9
Crash-info is for Cisco IOS coders, not for
8
7
forensics
6
5
Stack trace is misleading at best in more than 80% of
4
3 all crash cases (software forced reload)
2
1
After exploitation of heap overflows, the wrong heap
0
sections are shown
It’s just not enough info for forensics
25. What do binary exploits do?
Binary modification of the runtime image
E
Patch user access credential checking (backdoor)
D
C
Patch logging mechanisms
B
A
Patch firewall functionality
9
8
Data structure patching
7
6
5
Change access levels of VTYs (shells)
4
3 Bind additional VTYs (Michael Lynn’s attack)
2
1
Terminate processes
0
It actually depends … we will come back to it
26. Forensics for Binary Exploits
What we need:
E
D
Evidence acquisition
C
B
Recovering of information from raw data
A
9
8
Analysis of information
7
6
5
Plus:
4
3
2
Good understanding of Cisco IOS
1
0
internals
27. Inside Cisco IOS
One large ELF binary
Essentially a large, statically linked UNIX program, loaded by ROMMON
E
D Runs directly on the router’s main CPU
C
If the CPU provides privilege separation, it will not be used
B
e.g. privilege levels on PPC
A
9 Virtual Memory Mapping will be used, minimally
8
Processes are rather like threads
7
No virtual memory mapping per process
6
5
Run-to-completion, cooperative multitasking
4
3 Interrupt driven handling of critical events
2
System-wide global data structures
1
0
Common heap
Very little abstraction around the data structures, no way to force
28. Cisco IOS Device Memory
IOS devices start from the ROMMON
E Loading an IOS image from Flash or network into RAM
D
The image may be self-decompressing
C
B
The image may contain firmware for additional hardware
A
9
Configuration is loaded as ASCII text from NVRAM or
8
7 network
6
5 Parsed on load
4
Mixed with image version dependent defaults of configuration
3
2 settings
1
Everything is kept in RAM
0
Configuration changes have immediate effect
Configuration is written back into NVRAM by command
29. Evidence Acquisition
Common operating system:
E
Most evidence is non-volatile
D
C
Imaging the hard-drive is the acquisition method
B
A
Capturing volatile data is optional
9
8
Cisco IOS:
7
6
5
Almost all evidence is volatile
4
3 What we need is memory imaging
2
1
On-demand or when the device restarts
0
Restarting is the default behavior on errors!
30. Non-volatile Cisco Evidence
Flash file system
E
D If the attacker modified the IOS image
C
B statically
A
9
NVRAM
8
7
6
If the attacker modified the configuration and
5
4
wrote it back into NVRAM
3
2
Both cases are rare for binary exploits
1
0
31. Evidence Acquisition: Cores
Using debugging features for evidence
E
acquisition:
D
C
IOS can write complete core dump files
B
A
Dump targets: TFTP (broken), FTP, RCP, Flash
9
8
Complete dump
7
6
Includes Main Memory
5
4 Includes IO Memory
3
2 Includes PCI Memory
1
Raw dump, perfect evidence
0
32. Evidence must be configured
Core dumps are enabled by configuration
Configuration change has no effect on the
E
D router’s operation or performance
C
Configure all IOS devices to dump core onto one or
B
A
more centrally located FTP servers
9
8
Minimizes required monitoring of devices
7
6 Preserves evidence
5
Allows crash correlation between different routers
4
3
Why wasn’t it used before?
2
1
Core dumps were useless, except for Cisco developers
0
and exploit writers
33. CIR – Cisco Incident Response
Publicly available core dump analyzer:
E
http://cir.recurity-labs.com
D
C
B
Currently supports 1700 and 2600 series
A
9
Server side processing of core dumps
8
7
6
Entirely written in .NET
5
4
We don’t want to get owned by malicious core
3
2 dumps
1
0
34. Rootkit Detection Arms Race
E
D
C Next Attack Detection
B
A Rootkit code patching core dump GDB debug protocol memory
9 writing acquisition
8
7 GDB debugger stub patching ROMMON privilege mode memory
6
acquisition
5
4
3
2
1
0
35. The Image Blueprint
The IOS image (ELF file) contains all required
E
information about the memory mapping on the router
D
C The image serves as the memory layout blueprint, to be applied
B
to the core files
A
9 We wish it were as easy as it sounds
8
7 Using a known-to-be-good image also allows verification
6
of the code and read-only data segments
5
4
Now we can easily and reliably detect runtime patched images
3
2
1
0
36. Image vs. Core
IO Memory
ELF Header
E
D Code Segment Code Segment
C
B
A
9
Read-Only Data Read-Only Data
8
7
6
5
4 Data Data
3
2
1
0
BSS data
37. Simple Detections Work Best
Recurity Labs CIR vs. Topo‘s DIK
E
D
(at PH-Neutral 0x7d8)
C
B
A
9
8
7
6
5
4
3
2
1
0
CIR Online case: 120EF269A5BC2320730E60289A4B84D9047CECEE
38. Heap Reconstruction
IOS uses one large heap
E
The IOS heap contains plenty of meta-data for
D
C
debugging purposes
B
A
40 bytes overhead per heap block in IOS up to 12.3
9
8 48 bytes overhead per heap block in IOS 12.4
7
6
Reconstructing the entire heap allows extensive
5
4 integrity and validity checks
3
2 Exceeding by far the on-board checks IOS performs
1
during runtime
0
Showing a number of things that would have liked to
stay hidden in the shadows
39. Heap Verification
Full functionality of “CheckHeaps”
Verify the integrity of the allocated and free heap block doubly
E
D linked lists
C
B Find holes in addressable heap
A
Invisible to CheckHeaps
9
8
Identify heap overflow footprints
7
6 Values not verified by CheckHeaps
5
Heuristics on rarely used fields
4
3
Map heap blocks to referencing processes
2
1
Identify formerly allocated heap blocks
0
Catches memory usage peaks from the recent past
40. Process List
Extraction of the IOS Process List
E
Identify the processes’ stack block
D
C Create individual, per process back-traces
B
Identify return address overwrites
A
9
Obtain the processes’ scheduling state
8
7
Obtain the processes’ CPU usage history
6
5
Obtain the processes’ CPU context
4
3
Almost any post mortem analysis method known
2
can be applied, given the two reconstructed data
1
0
structures.
41. TCL Backdoor Detection
We can extract any TCL script “chunk”
E
from the memory dump
D
C
B
Currently only rare chunks
A
9
There is still some reversing to do
8
7
6
Potentially, a TCL decompiler will be required
5
4
3
2
1
0
42. IOS Packet Forwarding Memory
IOS performs routing either as:
Process switching
E
D Fast switching
C
Particle systems
B
Hardware accelerated switching
A
9
Entirely incomprehensible voodoo
8
At least access layer router all use IO memory
7
6
IO memory is written as separate code dump
5
4 By default, about 6% of the router’s memory is dedicated as IO
3 memory
2
Hardware switched packets use PCI memory
1
0
PCI memory is written as separate core dump
Bigger iron?
Should provide a respective core file as well
43. IO Memory Buffers
Routing (switching) ring buffers are grouped by packet
size
E
D
Small
C
B Medium
A
Big
9
8 Huge
7
Interfaces have their own buffers for locally handled
6
5
traffic
4
3
IOS tries really hard to not copy packets around in
2
memory
1
0
New traffic does not automatically erase older traffic in a
linear way
44. Traffic Extraction
CIR dumps packets that were process switched
E
by the router from IO memory into a PCAP file
D
C
Traffic addressed to and from the router itself
B
A
Traffic that was process switching inspected
9
8
Access List matching
7
6
QoS routed traffic
5
4
CIR could dump packets that were forwarded
3
2
through the router too
1
0
Reconstruction of packet fragments possible
Currently not in focus, but can be done
45. Challenges with IOS
The challenge with IOS is the combinatory explosion
of platform, IOS version, Feature Set and additional
E
D
hardware
C
B
Every IOS image is compiled individually
A
9
Over 100.000 IOS images currently used in the wild
8
7
(production networks)
6
5
Around 15.000 officially supported by Cisco
4
Cisco IOS is rarely updated and cannot be patched
3
2
This is a great headache for IOS forensics, but also
1
0
for IOS exploit writers
46. Reality Check IOS Exploits
The entire code is in the image
E
Remotely, you have a 1-in-100.000 chance to
D
C
guess the IOS image (conservative estimate)
B
A
Any exception causes the router to restart
9
8
This is inherent to a monolithic firmware design, as it
7
looses integrity entirely with a single error
6
5
Stacks are heap blocks
4
3
2 Always at different memory addresses
1
Addresses vary even within the same image
0
47. Reality Check IOS Exploits
So far, all IOS exploits published use fixed
E
addresses that depend on the exact IOS image
D
C
being known before the attack
B
A
IOS’s address diversity is a similar “protection” as the
9
8
Source Port Randomization patch you applied to your
7
DNS servers in summer 2008
6
5
4
Performing your own research in this area is vital
3
2
to understand weaponized exploits
1
0
It is always hard to detect something you could not
get to work yourself
48. Where to (re)turn to?
The complete address layout changes with
E
every image
D
C
B
IO memory even changes based on
A
9
configuration and is not executable
8
7
6
5 Start End Size(b) Class Media Name
4 0x03C00000 0x03FFFFFF 4194304 Iomem R/W iomem
3 0x60000000 0x60FFFFFF 16777216 Flash R/O flash
2 0x80000000 0x83BFFFFF 62914560 Local R/W main
1
0x8000808C 0x8095B087 9777148 IText R/O main:text
0
0x8095B088 0x80CDBFCB 3673924 IData R/W main:data
0x80CDBFCC 0x80DECEE7 1117980 IBss R/W main:bss
0x80DECEE8 0x83BFFFFF 48312600 Local R/W main:heap
49. The ROMMON code
ROMMON code (System Version 11.3(2)XA4
Bootstrap) is mapped in
E Version 12.1(3r)T1
D
memory and stays there
C Version 12.1(3r)T2
B Version 12.2(10r)1
0xFFF00000 is the
A Version 12.2(6r)
9 exception vector base upon Version 12.2(7r) [cmong 7r]
8
startup, followed by Version 12.2(7r)XM1
7
ROMMON code
6 Version 12.2(8r) [cmong 8r]
5
4
Version distribution is much smaller
3
2
The figure shows System Bootstrap versions for the 2600
1
platform, based on Internet posted boot screen captures
0
ROMMON is almost never updated (and often cannot)
Versions depend on shipping data (bulk sales rocks!)
50. Return Oriented Programming*
Chaining together function epilogs before
E
return to gain arbitrary functionality
D
C
B
One of these hacking techniques that every
A
9
sufficiently talented hacker with a need came
8
7
up with independently
6
5
Has been shown to work nicely on IA-32
4
3
2
and SPARC code using an entire glibc
1
0
We have 146556 bytes (36639 instructions)
and a PowerPC CPU that returns via LR
* „Return-oriented Programming: Exploitation without Code Injection“
Erik Buchanan, Ryan Roemer, Stefan Savage, Hovav Shacham - University of California, San Diego
http://www.blackhat.com/presentations/bh-usa-08/Shacham/BH_US_08_Shacham_Return_Oriented_Programming.pdf
52. Too Much Cache
PowerPC has separate
E
instruction and data caches
D
C
Executing data you just wrote
B
A
doesn’t work
9 AAAA…AAAAA
8
7
6
5
4 memcpy() D-Cache Memory
3 return
2
1 CPU
AAAA…AAAAA
0
I-Cache
53. More Code Reuse
stwu %sp, -0x10(%sp)
The Bootstrap code mflr %r0
stw %r31, 0x10+var_4(%sp)
E stw %r0, 0x10+arg_4(%sp)
already brings
D bl Disable_Interrupts
C mr %r31, %r3
mfspr %r0, dc_cst
functionality that we
B cmpwi cr1, %r0, 0
A
bge cr1, NoDataCache
9
need: bl Flush_Data_Cache
8 bl Unlock_Data_Cache
7 bl Disable_Data_Cache
Disable all caches! NoDataCache:
6
bl Invalidate_Instruction_Cache
5
bl Unlock_Instruction_Cache
4 bl Disable_Instruction_Cache
3 mfmsr %r0
2 rlwinm %r0, %r0, 0,28,25
IOS doesn’t care
1 mtmsr %r0
cmpwi cr1, %r31, 0
0
beq cr1, InterruptsAreOff
But we do! bl EnableInterrupts
InterruptsAreOff:
lwz %r0, 0x10+arg_4(%sp)
mtlr %r0
lwz %r31, 0x10+var_4(%sp)
addi %sp, %sp, 0x10
blr
54. Reliable Code Execution
IO Memory
AAAAAAAAAAAA
AAAAAAAAA…
mtctr SP
Globals
P bctr
Return oriented
mtctr S
E
Code Segment
Cache Disable
D 6
10
bctr
C Return oriented
B
FE
B memory write
E
A
xF
Return oriented
0
9 Read-Only Data
memory write
h
c
ar
8 Execute written
se
7 copy
data (code)
6
Second Stage
5
Data
Code:
4
3 Search for full
2 packet in
IO Memory
1
0
Run third stage
code
STACK
Heap
ROMMON
55. Reliability Notes
The return oriented ROMMON method is reliable
E
for a known System Bootstrap version
D
C
Successfully implemented an exploit for the IP
B
A
options vulnerability*
9
8
Successfully ported Andy Davis’ FTP server** exploit
7
to the method
6
5
4
The second stage code is actually less reliable:
3
2
Devices using the same ROMMON code may
1
0
place their IO Memory at different base
addresses (e.g. 2611 vs. 2621)
* cisco-sa-20070124-crafted-ip-option
** cisco-sa-20070509-iosftp
56. Getting away with it
Reliable code execution is nice, but an
E
attacker needs the device to stay running
D
C
B
Andy Davis et al have called the
A
9
TerminateProcess function of IOS
8
7
6
Needs the address of this function, which is
5
4
again image dependent
3
2
Exactly what is not wanted!
1
0
Crucial processes should not be terminated
IP Options vulnerability exploits “IP Input”
57. Getting away with it
41414141
Buffer
41414141
Buffer
Remember the stack layout?
41414141
Buffer
E
We search the stack for a stack frame
D 41414141
Buffer
C
sequence of SP&LR upwards
B VALUE R30
saved
A
DEST.PTR
saved R31
9 Once found, we restore the stack pointer
8 41414141 SP
saved
and return to the caller
7
FUNC_02 LR
saved
6
This is reliable across images, as the
5
saved R28
4
call stack layout does not change
3 saved R29
2
dramatically over releases
1 saved R30
0
saved R31
This has been shown to be mostly true
saved SP
on other well exploited platforms
saved LR
stuff
59. On IOS Shellcode
Image independent exploits require image
E
independent shellcode
D
C
B
Earlier, image dependent exploits use fixed
A
9
addresses for function calls and data
8
7
structures
6
5
Signature based shellcode by Andy Davis
4
3
searches code but still uses fixed data
2
1
structure offsets, which are not stable
0
60. Disassembling Shellcode
When searching for code manually, one
E
often follows string references
D
C
B
A
9
8
7
6
5
4
3
2
1
0
61. Disassembling Shellcode
Shellcode can do the same:
E
D 1. Find a unique string to determine its address
C
B
2. Find a code sequence of LIS / ADDI loading
A
9
the address of this string
8
7
3. Go backwards until you find the STWU %SP
6
5
instruction, marking the beginning of the
4
3
function
2
1
0
4. Patch the function to always return TRUE
63. IOS Shellcode Options
Port bind shell (VTY)
Full interaction + logging
E Requires port 22 or 23 open and reachable
D Doesn’t work for AAA configurations
C
Connect-back VTY shellcode
B
Full interaction + logging
A
9 Requires outgoing connections to the connect-back target
8 Doesn’t work for AAA configurations
7 Single command execution shellcode
6
One packet – one command
5
Requires no back-channel
4
Works with AAA configurations
3
2 Cannot change the configuration easily
1 Image patching shellcode
0
The most powerful and flexible method, but can get really big
Further work is required in this area, so we know what to
look for in forensics
64. Summary
The best defense is still to block traffic
terminating at the router’s interface
E
D
C
IOS forensic tools (e.g. CIR) are capable of
B
detecting current rootkits and shellcodes in
A
9
action, if they persist
8
7
Non-persistent exploits are really hard to detect
6
5
Reliable code execution is possible
4
3
At least on the PowerPC based platforms
2
1
It is highly likely that the $badguys have
0
significant better exploits at their disposal
65. Thanks
Nicolas Fischbach for pointing out Bootloader
and ROMMON code
E
D
Mac + souls for Cisco equipment
C
B
Cloudsky for the initial question on stack
A
9
overflow exploitation
8
7
Zynamics for BinDiff and BinNavi
6
5
nowin for finding and defending research time
4
3
2
Mumpi for awesomeness
1
0
Ilker from Cisco PSIRT for a presentation on
IOS attacks without the word “Phenoelit” in it
