Presented @ BSidesDC 2013 Washington, DC, October 20, 2013 Measuring the performance of network protocols that require determinism can be difficult with the existing set of tools. Tools like Wireshark can give you the details of the protocols themselves and some general statistics about the packet streams, but they don’t easily show the full set of traffic for those streams. Visual tools like Etherape can show you the full set of traffic streams, but don’t give you any idea of the nuances of the performance represented in those traffic streams. While at the National Institute of Standards and Technology (NIST), I built a tool capable of analyzing and displaying the performance of network protocols. The first generation of the tool was called the Industrial Ethernet Network Performance (IENetP) test tool and the second generation of the tool is called the Factory Equipment Network Testing (FENT) framework. Both these tools are available on SourceForge and are public domain. I have since left NIST and the tools haven’t been picked up by anyone. The FENT framework is useful for analyzing the performance of any deterministic protocol and reporting certain performance characteristics. It was originally designed for EtherNet/IP (Ethernet / Industrial Protocol), Modbus, Profinet, and other industrial Ethernet-based protocols, but has proven to be useful for many other protocols as well. The most beneficial part of the software has actually been the graphical analysis, which in many cases resemble Rorschach plots due to the subtle performance problems that show up as strange patterns in the data. My presentation will describe the FENT framework, present the tool in its current state, and display some of the more interesting results. It will also be a plea for someone to take up the open-source development of this project and move it forward. My new position does not leave me with enough time to dedicate to the project, so the project has been dormant for the last few months. I’ve received complements on the project from many industrial partners in the past and they would like to see further development, but that means that someone else has to take on the task.

1. Rorschach Plots and Network
Performance Analysis
Jim Gilsinn
Kenexis Consulting Corporation
October 19-20, 2013
BSidesDC 2013
1

2. Rorschach?
October 19-20, 2013
BSidesDC 2013
2

3. “Rorschach” Plots
October 19-20, 2013
BSidesDC 2013
3

4. “Rorschach” Plots
October 19-20, 2013
BSidesDC 2013
4

5. “Rorschach” Plots
October 19-20, 2013
BSidesDC 2013
5

6. ICS Environment
October 19-20, 2013
BSidesDC 2013
6

7. ICS Environment
October 19-20, 2013
BSidesDC 2013
7

8. ICS Systems
October 19-20, 2013
BSidesDC 2013
8

9. What’s This All About?
• I used to work at NIST
• I left about a year ago
• I worked on ICS network performance metrics, tests, and tools
• The test tools I developed have been dormant since leaving
• The vendors I worked with while at NIST want to tool
• My new employer won’t support open-source development
• I’m here to beg for help!
October 19-20, 2013
BSidesDC 2013
9

10. Performance Testing Methodology:
Performance Metrics
• Publish/subscribe or peer-to-peer communications
• Main performance metric: Cyclic frequency variability/jitter
• Real-time EtherNet/IP uses publish/subscribe
• Requested/Accepted Packet Interval (RPI/API)
• Measured Packet Interval (MPI)
October 19-20, 2013
BSidesDC 2013
10

11. Performance Testing Methodology:
Performance Metrics
• Command/response or master/slave communications
• Main performance metric: Latency
• Large numbers of protocols use this
• Most (All?) PC-based server/client protocols – HTTP(S), (S)FTP, etc.
• Most industrial protocols – Modbus/TCP, Profinet, Ethercat, etc.
October 19-20, 2013
BSidesDC 2013
11

12. IENetP Test Tool
• Industrial Ethernet Network Performance (IENetP)
• http://sourceforge.net/projects/ienetp/
• Current Version = 1.1.2, Released 2011-02-11
• Software Features
•
•
•
•
•
•
•
Analyze existing Wireshark captures
Allows user to override default EtherNet/IP filter
Isolates individual traffic streams
Determine cyclic jitter of those streams
Generates HTML report
Generates time-space & histogram graphs
Graphs allow zooming
October 19-20, 2013
BSidesDC 2013
12

13. NIST Performance Test Tool
• Industrial Ethernet Network Performance (IENetP) Test Tool
• Factory Equipment Network Testing (FENT) Framework

14. FENT Framework
Universal Client Application
Testing
Testing
Module
Testing
Module
Testing
Module
Module
Analysis
Analysis
Engine
Engine
Reporting
Engine
Universal Client Application API
Personality
Module
Personality
Module
Personality
Module
Ethernet
Sensor
Gateway
Internet
Fieldbus
October 19-20, 2013
BSidesDC 2013
14

15. FENT Features
• All Analysis Features From IENetP
• Analyze Wireshark capture files
• Build graphs and reports of results
• Added Features
• True multi-protocol support
• Real-time testing capability
• Extensible framework
October 19-20, 2013
BSidesDC 2013
15

16. FENT Personality Modules
• Wrapper for Driver
Application
• Implement a TCP-socket
interface for UCA-API
messaging
• Build a simple XML-based PM
Descriptor file
UCA – API
Protocol PM
PM Descriptor
• Features
• Describes Wireshark
parameters
• Allows any protocol to be used
• Can be built/loaded at run-time
October 19-20, 2013
BSidesDC 2013
Driver App
16

17. FENT Framework Run-Time
UCA
Testing
Module
Analysis
Engine
1.
Reporting
Engine
2.
PSML File
3.
4.
UCA – API
Wireshark
Protocol PM
PM
Descriptor
5.
Driver
App
6.
Testing Module  Protocol PM –
Grab protocol-specific Wireshark
parameters via UCA-API
Testing Module  Wireshark – Start
capturing traffic
Testing Module  Protocol PM –
Command driver app to communicate
with DUT
Testing Module  Wireshark – Stop
Capturing traffic, process capture file
using desired protocol and user
parameters, generate PSML file
Analysis Engine – Read PSML file,
analyze packets for desired metrics
Reporting Engine – Report data to
user
DUT
October 19-20, 2013
BSidesDC 2013
17

18. FENT UCA-API Schema
October 19-20, 2013
BSidesDC 2013
18

19. FENT Framework
• Project Home:
• http://sourceforge.net/projects/fent/
• What’s Available:
• SVN repository & schema
• FENT software
• Conduct real-time testing
• Analyze results
• Build graphs on-screen
• NIST SensorSim PM, IEEE 1451 PM
• EtherNet/IP PlugFest “Gold Standard” Background Traffic
October 19-20, 2013
BSidesDC 2013
19

20. FENT Framework
• Known Problems & Issues
• Doesn’t work with Wireshark 1.9+
• Tshark argument for getting fields changed
•
•
•
•
Logic problems with using multi-protocol Wireshark headers
Software doesn’t use true database
Testing automation not integrated
No installer
October 19-20, 2013
BSidesDC 2013
20

21. FENT Demo
October 19-20, 2013
BSidesDC 2013
21

22. “Gold Standard” Background Traffic
October 19-20, 2013
BSidesDC 2013
22

23. “Gold Standard” Background Traffic
• What Is It?
• A set of Wireshark captures, Linux scripts, and analysis results
• Based on EtherNet/IP PlugFest performance testing requirements
• High precision and accuracy Wireshark captures of PlugFest
performance background traffic
• Linux scripts designed for use in BackTrack Linux
(http://www.backtrack-linux.org/)
• Analysis results show validation for use in PlugFest performance
testing
• Where Can You Get It?
• http://ienetp.sourceforge.net/EtherNet-IP_Testing.zip or
• FENT SVN in Background_Traffic folder
October 19-20, 2013
BSidesDC 2013
23

24. ARP Request Broadcasts
180
Gratuitous ARP Broadcasts
180
DHCP Request Broadcasts
100
ICMP (ping) Request Broadcasts
100
NTP Multicasts
10
EtherNet/IP ListIdentity Request
10
EtherNet/IP Class 1
1800
ARP Burst Requests
240 pkts @
4k Hz
Burst
Unmanaged
Burst Managed
Steady-State
Unmanaged
Rate (pps)
Steady-State
Managed
Traffic Type
Baseline
PlugFest Performance Traffic

25. “Gold Standard” Captures
• Built From Individual Traffic Streams
• Each traffic stream generated and captured using NIST Ixia system (a
few microseconds jitter)
• Assembled using editcap and mergecap scripts
• Final captures are 60-seconds long
• Can’t just loop continuously
• Longer test captures require rebuilding (not hard)
• Analyzed Using IENetP
• Analysis results are included in package
• Well within spec for PlugFest performance testing needs (<25% of
desired packet intervals)
October 19-20, 2013
BSidesDC 2013
25

26. Licensing?
• The project is Public Domain!!!
• There are NO LICENSING ISSUES!!!
October 19-20, 2013
BSidesDC 2013
26

27. What’s Next?
• Contact Me
•
•
•
•
•
Jim Gilsinn
301-706-9985
jim.gilsinn@kenexis.com
Twitter – @JimGilsinn
LinkedIn – http://www.linkedin.com/in/jimgilsinn/
• Review the FENT SourceForge Project
• http://sourceforge.net/projects/fent/
• Fork the Project
October 19-20, 2013
BSidesDC 2013
27