Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1CpR2Pz. Martin Thompson focuses on the design of Aeron and what they learned trying to achieve very consistent performance. He explores the challenges of dealing with reliable message delivery over UDP and the data structures necessary to support transmission and retransmission in a lock-free and wait-free manner. Filmed at qconlondon.com. Martin Thompson is a high-performance and low-latency specialist, with experience gained over two decades working on large scale transactional and big-data systems. He believes in Mechanical Sympathy, i.e. applying an understanding of the hardware to the creation of software as being fundamental to delivering elegant high-performance solutions.

1. Aeron
High-Performance
Open Source
Message Transport
Martin Thompson - @mjpt777

2. InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
http://www.infoq.com/presentations
/aeron-messaging

3. Presented at QCon London
www.qconlondon.com
Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide

5. 1. Why build another Product?
2. What Features are really needed?
3. How does one Design for this?
4. What did we Learn on the way?
5. What’s the Roadmap?

6. 1. Why build another
product?

7. Not Invented Here!

8. There’s a story here...

9. Matching or
Trading
Engine
Matching or
Trading
Engine
Gateway
Clients
Gateway Gateway Gateway Gateway Gateway

10. Matching or
Trading
Engine
Matching or
Trading
Engine
Gateway
Clients
Gateway Gateway Gateway Gateway Gateway

11. But many others could benefit

12. Feature Bloat & Complexity

13. Not Fast Enough

14. Low & Predictable Latency is key

15. We are in a new world
Multi-core, Multi-socket,
Cloud...

16. We are in a new world
UDP, IPC, InfiniBand,
RDMA, PCI-e
Multi-core, Multi-socket,
Cloud...

17. Aeron is trying a new approach

18. The Team
Todd Montgomery Richard Warburton
Martin Thompson

19. 2. What features
are really needed?

20. Publishers SubscribersChannel
Stream
Messaging
Channel

21. A library, not a framework, on
which other abstractions and
applications can be built

22. Composable Design

23. OSI layer 4 Transport for
message oriented streams

24. OSI Layer 4 (Transport) Services
1. Connection Oriented Communication
2. Reliability
3. Flow Control
4. Congestion Avoidance/Control
5. Multiplexing

25. Connection Oriented Communication

26. Reliability

27. Flow Control

28. Congestion Avoidance/Control

29. Multiplexing

30. Multi-Everything World!

31. Publishers Subscribers
Channel
Stream
Multi-Everything World

32. 3. How does one
design for this?

33. Design Principles
1. Garbage free in steady state running
2. Smart Batching in the message path
3. Wait-free algos in the message path
4. Non-blocking IO in the message path
5. No exceptional cases in message path
6. Apply the Single Writer Principle
7. Prefer unshared state
8. Avoid unnecessary data copies

34. It’s all about 3 things

35. It’s all about 3 things
1. System Architecture

36. It’s all about 3 things
1. System Architecture
2. Data Structures

37. It’s all about 3 things
1. System Architecture
2. Data Structures
3. Protocols of Interaction

38. Publisher
Subscriber
Subscriber
Publisher
Architecture
IPC Log Buffer

39. Sender
Receiver
Receiver
Sender
Publisher
Subscriber
Subscriber
Publisher
Architecture
Media
IPC Log Buffer
Media (UDP, InfiniBand, PCI-e 3.0)

40. Conductor
Sender
Receiver
Conductor
Receiver
Sender
Publisher
Subscriber
Subscriber
Publisher
Admin
Events
Architecture
Admin
EventsMedia
IPC Log Buffer
Media (UDP, InfiniBand, PCI-e 3.0)
Function/Method Call
Volatile Fields & Queues

41. ClientMedia DriverMedia Driver
Conductor
Sender
Receiver
Conductor
Receiver
Sender
Client
Publisher
Conductor Conductor
Subscriber
Subscriber
Publisher
Admin
Events
Architecture
Admin
EventsMedia
IPC Log Buffer
IPC Ring/Broadcast Buffer
Media (UDP, InfiniBand, PCI-e 3.0)
Function/Method Call
Volatile Fields & Queues

42. Data Structures
• Maps
• IPC Ring Buffers
• IPC Broadcast Buffers
• ITC Queues
• Dynamic Arrays
• Log Buffers

43. Creates a
replicated persistent log
of messages
What Aeron does

44. How would you design a log?

45. Tail
File

46. Tail
File
Message 1
Header

47. Tail
File
Message 1
Header
Message 2
Header

48. Tail
File
Message 1
Header
Message 2
Header

49. Tail
File
Message 1
Header
Message 2
Header
Message 3

50. Tail
File
Message 1
Header
Message 2
Header
Message 3
Header

51. Persistent data structures can
be safe to read without locks

52. One big file that
goes on forever?

53. No!!!
Page faults, page cache churn,
VM pressure, ...

54. ActiveDirtyClean
Tail
Message
Header
Message
Header
Message
Header
Message
Header
Message
Header
Message
Header
Message
Header
Message
Header

55. How do we stay “wait-free”?

56. Tail
File
Message 1
Header
Message 2
Header
Message 3
Header
Message X
Message Y

57. Tail
File
Message 1
Header
Message 2
Header
Message 3
Header
Message X
Message Y

58. Tail
File
Message 1
Header
Message 2
Header
Message 3
Header
Message X
Message Y

59. Tail
File
Message 1
Header
Message 2
Header
Message 3
Header
Message X
Message Y
Header

60. Tail
File
Message 1
Header
Message 2
Header
Message 3
Header
Message X
Message Y
Header
Padding

61. Tail
File
Message 1
Header
Message 2
Header
Message 3
Header
Message Y
Header
Padding
File
Message X

62. Tail
File
Message 1
Header
Message 2
Header
Message 3
Header
Message X
Message Y
Header
Padding
File
Header

63. What’s in a header?

64. Data Message Header
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Version |B|E| Flags | Type |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-------------------------------+
|R| Frame Length |
+-+-------------------------------------------------------------+
|R| Term Offset |
+-+-------------------------------------------------------------+
| Session ID |
+---------------------------------------------------------------+
| Stream ID |
+---------------------------------------------------------------+
| Term ID |
+---------------------------------------------------------------+
| Encoded Message ...
... |
+---------------------------------------------------------------+

65. Unique identification of a byte
within each stream across time
(streamId, sessionId,
termId, termOffset)

66. How do we replicate a log?

67. We need a Protocol of
messages

68. Sender Receiver

69. Receiver
Setup
Sender

70. Sender
Status
Receiver

71. DataData
Sender Receiver

72. DataData
Status
Sender Receiver

73. DataDataHeartbeat
Sender Receiver

74. DataData
NAK
Sender Receiver

75. Data
Sender Receiver

76. How are message streams
reassembled?

77. High Water Mark
File
Completed

78. High Water Mark
File
Message 1
Header
Completed

79. High Water Mark
File
Message 1
Header
Message 3
Header
Completed

80. File
Message 1
Header
Message 2
Header
Message 3
Header
Completed High Water Mark

81. What if a gap is never filled?

82. How do we know what is
consumed?

83. Publishers, Senders,
Receivers, and Subscribers
all keep position counters

84. Counters are the key to
flow control and monitoring

85. Protocols can be more subtle
than you think…

86. What about
“Self similar behaviour”?

87. 4. What did we learn
on the way?

88. Monitoring and Debugging

89. Loss, throughput, and buffer size
are all strongly related!!!

90. Pro Tip:
Know your OS network
parameters and how
to tune them

91. Some parts of Java really suck!

92. Some parts of Java really suck!
Unsigned Types?

93. Some parts of Java really suck!
Unsigned Types?
NIO (most of) - Locks

94. Some parts of Java really suck!
Unsigned Types?
NIO (most of) - Locks
Off-heap, PAUSE, Signals, etc.

95. Some parts of Java really suck!
Unsigned Types?
String Encoding
NIO (most of) - Locks
Off-heap, PAUSE, Signals, etc.

96. Some parts of Java really suck!
Unsigned Types?
String Encoding
NIO (most of) - Locks
Off-heap, PAUSE, Signals, etc.
Managing External Resources

97. Some parts of Java really suck!
Unsigned Types?
Off-heap, PAUSE, Signals, etc.
Selectors - GC
String Encoding
NIO (most of) - Locks
Managing External Resources

98. Bytes!!!
public void main(final String[] args)
{
byte a = 0b0000_0001;
byte b = 0b0000_0010;
byte flags = a | b;
System.out.printf(
"flags=%sn",
Integer.toBinaryString(flags));
}

99. Bytes!!!
public void main(final String[] args)
{
byte a = 0b0000_0001;
byte b = 0b0000_0010;
byte flags = a | b;
System.out.printf(
"flags=%sn",
Integer.toBinaryString(flags));
}

100. Bytes!!!
public void main(final String[] args)
{
byte a = 0b0000_0001;
byte b = 0b0000_0010;
byte flags = a | b;
System.out.printf(
"flags=%sn",
Integer.toBinaryString(flags));
}

101. Some parts of Java are really nice!

102. Some parts of Java are really nice!
Tooling – IDEs, Gradle, HdrHistogram

103. Some parts of Java are really nice!
Tooling – IDEs, Gradle, HdrHistogram
Profiling – Flight Recorder

104. Some parts of Java are really nice!
Tooling – IDEs, Gradle, HdrHistogram
Bytecode Instrumentation
Profiling – Flight Recorder

105. Some parts of Java are really nice!
Tooling – IDEs, Gradle, HdrHistogram
Bytecode Instrumentation
Unsafe!!! + Java 8
Profiling – Flight Recorder

106. Some parts of Java are really nice!
Tooling – IDEs, Gradle, HdrHistogram
Bytecode Instrumentation
Profiling – Flight Recorder
The Optimiser
Unsafe!!! + Java 8

107. Some parts of Java are really nice!
Tooling – IDEs, Gradle, HdrHistogram
Bytecode Instrumentation
Profiling – Flight Recorder
The Optimiser – Love/Hate
Unsafe!!! + Java 8

108. Some parts of Java are really nice!
Tooling – IDEs, Gradle, HdrHistogram
Bytecode Instrumentation
Garbage Collection!!!
Profiling – Flight Recorder
Unsafe!!! + Java 8
The Optimiser – Love/Hate

109. 5. What’s the Roadmap?

110. Finished a few passes of
Profiling and Tuning

111. Things are looking very good

112. 20+ Million 40 byte
messages per second!!!

113. 50345330
4596
26772645
23622184
2577
11061028
13111423
1014
654612612579622
313313316327301
151156159152163
7575788282
42
29
383942
1818
202120
9
8
1011
9
6
4 4
5 5
3
2 2
3
2 2
1 1 1 1 1 1 1 1 1 1
1
10
100
1000
10000
7.0
7.1
7.2
7.2
7.2
7.3
7.4
9.2
9.6
9.9
10.5
11.0
11.2
11.7
12.2
12.5
13.3
13.9
14.5
15.8
18.1
18.8
19.5
20.9
23.9
26.7
37.9
Latency Distribution (µs)

115. Replication
Services
Aeron Core
Persistence
Queueing
Performance
C++ Port
Batch Send/Recv
IPC
Infiniband
Multi Unicast Send
Stream Query

116. In closing…

118. https://github.com/real-logic/Aeron
Where can I find it?

119. Blog: http://mechanical-sympathy.blogspot.com/
Twitter: @mjpt777
“Any intelligent fool can make things bigger,
more complex, and more violent.
It takes a touch of genius, and a lot of
courage, to move in the opposite direction.”
- Albert Einstein
Questions?

120. Watch the video with slide synchronization on
InfoQ.com!
http://www.infoq.com/presentations/aeron-
messaging