Presentation to the OMG Real-Time Workshop in May 2010 on system integration patterns, especially (but not exclusively) with respect to OMG Data Distribution Service (DDS) technology.

1. The Real-Time
Middleware Experts
Large-Scale System Integration with
DDS for SCADA, C2, and Finance
Rick Warren, Principal Engineer rick.warren@rti.com

2. What do I mean by “large system”?
 Systems of systems
– Modular, hierarchical design
– Legacy components,
subsystems
– Multiple technologies
– Global scale
 Decoupled subsystem
lifecycles
– Independent development
– Independent deployment
and use
– Independent management
– Independent revision and
retirement
 Multiple communities of
interest
– Different data interest,
entitlements
– Non-uniform levels of trust
© 2009 Real-Time Innovations, Inc. 2
LAN
DDS #1
(Initech)
JMS
(Dunder
Mifflin)
DDS #2
(Acme)
Legacy
(COBOL ‘R’
US)
Satellite
Links

3. What matters to integrators of large systems?
 Governance (including over security)
– What information will be exchanged?
– Under what conditions will it be exchanged?
– Who is allowed to exchange this information?
– If these SLAs are violated, can the exchange be prevented?
Can I be notified?
– (In the past, what has occurred wrt these SLAs?)
 Isolation
– When I connect A and B, ensure they don’t break (each other)
– When I disconnect A, ensure B doesn’t break
– When I connect C, don’t change A or B
 Scalability (like in any system,
 Fault Tolerance but stakes are higher)
© 2009 Real-Time Innovations, Inc. 3

4. Part 1: Architecture
(we’ll get to technology later)
© 2009 Real-Time Innovations, Inc. 4

5. System
Component
Component
Component
Class
A Modest Proposition
© 2009 Real-Time Innovations, Inc. 5
State
Behavior
Class
 Fundamental design principles scale
– Abstraction: Provide interface based on relevant concepts
– Encapsulation: Hide internal implementation, communication
– Composition: Combine existing capabilities into new ones

6. Subsystem 2
P P
PP P
Subsystem 1
P P
PP P
Schematic of a Composed System
 Subsystems may have different network environments
 Integration may have different network environment
than subsystems themselves
 Data may need to be transformed / cleansed as it
moves among subsystems
 Routing / gateway services will adapt data types /
formats / protocols
LAN LAN
WAN
Router/G
ateway
Router/G
ateway
Isolation
Additional
Governance

7. Schematic of a Composed System
 Wash, rinse, repeat
Subsystem 1
P P
PP P
Subsystem 2
P P
PP P
Router/G
ateway
Router/G
ateway

8. Same data model?
Same network env.?
Same lifecycle?
Behavior unaffected?
Understandable?
Apples and Oranges
P P
PP P
Subsystem 1 + Subsystem 2
P P
PP P
Subsystem 2
P P
PP P
Subsystem 1
P P
PP P
Router/
Gateway
Router/
Gateway

9. Nothing New Under the Sun
© 2009 Real-Time Innovations, Inc. 9
Subsystem
Data Space
Subsystem
Data Space
Subsystem
Data Space
Integration
Data Space
Router/
Gateway
Router/
Gateway
Router/
Gateway

10. Nothing New Under the Sun
© 2009 Real-Time Innovations, Inc. 10
U.S. Combat
Data Space
Allied Combat
Data Space
U.S. C4I
Data Space
Integration
Data Space
Router/
Gateway
Router/
Gateway
Router/
Gateway
Defense
Industry

11. Nothing New Under the Sun
© 2009 Real-Time Innovations, Inc. 11
NYC Trading
Data Space
London Trading
Data Space
Tokyo Trading
Data Space
Integration
Data Space
Router/
Gateway
Router/
Gateway
Router/
Gateway
Financial
Services
Industry

12. Nothing New Under the Sun
© 2009 Real-Time Innovations, Inc. 12
Generation
Data Space
Substation #1
Data Space
Substation #2
Data Space
Integration
Data Space
Router/
Gateway
Router/
Gateway
Router/
Gateway
Power
Industry

13. Scalability and Fault Tolerance
 Fault tolerance: Router/gateway can’t be single point of
failure. 3 answers:
– Persistent data survives system failures
– Redundancy allows continued service during failure
– Segmented/load-balanced configuration limits scope of failures
 Scalability: How big can a subsystem be?
1. How big a subsystem can you understand, test, and debug?
2. How well does infrastructure scale?
© 2009 Real-Time Innovations, Inc. 13

14. Part 2: DDS
© 2009 Real-Time Innovations, Inc. 14

15. How Does DDS Stack Up?
 Governance
– Structural SLA: data type
– Behavioral SLA: delivery QoS
– Security
– Problem
detection/prevention/notification
– System monitoring and recording
 Isolation
– Prevent data “leaks” and side
effects
– Allow dynamic (dis-|re-)connection
– Support independent integration
and evolution
 Fault tolerance
– Data persistence
– Segmented/load-balanced routing
– Redundant routing
 Scalability
– WAN connectivity
– Peer-to-peer communication
– Brokered/managed communication
– Built in
– Built in
– Products available; future standards
– Built in
– Products available
– Built in: domains, partitions
– Built in
– Built in, esp. w/ DDS-XTypes
– Built in
– Built in if router uses DDS
– Built in if router uses DDS
– Products available; future standards
– Highly scalable
– Protocol support if necessary
© 2009 Real-Time Innovations, Inc. 15

16. DDS-RTPS Supports Redundant Data Routing
© 2009 Real-Time Innovations, Inc. 16
Write
MyData
Read
MyData'
Transform
and Forward
Router 1
Router 2
Multiple
readers:
no problem
Multiple
writers:
prevent
duplicates!
Fortunately,
built into
DDS-RTPS

17. DDS-RTPS Supports Redundant Data Routing
© 2009 Real-Time Innovations, Inc. 17
Read
MyData'
DDS-RTPS
Message
Inline
QoS
Data
Inline QoS
Orig.
Writer
Info
Other
Metadata
Original
Writer Info
Orig.
Writer
GUID
Orig.
Writer
Seq. #
Identity of original writer
Identity of original sample
Reader discards
duplicates
based on these

18. DDS-RTPS Supports Redundant Data Routing
© 2009 Real-Time Innovations, Inc. 18
Read
MyData'
Transform
and Forward
Router 1
Router 2
Writer 5AC2
Data
Seq. # 42
Writer 5AC2
Data
Seq. # 42
Writer 5AC2
Data
Seq. # 42
Writer 5AC2
Data
Seq. # 42
Writer 5AC2
Writer 5AC2
Seq. # 42
Seq. # 42
If your impl.
doesn’t support,
fall back to
ownership.
Write
MyData

19. DDS Scalability: Two Aspects
1. Application data
– How does performance fall off as # participants, writers,
readers increase?
– Any single writer/reader pair can saturate gig-E:
achieving high aggregate throughput is not main issue
– Interesting issue: Fan-out – number of readers per writer
2. Discovery
– How many DDS applications can discover one another?
– Interesting issue: How many applications need to discover
one another?
 Hard limits, if any, very dependent on DDS
implementation, machine configuration, network, etc.
© 2009 Real-Time Innovations, Inc. 19

20. 1. DDS Scalability: Application Data
 DDS-RTPS reliable multicast scales at least:
– …to hundreds of readers per writer
– …with very little degradation in throughput.
– Larger testing facilities welcomed. 
© 2009 Real-Time Innovations, Inc. 20
< 15% decrease
1~900 readers
RTI Data Distribution Service 4.3
Red Hat EL 5.0, 32-bit
2.4 GHz processors
Gigabit Ethernet
UDP/IPv4
Reliable ordered delivery

21. 2. DDS Scalability: P2P Discovery Scenarios
 Single domain, symmetric discovery
– Everyone discovers everyone else
– Easiest to configure; most challenging wrt scalability
– RTI test results: 1,800 participants; 3.2M endpoint matches
 Single domain, asymmetric discovery
– Each participant only knows of certain others in its domain
– Good for partitioning domains in which not everyone talks
 Multiple domains
– Greatest separation for subsystems that rarely exchange data
• DDS-RTPS maps to different IP ports
© 2009 Real-Time Innovations, Inc. 21
1
…
n

22. Summary: Large-Scale P2P Scalability
 Experimental results: DDS-RTPS allows participant to
talk to ≥ 1-2K others in single symmetric domain
 Implication: Compose arbitrarily large systems out of
subsystems this size or smaller
 Large (sub)systems often have requirements for:
– Increased governance, e.g. over security boundaries
– Data transformation
– Protocol mediation
– These typically require data brokering anyway
© 2009 Real-Time Innovations, Inc. 22

23. DDS Scalability: Stronger Measures
Not big enough?
1. Brokered application data
– Covered this
2. Brokered discovery
© 2009 Real-Time Innovations, Inc. 23
…
…
m
…
n
n < m
m
n

24. DDS Scalability: Brokered Discovery
 Can be built with DDS-RTPS-standard building blocks
© 2009 Real-Time Innovations, Inc. 24
Application Topic
Built-In Topic
RTPS Metadata
Data
RTPS Metadata
Data
Orig. Metadata
RTPS Metadata
Data
RTPS InfoSource submessage provides data forwarding capability:
“Treat the following as if it came from that guy instead of from me.”

25. Future Direction: Enhanced Built-in Security
 System of systems often have non-uniform trust
– Multiple communities of interest w/ different
entitlements/authorization
– Infrastructure components may (not) be trusted
(e.g. brokers, daemons, persistence, recording/playback, etc.)
– There is no “system high”
 Desirable new specifications:
– Fine-grained access control: topic, instance, type member levels
– Sample-level tagging and labeling
– Fine-grained signing and/or encryption: topic, member level
© 2009 Real-Time Innovations, Inc. 25

26. Future Direction: Standardized WAN Support
 Site-to-site comms need DDS across WAN(s)
– Including untrusted networks
– Including firewall, NAT traversal
– Challenge: UDP may not be routable, especially if multicast
 DDS-RTPS protocol designed for layering
atop diverse lower-level protocols
– Proven: UDP, TCP, serial, switched fabric, Infiniband, VME,
shared memory…
– Blessed for interop by OMG: UDP
 Additional OMG-recognized protocol layerings
can improve site-to-site interop
– TCP for WAN routing
– (D)TLS for transport-level security (a clarification, not new)
– Could be standardized quickly
© 2009 Real-Time Innovations, Inc. 26

27. Recap
 Large systems usually composed of smaller systems
– Developed independently
– …With different network environments
– …And different data interest/entitlements
 Not an accident: hierarchical design best practice at
every scale
– Subsystems guarded by DDS router/gateway
• Restrict access to internal topics/flows
• Mediate internal/external protocols
• Transform/cleanse internal/external data types
• Impose governance: enforce policy, attach monitoring
– Inter-router integration space is itself a “subsystem” for further
hierarchical composition
© 2009 Real-Time Innovations, Inc. 27

28. Recap
 DDS uniquely qualified to meet these needs
– Strong SLAs for governance
– Multiple topologies for fault-tolerant subsystem isolation
– High-performance, scalable, deterministic interoperability
– What other standard technology offers this combination?
(Not JMS. Not CORBA. Not AMQP. Not web services. …)
 Promising future directions:
– DDS-RTPS/TCP protocol layering for WAN traversal
– Enhanced security model
• Fine-grained access control
• Sample-level tagging, signing, and encrypting
© 2009 Real-Time Innovations, Inc. 28

29. Thank You
© 2009 Real-Time Innovations, Inc. 29