In the last 30 years, messaging middleware software (MOM) has been offering store and forward services by decoupling senders (publishers) and receivers (subscribers) while offering various levels of QOS and transport protocol options. As hardware options became multi-core and with the aid of developments in Service Oriented Architecture (SOA), Event Driven Architecture (EDA) and cloud computing, a plethora of MOM products have emerged most of them based on the popular publish-subscribe paradigm. This has also led to standardisation efforts both at the API level (JMS) or the transport protocol level (MQTT, AMQP) and an ever increasing developer user base. The majority of today’s IOT architectures involve a gateway node that bridges sensor/field networks with the internet. This is typically done either by using the consumer internet subscription or by leveraging new long range WAN technology such as SigFox and LoRa. In the majority of real world cases, the gateways use advanced software to buffer / support back pressure, an unavoidable result of the network diversity and intermittent connectivity of IOT and mobile nodes. In an effort to find a killer application for DTN that is immediately useful and based on technology available today, we have researched an IOT gateway architecture that acts as a bi-directional convergence layer between MQTT events and RFC 5050 Bundle protocol packets. This allows existing MQTT based sensor applications to leverage DTN services without the need to make any code modifications. At the same time, the approach allows the definition of alternate DTN based routing paths between the gateway and the accompanying data ingestion cloud services, without being restricted from the typically tiny routing table size. Geographically dispersed such gateways can now also provide routing services for each other by further converging over Licklider Transmission Protocol (LTP) for all or a subset of the messaging namespace payloads. Finally the cost of data ingestion from occasionally connected field networks is greatly reduced due to the simplicity of developing applications for them.

1. Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks
BIT 5th Annual World Congress
Emerging InfoTech 2016
An IOT gateway Architecture
using a MQTT bundle transport &
LTP Convergence Layer
11/11/16
1
By Alex Kritikos

2. . Senior director R&D
Universal Messaging
Software AG UK
. OASIS MQTT TC member
Email: alex.kritikos@softwareag.com
Twitter: @krital
LinkedIn: https://uk.linkedin.com/in/alexkritikos
ALEX KRITIKOS
About the speaker
2
Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks

3. Growing fast
Internet of (Earth) Things
3
11/11/16 Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks

4. DTN based Solar System
Internet
4
11/11/16
Internet of (Space) Things
Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks

5. Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks
Earth & Space IOT research
What if we
combined them?
• Link layer agnostic propagation (bundles)
• MQTT encapsulation: payloads to local subscribers
5
11/11/16
• DTN encapsulation: payloads are routed progressively via
intermediary nodes
• Full MQTT & DTN stack on each node
• Lower latency and higher throughput

6. Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks
DTN to MQTT
Convergence Layer
• /temperature => dtn://node/temperature
• /consumption => dtn://node/consumption
• /ac/actions => dtn://node/ac/actions
• MQTT topic => DTN EID
6
11/11/16
• bi directional
• MQTT publish DTN receive
• DTN send MQTT subscribe

7. 7
11/11/16
• MQTT integration with DTN
• Hardware sharing = cost reduction
• Software sharing = cost reduction
• MQTT bridging
• via DTN BP
• Via Licklider Transmission Protocol (LTP)
DTN to MQTT
Value
Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks

8. MQTT PUBLISH BP TRAFFIC MQTT
SUBSCRIBE
8
11/11/16
DTN to MQTT integration
Hardware and Software
Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks

9. MQTT ROUTING via DTN
Best effort MQTT Bridging via bundles
9
11/11/16 Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks

10. 10
11/11/16
MQTT ROUTING via DTN
Assured MQTT Bridging via custody transfer bundles
Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks

11. • Schedules best effort or assured data propagation
• Decouples local fanout from bridging propagation
11
11/11/16
• Events are routed progressively via intermediary
nodes
• Results in lower latency and higher throughput
MQTT ROUTING via DTN
Thoughts
Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks

12. 12
11/11/16
MQTT ROUTING via LTP
Bridging MQTT brokers via LTP UDPCL
Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks

13. • Unidirectional, Connectionless, Efficient flow
• Fragmentation support
• Flow control via LTP sessions
• Multi stream (red, green) assured data
propagation
• Link layer agnostic
13
11/11/16
MQTT ROUTING via LTP
Thoughts
Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks

14. Interested? Contact me
• To find out more
• To collaborate
14
11/11/16
Earth & Space IOT research
What if we
combined them?
Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks

15. Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks
References
• Advances in Delay-tolerant Networks
(DTNs) – Architecture and Enhanced
Performance
By Woodhead Publishing 2015
11/11/16
15
• Brenda Lyons, NASA NDBS 2016
• Appendix A: What is IOT?
• Appendix B: What is DTN?
• Appendix C: What is MQTT?
• Further Research A: DTN routing via MQTT
• Further Research B: High availability with MQTT
• Further Research C: High availability with MQTT & DTN

16. Appendix A
Never heard of Internet of Things?
Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks 1611/11/16
• Read about
• History and status
• M2M Comparison
• Market & estimates
• Example architectures

17. A
IOT & M2M
Comparison…
• M2M came first during the telecom era
• M2M: proprietary, closed, private
machine communication (e.g. CallerID)
• IOT is much more than machine
communication
Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks 1711/11/16

18. A
IOT & M2M
Comparison…(2)
• IOT: devices connect to a larger
network, usually via a gateway, are
cloud enabled, can interop, enables
device physical reach (e.g. Samsung
SmartThings)
• IOT – M2M common: machine comms
Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks 1811/11/16

19. IOT Market
Estimates…
• 2020:
• 25-30 billion
Connected devices
• Today:
• 6.4-9 billion
Connected devices
!This excludes phones, tablets, PCs!
Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks 1911/11/16

20. IOT IMPORTANCE
Status…
• Ongoing transport standardization efforts
• MQTT, COAP, AMQP, Websocket, Node
• Enormous telco traffic challenge
• Global M2M traffic:1 to 6.3 Exabytes / month by 2020 SOURCE
• Real time streaming analytics
• Security nightmare
• 1TB DDOS by hacked CCTVs
• PCs, PVRs, routers, smart fridge
• MCUs + PKI = slow
• Privacy concerns
Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks 2011/11/16

21. IOT Example
Smart Home…
Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks 21
11/11/16

22. IOT Example
Smart Home…
Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks 22
11/11/16
Why cant all things use WiFi?
• If mains powered, sometimes they do
• WiFi may not reach your boiler room
• A lot of battery powered – low energy sensors
• WiFi lacks ad-hoc network features (e.g. mesh)
Things tend to be in their own sensor/field net
• Based on low power RF (e.g. zigbee, zwave)
• Needs a gateway to bridge to local IP network

23. Smart Home IOT Architecture
GSM/DSL/
FIBRE
HTTP
MQTT
AMQP
Ethernet /
WiFi
HTTP
MQTT
AMQP
Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks 23
11/11/16
Users interact with things via a device …

24. Smart Home IOT Architecture (2)
GSM/DSL/
FIBRE
HTTP
MQTT
AMQP
RF
ZigBee
ZWave
BLE
Ethernet /
WiFi
HTTP
MQTT
AMQP
Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks 2411/11/16
Things interact with environment via gateway…

25. Smart Home IOT Architecture (3)
GSM/DSL/
FIBRE
HTTP
MQTT
AMQP
RF
ZigBee
ZWave
BLE
Ethernet /
WiFi
HTTP
MQTT
AMQP
Cloud Platform Services
Cloud services add value & remote access

26. Smart Home IOT Architecture (4)
GSM/DSL/
FIBRE
HTTP
MQTT
AMQP
RF
ZigBee
ZWave
BLE
Ethernet /
WiFi
HTTP
MQTT
AMQP
Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks 2611/11/16

27. Appendix B
Never heard of DTN?
Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks 2711/11/16
• Read about
• history and status
• Protocol overview
• TCP/IP Comparison
• Terrestrial uses

28. DTN
History…
Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks 2811/11/16
initiative to
develop an
“Interplanetary
Internet”
1998
NASA starts
Delay / Disruption
Tolerant
Networking
1998
NASA leads
worldwide
DTN
development
2005
IOAG
Recommends
Solar System
Internetwork
(SSI)
2010
IOAG
Suggests
DTN
a core service
of SSI
2010
DTN
is being
standardized
by IETF
DTN
is being
standardized
by CCSDS

29. DTN
Solar System Internet…
Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks 2911/11/16

30. DTN
• The (DTN) protocol suite
• Extends the terrestrial Internet capabilities
• Works
• in challenged communication environments
• over unidirectional/asymmetric links
• Tolerates
• frequent disruptions
• long delays / high error rates
• Provides
• assured delivery of data
• automatic store-and-forward
• Can run over IP or as a full Internetworking protocol
Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks 3011/11/16
Overview

31. DTN
Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks 3111/11/16
Overview (2)

32. DTN
Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks 3211/11/16
TCP/IP Comparison

33. DTN
IP: Must wait for complete path
Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks 3311/11/16
TCP/IP Comparison (2)

34. DTN
TCP/IP Comparison (3)
DTN: Incremental progress without complete path
Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks 3411/11/16

35. DTN
DTN decouples data propagation from link layer
characteristics
• Can be used to propagate data not needed in
real time (e.g. wearables, home automation
etc)
• Introduces some node storage requirements
• Use cases
• sea-going vessels
• outdoor explorers
• wildlife research
• vehicular networks
• disaster area networks
Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks 3511/11/16
Terrestrial use

36. Appendix C
Never heard of MQTT?
Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks 3611/11/16
• Read about
• History and status
• Protocol overview
• Role in IOT architectures

37. MQTT
History…
Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks 3711/11/16
IBM &
Eurotech
develop
1990s
IBM &
Eurotech
Open
Source
2012
Software
AG
implements
2013
OASIS
MQTT TC
Created
2013
Many
others
join
MQTT
becomes
OASIS
standard
2014
MQTT
becomes
ISO/IEC
20922
2016

38. MQTT
Description
• MQTT is
• Is an open, simple, lightweight publish-
subscribe binary messaging protocol
• Is easy to implement (15 protocol
commands)
• has a fixed 2 byte over-the-wire overhead
header
• Has 256MB max payload
• runs on top of TCP/TLS/WS/WSS stacks
Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks 3811/11/16

39. MQTT
Description (2)
• MQTT offers 3 quality of service levels:
• QOS 0: At most once (Fire and forget)
• QOS 1: At least once (ACKed delivery)
• QOS 2: Exactly once (ACKed / No
duplicates)
Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks 3911/11/16

40. MQTT
Description (3)
• Information on software/hardware that uses
MQTT: http://ow.ly/wHm2e
• The Eclipse Paho project offers some great
MQTT clients in minimal C, regular C, C++,
Java, Javascript, Python, Lua, Go, Objective-
C: http://ow.ly/wHl8c
Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks 4011/11/16

41. MQTT in IOT
Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks 4111/11/16

42. MQTT in IOT (2)
Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks 4211/11/16

43. MQTT in IOT (3)
MQTT BRIDGE
MQTT PUB &
SUB MQTT SUB
Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks 4311/11/16

44. Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks 4411/11/16
Further Research A
DTN routing via MQTT

45. DTN ROUTING via MQTT
One to many
Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks 4511/11/16

46. DTN ROUTING via MQTT
Many to one
Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks 4611/11/16

47. DTN ROUTING via MQTT
Thoughts
• Unicast routing focus
• Data sharing importance:
• Vehicular networks
• Disaster area networks
• Delegates shared custodian to MQTT broker
• Can use various MQTT QOS
Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks 4711/11/16

48. Further Research B
High availability with MQTT
Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks 4811/11/16

49. IOT with MQTT
HA architecture
MQTT PUB & BRIDGE
Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks 4911/11/16

50. IOT with MQTT
HA active – active cluster architecture
MQTT PUB
PROPRIETRY
QUORUM
REPLICATE
SYNC
COORDINATE
Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks 5011/11/16

51. IOT with MQTT
HA active – active: master failure
MQTT PUB
PROPRIETRY
QUORUM
REPLICATE
SYNC
COORDINATE
Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks 5111/11/16

52. IOT with MQTT
HA active – active: publisher failover
MQTT PUB
PROPRIETRY
QUORUM
REPLICATE
SYNC
COORDINATE
Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks 5211/11/16

53. IOT with MQTT
HA active – active: node rejoin
MQTT PUB
PROPRIETRY
QUORUM
REPLICATE
SYNC
COORDINATE
Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks 5311/11/16

54. IOT with MQTT
HA active – active: multiple streams
MQTT PUB STREAM
1
MQTT PUB STREAM
2
MQTT PUB STREAM
3
Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks 5411/11/16

55. Further Research C
High availability with MQTT & DTN
Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks 5511/11/16

56. MQTT HA via DTN
Eventual consistency …
Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks 5611/11/16

57. MQTT HA via DTN
Master failure
Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks 5711/11/16

58. MQTT HA via DTN
Partial sync while offline
Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks 5811/11/16

59. MQTT HA via DTN
Fast node cluster rejoin
Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks 5911/11/16

60. MQTT HA via DTN
Thoughts
• Schedules best effort or assured data
propagation
• Decouples local fanout from slave propagation
• Bundles continue to flow while broker offline
• Allows fast cluster recovery
Alex Kritikos - Thoughts on the Future Development of Ad hoc and Wireless Sensors Networks 6011/11/16