RedisDay London 2018

1. Atom: Redis for Robotics
Dan Pipe-Mazo, Elementary Robotics

2. ELEMENTARY ROBOTICS NOV 15
Elementary Robotics:
Who We Are
We believe that there are two main
items holding back the robotics industry:
Cost & Ease of programming.
We’re building low-cost, data-rich robots
from the ground up with an emphasis on
performance and ease of use.

3. ELEMENTARY ROBOTICS NOV 15
Robotics Stack: Basics
HARDWARE DRIVERS SOFTWARE
Joints
Motors
Sensors
Talk to all hardware
in low-level
Programming languages/
kernel modules
Proprietary communications
protocols
Implements high-level logic
and algorithms for the system.
Calls driver APIs as necessary
to get things done.

4. ELEMENTARY ROBOTICS NOV 15
ABSTRACTION
SCALABILITY LANGUAGE SUPPORT
Standardize complex
hardware and algorithms
into simple paradigms
Easy horizontal scaling
for compute-heavy
tasks
Certain tasks are
better suited to
certain languages
Software Stack:
Objectives
DEPENDENCIES
Eliminate
dependency issues
INTROSPECTION
Easily audit the system
for debug/monitoring

5. ELEMENTARY ROBOTICS NOV 15
Software Stack:
Solution
+
API
Create a specification, API, and set of libraries
called atom prebuilt with all dependencies and
shipped in a docker container.
MICRO-SERVICES
Create micro services called elements that use
the atom libraries, in their preferred language,
to implement higher-level APIs
TECHNOLOGY
Use redis streams for performant, flexible distributed
data publishing and command/response paradigms.

6. ELEMENTARY ROBOTICS NOV 15
Redis Streams: overview
XADD s1 MAXLEN ~ X k1 v1 k2 v2 …
...
stream: s1
0
-
k1
v1
k2
v2
1
-
k1
v1
k2
v2
2
-
k1
v1
k2
v2
X
-
k1
v1
k2
v2

7. ELEMENTARY ROBOTICS NOV 15
Redis Streams: overview
XADD s2 MAXLEN ~ X kA vA kB vB …
XADD s1 MAXLEN ~ X k1 v1 k2 v2 …
stream: s2
...
0
-
kA
vA
kB
vB
1
-
kA
vA
kB
vB
2
-
kA
vA
kB
vB
X
-
kA
vA
kB
vB
...
stream: s1
0
-
k1
v1
k2
v2
1
-
k1
v1
k2
v2
2
-
k1
v1
k2
v2
X
-
k1
v1
k2
v2

8. ELEMENTARY ROBOTICS NOV 15
Redis Streams: overview
XADD s2 MAXLEN ~ X kA vA kB vB …
XADD s1 MAXLEN ~ X k1 v1 k2 v2 …
stream: s2
...
stream: s1
0
-
k1
v1
k2
v2
1
-
k1
v1
k2
v2
2
-
k1
v1
k2
v2
X
-
k1
v1
k2
v2
XREAD BLOCK N STREAMS s1 $
XREAD BLOCK N STREAMS s1 ID
Subscribe to all entries from one stream
...
0
-
kA
vA
kB
vB
1
-
kA
vA
kB
vB
2
-
kA
vA
kB
vB
X
-
kA
vA
kB
vB

9. ELEMENTARY ROBOTICS NOV 15
Redis Streams: overview
XADD s2 MAXLEN ~ X kA vA kB vB …
XADD s1 MAXLEN ~ X k1 v1 k2 v2 …
stream: s2
...
stream: s1
0
-
k1
v1
k2
v2
1
-
k1
v1
k2
v2
2
-
k1
v1
k2
v2
X
-
k1
v1
k2
v2
XREAD BLOCK N STREAMS s1 $
XREAD BLOCK N STREAMS s1 ID
Subscribe to all entries from one stream
XREAD BLOCK N STREAMS s1 s2 ID1 ID2
Subscribe to all entries from multiple streams
...
0
-
kA
vA
kB
vB
1
-
kA
vA
kB
vB
2
-
kA
vA
kB
vB
X
-
kA
vA
kB
vB

10. ELEMENTARY ROBOTICS NOV 15
Redis Streams: overview
XADD s2 MAXLEN ~ X kA vA kB vB …
XADD s1 MAXLEN ~ X k1 v1 k2 v2 …
stream: s2
...
stream: s1
0
-
k1
v1
k2
v2
1
-
k1
v1
k2
v2
2
-
k1
v1
k2
v2
X
-
k1
v1
k2
v2
XREAD BLOCK N STREAMS s1 $
XREAD BLOCK N STREAMS s1 ID
Subscribe to all entries from one stream
XREAD BLOCK N STREAMS s1 s2 ID1 ID2
Subscribe to all entries from multiple streams
XREVRANGE s1 + - COUNT N
Get the latest N entries from a stream
...
0
-
kA
vA
kB
vB
1
-
kA
vA
kB
vB
2
-
kA
vA
kB
vB
X
-
kA
vA
kB
vB

11. ELEMENTARY ROBOTICS NOV 15
Redis Streams: overview
XADD s2 MAXLEN ~ X kA vA kB vB …
XADD s1 MAXLEN ~ X k1 v1 k2 v2 …
stream: s2
...
stream: s1
0
-
k1
v1
k2
v2
1
-
k1
v1
k2
v2
2
-
k1
v1
k2
v2
X
-
k1
v1
k2
v2
XREAD BLOCK N STREAMS s1 $
XREAD BLOCK N STREAMS s1 ID
Subscribe to all entries from one stream
XREAD BLOCK N STREAMS s1 s2 ID1 ID2
Subscribe to all entries from multiple streams
XREVRANGE s1 + - COUNT N
Get the latest N entries from a stream
XRANGE s1 - + COUNT N
Get the oldest N entries from a stream
...
0
-
kA
vA
kB
vB
1
-
kA
vA
kB
vB
2
-
kA
vA
kB
vB
X
-
kA
vA
kB
vB

12. ELEMENTARY ROBOTICS NOV 15
Atom
Specification and Examples

13. ELEMENTARY ROBOTICS NOV 15
Atom: Data Publishing
PUBLISHER:
element “foo”, streams s1 and s2
SUBSCRIBER：
Options to read from the stream
XADD data:foo:s1 MAXLEN ~ X k1 v1 k2 v2 …
XADD data:foo:s2 MAXLEN ~ X k3 v3 k4 v4 …
XREAD BLOCK T STREAMS data:foo:s1 data:foo:s2 idS1 idS2
XREVRANGE data:foo:s1 + - COUNT N
XRANGE data:foo:s1 - + COUNT N

14. ELEMENTARY ROBOTICS NOV 15
Atom: Data Publishing Example
CAMERA
USB
camera
element
MAIN COMPUTE
XADD
frames
elem:cam:frames
REDIS

15. ELEMENTARY ROBOTICS NOV 15
Atom: Data Publishing Example
CAMERA
USB
camera
element
MAIN COMPUTE
XADD
frames
elem:cam:frames
REDIS
video
packaging
element
XREAD BLOCK N

16. ELEMENTARY ROBOTICS NOV 15
Atom: Data Publishing Example
CAMERA
USB
camera
element
MAIN COMPUTE
XADD
frames
elem:cam:frames
REDIS
video
packaging
element
ML
classification
element
XREAD BLOCK N
XREVRANGE + -

17. ELEMENTARY ROBOTICS NOV 15
Atom: Data Publishing Example
CAMERA
USB
camera
element
MAIN COMPUTE
XADD
frames
elem:cam:frames
REDIS
video
packaging
element
ML
classification
element
ML
retraining
element
XREAD BLOCK N
XREVRANGE + -
XRANGE last +

18. ELEMENTARY ROBOTICS NOV 15
Atom: Sync or Async Req/Rep
Requestor: “foo” Responder: “bar”REDIS
elem:bar:req XREAD BLOCK 0 STREAMS elem:bar:req lastEntryID
elem:foo:rep

19. ELEMENTARY ROBOTICS NOV 15
Atom: Sync or Async Req/Rep
Requestor: “foo” Responder: “bar”REDIS
elem:bar:req XREAD BLOCK 0 STREAMS elem:bar:req lastEntryIDXADD elem:bar:req MAXLEN ~ X elem foo cmd Y data Z
elem:foo:rep

20. ELEMENTARY ROBOTICS NOV 15
Atom: Sync or Async Req/Rep
Requestor: “foo” Responder: “bar”REDIS
elem:bar:req
(entryID, {“elem”: foo, “cmd”: Y, “data”: Z})
XREAD BLOCK 0 STREAMS elem:bar:req lastEntryIDXADD elem:bar:req MAXLEN ~ X elem foo cmd Y data Z
elem:foo:rep

21. ELEMENTARY ROBOTICS NOV 15
Atom: Sync or Async Req/Rep
Requestor: “foo” Responder: “bar”REDIS
elem:bar:req
(entryID, {“elem”: foo, “cmd”: Y, “data”: Z})
XREAD BLOCK 0 STREAMS elem:bar:req lastEntryIDXADD elem:bar:req MAXLEN ~ X elem foo cmd Y data Z
XREAD BLOCK 1000 STREAMS elem:foo:rep lastEntryID
elem:foo:rep

22. ELEMENTARY ROBOTICS NOV 15
Atom: Sync or Async Req/Rep
Requestor: “foo” Responder: “bar”REDIS
elem:bar:req
(entryID, {“elem”: foo, “cmd”: Y, “data”: Z})
XREAD BLOCK 0 STREAMS elem:bar:req lastEntryID
XADD elem:foo:rep MAXLEN ~ X elem bar id ID time T
XADD elem:bar:req MAXLEN ~ X elem foo cmd Y data Z
XREAD BLOCK 1000 STREAMS elem:foo:rep lastEntryID
elem:foo:rep

23. ELEMENTARY ROBOTICS NOV 15
Atom: Sync or Async Req/Rep
Requestor: “foo” Responder: “bar”REDIS
elem:bar:req
(entryID, {“elem”: foo, “cmd”: Y, “data”: Z})
XREAD BLOCK 0 STREAMS elem:bar:req lastEntryID
XADD elem:foo:rep MAXLEN ~ X elem bar id ID time T
XADD elem:bar:req MAXLEN ~ X elem foo cmd Y data Z
(entryID, {“elem”: bar, “id”: ID, “time”: T})
XREAD BLOCK 1000 STREAMS elem:foo:rep lastEntryID
elem:foo:rep

24. ELEMENTARY ROBOTICS NOV 15
Atom: Sync or Async Req/Rep
Requestor: “foo” Responder: “bar”REDIS
elem:bar:req
(entryID, {“elem”: foo, “cmd”: Y, “data”: Z})
XREAD BLOCK 0 STREAMS elem:bar:req lastEntryID
XADD elem:foo:rep MAXLEN ~ X elem bar id ID time T
XADD elem:bar:req MAXLEN ~ X elem foo cmd Y data Z
(entryID, {“elem”: bar, “id”: ID, “time”: T})
XREAD BLOCK 1000 STREAMS elem:foo:rep lastEntryID
elem:foo:rep
XREAD BLOCK T STREAMS elem:foo:rep lastEntryID

25. ELEMENTARY ROBOTICS NOV 15
Atom: Sync or Async Req/Rep
Requestor: “foo” Responder: “bar”REDIS
elem:bar:req
(entryID, {“elem”: foo, “cmd”: Y, “data”: Z})
XREAD BLOCK 0 STREAMS elem:bar:req lastEntryID
XADD elem:foo:rep MAXLEN ~ X elem bar id ID time T
XADD elem:foo:rep MAXLEN ~ X elem bar id ID resp R
XADD elem:bar:req MAXLEN ~ X elem foo cmd Y data Z
(entryID, {“elem”: bar, “id”: ID, “time”: T})
XREAD BLOCK 1000 STREAMS elem:foo:rep lastEntryID
elem:foo:rep
XREAD BLOCK T STREAMS elem:foo:rep lastEntryID

26. ELEMENTARY ROBOTICS NOV 15
Atom: Sync or Async Req/Rep
Requestor: “foo” Responder: “bar”REDIS
elem:bar:req
(entryID, {“elem”: foo, “cmd”: Y, “data”: Z})
XREAD BLOCK 0 STREAMS elem:bar:req lastEntryID
XADD elem:foo:rep MAXLEN ~ X elem bar id ID time T
XADD elem:foo:rep MAXLEN ~ X elem bar id ID resp R
XADD elem:bar:req MAXLEN ~ X elem foo cmd Y data Z
(entryID, {“elem”: bar, “id”: ID, “time”: T})
XREAD BLOCK 1000 STREAMS elem:foo:rep lastEntryID
elem:foo:rep
(entryID, {“elem”: bar, “id”: ID, “resp”: R})
XREAD BLOCK T STREAMS elem:foo:rep lastEntryID

27. ELEMENTARY ROBOTICS NOV 15
Atom: Req/Rep Example
MAIN COMPUTE
REDIS
user
element
robot
element
elem:robot:req
elem:user:rep
Go to (x, y, z)

28. ELEMENTARY ROBOTICS NOV 15
Atom: Req/Rep Example
MAIN COMPUTE
REDIS
user
element
robot
element
elem:robot:req
elem:user:rep
Go to (x, y, z)
ACK

29. ELEMENTARY ROBOTICS NOV 15
Atom: Req/Rep Example
COMMS
MAIN COMPUTE
REDIS
MOTOR 1
MOTOR 2
MOTOR N
...
user
element
robot
element
elem:robot:req
elem:user:rep
Go to (x, y, z)
ACK

30. ELEMENTARY ROBOTICS NOV 15
Atom: Req/Rep Example
COMMS
MAIN COMPUTE
REDIS
MOTOR 1
MOTOR 2
MOTOR N
...
user
element
robot
element
elem:robot:req
elem:user:rep
Go to (x, y, z)
ACK
path
planning
element
camera
element

31. ELEMENTARY ROBOTICS NOV 15
Atom: Req/Rep Example
COMMS
MAIN COMPUTE
REDIS
MOTOR 1
MOTOR 2
MOTOR N
...
user
element
robot
element
elem:robot:req
elem:user:rep
Go to (x, y, z)
ACK
Done
path
planning
element
camera
element

32. ELEMENTARY ROBOTICS NOV 15
Atom:
Scaling &
Updates
REDIS STREAM CONSUMER GROUPS
XGROUP CREATE elem:bar:req consumers $
UPDATES (ASSUMING SAME API)
Spin up the new instance, monitor performance,
spin down the old instance.
SCALING
Just spin up another instance of the element on
the same redis master.
XREADGROUP consumers me BLOCK 0 COUNT 1 STREAMS elem:bar:req <
XACK elem:bar:req consumers entryID

33. ELEMENTARY ROBOTICS NOV 15
Atom: Scaling/Update Example
MAIN COMPUTE
REDISML
classification
element
(copy 1) elem:class:req
elem:user:rep
user element
Classify A, B, C, D
A

34. ELEMENTARY ROBOTICS NOV 15
Atom: Scaling/Update Example
MAIN COMPUTE
REDISML
classification
element
(copy 1)
ML
classification
element
(copy 2)
elem:class:req
elem:user:rep
user element
Classify A, B, C, D
A
B

35. ELEMENTARY ROBOTICS NOV 15
Atom: Scaling/Update Example
MAIN COMPUTE
REDISML
classification
element
(copy 1)
ML
classification
element
(copy 2)
elem:class:req
elem:user:rep
user element
Classify A, B, C, D
Result A
Results
Result B

36. ELEMENTARY ROBOTICS NOV 15
Atom: Scaling/Update Example
MAIN COMPUTE
REDISML
classification
element
(copy 1)
ML
classification
element
(copy 2)
elem:class:req
elem:user:rep
user element
Classify A, B, C, D
C
D

37. ELEMENTARY ROBOTICS NOV 15
Atom: Scaling/Update Example
MAIN COMPUTE
REDISML
classification
element
(copy 1)
ML
classification
element
(copy 2)
elem:class:req
elem:user:rep
user element
Classify A, B, C, D
Result C
Results
Result D

38. ELEMENTARY ROBOTICS NOV 15
Atom:
Basic Discovery
ELEMENT
DISCOVERY
DATA STREAM
DISCOVERY
SCAN cursor MATCH elem: SCAN cursor MATCH data:
SCAN cursor MATCH data:foo
COMMAND
DISCOVERY
In the works!

39. ELEMENTARY ROBOTICS NOV 15
Atom:
Serialization
OVERALL COMMAND/
RESPONSE
DATA
JPEG/TIFF best for
camera/PointCloud data,
Protobuf pretty good for
most other data.
No single serialization
protocol was best/optimal
for all payloads/use cases.
No Enforced Serialization
Protobuf performed well and
RPC service specification
worked well for strongly typed
command/response types.

40. ELEMENTARY ROBOTICS NOV 15
Atom:
API
class Element():
__init__(self, name)
__del__(self)
# Commands
command_add(
self,
name,
handler,
timeout=DEFAULT_TIMEOUT)
command_loop(
self)
command_send(
self,
element,
command,
data,
block=True)
# Data
entry_write(
self,
name,
{“k1”: v1, “k2”:v2, … },
maxlen=DEFAULT_MAXLEN)
entry_read_n(
self,
element,
stream,
n)
entry_read_loop(
self,
{“s1”: h1, “s2”: h2, …},
n_loops=None)
# Discovery
get_all_elements(
self)
get_all_streams(
self,
element_name=None)

41. ELEMENTARY ROBOTICS NOV 15
Atom:
Docker
NUCLEUS
DOCKER
CONTAINER
ATOM
DOCKER
CONTAINER
ELEMENT
DOCKER
CONTAINERS
User-developed
elements
FROM
elementaryrobotics/atom
FROM
redis/alpine
Contains custom redis
conf + atom CLI
Latest tested atom
libraries
All dependencies,
protobuf, etc.
MOLECULE
DOCKER
COMPOSE FILE
Launch nucleus
and all elements
Set up shared tmpfs
between containers &
unix socket for redis

42. ELEMENTARY ROBOTICS NOV 15
Atom:
User Development Flow
1
Copy template from
atom github repo.
2
Implement element in
language of choosing
3
Build element with
4
Launch element, nucleus
and all dependencies with
docker-compose up -d
docker-compose build
5
Done!

43. ELEMENTARY ROBOTICS NOV 15
Summary
ATOM SET
OF TOOLS
OUR
WEBSITE
elementaryrobotics.com.
Please visit and sign up
for more info on atom.
Allow for powerful yet
simple new ways of
programming
robotic systems.
WE’RE
HIRING!
REDIS
STREAMS
We're working on atom to
bring the power of redis
streams to robotics.
We build robots in LA! Send
me an email directly at
dan@elementaryrobotics.com

44. Thank you!
Please check out our website at
elementaryrobotics.com
and sign up for more info about atom!