The Assimilation Monitoring Project is an open source project providing both continuous Stealth Discovery(TM) and extreme scale monitoring.

1. Modeling IT Infrastucture
using
The Assimilation Project
#AssimProj
@OSSAlanR
http://assimproj.org/
http://bit.ly/AssimGC2013
Alan Robertson <alanr@unix.sh>
Assimilation Systems Limited
http://assimilationsystems.com
G
R
A
P
H
C
O
N
N
E
C
T

2. Upcoming Events
GraphConnect San Francisco (today!)
Open Source Monitoring Conference - Nürnberg
NSA / Homeland Security Assimilation Technical Talk
Large Installation System Administration Conference - DC
Colorado Springs Open Source User’s Group
linux.conf.au – Awesome Australian Linux Conf - Perth
Details on http://assimilationsystems.com/
G
R
A
P
H
C
O
N
N
E
C
T
GraphConnect
4 October
2013
© 2013 Assimilation Systems Limited
2/25

3. Talk Outline
●
Project Overview
●
Infrastructure Schema
●
Python Object-Graph-Mapping API
G
R
A
P
H
C
O
N
N
E
C
T
GraphConnect
4 October
2013
© 2013 Assimilation Systems Limited
3/25

4. Assimilation Project History
●
Inspired by 2 million core computer (cyclops64)
●
Concerns for extreme scale
●
Topology aware monitoring
●
Topology discovery w/out security issues
=►Discovery of everything!
G
R
A
P
H
C
O
N
N
E
C
T
GraphConnect
4 October
2013
© 2013 Assimilation Systems Limited
4/25

5. Project Scope
Zero-network-footprint continuous Discovery
integrated with extreme-scale Monitoring
●
Continuous extensible discovery
–
●
Extensible exception monitoring
–
●
systems, switches, services, dependencies
– zero network footprint
G
R
A
P
H
C
O
N
N
E
C
T
more than 100K systems
All data goes into central graph database
© 2013 Assimilation Systems Limited
GraphConnect
4 October
2013
5/25

6. Discovery
Discovering
●
systems you've forgotten
●
what you're not monitoring
●
whatever you'd like
●
without setting off security alarms
G
R
A
P
H
C
O
N
N
E
C
T
GraphConnect
4 October
2013
© 2013 Assimilation Systems Limited
6/25

7. Why Discovery?
●
Documentation: incomplete, incorrect
●
Dependencies: unknown
●
Planning: Needs accurate data
●
●
Best Practices: Verification needs
data
ITIL CMDB (Configuration Mgmt
DataBase)
Our Discovery: continuous, low-profile
© 2013 Assimilation Systems Limited
G
R
A
P
H
C
O
N
N
E
C
T
GraphConnect
4 October
2013
7/25

8. Why Neo4j (graph db)?
●
●
●
●
●
●
Dependency & Discovery information: graph
Speed of graph traversals depends on size
of subgraph, not total graph size
Root cause queries  graph traversals –
notoriously slow in relational databases
Visualization of relationships
Schema-less design: good for constantly
changing heterogeneous environment
Graph Model === Object Model
© 2013 Assimilation Systems Limited
G
R
A
P
H
C
O
N
N
E
C
T
GraphConnect
4 October
2013
8/25

9. Assimilation Communication
Neighbor-Rings
G
R
A
P
H
C
O
N
N
E
C
T
GraphConnect
4 October
2013
© 2013 Assimilation Systems Limited
9/25

10. Ring Representation Schema
G
R
A
P
H
C
O
N
N
E
C
T
GraphConnect
4 October
2013
© 2013 Assimilation Systems Limited
10/25

11. ssh -> sshd dependency graph
G
R
A
P
H
C
O
N
N
E
C
T
GraphConnect
4 October
2013
© 2013 Assimilation Systems Limited
11/25

12. Switch Discovery Data
from LLDP (or CDP)
CRM transforms LLDP (CDP) Data to JSON
© 2013 Assimilation Systems Limited
G
R
A
P
H
C
O
N
N
E
C
T
GraphConnect
4 October
2013
12/25

13. OGM – Object Graph Mapping
●
Managing the Graph Nodes “disappears”
●
The Object Model is the Graph Model
●
Significant Improvement in Thinking
Clarity
–
The “mud” is gone
G
R
A
P
H
C
O
N
N
E
C
T
GraphConnect
4 October
2013
© 2013 Assimilation Systems Limited
13/25

14. OGM rules
●
●
●
Don't use Constructors
Relationships replace hash tables and
object references and so on
Constructor parameter names match
attribute names
G
R
A
P
H
C
O
N
N
E
C
T
GraphConnect
4 October
2013
© 2013 Assimilation Systems Limited
14/25

15. OGM sample
@RegisterGraphClass
class Person(GraphNode):
'A Person - or at least an electronic representation of one'
def __init__(self, firstname, lastname, dateofbirth=None):
GraphNode.__init__(self, domain='global')
self.firstname = firstname
self.lastname = lastname
if dateofbirth is not None:
self.dateofbirth = dateofbirth
else:
self.dateofbirth='unknown'
@staticmethod
def __meta_keyattrs__():
'Return our key attributes in order of significance'
return ['lastname', 'firstname']
© 2013 Assimilation Systems Limited
G
R
A
P
H
C
O
N
N
E
C
T
GraphConnect
4 October
2013
15/25

16. OGM sample
@RegisterGraphClass
class TestSystem(GraphNode):
'Some kind of semi-intelligent system'
def __init__(self, designation, roles=[]):
GraphNode.__init__(self, domain)
self.designation = designation.lower()
if roles == []:
roles = ['']
self.roles = roles
def addroles(self, role):
if self.roles[0] == '''':
del self.roles[0]
if isinstance(role, list):
for arole in role:
self.addroles(arole)
elif role not in self.roles:
self.roles.append(role)
© 2013 Assimilation Systems Limited
G
R
A
P
H
C
O
N
N
E
C
T
GraphConnect
4 October
2013
16/25

17. OGM sample
@staticmethod
def __meta_keyattrs__():
'Return our key attributes in order of significance'
return ['designation']
G
R
A
P
H
C
O
N
N
E
C
T
GraphConnect
4 October
2013
© 2013 Assimilation Systems Limited
17/25

18. OGM sample
# (seven)-[:formerly]->(Annika)
Annika = store.load_or_create(Person,
firstname='Annika',
lastname='Hansen')
seven = store.load_or_create(Drone,
designation='SevenOfNine', roles='Borg')
store.relate(seven, 'formerly', Annika)
store.commit()
G
R
A
P
H
C
O
N
N
E
C
T
GraphConnect
4 October
2013
© 2013 Assimilation Systems Limited
18/25

19. OGM sample
@RegisterGraphClass
class TestDrone(TestSystem):
def __init__(self, designation, roles=[]):
TestSystem.__init__(self, designation=designation)
if isinstance(roles, str):
roles = [roles]
roles.extend(['host', 'Drone'])
System.__init__(self, designation, roles=roles)
G
R
A
P
H
C
O
N
N
E
C
T
GraphConnect
4 October
2013
© 2013 Assimilation Systems Limited
19/25

20. Current State
●
First release was April 2013
●
Great unit test infrastructure
●
Nanoprobe code – works well
●
Service monitoring works
●
Lacks digital signatures, encryption, compression
●
Reliable UDP comm code working
●
Several discovery methods written
●
CMA and database code restructuring near-complete
●
G
R
A
P
H
C
O
N
N
E
C
T
UI development underway
●
Licensed under the GPL, commercial license available
© 2013 Assimilation Systems Limited
GraphConnect
4 October
2013
20/25

21. Future Plans
●
Production grade by end of year
●
Purchased support
●
“Real digital signatures, compression, encryption
●
Other security enhancements
●
Much more discovery
●
GUI
●
Alerting
●
Reporting
●
Add Statistical Monitoring
●
Best Practice Audits
●
Dynamic (aka cloud) specialization
●
G
R
A
P
H
C
O
N
N
E
C
T
Hundreds more ideas
–
See: https://trello.com/b/OpaED3AT
© 2013 Assimilation Systems Limited
GraphConnect
4 October
2013
21/25

22. Get Involved!
Powerful Ideas and Infrastucture
Fun, ground-breaking project
Looking for early adopters, testers!!
Needs for every kind of skill
●
●
●
●
●
●
●
Awesome User Interfaces (UI/UX)
Evangelism, community building
Test Code (simulate 106 servers!)
Python, C, script coding
Documentation
Feedback: Testing, Ideas, Plans
Many others!
© 2013 Assimilation Systems Limited
G
R
A
P
H
C
O
N
N
E
C
T
GraphConnect
4 October
2013
22/25

23. Resistance Is Futile!
#AssimProj
@OSSAlanR
#AssimMon
Project Web Site
http://assimproj.org
G
R
A
P
H
C
O
N
N
E
C
T
Blog
techthoughts.typepad.com
lists.community.tummy.com/cgi-bin/mailman/admin/assimilation
© 2013 Assimilation Systems Limited
GraphConnect
4 October
2013
23/25

24. The Elder GeekGirl
G
R
A
P
H
C
O
N
N
E
C
T
GraphConnect
4 October
2013
© 2013 Assimilation Systems Limited
24/25

25. Younger GeekGirl's Computer
Running Linux
Of Course!
G
R
A
P
H
C
O
N
N
E
C
T
GraphConnect
4 October
2013
© 2013 Assimilation Systems Limited
25/25