1.
Logging, Metrics and Monitoring
as a Service
Lakshminarayanan (LN) Renganarayana
Architect, Cloud Platform Engineering
LMM, Cloud Platform Engineering Nov 4, 2014 1

2.
LMM, Cloud Platform Engineering Nov 4, 2014
2
The Symantec Team
• Cloud Platform Engineering
– We are building a consolidated cloud platform that provides infrastructure
and platform services for next generation Symantec products and services
– starting small, but scaling to tens of thousands of nodes across multiple data
centers
• Me
– Architect / Lead for LMM in Cloud Platform Engineering
– Love to build and break systems
– Previous life: Research Staff Member, IBM Research
– Automatic parallelization, Compilers, Deployment Automation (Heat)

3.
Context for LMM: Symantec Cloud Platform
Common
Services
LMM, Cloud Platform Engineering
3
Symantec Services & Apps
Platform Services
BP, SP, KV, OBS
IaaS (OpenStack)
Bare Metal
Nov 4, 2014
LMM
Identity
Manager
CI / CD

4.
LMM : Goals we started with …
• Design for both Developers and Ops
• Multi Tenant, Elastic Scalability
• One system for logs, metrics, monitoring, & other events
• Designed for under and over the cloud
• Build using open source tools and for open sourcing
LMM, Cloud Platform Engineering Nov 4, 2014
4

5.
Guiding principle
Make it extremely simple to generate logs & metrics
Provide powerful query and visualizations for free
LMM, Cloud Platform Engineering Nov 4, 2014
5
Gen with 1 line: logger.warn(“WARN KVaaS too many failed writes”)
Query with all the power of ElasticSearch and visualize with Kibana
Gen with 1 line: statsd_client.incr(‘myapp_num_requests’)
Query with all the power of InfluxDB and visualize with Grafana

6.
LMM Client Interface
expose
metrics
JMX
Push
Metrics
LMM, Cloud Platform Engineering Nov 4, 2014
6
App
Platform
Services
Writes app
metrics directly
Infrastructure
SNMP Vars
Pull
Metrics
LMM
Shipper
VM
logs
LMM
metrics log events
Alerts &
Config UI
Push: StatsD, metrics extension for openstack
Pull: CollectD. Shipper: logstash, lumberjack

7.
Security and Multi-tenancy
• User authentication with Keystone for Query API & UI
• Tenant id and API key used for events sent to LMM
– Tenant ids from Keystone and API keys generated by LMM
• Every event is tagged with a tenant id
– Log events: tenant id as a field
– Metric events: tenant id prefixed to the metric name
• Keystone group membership used for sophisticated cross-tenant
event visibility / correlation
LMM, Cloud Platform Engineering Nov 4, 2014
7

8.
Client App / Service
log & metrics shipper
log, metric, alert events
log, metric & alert events
Authentication, Validation, Alerts Processing
s1 s2 s3 s4 ... sn
LMM, Cloud Platform Engineering Nov 4, 2014
8
Kafka cluster
knode1
Keystone
knode2 knode3 ... knodeN
Storm cluster
Front End Cluster: Multi-tenancy and Kibana, Graphana Proxies
Elastic
Search
Elastic
ReSdeiasrch
Alerts email &
callbacks
Load Balancer
InfluxDB
InfluxDB
InfluxDB
Metrics Store
Elastic
Search
Elastic
Search
Elastic
Search
Log Store
LMM Under the Hood
Quota
&
Policy

9.
Current deployment
• Multiple deployments : on bare KVM nodes, on OpenStack VMs
– On KVM nodes: 40+ VMs, 80+ TB storage, many large memory nodes
– All components are deployed in clustered mode for HA
– Some with active/active replication, some with active/passive
• Use by Platform and Infrastructure Services
– Hundreds of thousands of events per second
– Few TBs of data collected and indexed per day
– Queries are currently coming from Kibana and Grafana, in future from APIs
LMM, Cloud Platform Engineering Nov 4, 2014
9

10.
In the Next Iteration …
• Support Health checks
– Exploring Sensu and related technologies
• Multi data center deployment
• Refactor to a data pipeline architecture
• Deployment automation using Murano
• Open Source LMM
LMM, Cloud Platform Engineering Nov 4, 2014
10

11.
Thank you!
Questions, Comments, Suggestions?
We are interested in open sourcing LMM
and are looking for collaborators …
Interested? Feel free to contact me
LN_Renganarayana@Symantec.com
LMM, Cloud Platform Engineering Nov 4, 2014 11