Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
Logging, Metrics and Monitoring 
as a Service 
Lakshminarayanan (LN) Renganarayana 
Architect, Cloud Platform Engineering ...
LMM, Cloud Platform Engineering Nov 4, 2014 
2 
The Symantec Team 
• Cloud Platform Engineering 
– We are building a conso...
Context for LMM: Symantec Cloud Platform 
Common 
Services 
LMM, Cloud Platform Engineering 
3 
Symantec Services & Apps 
...
LMM : Goals we started with … 
• Design for both Developers and Ops 
• Multi Tenant, Elastic Scalability 
• One system for...
Guiding principle 
Make it extremely simple to generate logs & metrics 
Provide powerful query and visualizations for free...
LMM Client Interface 
expose 
metrics 
JMX 
Push 
Metrics 
LMM, Cloud Platform Engineering Nov 4, 2014 
6 
App 
Platform 
...
Security and Multi-tenancy 
• User authentication with Keystone for Query API & UI 
• Tenant id and API key used for event...
Client App / Service 
log & metrics shipper 
log, metric, alert events 
log, metric & alert events 
Authentication, Valida...
Current deployment 
• Multiple deployments : on bare KVM nodes, on OpenStack VMs 
– On KVM nodes: 40+ VMs, 80+ TB storage,...
In the Next Iteration … 
• Support Health checks 
– Exploring Sensu and related technologies 
• Multi data center deployme...
Thank you! 
Questions, Comments, Suggestions? 
We are interested in open sourcing LMM 
and are looking for collaborators …...
Prochain SlideShare
Chargement dans…5
×

A Scalable Multi-tenant Logging, Metrics and Monitoring as Service for OpenStack and Cloud Applications

4 420 vues

Publié le

These slides are from my talk in the BrownBag session in OpenStack Summit Paris. In this talk, I share our experience with building a highly scalable multi-tenant cloud service for logging, metrics and monitoring. Talking to other groups that are building similar services, I found that what we are building is unique in some aspects and the scale we are already using it is pretty cool! The video was also recorded and you can check that out here: http://youtu.be/wers9XNO26Q

Publié dans : Logiciels
  • Identifiez-vous pour voir les commentaires

A Scalable Multi-tenant Logging, Metrics and Monitoring as Service for OpenStack and Cloud Applications

  1. 1. Logging, Metrics and Monitoring as a Service Lakshminarayanan (LN) Renganarayana Architect, Cloud Platform Engineering LMM, Cloud Platform Engineering Nov 4, 2014 1
  2. 2. LMM, Cloud Platform Engineering Nov 4, 2014 2 The Symantec Team • Cloud Platform Engineering – We are building a consolidated cloud platform that provides infrastructure and platform services for next generation Symantec products and services – starting small, but scaling to tens of thousands of nodes across multiple data centers • Me – Architect / Lead for LMM in Cloud Platform Engineering – Love to build and break systems – Previous life: Research Staff Member, IBM Research – Automatic parallelization, Compilers, Deployment Automation (Heat)
  3. 3. Context for LMM: Symantec Cloud Platform Common Services LMM, Cloud Platform Engineering 3 Symantec Services & Apps Platform Services BP, SP, KV, OBS IaaS (OpenStack) Bare Metal Nov 4, 2014 LMM Identity Manager CI / CD
  4. 4. LMM : Goals we started with … • Design for both Developers and Ops • Multi Tenant, Elastic Scalability • One system for logs, metrics, monitoring, & other events • Designed for under and over the cloud • Build using open source tools and for open sourcing LMM, Cloud Platform Engineering Nov 4, 2014 4
  5. 5. Guiding principle Make it extremely simple to generate logs & metrics Provide powerful query and visualizations for free LMM, Cloud Platform Engineering Nov 4, 2014 5 Gen with 1 line: logger.warn(“WARN KVaaS too many failed writes”) Query with all the power of ElasticSearch and visualize with Kibana Gen with 1 line: statsd_client.incr(‘myapp_num_requests’) Query with all the power of InfluxDB and visualize with Grafana
  6. 6. LMM Client Interface expose metrics JMX Push Metrics LMM, Cloud Platform Engineering Nov 4, 2014 6 App Platform Services Writes app metrics directly Infrastructure SNMP Vars Pull Metrics LMM Shipper VM logs LMM metrics log events Alerts & Config UI Push: StatsD, metrics extension for openstack Pull: CollectD. Shipper: logstash, lumberjack
  7. 7. Security and Multi-tenancy • User authentication with Keystone for Query API & UI • Tenant id and API key used for events sent to LMM – Tenant ids from Keystone and API keys generated by LMM • Every event is tagged with a tenant id – Log events: tenant id as a field – Metric events: tenant id prefixed to the metric name • Keystone group membership used for sophisticated cross-tenant event visibility / correlation LMM, Cloud Platform Engineering Nov 4, 2014 7
  8. 8. Client App / Service log & metrics shipper log, metric, alert events log, metric & alert events Authentication, Validation, Alerts Processing s1 s2 s3 s4 ... sn LMM, Cloud Platform Engineering Nov 4, 2014 8 Kafka cluster knode1 Keystone knode2 knode3 ... knodeN Storm cluster Front End Cluster: Multi-tenancy and Kibana, Graphana Proxies Elastic Search Elastic ReSdeiasrch Alerts email & callbacks Load Balancer InfluxDB InfluxDB InfluxDB Metrics Store Elastic Search Elastic Search Elastic Search Log Store LMM Under the Hood Quota & Policy
  9. 9. Current deployment • Multiple deployments : on bare KVM nodes, on OpenStack VMs – On KVM nodes: 40+ VMs, 80+ TB storage, many large memory nodes – All components are deployed in clustered mode for HA – Some with active/active replication, some with active/passive • Use by Platform and Infrastructure Services – Hundreds of thousands of events per second – Few TBs of data collected and indexed per day – Queries are currently coming from Kibana and Grafana, in future from APIs LMM, Cloud Platform Engineering Nov 4, 2014 9
  10. 10. In the Next Iteration … • Support Health checks – Exploring Sensu and related technologies • Multi data center deployment • Refactor to a data pipeline architecture • Deployment automation using Murano • Open Source LMM LMM, Cloud Platform Engineering Nov 4, 2014 10
  11. 11. Thank you! Questions, Comments, Suggestions? We are interested in open sourcing LMM and are looking for collaborators … Interested? Feel free to contact me LN_Renganarayana@Symantec.com LMM, Cloud Platform Engineering Nov 4, 2014 11

×