Building Rackspace Cloud Monitoring

•

2 likes•1,372 views

Rackspace monitors tens of thousands of servers using several open source tools like Apache Cassandra, Zookeeper, and Scribe. They developed their own tools like Virgo and Dreadnot to deploy agents and configure clusters across datacenters. Regular testing, deployment automation with Dreadnot and Chef, and documentation help support the large-scale monitoring system.

Technology Business

Monitoring @ Scale
How Rackspace does it.

Gary Dusbabek

Outline
1. Our Problems
2. Other Requirements
(Hopes and Dreams)
3. How we did it

• Many tens of thousands of servers
• Several solutions in place
– Merged segments
– Wouldn’t scale to all
– Tedious (Manpower)
– Expensive
• Wanted more

ExternalUsers

No Special Cases

Internal Users

Know about problems

Before customers do.

Gain
Insight
Trending
Capacity Planning
More

MONITORING

Throwing away data
Throwing away knowledge

Y U NO BIG DATA?

API endpoints
Other services that write to Cassandra

node-whiskey˚
node-cassandra-client˚
node-swiz˚
node-elementtree˚

C projects
Reconnoiter
– noitdˆ
– stratcondˆ
Scribe

Java projects
Mainly for concurrency
Apache Cassandraˆ
Apache Zookeeper
Apache Thriftˆ
Metric Ingestion
Complex events (Esper)

A Tale of Two Clusters
Used Differently

Control Cluster
Data Cluster

Data Model
Relational Mismatch
Our Approach
One row per account
Prefixed Ids for col names
Concatenate for parent/child

Data Model
Foo objects get ids like ‘foXXX’

Bar objects get ids like ‘baXXX’

Assume 1:* relationship from foo:bar

Bar 456 that is attached to Foo 123 gets
column name ‘fo123:ba456’

Select ‘fo123:ba’..’fo123:baxefxbfxfb’

Lua/Luvitˆ
Virgo˚ (host agent)
https://github.com/racker/virgo

Vagrant/Chef
Deployment
Development

Open Source Contribution

Dreadnot˚
Inspired by Deployinator
Multi-region deployment tool

https://github.com/racker/dreadnot
Blog Post - http://bit.ly/xks0qT

Supportability
Dashboard
KPI Metrics
Logging
Dump/Load (ops tools)

Automation
• Testing continuously on
buildbot

Deployment
• Dreadnot (bb + chef)
• Constant deployment
and self-monitoring
(nagios + graphite)
• Single-region upgrades

Documentation
Written by programmers

Examples generated
from tests

Thanks!
Private Beta
https://cmbeta.api.rackspacecloud.com/

@gdusbabek

https://github.com/racker

Image Credits
Scale http://www.flickr.com/photos/puuikibeach/4765115333
Pyramids http://www.flickr.com/photos/gracewong/93631410
Hercules http://www.flickr.com/photos/istolethetv/2203377554/
Dandelion http://www.flickr.com/photos/8047705@N02/5572197407/
Ear http://www.flickr.com/photos/perpetualplum/3974880498
Hourglass http://www.flickr.com/photos/22244945@N00/3278869535
Eyeball http://www.flickr.com/photos/miran/6567911705
few gears http://www.flickr.com/photos/arthurjohnpicton/5364226117
Many Gears http://www.flickr.com/photos/mwichary/2294174641
Tools http://www.flickr.com/photos/zzpza/3269784239/
Crane http://www.flickr.com/photos/katatoniq/2075966238/
Ants http://www.flickr.com/photos/dendroica/6170146527
Choice http://www.flickr.com/photos/-bast-/349497988
Old Car http://www.flickr.com/photos/dok1/353601845/
Monster Truck http://www.flickr.com/photos/beadmobile/3279378483/
Clusters http://www.flickr.com/photos/nanagyei/6318995952/
Model http://www.flickr.com/photos/inl/5097547405/
Agent http://www.flickr.com/photos/erix/191965832/
Chef http://www.flickr.com/photos/londonmatt/417683733/
Arrows http://www.flickr.com/photos/generated/2084287794/
Dashboard http://www.flickr.com/photos/80502454@N00/4172458435/
Legos http://www.flickr.com/photos/great8/6820722517/
Xray http://www.flickr.com/photos/karen_roe/4417259305/
Document http://www.flickr.com/photos/tusnelda/6140792529/
Fowers http://www.flickr.com/photos/petercastleton/5905455717/

What's hot

Boot It Up

Joshua Long

This talk describes the tools and techniques I use to diagnose performance problems in .NET and SQL Server web apps. The talk is based on a series of real world examples that allow you to build a picture of the sort of outside-in approach that works well for figuring out performance bottlenecks. Often, once you've isolated the problem, the fix is fairly simple. Where I tend to find people struggle is in isolating problems in the first place so my aim here is to equip you to understand why your app is performing poorly. Increasingly applications are becoming more complex with dependencies not just on the database (or several databases), but often other services. Infrastructure, networking, storage, and the client also increasingly have an impact nowadays, and this is something I try to emphasise throughout, giving you an overview of which tools to use in each case. This is a longer version of the talk than I usually give so I take advantage of the extra time available to discuss an issue that many developers struggle with: memory management. Normally I have to skim over this but in this case I've gone into some depth about how memory is managed in runtimes such as the .NET CLR, the JVM, and JavaScript works. This will help you write code that works with the garbage collector rather than against it.

Extended edition: How to speed up .NET and SQL Server web apps (2 x 45 mins w...

Bart Read

2012-04-21-ignite-offline-mobile-app-has-great-potential

Kenichi Murahashi

5 W's of Hookin'

Nowell VanHoesen

Why you should add React to your Rails application now!

David Roberts

Interested in how spark can fit into a data scientist toolbox? Let Rajiv share his knowledge of using Spark alongside other tools. This talk is from someone that lives in R/Python for most of the day. He will provide an introduction and comparison to how Spark fits in and when its best used. Along the way he will share his thinking on various vendors, languages, and algorithms in spark. As part of the presentation, Rajiv will share some simple examples of using spark streaming, recommenders, as well as other java apps (e.g,H2O) with spark. By the end, you will have one perspective on when its best to use spark, how to use it, and where it is going.

Using Spark Part Time

Rajiv Shah

End-to-End Testing with Cypress

kitconcept GmbH

What's hot (7)

Boot It Up

Extended edition: How to speed up .NET and SQL Server web apps (2 x 45 mins w...

2012-04-21-ignite-offline-mobile-app-has-great-potential

5 W's of Hookin'

Why you should add React to your Rails application now!

Using Spark Part Time

End-to-End Testing with Cypress

Similar to Building Rackspace Cloud Monitoring

Mobile Web Speed Bumps

Nicholas Zakas

Log data contains some of the most valuable raw information you can gather and analyze about your infrastructure and applications. Amid the mess of confusing lines of seemingly random text can be hints about performance, security, flaws in code, user access patterns, and other operational data. Without the proper tools, finding insights in these logs can be like searching for a hay-colored needle in a haystack. In this session you learn what practices and patterns you can easily implement that can help you better understand your log files. You see how you can customize web logs to add more information to them, how to digest logs from around your infrastructure, and how to analyze your log files in near real time.

(WEB301) Operational Web Log Analysis | AWS re:Invent 2014

Amazon Web Services

How Rackspace Cloud Monitoring uses Cassandra

gdusbabek

JRuby, Ruby, Rails and You on the Cloud

Hiro Asari

Using ArcGIS Server with Ruby on Rails

Dave Bouwman

Http/2 - What's it all about?

Andy Davies

D Baker - Galaxy Update

Jan Aerts

This presentation will be useful to those who would like to get acquainted with Apache Spark architecture, top features and see some of them in action, e.g. RDD transformations and actions, Spark SQL, etc. Also it covers real life use cases related to one of ours commercial projects and recall roadmap how we’ve integrated Apache Spark into it. Was presented on Morning@Lohika tech talks in Lviv. Design by Yarko Filevych: http://www.filevych.com/

Introduction to real time big data with Apache Spark

Taras Matyashovsky

Fix me if you can - DrupalCon prague

hernanibf

As a Tester you need to level up. You can do more than functional verification or reporting Response Time In my Performance Clinic Workshops I show you real life exampls on why Applications fail and what you can do to find these problems when you are testing these applications. I am using Free Tools for all of these excercises - especially Dynatrace which gives full End-to-End Visibility (Browser to Database). You can test and download Dynatrace for Free @ http://bit.ly/atd2014challenge

From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

Andreas Grabner

It’s easy to get seduced by being able to quickly deploy and scale applications by using containers. However, when things inevitably go wrong, how do you debug your application? This session covers various pro bug hunting tips and tricks. It shows live demos of tools such as the Docker stats API, Docker exec (and top, vmstat, and netstat), and how to use the ELK stack for centralized logging. It also dives into other more sophisticated tools that operate at the application and (micro)service layer, such as Twitter’s Zipkin tracing app, Spring Boot’s Actuator, and DropWizard’s Metrics library. Keep those container-based nightmares away by ensuring that when the worst does happen, you have the tools, info, and experience to debug containerized applications. Presented at JavaOne 2015 with Steve Poole

J1 2015 "Debugging Java Apps in Containers: No Heavy Welding Gear Required"

Daniel Bryant

An API Your Parents Would Be Proud Of

Jose Alfredo Alvarez Aldana

Rails Performance Tricks and Treats

Marshall Yount

The Case for HTTP/2 - Internetdagarna 2015 - Stockholm

Andy Davies

Whether you are building a mobile app or a web app, Apache Usergrid (incubating) can provide you with a complete backend that supports authentication, persistence and social features like activities and followers all via a comprehensive REST API — and backed by Cassandra, giving you linear scalability. This session will tell you what you need to know to be a Usergrid contributor, starting with the basics of building and running Usergrid from source code. You’ll learn how to find your way around the Usergrid code base, how the code for the Stack, Portal and SDKs and how to use the test infrastructure to test your changes to Usergrid. You’ll learn the Usergrid contributor workflow, how the project uses JIRA and Github to manage change and how to contribute your changes to the project. The session will also cover the Usergrid roadmap and what the community is currently working on.

How to Contribute to Apache Usergrid

David M. Johnson

Our Puppet Story (GUUG FFG 2015)

DECK36

Hadoop application architectures - using Customer 360 as an example

hadooparchbook

Sysdig Monitorama Slides

Loris Degioanni

Machine Learning (ML) is quite popular today in the academic/research world. However, it is quite difficult to put into a product, especially a product with a huge customer base. This session will give Kubeflow, the open source ML toolkit on top of Kubernetes, a deep look from the MLOps perspective. Furthermore, we will have a brief look at Distributed MLSys and how Kubeflow copes with scalability. Last, a demo of the Kubeflow stack setup and ML project pipeline deployment will be demonstrated in the cloud.

Bring-your-ML-Project-into-Production-v2.pdf

Liang Yan

Google App Engine Java, Groovy and Gaelyk

Guillaume Laforge

Similar to Building Rackspace Cloud Monitoring (20)

Mobile Web Speed Bumps

(WEB301) Operational Web Log Analysis | AWS re:Invent 2014

How Rackspace Cloud Monitoring uses Cassandra

JRuby, Ruby, Rails and You on the Cloud

Using ArcGIS Server with Ruby on Rails

Http/2 - What's it all about?

D Baker - Galaxy Update

Introduction to real time big data with Apache Spark

Fix me if you can - DrupalCon prague

From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

J1 2015 "Debugging Java Apps in Containers: No Heavy Welding Gear Required"

An API Your Parents Would Be Proud Of

Rails Performance Tricks and Treats

The Case for HTTP/2 - Internetdagarna 2015 - Stockholm

How to Contribute to Apache Usergrid

Our Puppet Story (GUUG FFG 2015)

Hadoop application architectures - using Customer 360 as an example

Sysdig Monitorama Slides

Bring-your-ML-Project-into-Production-v2.pdf

Google App Engine Java, Groovy and Gaelyk

More from gdusbabek

My Futuristic Vision of the Future of Cassandra's Future - NGCC 2015

gdusbabek

Releasing an open source project while maintaining a shipping product is hard! Different behaviors, attitudes and actions can help or hinder your cause; and they are not always obvious. The Blueflood distributed metrics engine was released as open source software by Rackspace in August 2012. In the succeeding months the team had to strike a manageable balance between the challenges of growing a community, being good open source stewards, and maintaining a shipping product for Rackspace. Find out what worked, what did not work, and the lessons that can be applied as you endeavor to take your project out into the open. In this presentation you will learn about strategies for releasing open source products, pitfalls to avoid, and the potential benefits of moving more of your development out in the open. We have also made a few realizations about the community growing up around metrics. It is still young, and there are problems that come with that youth. I'll talk about some things we can do to make a better software ecosystem.

How To (Not) Open Source - Javazone, Oslo 2014

gdusbabek

Blueflood and Beyond: The Future of Metrics - Berlin Buzzwords 2014

gdusbabek

Measure All the Things! - Austin Data Day 2014

gdusbabek

Blueflood: Open Source Metrics Processing at CassandraEU 2013

gdusbabek

Introduction to Blueflood at Berlin Buzzwords 2013

gdusbabek

Rackspace Cloud Monitoring - Strata NYC

gdusbabek

Breaking the Relational Headlock: A Survey of NoSQL Datastores

gdusbabek

Cassandra Codebase 2011

gdusbabek

Data Modeling with Cassandra Column Families

gdusbabek

Getting to Know the Cassandra Codebase

gdusbabek

Introduction to Cassandra (June 2010)

gdusbabek

Cassandra Presentation for San Antonio JUG

gdusbabek

More from gdusbabek (13)

My Futuristic Vision of the Future of Cassandra's Future - NGCC 2015

How To (Not) Open Source - Javazone, Oslo 2014

Blueflood and Beyond: The Future of Metrics - Berlin Buzzwords 2014

Measure All the Things! - Austin Data Day 2014

Blueflood: Open Source Metrics Processing at CassandraEU 2013

Introduction to Blueflood at Berlin Buzzwords 2013

Rackspace Cloud Monitoring - Strata NYC

Breaking the Relational Headlock: A Survey of NoSQL Datastores

Cassandra Codebase 2011

Data Modeling with Cassandra Column Families

Getting to Know the Cassandra Codebase

Introduction to Cassandra (June 2010)

Cassandra Presentation for San Antonio JUG

Recently uploaded

Three things you will take away from the session: • How to run an effective tenant-to-tenant migration • Best practices for before, during, and after migration • Tips for using migration as a springboard to prepare for Copilot in Microsoft 365 Main ideas: Migration Overview: The presentation covers the current reality of cross-tenant migrations, the triggers, phases, best practices, and benefits of a successful tenant migration Considerations: When considering a migration, it is important to consider the migration scope, performance, customization, flexibility, user-friendly interface, automation, monitoring, support, training, scalability, data integrity, data security, cost, and licensing structure Next Wave: The next wave of change includes the launch of Copilot, which requires businesses to be prepared for upcoming changes related to Copilot and the cloud, and to consolidate data and tighten governance ShareGate: ShareGate can help with pre-migration analysis, configurable migration tool, and automated, end-user driven collaborative governance

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

sammart93

How to convert PDF to text with Nanonets

naman860154

Axa Assurance Maroc - Insurer Innovation Award 2024

The Digital Insurer

In this session, we will delve into strategic approaches for optimizing knowledge management within Microsoft 365, amidst the evolving landscape of Copilot. From leveraging automatic metadata classification and permission governance with SharePoint Premium, to unlocking Viva Engage for the cultivation of knowledge and communities, you will gain actionable insights to bolster your organization's knowledge-sharing initiatives. In this session, we will also explore how to facilitate solutions to enable your employees to find answers and expertise within Microsoft 365. You will leave equipped with practical techniques and a deeper understanding of how there is more to effective knowledge management than just enabling Copilot, but building actual solutions to prepare the knowledge that Copilot and your employees can use.

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Drew Madelung

Evaluating the top large language models.pdf

ChristopherTHyatt

BooK Now Call us at +918448380779 to hire a gorgeous and seductive call girl for sex. Take a Delhi Escort Service. The help of our escort agency is mostly meant for men who want sexual Indian Escorts In Delhi NCR. It should be noted that any impersonator will get 100 attention from our Young Girls Escorts in Delhi. They will assume the position of reliable allies. VIP Call Girl With Original Photos Book Tonight +918448380779 Our Cheap Price 1 Hour not available 2 Hours 5000 Full Night 8000 TAG: Call Girls in Delhi, Noida, Gurgaon, Ghaziabad, Connaught Place, Greater Kailash Delhi, Lajpat Nagar Delhi, Mayur Vihar Delhi, Chanakyapuri Delhi, New Friends Colony Delhi, Majnu Ka Tilla, Karol Bagh, Malviya Nagar, Saket, Khan Market, Noida Sector 18, Noida Sector 76, Noida Sector 51, Gurgaon Mg Road, Iffco Chowk Gurgaon, Rajiv Chowk Gurgaon All Delhi Ncr Free Home Deliver

08448380779 Call Girls In Greater Kailash - I Women Seeking Men

Delhi Call girls

As privacy and data protection regulations evolve rapidly, organizations operating in multiple jurisdictions face mounting challenges to ensure compliance and safeguard customer data. With state-specific privacy laws coming up in multiple states this year, it is essential to understand what their unique data protection regulations will require clearly. How will data privacy evolve in the US in 2024? How to stay compliant? Our panellists will guide you through the intricacies of these states' specific data privacy laws, clarifying complex legal frameworks and compliance requirements. This webinar will review: - The essential aspects of each state's privacy landscape and the latest updates - Common compliance challenges faced by organizations operating in multiple states and best practices to achieve regulatory adherence - Valuable insights into potential changes to existing regulations and prepare your organization for the evolving landscape

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

TrustArc

Tech Trends Report 2024 Future Today Institute.pdf

hans926745

What is a good lead in your organisation? Which leads are priority? What happens to leads? When sales and marketing give different answers to these questions, or perhaps aren't sure of the answers at all, frustrations build and opportunities are left on the table. Join us for an illuminating session with Cian McLoughlin, HubSpot Principal Customer Success Manager, as we look at that crucial piece of the customer journey in which leads are transferred from marketing to sales.

04-2024-HHUG-Sales-and-Marketing-Alignment.pptx

HampshireHUG

Discord is a free app offering voice, video, and text chat functionalities, primarily catering to the gaming community. It serves as a hub for users to create and join servers tailored to their interests. Discord’s ecosystem comprises servers, each functioning as a distinct online community with its own channels dedicated to specific topics or activities. Users can engage in text-based discussions, voice calls, or video chats within these channels. Understanding Discord Servers Discord servers are virtual spaces where users congregate to interact, share content, and build communities. Servers may revolve around gaming, hobbies, interests, or fandoms, providing a platform for like-minded individuals to connect. Communication Features Discord offers a range of communication tools, including text channels for messaging, voice channels for real-time audio conversations, and video channels for face-to-face interactions. These features facilitate seamless communication and collaboration. What Does NSFW Mean? The acronym NSFW stands for “Not Safe For Work,” indicating content that may be inappropriate for professional or public settings. NSFW Content NSFW content encompasses material that is sexually explicit, violent, or otherwise graphic in nature. It often includes nudity, profanity, or depictions of sensitive topics.

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

UK Journal

08448380779 Call Girls In Diplomatic Enclave Women Seeking Men

Delhi Call girls

Enterprise Knowledge’s Urmi Majumder, Principal Data Architecture Consultant, and Fernando Aguilar Islas, Senior Data Science Consultant, presented "Driving Behavioral Change for Information Management through Data-Driven Green Strategy" on March 27, 2024 at Enterprise Data World (EDW) in Orlando, Florida. In this presentation, Urmi and Fernando discussed a case study describing how the information management division in a large supply chain organization drove user behavior change through awareness of the carbon footprint of their duplicated and near-duplicated content, identified via advanced data analytics. Check out their presentation to gain valuable perspectives on utilizing data-driven strategies to influence positive behavioral shifts and support sustainability initiatives within your organization. In this session, participants gained answers to the following questions: - What is a Green Information Management (IM) Strategy, and why should you have one? - How can Artificial Intelligence (AI) and Machine Learning (ML) support your Green IM Strategy through content deduplication? - How can an organization use insights into their data to influence employee behavior for IM? - How can you reap additional benefits from content reduction that go beyond Green IM?

Driving Behavioral Change for Information Management through Data-Driven Gree...

Enterprise Knowledge

In an era where artificial intelligence (AI) stands at the forefront of business innovation, Information Architecture (IA) is at the core of functionality. See “There’s No AI Without IA” – (from 2016 but even more relevant today) Understanding and leveraging how Information Architecture (IA) supports AI synergies between knowledge engineering and prompt engineering is critical for senior leaders looking to successfully deploy AI for internal and externally facing knowledge processes. This webinar be a high-level overview of the methodologies that can elevate AI-driven knowledge processes supporting both employees and customers. Core Insights Include: Strategic Knowledge Engineering: Delve into how structuring AI's knowledge base is required to prevent hallucinations, enable contextual retrieval of accurate information. This will include discussion of gold standard libraries of use cases support testing various LLMs and structures and configurations of knowledge base. Precision in Prompt Engineering: Learn the art of crafting prompts that direct AI to deliver targeted, relevant responses, thereby optimizing customer experiences and business outcomes. Unified Approach for Enhanced AI Performance: Explore the intersection of knowledge and prompt engineering to develop AI systems that are not only more responsive but also aligned with overarching business strategies. Guiding Principles for Implementation: Equip yourself with best practices, ethical guidelines, and strategic considerations for embedding these technologies into your business ecosystem effectively. This webinar is designed to empower business and technology leaders with the knowledge to harness the full potential of AI, ensuring their organizations not only keep pace with digital transformation but lead the charge. Join us to map a roadmap to fully leverage Information Architecture (IA) and AI chart a course towards a future where AI is a key pillar of strategic innovation and business success.

EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx

Earley Information Science

GenAI Risks & Security Meetup 01052024.pdf

lior mazor

This presentations targets students or working professionals. You may know Google for search, YouTube, Android, Chrome, and Gmail, but did you know Google has many developer tools, platforms & APIs? This comprehensive yet still high-level overview outlines the most impactful tools for where to run your code, store & analyze your data. It will also inspire you as to what's possible. This talk is 50 minutes in length.

Powerful Google developer tools for immediate impact! (2023-24 C)

wesley chun

08448380779 Call Girls In Civil Lines Women Seeking Men

Delhi Call girls

Building Digital Trust in a Digital Economy Veronica Tan, Director - Cyber Security Agency of Singapore Apidays Singapore 2024: Connecting Customers, Business and Technology (April 17 & 18, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

apidays

Partners Life - Insurer Innovation Award 2024

The Digital Insurer

Handwritten Text Recognition for manuscripts and early printed texts

Maria Levchenko

MySQL Webinar, presented on the 25th of April, 2024. Summary: MySQL solutions enable the deployment of diverse Database Architectures tailored to specific needs, including High Availability, Disaster Recovery, and Read Scale-Out. With MySQL Shell's AdminAPI, administrators can seamlessly set up, manage, and monitor these solutions, ensuring efficiency and ease of use in their administration. MySQL Router, on the other hand, provides transparent routing from the application traffic to the backend servers in the architectures, requiring minimal configuration. Completely built in-house and supported by Oracle, these solutions have been adopted by enterprises of all sizes for their business-critical applications. In this presentation, we'll delve into various database architecture solutions to help you choose the right one based on your business requirements. Focusing on technical details and the latest features to maximize the potential of these solutions.

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Miguel Araújo

Recently uploaded (20)

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

How to convert PDF to text with Nanonets

Axa Assurance Maroc - Insurer Innovation Award 2024

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Evaluating the top large language models.pdf

08448380779 Call Girls In Greater Kailash - I Women Seeking Men

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

Tech Trends Report 2024 Future Today Institute.pdf

04-2024-HHUG-Sales-and-Marketing-Alignment.pptx

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

08448380779 Call Girls In Diplomatic Enclave Women Seeking Men

Driving Behavioral Change for Information Management through Data-Driven Gree...

EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx

GenAI Risks & Security Meetup 01052024.pdf

Powerful Google developer tools for immediate impact! (2023-24 C)

08448380779 Call Girls In Civil Lines Women Seeking Men

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

Partners Life - Insurer Innovation Award 2024

Handwritten Text Recognition for manuscripts and early printed texts

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Building Rackspace Cloud Monitoring

1. Monitoring @ Scale How Rackspace does it. Gary Dusbabek

2. Outline 1. Our Problems 2. Other Requirements (Hopes and Dreams) 3. How we did it

3. Our Problems

4. • Many tens of thousands of servers • Several solutions in place – Merged segments – Wouldn’t scale to all – Tedious (Manpower) – Expensive • Wanted more

5. Hopes and Dreams

6. ExternalUsers No Special Cases Internal Users

7. Know about problems Before customers do.

8. Gain Insight Trending Capacity Planning More

9. MONITORING Throwing away data Throwing away knowledge Y U NO BIG DATA?

10. Availability Component Failure

11. Availability Datacenter Failure

12. Supportable Debuggable Deployable

13. How We Built It

14. 100% Ownership 5 Devs 2 Ops

15. Technology Choices

16. API endpoints Other services that write to Cassandra node-whiskey˚ node-cassandra-client˚ node-swiz˚ node-elementtree˚

17. C projects Reconnoiter – noitdˆ – stratcondˆ Scribe

18. Java projects Mainly for concurrency Apache Cassandraˆ Apache Zookeeper Apache Thriftˆ Metric Ingestion Complex events (Esper)

19. A Tale of Two Clusters Used Differently Control Cluster Data Cluster

20. Data Model Relational Mismatch Our Approach One row per account Prefixed Ids for col names Concatenate for parent/child

21. Data Model Foo objects get ids like ‘foXXX’ Bar objects get ids like ‘baXXX’ Assume 1:* relationship from foo:bar Bar 456 that is attached to Foo 123 gets column name ‘fo123:ba456’ Select ‘fo123:ba’..’fo123:baxefxbfxfb’

22. Lua/Luvitˆ Virgo˚ (host agent) https://github.com/racker/virgo Vagrant/Chef Deployment Development

23. Open Source Contribution Dreadnot˚ Inspired by Deployinator Multi-region deployment tool https://github.com/racker/dreadnot Blog Post - http://bit.ly/xks0qT

24. Oh noes!

25. Supportability Dashboard KPI Metrics Logging Dump/Load (ops tools)

26. Automation • Testing continuously on buildbot Deployment • Dreadnot (bb + chef) • Constant deployment and self-monitoring (nagios + graphite) • Single-region upgrades

27. Documentation Written by programmers Examples generated from tests

28. Thanks! Private Beta https://cmbeta.api.rackspacecloud.com/ @gdusbabek https://github.com/racker

29. Image Credits Scale http://www.flickr.com/photos/puuikibeach/4765115333 Pyramids http://www.flickr.com/photos/gracewong/93631410 Hercules http://www.flickr.com/photos/istolethetv/2203377554/ Dandelion http://www.flickr.com/photos/8047705@N02/5572197407/ Ear http://www.flickr.com/photos/perpetualplum/3974880498 Hourglass http://www.flickr.com/photos/22244945@N00/3278869535 Eyeball http://www.flickr.com/photos/miran/6567911705 few gears http://www.flickr.com/photos/arthurjohnpicton/5364226117 Many Gears http://www.flickr.com/photos/mwichary/2294174641 Tools http://www.flickr.com/photos/zzpza/3269784239/ Crane http://www.flickr.com/photos/katatoniq/2075966238/ Ants http://www.flickr.com/photos/dendroica/6170146527 Choice http://www.flickr.com/photos/-bast-/349497988 Old Car http://www.flickr.com/photos/dok1/353601845/ Monster Truck http://www.flickr.com/photos/beadmobile/3279378483/ Clusters http://www.flickr.com/photos/nanagyei/6318995952/ Model http://www.flickr.com/photos/inl/5097547405/ Agent http://www.flickr.com/photos/erix/191965832/ Chef http://www.flickr.com/photos/londonmatt/417683733/ Arrows http://www.flickr.com/photos/generated/2084287794/ Dashboard http://www.flickr.com/photos/80502454@N00/4172458435/ Legos http://www.flickr.com/photos/great8/6820722517/ Xray http://www.flickr.com/photos/karen_roe/4417259305/ Document http://www.flickr.com/photos/tusnelda/6140792529/ Fowers http://www.flickr.com/photos/petercastleton/5905455717/

Editor's Notes

Problems = Specifically what we’re trying to solveHopes/Dreams = Things that would be really nice.How We Did It = Processes & tools
As business has changed/segments were merged, others were added.Different teamsNone of the existing solutions will scale to meet the needs of everyone.Not harnessing the data.
Different needs.Internal – hook into ticketing and support.External – monitor anythingOne API for everybody.
Obvious reallyProactive
TrendingCapacity planningAuto scaling
ToolingAPI Driven – Self-serviceAwesome reporting
Open sourceDistributed systemsEmbeddedDevOps
Prefer open source
Polyglot system
Control:Metadata + stateHigh high replication (5 DC)Wide rowsEasy dump and loadData:ArchivalConstant writes3 DC replication
Crud operationsData descriptionSimple parent-child – Simplified relationsPrefixed IDsObject called ‘foo’, id becomes ‘foXXXXXXXX’.Object called ‘bar’, id becomes ‘baXXXXXXX’.1:* relationship from foo:bar, then concatenate column names: foXXXXXXXX:baXXXXXXXXSelect all the bar that belong to fo12345678 becomes a slice from fo12345678:ba..fo12345678:ba\\xef\\xbf\\xbfOne to many from Objects are slices.
Crud operationsData descriptionSimple parent-child – Simplified relationsPrefixed IDsObject called ‘foo’, id becomes ‘foXXXXXXXX’.Object called ‘bar’, id becomes ‘baXXXXXXX’.1:* relationship from foo:bar, then concatenate column names: foXXXXXXXX:baXXXXXXXXSelect all the bar that belong to fo12345678 becomes a slice from fo12345678:ba..fo12345678:ba\\xef\\xbf\\xbfOne to many from Objects are slices.
Stress VISIBILITY!
Deployed 120 times in JanuarySystem changes treated the same way as code changes.

Building Rackspace Cloud Monitoring

Recommended

Recommended

More Related Content

What's hot

What's hot (7)

Similar to Building Rackspace Cloud Monitoring

Similar to Building Rackspace Cloud Monitoring (20)

More from gdusbabek

More from gdusbabek (13)

Recently uploaded

Recently uploaded (20)

Building Rackspace Cloud Monitoring

Editor's Notes