Monitoring and metrics in the cloud

•

0 j'aime•1,025 vues

David Lutz

Technologie

The objective of metrics is to
make pretty graphs…

The objective of metrics is to
make pretty graphs...
in order to understand the performance
and capacity
of your systems and how they vary over time.

The objective of monitoring is to…
make the Operations-guy-on-call’s life hell.

The objective of monitoring is to
check that the system is working as expected
and take action if some component isn't.

“Those who cannot remember the past are
condemned to repeat it” - George Santayana

So here’s a case study…

A long time ago in a data centre far,
far away….

Complete system includes humans to run it!
Human Factors Engineering.
http://en.wikipedia.org/wiki/Human_factors

2 x Linux Engineers
1 x Network Engineer
1 x Do Anything Guy
1 x Developer

No Monitoring or Metrics.

Black Box. Completely blind.

Large Development team
External Consultants
ITIL Process people

5 x Linux Engineers
1 x Network Engineer
2 x Database Administrators
and
Part of an Infrastructure team that included
Virtualization specialists
Storage specialists
Hardware specialists

WTF happened? It grew…

Virtualization / Cloud

Cloud / Virtualization

Approximately 400 servers

Still using Nagios and Cacti

15 minutes to add server manually.
1 hour or more to add a new check.

And Ganglia.
And External SAAS tools:
New Relic. Gomez. Omniture.

What’s different about the cloud?

• Servers come and go
• Sometimes automatically with auto-scaling
• Topologies and Architectures change rapidly
• Driven from Configuration Management
Systems

The problems with Nagios

• Clunky UI.
• Monolithic design.
• Hard to scale.
• Hard to add nodes
dynamically.

sensu-server
sensu-client
sensu-api
sensu-dashboard

• JSON everywhere
• Can re-use Nagios checks
• Messaging oriented architecture
• Designed to be driven from
Config Management tools
• Supports dynamic topologies

David Lutz

99designs

@dlutzy

meetup.com/Infrastructure-Coders/

Contenu connexe

Similaire à Monitoring and metrics in the cloud

Webinar: Code Faster on Kubernetes

Ambassador Labs

Infrastructure - a journey from datacentres to cloud

Equal Experts

Tracing Summit 2014, Düsseldorf. What can Linux learn from DTrace: what went well, and what didn't go well, on its path to success? This talk will discuss not just the DTrace software, but lessons from the marketing and adoption of a system tracer, and an inside look at how DTrace was really deployed and used in production environments. It will also cover ongoing problems with DTrace, and how Linux may surpass them and continue to advance the field of system tracing. A world expert and core contributor to DTrace, Brendan now works at Netflix on Linux performance with the various Linux tracers (ftrace, perf_events, eBPF, SystemTap, ktap, sysdig, LTTng, and the DTrace Linux ports), and will summarize his experiences and suggestions for improvements. He has also been contributing to various tracers: recently promoting ftrace and perf_events adoption through articles and front-end scripts, and testing eBPF.

From DTrace to Linux

Brendan Gregg

POD-Diagnosis: Error Detection and Diagnosis of Sporadic Operations on Cloud ...

Liming Zhu

Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...

Demi Ben-Ari

Once you start working with Big Data systems, you discover a whole bunch of problems you won’t find in monolithic systems. Monitoring all of the components becomes a big data problem itself. In the talk we’ll mention all of the aspects that you should take in consideration when monitoring a distributed system using tools like: Web Services,Spark,Cassandra,MongoDB,AWS. Not only the tools, what should you monitor about the actual data that flows in the system? We’ll cover the simplest solution with your day to day open source tools, the surprising thing, that it comes not from an Ops Guy.

Demi Ben-Ari - Monitoring Big Data Systems Done "The Simple Way" - Codemotion...

Codemotion

Lecture 12 monitoring the network

Wiliam Ferraciolli

Once you start working with Big Data systems, you discover a whole bunch of problems you won’t find in monolithic systems. Monitoring all of the components becomes a big data problem itself. In the talk, we’ll mention all of the aspects that you should take into consideration when monitoring a distributed system using tools like Web Services, Spark, Cassandra, MongoDB, AWS. Not only the tools, what should you monitor about the actual data that flows in the system? We’ll cover the simplest solution with your day to day open source tools, the surprising thing, that it comes not from an Ops Guy.

Monitoring Big Data Systems Done "The Simple Way" - Demi Ben-Ari - Codemotion...

Codemotion

Monitoring Big Data Systems "Done the simple way" - Demi Ben-Ari - Codemotion...

Demi Ben-Ari

Which watcher watches CloudWatch

David Lutz

Spaghetti devops

Alessandro Franceschi

Skynet project: Monitor, analyze, scale, and maintain a system in the Cloud

Sylvain Kalache

Conventional wisdom is that microservices is an architecture that is the spiritual successor to service-oriented architecture. While true, this myopic view of microservices ignores some of the profound workflow shifts in today’s microservices organizations. The reality is that microservices is an architecture _and_ workflow. In this talk, we’ll introduce the workflow of service-oriented development. Rafael will talk about how the real goal of microservices is to break up a monolithic development workflow. We’ll show you how, by breaking up your workflow, you can build software that lets you move fast and make things.

QCon SF 2017 - Microservices: Service-Oriented Development

Ambassador Labs

Abstract The idea of this talk is to help development teams to make correct architectural decisions. Andrei will highlight the basic architectural principles and show ways to achieve architecture that is good enough to cover the project requirements and evolve in the future. He will also present several cases from real projects, where wrong, missing, or over-sophisticated architecture decisions really hurt the development teams: - Painful sharing: do shared modules increase reusability or will be the source of problems? - Microservices are the solution to every problem! - Non-extensible extensibility: too sophisticated configuration hurts - Over fine-grained: incorrect splitting to Microservices can make life even harder as with monolith - Convey horizontal split: how organizational driven split can jeopardise the architecture - Model-driven: central responsibility blocks and limits the team - Cargo cult: blindly following patterns and rule can produce an unmaintainable system - Freestyle architecture: what happens if teams completely ignore architecture - Improve with less intelligence: smart endpoint and dumb pipes

Architectural Decisions: Smoothly and Consistently

Comsysto Reply GmbH

Architectural Decisions: Smoothly and Consistently

Comsysto Reply GmbH

The lines between Development and Operations people have gotten blurry and lots of skills needs to be held by both sides. In the talk we'll talk about all of the considerations that are needed to be taken when creating a development and production environment, mentioning Continuous Integration, Continuous Deployment and the Buzzword "DevOps", also talking about some real implementations in the industry. Of course how can we leave out the real enabler of the whole deal, "The Cloud", Giving us a tool set that makes life much easier when implementing all of these practices.

Thinking DevOps in the era of the Cloud - Demi Ben-Ari

Demi Ben-Ari

Once you start working with distributed Big Data systems, you start discovering a whole bunch of problems you won’t find in monolithic systems. All of a sudden to monitor all of the components becomes a big data problem itself. In the talk we’ll mention all of the aspects that you should take in consideration when monitoring a distributed system once you’re using tools like: Web Services, Apache Spark, Cassandra, MongoDB, Amazon Web Services. Not only the tools, what should you monitor about the actual data that flows in the system? And we’ll cover the simplest solution with your day to day open source tools, the surprising thing, that it comes not from an Ops Guy. Demi Ben-Ari is a Co-Founder and CTO @ Panorays. Demi has over 9 years of experience in building various systems both from the field of near real time applications and Big Data distributed systems. Describing himself as a software development groupie, Interested in tackling cutting edge technologies. Demi is also a co-founder of the “Big Things” Big Data community: http://somebigthings.com/big-things-intro/

Monitoring Big Data Systems - "The Simple Way"

Demi Ben-Ari

Continuous Delivery Amsterdam - Microservices in action at the Dutch National...

Bert Jan Schrijver

As data science workloads grow, so does their need for infrastructure. But, is it fair to ask data scientists to also become infrastructure experts? If not the data scientists, then, who is responsible for spinning up and managing data science infrastructure? This talk will address the context in which ML infrastructure is emerging, walk through two examples of ML infrastructure tools for launching hyperparameter optimization jobs, and end with some thoughts for building better tools in the future. Originally given as a talk at the PyData Ann Arbor meetup (https://www.meetup.com/PyData-Ann-Arbor/events/260380989/)

Machine Learning Infrastructure

SigOpt

Scaling Pinterest's Monitoring

Brian Overstreet

Similaire à Monitoring and metrics in the cloud (20)

Webinar: Code Faster on Kubernetes

Infrastructure - a journey from datacentres to cloud

From DTrace to Linux

POD-Diagnosis: Error Detection and Diagnosis of Sporadic Operations on Cloud ...

Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...

Demi Ben-Ari - Monitoring Big Data Systems Done "The Simple Way" - Codemotion...

Lecture 12 monitoring the network

Monitoring Big Data Systems Done "The Simple Way" - Demi Ben-Ari - Codemotion...

Monitoring Big Data Systems "Done the simple way" - Demi Ben-Ari - Codemotion...

Which watcher watches CloudWatch

Spaghetti devops

Skynet project: Monitor, analyze, scale, and maintain a system in the Cloud

QCon SF 2017 - Microservices: Service-Oriented Development

Architectural Decisions: Smoothly and Consistently

Thinking DevOps in the era of the Cloud - Demi Ben-Ari

Monitoring Big Data Systems - "The Simple Way"

Continuous Delivery Amsterdam - Microservices in action at the Dutch National...

Machine Learning Infrastructure

Scaling Pinterest's Monitoring

Plus de David Lutz

2014 devops conferences

David Lutz

antifragility - DevOps Melbourne November 2013

David Lutz

Devops, Dungeons & Dragons

David Lutz

Infrastructure coders logstash

David Lutz

Graphite

David Lutz

Kanban thank you maam

David Lutz

Plus de David Lutz (6)

2014 devops conferences

antifragility - DevOps Melbourne November 2013

Devops, Dungeons & Dragons

Infrastructure coders logstash

Graphite

Kanban thank you maam

Dernier

DBX First Quarter 2024 Investor Presentation

Dropbox

Following the popularity of “Cloud Revolution: Exploring the New Wave of Serverless Spatial Data,” we’re thrilled to announce this much-anticipated encore webinar. In this sequel, we’ll dive deeper into the Cloud-Native realm by uncovering practical applications and FME support for these new formats, including COGs, COPC, FlatGeoBuf, GeoParquet, STAC, and ZARR. Building on the foundation laid by industry leaders Michelle Roby of Radiant Earth and Chris Holmes of Planet in the first webinar, this second part offers an in-depth look at the real-world application and behind-the-scenes dynamics of these cutting-edge formats. We will spotlight specific use-cases and workflows, showcasing their efficiency and relevance in practical scenarios. Discover the vast possibilities each format holds, highlighted through detailed discussions and demonstrations. Our expert speakers will dissect the key aspects and provide critical takeaways for effective use, ensuring attendees leave with a thorough understanding of how to apply these formats in their own projects. Elevate your understanding of how FME supports these cutting-edge technologies, enhancing your ability to manage, share, and analyze spatial data. Whether you’re building on knowledge from our initial session or are new to the serverless spatial data landscape, this webinar is your gateway to mastering cloud-native formats in your workflows.

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Safe Software

ICT role in 21st century education and its challenges

rafiqahmad00786416

CNIC Information System with Pakdata Cf In Pakistan

danishmna97

Discover the innovative features and strategic vision that keep WSO2 an industry leader. Explore the exciting 2024 roadmap of WSO2 API management, showcasing innovations, unified APIM/APK control plane, natural language API interaction, and cloud native agility. Discover how open source solutions, microservices architecture, and cloud native technologies unlock seamless API management in today's dynamic landscapes. Leave with a clear blueprint to revolutionize your API journey and achieve industry success!

WSO2's API Vision: Unifying Control, Empowering Developers

WSO2

Whatsapp Number Escorts Call girls 8617370543 Available 24x7 Mcleodganj Call Girls Service Offer Genuine VIP Model Escorts Call Girls in Your Budget. Mcleodganj Call Girls Service Provide Real Call Girls Number. Make Your Sexual Pleasure Memorable with Our Mcleodganj Call Girls at Affordable Price. Top VIP Escorts Call Girls, High Profile Independent Escorts Call Girls, Housewife Women Escorts Call Girl, College Girls Escorts Call Girls, Russian Escorts Call girls Service in Your Budget.

Mcleodganj Call Girls 🥰 8617370543 Service Offer VIP Hot Model

Deepika Singh

Webinar Recording: https://www.panagenda.com/webinars/why-teams-call-analytics-is-critical-to-your-entire-business Nothing is as frustrating and noticeable as being in an important call and being unable to see or hear the other person. Not surprising then, that issues with Teams calls are among the most common problems users call their helpdesk for. Having in depth insight into everything relevant going on at the user’s device, local network, ISP and Microsoft itself during the call is crucial for good Microsoft Teams Call quality support. To ensure a quick and adequate solution and to ensure your users get the most out of their Microsoft 365. But did you know that ‘bad calls’ are also an excellent indicator of other problems arising? Precisely because it is so noticeable!? Like the canary in the mine, bad calls can be early indicators of problems. Problems that might otherwise not have been noticed for a while but can have a big impact on productivity and satisfaction. Join this session by Christoph Adler to learn how true Microsoft Teams call quality analytics helped other organizations troubleshoot bad calls and identify and fix problems that impacted Teams calls or the use of Microsoft365 in general. See what it can do to keep your users happy and productive! In this session we will cover - Why CQD data alone is not enough to troubleshoot call problems - The importance of attributing call problems to the right call participant - What call quality analytics can do to help you quickly find, fix-, and prevent problems - Why having retrospective detailed insights matters - Real life examples of how others have used Microsoft Teams call quality monitoring to problem shoot problems with their ISP, network, device health and more.

Why Teams call analytics are critical to your entire business

panagenda

Exploring Multimodal Embeddings with Milvus

Zilliz

Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows. We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases. This video focuses on the deployment of external web forms using Jotform for Bonterra Impact Management. This solution can be customized to your organization’s needs and deployed to support the common use cases below: - Intake and consent - Assessments - Surveys - Applications - Program registration Interested in deploying web form automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.

Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...

Jeffrey Haguewood

Join our latest Connector Corner webinar to discover how UiPath Integration Service revolutionizes API-centric automation in a 'Quote to Cash' process—and how that automation empowers businesses to accelerate revenue generation. A comprehensive demo will explore connecting systems, GenAI, and people, through powerful pre-built connectors designed to speed process cycle times. Speakers: James Dickson, Senior Software Engineer Charlie Greenberg, Host, Product Marketing Manager

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

DianaGray10

Six Myths about Ontologies: The Basics of Formal Ontology

johnbeverley2021

Tracing the root cause of a performance issue requires a lot of patience, experience, and focus. It’s so hard that we sometimes attempt to guess by trying out tentative fixes, but that usually results in frustration, messy code, and a considerable waste of time and money. This talk explains how to correctly zoom in on a performance bottleneck using three levels of profiling: distributed tracing, metrics, and method profiling. After we learn to read the JVM profiler output as a flame graph, we explore a series of bottlenecks typical for backend systems, like connection/thread pool starvation, invisible aspects, blocking code, hot CPU methods, lock contention, and Virtual Thread pinning, and we learn to trace them even if they occur in library code you are not familiar with. Attend this talk and prepare for the performance issues that will eventually hit any successful system. About authorWith two decades of experience, Victor is a Java Champion working as a trainer for top companies in Europe. Five thousands developers in 120 companies attended his workshops, so he gets to debate every week the challenges that various projects struggle with. In return, Victor summarizes key points from these workshops in conference talks and online meetups for the European Software Crafters, the world’s largest developer community around architecture, refactoring, and testing. Discover how Victor can help you on victorrentea.ro : company training catalog, consultancy and YouTube playlists.

Finding Java's Hidden Performance Traps @ DevoxxUK 2024

Victor Rentea

[BuildWithAI] Introduction to Gemini.pdf

Sandro Moreira

Scaling API-first – The story of a global engineering organization Ian Reasor, Senior Computer Scientist - Adobe Radu Cotescu, Senior Computer Scientist - Adobe Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

apidays

Architecting Cloud Native Applications

WSO2

Keynote 2: APIs in 2030: The Risk of Technological Sleepwalk Paolo Malinverno, Growth Advisor - The Business of Technology Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...

apidays

Three things you will take away from the session: • How to run an effective tenant-to-tenant migration • Best practices for before, during, and after migration • Tips for using migration as a springboard to prepare for Copilot in Microsoft 365 Main ideas: Migration Overview: The presentation covers the current reality of cross-tenant migrations, the triggers, phases, best practices, and benefits of a successful tenant migration Considerations: When considering a migration, it is important to consider the migration scope, performance, customization, flexibility, user-friendly interface, automation, monitoring, support, training, scalability, data integrity, data security, cost, and licensing structure Next Wave: The next wave of change includes the launch of Copilot, which requires businesses to be prepared for upcoming changes related to Copilot and the cloud, and to consolidate data and tighten governance ShareGate: ShareGate can help with pre-migration analysis, configurable migration tool, and automated, end-user driven collaborative governance

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

sammart93

"I see eyes in my soup": How Delivery Hero implemented the safety system for ...

Zilliz

The value of a flexible API Management solution for Open Banking Steve Melan, Manager for IT Innovation and Architecture - State's and Saving's Bank of Luxembourg Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The value of a flexible API Management solution for O...

apidays

Artificial Intelligence Chap.5 : Uncertainty

Khushali Kathiriya

Dernier (20)

DBX First Quarter 2024 Investor Presentation

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

ICT role in 21st century education and its challenges

CNIC Information System with Pakdata Cf In Pakistan

WSO2's API Vision: Unifying Control, Empowering Developers

Mcleodganj Call Girls 🥰 8617370543 Service Offer VIP Hot Model

Why Teams call analytics are critical to your entire business

Exploring Multimodal Embeddings with Milvus

Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

Six Myths about Ontologies: The Basics of Formal Ontology

Finding Java's Hidden Performance Traps @ DevoxxUK 2024

[BuildWithAI] Introduction to Gemini.pdf

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

Architecting Cloud Native Applications

Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

"I see eyes in my soup": How Delivery Hero implemented the safety system for ...

Apidays New York 2024 - The value of a flexible API Management solution for O...

Artificial Intelligence Chap.5 : Uncertainty

Monitoring and metrics in the cloud

1. Metrics and Monitoring in the cloud David Lutz @dlutzy

2. The objective of metrics is to make pretty graphs…

3. The objective of metrics is to make pretty graphs... in order to understand the performance and capacity of your systems and how they vary over time.

4. The objective of monitoring is to… make the Operations-guy-on-call’s life hell.

5. The objective of monitoring is to check that the system is working as expected and take action if some component isn't.

6. “Those who cannot remember the past are condemned to repeat it” - George Santayana So here’s a case study…

7. A long time ago in a data centre far, far away….

9. Complete system includes humans to run it! Human Factors Engineering. http://en.wikipedia.org/wiki/Human_factors 2 x Linux Engineers 1 x Network Engineer 1 x Do Anything Guy 1 x Developer

10. No Monitoring or Metrics. Black Box. Completely blind.

11. MRTG Net Saint

12.

13. Large Development team External Consultants ITIL Process people 5 x Linux Engineers 1 x Network Engineer 2 x Database Administrators and Part of an Infrastructure team that included Virtualization specialists Storage specialists Hardware specialists

14. WTF happened? It grew… Virtualization / Cloud Cloud / Virtualization

15. Approximately 400 servers Still using Nagios and Cacti 15 minutes to add server manually. 1 hour or more to add a new check.

16. And Ganglia. And External SAAS tools: New Relic. Gomez. Omniture.

17. #monitoringsucks

18. Getting it right

19. Getting it wrong

20. What’s different about the cloud? • Servers come and go • Sometimes automatically with auto-scaling • Topologies and Architectures change rapidly • Driven from Configuration Management Systems

21. The problems with Nagios • Clunky UI. • Monolithic design. • Hard to scale. • Hard to add nodes dynamically.

22. #doingitright

23. #doingitwrong

24. Sensu… Is it the Nagios killer?

25. sensu-server sensu-client sensu-api sensu-dashboard

26. • JSON everywhere • Can re-use Nagios checks • Messaging oriented architecture • Designed to be driven from Config Management tools • Supports dynamic topologies

27. ?

28. ?

29. David Lutz 99designs @dlutzy meetup.com/Infrastructure-Coders/

Monitoring and metrics in the cloud

Recommandé

Recommandé

Contenu connexe

Similaire à Monitoring and metrics in the cloud

Similaire à Monitoring and metrics in the cloud (20)

Plus de David Lutz

Plus de David Lutz (6)

Dernier

Dernier (20)

Monitoring and metrics in the cloud