Денис Резник "Моя база данных не справляется с нагрузкой. Что делать?"

•Télécharger en tant que PPTX, PDF•

0 j'aime•565 vues

В течении доклада мы с вами рассмотрим ряд принципов и техник, которые позволят вашей базе данных справляться с большей нагрузкой. P.S. Все примеры и демо будут проводиться на базе данных MS SQL Server. Все совпадения с другими базами данными случайны, но вполне вероятны :) так что знания, полученные в ходе доклада, могут вам пригодиться даже если вы работаете с другой базой данных.

Technologie

Denis Reznik
Intapp, Inc.
My Database can’t handle the load.
What to do?

Agenda
• Highload and Scalability
• My database can’t handle the load
• What to do?

Ok, here we go. My Database
can’t handle the load.
What to do?

STEP 1: Questions
we should know this for sure

Questions
• How do you know that your database can’t
handle a load?
• Which load your database can handle?
• Is load predictable or unpredictable?
• Does only database can’t handle it?
• Has anything been done to improve this?
• Do you have any information which prove that
database is not efficient?

Time for the Questions
• How we came to the current state?
• Were there any particular actions which increase the
load?
• Does load predictable or sporadic?
• Has database started to operate slowly at one
moment or it was a continuous process?
• Which information do we have?

What to do
• Collect Information (current state and
historical)
• Talk to people (well.. we are already doing this )
• Environment information (OS, Database, Hypervisor,
etc)
• Server OS logs
• System resources utilization
• Database queries trace
• Any third-party dashboards and logs

System Resources
• CPU
• Memory
• Storage Subsystem
• Network

CPU – Preemptive Multitasking
Thread 1
Context Switch
End of time Quantum
High Priority Thread
IO
Thread 2

CPU – Cooperative Multitasking
0
SchedulersRunnable
Queue
1
2
3
Suspended Queue
Time Quantum Length for each Scheduler in SQL Server is 4
ms

Memory – OS Management
0001 0010 0101
0000 1001 0101
0011 1010 1001
1110 0010 0110
0001 0010 0101
0000 1001 0101
0011 1010 1001
Physical Memory
(RAM)
Page File
VirtualMemory
Virtual Memory
Manager
SQL Server
Browser
Word

Memory – Database
Data Cache Query Cache
Locks Memory Memory Grants
Some Small Caches
External Memory
Buffer Pool

Storage Subsystem – Database
Buffer Pool
Data Cache
Database File
Access Methods
Buffer Manager
Transaction Log
Log Manager
?
Background
Processes

Network
Bandwidth
TCP Packages
TCP Packages
TCP Packages
Bandwidth
TCP Packages

Solution Variations
• Code or SQL optimization
• Adding more resources to the server (Scale-Up)
• Patching database server or OS
• Distribute database data among several servers
(Scale-Out) (?)
• Switch to another database server (?)

So, here is the end of story
well.. actually not..

STEP 4: Planning
turning off the Emergency mode

Planning
• Hardware capacity planning
• Maintenance planning
• Forecasting workload
• Load testing (ideally)

Summary
• STEP 1: Questions
• STEP 2: Root Cause
• STEP 3: Solution
• STEP 4: Planning

Thank You!
@denisreznik
denisreznik@gmail.com
http://reznik.uneta.com.ua/
https://www.facebook.com/denis.reznik.5
https://www.linkedin.com/pub/denis-reznik/3/502/234

Contenu connexe

Tendances

Benchmarking Top NoSQL Databases: Apache Cassandra, Apache HBase and MongoDB

Athiq Ahamed

Presto at Twitter

Bill Graham

Cassandra is a distributed database with features included but not limited to Secundary Indexes, UDF, Materialized Views, etc. and not so strict hardware requirements. It is important to use those features and select hardware correctly to make sure the use of Cassandra in your business can be as painless as possible. I will address how these features are used in the wrong way, how hardware should be selected, and how to make Cassandra work in the best possible way. Learning Objective #1: Learn that Cassandra hardware requirements exist (and why) and the shortcomings in some of features(Secundary Indexes, Compaction Strategies, etc). Learning Objective #2: The most misused features and common hardware errors. How they might seem harmeless at first (either small cluster or even single node). Learning Objective #3: How to correctly use Cassandra and it's features and go for perfect operation. About the Speaker Carlos Rolo Cassandra Consultant, Pythian Carlos Rolo is a Cassandra MVP, and has deep expertise with distributed architecture technologies. Carlos is driven by challenge, and enjoys the opportunities to discover new things.. He has become known and trusted by customers and colleagues for his ability to understand complex problems, and to work well under pressure. When Carlos isn't working he can be found playing water polo or enjoying the his local community.

Tales From the Field: The Wrong Way of Using Cassandra (Carlos Rolo, Pythian)...

DataStax

Cassandra is getting more and more buzz and that means two things, more development and more issues. Some issues are unavoidable, but some of them are, just by understanding how our tooling works. In this talk I'd like to review the core concepts on which Cassandra is built and how they impose the way we should work with it using some examples that will hopefully give you both a 'Quick Reference' and a 'Checklist' to go through every time you want to build scalable data models. About the Speaker Carlos Alonso Software Engineer, Job and Talent Carlos received his Masters CS at Salamanca University, Spain. He worked a few years there in a digital agency, gaining expertise on a very wide range of technologies before moving to London where he narrowed down the focus on to the backend and data engineering disciplines. The latest step in his professional career was to move back to Madrid to work for Job and Talent where he currently helps on building the best candidate-job opening matching technology. Aside from work he likes sharing as much as he can by public speaking, mentoring or getting involved in OSS or OpenData initiatives.

Scalable Data Modeling by Example (Carlos Alonso, Job and Talent) | Cassandra...

DataStax

No sq lv1_0

Tuan Luong

Efficient in situ processing of various storage types on apache tajo

Hyunsik Choi

ScyllaDB is a distributed database designed to scale horizontally and vertically — in theory. What about in practice? ScyllaDB’s Benny Halevy, Director, Software Engineering, will take you through the process and results of benchmarking our NoSQL database at the petabyte level, showing how you can use advanced features like workload prioritization to control priorities of transactional (read-write) and analytic (read-only) queries on the same cluster with smooth and predictable performance. To watch all of the recordings hosted during Scylla Summit 2022 visit our website here: https://www.scylladb.com/summit.

Scylla Summit 2022: Operating at Monstrous Scales: Benchmarking Petabyte Work...

ScyllaDB

MongoDB Capacity Planning

Norberto Leite

Gruter TECHDAY 2014 Realtime Processing in Telco

Gruter

n this session, you will learn about a solution developed in partnership between Intel and the Michael J. Fox foundation to enable breakthroughs in Parkinson's disease (PD) research, by leveraging wearable sensors and smartphone to monitor PD patient's motor movements 24/7. We'll elaborate on how we're using HBase for time-series data storage and integrating it with various stream, batch, and interactive technologies. We'll also review our efforts to create an interactive querying solution over HBase.

HBaseCon 2015: HBase as an IoT Stream Analytics Platform for Parkinson's Dise...

HBaseCon

Optimizing Presto Connector on Cloud Storage

Kai Sasaki

AWS big-data-demystified #1.1 | Big Data Architecture Lessons Learned | English

Omid Vahdaty

HBaseCon 2013: Near Real Time Indexing for eBay Search

Cloudera, Inc.

Hadoop Meetup Jan 2019 - TonY: TensorFlow on YARN and Beyond

Erik Krogen

Deploying any software can be a challenge if you don't understand how resources are used or how to plan for the capacity of your systems. Whether you need to deploy or grow a single MongoDB instance, replica set, or tens of sharded clusters then you probably share the same challenges in trying to size that deployment. This webinar will cover what resources MongoDB uses, and how to plan for their use in your deployment. Topics covered will include understanding how to model and plan capacity needs for new and growing deployments. The goal of this webinar will be to provide you with the tools needed to be successful in managing your MongoDB capacity planning tasks.

Capacity Planning

MongoDB

MongoDB Deployment Checklist

MongoDB

Presenter: Duy Hai Doan, Technical Advocate at Datastax Libon is a messaging service designed to improve mobile communications through free calls, chat and a voicemail services regardless of operator or Internet access provider. As a mobile communications application, Libon processes billions of messages and calls while backing up billions of contact data. Join this webinar to learn best practices and pitfalls to avoid when tackling a migration project from Relational Database (RDBMS) to Cassandra and how Libon is now able to ingest massive volumes of high velocity data with read and write latency below 10 milliseconds.

Migration Best Practices: From RDBMS to Cassandra without a Hitch

DataStax Academy

Gruter_TECHDAY_2014_03_ApacheTajo (in Korean)

Gruter

DatEngConf SF16 - Apache Kudu: Fast Analytics on Fast Data

Hakka Labs

When it comes time to select database software for your project, there are a bewildering number of choices. How do you know if your project is a good fit for a relational database, or whether one of the many NoSQL options is a better choice? In this webinar you will learn when to use MongoDB and how to evaluate if MongoDB is a fit for your project. You will see how MongoDB's flexible document model is solving business problems in ways that were not previously possible, and how MongoDB's built-in features allow running at scale. Topics covered include: Performance and Scalability MongoDB's Data Model Popular MongoDB Use Cases Customer Stories

Webinar: When to Use MongoDB

MongoDB

Tendances (20)

Benchmarking Top NoSQL Databases: Apache Cassandra, Apache HBase and MongoDB

Presto at Twitter

Tales From the Field: The Wrong Way of Using Cassandra (Carlos Rolo, Pythian)...

Scalable Data Modeling by Example (Carlos Alonso, Job and Talent) | Cassandra...

No sq lv1_0

Efficient in situ processing of various storage types on apache tajo

Scylla Summit 2022: Operating at Monstrous Scales: Benchmarking Petabyte Work...

MongoDB Capacity Planning

Gruter TECHDAY 2014 Realtime Processing in Telco

HBaseCon 2015: HBase as an IoT Stream Analytics Platform for Parkinson's Dise...

Optimizing Presto Connector on Cloud Storage

AWS big-data-demystified #1.1 | Big Data Architecture Lessons Learned | English

HBaseCon 2013: Near Real Time Indexing for eBay Search

Hadoop Meetup Jan 2019 - TonY: TensorFlow on YARN and Beyond

Capacity Planning

MongoDB Deployment Checklist

Migration Best Practices: From RDBMS to Cassandra without a Hitch

Gruter_TECHDAY_2014_03_ApacheTajo (in Korean)

DatEngConf SF16 - Apache Kudu: Fast Analytics on Fast Data

Webinar: When to Use MongoDB

Similaire à Денис Резник "Моя база данных не справляется с нагрузкой. Что делать?"

Optimizing IT Infrastructure For Peak Database Performance

Aventis Systems, Inc.

MySQL Optimization from a Developer's point of view

Sachin Khosla

Average Active Sessions - OaktableWorld 2013

John Beresniewicz

Caching: A Guided Tour - 10/12/2010

Jason Ragsdale

It's harder than ever to predict the load your application will need to handle in advance, so how do you design your architecture so you can afford to implement as you go and be ready for whatever comes your way. It's easy to focus on optimizing each part of your application but your application architecture determines the options you have to make big leaps in scalability. In this talk we'll cover practical patterns you can build today to meet the needs of rapid development while still creating systems that can scale up and out. Specific code examples will focus on .NET but the principles apply across many technologies. Real world systems will be discussed based on our experience helping customers around the world optimize their enterprise applications.

Scaling Systems: Architectures that grow

Gibraltar Software

Data Care, Feeding, and Maintenance

Mercedes Coyle

Doc 2011101412020074

Rhythm Sun

Operations for databases – the agile/devops journey

Eduardo Piairo

10135 b 11

Wichien Saisorn

1. SQL Server forSharePoint geeksA gentle introductionThomas Vochten • Septem...

BIWUG

Whats A Data Warehouse

None None

Ensuring Consistency in a Replicated World

Yelp Engineering

Oracle Database : Addressing a performance issue the drilldown approach

Laurent Leturgez

Today, many businesses around the world are using an Oracle product and in many of these at the core there is an Oracle Database. Many of us who started as a Database administrator where put in this position because we were good PL/SQL programmers or good Sysadmins, but knew very little of what it took to be a DBA. In this session you will learn the core architecture of an Oracle Database in 12c as well as what it takes to administer and apply this new knowledge the day you go back to your office.

DBA 101 : Calling all New Database Administrators (PPT)

Gustavo Rene Antunez

Performance Scenario: Diagnosing and resolving sudden slow down on two node RAC

Kristofferson A

Scaling apps for the big time

proitconsult

Performance

Christophe Marchal

PlayStation and Lucene - Indexing 1M documents per second: Presented by Alexa...

Lucidworks

Performance Tuning

Jannet Peetz

Powering Interactive Data Analysis at Pinterest by Amazon Redshift

Jie Li

Similaire à Денис Резник "Моя база данных не справляется с нагрузкой. Что делать?" (20)

Optimizing IT Infrastructure For Peak Database Performance

MySQL Optimization from a Developer's point of view

Average Active Sessions - OaktableWorld 2013

Caching: A Guided Tour - 10/12/2010

Scaling Systems: Architectures that grow

Data Care, Feeding, and Maintenance

Doc 2011101412020074

Operations for databases – the agile/devops journey

10135 b 11

1. SQL Server forSharePoint geeksA gentle introductionThomas Vochten • Septem...

Whats A Data Warehouse

Ensuring Consistency in a Replicated World

Oracle Database : Addressing a performance issue the drilldown approach

DBA 101 : Calling all New Database Administrators (PPT)

Performance Scenario: Diagnosing and resolving sudden slow down on two node RAC

Scaling apps for the big time

Performance

PlayStation and Lucene - Indexing 1M documents per second: Presented by Alexa...

Performance Tuning

Powering Interactive Data Analysis at Pinterest by Amazon Redshift

Plus de Fwdays

"How Preply reduced ML model development time from 1 month to 1 day",Yevhen Y...

Fwdays

In my talk, I will tell about the world of GenAI services beyond GPT-wrappers and how we developed and scaled GenAI-centric applications. I'll share personal experiences about the obstacles, lessons, and strategic tools and methodologies that were key in taking GenAI applications from 0 to 1. I'll talk about the challenges we faced when launching LLM-based and image generative applications and delivering them to end users, and what conclusions and solutions were made.

"GenAI Apps: Our Journey from Ideas to Production Excellence",Danil Topchii

Fwdays

Python engineers are introduced to the transformative potential of Large Language Models (LLMs) in the realm of advanced data analysis and the application of Semantic Kernel techniques. We will talk about how LLMs like ChatGPT can be integrated into Python environments to automate data processing, enhance predictive modeling, and unlock deeper insights from complex datasets. The session will delve into practical strategies for embedding Semantic Kernel methods within Python projects, illustrating how these advanced techniques can refine the accuracy of machine learning models by embedding domain-specific knowledge directly into the analysis process. Attendees will leave with a clear roadmap for leveraging the combined power of LLMs and Semantic Kernels, equipped with actionable knowledge to drive innovation in their data analysis projects and beyond, marking a significant leap forward in the evolution of Python engineering practices.

"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...

Fwdays

"Federated learning: out of reach no matter how close",Oleksandr Lapshyn

Fwdays

"What is a RAG system and how to build it",Dmytro Spodarets

Fwdays

"Debugging python applications inside k8s environment", Andrii Soldatenko

Fwdays

"ML in Production",Oleksandr Bagan

Fwdays

Ever seen a code base where understanding a simple method meant jumping through tangled class hierarchies? We all have! And while "Favor composition over inheritance!" is almost as old as object-oriented programming, strictly avoiding all types of subclassing leads to verbose, un-Pythonic code. So, what to do? The discussion on composition vs. inheritance is so frustrating because far-reaching design decisions like this can only be made with the ecosystem in mind – and because there's more than one type of subclassing! Let's take a dogma-free stroll through the types of subclassing through a Pythonic lens and untangle some patterns and trade-offs together. By the end, you'll be more confident in deciding when subclassing will make your code more Pythonic and when composition will improve its clarity.

"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack

Fwdays

The current architecture of Prom.ua is built on microservices and GraphQL API, but it was not always like that. In this talk, I'll tell you how far we've come and how we've made using graphs in a microservice architecture convenient and simple. I will talk about the problems we faced and how we overcame them, made our development process more accessible, deployments faster, and the remains of the monolith less loaded.

"Distributed graphs and microservices in Prom.ua", Maksym Kindritskyi

Fwdays

ETL stands for extract, transform, load. It's a process that combines data from different sources into a single repository for further processing, analysis, and utilization. This talk provides an example of how pandas can be used to solve ETL tasks as a stage in the evolution of the data intake component. This involves preliminary validation, filtering, and conversion of data according to a set of business rules and internal representation, with intermediate combination with other sources.

"Rethinking the existing data loading and processing process as an ETL exampl...

Fwdays

I’m confident that many IT professionals are currently facing the same situation I was in a few months ago. Mobilization, uncertainty. How can I be maximally beneficial to the country with my experience and continue professional development in such circumstances? Since the onset of the full-scale invasion, I've been actively volunteering and assisting the army. Mobilization became the next logical step. I want to share: My journey in IT, volunteering, and the beginning of my service in the Armed Forces Impressions from the first few months Which Soft Skills are helpful in this context I aim to dispel myths about the mobilization process and projects of the Armed Forces. Address your questions And yes, military personnel can travel abroad during their leave.

"How Ukrainian IT specialist can go on vacation abroad without crossing the T...

Fwdays

The leader must be strong all the time. The leader cannot afford to make mistakes, let alone fail in front of their team. Is that really true? Nick Gicinto, a cybersecurity leader with over 25 years of experience, who has worked for the CIA and has built security systems from scratch at Tesla and Uber, fully hiring teams for these projects, will talk about the importance of being vulnerable to build trust within a team.

"The Strength of Being Vulnerable: the experience from CIA, Tesla and Uber", ...

Fwdays

Sharing open feedback can be difficult because it equals much work on yourself. However, feedback needs attention and a special place in the corporate culture. It helps to grow dynamically, build a team of like-minded people and achieve powerful results. In the presentation, I will talk about: The ability to work with feedback as a soft, solid skill in developing technical specialists. A list of difficulties that prevent quality work with feedback. The 4A Framework is a tool for successful giving and receiving feedback. I will also help specialists learn the following: Form constructive feedback and understand how and when to give it. Work analytically with the received feedback. Feel free to share your thoughts and be heard.

"[QUICK TALK] Radical candor: how to achieve results faster thanks to a cultu...

Fwdays

Will discuss: Current communication challenges, including mishaps and toxic versus productive interactions. Ever wondered about PDP? It’s likely because its relevance to career planning, even outside your current company, hasn’t been fully spotlighted. Exploring how PDP functions within career planning, applicable even if you’re eyeing an exit. “Who do I aspire to become?” Summarizing key points with a reference to a practical form you can download to use.

"[QUICK TALK] PDP Plan, the only one door to raise your salary and boost care...

Fwdays

This talk will reveal four destructive communication patterns that can undermine team spirit, reduce productivity and cause conflict, and offer effective strategies for neutralizing them. Let's start with exciting storytelling about a fictional team of developers working on Scrum. You will learn about situations that their team member noticed during team meetings. Next, we will analyze "The Gottman Four Horsemen" model, which describes the four "horsemen of the apocalypse" of work relationships: criticism, defensiveness, contempt, and stonewalling. For each of these patterns, specific "antidotes" will be offered that allow you to build healthier and more productive relationships in the team. Finally, we'll look at why this topic is critical to team productivity, drawing on Google's "Project Aristotle" research. Special attention will be paid to the concept of psychological safety, which is a key factor in the success of high-performance teams. This talk will not only provide valuable insights and tools for improving communication and management in Tech teams, but will also help each member better understand their own contribution to the overall success of the team.

"4 horsemen of the apocalypse of working relationships (+ antidotes to them)"...

Fwdays

We are all living in exceptionally turbulent times. Ukrainians are facing numerous crises and challenges, ranging from war to lay-offs. Work provides us with a sense of stability, which many of us deeply value and strive to maintain. However, there are times when we may not realize that our workload becomes overwhelming or that the stress we experience in our jobs becomes highly toxic. This often leads to burnout. During our discussion, we will cover: - The factors that contribute to burnout; - How to identify it; - Short-term strategies for addressing it; - Long-term approaches to finding fulfillment and satisfaction in our work; I will also share my personal experiences and insights, as well as those of my colleagues in the IT field. - Burnout - Stress

"Reconnecting with Purpose: Rediscovering Job Interest after Burnout", Anast...

Fwdays

As you grow in experience as a tech specialist and close the needs of Maslow's hierarchy, you realize that you can help beginners in your specialization. But how to do it effectively? Where to look for future mentees? How to avoid mistakes that demotivate both sides? And, in general, how can you benefit from investing your time in the success of a mentee? I'll discuss this based on my 1.5 years of experience mentoring beginners. It will be useful for professionals with at least six months of experience in IT and a willingness to share their knowledge with the community.

"Mentoring 101: How to effectively invest experience in the success of others...

Fwdays

"Mission (im) possible: How to get an offer in 2024?", Oleksandra Myronova

Fwdays

Historically, Ukrainians are not good at presenting themselves (read: selling) because they are not self-confident. But we were never taught to be confident or to understand our strengths, moreover to pitch ourselves well in 3 minutes at an interview. Therefore, my task is to teach you in a few minutes how to: Create your personal pitch correctly (sales points, structure) Understand, depending on the type of company, what managers want to hear about you at the interview Be able to package irrelevant or no experience Understand your strengths Create an example of a personal pitch No, there will be no information here on how to get more $ at the interview, but it's not sure 😂

"Why have we learned how to package products, but not how to 'package ourselv...

Fwdays

Imposter syndrome is a psychological phenomenon that most of us suffer from to one degree or another. I'm no exception and have faced this for most of my IT career, which has been around 10 years. I did not immediately understand that the impostor syndrome is one of the biggest blockers for my personal and career development. When I understood it, I gradually started, with small and large steps, fighting with a syndrome in all available ways. In my talk, I will share my story and how it helped me grow from an ordinary developer to a team leader and, eventually, the leader of the entire department.

"How to tame the dragon, or leadership with imposter syndrome", Oleksandr Zin...

Fwdays

Plus de Fwdays (20)