Gogolook SQS lesson learnt

•

1 like•565 views

cc liu

SQS lesson learnt

Technology

Hello!
I am Kakashi
You can find me at:
Twitter: @kakashiliu

Previous Status
◎ Worker: c3.large * 30~50, spot instance
◎ CPU utilization: less than 20%
◎ Messages: over than 1 billion / per day

SQS 基本設定 2
◎ First 1 million SQS requests per month are free
◎ $0.476 / per 1 million SQS requests
◎ A single request can have from 1 to 10 messages,
up to maximum total payload of 256 kb
◎ requests including:
CreateQueue, ListQueues, DeleteQueue, SendMessage,
SendMessageBatch, ReceiveMessage, ChangeMessageVisibility,
ChangeMessageVisibilityBatch, DeleteMessage, DeleteMessageBatch,
SetQueueAttributes, GetQueueAttributes, GetQueueUrl, AddPermission,
and RemovePermission.

SQS新政1 : 流程改善
◎ 每個worker每隔1min, 用crontab叫他們復活, 沒事幹就死掉
=> cpu utilization low, cpu loading high
◎ Solution:
利用 while True 加上 SQS long polling (20s) , 沒事做的時候
sleep 60s
◎ 結果: Loading降低, CPU utilization 沒提升很多

SQS新政2 : 斷開鎖鏈
◎ 發現有network bound: 3rd library loggly 沒有使用keep-alive
的方式連接
◎ Solution:
1) python lib 替換 urllib2 為 requests.session
2) 並且使用batch sent log
◎ 結果: 清queue速度快一倍
worker cpu utilization 上升數倍

SQS新政3 : 減少request量
◎ 利用sent message batch的方式, Batch sent 100 messages or
塞滿256kb在發送
◎ Solution:
patch掉原本送queue的function, 並且利用 atexit 去避免
worker被關閉時, 有些message還在 local queue裡面
◎ 結果: sent, receive , delete 量減少99%

SQS新政4 : 調整單機工人數目
◎ Message 多的開多一點 thread去處理
◎ 其中還可以利用 gevent 去避開一些 network bound的問題

Result
◎ Spot instance: 30~50 -> 2
◎ CPU utilization: 20% -> 80%
◎ Message Counts: 1 billion -> 10million
◎ 處理時間

Traditional Flow
AP server SQS Worker Mongodb
Big Query

Fluentd Flow
AP server
Fluentd
Forwarder
Fluentd
Station
Mongodb
Big Query

Fluentd Flow
AP server
Fluentd
Forwarder
Fluentd
Station
ELB
Fluentd
Station
Fluentd
Station
Fluentd
Station

Thanks!
Any questions?
You can find me at:
@kakashiliu
kakashi@gogolook.com

Viewers also liked

Project Managment Program

Rajendran Mohanram

Project Managment

Koshy Tk

Project Managment Thinking

Emily Clasper

Mjedisi

Klarisa Klara

ENGLAND PRESENTATION

adamson1218

Project managment ppt

Gezehagne Andarssa Hunde

Project Managment Good Practices

Hari Thapliyal

England Presentation

mohamedreda

184 slides includes: the definition of projects, characteristics of projects, what project is not, project management basics, characteristics, and successful project management. The process of project management, role of a project manager, learning from best practices, project management practices, the project management institute, PMBOK body of knowledge, connecting project with strategy, project management process, how to create a project plan, project management challenges and advantages, the cycles and challenges of project management, project scheduling, building project teams, project requirements, project processes, celebrations, recognitions, rewards, how to's and more.

Project Management PowerPoint PPT Content Modern Sample

Andrew Schwartz

Project Management PPT

bletindia

PMI Project Management Principles

tltiede

Fundamentals of Project Management

Rodolfo Siles

Viewers also liked (12)

Project Managment Program

Project Managment

Project Managment Thinking

Mjedisi

ENGLAND PRESENTATION

Project managment ppt

Project Managment Good Practices

England Presentation

Project Management PowerPoint PPT Content Modern Sample

Project Management PPT

PMI Project Management Principles

Fundamentals of Project Management

Recently uploaded

Three things you will take away from the session: • How to run an effective tenant-to-tenant migration • Best practices for before, during, and after migration • Tips for using migration as a springboard to prepare for Copilot in Microsoft 365 Main ideas: Migration Overview: The presentation covers the current reality of cross-tenant migrations, the triggers, phases, best practices, and benefits of a successful tenant migration Considerations: When considering a migration, it is important to consider the migration scope, performance, customization, flexibility, user-friendly interface, automation, monitoring, support, training, scalability, data integrity, data security, cost, and licensing structure Next Wave: The next wave of change includes the launch of Copilot, which requires businesses to be prepared for upcoming changes related to Copilot and the cloud, and to consolidate data and tighten governance ShareGate: ShareGate can help with pre-migration analysis, configurable migration tool, and automated, end-user driven collaborative governance

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

sammart93

Following the popularity of “Cloud Revolution: Exploring the New Wave of Serverless Spatial Data,” we’re thrilled to announce this much-anticipated encore webinar. In this sequel, we’ll dive deeper into the Cloud-Native realm by uncovering practical applications and FME support for these new formats, including COGs, COPC, FlatGeoBuf, GeoParquet, STAC, and ZARR. Building on the foundation laid by industry leaders Michelle Roby of Radiant Earth and Chris Holmes of Planet in the first webinar, this second part offers an in-depth look at the real-world application and behind-the-scenes dynamics of these cutting-edge formats. We will spotlight specific use-cases and workflows, showcasing their efficiency and relevance in practical scenarios. Discover the vast possibilities each format holds, highlighted through detailed discussions and demonstrations. Our expert speakers will dissect the key aspects and provide critical takeaways for effective use, ensuring attendees leave with a thorough understanding of how to apply these formats in their own projects. Elevate your understanding of how FME supports these cutting-edge technologies, enhancing your ability to manage, share, and analyze spatial data. Whether you’re building on knowledge from our initial session or are new to the serverless spatial data landscape, this webinar is your gateway to mastering cloud-native formats in your workflows.

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Safe Software

Angeliki Cooney has spent over twenty years at the forefront of the life sciences industry, working out of Wynantskill, NY. She is highly regarded for her dedication to advancing the development and accessibility of innovative treatments for chronic diseases, rare disorders, and cancer. Her professional journey has centered on strategic consulting for biopharmaceutical companies, facilitating digital transformation, enhancing omnichannel engagement, and refining strategic commercial practices. Angeliki's innovative contributions include pioneering several software-as-a-service (SaaS) products for the life sciences sector, earning her three patents. As the Senior Vice President of Life Sciences at Avenga, Angeliki orchestrated the firm's strategic entry into the U.S. market. Avenga, a renowned digital engineering and consulting firm, partners with significant entities in the pharmaceutical and biotechnology fields. Her leadership was instrumental in expanding Avenga's client base and establishing its presence in the competitive U.S. market.

Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...

Angeliki Cooney

MS Copilot expands with MS Graph connectors

Nanddeep Nachan

Elevate Developer Efficiency & build GenAI Application with Amazon Q

Bhuvaneswari Subramani

Whatsapp Number Escorts Call girls 8617370543 Available 24x7 Mcleodganj Call Girls Service Offer Genuine VIP Model Escorts Call Girls in Your Budget. Mcleodganj Call Girls Service Provide Real Call Girls Number. Make Your Sexual Pleasure Memorable with Our Mcleodganj Call Girls at Affordable Price. Top VIP Escorts Call Girls, High Profile Independent Escorts Call Girls, Housewife Women Escorts Call Girl, College Girls Escorts Call Girls, Russian Escorts Call girls Service in Your Budget.

Mcleodganj Call Girls 🥰 8617370543 Service Offer VIP Hot Model

Deepika Singh

The value of a flexible API Management solution for Open Banking Steve Melan, Manager for IT Innovation and Architecture - State's and Saving's Bank of Luxembourg Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The value of a flexible API Management solution for O...

apidays

Dubai, often portrayed as a shimmering oasis in the desert, faces its own set of challenges, including the occasional threat of flooding. Despite its reputation for opulence and modernity, the emirate is not immune to the forces of nature. In recent years, Dubai has experienced sporadic but significant floods, testing the resilience of its infrastructure and communities. Among the critical lifelines in this bustling metropolis is the Dubai International Airport, a bustling hub that connects the city to the world. This article explores the intersection of Dubai flood events and the resilience demonstrated by the Dubai International Airport in the face of such challenges.

Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...

Orbitshub

Discover the innovative features and strategic vision that keep WSO2 an industry leader. Explore the exciting 2024 roadmap of WSO2 API management, showcasing innovations, unified APIM/APK control plane, natural language API interaction, and cloud native agility. Discover how open source solutions, microservices architecture, and cloud native technologies unlock seamless API management in today's dynamic landscapes. Leave with a clear blueprint to revolutionize your API journey and achieve industry success!

WSO2's API Vision: Unifying Control, Empowering Developers

WSO2

Keynote 2: APIs in 2030: The Risk of Technological Sleepwalk Paolo Malinverno, Growth Advisor - The Business of Technology Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...

apidays

💥 You’re lucky! We’ve found two different (lead) developers that are willing to share their valuable lessons learned about using UiPath Document Understanding! Based on recent implementations in appealing use cases at Partou and SPIE. Don’t expect fancy videos or slide decks, but real and practical experiences that will help you with your own implementations. 📕 Topics that will be addressed: • Training the ML-model by humans: do or don't? • Rule-based versus AI extractors • Tips for finding use cases • How to start 👨‍🏫👨‍💻 Speakers: o Dion Morskieft, RPA Product Owner @Partou o Jack Klein-Schiphorst, Automation Developer @Tacstone Technology

DEV meet-up UiPath Document Understanding May 7 2024 Amsterdam

UiPathCommunity

Vector Search -An Introduction in Oracle Database 23ai.pptx

Remote DBA Services

ICT role in 21st century education and its challenges

rafiqahmad00786416

Strategies for Landing an Oracle DBA Job as a Fresher

Remote DBA Services

DBX First Quarter 2024 Investor Presentation

Dropbox

Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows. We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases. This video focuses on the deployment of external web forms using Jotform for Bonterra Impact Management. This solution can be customized to your organization’s needs and deployed to support the common use cases below: - Intake and consent - Assessments - Surveys - Applications - Program registration Interested in deploying web form automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.

Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...

Jeffrey Haguewood

Scaling API-first – The story of a global engineering organization Ian Reasor, Senior Computer Scientist - Adobe Radu Cotescu, Senior Computer Scientist - Adobe Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

apidays

Architecting Cloud Native Applications

WSO2

Following the popularity of "Cloud Revolution: Exploring the New Wave of Serverless Spatial Data," we're thrilled to announce this much-anticipated encore webinar. In this sequel, we'll dive deeper into the Cloud-Native realm by uncovering practical applications and FME support for these new formats, including COGs, COPC, FlatGeoBuf, GeoParquet, STAC, and ZARR. Building on the foundation laid by industry leaders Michelle Roby of Radiant Earth and Chris Holmes of Planet in the first webinar, this second part offers an in-depth look at the real-world application and behind-the-scenes dynamics of these cutting-edge formats. We will spotlight specific use-cases and workflows, showcasing their efficiency and relevance in practical scenarios. Discover the vast possibilities each format holds, highlighted through detailed discussions and demonstrations. Our expert speakers will dissect the key aspects and provide critical takeaways for effective use, ensuring attendees leave with a thorough understanding of how to apply these formats in their own projects. Elevate your understanding of how FME supports these cutting-edge technologies, enhancing your ability to manage, share, and analyze spatial data. Whether you're building on knowledge from our initial session or are new to the serverless spatial data landscape, this webinar is your gateway to mastering cloud-native formats in your workflows.

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Safe Software

Corporate and higher education. Two industries that, in the past, have had a clear divide with very little crossover. The difference in goals, learning styles and objectives paved the way for differing learning technologies platforms to evolve. Now, those stark lines are blurring as both sides are discovering they have content that’s relevant to the other. Join Tammy Rutherford as she walks through the pros and cons of corporate and higher ed collaborating. And the challenges of these different technology platforms working together for a brighter future.

Corporate and higher education May webinar.pptx

Rustici Software

Recently uploaded (20)

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...

MS Copilot expands with MS Graph connectors

Elevate Developer Efficiency & build GenAI Application with Amazon Q

Mcleodganj Call Girls 🥰 8617370543 Service Offer VIP Hot Model

Apidays New York 2024 - The value of a flexible API Management solution for O...

Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...

WSO2's API Vision: Unifying Control, Empowering Developers

Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...

DEV meet-up UiPath Document Understanding May 7 2024 Amsterdam

Vector Search -An Introduction in Oracle Database 23ai.pptx

ICT role in 21st century education and its challenges

Strategies for Landing an Oracle DBA Job as a Fresher

DBX First Quarter 2024 Investor Presentation

Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

Architecting Cloud Native Applications

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Corporate and higher education May webinar.pptx

Gogolook SQS lesson learnt

1. Gogolook SQS lesson learnt

2. Hello! I am Kakashi You can find me at: Twitter: @kakashiliu

3. The story is ...

5. Previous Status ◎ Worker: c3.large * 30~50, spot instance ◎ CPU utilization: less than 20% ◎ Messages: over than 1 billion / per day

6. SQS 基本介紹

7. SQS 基本設定 2 ◎ First 1 million SQS requests per month are free ◎ $0.476 / per 1 million SQS requests ◎ A single request can have from 1 to 10 messages, up to maximum total payload of 256 kb ◎ requests including: CreateQueue, ListQueues, DeleteQueue, SendMessage, SendMessageBatch, ReceiveMessage, ChangeMessageVisibility, ChangeMessageVisibilityBatch, DeleteMessage, DeleteMessageBatch, SetQueueAttributes, GetQueueAttributes, GetQueueUrl, AddPermission, and RemovePermission.

8. SQS新政1 : 流程改善 ◎ 每個worker每隔1min, 用crontab叫他們復活, 沒事幹就死掉 => cpu utilization low, cpu loading high ◎ Solution: 利用 while True 加上 SQS long polling (20s) , 沒事做的時候 sleep 60s ◎ 結果: Loading降低, CPU utilization 沒提升很多

9. SQS新政2 : 斷開鎖鏈 ◎ 發現有network bound: 3rd library loggly 沒有使用keep-alive 的方式連接 ◎ Solution: 1) python lib 替換 urllib2 為 requests.session 2) 並且使用batch sent log ◎ 結果: 清queue速度快一倍 worker cpu utilization 上升數倍

10. SQS新政3 : 減少request量 ◎ 利用sent message batch的方式, Batch sent 100 messages or 塞滿256kb在發送 ◎ Solution: patch掉原本送queue的function, 並且利用 atexit 去避免 worker被關閉時, 有些message還在 local queue裡面 ◎ 結果: sent, receive , delete 量減少99%

11. SQS新政4 : 調整單機工人數目 ◎ Message 多的開多一點 thread去處理 ◎ 其中還可以利用 gevent 去避開一些 network bound的問題

12. Result ◎ Spot instance: 30~50 -> 2 ◎ CPU utilization: 20% -> 80% ◎ Message Counts: 1 billion -> 10million ◎ 處理時間

13. Result Cost Reduction around 70%

14. Bonus: SQS alternative

15. Fluentd (Before)

16. Fluentd (After)

17. Traditional Flow AP server SQS Worker Mongodb Big Query

18. Fluentd Flow AP server Fluentd Forwarder Fluentd Station Mongodb Big Query

19. Fluentd Flow AP server Fluentd Forwarder Fluentd Station ELB Fluentd Station Fluentd Station Fluentd Station

20. Thanks! Any questions? You can find me at: @kakashiliu kakashi@gogolook.com