Submit Search
Upload
Kiwipycon2011 async-with-gevent-redis
•
Download as ODP, PDF
•
11 likes
•
2,478 views
alexdong
Follow
Technology
Education
Report
Share
Report
Share
1 of 13
Download now
Recommended
API analytics with Redis and Google Bigquery. NoSQL matters edition
API analytics with Redis and Google Bigquery. NoSQL matters edition
javier ramirez
Building highly reliable data pipeline @datadog par Quentin François
Building highly reliable data pipeline @datadog par Quentin François
Paris Data Engineers !
Clickhouse at Cloudflare. By Marek Vavrusa
Clickhouse at Cloudflare. By Marek Vavrusa
Valery Tkachenko
Spark Summit EU talk by William Benton
Spark Summit EU talk by William Benton
Spark Summit
Speed Up Uber's Presto with Alluxio
Speed Up Uber's Presto with Alluxio
Alluxio, Inc.
RESTful API – How to Consume, Extract, Store and Visualize Data with InfluxDB...
RESTful API – How to Consume, Extract, Store and Visualize Data with InfluxDB...
InfluxData
Let's Compare: A Benchmark review of InfluxDB and Elasticsearch
Let's Compare: A Benchmark review of InfluxDB and Elasticsearch
InfluxData
DOWNSAMPLING DATA
DOWNSAMPLING DATA
InfluxData
Recommended
API analytics with Redis and Google Bigquery. NoSQL matters edition
API analytics with Redis and Google Bigquery. NoSQL matters edition
javier ramirez
Building highly reliable data pipeline @datadog par Quentin François
Building highly reliable data pipeline @datadog par Quentin François
Paris Data Engineers !
Clickhouse at Cloudflare. By Marek Vavrusa
Clickhouse at Cloudflare. By Marek Vavrusa
Valery Tkachenko
Spark Summit EU talk by William Benton
Spark Summit EU talk by William Benton
Spark Summit
Speed Up Uber's Presto with Alluxio
Speed Up Uber's Presto with Alluxio
Alluxio, Inc.
RESTful API – How to Consume, Extract, Store and Visualize Data with InfluxDB...
RESTful API – How to Consume, Extract, Store and Visualize Data with InfluxDB...
InfluxData
Let's Compare: A Benchmark review of InfluxDB and Elasticsearch
Let's Compare: A Benchmark review of InfluxDB and Elasticsearch
InfluxData
DOWNSAMPLING DATA
DOWNSAMPLING DATA
InfluxData
Netflix running Presto in the AWS Cloud
Netflix running Presto in the AWS Cloud
Zhenxiao Luo
Developing high frequency indicators using real time tick data on apache supe...
Developing high frequency indicators using real time tick data on apache supe...
Zekeriya Besiroglu
Introduction to Meteor - revised edition
Introduction to Meteor - revised edition
Stephan Hochhaus
Clickhouse MeetUp@ContentSquare - ContentSquare's Experience Sharing
Clickhouse MeetUp@ContentSquare - ContentSquare's Experience Sharing
Vianney FOUCAULT
The Practice of Alluxio in JD.com
The Practice of Alluxio in JD.com
Alluxio, Inc.
RedisConf17 - Redis Graph
RedisConf17 - Redis Graph
Redis Labs
Taking Your Database Global with Kubernetes
Taking Your Database Global with Kubernetes
Christopher Bradford
Spark Pipelines in the Cloud with Alluxio
Spark Pipelines in the Cloud with Alluxio
Alluxio, Inc.
H2O World - PySparkling Water - Nidhi Mehta
H2O World - PySparkling Water - Nidhi Mehta
Sri Ambati
Best Practices for Using Alluxio with Spark
Best Practices for Using Alluxio with Spark
Alluxio, Inc.
Getting to Know Airflow
Getting to Know Airflow
Rosanne Hoyem
presto-at-netflix-hadoop-summit-15
presto-at-netflix-hadoop-summit-15
Zhenxiao Luo
Setting up InfluxData for IoT
Setting up InfluxData for IoT
InfluxData
Pig on spark
Pig on spark
Sigmoid
Deep Learning to Production with MLflow & RedisAI
Deep Learning to Production with MLflow & RedisAI
Databricks
Yaroslav Nedashkovsky "How to manage hundreds of pipelines for processing da...
Yaroslav Nedashkovsky "How to manage hundreds of pipelines for processing da...
Lviv Startup Club
Intro to hadoop
Intro to hadoop
Haden Pereira
BeeGFS Enterprise Deployment
BeeGFS Enterprise Deployment
Dirk Petersen
Rental Cars and Industrialized Learning to Rank with Sean Downes
Rental Cars and Industrialized Learning to Rank with Sean Downes
Databricks
Pachyderm: Building a Big Data Beast On Kubernetes
Pachyderm: Building a Big Data Beast On Kubernetes
KubeAcademy
Clug 2011 March web server optimisation
Clug 2011 March web server optimisation
grooverdan
Startup Case Study: Leveraging the Broad Hadoop Ecosystem to Develop World-Fi...
Startup Case Study: Leveraging the Broad Hadoop Ecosystem to Develop World-Fi...
DataWorks Summit
More Related Content
What's hot
Netflix running Presto in the AWS Cloud
Netflix running Presto in the AWS Cloud
Zhenxiao Luo
Developing high frequency indicators using real time tick data on apache supe...
Developing high frequency indicators using real time tick data on apache supe...
Zekeriya Besiroglu
Introduction to Meteor - revised edition
Introduction to Meteor - revised edition
Stephan Hochhaus
Clickhouse MeetUp@ContentSquare - ContentSquare's Experience Sharing
Clickhouse MeetUp@ContentSquare - ContentSquare's Experience Sharing
Vianney FOUCAULT
The Practice of Alluxio in JD.com
The Practice of Alluxio in JD.com
Alluxio, Inc.
RedisConf17 - Redis Graph
RedisConf17 - Redis Graph
Redis Labs
Taking Your Database Global with Kubernetes
Taking Your Database Global with Kubernetes
Christopher Bradford
Spark Pipelines in the Cloud with Alluxio
Spark Pipelines in the Cloud with Alluxio
Alluxio, Inc.
H2O World - PySparkling Water - Nidhi Mehta
H2O World - PySparkling Water - Nidhi Mehta
Sri Ambati
Best Practices for Using Alluxio with Spark
Best Practices for Using Alluxio with Spark
Alluxio, Inc.
Getting to Know Airflow
Getting to Know Airflow
Rosanne Hoyem
presto-at-netflix-hadoop-summit-15
presto-at-netflix-hadoop-summit-15
Zhenxiao Luo
Setting up InfluxData for IoT
Setting up InfluxData for IoT
InfluxData
Pig on spark
Pig on spark
Sigmoid
Deep Learning to Production with MLflow & RedisAI
Deep Learning to Production with MLflow & RedisAI
Databricks
Yaroslav Nedashkovsky "How to manage hundreds of pipelines for processing da...
Yaroslav Nedashkovsky "How to manage hundreds of pipelines for processing da...
Lviv Startup Club
Intro to hadoop
Intro to hadoop
Haden Pereira
BeeGFS Enterprise Deployment
BeeGFS Enterprise Deployment
Dirk Petersen
Rental Cars and Industrialized Learning to Rank with Sean Downes
Rental Cars and Industrialized Learning to Rank with Sean Downes
Databricks
Pachyderm: Building a Big Data Beast On Kubernetes
Pachyderm: Building a Big Data Beast On Kubernetes
KubeAcademy
What's hot
(20)
Netflix running Presto in the AWS Cloud
Netflix running Presto in the AWS Cloud
Developing high frequency indicators using real time tick data on apache supe...
Developing high frequency indicators using real time tick data on apache supe...
Introduction to Meteor - revised edition
Introduction to Meteor - revised edition
Clickhouse MeetUp@ContentSquare - ContentSquare's Experience Sharing
Clickhouse MeetUp@ContentSquare - ContentSquare's Experience Sharing
The Practice of Alluxio in JD.com
The Practice of Alluxio in JD.com
RedisConf17 - Redis Graph
RedisConf17 - Redis Graph
Taking Your Database Global with Kubernetes
Taking Your Database Global with Kubernetes
Spark Pipelines in the Cloud with Alluxio
Spark Pipelines in the Cloud with Alluxio
H2O World - PySparkling Water - Nidhi Mehta
H2O World - PySparkling Water - Nidhi Mehta
Best Practices for Using Alluxio with Spark
Best Practices for Using Alluxio with Spark
Getting to Know Airflow
Getting to Know Airflow
presto-at-netflix-hadoop-summit-15
presto-at-netflix-hadoop-summit-15
Setting up InfluxData for IoT
Setting up InfluxData for IoT
Pig on spark
Pig on spark
Deep Learning to Production with MLflow & RedisAI
Deep Learning to Production with MLflow & RedisAI
Yaroslav Nedashkovsky "How to manage hundreds of pipelines for processing da...
Yaroslav Nedashkovsky "How to manage hundreds of pipelines for processing da...
Intro to hadoop
Intro to hadoop
BeeGFS Enterprise Deployment
BeeGFS Enterprise Deployment
Rental Cars and Industrialized Learning to Rank with Sean Downes
Rental Cars and Industrialized Learning to Rank with Sean Downes
Pachyderm: Building a Big Data Beast On Kubernetes
Pachyderm: Building a Big Data Beast On Kubernetes
Similar to Kiwipycon2011 async-with-gevent-redis
Clug 2011 March web server optimisation
Clug 2011 March web server optimisation
grooverdan
Startup Case Study: Leveraging the Broad Hadoop Ecosystem to Develop World-Fi...
Startup Case Study: Leveraging the Broad Hadoop Ecosystem to Develop World-Fi...
DataWorks Summit
WE18_Performance_Up.ppt
WE18_Performance_Up.ppt
webhostingguy
Handout3o
Handout3o
Shahbaz Sidhu
Improving Hadoop Resiliency and Operational Efficiency with EMC Isilon
Improving Hadoop Resiliency and Operational Efficiency with EMC Isilon
DataWorks Summit/Hadoop Summit
Voldemort & Hadoop @ Linkedin, Hadoop User Group Jan 2010
Voldemort & Hadoop @ Linkedin, Hadoop User Group Jan 2010
Bhupesh Bansal
Hadoop and Voldemort @ LinkedIn
Hadoop and Voldemort @ LinkedIn
Hadoop User Group
Pilot Hadoop Towards 2500 Nodes and Cluster Redundancy
Pilot Hadoop Towards 2500 Nodes and Cluster Redundancy
Stuart Pook
Hadoop World 2011: Building Web Analytics Processing on Hadoop at CBS Interac...
Hadoop World 2011: Building Web Analytics Processing on Hadoop at CBS Interac...
Cloudera, Inc.
Big data at United Airlines
Big data at United Airlines
DataWorks Summit
Eagle from eBay at China Hadoop Summit 2015
Eagle from eBay at China Hadoop Summit 2015
Hao Chen
Inroduction to Big Data
Inroduction to Big Data
Omnia Safaan
Hug syncsort etl hadoop big data
Hug syncsort etl hadoop big data
Stéphane Heckel
Syncsort et le retour d'expérience ComScore
Syncsort et le retour d'expérience ComScore
Modern Data Stack France
Composing re-useable ETL on Hadoop
Composing re-useable ETL on Hadoop
Paul Lam
Hands on Hadoop and pig
Hands on Hadoop and pig
Sudar Muthu
RAPIDS: GPU-Accelerated ETL and Feature Engineering
RAPIDS: GPU-Accelerated ETL and Feature Engineering
Keith Kraus
Arun Rathinasabapathy, Senior Software Engineer, LexisNexis at MLconf ATL 2016
Arun Rathinasabapathy, Senior Software Engineer, LexisNexis at MLconf ATL 2016
MLconf
LLAP: Sub-Second Analytical Queries in Hive
LLAP: Sub-Second Analytical Queries in Hive
DataWorks Summit/Hadoop Summit
Globo.com & Varnish
Globo.com & Varnish
lokama
Similar to Kiwipycon2011 async-with-gevent-redis
(20)
Clug 2011 March web server optimisation
Clug 2011 March web server optimisation
Startup Case Study: Leveraging the Broad Hadoop Ecosystem to Develop World-Fi...
Startup Case Study: Leveraging the Broad Hadoop Ecosystem to Develop World-Fi...
WE18_Performance_Up.ppt
WE18_Performance_Up.ppt
Handout3o
Handout3o
Improving Hadoop Resiliency and Operational Efficiency with EMC Isilon
Improving Hadoop Resiliency and Operational Efficiency with EMC Isilon
Voldemort & Hadoop @ Linkedin, Hadoop User Group Jan 2010
Voldemort & Hadoop @ Linkedin, Hadoop User Group Jan 2010
Hadoop and Voldemort @ LinkedIn
Hadoop and Voldemort @ LinkedIn
Pilot Hadoop Towards 2500 Nodes and Cluster Redundancy
Pilot Hadoop Towards 2500 Nodes and Cluster Redundancy
Hadoop World 2011: Building Web Analytics Processing on Hadoop at CBS Interac...
Hadoop World 2011: Building Web Analytics Processing on Hadoop at CBS Interac...
Big data at United Airlines
Big data at United Airlines
Eagle from eBay at China Hadoop Summit 2015
Eagle from eBay at China Hadoop Summit 2015
Inroduction to Big Data
Inroduction to Big Data
Hug syncsort etl hadoop big data
Hug syncsort etl hadoop big data
Syncsort et le retour d'expérience ComScore
Syncsort et le retour d'expérience ComScore
Composing re-useable ETL on Hadoop
Composing re-useable ETL on Hadoop
Hands on Hadoop and pig
Hands on Hadoop and pig
RAPIDS: GPU-Accelerated ETL and Feature Engineering
RAPIDS: GPU-Accelerated ETL and Feature Engineering
Arun Rathinasabapathy, Senior Software Engineer, LexisNexis at MLconf ATL 2016
Arun Rathinasabapathy, Senior Software Engineer, LexisNexis at MLconf ATL 2016
LLAP: Sub-Second Analytical Queries in Hive
LLAP: Sub-Second Analytical Queries in Hive
Globo.com & Varnish
Globo.com & Varnish
Recently uploaded
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
ThousandEyes
presentation ICT roal in 21st century education
presentation ICT roal in 21st century education
jfdjdjcjdnsjd
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
ThousandEyes
Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Script
wesley chun
🐬 The future of MySQL is Postgres 🐘
🐬 The future of MySQL is Postgres 🐘
RTylerCroy
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
Andrey Devyatkin
A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?
Igalia
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
The Digital Insurer
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
debabhi2
GenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdf
lior mazor
Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024
The Digital Insurer
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organization
Radu Cotescu
Developing An App To Navigate The Roads of Brazil
Developing An App To Navigate The Roads of Brazil
V3cube
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Miguel Araújo
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
?#DUbAI#??##{{(☎️+971_581248768%)**%*]'#abortion pills for sale in dubai@
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Neo4j
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
vu2urc
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
Joaquim Jorge
Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024
The Digital Insurer
Recently uploaded
(20)
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
presentation ICT roal in 21st century education
presentation ICT roal in 21st century education
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Script
🐬 The future of MySQL is Postgres 🐘
🐬 The future of MySQL is Postgres 🐘
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
GenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdf
Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organization
Developing An App To Navigate The Roads of Brazil
Developing An App To Navigate The Roads of Brazil
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024
Kiwipycon2011 async-with-gevent-redis
1.
Writing a
distributed crawler system using gevent and redis Alex Dong @alexdong
2.
3.
Async 101
4.
Gevent: the monkey
king
5.
Redis: data structure
server
6.
Lessons learned
7.
8.
How many links
today?
9.
What was the
project Google employee #1 working on?
10.
11.
DNS resolution
12.
Request HTTP Header
13.
Download full content
14.
Store to local
file store, database and index
15.
Scheduling, throttling, status
monitoring, scale up by flicking on more servers.
16.
17.
Yield on IO/socket,
but
18.
19.
Context switch overhead
20.
21.
Cooperative multitasking
22.
epollfd =
epoll_create ();
23.
epoll_ctl ( epollfd
, EPOLL_CTL_ADD, listen_sock , &ev)
24.
epoll_wait ( epollfd
, events , MAX_EVENTS, -1)
25.
26.
27.
28.
Working: Hashtable -
HSET, HDEL, HEXISTS
29.
One and only
one: Sets - SADD, SPOP
30.
31.
32.
25% code for
status update and monitoring
33.
34.
DNS is down?
35.
ISP's bandwidth?
36.
Large file download?
37.
Scheduler re-submit tasks?
38.
39.
40.
Carefully profile to
make sure socket.recv becomes the main bottleneck.
41.
The get_title
crisis
42.
Rule of thumb:
load average < 1 to saturate 10M bandwidth
43.
Twitter: @alexdong
trunk.ly/?q= from:alexdong + gevent
Download now