SlideShare une entreprise Scribd logo
1  sur  27
Télécharger pour lire hors ligne
Celery for internal API in
  SOA infrastructure
        Roman Imankulov
     PyCon Russia, Feb 2013
1. Internal API. What's the deal
SOA service architecture example
1. Internal API. What's the deal

● DCOM: too Microsoftish
● CORBA: too Enterprisy
● SOAP: XML inside
● RESTful: too sloppy



Celery to the rescue?
Agenda
1. Internal API. What's the deal

2. How to use Celery for internal API

3. Organizing Celery-based API. Routing

4. Celery-based API. Benefits, pitfalls and

   security notes
2. How to use Celery for internal API
             Celery Worker
2. How to use Celery for internal API
  Classic celery client imports task function
2. How to use Celery for internal API
Classic celery client imports makes you:
● install everything
   ○ slows down installation

● import everything
   ○ python processes grow in size
   ○ circular dependencies
   ○ tight coupling

● share everything
   ○ ... with outsource developers
2. How to use Celery for internal API
            "Task implementation is not to
             be exposed without necessity"
                           William of Ockham
2. How to use Celery for internal API
  Low-level interface defines only task name
2. How to use Celery for internal API
Convenient API requires some boilerplate code
2. How to use Celery for internal API
Make it even more convenient with
introspection
2. How to use Celery for internal API
Introspection magic with celery-api




http://github.com/imankulov/celery-api
3. Celery-based API. Routing
3. Celery-based API. Routing
Creating API endpoint

1. Common package for all API endpoints
2. Every worker imports celery instance
3. Expose tasks for worker with
   CELERY_IMPORTS
4. Launch worker with
   celery worker -A module_name
                     -Q foo
                     -n foo
3. Celery-based API. Routing
Send tasks with

add.apply_async(queue='foo',
                args=(1, 1))
3. Celery-based API. Routing
Define routing rules
3. Celery-based API. Routing
Task in the wire
{ "body": <base64 encoded string with task name and args>,
  "properties": {
     "body_encoding": "base64",
     "delivery_info": {
        "priority": 0, "routing_key": "foo", "exchange":
"foo"
     },
     "delivery_mode": 2,
     "delivery_tag": <UUID>
  },
  ...}
3. Celery-based API. Routing
Task in the wire


     "delivery_info": {
       "priority": 0,
       "routing_key": "foo",
       "exchange": "foo"
     }
3. Celery-based API. Routing
AMQP for dummies
3. Celery-based API. Routing
Define all queues (API endpoints) you have
4. Celery-based API. Benefits
High level protocol.

Takes the responsibility for

                   data serialization
                   routing
                   exception handling
4. Celery-based API. Benefits
Support for wide range of brokers.

You can switch between brokers with no major
changes in your code

Brokers:
RabbitMQ Redis SQLAlchemy
Django MongoDB Amazon SQS
CouchDB Beanstalk
4. Celery-based API. Benefits
Built-in extra stuff which you were afraid to ever
dream of.

asynchronous execution        parallel execution
asynchronous parallel execution
delayed execution         throttling
automatic retrying of failed tasks
limiting the time of execution
autoscaling API inspection
public key cryptography (message signing)
4. Celery-based API. Pitfalls
Python-centric.


Celery is written in Python and meant to be
used with Python code exclusively
4. Celery-based API. Pitfalls
Increasing complexity.

Celery codebase          46k LOC
Kombu codebase           19k LOC

Compare
Flask codebase           10k LOC
Werkzeug codebase        28k LOC
Jinja2 codebase            16k LOC
4. Celery-based API. Security note




Never ever expose Celery broker to the
Web!
4. Celery-based API. Security note

1. Try
>>> redis_url = 'redis://:pycon2013@broker.imankulov.name'
>>> celery = Celery(broker=redis_url, result=redis_url)



2. Get the contents of /etc/pycon_secret

3. Exchange it for a secret prize ;)

Contenu connexe

Tendances

Controlling multiple VMs with the power of Python
Controlling multiple VMs with the power of PythonControlling multiple VMs with the power of Python
Controlling multiple VMs with the power of PythonYurii Vasylenko
 
Introduction to Ansible (Pycon7 2016)
Introduction to Ansible (Pycon7 2016)Introduction to Ansible (Pycon7 2016)
Introduction to Ansible (Pycon7 2016)Ivan Rossi
 
Advanced task management with Celery
Advanced task management with CeleryAdvanced task management with Celery
Advanced task management with CeleryMahendra M
 
Background Jobs with Resque
Background Jobs with ResqueBackground Jobs with Resque
Background Jobs with Resquehomanj
 
Background processing with Resque
Background processing with ResqueBackground processing with Resque
Background processing with ResqueNicolas Blanco
 
Gruntwork Executive Summary
Gruntwork Executive SummaryGruntwork Executive Summary
Gruntwork Executive SummaryYevgeniy Brikman
 
GPerf Using Jesque
GPerf Using JesqueGPerf Using Jesque
GPerf Using Jesquectoestreich
 
Python tools for testing web services over HTTP
Python tools for testing web services over HTTPPython tools for testing web services over HTTP
Python tools for testing web services over HTTPMykhailo Kolesnyk
 
Ansible module development 101
Ansible module development 101Ansible module development 101
Ansible module development 101yfauser
 
Puppet in the Pipeline
Puppet in the PipelinePuppet in the Pipeline
Puppet in the PipelinePuppet
 
Getting Started with Ansible
Getting Started with AnsibleGetting Started with Ansible
Getting Started with AnsibleAhmed AbouZaid
 
Infrastructure Automation with Chef & Ansible
Infrastructure Automation with Chef & AnsibleInfrastructure Automation with Chef & Ansible
Infrastructure Automation with Chef & Ansiblewajrcs
 
Rhebok, High Performance Rack Handler / Rubykaigi 2015
Rhebok, High Performance Rack Handler / Rubykaigi 2015Rhebok, High Performance Rack Handler / Rubykaigi 2015
Rhebok, High Performance Rack Handler / Rubykaigi 2015Masahiro Nagano
 
What Is Async, How Does It Work, And When Should I Use It?
What Is Async, How Does It Work, And When Should I Use It?What Is Async, How Does It Work, And When Should I Use It?
What Is Async, How Does It Work, And When Should I Use It?emptysquare
 
Javascript TDD with Jasmine, Karma, and Gulp
Javascript TDD with Jasmine, Karma, and GulpJavascript TDD with Jasmine, Karma, and Gulp
Javascript TDD with Jasmine, Karma, and GulpAll Things Open
 

Tendances (20)

Celery introduction
Celery introductionCelery introduction
Celery introduction
 
Celery
CeleryCelery
Celery
 
Controlling multiple VMs with the power of Python
Controlling multiple VMs with the power of PythonControlling multiple VMs with the power of Python
Controlling multiple VMs with the power of Python
 
Introduction to Ansible (Pycon7 2016)
Introduction to Ansible (Pycon7 2016)Introduction to Ansible (Pycon7 2016)
Introduction to Ansible (Pycon7 2016)
 
Advanced task management with Celery
Advanced task management with CeleryAdvanced task management with Celery
Advanced task management with Celery
 
Php resque
Php resquePhp resque
Php resque
 
Background Jobs with Resque
Background Jobs with ResqueBackground Jobs with Resque
Background Jobs with Resque
 
Background processing with Resque
Background processing with ResqueBackground processing with Resque
Background processing with Resque
 
Introducing Ansible
Introducing AnsibleIntroducing Ansible
Introducing Ansible
 
Gruntwork Executive Summary
Gruntwork Executive SummaryGruntwork Executive Summary
Gruntwork Executive Summary
 
GPerf Using Jesque
GPerf Using JesqueGPerf Using Jesque
GPerf Using Jesque
 
Python tools for testing web services over HTTP
Python tools for testing web services over HTTPPython tools for testing web services over HTTP
Python tools for testing web services over HTTP
 
Ansible module development 101
Ansible module development 101Ansible module development 101
Ansible module development 101
 
Ansible Crash Course
Ansible Crash CourseAnsible Crash Course
Ansible Crash Course
 
Puppet in the Pipeline
Puppet in the PipelinePuppet in the Pipeline
Puppet in the Pipeline
 
Getting Started with Ansible
Getting Started with AnsibleGetting Started with Ansible
Getting Started with Ansible
 
Infrastructure Automation with Chef & Ansible
Infrastructure Automation with Chef & AnsibleInfrastructure Automation with Chef & Ansible
Infrastructure Automation with Chef & Ansible
 
Rhebok, High Performance Rack Handler / Rubykaigi 2015
Rhebok, High Performance Rack Handler / Rubykaigi 2015Rhebok, High Performance Rack Handler / Rubykaigi 2015
Rhebok, High Performance Rack Handler / Rubykaigi 2015
 
What Is Async, How Does It Work, And When Should I Use It?
What Is Async, How Does It Work, And When Should I Use It?What Is Async, How Does It Work, And When Should I Use It?
What Is Async, How Does It Work, And When Should I Use It?
 
Javascript TDD with Jasmine, Karma, and Gulp
Javascript TDD with Jasmine, Karma, and GulpJavascript TDD with Jasmine, Karma, and Gulp
Javascript TDD with Jasmine, Karma, and Gulp
 

En vedette

Celery - A Distributed Task Queue
Celery - A Distributed Task QueueCelery - A Distributed Task Queue
Celery - A Distributed Task QueueDuy Do
 
Microservices in the Enterprise: A Research Study and Reference Architecture
Microservices in the Enterprise: A Research Study and Reference ArchitectureMicroservices in the Enterprise: A Research Study and Reference Architecture
Microservices in the Enterprise: A Research Study and Reference ArchitectureJesus Rodriguez
 
도커(Docker) 메트릭스 & 로그 수집
도커(Docker) 메트릭스 & 로그 수집도커(Docker) 메트릭스 & 로그 수집
도커(Docker) 메트릭스 & 로그 수집Daegwon Kim
 
An Introduction to Celery
An Introduction to CeleryAn Introduction to Celery
An Introduction to CeleryIdan Gazit
 
Scaling Symfony2 apps with RabbitMQ - Symfony UK Meetup
Scaling Symfony2 apps with RabbitMQ - Symfony UK MeetupScaling Symfony2 apps with RabbitMQ - Symfony UK Meetup
Scaling Symfony2 apps with RabbitMQ - Symfony UK MeetupKacper Gunia
 
Apache Spark - Intro to Large-scale recommendations with Apache Spark and Python
Apache Spark - Intro to Large-scale recommendations with Apache Spark and PythonApache Spark - Intro to Large-scale recommendations with Apache Spark and Python
Apache Spark - Intro to Large-scale recommendations with Apache Spark and PythonChristian Perone
 
Word Embeddings - Introduction
Word Embeddings - IntroductionWord Embeddings - Introduction
Word Embeddings - IntroductionChristian Perone
 
New Ceph capabilities and Reference Architectures
New Ceph capabilities and Reference ArchitecturesNew Ceph capabilities and Reference Architectures
New Ceph capabilities and Reference ArchitecturesKamesh Pemmaraju
 
TeorÍa Del Consumo
TeorÍa Del ConsumoTeorÍa Del Consumo
TeorÍa Del Consumoalbarrason
 
왜 레진코믹스는 구글앱엔진을 선택했나
왜 레진코믹스는 구글앱엔진을 선택했나왜 레진코믹스는 구글앱엔진을 선택했나
왜 레진코믹스는 구글앱엔진을 선택했나소리 강
 
RestMQ - HTTP/Redis based Message Queue
RestMQ - HTTP/Redis based Message QueueRestMQ - HTTP/Redis based Message Queue
RestMQ - HTTP/Redis based Message QueueGleicon Moraes
 
Deep Learning - Convolutional Neural Networks - Architectural Zoo
Deep Learning - Convolutional Neural Networks - Architectural ZooDeep Learning - Convolutional Neural Networks - Architectural Zoo
Deep Learning - Convolutional Neural Networks - Architectural ZooChristian Perone
 
REST vs. Messaging For Microservices
REST vs. Messaging For MicroservicesREST vs. Messaging For Microservices
REST vs. Messaging For MicroservicesEberhard Wolff
 
20170227 파이썬으로 챗봇_만들기
20170227 파이썬으로 챗봇_만들기20170227 파이썬으로 챗봇_만들기
20170227 파이썬으로 챗봇_만들기Kim Sungdong
 
Developing applications with a microservice architecture (SVforum, microservi...
Developing applications with a microservice architecture (SVforum, microservi...Developing applications with a microservice architecture (SVforum, microservi...
Developing applications with a microservice architecture (SVforum, microservi...Chris Richardson
 
The Future of Everything
The Future of EverythingThe Future of Everything
The Future of EverythingMichael Ducy
 

En vedette (16)

Celery - A Distributed Task Queue
Celery - A Distributed Task QueueCelery - A Distributed Task Queue
Celery - A Distributed Task Queue
 
Microservices in the Enterprise: A Research Study and Reference Architecture
Microservices in the Enterprise: A Research Study and Reference ArchitectureMicroservices in the Enterprise: A Research Study and Reference Architecture
Microservices in the Enterprise: A Research Study and Reference Architecture
 
도커(Docker) 메트릭스 & 로그 수집
도커(Docker) 메트릭스 & 로그 수집도커(Docker) 메트릭스 & 로그 수집
도커(Docker) 메트릭스 & 로그 수집
 
An Introduction to Celery
An Introduction to CeleryAn Introduction to Celery
An Introduction to Celery
 
Scaling Symfony2 apps with RabbitMQ - Symfony UK Meetup
Scaling Symfony2 apps with RabbitMQ - Symfony UK MeetupScaling Symfony2 apps with RabbitMQ - Symfony UK Meetup
Scaling Symfony2 apps with RabbitMQ - Symfony UK Meetup
 
Apache Spark - Intro to Large-scale recommendations with Apache Spark and Python
Apache Spark - Intro to Large-scale recommendations with Apache Spark and PythonApache Spark - Intro to Large-scale recommendations with Apache Spark and Python
Apache Spark - Intro to Large-scale recommendations with Apache Spark and Python
 
Word Embeddings - Introduction
Word Embeddings - IntroductionWord Embeddings - Introduction
Word Embeddings - Introduction
 
New Ceph capabilities and Reference Architectures
New Ceph capabilities and Reference ArchitecturesNew Ceph capabilities and Reference Architectures
New Ceph capabilities and Reference Architectures
 
TeorÍa Del Consumo
TeorÍa Del ConsumoTeorÍa Del Consumo
TeorÍa Del Consumo
 
왜 레진코믹스는 구글앱엔진을 선택했나
왜 레진코믹스는 구글앱엔진을 선택했나왜 레진코믹스는 구글앱엔진을 선택했나
왜 레진코믹스는 구글앱엔진을 선택했나
 
RestMQ - HTTP/Redis based Message Queue
RestMQ - HTTP/Redis based Message QueueRestMQ - HTTP/Redis based Message Queue
RestMQ - HTTP/Redis based Message Queue
 
Deep Learning - Convolutional Neural Networks - Architectural Zoo
Deep Learning - Convolutional Neural Networks - Architectural ZooDeep Learning - Convolutional Neural Networks - Architectural Zoo
Deep Learning - Convolutional Neural Networks - Architectural Zoo
 
REST vs. Messaging For Microservices
REST vs. Messaging For MicroservicesREST vs. Messaging For Microservices
REST vs. Messaging For Microservices
 
20170227 파이썬으로 챗봇_만들기
20170227 파이썬으로 챗봇_만들기20170227 파이썬으로 챗봇_만들기
20170227 파이썬으로 챗봇_만들기
 
Developing applications with a microservice architecture (SVforum, microservi...
Developing applications with a microservice architecture (SVforum, microservi...Developing applications with a microservice architecture (SVforum, microservi...
Developing applications with a microservice architecture (SVforum, microservi...
 
The Future of Everything
The Future of EverythingThe Future of Everything
The Future of Everything
 

Similaire à Celery for internal API in SOA infrastructure

Developing Brilliant and Powerful APIs in Ruby & Python
Developing Brilliant and Powerful APIs in Ruby & PythonDeveloping Brilliant and Powerful APIs in Ruby & Python
Developing Brilliant and Powerful APIs in Ruby & PythonSmartBear
 
Build your APIs with apigility
Build your APIs with apigilityBuild your APIs with apigility
Build your APIs with apigilityChristian Varela
 
DEVNET-1128 Cisco Intercloud Fabric NB Api's for Business & Providers
DEVNET-1128	Cisco Intercloud Fabric NB Api's for Business & ProvidersDEVNET-1128	Cisco Intercloud Fabric NB Api's for Business & Providers
DEVNET-1128 Cisco Intercloud Fabric NB Api's for Business & ProvidersCisco DevNet
 
Part 2 of the REAL Webinars on Oracle Cloud Native Application Development (J...
Part 2 of the REAL Webinars on Oracle Cloud Native Application Development (J...Part 2 of the REAL Webinars on Oracle Cloud Native Application Development (J...
Part 2 of the REAL Webinars on Oracle Cloud Native Application Development (J...Lucas Jellema
 
Using OSGi for script deployment in Apache Sling
Using OSGi for script deployment in Apache SlingUsing OSGi for script deployment in Apache Sling
Using OSGi for script deployment in Apache SlingRadu Cotescu
 
Introducing the All New RESTful API for API Management
Introducing the All New RESTful API for API ManagementIntroducing the All New RESTful API for API Management
Introducing the All New RESTful API for API ManagementWSO2
 
Coding 100-session-slides
Coding 100-session-slidesCoding 100-session-slides
Coding 100-session-slidesCisco DevNet
 
Itb 2021 - Bulding Quick APIs by Gavin Pickin
Itb 2021 - Bulding Quick APIs by Gavin PickinItb 2021 - Bulding Quick APIs by Gavin Pickin
Itb 2021 - Bulding Quick APIs by Gavin PickinGavin Pickin
 
Rest API with Swagger and NodeJS
Rest API with Swagger and NodeJSRest API with Swagger and NodeJS
Rest API with Swagger and NodeJSLuigi Saetta
 
SharePoint Fest Seattle - SharePoint Framework, Angular & Azure Functions
SharePoint Fest Seattle - SharePoint Framework, Angular & Azure FunctionsSharePoint Fest Seattle - SharePoint Framework, Angular & Azure Functions
SharePoint Fest Seattle - SharePoint Framework, Angular & Azure FunctionsSébastien Levert
 
Apigility-Powered APIs on IBM i
Apigility-Powered APIs on IBM iApigility-Powered APIs on IBM i
Apigility-Powered APIs on IBM ichukShirley
 
Hacking Adobe Experience Manager sites
Hacking Adobe Experience Manager sitesHacking Adobe Experience Manager sites
Hacking Adobe Experience Manager sitesMikhail Egorov
 
Доклад Михаила Егорова на PHDays
Доклад Михаила Егорова на PHDaysДоклад Михаила Егорова на PHDays
Доклад Михаила Егорова на PHDaysru_Parallels
 
Resting with OroCRM Webinar
Resting with OroCRM WebinarResting with OroCRM Webinar
Resting with OroCRM WebinarOro Inc.
 
Kонструируем микросервисную архитектуру на Symfony и связываем с Kong
Kонструируем микросервисную архитектуру на Symfony и связываем с KongKонструируем микросервисную архитектуру на Symfony и связываем с Kong
Kонструируем микросервисную архитектуру на Symfony и связываем с KongAlex Demchenko
 
Survival Strategies for API Documentation: Presentation to Southwestern Ontar...
Survival Strategies for API Documentation: Presentation to Southwestern Ontar...Survival Strategies for API Documentation: Presentation to Southwestern Ontar...
Survival Strategies for API Documentation: Presentation to Southwestern Ontar...Tom Johnson
 
Serverless Pune meetup 3
Serverless Pune meetup 3Serverless Pune meetup 3
Serverless Pune meetup 3Vishal Biyani
 

Similaire à Celery for internal API in SOA infrastructure (20)

Developing Brilliant and Powerful APIs in Ruby & Python
Developing Brilliant and Powerful APIs in Ruby & PythonDeveloping Brilliant and Powerful APIs in Ruby & Python
Developing Brilliant and Powerful APIs in Ruby & Python
 
Build your APIs with apigility
Build your APIs with apigilityBuild your APIs with apigility
Build your APIs with apigility
 
DEVNET-1128 Cisco Intercloud Fabric NB Api's for Business & Providers
DEVNET-1128	Cisco Intercloud Fabric NB Api's for Business & ProvidersDEVNET-1128	Cisco Intercloud Fabric NB Api's for Business & Providers
DEVNET-1128 Cisco Intercloud Fabric NB Api's for Business & Providers
 
Part 2 of the REAL Webinars on Oracle Cloud Native Application Development (J...
Part 2 of the REAL Webinars on Oracle Cloud Native Application Development (J...Part 2 of the REAL Webinars on Oracle Cloud Native Application Development (J...
Part 2 of the REAL Webinars on Oracle Cloud Native Application Development (J...
 
Using OSGi for script deployment in Apache Sling
Using OSGi for script deployment in Apache SlingUsing OSGi for script deployment in Apache Sling
Using OSGi for script deployment in Apache Sling
 
Crafting APIs
Crafting APIsCrafting APIs
Crafting APIs
 
Introducing the All New RESTful API for API Management
Introducing the All New RESTful API for API ManagementIntroducing the All New RESTful API for API Management
Introducing the All New RESTful API for API Management
 
Coding 100-session-slides
Coding 100-session-slidesCoding 100-session-slides
Coding 100-session-slides
 
Itb 2021 - Bulding Quick APIs by Gavin Pickin
Itb 2021 - Bulding Quick APIs by Gavin PickinItb 2021 - Bulding Quick APIs by Gavin Pickin
Itb 2021 - Bulding Quick APIs by Gavin Pickin
 
Rest API with Swagger and NodeJS
Rest API with Swagger and NodeJSRest API with Swagger and NodeJS
Rest API with Swagger and NodeJS
 
SharePoint Fest Seattle - SharePoint Framework, Angular & Azure Functions
SharePoint Fest Seattle - SharePoint Framework, Angular & Azure FunctionsSharePoint Fest Seattle - SharePoint Framework, Angular & Azure Functions
SharePoint Fest Seattle - SharePoint Framework, Angular & Azure Functions
 
Apigility-Powered APIs on IBM i
Apigility-Powered APIs on IBM iApigility-Powered APIs on IBM i
Apigility-Powered APIs on IBM i
 
Hacking Adobe Experience Manager sites
Hacking Adobe Experience Manager sitesHacking Adobe Experience Manager sites
Hacking Adobe Experience Manager sites
 
Доклад Михаила Егорова на PHDays
Доклад Михаила Егорова на PHDaysДоклад Михаила Егорова на PHDays
Доклад Михаила Егорова на PHDays
 
Resting with OroCRM Webinar
Resting with OroCRM WebinarResting with OroCRM Webinar
Resting with OroCRM Webinar
 
API SECURITY
API SECURITYAPI SECURITY
API SECURITY
 
Kонструируем микросервисную архитектуру на Symfony и связываем с Kong
Kонструируем микросервисную архитектуру на Symfony и связываем с KongKонструируем микросервисную архитектуру на Symfony и связываем с Kong
Kонструируем микросервисную архитектуру на Symfony и связываем с Kong
 
Survival Strategies for API Documentation: Presentation to Southwestern Ontar...
Survival Strategies for API Documentation: Presentation to Southwestern Ontar...Survival Strategies for API Documentation: Presentation to Southwestern Ontar...
Survival Strategies for API Documentation: Presentation to Southwestern Ontar...
 
Introduction to Flask Micro Framework
Introduction to Flask Micro FrameworkIntroduction to Flask Micro Framework
Introduction to Flask Micro Framework
 
Serverless Pune meetup 3
Serverless Pune meetup 3Serverless Pune meetup 3
Serverless Pune meetup 3
 

Celery for internal API in SOA infrastructure

  • 1. Celery for internal API in SOA infrastructure Roman Imankulov PyCon Russia, Feb 2013
  • 2. 1. Internal API. What's the deal SOA service architecture example
  • 3. 1. Internal API. What's the deal ● DCOM: too Microsoftish ● CORBA: too Enterprisy ● SOAP: XML inside ● RESTful: too sloppy Celery to the rescue?
  • 4. Agenda 1. Internal API. What's the deal 2. How to use Celery for internal API 3. Organizing Celery-based API. Routing 4. Celery-based API. Benefits, pitfalls and security notes
  • 5. 2. How to use Celery for internal API Celery Worker
  • 6. 2. How to use Celery for internal API Classic celery client imports task function
  • 7. 2. How to use Celery for internal API Classic celery client imports makes you: ● install everything ○ slows down installation ● import everything ○ python processes grow in size ○ circular dependencies ○ tight coupling ● share everything ○ ... with outsource developers
  • 8. 2. How to use Celery for internal API "Task implementation is not to be exposed without necessity" William of Ockham
  • 9. 2. How to use Celery for internal API Low-level interface defines only task name
  • 10. 2. How to use Celery for internal API Convenient API requires some boilerplate code
  • 11. 2. How to use Celery for internal API Make it even more convenient with introspection
  • 12. 2. How to use Celery for internal API Introspection magic with celery-api http://github.com/imankulov/celery-api
  • 14. 3. Celery-based API. Routing Creating API endpoint 1. Common package for all API endpoints 2. Every worker imports celery instance 3. Expose tasks for worker with CELERY_IMPORTS 4. Launch worker with celery worker -A module_name -Q foo -n foo
  • 15. 3. Celery-based API. Routing Send tasks with add.apply_async(queue='foo', args=(1, 1))
  • 16. 3. Celery-based API. Routing Define routing rules
  • 17. 3. Celery-based API. Routing Task in the wire { "body": <base64 encoded string with task name and args>, "properties": { "body_encoding": "base64", "delivery_info": { "priority": 0, "routing_key": "foo", "exchange": "foo" }, "delivery_mode": 2, "delivery_tag": <UUID> }, ...}
  • 18. 3. Celery-based API. Routing Task in the wire "delivery_info": { "priority": 0, "routing_key": "foo", "exchange": "foo" }
  • 19. 3. Celery-based API. Routing AMQP for dummies
  • 20. 3. Celery-based API. Routing Define all queues (API endpoints) you have
  • 21. 4. Celery-based API. Benefits High level protocol. Takes the responsibility for data serialization routing exception handling
  • 22. 4. Celery-based API. Benefits Support for wide range of brokers. You can switch between brokers with no major changes in your code Brokers: RabbitMQ Redis SQLAlchemy Django MongoDB Amazon SQS CouchDB Beanstalk
  • 23. 4. Celery-based API. Benefits Built-in extra stuff which you were afraid to ever dream of. asynchronous execution parallel execution asynchronous parallel execution delayed execution throttling automatic retrying of failed tasks limiting the time of execution autoscaling API inspection public key cryptography (message signing)
  • 24. 4. Celery-based API. Pitfalls Python-centric. Celery is written in Python and meant to be used with Python code exclusively
  • 25. 4. Celery-based API. Pitfalls Increasing complexity. Celery codebase 46k LOC Kombu codebase 19k LOC Compare Flask codebase 10k LOC Werkzeug codebase 28k LOC Jinja2 codebase 16k LOC
  • 26. 4. Celery-based API. Security note Never ever expose Celery broker to the Web!
  • 27. 4. Celery-based API. Security note 1. Try >>> redis_url = 'redis://:pycon2013@broker.imankulov.name' >>> celery = Celery(broker=redis_url, result=redis_url) 2. Get the contents of /etc/pycon_secret 3. Exchange it for a secret prize ;)