1. Celery for internal API in
SOA infrastructure
Roman Imankulov
PyCon Russia, Feb 2013
2. 1. Internal API. What's the deal
SOA service architecture example
3. 1. Internal API. What's the deal
● DCOM: too Microsoftish
● CORBA: too Enterprisy
● SOAP: XML inside
● RESTful: too sloppy
Celery to the rescue?
4. Agenda
1. Internal API. What's the deal
2. How to use Celery for internal API
3. Organizing Celery-based API. Routing
4. Celery-based API. Benefits, pitfalls and
security notes
5. 2. How to use Celery for internal API
Celery Worker
6. 2. How to use Celery for internal API
Classic celery client imports task function
7. 2. How to use Celery for internal API
Classic celery client imports makes you:
● install everything
○ slows down installation
● import everything
○ python processes grow in size
○ circular dependencies
○ tight coupling
● share everything
○ ... with outsource developers
8. 2. How to use Celery for internal API
"Task implementation is not to
be exposed without necessity"
William of Ockham
9. 2. How to use Celery for internal API
Low-level interface defines only task name
10. 2. How to use Celery for internal API
Convenient API requires some boilerplate code
11. 2. How to use Celery for internal API
Make it even more convenient with
introspection
12. 2. How to use Celery for internal API
Introspection magic with celery-api
http://github.com/imankulov/celery-api
14. 3. Celery-based API. Routing
Creating API endpoint
1. Common package for all API endpoints
2. Every worker imports celery instance
3. Expose tasks for worker with
CELERY_IMPORTS
4. Launch worker with
celery worker -A module_name
-Q foo
-n foo
21. 4. Celery-based API. Benefits
High level protocol.
Takes the responsibility for
data serialization
routing
exception handling
22. 4. Celery-based API. Benefits
Support for wide range of brokers.
You can switch between brokers with no major
changes in your code
Brokers:
RabbitMQ Redis SQLAlchemy
Django MongoDB Amazon SQS
CouchDB Beanstalk
23. 4. Celery-based API. Benefits
Built-in extra stuff which you were afraid to ever
dream of.
asynchronous execution parallel execution
asynchronous parallel execution
delayed execution throttling
automatic retrying of failed tasks
limiting the time of execution
autoscaling API inspection
public key cryptography (message signing)
24. 4. Celery-based API. Pitfalls
Python-centric.
Celery is written in Python and meant to be
used with Python code exclusively
25. 4. Celery-based API. Pitfalls
Increasing complexity.
Celery codebase 46k LOC
Kombu codebase 19k LOC
Compare
Flask codebase 10k LOC
Werkzeug codebase 28k LOC
Jinja2 codebase 16k LOC