Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
Dockerizing Django
PDX Portland

October 27th, 2016
Who are we?
Michael Dougherty

@maackle

Senior Front-end Engineer

CrowdStreet, Inc.
Hannes Hapke

@hanneshapke

Software...
Our pre-Docker World …
• Single instance world

(e.g. celery ran on the web
server)
• Outdated Amazon machine
image
• No d...
Our post-Docker World …
• Single instance world

(e.g. celery ran on the web
server)
• Outdated Amazon machine
image
• No ...
What is Docker?
Docker • Compose • Machine • Swarm
Docker containers …
… wrap a piece of software in a complete filesystem that
contains everything needed to run: code, runti...
Docker vs. Vagrant
Where is the difference?
Images from https://www.docker.com/what-docker
VM includes OS No OS needed
How does Docker
work?
• Create a Dockerfile
• Build the Docker image and push it to the docker registry
Plain Docker
FROM ubuntu:16.04
RUN apt-ge...
Plain Docker
• Run a shell in a docker container



-i start an interactive container

-t creates “Pseudo interface” with ...
What if we need
multiple services?
Docker • Compose • Machine • Swarm
Docker Compose
Compose is a tool for orchestrating the building,
running, and intercommunication of
multi-container Docker...
How does it work?
1) Define a Dockerfile for every service
2) Define a Docker compose description of the environment
3) Use d...
How can I easily
provision a server with
the containers?
Docker • Compose • Machine • Swarm
Docker Machine
… is a great tool which creates Docker hosts anywhere.

Yes, anywhere.

Locally, AWS EC2, Digital Ocean, MS...
What if I need multiple
instances with multiple
services?
Docker • Compose • Machine • Swarm
Docker Swarm
Dockerize for real …
If you start from scratch …
• Docker documentation includes a great Django
setup
• Too much work? The Django Cookie Cutter...
If you convert a
project like us …
Reorganize your folder
structure
Normalize folders
• Create folders for every service
• docker-compose-{env}.yml go into the project root
• Dockerfiles go i...
Reorganized folders
Project Root

|-apps

|-settings

|-static

|-templates

|-manage.py

|-fabfile.py

|-urls.py

 require...
Build a base image
Base Image
• Create one (or more) base Dockerfile(s) with all
common packages
• Service containers can use this base image ...
Base Image
FROM ubuntu:16.04
RUN apt-get update && apt-get upgrade -y
RUN apt-get install -y vim # Install some useful edi...
Base Image
Add image of the Docker registry
Set up Docker compose
for the different
environments
Docker Compose
• For every environment, local, QA, staging,
production, define a docker-compose-{env}.yml file
• The files de...
Docker Composeversion: '2'
volumes:
postgres_data_dev: {}
redisdata: {}
webpack_data: {}
services:
postgres:
image: postgr...
Docker Composedjango:
build:
context: .
dockerfile: ./django/Dockerfile-dev
command: python /crowdstreet-src/manage.py run...
Docker Compose
• Build your service stack with
• Start the container stack with
• Access a single container with
$ docker-...
Set up Docker machine
and deploy to the world
Docker machine is
awesome!
Docker Machine
• With









will provision you an AWS instance
• “Activate” the instance with

• Afterwards, any docker...
Lessons Learned
Or... how to cowboy code
with Docker
• Sometimes you just need to manually change
something
• Docker provides ways to get ...
The Disciplined Way:
The Cowboy Way:
$ docker-compose run django bash
$ docker exec -it {container_id} bash
How does QA work with
Docker?
• No QA bottleneck anymore
• No database gridlock anymore
• Each feature branch gets its own...
How can I access the
manage.py shell/migrate?
• Access the bash of the django container with



• Continue as usual with

...
Help, ipdb doesn’t work
anymore …
• Start the Django container with the service ports
enabled

• If no command is specified...
How to run tests?
• Start the Django container with your test command



$ docker-compose -f … run django manage.py test
CI Testing is convenient
• Setup for Circle CI
machine:
pre:
- curl -sSL https://s3.amazonaws.com/circle-downloads/install...
WTF, the files I copied into
my container are missing??
• If a volume is mounted at the same directory
where you copied oth...
Sharing Docker Machine
credentials
• Docker machine is great, but there is no concept
of sharing credentials
• All credent...
General Troubleshooting
• Confirm that the correct docker-machine environment is active
• Rebuild your container stack
• Re...
So, what does our setup
look like now?
Dev Environment
• You can use the same image as in your production
builds
• All services run at once, all output piped to ...
How does the deployment
work now?
• Create AWS instance with docker-machine
• Activate the docker machine
• Use docker-com...
Summary of technologies
• Learned about Docker
• How to use docker to define images and containers
• Learned about Docker-c...
Summary of benefits
• Explicit, declarative server setup
• Zero down time deployments
• All dev services in one "window" an...
Thank you!
Q&A
PDXPortland - Dockerize Django
PDXPortland - Dockerize Django
PDXPortland - Dockerize Django
Prochain SlideShare
Chargement dans…5
×

PDXPortland - Dockerize Django

684 vues

Publié le

  • Soyez le premier à commenter

PDXPortland - Dockerize Django

  1. 1. Dockerizing Django PDX Portland
 October 27th, 2016
  2. 2. Who are we? Michael Dougherty
 @maackle
 Senior Front-end Engineer
 CrowdStreet, Inc. Hannes Hapke
 @hanneshapke
 Software Engineer
 Talentpair, Inc.
  3. 3. Our pre-Docker World … • Single instance world
 (e.g. celery ran on the web server) • Outdated Amazon machine image • No documentation about the setup, consultancy work • Live data monkey patching • Scaling/Recovery time > 8 hours • Clunky QA setup > bottleneck
  4. 4. Our post-Docker World … • Single instance world
 (e.g. celery ran on the web server) • Outdated Amazon machine image • No documentation about the setup, consultancy work • Live data monkey patching • Scaling/Recovery time > 8 hours • Clunky QA setup > bottleneck • One service per container, redundancy of instances • One common base image shared across all instances • Explicit, declarative server setup • Immutable infrastructure (mostly) • Scaling/Recovery time ~ 20min • As many QA instances as we want
  5. 5. What is Docker? Docker • Compose • Machine • Swarm
  6. 6. Docker containers … … wrap a piece of software in a complete filesystem that contains everything needed to run: code, runtime, system tools, system libraries – anything that can be installed on a server. This guarantees that the software will always run the same, regardless of its environment. * Basically a virtual env for your operating system. * from https://www.docker.com/what-docker
  7. 7. Docker vs. Vagrant
  8. 8. Where is the difference? Images from https://www.docker.com/what-docker VM includes OS No OS needed
  9. 9. How does Docker work?
  10. 10. • Create a Dockerfile • Build the Docker image and push it to the docker registry Plain Docker FROM ubuntu:16.04 RUN apt-get update && apt-get upgrade -y RUN pip install Django COPY requirements.txt . WORKDIR /dev $ docker build -t your_project/your-whale . $ docker images
 $ docker tag {image_hash} your_project/your-whale:latest
 $ docker push
  11. 11. Plain Docker • Run a shell in a docker container
 
 -i start an interactive container
 -t creates “Pseudo interface” with stdin and stdout • Run the Django server in a container
 
 -d run container in detached mode
 -P maps all ports to the host machine $ docker run -i -t ubuntu /bin/bash $ docker run -d -P my-container python manage.py run server
  12. 12. What if we need multiple services? Docker • Compose • Machine • Swarm
  13. 13. Docker Compose Compose is a tool for orchestrating the building, running, and intercommunication of multi-container Docker applications.
  14. 14. How does it work? 1) Define a Dockerfile for every service 2) Define a Docker compose description of the environment 3) Use docker-compose build/up to start all services version: '2' services: web: build: . ports: - "5000:5000" volumes: - .:/code redis: image: redis
  15. 15. How can I easily provision a server with the containers? Docker • Compose • Machine • Swarm
  16. 16. Docker Machine … is a great tool which creates Docker hosts anywhere.
 Yes, anywhere.
 Locally, AWS EC2, Digital Ocean, MS Azure, you name it. No Ansible, Puppet, Chef, fabric, etc. required.
  17. 17. What if I need multiple instances with multiple services? Docker • Compose • Machine • Swarm
  18. 18. Docker Swarm
  19. 19. Dockerize for real …
  20. 20. If you start from scratch … • Docker documentation includes a great Django setup • Too much work? The Django Cookie Cutter template includes a great Docker setup
 Other projects: • django-docker on github
  21. 21. If you convert a project like us …
  22. 22. Reorganize your folder structure
  23. 23. Normalize folders • Create folders for every service • docker-compose-{env}.yml go into the project root • Dockerfiles go into every service folder • startup.sh scripts go into the service folders • Keep your local folder structure similar to the folder structure within the container(s) - for sanity
  24. 24. Reorganized folders Project Root
 |-apps
 |-settings
 |-static
 |-templates
 |-manage.py
 |-fabfile.py
 |-urls.py
 requirements.txt Project Root
 |-django
 | |-apps
 | |- …
 | |-Dockerfile
 | |-startup-django.sh
 | -manage.py
 |-nginx
 |-webpack
 |-docker-compose.yml
 |-urls.py
 requirements.txt
  25. 25. Build a base image
  26. 26. Base Image • Create one (or more) base Dockerfile(s) with all common packages • Service containers can use this base image - this will increase build speed • If you store the base image(s) in a separate git repo, the docker registry will build them automatically for you
  27. 27. Base Image FROM ubuntu:16.04 RUN apt-get update && apt-get upgrade -y RUN apt-get install -y vim # Install some useful editor RUN apt-get install -y build-essential git software-properties-common RUN apt-get install -y python python-dev python-setuptools build-essential RUN apt-get install -y nodejs npm RUN npm install -g n # upgrading the npm version RUN n stable ...
  28. 28. Base Image Add image of the Docker registry
  29. 29. Set up Docker compose for the different environments
  30. 30. Docker Compose • For every environment, local, QA, staging, production, define a docker-compose-{env}.yml file • The files describe the environment stack • Each service within the docker-compose file can have it’s own Dockerfile
  31. 31. Docker Composeversion: '2' volumes: postgres_data_dev: {} redisdata: {} webpack_data: {} services: postgres: image: postgres:9.5 volumes: - postgres_data_dev:/var/lib/postgresql/data restart: always environment: - POSTGRES_USER=postgres_user - POSTGRES_DB=my_fancy_db - POSTGRES_PASSWORD= webpack: image: crowdstreet/crowdstreet-whale:latest command: npm run watch environment: - NODE_PATH=/node_modules volumes: - ./webpack/frontend-src:/frontend-src - ./django:/crowdstreet-src - webpack_data:/webpack_data/ ports: - "3000:3000" restart: always
  32. 32. Docker Composedjango: build: context: . dockerfile: ./django/Dockerfile-dev command: python /crowdstreet-src/manage.py runserver 0.0.0.0:8000 --settings=settings.dev depends_on: - postgres environment: - ENV=dev - DJANGO_SETTINGS_MODULE=settings volumes: - ./django:/crowdstreet-src - ./webpack/frontend-src:/frontend-src - webpack_data:/webpack_data/ ports: - "8000:8000" - "80:8000" links: - postgres - redis - webpack - memcached redis: restart: always image: redis:latest volumes: - redisdata:/data restart: always
  33. 33. Docker Compose • Build your service stack with • Start the container stack with • Access a single container with $ docker-compose -f docker-compose-{env}.yml build $ docker-compose -f docker-compose-{env}.yml up $ docker-compose -f docker-compose-{env}.yml run django bash $ docker-compose -f docker-compose-{env}.yml 
 run container name command
  34. 34. Set up Docker machine and deploy to the world
  35. 35. Docker machine is awesome!
  36. 36. Docker Machine • With
 
 
 
 
 will provision you an AWS instance • “Activate” the instance with
 • Afterwards, any docker-compose command will be executed on the active machine • Easy to start/stop/terminate machines $ docker-machine create --driver amazonec2 
 --amazonec2-region [e.g. us-west-2] 
 --amazonec2-vpc-id [YOUR_VPC_ID vpc-xxxxxx] 
 --amazonec2-instance-type [e.g. t2.small] [INSTANCE_NAME] $ docker-machine env [INSTANCE_NAME]
  37. 37. Lessons Learned
  38. 38. Or... how to cowboy code with Docker • Sometimes you just need to manually change something • Docker provides ways to get a shell inside a running instance and copy files back and forth • Your changes will of course be lost next time you spin up a new container
  39. 39. The Disciplined Way: The Cowboy Way: $ docker-compose run django bash $ docker exec -it {container_id} bash
  40. 40. How does QA work with Docker? • No QA bottleneck anymore • No database gridlock anymore • Each feature branch gets its own instance • Once feature is tested, instance gets terminated
  41. 41. How can I access the manage.py shell/migrate? • Access the bash of the django container with
 
 • Continue as usual with
 
 
 Some for migrations, make_migrations, etc. • Or run it from outside of the container stack with $ docker-compose -f docker-compose-{env}.yml run django bash # ./manage.py shell docker-compose -f … run django python manage.py migrate
  42. 42. Help, ipdb doesn’t work anymore … • Start the Django container with the service ports enabled
 • If no command is specified, then Docker will default to the command in the docker-compose.yml file $ docker-compose -f dev.yml run --service-ports django
  43. 43. How to run tests? • Start the Django container with your test command
 
 $ docker-compose -f … run django manage.py test
  44. 44. CI Testing is convenient • Setup for Circle CI machine: pre: - curl -sSL https://s3.amazonaws.com/circle-downloads/install-circleci-docker.sh | bash -s -- 1.10.0 services: - docker dependencies: override: - sudo pip install docker-compose - docker login -e $DOCKER_EMAIL -u $DOCKER_USER -p $DOCKER_PASS - docker-compose -f docker-compose-circle.yml build - npm install -g jshint test: pre: - sudo killall postgres # not sure why, but port 5432 is already taken up sometimes! - docker-compose -f docker-compose-circle.yml up -d postgres override: - jshint ~/your_project/django/static/js/your_project* - docker-compose -f docker-compose-circle.yml run django /your_project/manage.py test --verbosity=2
  45. 45. WTF, the files I copied into my container are missing?? • If a volume is mounted at the same directory where you copied other files, you will essentially overwrite those files
  46. 46. Sharing Docker Machine credentials • Docker machine is great, but there is no concept of sharing credentials • All credentials are simple text files, no magic • npm tool `machine-share` solved the problem • Let’s you export and import machine credentials
  47. 47. General Troubleshooting • Confirm that the correct docker-machine environment is active • Rebuild your container stack • Rebuild with the --pull and/or --no-cache options • Restart the docker daemon • Restart your docker machine with docker-machine restart [INSTANCE NAME] • Restart your docker machine VirtualBox VM • Remove and recreate your docker machine (essentially recreates your dev environment from scratch)
  48. 48. So, what does our setup look like now?
  49. 49. Dev Environment • You can use the same image as in your production builds • All services run at once, all output piped to a single log stream (which we saw earlier) • You can still have live reloading via Docker Volumes (but be careful!)
  50. 50. How does the deployment work now? • Create AWS instance with docker-machine • Activate the docker machine • Use docker-compose to build the stack • Use docker-compose up -d • Switch the load balancer
  51. 51. Summary of technologies • Learned about Docker • How to use docker to define images and containers • Learned about Docker-compose to define relationships between containers • Learned about Docker-machine to seamlessly work with containers on local/remote machines
  52. 52. Summary of benefits • Explicit, declarative server setup • Zero down time deployments • All dev services in one "window" and start with one command • Easy provisioning of multiple QA instances • Quick onboarding for new devs
  53. 53. Thank you!
  54. 54. Q&A

×