A 2-hour training session delivered at AtlasCamp in Prague, June 9th 2015. * Docker vs virtual machines * Docker concepts * Docker for testing * Automation with Docker Compose * Continuous integration with Bamboo Docker support * Extracting test results from Docker containers * Continuous deployment with deployment environments

1. Continuous Integration with
Docker and Bamboo
STEVE SMITH • DEVOPS ADVOCATE • ATLASSIAN • @TARKASTEVE
https://bitbucket.org/ssmith/atlascamp2015-docker-ci

2. © http://www.amigosdosbichos.org/

3. Warning:
Bad analogies coming up…

4. VMs vs Containers
Huffington Post

5. Virtual Machine
Virtual HW
BIOS/EFI
Guest FS / Kernel
Applications
Physical Hardware
Host FS / Kernel
VM Hypervisor
BIOS/EFI
Virtual HW
BIOS/EFI
Guest FS / Kernel
Applications

6. Containers / Docker
Physical Hardware
Host FS / Kernel
Docker
Engine
BIOS/EFI
Guest FS
Application
Guest FS
Application Virtual HW
BIOS/EFI
Guest FS / Kernel
Applications
Physical Hardware
Host FS / Kernel
VM Hypervisor
BIOS/EFI
Virtual HW
BIOS/EFI
Guest FS / Kernel
Applications

7. Using features of the Linux kernel
Docker
libcontainer
systemd
nspawn
libvirt
cgroups
CPU accounting
Namespaces
Posix capabilities
btrfs
OverlayFS
LXC

8. Unix: chroot

9. Containers take this further…
Network
Abstraction
Process
Isolation
Filesystem
Isolation
Resource
Limits

10. Docker takes this to the next level…
Build Tools Packaging
Lifecycle Communication Dev (boot2docker)
Sharing

11. boot2docker
Virtual Hardware
Host FS / Kernel
BIOS/EFI
Guest FS
Application
Guest FS
Application

12. Quick Demo
Peter Kastner

13. Docker Concepts
Carlsberg

14. Docker containers are like OO classes…
Inheritance
Sourcefile
(Dockerfile) Compile
Dep. Injection Declarative Deps

15. FROM ubuntu:trusty
RUN apt-get update &&
apt-get install -y nginx &&
apt-get clean
EXPOSE 80
EXPOSE 443
CMD nginx -g daemon off;
Single inheritance
Add member
Override parent behaviour
Exposed linking points
Dockerfile ~= Sourcefile

16. Inheritance resolved at runtime
Ubuntu
Add nginx
Add run command
(sorta like JVM class assembly)
Union Filesystem
- AUFS
- BTRFS
- Devicemapper
- OverlayFS

17. ‘run’ instantiates a new container
Ubuntu
Add nginx
Add run command
Writable
Immutable
(kinda like ‘new’)
Runtime layer

18. [ssmith] $ docker build -q --tag=mynginx docker/mynginx/
Sending build context to Docker daemon 12.72 MB
Sending build context to Docker daemon
Step 0 : FROM ubuntu:trusty
---> 8eaa4ff06b53
Step 1 : RUN apt-get update && apt-get install -y nginx && apt-get clea
---> Running in f05a4a034fe3
---> 1e84ac75dd9a
Removing intermediate container f05a4a034fe3
Step 2 : EXPOSE 80
---> Running in 77602b8eb9ec
---> 4076f56233ff
Removing intermediate container 77602b8eb9ec
Step 3 : EXPOSE 443
---> Running in b627b8c1d328
---> f5a3ff140964
Removing intermediate container b627b8c1d328
Step 4 : CMD nginx -g daemon off;
---> Running in 132573fe4fc9
---> c1df0408cf70
‘build’ compiles to binary

19. Inspecting the stack
[ssmith] $ docker history mynginx
IMAGE CREATED CREATED BY
606f0f611cb2 6 days ago /bin/sh -c #(nop) CMD [nginx -g daemon off;]
27d0fad8620d 6 days ago /bin/sh -c #(nop) EXPOSE map[443/tcp:{}]
dc26b50e9806 6 days ago /bin/sh -c #(nop) EXPOSE map[80/tcp:{}]
7a04b014a0a7 6 days ago /bin/sh -c apt-get update && apt-get install
b39b81afc8ca 9 days ago /bin/sh -c #(nop) CMD [/bin/bash]
615c102e2290 9 days ago /bin/sh -c sed -i 's/^#s*(deb.*universe)$/
837339b91538 9 days ago /bin/sh -c echo '#!/bin/sh' > /usr/sbin/polic
53f858aaaf03 9 days ago /bin/sh -c #(nop) ADD file:ca5b63647c6b7a419e
511136ea3c5a 19 months ago

20. Linked Containers
(dependency injection; sort-of…)
Docker host
postgresql client
--link postgresql:pg
can refer to
hostname pg
in commands

21. Volumes
Docker host
/var/volume1
DATA
/var/volume2
/var/volume1
client1
/var/volume2
--volumes-from DATA
(haven’t got a class analogy for this one)

22. Discussion

23. Testing with Docker

24. The example project
?
https://bitbucket.org/ssmith/atlascamp2015-docker-ci

25. The example project
Transfer
Trigger / Async
Data
Data
(See http://bit.do/postgres-es for details)

26. How to test…
Vagrant?
Dependencies
everywhere?
Docker?

27. Docker testing
Reuse images Startup speed Idempotent tests
Dev / Test match Deploy to Docker

28. Creating our images

29. Elasticsearch Dockerfile
FROM java:openjdk-7-jre
ENV ES_VER 1.3.4
ENV ES_URL https://download.elasticsearch.org/elasticsearch/elasticsearch/
elasticsearch-${ES_VER}.tar.gz
WORKDIR /opt/
RUN adduser --system elasticsearch
RUN curl ${ES_URL} -o elasticsearch.tgz &&
tar xzf elasticsearch.tgz &&
ln -s elasticsearch-${ES_VER} elasticsearch &&
mkdir /opt/elasticsearch/data/ && chown elasticsearch /opt/elasticsearch/data/ &&
mkdir /opt/elasticsearch/logs/ && chown elasticsearch /opt/elasticsearch/logs/ &&
rm elasticsearch.tgz
EXPOSE 9200
EXPOSE 9300
USER elasticsearch
CMD /opt/elasticsearch/bin/elasticsearch -f

30. Elasticsearch build
[ssmith] docker build -q --tag=elasticsearch docker/elasticsearch/
Sending build context to Docker daemon 2.56 kB
Sending build context to Docker daemon
Step 0 : FROM java:openjdk-7-jre
---> b797b3ba124d
Step 1 : ENV ES_VER 1.3.4
---> Running in 4842001c48d0
---> 2717f23208b7
Removing intermediate container 4842001c48d0
Step 2 : ENV ES_URL https://download.elasticsearch.org/elasticsearch/elasticsearch/
elasticsearch-${ES_VER}.tar.gz
---> Running in b2f574153a18
---> a292b608b854
Removing intermediate container b2f574153a18
Step 3 : WORKDIR /opt/
---> Running in 0f138382e071
---> aa2ebf2e8061
Removing intermediate container 0f138382e071
Step 4 : RUN adduser --system elasticsearch
---> Running in 0800878ae8c0
---> 2b0c2619f328
Removing intermediate container 0800878ae8c0
Step 5 : RUN curl ${ES_URL} -o elasticsearch.tgz && tar xzf elasticsearch.tgz &&
ln -s elasticsearch-${ES_VER} elasticsearch && mkdir /opt/elasticsearch/data/ &&
chown elasticsearch /opt/elasticsearch/data/ && mkdir /opt/elasticsearch/logs/ &&

31. Postgresql Dockerfile
FROM postgres:9.2
ENV POSTGRES_USER customer
COPY testschema.sql /
COPY init-test-db.sh /docker-entrypoint-initdb.d/

32. Postgresql init file
#!/bin/bash
echo "#### Create test DB ####"
gosu postgres postgres --single <<EOF
CREATE USER customer PASSWORD 'customer';
CREATE DATABASE customer OWNER customer;
GRANT ALL PRIVILEGES ON DATABASE customer TO customer;
EOF
echo "#### Initialise test DB ####"
gosu postgres postgres --single customer -j < /testschema.sql
echo
echo "#### Test DB initialised ####"

33. Postgresql build
[ssmith] docker build -q --tag=postgres docker/postgres/
Sending build context to Docker daemon 5.12 kB
Sending build context to Docker daemon
Step 0 : FROM postgres:9.2
---> 898ba6de4072
Step 1 : ENV POSTGRES_USER customer
---> Running in 932957e34d5e
---> 91b679d67fd9
Removing intermediate container 932957e34d5e
Step 2 : COPY testschema.sql /
---> 65b67419fe54
Removing intermediate container 54298baaa088
Step 3 : COPY init-test-db.sh /docker-entrypoint-initdb.d/
---> b6d84236c1f7
Removing intermediate container f9000db9da92
Successfully built b6d84236c1f7

34. Transfer Dockerfile
FROM java:openjdk-7-jre
# Install netcat for waitport
RUN apt-get update &&
apt-get install -y netcat-openbsd &&
apt-get clean
COPY docker/waitport /usr/local/bin/
RUN curl https://raw.githubusercontent.com/technomancy/leiningen/stable/bin/lein -o /
usr/local/bin/lein &&
chmod a+rx /usr/local/bin/lein
RUN adduser --disabled-password --gecos '' transfer
ADD . /code
RUN chown -R transfer.transfer /code
USER transfer
WORKDIR /code
RUN lein deps
CMD waitport elasticsearch 9200 &&
waitport postgres 5432 &&
lein test-out junit

35. Running Tests Manually

36. Manual run
[ssmith] docker run -d —name=postgres postgres
181470ed78836c9285664c18f0d5b4c6a9069d28c4f71b9d621d4c24762ad938
[ssmith] docker run -d --name=elasticsearch elasticsearch
5718e1ba9b7caf0e25f53d803a128d2a3d4d518ae8d747b843cc2cbdd78620ce
[smith] docker run --link postgres:postgres --link elasticsearch:elasticsearch transfer
Waiting for TCP connection to elasticsearch:9200...OK
Waiting for TCP connection to postgres:5432...OK
Received notification for accountupdate for 55d3290b-5baa-4ca6-b353-cacf5b104c60
Performing update of 55d3290b-5baa-4ca6-b353-cacf5b104c60
Indexing row 55d3290b-5baa-4ca6-b353-cacf5b104c60 Mr. 55d3290b 55d3290b Esq.
Dispatch complete

37. Manual cleanup
[ssmith] docker ps
CONTAINER ID IMAGE COMMAND CREATED …
5718e1ba9b7c elasticsearch:latest "/bin/sh -c '/opt/el 5 minutes ago …
181470ed7883 postgres:latest "/docker-entrypoint. 6 minutes ago …
[ssmith] docker kill elasticsearch postgres
elasticsearch
postgres
[ssmith] docker ps
CONTAINER ID IMAGE COMMAND CREATED …
[ssmith] docker ps -a
CONTAINER ID IMAGE COMMAND CREATED
ff6e5409d8ba transfer:latest "/bin/sh -c 'waitpor 11 minutes ago
5718e1ba9b7c elasticsearch:latest "/bin/sh -c '/opt/el 11 minutes ago
181470ed7883 postgres:latest "/docker-entrypoint. 11 minutes ago
[smith] docker rm transfer postgres elasticsearch
transfer
postgres
elasticsearch

38. Simplifying with Compose
feedyourskull.com

39. Compose automates Docker

40. Compose
Declarative Deps Fetch Images Build Images
Link Containers

41. Our docker-compose.yml
transfer:
build: .
links:
- postgres
- elasticsearch
postgres:
build: docker/postgres
elasticsearch:
build: docker/elasticsearch

42. Compose Run
[ssmith] docker-compose up
Creating devweek15code_postgres_1...
Creating devweek15code_elasticsearch_1...
Creating devweek15code_transfer_1...
Attaching to devweek15code_postgres_1, devweek15code_elasticsearch_1, devweek15code_transfer_1
postgres_1 | The files belonging to this database system will be owned by user "postgres".
postgres_1 | This user must also own the server process.
<SNIP> <SNIP> <SNIP> <SNIP> <SNIP>
transfer_1 | Waiting for TCP connection to postgres:5432...OK
transfer_1 | Received notification for accountupdate for dcf286bf-7011-40f5-abb0-153d94acde
transfer_1 | Performing update of dcf286bf-7011-40f5-abb0-153d94acde69
transfer_1 | Indexing row dcf286bf-7011-40f5-abb0-153d94acde69 Mr. dcf286bf dcf286bf Esq.
elasticsearch_1 | creating index, cause [auto(index api)], shards [5]/[1], mappings []
elasticsearch_1 | update_mapping [account] (dynamic)
transfer_1 | Dispatch complete
devweek15code_transfer_1 exited with code 0
Gracefully stopping... (press Ctrl+C again to force)
Stopping devweek15code_elasticsearch_1...
Stopping devweek15code_postgres_1...

43. Compose Cleanup
[ssmith] docker-compose ps
Name Command State Ports
--------------------------------------------------------------------------------
devweek15code_elasticsearch_1 /bin/sh -c /opt/elasticsea ... Exit -1
devweek15code_postgres_1 /docker-entrypoint.sh postgres Exit 0
devweek15code_transfer_1 /bin/sh -c waitport elasti ... Exit 0
[ssmith] docker-compose rm —force
Going to remove devweek15code_transfer_1, devweek15code_elasticsearch_1, devweek15code_postgres_1
Removing devweek15code_postgres_1...
Removing devweek15code_elasticsearch_1...
Removing devweek15code_transfer_1…
[ssmith] docker-compose ps
Name Command State Ports
--------------------------------------------------------------------------------
[ssmith]

44. Discussion
Rafael Neff

45. Running With Bamboo
Plan
Stage
Job
Tasks
QA
Staging
Prod
Artifacts
Deployment
Environments
Git Repo

46. Bamboo Live
https://atlascamp2015.atlassian.net/

47. All New! Bamboo Docker Task!

48. AWS cloud agents with Docker
With Docker
Or create
your own

49. Require Docker support
Bamboo will start
on-demand

50. Building

51. Running

52. Linking

53. Compose with BambooSouth Florida
Classical Review

54. No Compose? No problem…

55. Running Compose

56. Cleaning up

57. Extracting ResultsAtlantic Sentinel

58. Fetching container files
[ssmith] docker-compose ps
Name Command State Ports
--------------------------------------------------------------------------------
devweek15code_elasticsearch_1 /bin/sh -c /opt/elasticsea ... Exit -1
devweek15code_postgres_1 /docker-entrypoint.sh postgres Exit 0
devweek15code_transfer_1 /bin/sh -c waitport elasti ... Exit 0
[ssmith] docker-compose ps -q transfer
fd7e728e3e9a361d96c253f5aeadab1a3506538d1b0e19d27c82848d9bf48bf8
[ssmith] ID=`docker-compose ps -q transfer`
[ssmith] docker cp $ID:/code/testreports.xml .
[ssmith] file testreports.xml
testreports.xml: XML document text

59. Fetching in Bamboo

60. Using JUnit Parser

61. Using JUnit results

62. Using JUnit results

63. Deployment Environments
Plan
Stage
Job
Tasks
QA
Staging
Prod
Artifacts
Deployment
Environments
Git Repo

64. Deployment Environments
QA
Staging
Prod
Artifacts
Deployment
Environments

65. Sharing Artifacts

66. Configuring the Environment

67. Configuring the Environment

68. Configuring the Environment

69. 2
3
Useful Links
https://www.docker.com/subscribe_newsletter/
1 https://www.docker.com/
4 http://www.nkode.io/2014/08/24/valuable-docker-links.html
http://boot2docker.io/

70. Thank you!
STEVE SMITH • DEVOPS ADVOCATE • ATLASSIAN • @TARKASTEVE