Testing is important for any system you write and at eBay it is no different. We have a number of complex Scala and Akka based applications with a large number of external dependencies. One of the challenges of testing this kind of application is replicating the complete system across all your environments: development, different flavors of testing (unit, functional, integration, capacity and acceptance) and production. This is especially true in the case of integration and capacity testing where there are a multitude of ways to manage system complexity. Wouldn’t it be nice to define the testing system architecture in one place that we can reuse in all our tests? It turns out we can do exactly that using Docker. In this talk, we will first look at how to take advantage of Docker for integration testing your Scala application. After that we will explore how this has helped us reduce the duration and complexity of our tests.

1. Daniel Brown & Mario Camou
Scala, Docker
and Testing
oh my!
[Road]

2. Who are we?
Daniel Brown
Software engineer at eBay
Tinkerer and
hacker of electronic devices
Resident mad scientist
@_dlpb / https://github.com/dlpb
Mario Camou
Software engineer at eBay
3D printing enthusiast
and Doctor Who fan
“Do what I do. Hold tight and pretend it’s
a plan!”
—The Doctor, Season 7, Christmas
Special
@thedoc / https://github.com/mcamou/

3. Agenda
● Introduction to Docker
● Why would you use Docker in tests?
● How do you integrate Docker into a Scala project?
● Lessons learned

4. Intro to Docker
[Docker]

5. Virtual Machines
● eBay’s default way of working
● Easy to provision (through a UI)
● In the “cloud”
● Choice of OS’s

6. Virtual Machines
● Slow to provision internally
● Both API and UI
● Machines can, and do, disappear
● Internal connection latency

7. Virtual Machines
● Time consuming to set up
● Not possible easily to automate setup
● e.g. databases, web servers,
message queues (even with chef)

8. Enter Docker
Docker • /ˈdɒkər/
Noun
An open platform for developers and sysadmins
to build, ship and run applications
“
[Docker]

9. Enter Docker
Dependencies
● Create images of dependencies
[Docker]

10. Enter Docker
Dependencies
● Create images of dependencies
● Doesn't solve all setup issues
● But only needs to be done once
[Docker]

11. Enter Docker
Dependencies
● Create images of dependencies
● Doesn't solve all setup issues
● But only needs to be done once
● Less storage overhead than VM
● Databases and webservers = GBs of
data
[Docker]

12. Enter Docker
Dependencies
WIN!
[Docker]

13. Welcome to
The
Real World
[Bird]

14. Background: Our Service
[MFS]

15. Background: Our Service
[MFS]

16. Background: Our Goal
Decouple our tests from other services
[MFS]

17. Dependency Nightmares
Untested
Unreliable
Slow
Fire

18. Isolation is Key
● Decompose the monolith
● Identify services we are dependent upon
● Stub the contract
● Run your tests
[MFS]

19. Decompose the Monolith
[Stonehenge]

20. The Monolith
A common way of working
● Everything is “in the library”
● Antipatterns
● Black Magic

21. Identify Services we are Dependent Upon
This can be the tricky step when everything is “in the framework”
● May require the use of debuggers, network sniffers etc
● Investment is worth it in the long run

22. Identify Services we are Dependent Upon
Move configuration of services out to easily controllable (and
versionable) config
● Implement, or update, a client to use the new config
● Decoupling production from tests

23. Stub the Contract
Here you need to know
● The API mappings
● Expected Responses

24. Stub the Contract
Golden Rule: Keep it Simple!
● Have one response per stub
● Keep them versioned
● Put them in containers

25. Run your Tests
Now that we know our dependencies, contracts, and have stubs, we can
write tests
● These are focussed only on testing that our system behaves as
expected when downstream services offer varying responses
● We are NOT testing the downstream services

26. Run your Tests
● Create a number of different stubs in different docker images
● Spin up the ones you need for your specific test
● When you are done with the test, tear them down and start again

27. So How do you go about it?So, how do we go about it?
[Hands]

28. Dockerizing Scala
[Cat]

29. Normal Docker flow
● Create a Dockerfile
● Build the Docker image
● Push it to the registry
● Pull the image from every server

30. Normal Docker flow
● No static checking of the Dockerfile
● Artifact build is separate from image build
○ Do you have the right artifacts?
○ Did you run the tests before building?
○ Are the latest bits in the image?

31. Integrating Docker with sbt
[Construction Kit]

32. Integrating Docker with sbt
sbt-docker
An sbt plugin that:
● Creates a Dockerfile
● Creates an image based on that file
● Pushes the image to a registry
https://github.com/marcuslonnberg/sbt-docker

33. Setting the Image Name
imageNames in docker := Seq(
ImageName(
namespace = Some("myOrg"),
repository = name.value,
tag = Some(s"v${version.value}")
),
ImageName(
namespace = Some("myOrg"),
repository = name.value,
tag = Some("latest")
)
)

34. Some Useful vals
val artifact = (assemblyOutputPath in assembly).value
val baseDir = "/srv"
val preInstall = Seq(
"/usr/bin/apt-get update",
s"/usr/sbin/useradd -r -s /bin/false -d $baseDir myUser"
).mkString(" && ")

35. Configuring the Dockerfile
dockerfile in docker := {
new Dockerfile {
from("java:openjdk-8-jre")
user("myUser")
entryPoint("bin/run.sh")
runRaw(preInstall)
copy(new File("bin/run-docker.sh"), s"$baseDir/run.sh")
copy(new File("local/etc"), "$baseDir/etc")
copy(artifact, s"$baseDir/app.jar")
runRaw("chown -R myUser $baseDir && chmod 0544 $baseDir/run.sh")
}
}

36. Integrating with sbt-assembly
Ensure assembly always runs before Docker
docker <<= (docker dependsOn assembly)

37. Creating and Publishing the Image
sbt> docker
sbt> dockerPush

38. Caveats and Recommendations
● Create a fat JAR: sbt-assembly or sbt-native-packager
● In non-Linux platforms, start up docker-machine and set up the
environment variables before starting sbt:
$ docker-machine start theMachine
$ eval $(docker-machine env theMachine)
https://velvia.github.io/Docker-Scala-Sbt/

39. [Explosion]

40. Integration Testing with Docker
Create Docker image(s) containing stubbed external resources
● Tests always run with clean data
● Resource startup is standardized
● Does not require multiple VMs or network calls

41. Integration Testing with Docker
Before your tests:
● Start up the stubbed resource containers
● Wait for the containers to start
After your tests:
● Stop the containers
Can we automate all of this?

42. Container Orchestration during Tests
Orchestration platforms
● Docker Compose
● Kubernetes
● …

43. Container Orchestration during Tests
Orchestration platforms
● Docker Compose
● Kubernetes
● …
Orchestrate inside the test code

44. Using the Docker Java API
val config = DockerClientConfig.createDefaultConfigBuilder()
.withServerAddress("tcp://192.168.99.100:2376")
.withDockerCertPath("/path/to/certificates")
.build
val docker = DockerClientBuilder.getInstance(config).build
val callback = new PullImageResultCallback
docker.pullImageCmd("mongo:2.6.12").exec(callback)
callback.awaitSuccess
val container = docker.createContainerCmd("mongo:2.6.12")
.withCmd("mongod", "--nojournal", "--smallfiles", "--syncdelay", "0")
.exec
docker.startContainerCmd(container.getId).exec
docker.stopContainerCmd(container.getId).exec
val exitcode = docker.waitContainerCmd(container.getId).exec

45. Scala-native Solutions
reactive-docker and tugboat
● Use the Docker REST API directly -> versioning problems
● Unmaintained for > 1 year (not updated to Docker 1.2 API)
● No TLS support

46. Using reactive-docker
implicit val docker = Docker("192.168.99.100", 2375)
val timeout = 30.seconds
val name = "mongodb-test"
val cmd = Seq("mongod", "--nojournal", "--smallfiles", "--syncdelay",
"0")
val cfg = ContainerConfiguration(Some("mongo:2.6.12"), Some(cmd))
val (containerId, _) = Await.result(
docker.containerCreate("mongo:2.6.5", cfg, Some(name)), timeout)
val started = Await.result(docker.containerStart(containerId), timeout)
val stopped = Await.ready(docker.containerStop(containerId), timeout)
https://github.com/almoehi/reactive-docker

47. Caveats and Recommendations
● Use beforeAll to ensure containers start up before tests
● Use afterAll to ensure containers stop after tests
Caveats:
● Multiple container start/stops can make tests run much slower
● Need to check when resource (not just container) is up
● Single start/stop means testOnly/testQuick will start up all
resources
● Ctrl+C will not stop the stubbed resource containers

48. Introducing docker-it-scala
● Uses the (official) docker-java library
● Starts up resource containers in parallel
○ Only when they are needed
○ Once when the tests start
○ Waits for service (not just container) startup
● Automatically shuts down all started stub containers
● Configured via code or via Typesafe Config
https://github.com/whisklabs/docker-it-scala
[Docker-It-Scala]

49. Defining a Resource Container
● Resources are declared as traits and mixed into tests
● Sample implementations available for Cassandra, ElasticSearch,
Kafka, MongoDB, Neo4j, PostgreSQL, Zookeeper (in the docker-
testkit-samples package)
[Docker-It-Scala]

50. Defining Resource Container (Neo4j)
trait DockerNeo4jService extends DockerKit {
val neo4jContainer = DockerContainer("whisk/neo4j:2.1.8")
.withPorts(7474 -> None)
.withReadyChecker(
DockerReadyChecker.HttpResponseCode(7474, "/db/data/")
.within(100.millis)
.looped(20, 1250.millis)
))
abstract override def dockerContainers: List[DockerContainer] =
neo4jContainer :: super.dockerContainers
}
[Docker-It-Scala]

51. Defining Resource Container (PostgreSQL)
docker {
postgres {
image-name = "postgres:9.4.4"
environmental-variables = ["POSTGRES_USER=nph",
"POSTGRES_PASSWORD=suitup"]
ready-checker {
log-line = "database system is ready to accept connections"
}
port-maps {
Default-postgres-port.internal = 5432
}
}
}
[Docker-It-Scala]

52. Defining Resource Container (PostgreSQL)
trait DockerPostgresService extends DockerKitConfig {
val postgresContainer = configureDockerContainer("docker.postgres")
abstract override def dockerContainers: List[DockerContainer] =
postgresContainer :: super.dockerContainers
}
[Docker-It-Scala]

53. Writing Your Tests
class MyMongoSpec extends FunSpec with DockerMongodbService {
// Test assumes the MongoDB container is running
}
class MyPostgresSpec extends FunSpec with DockerNeo4jService {
// Test assumes the Neo4j container is running
}
class MyAllSpec extends FunSpec with DockerMongodbService
with DockerNeo4jService with DockerPostgresService{
// Test assumes all 3 containers are running
}
https://github.com/whisklabs/docker-it-scala
[Docker-It-Scala]

54. What Did We Achieve?
[Space needle]

55. The Effect
● We cut our end to end testing time
120 minutes -> 10 minutes
[Clock]

56. The Effect
● We cut our end to end testing time
120 minutes -> 10 minutes
● Confidence
[Clock]

57. Going Further
● Performance Measurements
● Business Decisions

58. Performance measurements (from Stubbed Tests)
Create mocks that can simulate (or approximate) real-world conditions
● E.g.
○ Drop every third request
○ Delay 5 seconds before responding
○ Respond instantly for every request

59. Performance measurements (from Stubbed Tests)
Use these new stubs to gather data about how your system performs
● When it is stressed;
● When downstream services are stressed

60. Performance measurements (from Stubbed Tests)
Use these new stubs to gather data about how your system performs
● When it is stressed;
● When downstream services are stressed
Gather the metrics!

61. Performance measurements (from Stubbed Tests)
[MFS]

62. Going Further
What did we decide from the graph?

63. Going Further
What did we decide from the graph?
● Technical limitations for our product

64. Going Further
What did we decide from the graph?
● Technical limitations for our product
● Business policies for the product

65. Summary
Isolation is key to gathering meaningful test data and keeping testing
strategies sane
Docker can ease the pain of managing stub dependencies during
integration testing
Meaningful tests can tell you a lot about the behaviour of your system
and therefore influence both architecture and UX

66. Q & A
[Audience]

67. Acknowledgements & Notes
[Docker] Docker: www.docker.com
Docker and the Docker logo are trademarks or registered trademarks of
Docker, Inc. in the United States and/or other countries. Docker, Inc. and
other parties may also have trademark rights in other terms used herein.
[Fire], Fire image, Creative Commons CC0 License https://www.pexels.
com/photo/fire-orange-emergency-burning-1749/
[Clock], Clock Image, Creative Commons
https://www.flickr.
com/photos/75680924@N08/6830220892/in/photostream/
[Space needle] Space needle, Creative Commons Zero
https://unsplash.com/photos/-48aJfQpFCE
[Explosion] Vehicle and building on fire, Creative Commons Zero
https://unsplash.com/photos/28v9cq7ytNU
[Stonehenge] Stonehenge, Public Domain
[Cat] Firsalar the fluffy cat loves to sit in boxes, CC-BY 2.0
[Construction Kit] Free Universal Construction Kit by F.A.T. Lab and Sy-
Lab http://fffff.at/free-universal-construction-kit/
[MFS] While MFS was released to production for a short time, it was not
released for external customer use.
[Audience] Audience, Creative Commons Zero
https://unsplash.com/photos/bBQ9lhB-wpY
[Bird] Bird Stare, Creative Commons Zero
https://unsplash.com/photos/ig9lRTGT0h8
[Road] Yellow Brick road to Lost Hatch, CC-BY-NC 2.0 https://www.flickr.
com/photos/wvs/352414272
[Hands] Mud Hands, CC-BY-NC 2.0 https://www.flickr.
com/photos/migueltejadaflores/13783685515
[Docker-it-scala] Docker IT Scala, Whisk Labs, MIT https://github.
com/whisklabs/docker-it-scala