True End-to-End Testing in Scala

TRUE
END-TO-END
TESTING
SCALA
Orr Sella
IN
Scalapeño 2014

AGENDA
WHAT ARE THEY
WHY WE NEED THEM
HOW TO WRITE THEM
END-TO-END
TESTS
TEST HARNESS
DRIVERS
FAKES
…

AGENDA
WHAT ARE THEY
WHY WE NEED THEM
HOW TO WRITE THEM
END-TO-END
TESTS
TEST HARNESS
DRIVERS
FAKES
… CO N C R E T E E X A M P L E
github.com/orrsella/scala-e2e-testing

Disclaimer
BIG SUBJECT,
WE CAN’T COVER
EVERYTHING

Why
SCALA & END-TO-END
TESTS
1 THE APPS
Typical Scala applications are ripe for end-to-end
tests
!
THE LANGUAGE & ECOSYSTEM
Build tools, testing libraries and language
features are a great fit
2

The Bible: GOOS
INTRO TO TDD
!
TESTING
TECHNIQUES
!
LOTS OF CODE &
EXAMPLES

THELEVELS OF
TESTING KNOWN TO MAN
1 UNIT
Do our objects do the right thing?
!
INTEGRATION
Does our code work against code we can’t change?
!
END-TO-END
Does the whole system work?
2
3
(not really)

WHAT IS
THE
RIGHT BALANCE
OF THE THREE?

THEINVERSE
ICE CREAMCONE(or pyramid)
NUMBER OF
TESTS

THEINVERSE
END-TO-
END
INTEGRATION
UNIT
NUMBER OF
TESTS

THEINVERSE
END-TO-
END
INTEGRATION
UNIT
MANUAL TESTING
(if you must) NUMBER OF
TESTS

END-TO-END TESTS
Interact with the system from the outside
Black Box
MOST IMPORTANTLY:
Exercising both the system and the
process by which it’s built and deployed
Include interaction with external
environment

AN IDEALAUTOMATIC
BUILDWOULD
1 Compile and unit-test the code
Integrate and package the system
Perform a production-like deployment to a
realistic environment
Exercise the system through its external access
points
2
3
4

Whoa,
THIS SOUNDS LIKE A
LOT OF WORK

Whoa,
THIS SOUNDS LIKE A
LOT OF WORK
IT IS.

But,
IS IT WORTH IT?
YES. (I hope to convince you)

WHY
ARE
END-TO-END
TESTS SO
IMPORTANT?

HERE ARE3 MAIN
(many more...)
REASONS

1
FLUSH OUT
THE UNKNOWNS
Requires asking (and answering) many
awkward questions
Expose uncertainty early, including
technical and organizational risks
Forces to understand how a system fits
into the world
Identify all the integration (read: potential
failure) points

1
FLUSH OUT
THE UNKNOWNS
Requires asking (and answering) many
awkward questions
Expose uncertainty early, including
technical and organizational risks
Forces to understand how a system fits
into the world
Identify all the integration (read: potential
failure) points
EXAMPLE IN
3 SLIDES

2
FEEDBACK &
CONFIDENCE
Feedback that would otherwise only show
in staging/production
Safety net of system-wide regression
Assurance that our chosen technology
stack (“plumbing”) works as expected
Confidence to make “risky” system-wide
changes (e.g: swap datastore, application
server)

3
FORCED
AUTOMATION
Everything must be automated so it
can be tested
Crucial for deployment which is
error-prone
Automatic Build-Deploy-Test cycle
means we can ship frequently

TIME
FOR A
CONCRETE
EXAMPLE:

TIME
FOR A
CONCRETE
EXAMPLE:
Memento

Memento?
NOTE-TAKING
WEB SERVICE

Memento?
NOTE-TAKING
WEB SERVICE
STOR E, RE TRIEVE, SEARC H, TRANSLATE

Memento?
NOTE-TAKING
WEB SERVICE
STOR E, RE TRIEVE, SEARC H, TRANSLATE “MICROSERVICE”

Memento?
NOTE-TAKING
WEB SERVICE
STOR E, RE TRIEVE, SEARC H, TRANSLATE “MICROSERVICE”REST API

LOAD
BALANCER
HAPROXY
REVERSE
PROXY
NGINX
APP
SERVER
FINATRA
DATASTORE
ELASTICSEARCH
TRANSLATION
SERVICE
YANDEX

LOAD
BALANCER
HAPROXY
REVERSE
PROXY
NGINX
APP
SERVER
FINATRA
DATASTORE
ELASTICSEARCH
TRANSLATION
SERVICE
YANDEX SCALA!..

LOAD
BALANCER
HAPROXY
REVERSE
PROXY
NGINX
APP
SERVER
FINATRA
DATASTORE
ELASTICSEARCH
TRANSLATION
SERVICE
YANDEX SCALA!..
PRETTY
STRAIGHTFORWARD

Ok. So.
WHAT CAN POSSIBLY
GO WRONG & BE
MISCONFIGURED?
i.e. integration (failure) points

LOAD
BALANCER
HAPROXY
PORT 80 /ETC/HAPROXY/
HAPROXY.CFG
LIST OF REVERSE
PROXIES:PORTS
/ETC/DEFAULT/
HAPROXY

REVERSE
PROXY
NGINX
APP
SERVER
FINATRA
/ETC/NGINX/
NGINX.CONF
PROXY
PORT 7770
INIT.D
SCRI P T
/ETC/NGINX/
SITES-AVAILABLE
APPLICATION.CONF
LO G BAC K . XM L
JAVA
DEPLOY NEW
VERSION

DATASTORE
ELASTICSEARCH
TRANSLATION
SERVICE
YANDEX
PORT 9300 MAPPING
URL API KEY
/ETC/ELSATICSEARCH.YML
JAVA

Ok,
SO A LOT
CAN
(read: will)
GO WRONG

Ok,
SO A LOT
CAN
(read: will)
GO WRONG
WE WANT OUR END-TO-END TESTS
TO E X E RC I S E A L L O F T H I S !

HOW DO
WE TEST
ALL OF THIS
END-TO-END?

HOW DO
WE TEST
ALL OF THIS
END-TO-END?
here is one approach…

HOW DO
WE TEST
ALL OF THIS
END-TO-END?
THERE ARE
OT H E R S

HOW DO
WE TEST
ALL OF THIS
END-TO-END?
THERE ARE
OT H E R S
BUT THIS IS THE
BEST, OBVIOUSLY

STEPSREQUIRED FOR
END-TO-ENDTESTS
0
1
BUILD & PACKAGE
Package the app for production deployment
2
3
VIRTUAL MACHINE
Fire-up a local VM instance with production-like env
CONFIGURATION & DEPLOYMENT
Run production config and deployment code,
having our entire system on one box
RUN TESTS
Execute end-to-end tests against the VM

HOW DO
WE
ORCHESTRATE
ALL OF THIS?

Enter:
THE TEST
HARNESS
EXTENSION OF THE BUILD
Runs all the necessary setup/teardown actions and
the tests themselves
!
(aka sbt)
STANDARD/SIMPLE/SCALA BUILD TOOL
Build and package our app (sbt-native-packager)
Fire-up the virtual machine (Vagrant)
Run configuration and deployment scripts (Ansible)

Meet
VAGRANT
PORTABLE DEV ENVIRONMENTS
Lightweight, reproducible and programmable
!
VIRTUALIZATION
Wrapper around VirtualBox, VMWare, more
!
SIMPLE & QUICK
Vagrantfile => `$ vagrant up`

THESTEPS REQUIRED FOR
END-TO-ENDTESTS
BUILD & PACKAGE
Package the app for production-like deployment
0
1 VIRTUAL MACHINE
2 CONFIGURATION & DEPLOYMENT
3
RUN TESTS

Test Harness:
SBT
MANAGING
VAGRANT
// project/Vagrant.scala
!
object Vagrant {
!
private lazy val vagrant = settingKey[Vagrant]("vagrant")
!
lazy val settings = Seq(
test in EndToEndTest <<= (test in EndToEndTest).dependsOn(publishLocal),
testOptions in EndToEndTest += Tests.Setup(() => vagrant.value.setup()),
testOptions in EndToEndTest += Tests.Cleanup(() => vagrant.value.cleanup())
)
}

Test Harness:
SBT
MANAGING
VAGRANT
// project/Vagrant.scala
!
object Vagrant {
!
private lazy val vagrant = settingKey[Vagrant]("vagrant")
!
lazy val settings = Seq(
test in EndToEndTest <<= (test in EndToEndTest).dependsOn(publishLocal),
testOptions in EndToEndTest += Tests.Setup(() => vagrant.value.setup()),
testOptions in EndToEndTest += Tests.Cleanup(() => vagrant.value.cleanup())
)
}
TEARDOWN
HOOK
SETUP HOOK

Test Harness:
SBT
MANAGING VAGRANT
class Vagrant(vagrantFile: File) {
!
// cli method wrappers
private def up() = Process("vagrant" :: "up" :: Nil, dir)!
private def provision() = Process("vagrant" :: "provision" :: Nil, dir)!
!
def setup(): Unit = {
prevStatus = status()
prevStatus match {
case Running => provision()
case Saved => up(); provision()
case NotCreated => up()
case Unknown => up()
}
}
!
def cleanup(): Unit = if (prevStatus != Running) suspend()
}

Test Harness:
SBT
MANAGING VAGRANT
class Vagrant(vagrantFile: File) {
!
// cli method wrappers
private def up() = Process("vagrant" :: "up" :: Nil, dir)!
private def provision() = Process("vagrant" :: "provision" :: Nil, dir)!
!
def setup(): Unit = {
Tip: VA G R A N T S TAT E
“J U G G L I N G ” , S AV E S A L O T
prevStatus = status()
prevStatus match {
case Running => provision()
case Saved => up(); provision()
case NotCreated => up()
case Unknown => up()
}
}
!
def cleanup(): Unit = if (prevStatus != Running) suspend()
}
OF TIME!

END-TO-ENDTESTS
0
1
BUILD & PACKAGE
3
VIRTUAL MACHINE
RUN TESTS

THEDEPLOYMENT
SCRIPTS
(Provision in Vagrant parlance)
CONFIGURATION MANAGEMENT
Ensure that our system is configured properly
(OS, settings, packages, file system, etc.)
!
DEPLOY THE APP
Upgrade to the latest version just compiled and
packaged, as would be done in production
!
APPLICATION CONFIG
Configure the application itself the same way it
would in production (but with test values)

Specifically,
ANSIBLE
BUT ANY
CM TOOL WILL DO
(Chef, Puppet, Shell scripts)

Provision:
VAGRANTFILE
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
!
config.vm.box = "ubuntu/trusty64"
config.vm.network "forwarded_port", guest: 7770, host: 7769
...
!
config.vm.provision "ansible" do |ansible|
ansible.playbook = "ansible/site.yml"
ansible.inventory_path = "ansible/inventories/vagrant"
end
!
end

Provision:
VAGRANTFILE
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
!
config.vm.box = "ubuntu/trusty64"
config.vm.network "forwarded_port", guest: 7770, host: 7769
...
!
config.vm.provision "ansible" do |ansible|
ansible.playbook = "ansible/site.yml"
ansible.inventory_path = "ansible/inventories/vagrant"
end
!
end
VA G R A N T
INVOKES THE
PROVISIONER

END-TO-ENDTESTS
BUILD & PACKAGE
VIRTUAL MACHINE
0
1
3 RUN TESTS
2

END-TO-ENDTESTS
BUILD & PACKAGE
VIRTUAL MACHINE
0
1
3 RUN TESTS
2
CHECK OUT
CO D E E X A M P L E

END-TO-ENDTESTS
BUILD & PACKAGE
0 Package the app for production-like deployment
1
VIRTUAL MACHINE
3 RUN TESTS

Finally,
TESTS
THEMSELVES
WRITING THE

SOMEGUIDELINES
FOR effectiveE2E TESTS
TEST SPARINGLY
End-to-End tests are slow; test few “sunny-day”
scenarios
!
ABSTRACTION
Tests validate features and business logic, not
implementation details
!
AVOID COUPLING
Tests should exercise the system from the
outside without reaching for its guts

Memento:
NOTES CONTROLLERTEST
class NotesControllerEndToEndTest
extends Specification
with NotesControllerDriver
with ResponseMatchers {
!
"Notes controller" should {
"add a note and then successfully get it" in {
!
val addResp = anAddNoteRequest.withText("Hello world!").execute()
addResp must beOk
addResp.noteId must not beEmpty
!
val getResp = aGetNoteRequest.withId(addResp.noteId).execute()
getResp must beOk
getResp.text must_== "Hello world!"
}
}
}

Memento:
!
USING
SPECS2
!
addResp must beOk
!
getResp must beOk
}
}
}

Memento:
!
!
addResp must beOk
!
getResp must beOk
}
}
}
TESTING THE
SYSTEM BY
USING ITSELF

Memento:
!
TEST IS DECOUPLED FROM
THE CONTROLLER API
ITSELF BY USING DRIVERS
!
addResp must beOk
!
getResp must beOk
}
}
}

Drivers:
ABSTRACTING THE SUT
Interact with the SUT directly
instead of the test
Coupled to the API/protocol
Run same test with different drivers

Drivers:
ABSTRACTING THE SUT
trait NotesControllerDriver {
!
def anAddNoteRequest = AddNoteRequest()
!
case class AddNoteRequest(text: String = "Lorem ipsum")
extends Request[AddNoteResponse] {
!
val method = HttpMethod.POST
val path = "/notes"
val params: Map[String, String] = Map()
val headers: Map[String, String] = Map()
val body: Option[String] = Some("{ "text": "" + text + "" }")
!
def withText(text: String) = copy(text = text)
}
...
}

Drivers:
ABSTRACTING THE SUT
!
!
!
val path = "/notes"
!
}
...
}
ONLY THE DRIVER
INTIMATELY KNOWS
THE CONTROLLER

Drivers:
ABSTRACTING THE SUT
!
!
!
val path = "/notes"
!
}
...
}
ONLY THE DRIVER
INTIMATELY KNOWS
THE CONTROLLER
class AddNoteResponse(response: Response)
extends BaseResponse(response)
with JsonResponse {
!
lazy val noteId = (json "noteId").extract[String]
}

Drivers:
ABSTRACTING THE SUT
!
!
!
val path = "/notes"
!
}
...
}
ONLY THE DRIVER
INTIMATELY KNOWS
THE CONTROLLER
with JsonResponse {
!
}
ONLY THE DRIVER
KNOWS THE
RESPONSE IS JSON

Drivers:
ABSTRACTING THE SUT
!
!
!
val path = "/notes"
!
}
...
}
ONLY THE DRIVER
INTIMATELY KNOWS
THE CONTROLLER
with JsonResponse {
!
}
ONLY THE DRIVER
KNOWS THE
RESPONSE IS JSON
OR THE FIELD
NAME

Matchers:
ABSTRACTING THE SUT
The other side of the Driver, i.e: the
response
Help decoupling from the protocol,
make the test be about features
Make test more readable

Memento:
!
trait ResponseMatchers extends Matchers {
def intToIntMatcher(t: Int): Matcher[Int] = beEqualTo(t)
private implicit !
def beOk = haveStatus(200)
def beBadRequest "add a note and = haveStatus(then successfully 400)
def haveStatus(404)
get it" in {
!
beNotFound = !
=
def haveStatus(status: Matcher[Int]): Matcher[Response] val addResp = anAddNoteRequest.withText("Hello world!").execute()
addResp must beOk
!
((_: Response).status) ^^ status
getResp must beOk
}
}
}
}

As always,
LIFE IS ABOUT
COMPROMISES
AND SETTING
BOUNDARIES

As always,
LIFE IS ABOUT
COMPROMISES
AND SETTING
BOUNDARIES
=> USE FAKES

FAKES FOR
USE
EXTERNAL SERVICES NOT EASILY AVAILABLE(Sometimes called
Simplicators)
Avoid making external network calls in tests
Create Fakes based on available documentation
Fakes should have minimal implementation to
only support testing
Test the Fake against the real service in a
contract test that’s manually run

!
private val port = 9921
private val server = new FakeYandexTranslateServer(port)
!
step {
server.start()
}
!
"add a note and then get it translated" in {
...
val translated = aTranslateRequest.withId(noteId).withLang("es").execute()
translated must beOk
translated.text must_== "Buenos días"
}
}
!
step {
server.stop()
}
}

!
private val port = 9921
private val server = new FakeYandexTranslateServer(port)
!
step {
class FakeYandexTranslateServer(port: Int) extends SimpleHttpServer(port) {
!
private val json =
server.start()
}
!
"""{
|"code": 200,
|"lang": "en-es",
|"text": ["Buenos días"]
|}
""".stripMargin
"add a note and then get it translated" in {
...
val translated = aTranslateRequest.withId(noteId).withLang("es").execute()
translated must beOk
translated.text must_== "Buenos días"
}
}
!
step {
!
override protected def onSimpleRequest(request: HttpRequest): String = json
}
server.stop()
}
}

How to
CONTINUE
FROM HERE
CHECK OUT THE EXAMPLE
!
END-TO-END TEST NEW PROJECTS
It is easier to get started with e2e tests on a new
project, integrating into an existing one is harder
!
READ
Growing Object-Oriented Software Guided by Tests

True End-to-End Testing in Scala

True End-to-End Testing in Scala

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (14)

Recently uploaded

Recently uploaded (20)

True End-to-End Testing in Scala