How often have you heard "it's gonna be hard or impossible to cover this part of our backend application"? I've heard a lot. Because our modern application always involve databases, message queues, 3rd party services. What if I tell you, that I can cover even logs with tests, values of custom application metrics, apps that involve Google Cloud Storage, Google Data Storage, Kafka, Redis, Lambdas, DynamoDB, Aerospike, S3. I work in highload project without manual QA's at all. And I'm confident in my everyday releases, so should be you.

2. It’s about
tests!
...and infrastructure around them!

3. Why do we need
tests*?
* by tests I mean autotests. I’m a
developer(R&D Team Lead). I hate
manual testing.

4. We want to have
less bugs in
production!

5. How our production looks like?
Frontend
Backend
DataBase
Web server
HTTP
Requests

6. How our backend tests should look like
Tests
Integration tests
MongoDB
Node.js
HTTP
Requests

7. Requirements for our tests
01
04
02
03
Easy to write Easy to setup
locally
Easy to setup
in CI/CD
As close to
production
environment

8. Simple Node.js app

10. Patron, dog mascot of the State Emergency
Service of Ukraine
*Patron translates as “bullet”

11. Mac
OS
Windo
ws
Linux
Intel 1 2 ∞
Arm 3 4 ∞
What are the options to run 3rd parties? (MongoDB)
Pros:
- easy to run in all
envs
01 02
Mock it Run real db on your computer
Cons:
- Will not represent
production
- Chance to
implement mocks
wrongly
Pros:
- Familiar way to
start DB
- Real DB
Cons:
- Different setup for
local envs and for CI

12. Where we can run 3rd parties? (MongoDB)
04
03 Use dedicated cloud infra
for tests
Run everything in docker
ecosystem
Pros:
- As close to
production as
possible
Cons:
- Hard to isolate test
runs
- Might require VPN
access to internal
resources
- Can cost money
Pros:
- easy to run in all
envs
- Close to production
Cons:
- Initial setup might
take some time
- Debug in containers
is harder

13. Running tests in Docker infra is
hard?
01
04
02
05
03
Build image for
an app
Spin all the
services
Run tests in
container
Change the
code or the
tests
Repeat 1 and 3
and 4?

14. What If I tell you that we can solve
all our problems with one command?
docker compose run --rm nodejs-app-test

15. docker compose run --rm nodejs-app-test
docker-compose.yaml

16. https://github.com/eagleeye/to-cover-uncoverable
https://gitlab.com/andrii.shumada/to-cover-uncoverable

17. Easy to setup in CI (gitlab)

20. …as well as on github.com

21. How our production looks like? (2)
Frontend
Backend
Queue
Node.js app
HTTP
Requests

22. What’s wrong with Kafka tests?
● Recreating topic is time
consuming
● There is no key-value access
● Failed test runs can produce
more messages then read

23. Sample Kafka producer app
https://github.com/eagleeye/to-cover-uncoverable/tree/master/02-kafka
https://gitlab.com/andrii.shumada/to-cover-uncoverable/-/tree/master/02-kafka

24. Simple test with kafka

26. Start zookeeper
Start Kafka
Build Node.js
Docker image
Run tests
Order of subtasks to run tests with Kafka

27. docker-compose.yaml

28. .gituhub/workflows/test-with-github.yaml .gitlab-ci.yml

31. How our production looks like? (3)
Frontend
Backend
Node.js app
HTTP
Requests

32. Localstack - mocks of AWS services
docker-compose.yml
https://localstack.cloud/
*
* better to fix version
●
- Lambda
- Kinesis
- Redshift
- DynamoDB
- ElastiCache
…and 45 more services

33. Node.js app
DataStore
Cloud functions
Big Query
Not all the 3rd parties we can run
locally…

34. For DataStore
Managing shared cloud resources could be
hard, specially when you have multiple
developers trying to reach your resources

35. Cloud function
Option 2
Spin up cloud function
emulator
Write unit test for
controller
Option 1

36. Testing cloud functions
https://cloud.google.com/functio
ns/docs/testing/test-http
Or
Personal
experience

37. What if our production production is
even more complex? (4)

38. We tested all the gears, but will it
work a gearbox..

39. No way that you can write
that complex test…*
You need more than
1 team to develop,
maintain and test
that
*I know you can,,,

40. Where we can run 3rd parties? (Google services)
04
03
Use dedicated cloud infra
for tests
Run everything in docker
ecosystem
Pros:
- As close to
production as
possible
Cons:
- Hard to setup
isolation for
separate tests
- Might require VPN
access to internal
resources
Pros:
- easy to run in all
envs
- Close to production
Cons:
- Initial setup might
take some time
- Debug in containers
is harder

41. Continuous QA in Qa environment to the rescue!
Service that
constantly
sends data
in QA
(emulates
clients)
HTTP
Requests
Grafana
(any monitoring)
Zoo of services
and 3rd parties

42. How grafana
appeared here?

43. How I know that my
product (apps and
infrastructure) works
properly in
production?
1) I don’t receive calls from
support and my boss
2) Metrics and alerts

44. Metrics and alerts on:
- Status codes
- Number of error logs
- Business metrics
- Container restarts
- Resources usage
How can I know, that
my product works in
QA?

45. Example of continuous test
Client
automation
HTTP
Requests
- 10 valid POST
/pets requests
with unique pets
- 5 invalid
requests
- 5 valid GET
/pet/:id requests
- 15 of 200 status codes
- 5 of 400 status codes
- Incoming records per second in
topic should be 10
- Increased number of items in
your database by 10
- Cloud function was executed 10
times
- Cloud function executions
finished without errors
- No logs with severity > error
- No restarts in all containers
- CPU, memory, disk metrics
should be < 80%
- …..
Sends similar
requests
Every second

46. Conclusion
- Use Docker compose infrastructure to
organize your local development
infrastructure and as infrastructure for
tests locally and on CI/CD
- Use continuous testing in your QA
environment by making automated
requests to your infra and monitoring
should catch problems if they occur

47. Thanks!
Do you have any questions?
Discord: #track-b, or Voice Chat: Andrii Shumada
eagleeyes91@gmail.com
https://eagleeye.github.io
https://gitlab.com/andrii.shumada/to-cover-uncoverable
https://github.com/eagleeye/to-cover-uncoverable
https://u24.gov.ua/
Remember to support Ukraine!