Applications without tests (legacy applications) present particular problems to developers when changes are required. Unless the change is very small, they require a lot of manual testing. This is even more true when we wish to completely replace such an application. At Nexthink, we replaced a legacy Ruby on Rails application with a web application written using Spray and Slick. There were two parts, a Rest API, accessed from lots of different version of our products, and an administration web interface to allow editing of the data. In replacing this application, we had to ensure that the new implementation returned exactly the same answers to requests as the old application for the same input criteria, without knowing in advance what the requests were. This presentation shows how a combination of technologies was used to achieve this goal, including: spray - for the reactive http container proxy - the new application was used as a proxy to the old application, and the results were compared with the new application (in production) scalacheck - to generate structured queries on the old and new application, for testing gatling - for performance tests This presentation covers both how the replacement was done, to ensure the minimum number of bugs in production, along with our subsequent experience in production.

1. Nexthink Library
in Scala
Implementation & Techniques
Replacing a Ruby on Rails Application with
Scala / Spray

2. Matthew Farwell
Senior Developer @ Nexthink in Lausanne
Java / Scala / JVM
Project lead on Scalastyle, the style checker for Scala
Contributor to other open source projects, JUnit
Co-author of “sbt in Action” with Josh Suereth

3. The Nexthink Library

4. The Nexthink Library

5. Project Goals
To rewrite the existing Ruby on Rails Library
To make it more ‘mainstream’ in the company
Increase the truck number
Have adequate performance for the future.
Assess technologies:
– Spray for web container, async, non-blocking
– Slick for database access (instead of Hibernate)
– Gatling for performance tests
– Angular JS for front end

6. Existing solution
Split into two parts:
API used by Finder and Portal, both GET and POST
5 different types of request
Administration section, for updating content
Requirements
– For the API, it must do exactly the same as the
legacy.
– The admin section can be improved if we want

7. Approach
A single rest server which serves both the API and the
admin screens (Spray)
The admin screens use Angular JS, which themselves
use the REST APIs
Server is stateless, therefore no sessions, etc.
Text search uses Lucene

8. Spray
Spray is a DSL to describe a REST API along with the
actions. It is a mixture of directives and actions:
path("1" / "ping") {
get {
respondWithMediaType(`application/json`) {
complete {
Ping(Settings.config.hostname, databaseOk())
}
}
}
}
implicit val pingFormat = jsonFormat2(Ping)

9. Spray
It makes things easy to do asynchronously and non-
blocking:
complete {
future {
val u = entityService.update(id, obj, user)
future { LibraryApi.refreshCache }
u
}
}
complete {
pipeline(Get("http://google.com"))
}

10. Testing with comparison

11. API Testing
We can test the new library by comparing directly with
the legacy library. So how do we generate the test
cases? We can:
use the access log for the legacy library to get a typical set of
requests
generate test cases using Scalacheck

12. Scalacheck
Scalacheck allows you to generate data for tests
It can generate random structured data
It then calls the Method Under Test and compares the result
using properties
It can then ‘reduce’ the use case to the smallest size which fails
In our case, we only use the generation methods

13. Scalacheck example
This generates one type of request:
val gen: Gen[Parameters] = for {
elementTypes <- Gen.oneOf("alert", "investigation")
tag <- Gen.containerOf[List, String](
Gen.oneOf("", "cloud", "security"))
version <- Gen.oneOf("", "3.1", "4", "5")
} yield Parameters("categories", 100, tag, elementTypes,
version, license)
We run this 5000 times per type of request for our basic tests.
Scalacheck is very good for exploring an API and finding edge cases.

14. Proxy/Comparison

15. Proxy/Comparison
The new library proxies the legacy. When we call the
legacy (non-blocking), we return those results.
When we return the results, we asynchronously compare
the results with the new library.
We can put this into production without fear
complete {
val f = pipeline(Get(legacyUrl(params)))
f onComplete { result =>
compare(result, localApi(params))
}
f
}

16. Proxy/Comparison
If there is a comparison failure, we store the parameters
that failed, which we can access from the FE.
We record all of the parameters from a GET or a POST.
We can also replay a failure
This approach means that we have to share the
database – so we didn't change it at all.
We had different comparison strategies depending upon
the type of API call.

17. Some numbers
Legacy RoR
~3400 lines (.rb, .erb, .xml.builder)
New:
~3400 lines Scala (500 lines comparison/proxy)
~1000 lines JavaScript (Angular JS)
~650 lines HTML
Spring app (similar) backend with DB: ~9000 lines Java
Frontend (GWT): ~16000 lines Java

18. Performance Tests – Gatling
Define a scenario:
val scn = scenario("basic usage")
.group("search") {
repeat(100) {
exec(http("google").get("http://google.com"))
.pause(0 milliseconds, 2 milliseconds)
.exec(http("yahoo").get("http://yahoo.com"))
}
}

19. Performance Tests – Gatling
Execute it:
setUp(scn.inject(ramp(10 users) over (10 seconds)))
.protocols(httpConf)
.assertions(
global.successfulRequests.percent.is(100),
details("search"/"google").responseTime.max.lessThan(100),
details("search"/"yahoo").responseTime.max.lessThan(100))
Gatling

20. Performance – RoR baseline
5 Users, 5 times, 10 requests == 250 requests
Total time: 227 seconds, req/s = 1
Caveats: unstable setup, production (single request) was
faster x 10
Mean Max
search 7600 8900
featured 2500 3200
copy 6300 7000

21. Performance – New
5 Users, 5 times, 10 requests == 250 requests
Total time: 5 seconds, req/s = 50
Mean Max
search 11 30
featured 5 10
copy 0 0

22. Performance – New
100 Users, 100 times, 10 requests == 100,000 requests
Total time: 28 seconds, req/s = 3571
3571 req/s == 300,000,000 / day
Mean Max
search 33 660
featured 24 500
copy 22 460

23. Performance – New
1000 Users, 100 times, 10 requests == 1,000,000 requests
Total time: 280 seconds, req/s = 3564
Required: 150-160k / day
We can support 1 day’s traffic in < 1 minute. We have
room for expansion :-)
Mean Max
search 383 15400
featured 259 23510
copy 260 33310

24. Some interesting numbers
Number of failures during performance tests: 0
This means that the system is very stable.
The throughput for 100 concurrent users (3571 req/s) is
about the same as for 1000 concurrent users (3564
req/s), but each request takes longer when there are
1000 concurrent users.
This is due to back pressure.

25. Production
Production: April 2014
Proxies were in place 6-8 weeks
Bugs found:
1 - incorrect version (attribute value) returned
2 – Finder was using empty url as a ping
3 – incorrect data returned (cause: data in database)
4 – encoding problem

26. Conclusion
Scala (language and tools)
compilation slow, tooling great (sbt & worksheet)
Spray
easy to use, simple to extend, no real ‘magic’
Scalacheck
very good for exploring an API, and finding edge cases, good for
data oriented testing
Slick
– easy to use, we were only doing simple things, but very similar
to working with direct SQL.

27. Conclusion
Angular JS
Great. Better than some of the other solutions we've had.
Async / non-blocking
Fairly easy to manage in Scala, harder in Java. Harder when
dealing with databases and non-web services.
Gatling
Great. Simple and easy. Miles better than the alternatives
(JMeter for example)

28. Questions?
Twitter: @matthewfarwell
Scalastyle: @scalastyle http://www.scalastyle.org
Sbt in Action:
https://www.manning.com/books/sbt-in-action
Nexthink are hiring! http://www.nexthink.com