SlideShare une entreprise Scribd logo
1  sur  47
Télécharger pour lire hors ligne
Software Testing
Roberto Casadei
{roby.casadei}@unibo.it
C.D.L. Magistrale in Ingegneria e Scienze Informatiche
Alma Mater Studiorum—Universit`a di Bologna, Cesena
a.a. 2016/2017
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 1 / 47
Outline
Outline
Recap on unit tests in JUnit and TDD
Operative introduction to “automated builds with tests and
dependencies” (in sbt)
Main kinds of tests: unit, integration, functional, stress, acceptance
Testing framework for Scala: ScalaTest
Test coverage
Introduction to test doubles and mocking
Some pointers to more advanced stuff on testing
Effective unit testing
Property-based testing (cf., QuickCheck, ScalaCheck)
Cucumber for behavior-driven development
Selenium for testing web apps via browser automation
Testing of GUI-based programs
Performance regression testing with ScalaMeter
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 2 / 47
Outline
1 Quick recap
2 sbt
3 Panorama on testing
4 Basics of testing in Scala with ScalaTest
5 Test doubles
6 More on testing
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 3 / 47
Recap: JUnit
1 public class DeviceImplTest {
2 private DeviceImpl device;
3
4 @Before public void init (){
5 this.device = new DeviceImpl ();
6 }
7
8 @Test public void on() throws Exception {
9 this.device.on();
10 assertTrue(this.device.isOn ());
11 }
12 // ..........
13 }
Concepts
Test suite, test cases, tests
Assertions
Fixture
Arrange, Act, Assert
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 4 / 47
Recap: Test-Driven Development (TDD)
What
“Development” is guided (“driven”) by “tests”
I.e., it is not a testing activity; tests as (useful) side-effect
Exercise your code before even writing it
Makes you explore the problem before attempting a solution
Forces you to think as the user of your code
Makes you focus on what is important right now (cf., KISS)
Red – Green – Refactor cycle
Benefits
Specification before implementation
Guides the (detailed) design process
You end up with regression tests (i.e., tests to catch regressions)
A regression is when something correct gets broken
A safety net against regressions is fundamental for refactoring activities
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 5 / 47
Outline
1 Quick recap
2 sbt
3 Panorama on testing
4 Basics of testing in Scala with ScalaTest
5 Test doubles
6 More on testing
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 6 / 47
The Scala Build Tool (sbt): intro
Goal: improve productivity through automation of project-related activities
sbt: the “standard” build tool for Scala
Belongs to the family of build tools, like Maven, Gradle and others
Gradle is the one that will be covered in this course
Not limited to Scala projects
Can seamlessly be used for Java projects
Plugins for other languages (e.g., Groovy, Clojure, C++a
, Jythonb
..)
a
https://github.com/d40cht/sbt-cpp
b
https://github.com/markchadwick/jython-sbt-plugin
Lightbend Activator
A custom version of sbt which adds two extra commands, activator
ui and activator new
Supports bootstrapping applications from a catalog of templates
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 7 / 47
The Scala Build Tool (sbt): basics
Key concepts
A build definition is defined in /build.sbt
It specifies a set of projects
Each projects is associated with a set of settings and tasks
Two modes of operation
1. Batch mode: $ sbt clean compile
2. Interactive mode
1 $ sbt
2 sbt > test
3 ...
4 sbt > console
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 8 / 47
The Scala Build Tool (sbt): basic usage
Directory structure (Maven-style)
1 projectRoot/
2 build.sbt
3 lib/
4 src/
5 main/resources/
6 main/scala/
7 main/java/
8 test/resources/
9 test/scala/
10 test/java/
11 project/
12 build.properties
13 plugins.sbt
14 target/
build.sbt
1 lazy val root = (project in file(".")).settings(
2 name := "mytest", version := "1.0", organization := "unibo.pps",
3 scalaVersion := "2.12.2",
4 libraryDependencies += "com.novocode" % "junit -interface" % "0.10
" % "test")
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 9 / 47
The Scala Build Tool (sbt): basic usage
Basic commands
help: shows help
settings: lists the settings for the current project
tasks: lists the tasks for the current project
<settingName>: shows the value of the setting
<taskName> ..args..: execute the task
Common tasks
clean: deletes files produced by build
compile: compiles sources
console: starts Scala interpreter with project classes in classpath
package: produces a main artifact (e.g., a JAR)
runMain <mainClass> ..args..
test: executes all tests
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 10 / 47
Outline
1 Quick recap
2 sbt
3 Panorama on testing
4 Basics of testing in Scala with ScalaTest
5 Test doubles
6 More on testing
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 11 / 47
Perspectives on testing1
“Schools” of testing
Analytic School: testing as a rigorous task, via formal methods
Motto: Programs are logical artifacts, subject to math laws
Factory School: testing to measure project progress
Motto: Testing must be planned/scheduled/managed
Quality School: testing as a quality process
Motto: A quality process for a quality product
Context-Driven School: testing as a stakeholder-oriented, adaptable
process
Key question: which testing would be more valuable right now?
Agile School: testing to facilitate change
Techniques: test automation, test-driven approaches
Key takeaway: perspectives on testing impact the what and how of testing
1
https://www.prismnet.com/~wazmo/papers/four_schools.pdf
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 12 / 47
Many different kinds of testing
Multiple dimensions
Technology-facing vs. Business-facing testing
Verification vs. validation
Granularity/scope of testing
Unit – integration – system-level/end-to-end/acceptance
Static vs. dynamic
Code-level vs. runtime
Manual (human-based) vs. automated (computer-based)
Examples of human testing: inspection (code reviews), walkthrough
Examples of computer-based testing: static analysis, “standard” testing
Structural (aka white-box) vs. functional (aka black-box)
Examples of structural testing: control flow testing, path testing
Examples of functional testing: equivalence partitioning, BVA
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 13 / 47
Levels of testing
Levels of testing based on granularity/scope
1. Unit testing: testing at the level of individual units (of functionality)
2. Integration testing: testing of the functionality provided by multiple
integrated/interacting units
3. System testing: testing the whole system for correctness
In a black-box way
Non-functional requirements (e.g., efficiency, reliability, security) are
typically tested at this level
4. Acceptance testing: testing the whole system against the needs and
expectations of users and stakeholders
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 14 / 47
Acceptance tests
Acceptance testing
Is the SW system “acceptable” (for delivery)?
It’s about testing whether the SW system satisfies the acceptance
criteria (e.g., as specified in the requirements)
Acceptability is matter of customers, product owners, stakeholders, or
users
It is very much about what has to be built
⇒ Acceptance test-driven planning/development (ATDP/D)
Tools: FitNesse (wiki-based), JBehave, Cucumber (see later)
Acceptance testing vs. system testing
System tests ⇒ build the thing right (things should behave correctly)
Acceptance tests ⇒ build the right thing (validate what you want)
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 15 / 47
Scalability testing
Scalability testing: how is the ability of the system to accomodate
increasing loads?
Stress testing
Testing the system under unusual conditions (e.g., DoS attack)
Testing the upper limits (e.g., by exceeding the max load or
diminishing resources) until the SW crashes
Strictly related to reliability and availability requirements
Load testing
Testing system performance under high-load conditions
Example metrics: maximum/average response time, throughput
Tools: Gatlinga, The Grinderb,
a
http://gatling.io/
b
http://grinder.sourceforge.net/
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 16 / 47
Code/Test coverage
Coverage
Several definitions/metrics (and inconsistent terminology)
Many “things” can be “covered”, and at many levels
Code coverage: which/how much code is executed during testing
Mainly used for finding untested codea
Note: 100% coverage doesn’t mean that tests cover all the scenarios
Test coverage: how much of the behaviour is tested
The goal is not 100% coverage → testing the “right things”
a
https://martinfowler.com/bliki/TestCoverage.html
Further readings
“Coverage is not strongly correlated with test suite effectiveness” (Inozemtseva and
Holmes 2014) [3]
“How to misuse code coverage” (Marick 1999) [5]
“A survey of coverage-based testing tools” (Yang, Li, and Weiss 2009) [9]
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 17 / 47
Code coverage in practice
Configuration in sbt
project/plugins.sbt
1 addSbtPlugin ("org.scoverage" % "sbt -scoverage" % "1.5.0")
Enable coverage in build.sbt
1 coverageEnabled := true
Usage
1 $ sbt
2 sbt > coverage // Enables coverage
3 sbt > test // Execute tests
4 sbt > coverageReport // Generate coverage reports
5 sbt > exit
6
7 $ open target/scala -2.11/ scoverage -report/index.html
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 18 / 47
Outline
1 Quick recap
2 sbt
3 Panorama on testing
4 Basics of testing in Scala with ScalaTest
5 Test doubles
6 More on testing
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 19 / 47
Testing in Scala: ScalaTest
Key features
Provides integration with many tools to provide a full, seamless
testing experience
JUnit/TestNG, ScalaCheck, JMock, ScalaMock, ...
Ant, Maven, sbt
Eclipse, NetBeans, IntelliJ
Supports several styles of testing out of the box
Recommendation: 1 style for unit testing, 1 style for acceptance testing
Configuration in sbt:
1 val scalatest = "org.scalatest" % " scalatest_2 .12" % "3.0.1" % "
test"
2
3 val commonsettings = Seq( scalaVersion := "2.12.2")
4
5 lazy val root = (project in file(".")).
6 settings( commonsettings ).
7 settings(name := "hello -scalatest",
8 libraryDependencies ++= Seq(scalatest))
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 20 / 47
ScalaTest: styles I
http://www.scalatest.org/user_guide/selecting_a_style
FunSuite: test functions as in classical XUnit libraries
1 class SetSuite extends FunSuite {
2 test("An empty Set should have size 0") {
3 assert(Set.empty.size == 0)
4 }
5
6 test("Invoking head on an empty Set should produce
NoSuchElementException ") {
7 assertThrows [ NoSuchElementException ] {
8 Set.empty.head
9 }
10 }
11 }
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 21 / 47
ScalaTest: styles II
FlatSpec: flat specifications `a la BDD
BDD (Behaviour-Driven Development) is a refinement of (A)TDD
From “tests” to “behaviour specifications”
Emphasis on communication
Stylistic impact + tooling impact (cf., Cucumber @ next slide)
1 class SetSpec extends FlatSpec {
2 "An empty Set" should "have size 0" in {
3 assert(Set.empty.size == 0)
4 }
5
6 it should "raise NoSuchElementException when head is invoked" in
{
7 assertThrows [ NoSuchElementException ] {
8 Set.empty.head
9 }
10 }
11 }
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 22 / 47
ScalaTest: styles III
FunSpec: specifications `a la BDD with (nested) describe blocks
1 class SetSpec extends FunSpec {
2 describe("A Set") {
3 describe("when empty") {
4 it("should have size 0") {
5 assert(Set.empty.size == 0)
6 }
7
8 it("should raise NoSuchElementException for head") {
9 assertThrows[ NoSuchElementException ] {
10 Set.empty.head
11 }
12 }
13 }
14 }
15 }
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 23 / 47
ScalaTest: styles IV
Many styles: recap
FunSuite: for xUnit people
FlatSpec: transition from xUnit to BDD-style
FunSpec: for BDD people
... other variants, e.g., giving more flexibility in the BDD style, ...
PropSpec: for property-based testing (see @ next slide)
FeatureSpec: for acceptance BDD (see @ next slide)
Guidelines for choice
One main style for unit testing (e.g., FlatSpec)
One main style for acceptance testing (e.g., FeatureSpec)
PropSpec when property-based testing suited
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 24 / 47
Outline
1 Quick recap
2 sbt
3 Panorama on testing
4 Basics of testing in Scala with ScalaTest
5 Test doubles
6 More on testing
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 25 / 47
Test doubles [6]: why and what
Test doubles: objects to be substituted for the real implementation for
testing purposes
Allow to gain full control if the context/environment of a piece of
code
Test doubles are essential for good test automation
Allowing isolation of code under test (from its dependencies)
Speeding-up test execution
Making random behavior deterministic
Simulating particular conditions that would be difficult to create
Observing state & interaction otherwise invisible
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 26 / 47
Kinds of test doubles
Kinds of test doubles
Dummy: objects passed around but never actually used; often used
to fill parameter lists (e.g., as null)
Stubs (“unusually short things”): simple implementations, e.g., with
methods that do nothing or return default values
Fake objects: have working implementations but provide a
fast/simple alternative (e.g., in-memory DB)
Test spies: stubs that record information about how they are called
Mocks: test spies configured/prescribed to behave in a certain way
under certain circumstances
Tests also verify mocks are used in a certain way (white-box)
References
http://xunitpatterns.com/TestDouble.html
https://martinfowler.com/articles/mocksArentStubs.html
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 27 / 47
Mocking with ScalaMock I
Usage
1 libraryDependencies += "org.scalamock" %% "scalamock -scalatest -support" % "3.5.0" % Test
ScalaMock can be used in tests written in ScalaTest (or Specs2)
In ScalaTest, mix trait MockFactory in your test class
Mocking styles in ScalaMock
Expectations-First style: expectations are set on mocks before
exercising the System Under Test (SUT)
If these expectations are not met, the test fails
Record-then-Verify style: expectations are verified on stubs after
the SUT has executed
In ScalaMock, this is supported by stubs
(1) for objects for which we have strict expectations, (2) otherwise
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 28 / 47
Mocking with ScalaMock II
SUT
1 case class Request(body: String)
2 case class Response(content: String)
3
4 class Server {
5 def processRequest (req: Request): Response = Response(req.body+ "’s Response")
6 def serve(req: Request): Response = processRequest (req)
7 }
8
9 trait Cache[K,T] { // Note: we provide no implementation class yet
10 def cached(key: K): Boolean
11 def get(key: K): T
12 def put(key: K, value: T): Unit
13 }
14
15 class ServerWithCache (private val cache: Cache[Request , Response],
16 private val cacheable: Request => Boolean) extends Server {
17 override def serve(req: Request) = if(! cacheable(req)) super.serve(req) else {
18 if (cache.cached(req)) {
19 cache.get(req)
20 } else {
21 val res = processRequest (req)
22 cache.put(req , res)
23 res
24 }
25 }
26 }
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 29 / 47
Mocking with ScalaMock III
1 class TestWithMock extends FunSuite with MockFactory with Matchers {
2 test("Test server with cache"){
3 // Mocks/stubs for dependencies
4 val cacheMock = mock[Cache[Request ,Response ]]
5 val cacheableStub = stubFunction [Request , Boolean]
6 cacheableStub .when (*).returns(true)
7
8 // Wire SUT with stubbed/mocked dependencies
9 val server = new ServerWithCache (cacheMock , cacheableStub )
10
11 // Arrange
12 val request = Request("Some request")
13 val expectedResponse = Response(request.body+"’s Response")
14
15 // Mock expectations
16 inSequence {
17 (cacheMock.cached _).expects(request).returning(false)
18 (cacheMock.put _).expects(request , *)
19 (cacheMock.cached _).expects(request).returning(true)
20 (cacheMock.get _).expects(request).returning( expectedResponse )
21 }
22
23 // Act + Assert
24 server.serve(request) shouldEqual expectedResponse
25 server.serve(request) shouldEqual expectedResponse
26 cacheableStub .verify(request).twice () // (cf., test spy)
27 }
28 }
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 30 / 47
Outline
1 Quick recap
2 sbt
3 Panorama on testing
4 Basics of testing in Scala with ScalaTest
5 Test doubles
6 More on testing
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 31 / 47
On effective unit testing
Key point: (software) quality should also be applied to testing code
3 key properties: readability, maintainability, trustworthiness
Good unit tests should
be automated and repeatable
be easy to implement
be relevant tomorrow
be runnable by anyone with ease
run quickly
be consistent in its results
have full control of the unit under test
be fully isolated
be communicative when failing
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 32 / 47
On testable design
Testability of software components
Refers to how much effort is needed for testing
Wrt a certain testing technique
Factors affecting testability include
Observability
Controllability
Number of dependencies and how they are resolved
Number of responsibilities
Principles of testable design
Modularity
SOLID (SRP, OCP, LSP, ISP, DIP)
Inversion of Control
Cf., Dependency Injection (note: not “Dependency Inversion”)
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 33 / 47
Property-based testing with ScalaCheck (i)
Property-based testing: key idea
Decoupling specification of behaviour from creation of test cases
1) You express properties that must hold for certain types of data
2) Tools automatically generate test cases trying to falsify those props
When a prop is falsified, the (minimal) counterexample is given
ScalaCheck
Key abstractions
org.scalacheck.Prop: a property, the testable unit in ScalaCheck
Kinds: universally quantified, existentially quantified, conditional, ..
Combinators to compose props, e.g., p && q, atLeastOne(p,q,r)
org.scalacheck.Gen: a generator, for producing test data
Combinators for creating custom generators
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 34 / 47
Property-based testing with ScalaCheck (ii)
libraryDependencies += "org.scalacheck" %% "scalacheck" % "1.13.4" % "test"
1 import org.scalacheck .{ Arbitrary , Prop , Properties}
2 import org.scalacheck.Prop .{ forAll , exists }
3
4 object TestOnNumbers extends Properties("Numbers") {
5 property("Sum is associative") = forAll{ (a: Int , b: Int , c: Int) =>
6 (a+b)+c == a+(b+c)
7 }
8 property("Sum is commutative") = forAll { (a: Int , b: Int) =>
9 a+b == b+a
10 }
11 property("Sum has zero as identity") = forAll { (a: Int) =>
12 a + 0 == a && 0 + a == a
13 }
14 property("Diff is not associative ") = forAll{ (a: Int , b: Int) =>
15 exists{ c:Int => (a-b)-c != a-(b-c) }
16 }
17 }
18
19 object TestOnLists extends Properties("Seqs") {
20 def genericSizeProp [A:Arbitrary ]: Prop = forAll{ (l1: Seq[A], l2: Seq[A]) =>
21 (l1++l2).size == l1.size + l2.size
22 }
23 property("Size of concatenation ") = Prop.all(
24 genericSizeProp [Int], genericSizeProp [String], genericSizeProp [( Boolean ,Double)])
25 property("Reverse") = forAll { (l1: Seq[Int], l2: Seq[Int ]) =>
26 l1.reverse.reverse == l1
27 }
28 }
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 35 / 47
ScalaTest: property-based testing support
PropSpec: for property-based test suites
Two styles for property checks:
1. ScalaTest style: mix PropertyChecks trait in
2. ScalaCheck style: mix Checkers trait in
1 class SetSpec extends PropSpec with PropertyChecks with Checkers with Matchers {
2 val examples = Table("set",BitSet.empty , HashSet.empty[Int], TreeSet.empty[Int])
3 val examplesGen = Gen.oneOf(BitSet.empty , HashSet.empty[Int], TreeSet.empty[Int])
4
5 property("an empty Set should have size 0") {
6 // ScalaTest ’s property -based testing style
7 forAll(examples) { set => set.size should be (0) }
8 // ScalaCheck ’s property -based testing style
9 check { Prop.forAll( examplesGen ) { set => set.size == 0 } }
10 }
11
12 property("invoking head on an empty set should produce NoSuchElementException ") {
13 // ScalaTest ’s property -based testing style
14 forAll(examples) { set => a [ NoSuchElementException ] should be thrownBy{set.head} }
15 // ScalaCheck ’s property -based testing style
16 check { Prop.forAll( examplesGen ) { set =>
17 Prop.throws(classOf[ NoSuchElementException ])(set.head) }
18 }
19 }
20 }
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 36 / 47
Behaviour-Driven Development (BDD) with Cucumber (i)
Motivation
Remember: acceptance testing = are we building the right thing?
Agile approach: iterative, incremental development and frequent
interactions with stakeholders to get feedback
But requirements need to be communicated, different stakeholders
may use their particular jargon, ..
Behaviour-Driven Development (BDD)
⇒ Idea: collaborative specification of acceptance tests
≈ Acceptance Test-Driven Dev. (ATDD) + customer-orientation
Acceptance tests as executable specs and a living documentation
Pros: ubiquitous language (cf., DDD) + “definitive source of truth”
Acceptance tests specified as examples of the way we want the
system to behave in particular scenarios
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 37 / 47
Behaviour-Driven Development (BDD) with Cucumber (ii)
Cucumber: basics
Features are high-level specs expressed by a user-perspective in a
English-like DSL called Gherkin
Features consist of multiple scenarios, representing concrete
examples/use cases
Scenarios are structured into multiple steps (given, when, then)
sbt configuration
1 val commons = Seq( scalaVersion := "2.11.8")
2 val cuke_runner = "com.waioeka.sbt" %% "cucumber -runner" % "0.0.8"
3 val cuke_scala = "info.cukes" %% "cucumber -scala" % "1.2.4" % Test
4 val cuke_junit = "info.cukes" % "cucumber -junit" % "1.2.4" % Test
5
6 lazy val root = (project in file(".")).settings(commons).settings(
7 name := "hello -scala -cucumber",
8 libraryDependencies ++= Seq(cuke_runner ,cuke_scala ,cuke_junit))
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 38 / 47
Behaviour-Driven Development (BDD) with Cucumber (iii)
src/test/resources/features/MyFeature.feature
1 Feature: Numbers
2
3 Scenario: I want to multiply two nums
4 Given one operand 7
5 And another operand 8
6 When I multiply them together
7 Then I should obtain result 56
src/test/scala/RunCucumberTests.scala
1 @RunWith(classOf[Cucumber ])
2 @CucumberOptions (features = Array("classpath:features/MyFeature.feature"),
3 tags = Array("~@Wip"), glue = Array("classpath:steps"),
4 plugin = Array("pretty", "html:target/cucumber/html"))
5 class RunCucumberTests
src/test/scala/steps/CucumberSteps.scala
1 class CucumberSteps extends ScalaDsl with EN {
2 var (a, b, sum) = (0, 0, 0)
3
4 Given("""^one operand (d+)$"""){ a = (_:Int) }
5 Given("""^another operand (d+)$"""){ b = (_:Int) }
6 When("""^I multiply them together$"""){ sum = a*b }
7 Then("""^I should obtain result (d+)$"""){ (expectedSum :Int) =>
8 Assert. assertEquals (expectedSum , sum)
9 }
10 }
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 39 / 47
ScalaTest: acceptance testing style
FeatureSpec: for acceptance testing suites
1 class TVSet {
2 private var on: Boolean = false
3 def isOn: Boolean = on
4 def pressPowerButton () { on = !on }
5 }
6
7 class TVSetSpec extends FeatureSpec with GivenWhenThen {
8 info("As a TV set owner")
9 info("I want to be able to turn the TV on and off")
10 info("So I can watch TV when I want")
11 info("And save energy when I’m not watching TV")
12
13 feature("TV power button") {
14 scenario("User presses power button when TV is off") {
15 Given("a TV set that is switched off")
16 val tv = new TVSet
17 assert (!tv.isOn)
18
19 When("the power button is pressed")
20 tv. pressPowerButton ()
21
22 Then("the TV should switch on")
23 assert(tv.isOn)
24 }
25 }
26
27 scenario("User presses power button when TV is on") { ??? }
28 }
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 40 / 47
Web application testing with Selenium
Testing web applications
Selenium is a tool that provides support for browser-based tests
ScalaTest includes a DSL to drive Selenium tests
Ref.: http://www.scalatest.org/user_guide/using_selenium
Sbt dep: "org.seleniumhq.selenium" % "selenium-java" % "2.35.0" % "test"
1 class GoogletestSpec extends FlatSpec with Matchers with HtmlUnit {
2 "Google search" should "work" in {
3 go to "http :// www.google.com"
4 pageTitle should be ("Google")
5 executeScript ("return document.title") shouldEqual pageTitle
6
7 click on "q" // By name of the field
8 textField("q").value = "selenium"
9 submit ()
10
11 eventually(timeout (2 seconds)) {
12 pageTitle should startWith ("selenium - ")
13 }
14 }
15 }R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 41 / 47
Testing GUI-based programs
Key question
What does it mean to test a GUI-based program?
Answer “what should I test” before “how”
Approaches
1. Test monkey
2. Test monkey + logging
3. Mouse/Keyboard capture & replay tools
Scripts
4. Programmatic access to GUI
5. Image recognition to identify/control GUI components
E.g.: SikuliXa
a
http://sikulix.com/
https://en.wikipedia.org/wiki/Comparison_of_GUI_testing_tools
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 42 / 47
Microbenchmarking with ScalaMeter (i)
A microbenchmark is a measurement of the running time of some
isolated piece of code
Issues in doing microbenchmarking
Microbenchmark conditions/inputs are hard to reproduce
Performance measurement in general introduces observer bias
In general, runtime behaviour is nondeterministic
Inaccuracies in performance measurements can be due to
JIT-compilation, classloading, automatic memory management (GC), ..
Performance metrics (such as running time) inherently give an
incomplete picture about the performance characteristics
However, microbenchmarking may still be useful, e.g.,
for verifying that an optimisation is an optimisation at all
for comparison of different implementations wrt specific conditions
for supporting performance regression testing
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 43 / 47
Microbenchmarking with ScalaMeter (ii)
1 object Mapper {
2 def map1[T](xs: List[T])(f: T=>T): List[T] = xs.map(f)
3 def map2[T](xs: List[T])(f: T=>T): List[T] = xs.reverse.reverse.map(f)
4 }
5
6 import org.scalameter.api._ // ... other imports
7
8 object RegressionTest extends Bench. OnlineRegressionReport {
9 override val persistor: Persistor = SerializationPersistor ()
10 override val reporter: Reporter[Double] = Reporter.Composite(
11 new RegressionReporter (tester ,historian), HtmlReporter (! online))
12
13 val sizes = Gen.range("size")(1000 , 10000 , 3000)
14 val lists = for (sz <- sizes) yield (0 until sz).toList
15
16 performance of "List" in {
17 measure method "map" in {
18 using(lists) config (
19 exec.benchRuns -> 20, exec. independentSamples -> 1
20 ) in { xs =>
21 Mapper.map1(xs)(_+1)
22 }
23 }
24 }
25 }
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 44 / 47
References and further readings
Effective unit testing
Roy Osherove. The Art of Unit Testing: With Examples in .Net. 1st. Greenwich, CT,
USA: Manning Publications Co., 2009. isbn: 1933988274, 9781933988276
L. Koskela. Effective Unit Testing: A Guide for Java Developers. Running Series.
Manning, 2013. isbn: 9781935182573
Gerard Meszaros. XUnit Test Patterns: Refactoring Test Code. Upper Saddle River, NJ,
USA: Prentice Hall PTR, 2006. isbn: 0131495054
On specific topics
Dorothy Graham and Mark Fewster. Experiences of Test Automation: Case Studies of
Software Test Automation. 1st. Addison-Wesley Professional, 2012. isbn: 0321754069,
9780321754066
Max Guernsey III. Test-Driven Database Development: Unlocking Agility. 1st.
Addison-Wesley Professional, 2013. isbn: 032178412X, 9780321784124
S. Rose, M. Wynne, and A. Hellesoy. The Cucumber for Java Book: Behaviour-Driven
Development for Testers and Developers. Pragmatic programmers. Pragmatic Bookshelf,
2015. isbn: 9781941222294
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 45 / 47
Bibliography I
Dorothy Graham and Mark Fewster. Experiences of Test Automation: Case
Studies of Software Test Automation. 1st. Addison-Wesley Professional, 2012.
isbn: 0321754069, 9780321754066.
Max Guernsey III. Test-Driven Database Development: Unlocking Agility. 1st.
Addison-Wesley Professional, 2013. isbn: 032178412X, 9780321784124.
Laura Inozemtseva and Reid Holmes. “Coverage is not strongly correlated with
test suite effectiveness”. In: Proceedings of the 36th International Conference on
Software Engineering. ACM. 2014, pp. 435–445.
L. Koskela. Effective Unit Testing: A Guide for Java Developers. Running Series.
Manning, 2013. isbn: 9781935182573.
Brian Marick et al. “How to misuse code coverage”. In: Proceedings of the 16th
Interational Conference on Testing Computer Software. 1999, pp. 16–18.
Gerard Meszaros. XUnit Test Patterns: Refactoring Test Code. Upper Saddle
River, NJ, USA: Prentice Hall PTR, 2006. isbn: 0131495054.
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 46 / 47
Bibliography II
Roy Osherove. The Art of Unit Testing: With Examples in .Net. 1st. Greenwich,
CT, USA: Manning Publications Co., 2009. isbn: 1933988274, 9781933988276.
S. Rose, M. Wynne, and A. Hellesoy. The Cucumber for Java Book:
Behaviour-Driven Development for Testers and Developers. Pragmatic
programmers. Pragmatic Bookshelf, 2015. isbn: 9781941222294.
Qian Yang, J Jenny Li, and David M Weiss. “A survey of coverage-based testing
tools”. In: The Computer Journal 52.5 (2009), pp. 589–597.
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 47 / 47

Contenu connexe

Tendances

Hrishikesh_iitg_internship_report
Hrishikesh_iitg_internship_reportHrishikesh_iitg_internship_report
Hrishikesh_iitg_internship_reportHrishikesh Malakar
 
Assessing Unit Test Quality
Assessing Unit Test QualityAssessing Unit Test Quality
Assessing Unit Test Qualityguest268ee8
 
Unit Testing And Mocking
Unit Testing And MockingUnit Testing And Mocking
Unit Testing And MockingJoe Wilson
 
Test Driven Development - The art of fearless programming
Test Driven Development - The art of fearless programmingTest Driven Development - The art of fearless programming
Test Driven Development - The art of fearless programmingChamil Jeewantha
 
Test Driven Development
Test Driven DevelopmentTest Driven Development
Test Driven DevelopmentDhaval Dalal
 
Mutation Testing Workshop at Ericsson, Kista, Sweden
Mutation Testing Workshop at Ericsson, Kista, SwedenMutation Testing Workshop at Ericsson, Kista, Sweden
Mutation Testing Workshop at Ericsson, Kista, SwedenSTAMP Project
 
Test Driven Development
Test Driven DevelopmentTest Driven Development
Test Driven DevelopmentMireia Sangalo
 
Introduction to TDD (Test Driven development) - Ahmed Shreef
Introduction to TDD (Test Driven development) - Ahmed ShreefIntroduction to TDD (Test Driven development) - Ahmed Shreef
Introduction to TDD (Test Driven development) - Ahmed ShreefAhmed Shreef
 
Software testing quiz questions and answers
Software testing quiz questions and answersSoftware testing quiz questions and answers
Software testing quiz questions and answersRajendraG
 
Test Driven Development
Test Driven DevelopmentTest Driven Development
Test Driven Developmentguestc8093a6
 
TDD And Refactoring
TDD And RefactoringTDD And Refactoring
TDD And RefactoringNaresh Jain
 
A Not-So-Serious Introduction to Test Driven Development (TDD)
A Not-So-Serious Introduction to Test Driven Development (TDD) A Not-So-Serious Introduction to Test Driven Development (TDD)
A Not-So-Serious Introduction to Test Driven Development (TDD) CodeOps Technologies LLP
 
What's software testing
What's software testingWhat's software testing
What's software testingLi-Wei Cheng
 
TDD Flow: The Mantra in Action
TDD Flow: The Mantra in ActionTDD Flow: The Mantra in Action
TDD Flow: The Mantra in ActionDionatan default
 
Unit testing in android
Unit testing in androidUnit testing in android
Unit testing in androidLi-Wei Cheng
 
Software testing concepts
Software testing conceptsSoftware testing concepts
Software testing conceptssatyatwrmca
 

Tendances (18)

Hrishikesh_iitg_internship_report
Hrishikesh_iitg_internship_reportHrishikesh_iitg_internship_report
Hrishikesh_iitg_internship_report
 
Assessing Unit Test Quality
Assessing Unit Test QualityAssessing Unit Test Quality
Assessing Unit Test Quality
 
Unit Testing And Mocking
Unit Testing And MockingUnit Testing And Mocking
Unit Testing And Mocking
 
Test Driven Development - The art of fearless programming
Test Driven Development - The art of fearless programmingTest Driven Development - The art of fearless programming
Test Driven Development - The art of fearless programming
 
Test Driven Development
Test Driven DevelopmentTest Driven Development
Test Driven Development
 
Mutation Testing Workshop at Ericsson, Kista, Sweden
Mutation Testing Workshop at Ericsson, Kista, SwedenMutation Testing Workshop at Ericsson, Kista, Sweden
Mutation Testing Workshop at Ericsson, Kista, Sweden
 
Test Driven Development
Test Driven DevelopmentTest Driven Development
Test Driven Development
 
Introduction to TDD (Test Driven development) - Ahmed Shreef
Introduction to TDD (Test Driven development) - Ahmed ShreefIntroduction to TDD (Test Driven development) - Ahmed Shreef
Introduction to TDD (Test Driven development) - Ahmed Shreef
 
Test Driven Development
Test Driven DevelopmentTest Driven Development
Test Driven Development
 
Software testing quiz questions and answers
Software testing quiz questions and answersSoftware testing quiz questions and answers
Software testing quiz questions and answers
 
Test Driven Development
Test Driven DevelopmentTest Driven Development
Test Driven Development
 
TDD - Agile
TDD - Agile TDD - Agile
TDD - Agile
 
TDD And Refactoring
TDD And RefactoringTDD And Refactoring
TDD And Refactoring
 
A Not-So-Serious Introduction to Test Driven Development (TDD)
A Not-So-Serious Introduction to Test Driven Development (TDD) A Not-So-Serious Introduction to Test Driven Development (TDD)
A Not-So-Serious Introduction to Test Driven Development (TDD)
 
What's software testing
What's software testingWhat's software testing
What's software testing
 
TDD Flow: The Mantra in Action
TDD Flow: The Mantra in ActionTDD Flow: The Mantra in Action
TDD Flow: The Mantra in Action
 
Unit testing in android
Unit testing in androidUnit testing in android
Unit testing in android
 
Software testing concepts
Software testing conceptsSoftware testing concepts
Software testing concepts
 

Similaire à Software Testing (in Scala): A Practitioner's Survey (Quickly)

Test automation with the Gauge framework: Experience and best practices -- SE...
Test automation with the Gauge framework: Experience and best practices -- SE...Test automation with the Gauge framework: Experience and best practices -- SE...
Test automation with the Gauge framework: Experience and best practices -- SE...Vahid Garousi
 
Testware Hierarchy for Test Automation
Testware Hierarchy for Test AutomationTestware Hierarchy for Test Automation
Testware Hierarchy for Test AutomationGregory Solovey
 
Testing: an Introduction and Panorama
Testing: an Introduction and PanoramaTesting: an Introduction and Panorama
Testing: an Introduction and PanoramaRoberto Casadei
 
Towards Automated Engineering for Collective Adaptive Systems: Vision and Res...
Towards Automated Engineering for Collective Adaptive Systems: Vision and Res...Towards Automated Engineering for Collective Adaptive Systems: Vision and Res...
Towards Automated Engineering for Collective Adaptive Systems: Vision and Res...Roberto Casadei
 
We continue checking Microsoft projects: analysis of PowerShell
We continue checking Microsoft projects: analysis of PowerShellWe continue checking Microsoft projects: analysis of PowerShell
We continue checking Microsoft projects: analysis of PowerShellPVS-Studio
 
Software Testing Tecniques
Software Testing TecniquesSoftware Testing Tecniques
Software Testing Tecniquesersanbilik
 
Testing in Craft CMS
Testing in Craft CMSTesting in Craft CMS
Testing in Craft CMSJustinHolt20
 
Nguyenvandungb seminar
Nguyenvandungb seminarNguyenvandungb seminar
Nguyenvandungb seminardunglinh111
 
Cinfony - Bring cheminformatics toolkits into tune
Cinfony - Bring cheminformatics toolkits into tuneCinfony - Bring cheminformatics toolkits into tune
Cinfony - Bring cheminformatics toolkits into tunebaoilleach
 
Szczepan.faber.gradle
Szczepan.faber.gradleSzczepan.faber.gradle
Szczepan.faber.gradlemagda3695
 
Role+Of+Testing+In+Sdlc
Role+Of+Testing+In+SdlcRole+Of+Testing+In+Sdlc
Role+Of+Testing+In+Sdlcmahendra singh
 
Software Architecture - Allocation taxonomies: building, deployment and distr...
Software Architecture - Allocation taxonomies: building, deployment and distr...Software Architecture - Allocation taxonomies: building, deployment and distr...
Software Architecture - Allocation taxonomies: building, deployment and distr...Jose Emilio Labra Gayo
 
Unit Testing Fundamentals
Unit Testing FundamentalsUnit Testing Fundamentals
Unit Testing FundamentalsRichard Paul
 
The_Little_Jenkinsfile_That_Could
The_Little_Jenkinsfile_That_CouldThe_Little_Jenkinsfile_That_Could
The_Little_Jenkinsfile_That_CouldShelley Lambert
 
Stopping the Rot - Putting Legacy C++ Under Test
Stopping the Rot - Putting Legacy C++ Under TestStopping the Rot - Putting Legacy C++ Under Test
Stopping the Rot - Putting Legacy C++ Under TestSeb Rose
 
Introduction to the CII Badge Programe, OW2con'16, Paris.
Introduction to the CII Badge Programe, OW2con'16, Paris. Introduction to the CII Badge Programe, OW2con'16, Paris.
Introduction to the CII Badge Programe, OW2con'16, Paris. OW2
 
manual-testing
manual-testingmanual-testing
manual-testingKanak Mane
 

Similaire à Software Testing (in Scala): A Practitioner's Survey (Quickly) (20)

Test automation with the Gauge framework: Experience and best practices -- SE...
Test automation with the Gauge framework: Experience and best practices -- SE...Test automation with the Gauge framework: Experience and best practices -- SE...
Test automation with the Gauge framework: Experience and best practices -- SE...
 
Gallio Crafting A Toolchain
Gallio Crafting A ToolchainGallio Crafting A Toolchain
Gallio Crafting A Toolchain
 
Testware Hierarchy for Test Automation
Testware Hierarchy for Test AutomationTestware Hierarchy for Test Automation
Testware Hierarchy for Test Automation
 
Testing: an Introduction and Panorama
Testing: an Introduction and PanoramaTesting: an Introduction and Panorama
Testing: an Introduction and Panorama
 
Towards Automated Engineering for Collective Adaptive Systems: Vision and Res...
Towards Automated Engineering for Collective Adaptive Systems: Vision and Res...Towards Automated Engineering for Collective Adaptive Systems: Vision and Res...
Towards Automated Engineering for Collective Adaptive Systems: Vision and Res...
 
We continue checking Microsoft projects: analysis of PowerShell
We continue checking Microsoft projects: analysis of PowerShellWe continue checking Microsoft projects: analysis of PowerShell
We continue checking Microsoft projects: analysis of PowerShell
 
Software Testing Tecniques
Software Testing TecniquesSoftware Testing Tecniques
Software Testing Tecniques
 
Testing in Craft CMS
Testing in Craft CMSTesting in Craft CMS
Testing in Craft CMS
 
Nguyenvandungb seminar
Nguyenvandungb seminarNguyenvandungb seminar
Nguyenvandungb seminar
 
Cinfony - Bring cheminformatics toolkits into tune
Cinfony - Bring cheminformatics toolkits into tuneCinfony - Bring cheminformatics toolkits into tune
Cinfony - Bring cheminformatics toolkits into tune
 
Szczepan.faber.gradle
Szczepan.faber.gradleSzczepan.faber.gradle
Szczepan.faber.gradle
 
The Test way
The Test wayThe Test way
The Test way
 
Role+Of+Testing+In+Sdlc
Role+Of+Testing+In+SdlcRole+Of+Testing+In+Sdlc
Role+Of+Testing+In+Sdlc
 
Software Architecture - Allocation taxonomies: building, deployment and distr...
Software Architecture - Allocation taxonomies: building, deployment and distr...Software Architecture - Allocation taxonomies: building, deployment and distr...
Software Architecture - Allocation taxonomies: building, deployment and distr...
 
Unit Testing Fundamentals
Unit Testing FundamentalsUnit Testing Fundamentals
Unit Testing Fundamentals
 
The_Little_Jenkinsfile_That_Could
The_Little_Jenkinsfile_That_CouldThe_Little_Jenkinsfile_That_Could
The_Little_Jenkinsfile_That_Could
 
Stopping the Rot - Putting Legacy C++ Under Test
Stopping the Rot - Putting Legacy C++ Under TestStopping the Rot - Putting Legacy C++ Under Test
Stopping the Rot - Putting Legacy C++ Under Test
 
Testing Throughout the Software Life Cycle - Section 2
Testing Throughout the Software Life Cycle - Section 2Testing Throughout the Software Life Cycle - Section 2
Testing Throughout the Software Life Cycle - Section 2
 
Introduction to the CII Badge Programe, OW2con'16, Paris.
Introduction to the CII Badge Programe, OW2con'16, Paris. Introduction to the CII Badge Programe, OW2con'16, Paris.
Introduction to the CII Badge Programe, OW2con'16, Paris.
 
manual-testing
manual-testingmanual-testing
manual-testing
 

Plus de Roberto Casadei

Programming (and Learning) Self-Adaptive & Self-Organising Behaviour with Sca...
Programming (and Learning) Self-Adaptive & Self-Organising Behaviour with Sca...Programming (and Learning) Self-Adaptive & Self-Organising Behaviour with Sca...
Programming (and Learning) Self-Adaptive & Self-Organising Behaviour with Sca...Roberto Casadei
 
A Presentation of My Research Activity
A Presentation of My Research ActivityA Presentation of My Research Activity
A Presentation of My Research ActivityRoberto Casadei
 
Self-Organisation Programming: a Functional Reactive Macro Approach (FRASP) [...
Self-Organisation Programming: a Functional Reactive Macro Approach (FRASP) [...Self-Organisation Programming: a Functional Reactive Macro Approach (FRASP) [...
Self-Organisation Programming: a Functional Reactive Macro Approach (FRASP) [...Roberto Casadei
 
Programming Distributed Collective Processes for Dynamic Ensembles and Collec...
Programming Distributed Collective Processes for Dynamic Ensembles and Collec...Programming Distributed Collective Processes for Dynamic Ensembles and Collec...
Programming Distributed Collective Processes for Dynamic Ensembles and Collec...Roberto Casadei
 
Aggregate Computing Research: an Overview
Aggregate Computing Research: an OverviewAggregate Computing Research: an Overview
Aggregate Computing Research: an OverviewRoberto Casadei
 
Introduction to the 1st DISCOLI workshop on distributed collective intelligence
Introduction to the 1st DISCOLI workshop on distributed collective intelligenceIntroduction to the 1st DISCOLI workshop on distributed collective intelligence
Introduction to the 1st DISCOLI workshop on distributed collective intelligenceRoberto Casadei
 
Digital Twins, Virtual Devices, and Augmentations for Self-Organising Cyber-P...
Digital Twins, Virtual Devices, and Augmentations for Self-Organising Cyber-P...Digital Twins, Virtual Devices, and Augmentations for Self-Organising Cyber-P...
Digital Twins, Virtual Devices, and Augmentations for Self-Organising Cyber-P...Roberto Casadei
 
FScaFi: A Core Calculus for Collective Adaptive Systems Programming
FScaFi: A Core Calculus for Collective Adaptive Systems ProgrammingFScaFi: A Core Calculus for Collective Adaptive Systems Programming
FScaFi: A Core Calculus for Collective Adaptive Systems ProgrammingRoberto Casadei
 
6th eCAS workshop on Engineering Collective Adaptive Systems
6th eCAS workshop on Engineering Collective Adaptive Systems6th eCAS workshop on Engineering Collective Adaptive Systems
6th eCAS workshop on Engineering Collective Adaptive SystemsRoberto Casadei
 
Augmented Collective Digital Twins for Self-Organising Cyber-Physical Systems
Augmented Collective Digital Twins for Self-Organising Cyber-Physical SystemsAugmented Collective Digital Twins for Self-Organising Cyber-Physical Systems
Augmented Collective Digital Twins for Self-Organising Cyber-Physical SystemsRoberto Casadei
 
Tuple-Based Coordination in Large-Scale Situated Systems
Tuple-Based Coordination in Large-Scale Situated SystemsTuple-Based Coordination in Large-Scale Situated Systems
Tuple-Based Coordination in Large-Scale Situated SystemsRoberto Casadei
 
Pulverisation in Cyber-Physical Systems: Engineering the Self-Organising Logi...
Pulverisation in Cyber-Physical Systems: Engineering the Self-Organising Logi...Pulverisation in Cyber-Physical Systems: Engineering the Self-Organising Logi...
Pulverisation in Cyber-Physical Systems: Engineering the Self-Organising Logi...Roberto Casadei
 
Collective Adaptive Systems as Coordination Media: The Case of Tuples in Spac...
Collective Adaptive Systems as Coordination Media: The Case of Tuples in Spac...Collective Adaptive Systems as Coordination Media: The Case of Tuples in Spac...
Collective Adaptive Systems as Coordination Media: The Case of Tuples in Spac...Roberto Casadei
 
On Context-Orientation in Aggregate Programming
On Context-Orientation in Aggregate ProgrammingOn Context-Orientation in Aggregate Programming
On Context-Orientation in Aggregate ProgrammingRoberto Casadei
 
Engineering Resilient Collaborative Edge-enabled IoT
Engineering Resilient Collaborative Edge-enabled IoTEngineering Resilient Collaborative Edge-enabled IoT
Engineering Resilient Collaborative Edge-enabled IoTRoberto Casadei
 
Aggregate Processes in Field Calculus
Aggregate Processes in Field CalculusAggregate Processes in Field Calculus
Aggregate Processes in Field CalculusRoberto Casadei
 
AWS and Serverless Computing
AWS and Serverless ComputingAWS and Serverless Computing
AWS and Serverless ComputingRoberto Casadei
 
Coordinating Computation at the Edge: a Decentralized, Self-organizing, Spati...
Coordinating Computation at the Edge: a Decentralized, Self-organizing, Spati...Coordinating Computation at the Edge: a Decentralized, Self-organizing, Spati...
Coordinating Computation at the Edge: a Decentralized, Self-organizing, Spati...Roberto Casadei
 
The Rust Programming Language: an Overview
The Rust Programming Language: an OverviewThe Rust Programming Language: an Overview
The Rust Programming Language: an OverviewRoberto Casadei
 
Akka Remoting and Clustering: an Introduction
Akka Remoting and Clustering: an IntroductionAkka Remoting and Clustering: an Introduction
Akka Remoting and Clustering: an IntroductionRoberto Casadei
 

Plus de Roberto Casadei (20)

Programming (and Learning) Self-Adaptive & Self-Organising Behaviour with Sca...
Programming (and Learning) Self-Adaptive & Self-Organising Behaviour with Sca...Programming (and Learning) Self-Adaptive & Self-Organising Behaviour with Sca...
Programming (and Learning) Self-Adaptive & Self-Organising Behaviour with Sca...
 
A Presentation of My Research Activity
A Presentation of My Research ActivityA Presentation of My Research Activity
A Presentation of My Research Activity
 
Self-Organisation Programming: a Functional Reactive Macro Approach (FRASP) [...
Self-Organisation Programming: a Functional Reactive Macro Approach (FRASP) [...Self-Organisation Programming: a Functional Reactive Macro Approach (FRASP) [...
Self-Organisation Programming: a Functional Reactive Macro Approach (FRASP) [...
 
Programming Distributed Collective Processes for Dynamic Ensembles and Collec...
Programming Distributed Collective Processes for Dynamic Ensembles and Collec...Programming Distributed Collective Processes for Dynamic Ensembles and Collec...
Programming Distributed Collective Processes for Dynamic Ensembles and Collec...
 
Aggregate Computing Research: an Overview
Aggregate Computing Research: an OverviewAggregate Computing Research: an Overview
Aggregate Computing Research: an Overview
 
Introduction to the 1st DISCOLI workshop on distributed collective intelligence
Introduction to the 1st DISCOLI workshop on distributed collective intelligenceIntroduction to the 1st DISCOLI workshop on distributed collective intelligence
Introduction to the 1st DISCOLI workshop on distributed collective intelligence
 
Digital Twins, Virtual Devices, and Augmentations for Self-Organising Cyber-P...
Digital Twins, Virtual Devices, and Augmentations for Self-Organising Cyber-P...Digital Twins, Virtual Devices, and Augmentations for Self-Organising Cyber-P...
Digital Twins, Virtual Devices, and Augmentations for Self-Organising Cyber-P...
 
FScaFi: A Core Calculus for Collective Adaptive Systems Programming
FScaFi: A Core Calculus for Collective Adaptive Systems ProgrammingFScaFi: A Core Calculus for Collective Adaptive Systems Programming
FScaFi: A Core Calculus for Collective Adaptive Systems Programming
 
6th eCAS workshop on Engineering Collective Adaptive Systems
6th eCAS workshop on Engineering Collective Adaptive Systems6th eCAS workshop on Engineering Collective Adaptive Systems
6th eCAS workshop on Engineering Collective Adaptive Systems
 
Augmented Collective Digital Twins for Self-Organising Cyber-Physical Systems
Augmented Collective Digital Twins for Self-Organising Cyber-Physical SystemsAugmented Collective Digital Twins for Self-Organising Cyber-Physical Systems
Augmented Collective Digital Twins for Self-Organising Cyber-Physical Systems
 
Tuple-Based Coordination in Large-Scale Situated Systems
Tuple-Based Coordination in Large-Scale Situated SystemsTuple-Based Coordination in Large-Scale Situated Systems
Tuple-Based Coordination in Large-Scale Situated Systems
 
Pulverisation in Cyber-Physical Systems: Engineering the Self-Organising Logi...
Pulverisation in Cyber-Physical Systems: Engineering the Self-Organising Logi...Pulverisation in Cyber-Physical Systems: Engineering the Self-Organising Logi...
Pulverisation in Cyber-Physical Systems: Engineering the Self-Organising Logi...
 
Collective Adaptive Systems as Coordination Media: The Case of Tuples in Spac...
Collective Adaptive Systems as Coordination Media: The Case of Tuples in Spac...Collective Adaptive Systems as Coordination Media: The Case of Tuples in Spac...
Collective Adaptive Systems as Coordination Media: The Case of Tuples in Spac...
 
On Context-Orientation in Aggregate Programming
On Context-Orientation in Aggregate ProgrammingOn Context-Orientation in Aggregate Programming
On Context-Orientation in Aggregate Programming
 
Engineering Resilient Collaborative Edge-enabled IoT
Engineering Resilient Collaborative Edge-enabled IoTEngineering Resilient Collaborative Edge-enabled IoT
Engineering Resilient Collaborative Edge-enabled IoT
 
Aggregate Processes in Field Calculus
Aggregate Processes in Field CalculusAggregate Processes in Field Calculus
Aggregate Processes in Field Calculus
 
AWS and Serverless Computing
AWS and Serverless ComputingAWS and Serverless Computing
AWS and Serverless Computing
 
Coordinating Computation at the Edge: a Decentralized, Self-organizing, Spati...
Coordinating Computation at the Edge: a Decentralized, Self-organizing, Spati...Coordinating Computation at the Edge: a Decentralized, Self-organizing, Spati...
Coordinating Computation at the Edge: a Decentralized, Self-organizing, Spati...
 
The Rust Programming Language: an Overview
The Rust Programming Language: an OverviewThe Rust Programming Language: an Overview
The Rust Programming Language: an Overview
 
Akka Remoting and Clustering: an Introduction
Akka Remoting and Clustering: an IntroductionAkka Remoting and Clustering: an Introduction
Akka Remoting and Clustering: an Introduction
 

Dernier

Deep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - DatacampDeep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - DatacampVICTOR MAESTRE RAMIREZ
 
Cybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadCybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadIvo Andreev
 
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfTobias Schneck
 
Why Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdfWhy Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdfBrain Inventory
 
JS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIJS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIIvo Andreev
 
Kawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies
 
Mastering Kubernetes - Basics and Advanced Concepts using Example Project
Mastering Kubernetes - Basics and Advanced Concepts using Example ProjectMastering Kubernetes - Basics and Advanced Concepts using Example Project
Mastering Kubernetes - Basics and Advanced Concepts using Example Projectwajrcs
 
Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsYour Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsJaydeep Chhasatia
 
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...Jaydeep Chhasatia
 
Growing Oxen: channel operators and retries
Growing Oxen: channel operators and retriesGrowing Oxen: channel operators and retries
Growing Oxen: channel operators and retriesSoftwareMill
 
20240330_고급진 코드를 위한 exception 다루기
20240330_고급진 코드를 위한 exception 다루기20240330_고급진 코드를 위한 exception 다루기
20240330_고급진 코드를 위한 exception 다루기Chiwon Song
 
eAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionseAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionsNirav Modi
 
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine HarmonyLeveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmonyelliciumsolutionspun
 
How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?AmeliaSmith90
 
Top Software Development Trends in 2024
Top Software Development Trends in  2024Top Software Development Trends in  2024
Top Software Development Trends in 2024Mind IT Systems
 
Watermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesWatermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesShyamsundar Das
 
IA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeIA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeNeo4j
 
OpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorOpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorShane Coughlan
 

Dernier (20)

Deep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - DatacampDeep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - Datacamp
 
Cybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadCybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and Bad
 
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
 
Why Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdfWhy Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdf
 
JS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIJS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AI
 
Kawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in Trivandrum
 
Mastering Kubernetes - Basics and Advanced Concepts using Example Project
Mastering Kubernetes - Basics and Advanced Concepts using Example ProjectMastering Kubernetes - Basics and Advanced Concepts using Example Project
Mastering Kubernetes - Basics and Advanced Concepts using Example Project
 
Program with GUTs
Program with GUTsProgram with GUTs
Program with GUTs
 
Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsYour Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
 
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
 
Growing Oxen: channel operators and retries
Growing Oxen: channel operators and retriesGrowing Oxen: channel operators and retries
Growing Oxen: channel operators and retries
 
20240330_고급진 코드를 위한 exception 다루기
20240330_고급진 코드를 위한 exception 다루기20240330_고급진 코드를 위한 exception 다루기
20240330_고급진 코드를 위한 exception 다루기
 
eAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionseAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspections
 
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine HarmonyLeveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
 
How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?
 
Top Software Development Trends in 2024
Top Software Development Trends in  2024Top Software Development Trends in  2024
Top Software Development Trends in 2024
 
Sustainable Web Design - Claire Thornewill
Sustainable Web Design - Claire ThornewillSustainable Web Design - Claire Thornewill
Sustainable Web Design - Claire Thornewill
 
Watermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesWatermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security Challenges
 
IA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeIA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG time
 
OpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorOpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS Calculator
 

Software Testing (in Scala): A Practitioner's Survey (Quickly)

  • 1. Software Testing Roberto Casadei {roby.casadei}@unibo.it C.D.L. Magistrale in Ingegneria e Scienze Informatiche Alma Mater Studiorum—Universit`a di Bologna, Cesena a.a. 2016/2017 R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 1 / 47
  • 2. Outline Outline Recap on unit tests in JUnit and TDD Operative introduction to “automated builds with tests and dependencies” (in sbt) Main kinds of tests: unit, integration, functional, stress, acceptance Testing framework for Scala: ScalaTest Test coverage Introduction to test doubles and mocking Some pointers to more advanced stuff on testing Effective unit testing Property-based testing (cf., QuickCheck, ScalaCheck) Cucumber for behavior-driven development Selenium for testing web apps via browser automation Testing of GUI-based programs Performance regression testing with ScalaMeter R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 2 / 47
  • 3. Outline 1 Quick recap 2 sbt 3 Panorama on testing 4 Basics of testing in Scala with ScalaTest 5 Test doubles 6 More on testing R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 3 / 47
  • 4. Recap: JUnit 1 public class DeviceImplTest { 2 private DeviceImpl device; 3 4 @Before public void init (){ 5 this.device = new DeviceImpl (); 6 } 7 8 @Test public void on() throws Exception { 9 this.device.on(); 10 assertTrue(this.device.isOn ()); 11 } 12 // .......... 13 } Concepts Test suite, test cases, tests Assertions Fixture Arrange, Act, Assert R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 4 / 47
  • 5. Recap: Test-Driven Development (TDD) What “Development” is guided (“driven”) by “tests” I.e., it is not a testing activity; tests as (useful) side-effect Exercise your code before even writing it Makes you explore the problem before attempting a solution Forces you to think as the user of your code Makes you focus on what is important right now (cf., KISS) Red – Green – Refactor cycle Benefits Specification before implementation Guides the (detailed) design process You end up with regression tests (i.e., tests to catch regressions) A regression is when something correct gets broken A safety net against regressions is fundamental for refactoring activities R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 5 / 47
  • 6. Outline 1 Quick recap 2 sbt 3 Panorama on testing 4 Basics of testing in Scala with ScalaTest 5 Test doubles 6 More on testing R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 6 / 47
  • 7. The Scala Build Tool (sbt): intro Goal: improve productivity through automation of project-related activities sbt: the “standard” build tool for Scala Belongs to the family of build tools, like Maven, Gradle and others Gradle is the one that will be covered in this course Not limited to Scala projects Can seamlessly be used for Java projects Plugins for other languages (e.g., Groovy, Clojure, C++a , Jythonb ..) a https://github.com/d40cht/sbt-cpp b https://github.com/markchadwick/jython-sbt-plugin Lightbend Activator A custom version of sbt which adds two extra commands, activator ui and activator new Supports bootstrapping applications from a catalog of templates R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 7 / 47
  • 8. The Scala Build Tool (sbt): basics Key concepts A build definition is defined in /build.sbt It specifies a set of projects Each projects is associated with a set of settings and tasks Two modes of operation 1. Batch mode: $ sbt clean compile 2. Interactive mode 1 $ sbt 2 sbt > test 3 ... 4 sbt > console R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 8 / 47
  • 9. The Scala Build Tool (sbt): basic usage Directory structure (Maven-style) 1 projectRoot/ 2 build.sbt 3 lib/ 4 src/ 5 main/resources/ 6 main/scala/ 7 main/java/ 8 test/resources/ 9 test/scala/ 10 test/java/ 11 project/ 12 build.properties 13 plugins.sbt 14 target/ build.sbt 1 lazy val root = (project in file(".")).settings( 2 name := "mytest", version := "1.0", organization := "unibo.pps", 3 scalaVersion := "2.12.2", 4 libraryDependencies += "com.novocode" % "junit -interface" % "0.10 " % "test") R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 9 / 47
  • 10. The Scala Build Tool (sbt): basic usage Basic commands help: shows help settings: lists the settings for the current project tasks: lists the tasks for the current project <settingName>: shows the value of the setting <taskName> ..args..: execute the task Common tasks clean: deletes files produced by build compile: compiles sources console: starts Scala interpreter with project classes in classpath package: produces a main artifact (e.g., a JAR) runMain <mainClass> ..args.. test: executes all tests R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 10 / 47
  • 11. Outline 1 Quick recap 2 sbt 3 Panorama on testing 4 Basics of testing in Scala with ScalaTest 5 Test doubles 6 More on testing R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 11 / 47
  • 12. Perspectives on testing1 “Schools” of testing Analytic School: testing as a rigorous task, via formal methods Motto: Programs are logical artifacts, subject to math laws Factory School: testing to measure project progress Motto: Testing must be planned/scheduled/managed Quality School: testing as a quality process Motto: A quality process for a quality product Context-Driven School: testing as a stakeholder-oriented, adaptable process Key question: which testing would be more valuable right now? Agile School: testing to facilitate change Techniques: test automation, test-driven approaches Key takeaway: perspectives on testing impact the what and how of testing 1 https://www.prismnet.com/~wazmo/papers/four_schools.pdf R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 12 / 47
  • 13. Many different kinds of testing Multiple dimensions Technology-facing vs. Business-facing testing Verification vs. validation Granularity/scope of testing Unit – integration – system-level/end-to-end/acceptance Static vs. dynamic Code-level vs. runtime Manual (human-based) vs. automated (computer-based) Examples of human testing: inspection (code reviews), walkthrough Examples of computer-based testing: static analysis, “standard” testing Structural (aka white-box) vs. functional (aka black-box) Examples of structural testing: control flow testing, path testing Examples of functional testing: equivalence partitioning, BVA R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 13 / 47
  • 14. Levels of testing Levels of testing based on granularity/scope 1. Unit testing: testing at the level of individual units (of functionality) 2. Integration testing: testing of the functionality provided by multiple integrated/interacting units 3. System testing: testing the whole system for correctness In a black-box way Non-functional requirements (e.g., efficiency, reliability, security) are typically tested at this level 4. Acceptance testing: testing the whole system against the needs and expectations of users and stakeholders R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 14 / 47
  • 15. Acceptance tests Acceptance testing Is the SW system “acceptable” (for delivery)? It’s about testing whether the SW system satisfies the acceptance criteria (e.g., as specified in the requirements) Acceptability is matter of customers, product owners, stakeholders, or users It is very much about what has to be built ⇒ Acceptance test-driven planning/development (ATDP/D) Tools: FitNesse (wiki-based), JBehave, Cucumber (see later) Acceptance testing vs. system testing System tests ⇒ build the thing right (things should behave correctly) Acceptance tests ⇒ build the right thing (validate what you want) R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 15 / 47
  • 16. Scalability testing Scalability testing: how is the ability of the system to accomodate increasing loads? Stress testing Testing the system under unusual conditions (e.g., DoS attack) Testing the upper limits (e.g., by exceeding the max load or diminishing resources) until the SW crashes Strictly related to reliability and availability requirements Load testing Testing system performance under high-load conditions Example metrics: maximum/average response time, throughput Tools: Gatlinga, The Grinderb, a http://gatling.io/ b http://grinder.sourceforge.net/ R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 16 / 47
  • 17. Code/Test coverage Coverage Several definitions/metrics (and inconsistent terminology) Many “things” can be “covered”, and at many levels Code coverage: which/how much code is executed during testing Mainly used for finding untested codea Note: 100% coverage doesn’t mean that tests cover all the scenarios Test coverage: how much of the behaviour is tested The goal is not 100% coverage → testing the “right things” a https://martinfowler.com/bliki/TestCoverage.html Further readings “Coverage is not strongly correlated with test suite effectiveness” (Inozemtseva and Holmes 2014) [3] “How to misuse code coverage” (Marick 1999) [5] “A survey of coverage-based testing tools” (Yang, Li, and Weiss 2009) [9] R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 17 / 47
  • 18. Code coverage in practice Configuration in sbt project/plugins.sbt 1 addSbtPlugin ("org.scoverage" % "sbt -scoverage" % "1.5.0") Enable coverage in build.sbt 1 coverageEnabled := true Usage 1 $ sbt 2 sbt > coverage // Enables coverage 3 sbt > test // Execute tests 4 sbt > coverageReport // Generate coverage reports 5 sbt > exit 6 7 $ open target/scala -2.11/ scoverage -report/index.html R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 18 / 47
  • 19. Outline 1 Quick recap 2 sbt 3 Panorama on testing 4 Basics of testing in Scala with ScalaTest 5 Test doubles 6 More on testing R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 19 / 47
  • 20. Testing in Scala: ScalaTest Key features Provides integration with many tools to provide a full, seamless testing experience JUnit/TestNG, ScalaCheck, JMock, ScalaMock, ... Ant, Maven, sbt Eclipse, NetBeans, IntelliJ Supports several styles of testing out of the box Recommendation: 1 style for unit testing, 1 style for acceptance testing Configuration in sbt: 1 val scalatest = "org.scalatest" % " scalatest_2 .12" % "3.0.1" % " test" 2 3 val commonsettings = Seq( scalaVersion := "2.12.2") 4 5 lazy val root = (project in file(".")). 6 settings( commonsettings ). 7 settings(name := "hello -scalatest", 8 libraryDependencies ++= Seq(scalatest)) R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 20 / 47
  • 21. ScalaTest: styles I http://www.scalatest.org/user_guide/selecting_a_style FunSuite: test functions as in classical XUnit libraries 1 class SetSuite extends FunSuite { 2 test("An empty Set should have size 0") { 3 assert(Set.empty.size == 0) 4 } 5 6 test("Invoking head on an empty Set should produce NoSuchElementException ") { 7 assertThrows [ NoSuchElementException ] { 8 Set.empty.head 9 } 10 } 11 } R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 21 / 47
  • 22. ScalaTest: styles II FlatSpec: flat specifications `a la BDD BDD (Behaviour-Driven Development) is a refinement of (A)TDD From “tests” to “behaviour specifications” Emphasis on communication Stylistic impact + tooling impact (cf., Cucumber @ next slide) 1 class SetSpec extends FlatSpec { 2 "An empty Set" should "have size 0" in { 3 assert(Set.empty.size == 0) 4 } 5 6 it should "raise NoSuchElementException when head is invoked" in { 7 assertThrows [ NoSuchElementException ] { 8 Set.empty.head 9 } 10 } 11 } R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 22 / 47
  • 23. ScalaTest: styles III FunSpec: specifications `a la BDD with (nested) describe blocks 1 class SetSpec extends FunSpec { 2 describe("A Set") { 3 describe("when empty") { 4 it("should have size 0") { 5 assert(Set.empty.size == 0) 6 } 7 8 it("should raise NoSuchElementException for head") { 9 assertThrows[ NoSuchElementException ] { 10 Set.empty.head 11 } 12 } 13 } 14 } 15 } R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 23 / 47
  • 24. ScalaTest: styles IV Many styles: recap FunSuite: for xUnit people FlatSpec: transition from xUnit to BDD-style FunSpec: for BDD people ... other variants, e.g., giving more flexibility in the BDD style, ... PropSpec: for property-based testing (see @ next slide) FeatureSpec: for acceptance BDD (see @ next slide) Guidelines for choice One main style for unit testing (e.g., FlatSpec) One main style for acceptance testing (e.g., FeatureSpec) PropSpec when property-based testing suited R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 24 / 47
  • 25. Outline 1 Quick recap 2 sbt 3 Panorama on testing 4 Basics of testing in Scala with ScalaTest 5 Test doubles 6 More on testing R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 25 / 47
  • 26. Test doubles [6]: why and what Test doubles: objects to be substituted for the real implementation for testing purposes Allow to gain full control if the context/environment of a piece of code Test doubles are essential for good test automation Allowing isolation of code under test (from its dependencies) Speeding-up test execution Making random behavior deterministic Simulating particular conditions that would be difficult to create Observing state & interaction otherwise invisible R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 26 / 47
  • 27. Kinds of test doubles Kinds of test doubles Dummy: objects passed around but never actually used; often used to fill parameter lists (e.g., as null) Stubs (“unusually short things”): simple implementations, e.g., with methods that do nothing or return default values Fake objects: have working implementations but provide a fast/simple alternative (e.g., in-memory DB) Test spies: stubs that record information about how they are called Mocks: test spies configured/prescribed to behave in a certain way under certain circumstances Tests also verify mocks are used in a certain way (white-box) References http://xunitpatterns.com/TestDouble.html https://martinfowler.com/articles/mocksArentStubs.html R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 27 / 47
  • 28. Mocking with ScalaMock I Usage 1 libraryDependencies += "org.scalamock" %% "scalamock -scalatest -support" % "3.5.0" % Test ScalaMock can be used in tests written in ScalaTest (or Specs2) In ScalaTest, mix trait MockFactory in your test class Mocking styles in ScalaMock Expectations-First style: expectations are set on mocks before exercising the System Under Test (SUT) If these expectations are not met, the test fails Record-then-Verify style: expectations are verified on stubs after the SUT has executed In ScalaMock, this is supported by stubs (1) for objects for which we have strict expectations, (2) otherwise R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 28 / 47
  • 29. Mocking with ScalaMock II SUT 1 case class Request(body: String) 2 case class Response(content: String) 3 4 class Server { 5 def processRequest (req: Request): Response = Response(req.body+ "’s Response") 6 def serve(req: Request): Response = processRequest (req) 7 } 8 9 trait Cache[K,T] { // Note: we provide no implementation class yet 10 def cached(key: K): Boolean 11 def get(key: K): T 12 def put(key: K, value: T): Unit 13 } 14 15 class ServerWithCache (private val cache: Cache[Request , Response], 16 private val cacheable: Request => Boolean) extends Server { 17 override def serve(req: Request) = if(! cacheable(req)) super.serve(req) else { 18 if (cache.cached(req)) { 19 cache.get(req) 20 } else { 21 val res = processRequest (req) 22 cache.put(req , res) 23 res 24 } 25 } 26 } R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 29 / 47
  • 30. Mocking with ScalaMock III 1 class TestWithMock extends FunSuite with MockFactory with Matchers { 2 test("Test server with cache"){ 3 // Mocks/stubs for dependencies 4 val cacheMock = mock[Cache[Request ,Response ]] 5 val cacheableStub = stubFunction [Request , Boolean] 6 cacheableStub .when (*).returns(true) 7 8 // Wire SUT with stubbed/mocked dependencies 9 val server = new ServerWithCache (cacheMock , cacheableStub ) 10 11 // Arrange 12 val request = Request("Some request") 13 val expectedResponse = Response(request.body+"’s Response") 14 15 // Mock expectations 16 inSequence { 17 (cacheMock.cached _).expects(request).returning(false) 18 (cacheMock.put _).expects(request , *) 19 (cacheMock.cached _).expects(request).returning(true) 20 (cacheMock.get _).expects(request).returning( expectedResponse ) 21 } 22 23 // Act + Assert 24 server.serve(request) shouldEqual expectedResponse 25 server.serve(request) shouldEqual expectedResponse 26 cacheableStub .verify(request).twice () // (cf., test spy) 27 } 28 } R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 30 / 47
  • 31. Outline 1 Quick recap 2 sbt 3 Panorama on testing 4 Basics of testing in Scala with ScalaTest 5 Test doubles 6 More on testing R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 31 / 47
  • 32. On effective unit testing Key point: (software) quality should also be applied to testing code 3 key properties: readability, maintainability, trustworthiness Good unit tests should be automated and repeatable be easy to implement be relevant tomorrow be runnable by anyone with ease run quickly be consistent in its results have full control of the unit under test be fully isolated be communicative when failing R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 32 / 47
  • 33. On testable design Testability of software components Refers to how much effort is needed for testing Wrt a certain testing technique Factors affecting testability include Observability Controllability Number of dependencies and how they are resolved Number of responsibilities Principles of testable design Modularity SOLID (SRP, OCP, LSP, ISP, DIP) Inversion of Control Cf., Dependency Injection (note: not “Dependency Inversion”) R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 33 / 47
  • 34. Property-based testing with ScalaCheck (i) Property-based testing: key idea Decoupling specification of behaviour from creation of test cases 1) You express properties that must hold for certain types of data 2) Tools automatically generate test cases trying to falsify those props When a prop is falsified, the (minimal) counterexample is given ScalaCheck Key abstractions org.scalacheck.Prop: a property, the testable unit in ScalaCheck Kinds: universally quantified, existentially quantified, conditional, .. Combinators to compose props, e.g., p && q, atLeastOne(p,q,r) org.scalacheck.Gen: a generator, for producing test data Combinators for creating custom generators R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 34 / 47
  • 35. Property-based testing with ScalaCheck (ii) libraryDependencies += "org.scalacheck" %% "scalacheck" % "1.13.4" % "test" 1 import org.scalacheck .{ Arbitrary , Prop , Properties} 2 import org.scalacheck.Prop .{ forAll , exists } 3 4 object TestOnNumbers extends Properties("Numbers") { 5 property("Sum is associative") = forAll{ (a: Int , b: Int , c: Int) => 6 (a+b)+c == a+(b+c) 7 } 8 property("Sum is commutative") = forAll { (a: Int , b: Int) => 9 a+b == b+a 10 } 11 property("Sum has zero as identity") = forAll { (a: Int) => 12 a + 0 == a && 0 + a == a 13 } 14 property("Diff is not associative ") = forAll{ (a: Int , b: Int) => 15 exists{ c:Int => (a-b)-c != a-(b-c) } 16 } 17 } 18 19 object TestOnLists extends Properties("Seqs") { 20 def genericSizeProp [A:Arbitrary ]: Prop = forAll{ (l1: Seq[A], l2: Seq[A]) => 21 (l1++l2).size == l1.size + l2.size 22 } 23 property("Size of concatenation ") = Prop.all( 24 genericSizeProp [Int], genericSizeProp [String], genericSizeProp [( Boolean ,Double)]) 25 property("Reverse") = forAll { (l1: Seq[Int], l2: Seq[Int ]) => 26 l1.reverse.reverse == l1 27 } 28 } R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 35 / 47
  • 36. ScalaTest: property-based testing support PropSpec: for property-based test suites Two styles for property checks: 1. ScalaTest style: mix PropertyChecks trait in 2. ScalaCheck style: mix Checkers trait in 1 class SetSpec extends PropSpec with PropertyChecks with Checkers with Matchers { 2 val examples = Table("set",BitSet.empty , HashSet.empty[Int], TreeSet.empty[Int]) 3 val examplesGen = Gen.oneOf(BitSet.empty , HashSet.empty[Int], TreeSet.empty[Int]) 4 5 property("an empty Set should have size 0") { 6 // ScalaTest ’s property -based testing style 7 forAll(examples) { set => set.size should be (0) } 8 // ScalaCheck ’s property -based testing style 9 check { Prop.forAll( examplesGen ) { set => set.size == 0 } } 10 } 11 12 property("invoking head on an empty set should produce NoSuchElementException ") { 13 // ScalaTest ’s property -based testing style 14 forAll(examples) { set => a [ NoSuchElementException ] should be thrownBy{set.head} } 15 // ScalaCheck ’s property -based testing style 16 check { Prop.forAll( examplesGen ) { set => 17 Prop.throws(classOf[ NoSuchElementException ])(set.head) } 18 } 19 } 20 } R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 36 / 47
  • 37. Behaviour-Driven Development (BDD) with Cucumber (i) Motivation Remember: acceptance testing = are we building the right thing? Agile approach: iterative, incremental development and frequent interactions with stakeholders to get feedback But requirements need to be communicated, different stakeholders may use their particular jargon, .. Behaviour-Driven Development (BDD) ⇒ Idea: collaborative specification of acceptance tests ≈ Acceptance Test-Driven Dev. (ATDD) + customer-orientation Acceptance tests as executable specs and a living documentation Pros: ubiquitous language (cf., DDD) + “definitive source of truth” Acceptance tests specified as examples of the way we want the system to behave in particular scenarios R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 37 / 47
  • 38. Behaviour-Driven Development (BDD) with Cucumber (ii) Cucumber: basics Features are high-level specs expressed by a user-perspective in a English-like DSL called Gherkin Features consist of multiple scenarios, representing concrete examples/use cases Scenarios are structured into multiple steps (given, when, then) sbt configuration 1 val commons = Seq( scalaVersion := "2.11.8") 2 val cuke_runner = "com.waioeka.sbt" %% "cucumber -runner" % "0.0.8" 3 val cuke_scala = "info.cukes" %% "cucumber -scala" % "1.2.4" % Test 4 val cuke_junit = "info.cukes" % "cucumber -junit" % "1.2.4" % Test 5 6 lazy val root = (project in file(".")).settings(commons).settings( 7 name := "hello -scala -cucumber", 8 libraryDependencies ++= Seq(cuke_runner ,cuke_scala ,cuke_junit)) R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 38 / 47
  • 39. Behaviour-Driven Development (BDD) with Cucumber (iii) src/test/resources/features/MyFeature.feature 1 Feature: Numbers 2 3 Scenario: I want to multiply two nums 4 Given one operand 7 5 And another operand 8 6 When I multiply them together 7 Then I should obtain result 56 src/test/scala/RunCucumberTests.scala 1 @RunWith(classOf[Cucumber ]) 2 @CucumberOptions (features = Array("classpath:features/MyFeature.feature"), 3 tags = Array("~@Wip"), glue = Array("classpath:steps"), 4 plugin = Array("pretty", "html:target/cucumber/html")) 5 class RunCucumberTests src/test/scala/steps/CucumberSteps.scala 1 class CucumberSteps extends ScalaDsl with EN { 2 var (a, b, sum) = (0, 0, 0) 3 4 Given("""^one operand (d+)$"""){ a = (_:Int) } 5 Given("""^another operand (d+)$"""){ b = (_:Int) } 6 When("""^I multiply them together$"""){ sum = a*b } 7 Then("""^I should obtain result (d+)$"""){ (expectedSum :Int) => 8 Assert. assertEquals (expectedSum , sum) 9 } 10 } R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 39 / 47
  • 40. ScalaTest: acceptance testing style FeatureSpec: for acceptance testing suites 1 class TVSet { 2 private var on: Boolean = false 3 def isOn: Boolean = on 4 def pressPowerButton () { on = !on } 5 } 6 7 class TVSetSpec extends FeatureSpec with GivenWhenThen { 8 info("As a TV set owner") 9 info("I want to be able to turn the TV on and off") 10 info("So I can watch TV when I want") 11 info("And save energy when I’m not watching TV") 12 13 feature("TV power button") { 14 scenario("User presses power button when TV is off") { 15 Given("a TV set that is switched off") 16 val tv = new TVSet 17 assert (!tv.isOn) 18 19 When("the power button is pressed") 20 tv. pressPowerButton () 21 22 Then("the TV should switch on") 23 assert(tv.isOn) 24 } 25 } 26 27 scenario("User presses power button when TV is on") { ??? } 28 } R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 40 / 47
  • 41. Web application testing with Selenium Testing web applications Selenium is a tool that provides support for browser-based tests ScalaTest includes a DSL to drive Selenium tests Ref.: http://www.scalatest.org/user_guide/using_selenium Sbt dep: "org.seleniumhq.selenium" % "selenium-java" % "2.35.0" % "test" 1 class GoogletestSpec extends FlatSpec with Matchers with HtmlUnit { 2 "Google search" should "work" in { 3 go to "http :// www.google.com" 4 pageTitle should be ("Google") 5 executeScript ("return document.title") shouldEqual pageTitle 6 7 click on "q" // By name of the field 8 textField("q").value = "selenium" 9 submit () 10 11 eventually(timeout (2 seconds)) { 12 pageTitle should startWith ("selenium - ") 13 } 14 } 15 }R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 41 / 47
  • 42. Testing GUI-based programs Key question What does it mean to test a GUI-based program? Answer “what should I test” before “how” Approaches 1. Test monkey 2. Test monkey + logging 3. Mouse/Keyboard capture & replay tools Scripts 4. Programmatic access to GUI 5. Image recognition to identify/control GUI components E.g.: SikuliXa a http://sikulix.com/ https://en.wikipedia.org/wiki/Comparison_of_GUI_testing_tools R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 42 / 47
  • 43. Microbenchmarking with ScalaMeter (i) A microbenchmark is a measurement of the running time of some isolated piece of code Issues in doing microbenchmarking Microbenchmark conditions/inputs are hard to reproduce Performance measurement in general introduces observer bias In general, runtime behaviour is nondeterministic Inaccuracies in performance measurements can be due to JIT-compilation, classloading, automatic memory management (GC), .. Performance metrics (such as running time) inherently give an incomplete picture about the performance characteristics However, microbenchmarking may still be useful, e.g., for verifying that an optimisation is an optimisation at all for comparison of different implementations wrt specific conditions for supporting performance regression testing R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 43 / 47
  • 44. Microbenchmarking with ScalaMeter (ii) 1 object Mapper { 2 def map1[T](xs: List[T])(f: T=>T): List[T] = xs.map(f) 3 def map2[T](xs: List[T])(f: T=>T): List[T] = xs.reverse.reverse.map(f) 4 } 5 6 import org.scalameter.api._ // ... other imports 7 8 object RegressionTest extends Bench. OnlineRegressionReport { 9 override val persistor: Persistor = SerializationPersistor () 10 override val reporter: Reporter[Double] = Reporter.Composite( 11 new RegressionReporter (tester ,historian), HtmlReporter (! online)) 12 13 val sizes = Gen.range("size")(1000 , 10000 , 3000) 14 val lists = for (sz <- sizes) yield (0 until sz).toList 15 16 performance of "List" in { 17 measure method "map" in { 18 using(lists) config ( 19 exec.benchRuns -> 20, exec. independentSamples -> 1 20 ) in { xs => 21 Mapper.map1(xs)(_+1) 22 } 23 } 24 } 25 } R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 44 / 47
  • 45. References and further readings Effective unit testing Roy Osherove. The Art of Unit Testing: With Examples in .Net. 1st. Greenwich, CT, USA: Manning Publications Co., 2009. isbn: 1933988274, 9781933988276 L. Koskela. Effective Unit Testing: A Guide for Java Developers. Running Series. Manning, 2013. isbn: 9781935182573 Gerard Meszaros. XUnit Test Patterns: Refactoring Test Code. Upper Saddle River, NJ, USA: Prentice Hall PTR, 2006. isbn: 0131495054 On specific topics Dorothy Graham and Mark Fewster. Experiences of Test Automation: Case Studies of Software Test Automation. 1st. Addison-Wesley Professional, 2012. isbn: 0321754069, 9780321754066 Max Guernsey III. Test-Driven Database Development: Unlocking Agility. 1st. Addison-Wesley Professional, 2013. isbn: 032178412X, 9780321784124 S. Rose, M. Wynne, and A. Hellesoy. The Cucumber for Java Book: Behaviour-Driven Development for Testers and Developers. Pragmatic programmers. Pragmatic Bookshelf, 2015. isbn: 9781941222294 R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 45 / 47
  • 46. Bibliography I Dorothy Graham and Mark Fewster. Experiences of Test Automation: Case Studies of Software Test Automation. 1st. Addison-Wesley Professional, 2012. isbn: 0321754069, 9780321754066. Max Guernsey III. Test-Driven Database Development: Unlocking Agility. 1st. Addison-Wesley Professional, 2013. isbn: 032178412X, 9780321784124. Laura Inozemtseva and Reid Holmes. “Coverage is not strongly correlated with test suite effectiveness”. In: Proceedings of the 36th International Conference on Software Engineering. ACM. 2014, pp. 435–445. L. Koskela. Effective Unit Testing: A Guide for Java Developers. Running Series. Manning, 2013. isbn: 9781935182573. Brian Marick et al. “How to misuse code coverage”. In: Proceedings of the 16th Interational Conference on Testing Computer Software. 1999, pp. 16–18. Gerard Meszaros. XUnit Test Patterns: Refactoring Test Code. Upper Saddle River, NJ, USA: Prentice Hall PTR, 2006. isbn: 0131495054. R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 46 / 47
  • 47. Bibliography II Roy Osherove. The Art of Unit Testing: With Examples in .Net. 1st. Greenwich, CT, USA: Manning Publications Co., 2009. isbn: 1933988274, 9781933988276. S. Rose, M. Wynne, and A. Hellesoy. The Cucumber for Java Book: Behaviour-Driven Development for Testers and Developers. Pragmatic programmers. Pragmatic Bookshelf, 2015. isbn: 9781941222294. Qian Yang, J Jenny Li, and David M Weiss. “A survey of coverage-based testing tools”. In: The Computer Journal 52.5 (2009), pp. 589–597. R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 47 / 47