Modern Testing provides concise summaries of key points from the document in 3 sentences:
The document discusses modern testing practices and frameworks for Python projects, emphasizing the importance of writing tests to ensure high quality software and outlining different types of tests like unit, integration, and acceptance tests. It presents examples of using the unittest and pytest frameworks for writing tests in Python as well as the Robot Framework for acceptance testing. The document concludes by advocating for adopting best practices like using tox for local test invocation and continuous integration to enforce testing standards.
2. The Goal of Testing
→ Produce high quality / correct software
“Program testing can be used to show
the presence of bugs, but never show
their absence!”
Edsger Dijkstra
4. What to test
● Requirements
● Design
● Interfaces
● Code / Implementation
● Documentation
● Conventions
5. Test Design / Types
V-ModelUser Requirements Acceptance Test
Requirements Verification Functional Test
System Design Validation System Integration Test
Detailed Design Subsystem Integration Tests
Software Architecture (Interfaces) Unit-Tests
Coding / Implementation
Classical Software Development Process / Governmental Standard
6. Open Source Development
The development process in Open Source Projects work differently,
but a lot of the concepts of the V-Model are reflected in other ways.
Documentation
→ Reflects Requirements, Intentions and Design
Tests reflects the requirements
⇒ TTD - Test Driven Development
7. “Testing leads to failure,
and failure leads to understanding.”
Burt Rutan
Writing tests is mandatory for good software
10. Test Case
A test case is the smallest unit of testing.
It checks for a specific response to a particular set of inputs.
11. Test Fixture
A test fixture represents the preparation needed to perform
one or more tests, and any associate cleanup actions
12. Test Suite
A test suite is a collection of test cases, test suites, or both.
It is used to aggregate tests that should be executed together.
13. Test Layer
(plone.testing / plone.app.testing)
A test layer represents the baseline for a specific test
→ Reflects Test-Level in V-Model
● Unit tests
● Integration tests
● Functional tests
● Acceptance tests
14. Test Runner
A test runner is a component which orchestrates the execution of tests
and provides the outcome to the user.
15. Test Invocation Tool
A test invocation tool is a component which provides the
required infrastructure to the test runner to execute the test sets.
16. Mock
mock is a library for testing in Python.
It allows you to replace parts of your system under test with mock objects and
make assertions about how they have been used.
→ Test Isolation
17. Writing test / What to test
● Requirements → Acceptance tests
● Design → Functional tests
● Interfaces → Integration tests
● Code / Implementation → Unit tests
● Documentation → Test if your code examples actually works
● Conventions → Test if the convention of the project is followed
(e.g. Coding Conventions)
19. Unittest
The Python unit testing framework, sometimes referred to as “PyUnit,” is a Python language version of JUnit, by Kent Beck
and Erich Gamma. JUnit is, in turn, a Java version of Kent’s Smalltalk testing framework. Each is the de facto standard unit
testing framework for its respective language.
unittest supports test automation, sharing of setup and shutdown code for tests, aggregation of tests into collections,
and independence of the tests from the reporting framework. The unittest module provides classes that make it easy to
support these qualities for a set of tests.
● Python Standard Library module
● Used in Plone for testing
● Specific BaseClasses, and assert methods necessary, setup and teardown
methods
20. Unitest example
(Plone context - Plone Training Documentation)
import unittest
class TalkIntegrationTest(unittest.TestCase):
layer = PLONECONF_SITE_INTEGRATION_TESTING
def setUp(self):
self.portal = self.layer['portal']
setRoles(self.portal, TEST_USER_ID, ['Manager'])
def test_fti(self):
fti = queryUtility(IDexterityFTI, name='talk')
self.assertTrue(fti)
def test_adding(self):
self.portal.invokeFactory('talk', 'talk')
self.assertTrue(self.portal.talk)
self.assertTrue(ITalk.providedBy(self.portal.talk))
…
suite = unittest.TestLoader().loadTestsFromTestCase(TalkIntegrationTest)
Assert Methods
Method Checks that
assertEqual(a, b) a == b
assertNotEqual(a, b) a != b
assertTrue(x) bool(x) is True
assertFalse(x) bool(x) is False
assertIs(a, b) a is b
assertIsNot(a, b) a is not b
assertIsNone(x) x is None7
assertIsNotNone(x) x is not None
assertIn(a, b) a in b
assertNotIn(a, b) a not in b
assertIsInstance(a, b) isinstance(a, b)
assertNotIsInstance(a, b) not isinstance(a, b)
21. “The pytest framework makes it easy to write small tests, yet scales to support
complex functional testing for applications and libraries.”
● De Facto Standard in the Python world
● Some magic to make writing tests simpler
● just assert
● Implicit test loader
● Plugable addon system
22. pytest example
(Plone context - RestrictedPython)
from tests import e_eval
import pytest
@pytest.mark.parametrize(*e_eval)
def test_Eq(e_eval):
assert e_eval('1 == 1') is True
@pytest.mark.parametrize(*e_eval)
def test_NotEq(e_eval):
assert e_eval('1 != 2') is True
@pytest.mark.parametrize(*e_eval)
def test_Gt(e_eval):
assert e_eval('2 > 1') is True
@pytest.mark.parametrize(*e_eval)
def test_Lt(e_eval):
assert e_eval('1 < 2')
23. unittest2 / nose / nose2
Forks of unitest that either enhance or backport functionality
Mostly outdated and not recommended anymore.
24. Robot Framework
Robot Framework is a generic test automation framework for acceptance testing
and acceptance test-driven development (ATDD).
● Focus: Web Applications
● Wrapper around Selenium
25. Robot tests example
(Plone context - Plone Training Documentation)
*** Settings ***********************************************
Resource plone/app/robotframework/selenium.robot
Resource plone/app/robotframework/keywords.robot
Library Remote ${PLONE_URL}/RobotRemote
Test Setup Open test browser
Test Teardown Close all browsers
*** Test Cases *********************************************
Scenario: As a site administrator I can add a Talk
Given a logged-in site administrator
and an add talk form
When I type 'My Talk' into the title field
and I type 'Awesome talk' into the details field
and I type 'Team Banzai' into the speakers field
and I type 'banzai@example.com' into the email field
and I submit the form
Then a talk with the title 'My Talk' has been created
Scenario: As a site administrator I can view a Talk
Given a logged-in site administrator
and a talk 'My Talk'
When I go to the talk view
Then I can see the talk title 'My Talk'
Scenario: As a visitor I can view the new talk list
When I go to the talk list view
Then I can see a talk about 'Diazo designs are great'
*** Keywords ****************************************
# --- Given ------------------------------------------
a logged-in site administrator
Enable autologin as Site Administrator
an add talk form
Go To ${PLONE_URL}/++add++talk
a talk 'My Talk'
Create content type=talk id=my-talk title=My Talk
# --- WHEN --------------------------------------------
I type '${title}' into the title field
Input Text name=form.widgets.IDublinCore.title ${title}
I type '${details}' into the details field
Select frame form-widgets-details_ifr
Input text tinymce ${details}
Unselect Frame
I type '${speaker}' into the speakers field
Input Text name=form.widgets.speaker ${speaker}
I type '${email}' into the email field
Input Text name=form.widgets.email ${email}
I submit the form
Click Button Save
I go to the talk view
Go To ${PLONE_URL}/my-talk
Wait until page contains Site Map
I go to the talk list view
Go To ${PLONE_URL}/demoview
Wait until page contains Site Map
# --- THEN ----------------------
a talk with the title '${title}' has been created
Wait until page contains Site Map
Page should contain ${title}
Page should contain Item created
I can see the talk title '${title}'
Wait until page contains Site Map
Page should contain ${title}
I can see a talk about '${topic}'
Wait until page contains Site Map
Page should contain ${topic}
26. “The first principle is that you must not
fool yourself - and you are the easiest
person to fool.”
Richard Feynman
27. Test runners
● unittest testrunner
● zope.testrunner
● pytest-testrunner
● gocept.pytestlayer
A test runners is a component which orchestrates the execution of tests and
provides the outcome to the user.
● collects tests
● presents outcome
● interact with other tools (e.g. coverage)
32. Test invocation tools
● tox - Translate the concept of continuous Integration to local development
○ De facto standard as a local test invocation tool
○ Groups environments, allow to test different Python versions support
Fantastic if you use additional helpers:
● pyenv - having multiple Python version
● git-hooks - run scripts on git commands → pre-commit hook
35. The Zen of Python - PEP20
Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
...
Lessons learned from Zope and Plone
→ we should embrace each tool that helps us to provide a fantastic products
36. My Wishes to better “Best Practices” for Plone
● Adopt tox on all packages
● Switch to a different package structure and enforce that → bobtemplates.plone
○ docs
○ src
○ tests
● detailed and enforced settings for conventions
○ .editorconf
○ setup.cfg
→ https://github.com/plone/plone_best_practices_discussion