We typically distinguish between functional and non-functional testing, which might mislead to under prioritize some important aspects of the quality of the application we are testing. In many cases when the system is not secure, performant or accessible, its functionality is affected or it’s not functional at all. In this talk, I will show techniques and tools that we use that will help you improve your ability to observe the system while you are performing functional testing in order to provide feedback about the so-called “non-functional” properties. I will also discuss how to properly prioritize the different characteristics of the system in order to focus your efforts on what is more important to your business at each moment.
3. Mabl
/ USA
#QSConf2023
+20 November 9th & 10th
2 SPEAKERS TRACKS | ONLINE + IN PERSON
International thought
leaders
Free for attendees!
Call for Sponsors is open
Call for Speakers opens soon
4. abstracta.us
80% of the testing focuses on the
system’s functionality
Non-functional testing is often left for later:
● For when the system is more stable
● For when we are more mature
● Right before the release
● After the first version, for the optimization stage
Motivation
5. abstracta.us
● Demystify non-functional testing
● Share basic concepts for performance, security and accessibility
● Share some tools to help you kick off
● Discuss if the approach would make sense in your context
Goals
10. abstracta.us
What’s your quality strategy?
Build bridges between teams
Don’t lose focus from your main goal
● Be mindful about what’s important for the
business, at each moment
Unveil Risks Sooner
12. abstracta.us
What is it?
Amount of useful work achieved by a system
● Related to
○ Velocity
○ Response times
○ Throughput
● Resources usage
○ CPU
○ Memory
○ Disk
○ Network
○ Etc.
Performance
13. abstracta.us
Goal:
● Detect performance degradations, risks or improvement
opportunities in every test cycle
How?
● Using tools to gather metrics and compare with expected
(or historical) behavior
● Using tools to perform automatic analysis
How to Incorporate Performance
into Functional Testing?
14. abstracta.us
Good to Know:
● Understand your system’s architecture, components and
their role and most relevant KPIs
● Have access to the tools your team is already using
○ Understand reports and be able to drill down
How to Incorporate Performance
into Functional Testing?
15. abstracta.us
● Test under load
● Volume of the testing database
● Analize how performance limitations can affect functionality
○ Simulate or generate resource limitations
Good to Know:
● Client side versus server side performance
Where Performance
meets Functionality
21. abstracta.us
How to Incorporate
Performance into Test
Automation?
Collect response times
● Analyze trends, assertions to identify degradations
Chrome DevTools:
● API to access performance metrics
● Assert deviations with previous versions or
comparing to an SLO
● Example using Chrome DevTools from Playwright
https://github.com/unlikelyzero/js-perf-toolkit
22. abstracta.us
How to Incorporate
Performance into Test
Automation?
From Functional Test Automation to
Performance Simulation
JMeter DSL:
● Convert Selenium scripts to JMeter
● Demo (repo)
Mabl:
● Reusing API testing to run performance tests
24. abstracta.us
What is it?
Capacity to protect system data and resources
from access, damage or non authorized uses.
Related to:
● Confidentiality
● Information integrity and availability
Security
25. abstracta.us
How to Incorporate Security
into Functional Testing?
Goal:
● Detect security vulnerabilities or risks
● Detect non compliance with standards
How?
● Using tools to scan for vulnerabilities
● Functional testing on access control features
26. abstracta.us
Where Security meets
Functionality
● Functional tests on the login, and access control (users, roles)
● Error handling (which information is exposed, i.e. stack traces)
● Input validation (SQL injection, prompt injection)
Good to Know:
● Most common attacks and vulnerabilities (SQL injection,
phishing, prompt injection, etc.)
● Applicable standards (PCI, OWASP, MASVS)
● Identify sensitive information (passwords, financial or medical
information, credit card numbers, etc)
30. abstracta.us
How to Incorporate Security
into Test Automation?
OWASP ZAP:
● Run audits during automated test runs to detect new issues or threats
● Setting OWASP ZAP as a proxy in the webdriver
33. abstracta.us
What is it?
Capacity of a system of being accessed by all the
people in an effective way, without barriers,
regardless of their abilities or disabilities and
their context.
Accessibility
34. abstracta.us
How to Incorporate Accessibility
into Functional Testing?
Goal:
● Detect accessibility issues
● Detect non compliance with standards
How?
● Using tools to perform accessibility audits
● Manually verifying accessibility criteria
35. abstracta.us
How to Incorporate Accessibility
into Functional Testing?
Good to Know:
● Applicable standards (WCAG, ADA, etc.)
● Understand how support tools work and alternative uses of the
system in accessibility mode (only keyboard, screen readers,
magnifiers, etc)
● Understand how a system must be adjusted in order to effectively use
a support tool (ARIA labels, alternative text, tabulation order, etc)
36. abstracta.us
● Power users can be more efficient
● Usability improves accessibility and vice versa
○ Check usability, use Nielsen Heuristics
Good to Know:
● Applicable standards (WCAG, ADA, etc.)
● Support tools and accessibility mode (only keyboard, screen
readers, magnifiers, etc)
● What’s needed to make them work well (ARIA labels,
alternative text, tabulation order, etc.)
Where Accessibility meets
Functionality
41. abstracta.us
● Keyboard only
● Screen reader
“Accessibility is not about compliance with a
standard or adding ARIA labels, it’s about
removing barriers.”
Accessibility Tools:
42. abstracta.us
How to Incorporate Accessibility
into Test Automation?
● axe DevTools API:
API to include accessibility audits to your scripts
● Mabl:
Low-code test automation tool, includes accessibility checks in runtime
45. abstracta.us
● Extra-functional testing is not rocket science
○ Learn the basic concepts
○ Get familiar with the tools (Lighthouse, Apptim, ZAP, axe, etc.)
○ Share learnings with your team
● Integrate it to your functional testing and test automation
○ Start simple
○ Filter what’s important
● Understand business priorities
● Report considering priorities
● Connect, build bridges
Extra-Functional Testing,
Final Thoughts:
46. abstracta.us
Food for Thought
Guides
● Continuous Testing Guide
Blogposts
● From Functional Tester to Performance Tester
● What is Observability and How is it Relevant to Software Testing
● An End-to-End Guide of Load Testing
● How to Make Testing Progress Visible
● How to Take the Security of your Mobile Apps to the Next Level of OWASP
Quality Sense Podcasts
● Leandro Melendez - Performance Testing Explained Simple
● Sofia Palamarchuk - Mobile App Performance
47. abstracta.us
Let’s discuss if the
approach makes sense in
your context
Federico Toledo
federico@abstracta.us
¡Gracias!
Extra-functional Testing:
Improve How you Observe the System During Functional Testing