Best Practices for Shifting Left Performance and Accessibility Testing

Perfecto by Perforce © 2020 Perforce Software, Inc.
Best Practices for Shifting Left Performance
and Accessibility Testing

perfecto.io2 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Today’s Speakers
Dylan BarrellEran Kinsbruner Tzvika Shahaf Henrik Rexed
• Chief Product Officer at Deque
• 20+ years in technology development,
sales, product management, product
marketing, and corporate development
• MBA from University of Michigan, BS
from University of Witwatersrand
• dylanbarrell
• Chief Evangelist and author at
Perfecto
• Blogger and speaker
• 20+ years in development & testing
• Author of “The Digital Quality
Handbook” and “Continuous
Testing for DevOps Professionals”
• ek121268
• Performance Engineer at Neotys
• Extensive background and experience in
software performance
• Founded Performance CoE in major
enterprises
• henrik-rexed-a85a8315
• @hrexed
• VP Product Management at Perfecto
• Software professional with over 12 years
in product management, development,
and testing
• Expert in implementing and advising on
test strategies for Fortune 500
enterprises and leading tech companies
• http://linkedin.com/in/tzvikashahaf/
• @Tzvika_Shahaf

Today’s Agenda
1
2
3
Setting the Stage: Agile Testing Manifesto
Definitions: Shift Left Testing, Performance Testing, Accessibility Testing
5
Fundamentals of Performance Testing for Mobile and Web Apps
Best Practices for Embedding Performance & Accessibility Testing in the Build Cycle
4 Fundamentals of Accessibility Testing for Mobile and Web Apps
6 Demos
7 Q&A

Digital Engagement Reality – More Relevant Than Ever

The Agile Testing Manifesto
THE DEVOPS MANIFESTO
✔ Continuous testing over testing at the end.
✔ Embracing all testing activities over only automated functional testing.
✔ Testing what gives value over testing everything.
✔ Testing across the team over testing in siloed testing departments.
✔ Product coverage over code coverage.

Valuable Test Automation to Optimize the Delivery Pipeline
vs
OPTIMIZED
DELIVERY PIPELINE Innovation
Throughput
Quality of
Output
Time Cost

Balancing Test Automation Coverage

• Accessibility is the practice of making your apps (mobile/web) usable by as many people as
possible. We traditionally think of this as being about people with disabilities, but the practice
of making sites accessible also benefits other groups — such as those with slow network
connections, etc.
• Apps need to be accessible for everyone, including those that have vision disabilities, hearing
disabilities, or otherwise impaired users (physical, cognitive, etc.).
• 15% of the world population experiences some form of disability (The World Bank).
• Cost of non-compliance with WCAG/508/ADA and other accessibility standards varies by
jurisdiction, with an average fines of $300,000- $600,000 (this does not include additional costs
— engineering, brand, etc.).
• https://getadaaccessible.com/ada-compliance-law-and-penalities/
• https://www.ada.gov/civil_penalties_2014.htm
• Varying levels of Accessibility Success Criteria: A, AA, AAA
• https://www.w3.org/WAI/standards-guidelines/wcag/
What Is Accessibility Testing?

• Most accessibility tests are done manually or not at all, most
cycles are being outsourced by enterprises.
• Accessibility testing is not part of the cycle and does not occur
as often as needed since it’s not automated —
• Real-life examrisk of penalties due to escaped defects and lack
of compliance.ple: $356,775 for an accessibility lawsuit after
total cost for litigation.
• Will disrupt sprint cycles and product defects are 100X more
expensive to fix after the fact.
Accessibility Testing — What’s the Pain Today?

• PERFORMANCE TESTING checks the speed, response time, reliability, resource usage,
and scalability of a software program under their expected workload.
• In performance testing, there are two types:
• Load Testing
• Stress Testing
• The purpose of performance testing is not to find functional defects, but to eliminate
performance bottlenecks in the software or device.
• The focus of performance testing is checking a software program’s following KPIs:
• Speed — Determines whether the application responds quickly (response time).
• Scalability — Determines maximum user load the software application can handle (bandwidth).
• Stability — Determines if the application is stable under varying loads.
• Reliability — Measurement of software consistency to give same results repeatedly.
What Is Performance Testing?

Neotys © . All Rights Reserved.
Change of
Methodology
Requirements Design Implementation Testing Acceptance Deployment
Inception Elaboration Construction Transition
Development Operations
DevOps
Big Bang
Waterfall
Incremental
Rational Unified
Process
Agile
Scrum, Kanban
Continuous
DevOps
Digital Singularity
Agile Event Horizon
6+
4
2
1
Change of Methodology

OK, but let’s
review our
architecture first
• From Monolithic
Presentation Layer
Business Layer
Data Layer
One application per host
Having the entire business logic
Let’s Review Our Architecture First

OK, but let’s
review our
architecture first
• To New Microservices
Front
Service
A
Service
B
Service
C
Sevice
D
Service
E
Service
F
Service
G
Every service is
managed by a
dedicated
team
Let’s Review Our Architecture First

Let’s move to
the cloud!
• Reduce the cost.
• Increase the availability of the application.
• Simplify, modernize.
Let’s Move to the Cloud

Efficient and
realistic
workload model
Efficient and Realistic Workload Model

Traditional
monitoring
Tracking
system
APM
Let’s Re-Use Monitoring Data

Test
Performance
Continuously
API Testing
Test a Component
Application Testing
Test Real World
Integration Testing
Test a System
Continuous Testing Embedded in CI/CD
Pipelines
Let’s Extract Data From Our APM to Learn How Our App is
Used

Architecture is
based on
components
Architecture Is Based on Components

Error
rate
Throughput + response time versus
objectives
Resources
check
Test
failure
Process to Analyse

File defining the
indicators
and the objectives
Quality
Gate
Collecting the
specification
Retrieving the data
from the datasource
Calculate the
score
Datasourc
e
Quality Gate

Most Accessibility Work Is Done Much Too Late in the Process
Production
Environment
Most accessibility activity
is conducted out here

Deque Enables You to Shift Left
WorldSpace
Attest
WorldSpace
Assure
WorldSpace
Comply
Deque Solution
Design Training and
Services

Where You Find and Fix Issues Matters
If issue caught
during
maintenance:
$7,600
If issue caught
during
QA/testing:
$960
If issue
caught
during
build:$240
If issue caught
in
development
$80 Good
Bad
Source: IBM
http://www-01.ibm.com/support/docview.wss?uid=swg27048410&aid=1

• Accessibility testing cannot be automated.
• Accessibility testing is usability testing.
Accessibility Myths

Generic Accessibility Automation
powered by Deque

No Code Merges Until #axeClean!

• Test while you develop.
Interactive Testing

Encode Design Annotation in Automated Tests
…
// Check the edit button’s text
// select the button and the heading
let button = fixture.querySelector(‘button.edit’)
let heading = fixture.querySelector(‘.Heading’)
// get the two elements’ accessible names
let headingText = axe.commons.text.accessibleText (heading)
let buttonText = axe.commons.text.accessibleText(button)
// compare
expect(buttonText).toBe(‘Edit ‘ + headingText)
…

Best Practices & Recommendations

Pipeline Testing Coverage Example: Where Does
Performance/Accessibility Fit?

Perfecto Integrates With Both Neotys & Deque Axe Solutions

Demo Environment
Load generators

Pipeline
1 Retrieve latest version of the Code/Test
Build and launch tests 2 Build the application and launch
the selenium tests
Generate Junit Report 3 Generate Junit report in Jenkins
Deploy load testing
infrastructure
4 Deploy NeoLoad Load Generator
Run Load Test 5 Run Load test with perfecto
integration

NeoLoad : Reduce Total Cost of Ownership (TCO)
• Digital, web, mobile, SAP Oracle EBS…
• 5-10x faster test design:
• Scriptless test design and validation — reduce errors.
• Reduce time to market — spend more time on
performance engineering vs. script writing.
• 20x faster script maintenance — Agile/DevOps.
• Reduce spend on tools:
• Much higher ROI.
• Flexible pricing model.
“We saw an 80% reduction in costs
while NeoLoad generated 1.2 million
virtual users from Neotys Cloud
Platform in 15 minutes to prep for our
busiest day of the year, Boxing Day.”

NeoLoad’s Platform
On-Prem Docker Stack
RESTful APIs
NeoLoad
Core
NeoLoad
Web
NeoLoad
SaaS
Managed/Hosted
Load Generators
&

NeoLoad
Web
Design
Desktop GUI
connection
On-
premise
SaaS
Runtime &
Analysis
NeoLoad Web

Collaboration Between Teams
• Performance engineers can update the project’s build by the developers from NeoLoad GUI.
• Early testing assets can be reused in system-wide testing.

One Platform for Component Testing & System Wide Testing

Free axe Browser Extension
Run automated accessibility tests against your website. It’s totally free and even
has cool beta features available with sign up.
✓ Single page tests ✓ Remediation help
axe Auditor
Perform comprehensive WCAG audits
of all content and applications.
✓ Comprehensive testing
methodology
✓ Predefined issues for all content
✓ Group and filter issues for easy
communication
axe DevTools
Integrate accessibility testing into
your existing automated testing
process with this toolkit.
✓ Automated testing
✓ Intelligent guided testing
✓ CI/CD integrations
✓ CLI
✓ Expert tools
axe Monitor
Dynamically scan, monitor, and report
on the accessibility status of your site.
✓ PDFs and web pages
✓ Scan whole sites
✓ Scan behind authentication
✓ Scan behind the firewall
✓ Drill-down dashboard
axe: the Standard in Accessibility Testing

Deque Enables You to Shift Left

What Is the Difference Between the Free Extension & axe DevTools?
Free extension axe DevTools
Axe-core library ✓ ✓
"Learn more" remediation resources ✓ ✓
Intelligent guided tests ✓* ✓
Test record manager & user dashboard ✓* ✓
Export issues & limit test scope ✓* ✓
Managed user data and analytics ✓
CLI ✓
CI/CD integration ✓
On-prem installation option ✓
Custom rules support ✓
Enterprise support ✓ *currently in beta

Perfecto’s 4 Key Pillars of Continuous Testing in DevOps
This is where money is spent (Time, Tools, Resources)
This is where value is being
realized, quality is improved
Unified solution serves DevOps with greater effectiveness.

Thank You!
Visit perfecto.io or follow us @perfectomobile

Best Practices for Shifting Left Performance and Accessibility Testing

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Best Practices for Shifting Left Performance and Accessibility Testing

Similaire à Best Practices for Shifting Left Performance and Accessibility Testing (20)

Plus de Perfecto by Perforce

Plus de Perfecto by Perforce (20)

Dernier

Dernier (20)

Best Practices for Shifting Left Performance and Accessibility Testing