How to Achieve Customer Satisfaction Through Beta Testing

2
Housekeeping Details
● This webinar is being recorded. We will send recording and slide deck to everyone.
● This webinar is part three of our four part Customer Validation webinar series.
○ Part One (Customer Validation) and Part Two (Alpha Testing)
○ How to Evaluate Feature Adoption Through Field Testing (October 25th)
● SoCal Event: Building Better Products With Customer Validation
○ Thurs., October 12th at 5:30 pm in Orange County, CA
● Q&A at the end

3
What You'll Learn Today
Whatexactly is Beta Testing?
Whyis it critical to the success of your product?
Whodoes it apply to in the industry?
Wheredoes it fit within your organization?
Whendoes it take place within the product life-cycle?
Howdoes it actually work (high level)?

About Me
Tom Peelen
Lead Test Manager
Centercode
Electrical, Construction, & Project Management Background
Team Lead to Centercode's Test Managers
Managed 10,000+ Testers
tom@centercode.com
linkedin.com/in/tom-peelen
4

ASSETS
Centercode Overview
Customer Validation (CV) is our core competency • Founded in 2001 • Industry leader
Tester Panel 180k+ deeply profiled global test candidates
Framework Scalable and adaptable Customer Validation process
Platform End-to-end cross-platform Customer Validation platform
OFFERINGS
Managed CV
Programs1
CV Platform
Subscription2
Customer
Research3
5

Industry Leading Customers
WE SERVICE AMAZING HIGH-GROWTH TECHNOLOGY AND MODERN ENTERPRISE COMPANIES 6

the speaker and Centercode
is Beta Testing?
is Beta Testing critical to the success of your product?
does Beta Testing apply to?
does Beta Testing belong within your organization?
does Beta Testing occur in PLM?
does Beta Testing actually work?
and next steps
About
What
Why
Who
Where
When
How
Q&A
7

8
Customer Validation
Targeted customers using your products in real environments
Distinct test phases designed to ensure a successful product launch
Many names: CAT, UAT, customer evaluation, pre-release, etc.

Defining Attributes of CV
9
Targeted Users • recruited based on demographic and technical
characteristics that are representative of your true audience
Actual Products • tangible pre-production products beyond the
prototype stage of development
Real Environments • the actual homes, workplaces, vehicles, and
lives of the representative customers
Over Time • extended study of actual real-world product usage
by over weeks or months

Customer Validation Test Stages
10
Field
Evaluate Adoption
Targeted customers focused on the real-world customer experience
Beta
Evaluate Satisfaction
Targeted customers guided through a comprehensive product tour
Alpha
Evaluate Stability
Technical users identifying quality issues in real environments

Customer Validation Test Stages
11
Field
Evaluate Adoption
Targeted customers focused on the real-world customer experience
Beta
Targeted customers guided through a comprehensive product tour
Alpha
Evaluate Stability
Technical users identifying quality issues in real environments

Beta Testing
Beta Tests guide a group of targeted representative customers
through the entire product experience. Throughout this stage,
testers will provide detailed qualitative feedback through a variety
of methods including both experiential (e.g. bug reports,
suggestions, discussions) and directed (e.g tasks, surveys). Beta
Testing results provide evidence-based recommendations focused
on assessing and improving feature-level customer acceptance,
user experience, product quality, and interoperability.
12

Core Concepts
REAL WORLD REAL PRODUCTSREAL PEOPLE
Targeted
Users
13

Customer Validation Beta Stage Details
BETA TEST
Guided Evaluation
Near/Feature Complete (80-90%)
Strategy
Product State
3-6 Weeks
(1-2 cycles)
25-200
Typical Test Length
Typical Testers
Do they like it?
Test Objective
Topic Question
14

is Beta Testing?
and next steps
About
What
Why
Who
Where
When
How
Q&A
15

The Beta Tour
16
Get your product out of the QA Lab1
2
3
Brings your product to a broader audience
Deep feedback from a targeted market
● Unique environments
● Primary experiences
● Specific market segments

is Beta Testing?
and next steps
About
What
Why
Who
Where
When
How
Q&A
17

Product Type Product Audience
18
What Products Benefit from Beta
Short Answer: Any product that passed Alpha
Hardware
Software
Services
Consumer
Business
Enterprise
Passed Alpha
Stable
Low Risk

is Beta Testing?
and next steps
About
What
Why
Who
Where
When
How
Q&A
19

20
Customer Validation Within an Organization
Quality UX/CX
Market
Research
Customer
Validation

21
Customer Validation Within an Organization
Quality UX/CX
Market
Research
Customer
Validation
Technology .
.
. Customer
Business
Product
Management

22
Beta Within an Organization
Quality UX/CX
Market
Research
Customer
Validation

is Beta Testing?
does CV actually work?
and next steps
About
What
Why
Who
Where
When
How
Q&A
23

24
Product Readiness for Beta
● 80-90% release ready
● Prepared for customer acceptance
● Guided evaluation via Product Tour
● Identified demographic and technical
segmentations

is Beta Testing?
and next steps
About
What
Why
Who
Where
When
How
Q&A
25

26
Typical Legacy Process (aka "beta testing")
1. Beg engineering for a beta release
2. Scavenge for people willing to be testers
3. Send out the beta product (with minimal context)
4. Beg testers for feedback (via email)
5. Shotgun distribution of results
6. Pray nothing fell through the cracks

27
The Source of the Problem
PROCESS
PEOPLE
TECHNOLOGY

Key Beta Challenges
28
Project design and scoping
Allocating qualified resources
Recruiting ideal testers
Persistent user engagement
Collecting useful feedback
Deriving actionable recommendations

29
Ongoing
Feedback
Directed
Research
Analysis and Recommendation
Beta Process Overview
Test Scoping and Product MappingPlanning
Recruitment and SelectionPrep
Weekly
Tester
Support
Engagement
Management
Collection and Wrap-upClosure

Test Scoping and Product Topic Mapping
Topics represent the individual features, functions, and experiences you want tested.
Product
Setup
The process of connecting the product
for first time use.
Find a spot to plug the device into both
the network and wall.
1.8 M
Account
Creation
The process of creating a personal
account to customize the product.
Log into www.xyz.com and create
a new user account.
1.2 S
Topic
Out of
the Box
Onboarding
Description
The initial experience opening the
product package.
Activity
Receive and unbox the product.
Impact
1.2
Size
S
30

31
Recruitment and Selection (Beta Example)
US Resident, Own 2+ Smart Home ProductsCore Requirements
Android iOS .Technical Segmentations
Smart Home Interest + ExperienceAdditional Traits
Demographic Segmentations Age 18-25 Age 26-35 Age 36-50 . Targeted
Enthusiastic
Strangers
Profiled Panel +

32
Ongoing Feedback Management
All Feedback must be Filtered, Categorized, and Prioritized based on your objectives (Topics and concerns)
Bug Reports Test quality, interoperability, and real-world performance
Measure acceptance, prioritize backlog, generate new ideasSuggestions
Gather user insight and sentiment; study user experiencePrivate Journals
Emulate real-world discussion; drive focused collaborationOpen Discussions
32

Assign
Topic
Activities
Directed Research Management
Centercode
Analysis
Weekly Presentation including Recommendation Scorecards (FIPs)
33
Engage
and Support
Testers
Customer
Acceptance
Survey

Recommendation Overview (Beta Sample)
82%
Recommendations to FIX 3
TOPIC: Unboxing
Customer Acceptance Score
This score is Above Average
Recommendations to IMPROVE 5
Recommendations to PROMOTE 1
Recommendations
B
grade
34

Fix Recommendation Sample
Tester Quotes
RELATED FEEDBACK
15%
OF RESPONDENTS
P1
REC. STRENGTH
Major
SEVERITY
FIX
RECOMMENDATION
BUG-0015
Antenna Broke Pulling it Out
of the Box
SUG-0013
Better Packaging for the
Antenna
Some testers report frustration with
the antenna breaking while attempting
to remove it from the packaging.
Reinforcing the antenna packaging
could prevent consumers from
damaging the antenna during
unboxing.
The Antenna is Damaged
During Unboxing
“While trying to remove the product from the box
the antenna snapped off. I did not even apply
that much pressure.”
“I did not realize that the antenna was broken off
until I heard it banging around in the box. Very
cheaply made. Plus the packaging was so hard
to remove, it is no wonder it broke off."
35

Tester Quotes
RELATED FEEDBACK
15%
OF RESPONDENTS
P1
REC. STRENGTH
Major
SEVERITY
FIX
RECOMMENDATION
BUG-0015
of the Box
SUG-0013
Antenna
unboxing.
During Unboxing
36

Tester Quotes
RELATED FEEDBACK
15%
OF RESPONDENTS
P1
REC. STRENGTH
Major
SEVERITY
FIX
RECOMMENDATION
BUG-0015
of the Box
SUG-0013
Antenna
unboxing.
During Unboxing
37

Tester Quotes
RELATED FEEDBACK
15%
OF RESPONDENTS
P1
REC. STRENGTH
Major
SEVERITY
FIX
RECOMMENDATION
BUG-0015
of the Box
SUG-0013
Antenna
unboxing.
During Unboxing
38

Improve Recommendations Sample
Tester Quotes
RELATED FEEDBACK
20%
OF RESPONDENTS
P4
REC. STRENGTH
Minor
SEVERITY
IMPROVE
RECOMMENDATION
BUG-0041
Twist Ties Were Broken
SUG-0013
Needs Better Twist Ties or
Restraints
Testers feel that the twist ties are very
cheap compared to the rest of the
contents in the packaging. Some
testers reported that twist ties were
broken upon first inspection.
Higher quality twist ties ensure the
consumer receives their product intact.
This will help prevent any mindset that
the product is of lesser quality.
The Included Twist Ties
Feel Very Cheap
“Everything was going great as I pulled stuff out
of the box, except the twist ties that held the
ethernet cord together were already broken.”
“I enjoyed pulling everything out of the box.
Things were neat and easily accessible. But, the
twist ties around the cords looked like twist ties
on bread and took the feel of quality out of most
the experience.”
39

Promote Recommendation Sample
Tester Quotes
RECOMMENDATION
PROMOTE 30%
OF RESPONDENTS
P3
REC. STRENGTH
“WOW! This thing is clean, the
packaging felt very orderly making it
simple to unbox. I wish all the products
I purchased came like this."
“Packaging rocks. The box feels
awesome and the contents are neatly
stored. I was amazed with how neatly
things were stored."
"Very impressed by the packaging.
I'm going to end up holding onto it
just in case I move and need to store
all of the components again.
Everything fit into its nice little
compartment, for some reason I
even felt compelled to take a picture
of it. Got me in the mood to organize
some stuff at home."
Packaging Contents are
Neat and Orderly
Overall, testers felt like everything was
very well organized within the box. The
hub and the light were very nicely
stored and made it easy to pull them
out.
This should be promoted to your
project teams or other members that
can benefit from positive feedback.
40

41
Key Takeaways for Beta
● Product has passed the alpha stage and is 80-90% complete
● It's stable for testing with many testers
● Recruit targeted testers
● Tell a story of your product to those testers by touring them through
key features
● Ensure testers have the medium to provide you with feedback
● Trend identification will lead to actionable recommendations

is Beta Testing?
and next steps
About
What
Why
Who
Where
When
How
Q&A
42

43
Open Q&A
Tom Peelen
Lead Test Manager @ Centercode
Thank You!
tom@centercode.com
linkedin.com/in/tom-peelen

How to Achieve Customer Satisfaction Through Beta Testing

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à How to Achieve Customer Satisfaction Through Beta Testing

Similaire à How to Achieve Customer Satisfaction Through Beta Testing (20)

Dernier

Dernier (20)

How to Achieve Customer Satisfaction Through Beta Testing