14. Making Deployment
Decisions…
1. How to make (or at least communicate
recommendations to the people who do make)
decisions to release (with bugs), or delay, and whether
or not to rollback vs. hotfixing in production when bugs
do make it out into the wild
Release / delay?
Rollback / hotfix?
25. Because one of the scientists that worked on
the ground-based computer software
created output in units of pounds/seconds
instead of the SI units of newton/seconds
specified in the contract between NASA and
Lockheed.
35. • Caused by a glitch in 3rd party software provided by
Repricer Express
• Repriced thousands of products from mattresses to
Playstation 4’s to just 1p
• Small retailers lost tens of thousands of dollars overnight
and faced bankruptcy
• Both Amazon & RepricerExpress did not offer any
compensation to sellers
39. A disaster is (for the purposes of this presentation) a
software bug that harms:
o The client’s bottom line
o The users’ bottom line
o The client’s faith in you
o The users’ trust in the website / app
o Users’ ability to use the website/app
OR
o Your reputation
To an untenable degree.
40. Caveats
…Or causes maimings or deaths
(Therac 25, Toyota Camry--Barbara Schwarz)
….Otherwise probably has to affect
a significant number of users (if 2% of
users lose faith in your website, may not be a ‘disaster’)
41. Additionally…
• How will the bug impact internal
resources as you make reparations / fixes?
• Can the user recover by taking a
reasonable action?
• How captive is your audience?
• Have you exposed sensitive data?
43. Scenario 1: You work on a website that sells outdoor gear
and it is discovered a few hours after your latest release
that there is a javascript error that causes the quantity of
the last item in the cart to decrease when the user uses the
mouse to scroll down the page to click “Purchase,” leading
to incorrect totals/ missing items in the order that the user
may not notice on the following confirmation page—but
the incorrect total is displayed.
45. Impacts user’s trust in the website to an untenable
degree…(“I didn’t get my medium blue softshell jacket
before I left for vacation!”)
Unless the company makes quick reparations.
So NOT a disaster unless handled poorly by the company,
OR if the cost is too great in terms of internal resources
needed to make reparations, resulting in damage to the
company’s bottom line—was the bug released during a
time of high traffic? Are there a lot of orders to fix?
46. Scenario 2: You work on a website similar to jumponit and
as you navigate from the homepage to a specific deal and
then back to the homepage, the website loses track of your
geolocation and changes your ‘current location’ to a
random city. There is no way for the user to reselect their
location. This bug is not discovered until after being
released and is reported by a user in a large metropolitan
area almost 24 hours after the release.
48. • If verified, this impacts a large group of users, maybe all
users
• There is no way for users to recover
• Website is completely unusable to anyone who doesn’t
happen to be randomly traveling to wherever the
website decided they were located
• Impacts user’s trust in the website, BUT users will
probably come back because there’s good deals
• More importantly is impact on bottom line from lost
sales for period of time bug was out there
49. Scenario 3: The client calls and screams at your project
manager because they can’t log in as a full admin and view
important financial reports. They’ve been trying to reach
your company for hours because there is a big finance
review meeting in 10 minutes…
53. Maybe not ideal, but…
• Client reports terrible bug while smoke testing a new
release…turns out they haven’t cleared their cache
• Company admin reports terrible bug while
impersonating user…that does not affect actual users
• Homepage of website initially loads fine, but then goes
blank and reports no results found for your area after
about 20 seconds…but it’s ok because most users click
on something and go to another page in the first 5
seconds
55. Looks fine, really isn’t!
• Lack of integration testing so everyone knows it’s fine
within their part of the app/website but no one has
tested the whole thing
• Third party software changes that go unnoticed
• Subtle calculation errors when checking out, especially
in fees, taxes, or percentages paid out to vendors
• Works in your neck of the woods but nowhere else
(geolocation issues specific to other geolocations, time
zone issues specific to other time zones)
56. Looks fine, really isn’t!
• Everyone tests in Chrome but significant % of users has
issue in other browser
• Everyone tests web and completely forgets mobile
• Subtle calculation errors leading to falsely positive
results in business reports
58. Deploy or Delay?
• When possible do a little of both. Three hours of testing is
better than none.
• Start ups are a culture of higher risk—so be prepared to take
more risks and test less than you would in a more established
company—lean towards deploying
• Really focus whatever time for testing you have on core
functionality and on the devices used the most to access your
website/app
• Devote some sliver of time to testing admin functionality
• If core functionality is compromised, DELAY
• Consider the quality of what is currently out there—if latest
release is an improvement, release it even with the bugs
59. Deploy or Delay?
• Scenario 1: You’re testing an app that is map-based and
it doesn’t load smoothly. You can’t zoom in or out
effectively, and it takes forever for parts of the map to
load. You feel frustrated trying to use the app, so the
users will too. It’s slow and stutters on both iPhones and
Androids. You can find the information you’re looking
for, it just takes a long time and a lot of patience. The
current version of the app crashes every 30 seconds and
you cannot accomplish any basic tasks.
60. Deploy or Delay?
Scenario 2: The website you’re testing allows users to
barter for services with each other, review each other
publicly, and post items for sale. It is currently very stable
and usable but is only available in your local city. This next
release is to expand the website into 3 additional cities. You
discover in the eleventh hour of testing that new users
cannot successfully sign up—something broke in the most
recent round of fixes. Additionally, existing users can no
longer review each other.
61. Rollback?
• Have a rollback plan in place. Decide ahead of time what will
trigger a rollback.
• Is core functionality being impacted?
• Will a fix (either coding it or testing it) take a significant
amount of time?
• How easy is it to rollback? Was everything snapshotted at the
same time? Will info be lost in the meantime?
• How recent is your database backup?
• Are there database migrations that are either difficult, or
require more time to rollback?
• How long will it take to rollback vs. hotfixing?
• The risk of rollback failing is equal to the risk of failure to
deploy
62. Scenario 1: After the latest release you discover during
smoke testing that you cannot successfully purchase
anything on the website. The lead dev determines it will
take them a full day to fix things. You know the team has
been able to snapshot everything for a stable save point
across different components of the website.
63. Scenario 2: After the latest software release, you discover
that 3 important variables that are used to generate
financial reports concerning overall revenue are being
calculated at inflated values. All 3 are consistently inflated
by 12.5%. The developers need a few days to fix things. To
rollback would be difficult because it turns out there is a
problem with your database backup.
64. Hotfix?
• Is it a small change?
• Does it involve any dependencies?
• Is it impossible to rollback?
• Is core functionality being compromised?
65. Scenario 1: You’re smoke testing a release and notice that
the company name is spelled wrong on the homepage, in
the header.
66. Scenario 2: Your company just did a major release that
adds a new feature to your website where users can upload
photos and automatically create a slideshow for their
listings in your directory. Unfortunately all the existing
photos on the website are now not loading, and, users
cannot upload photos either. And even though everything
worked in your test environment, it also turns out that
users cannot create a new listing in the directory either.
68. Recipe for Disaster:
• Culture of ego, devs who assume they can pull it off
rather than asking thorough questions
• Lack of communication about requirements
• Project manager who always says yes to the client
• Culture that does not adequately weigh the risks of
moving too fast against the value of testing and doing
things right
69. Recipe for Disaster:
• Testing has been outsourced overseas
• Team has no diversity
• Lots of third party dependencies
• Lack of time for testing
• When devs are expected to do most of the testing
• When devs are not aware of widely accepted design
standards
• Party culture that is cavalier about business needs of
client
70. Preventing Disasters:
• When in doubt, ask questions!
• Involve QA in the design process
• Foster interdependence vs competition
• Be thorough about requirements gathering and then firm
with client about not changing them for that release
• Take the potential for disaster seriously. You are not
immune.
• If testing is going to be outsourced, up communication.
Preferably, keep testing in house.
71. Preventing Disasters:
• Value diversity within your team. If your team lacks
diversity seek other individuals to offer quick feedback.
• Be diligent about integration testing around third party
software. (And be clear about how they will handle bugs
in their software if it leads to loss of revenue.)
• QA time should be roughly equal to dev time. Really.
• Even the best devs mostly test that things work. QA will
actually try to break things. Don’t leave all the testing to
devs.
• Encourage devs to be aware of existing design
standards.
• Work hard, then play hard. Client’s needs are main
focus.
