In this session, Intuit presents how they prepared TurboTax to take the production load, and how they gained the confidence to run their 2017 peak activity entirely on AWS. They discuss resiliency testing, game days, operational run books, working with AWS Support, and how each of these activities impacted their confidence in their reliability and availability.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How Intuit TurboTax Ran Entirely
on AWS for 2017 Taxes
Jeffery Weber
Distinguished Architect
Intuit Inc. / Consumer Group
A R C 3 0 7

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Landing the Plane
Planning to make the run to AWS (Oct 2016)
The Strong Leg (Feb 2017)
Gaining Confidence (Oct 2017)
THE Capacity Test (Nov 2017)
AWS Launch (Dec 2017)
1st Peak (Jan/Feb 2018)
The Lull before the storm (March 2018)
2nd Peak (April 2018)
Heading East (Summer 2018)
All In AWS (October 2018)

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Tax: A highly seasonal business in the cloud
1st Peak 2nd Peak
3rd Peak

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Planning to make the run to AWS
Secure Resources – Had set allocation of resources per quarter which
then drove the plan and sequencing
Secure Funding – Leveraged internal tool, then AWS Simple Monthly
Calculator to feed estimates back into finance to address double bubble
Migration Principles – That guided decisions we made along the way
Migration Sequence Template – Statement of work for all teams to
follow
https://calculator.s3.amazonaws.com/index.html

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Migration principles
• Drive for simplicity, minimal viable cloud architecture
• Solve the most difficult problems first
• Lift & Shift, refactor opportunistically to accelerate
• Ability to dial to / from AWS is a must
• Operate in AWS for at least one large peak to gain confidence
• Maintain or improve user experience
• Secure the customers data!
• Use and contribute patterns

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The Strong Leg & the Dial (2 way doors)
Application Tier
Services Tier
Persistence Tier
Application Tier
Services Tier
Persistence Tier
Application Tier
Services Tier
Persistence Tier
Data Center A Data Center B west

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The right amount of refactoring
Examples:
Removed Oracle Dependency
TurboTax Online session management
Database consolidation
Things we didn’t do:
Cassandra to Amazon DynamoDB
Containers
Lift & Shift Refactor & Move
Sweet
spot
ALWAYS! Validate
refactoring in your data
center BEFORE lifting and
shifting it into AWS

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How small is to small?
Monolith Nano Services
Sweet
Spot

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Migration Sequence Template
1) Experiment - AWS 90 day program
2) Architecture and Refactoring, Pre Work - see principles
3) Pre-Prod Build Out - just get it to work
4) Perf Build Out - Optimization, Monitoring, Hardening, Secured
5) Production Build Out - Capacity Testing, Go Live Checklist
6) AWS West Region Launch - Toe Dip, Dial It Up, Retire 1st Data Center
7) Move into East - Toe Dip, Dial It Up, Retire 2nd Data Center
Gave us a language to communicate and track efforts across many teams
(used on the next slide)

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Migration work streams
TurboTax OnlineMy TurboTax (Home)Free Fillable Fileable Forms The Long Tail
Work Stream Q2 Q3
p1&2
Q4 Q1
p3
Q2 Q3 Q4 Q1
p3
Q2 Q3
p1&2
FFFF 1,2 3 4,5 6 6 6 7 7 7 7
My TurboTax 1,2 3 4,5 6 6 6 7 7 7 7
TurboTax 1 3 2,3 4,5 6 6 6 7 7 7
Data Platform 1 2 3,4 5,6 6 6 7 7 7 7
File Archive 3,4,5 6 7 7 7
Long Tail … 1 2 3,4 5,6

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Regular cadence of capacity tests
All with specific test objectives designed to break the ecosystem
(AWS GameDay)
The most valuable lessons happened during failures
(the intent was to run higher load here)
User count by our two data centers
3rd peak

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Capacity testing tools & methodology
Tank is a cloud-native open source performance testing tool that Intuit
developed to drive massive load at a reasonable cost
Coverage: ~8 difference use case models are run from Tank along with
>20 different supplemental load drivers to approximate the load that
we see
Scope: End to End, all teams needed to model user profiles we see at
our peak loads
Frequency: Weekly / Bi-Weekly

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Monitor, observe, and respond
Monitor: CPU, Memory, Disk, Availability, Throughput, Utilization, ….
Fundamental metrics that
are available without any
context of the domain

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Monitor, observe, and respond
Observe: Looking at the health of your ecosystem with context

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Monitor, observe, and respond
Respond: A portfolio of “levers” we can pull to “fly” the ecosystem
• Traffic moves, runtime configurations (automated and manual)
• That we practice on a regular cadence
• Ensuring all mitigations work to a common outcome
Architecture Use Case Impact How
Detected?
Level 1
(automated?)
Level 1
(automated?)
Level 1
(automated?)
Recovery Plan
Data Platform Lost
Connectivity
with Key Store
Unable to
encrypt /
decrypt (after
exhausting
cache ~5 min)
DP Failures
IDPS GTM
Monitoring
DP-01: Retry
Logic in place
- 3 retries with
longer back
offs
(automated)
IDPS-01:IDPS
invalidate
region/data
center that is
having trouble
(automated)
TTO-01,
MyTT-01,
FFFF-01,
CARE-01, DE-
01:
Application
traffic move
(not
automated)
• Automatic
• IDPS GTM
Config
• Rebalance
Traffic
…
Bold items refer to specific playbooks managed by dev teams and practiced in regular capacity tests
Asking if automated drove work
back to dev teams which makes
the system more resilient

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Gaining confidence—3rd peak tax year 2016
 Free Fillable Fileable Forms and toe dip
 TurboTax Online was proving refactored code
 My TurboTax toe dip
Discovery: As we dialed My TurboTax into west, we observed higher errors rates in west that we had not
observed in our synthetic load tests
Root cause: There was a specific use case where an API we were calling returned a subtly different
response in west than our data centers effecting about 1% of our My TurboTax users
Takeaway:
• We had confidence to go into season with the refactoring we had done
• We still had work to do …

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
A watershed moment—The capacity test
Context
• September into October we had been running capacity tests, finding limits, making fixes, and
rerunning those tests on our AWS stack
• We have made a number of adjustments in our stack when 3rd peak was upon us
THE Test
• We were running synthetic load in both our data centers and AWS (to match our planned
footprint for the upcoming 1st peak)
• The capacity test started out normal but quickly stumbled (as they often do)
• The issues were in our data centers where we dialed down synthetic traffic while we kept the
synthetic traffic running in AWS
• The teams were focused on ensuring we did not impact our real customers while trying to
resolve the issue so we can restart the test
• Our AWS stack continued to run smoothly almost doubling our previous capacity record
AWS was now our strong leg

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
West is down!
Team FMEA and playbooks
E2E FMEA playbooks
FMEA tested at regular cadence with capacity and at capacity
(AWS GameDay)
Timed west is down exercise (RTO <5 min to get out)
The gate rush problem

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Fail forward principles
Protect key customer experience!
Restore first, troubleshoot later
Move traffic away from problem first, Debug Later
Move new customers immediately
Turn the dial(s) to move new traffic away
Moving existing users only when there is significant customer impact
Move existing customers slow enough to avoid gate rush
Move Back as soon as it is safe
Once issues have been understood / mediated, move back quickly
Validate with a small sample, restore back to where you were
Test frequently, think E2E
Don’t execute a playbook that has not been rehearsed
Don't have so many playbook you can't possible rehearse them all
The dial became the key and most important
operation control in our playbooks

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Tax year 2017 launch
 TurboTax Online dialed 50% of their users to west
 My TurboTax dialed 50% of their users to west (error rate issues resolved)
Discovery: Some My TurboTax cases reach back into our data centers (previous year tax returns) and could
overload that connection
Root cause: We were still migrating data (as planned), attempt to accelerate caused a wobble we then
needed to mitigate and further delayed data migration
Takeaways:
Controls to ensure the numbers users My TurboTax was handling did not exceed set user count
(implies we would not be able to do 50% users for My TurboTax at peak)
Continue data migration on course and retest through December / January to re-evaluate the
threshold

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
1st peak tax year 2017—The war room

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
1st peak tax year 2017—The wobble
Real users behavior had identified a performance issue
Issue was seen in both our data centers and AWS
Refined dashboards showing the issue and refined metrics to gauge improvements we would
make
DEV and SRE working collaboratively identified a way the issue could be addressed with
minimal risk (at least in AWS)
AWS deployed in three days showed significant improvement
Equivalent fix in our data centers would take weeks with significant risk
Protecting the customer experience is most important
Dialed 80% of traffic into AWS to give most the best possible experience while still gaining
confidence in the AWS stack

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The lull before the storm
Implement changes to address wobbles we saw in first peak
From the outside, 1st peak seemed smooth (flying the ecosystem)
From the inside, we were pulling levers to keep it smooth
Identified specific changes planned to be in place for 2nd peak
We didn’t have much time
Needed to code and re-validate at capacity

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The lull before the storm
AWS WEST
Shift 80%
traffic to
AWS
Resumed capacity testing
Concurrent
users by data
center / region

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
2nd peak tax year 2017—Brace for impact
4/16 – back end
API wobble 4/17 – Dialed
100% into AWS4/17 1am – IRS filing
end point failing
4/18 – Extended
filing impact
AWS WEST
Failed end point to
alternative site
Determined alternative site
would wobble 4/17

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Another successful season

36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Heading East – What we did over the summer
Scaled down our AWS west foot print
Learned we had a lot to learn about managing cost in AWS
Decommissioned servers in one of our data centers
Build out east leg with the same rigor west
FMEA testing and failover for east

38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
October 1st non-incident
West region wobbles during one of our capacity tests
From: Reaction of west is down!
To: It wobbled, turn your dials, non-event
AWS EAST
AWS WEST
We could have failed back to our data center
We choose to fail forward to east region
TurboTax Online users by region

39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Smooth sailing through 3rd peak
We split traffic between west and east through 3rd peak
We have pulled out of our 2nd data centers
We looking forward to anther great season – All in AWS
Data Center A Data Center B west east
Tax Ecosystem Tax Ecosystem Tax Ecosystem Tax Ecosystem

40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Key takeaways
Get help—Get help from the many AWS programs available
Monitor, observe & respond—Take a deeper look at the health of your
system that involves both SRE and Dev
Game Days—Make this a part of your DNA, test under load, happy and
unhappy path, practice your levers
Blameless Root Cause Analysis—Create the virtuous improvement cycle
Dev + operations—These teams should be solving problems together as
opposed to in isolation of each other
Critical few—Move into AWS first, then refactor to take advantage of all
that is available

41. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Jeffery Weber
Jeff_weber@intuit.com

42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.