The document discusses Intuit's use of AWS services to implement a CI/CD pipeline using Slingshot. Slingshot automates the setup of AWS accounts, VPCs, and other resources to enable secure, highly available continuous integration and delivery. Wiremock is used to stub out unreliable dependencies to improve test pass rates. Chef recipes automate the deployment of Wiremock servers to further support resiliency testing and avoid blockers in the CI/CD pipeline. The combined solution of Slingshot and Wiremock provides benefits such as easy CI/CD onboarding, increased test pass rates, and the ability to find resiliency issues before production.

1. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Lucy Chang
Sr. Software Engineer in Quality, Intuit
CI/CD with Mocking &
Resiliency Testing Using AWS
Alfred Tan
Sr. DevOps Manager, Intuit

4. Session Overview
• CI/CD and AWS onboarding
• Cross team AWS strategy alignment
• Automating AWS deployment
• Development work is blocked by dependency
• Integration tests fails due to unreliable dependency
• Need to do resiliency testing

5. CI/CD in AWS

6. Our Challenges
• Security requirements
• Onboarding into CI/CD
• Highly Available

7. Our Solution - Slingshot
• Build security in
• Automate onboarding to CI/CD
• Build HA/DR in

8. CI/CD Pipeline
Continuous Integration
Continuous Delivery/Deployment Pipeline
Promotion Criteria:
• Build pass: 100%
• Unit Test pass: 100%
• Code Coverage: >80%
Build
Promotion Criteria:
• BAT pass: 100%
CI
Promotion Criteria:
• Regression Test
pass: 100%
QA
Promotion Criteria:
• E2E Test pass:
100%
Test Run:
• E2E Test
• Performance Test
E2E/Perf
Test Run:
• Smoke Test pass: 100%
Prod

9. Slingshot Setup
Initial Setup
GitHub Repo
CI/CD Pipeline
KMS/SSH keys
S3 Buckets
Egress Proxy and Bastion Host
Splunk Forwarder
AWS Account Setup
AWS Account
VPC, Subnets, Routing tables, Route 53 Zone Delegation
One time events

10. Slingshot Initial Setup
Region US-WEST-2
Bucket for Artifacts KMS Key for SecretsKMS Key for EBSBucket for Secrets
Internet
Gateway
Bastion
ASG
Splunk
Forwarder
Egress
ASG
Public Bastion Subnets Public Egress Subnets
Private DB Subnets
Private APP Subnets
Public ELB Subnets
Private WEB Subnets
Delegated DNS Zones

11. Slingshot Setup
Initial Setup
GitHub Repo
CI/CD Pipeline
KMS/SSH keys
S3 Buckets
Egress Proxy and Bastion Host
Splunk Forwarder
CI/CD
ELB
Web Tier
App Tier
CNAME
Recurring events
AWS Account Setup
AWS Account
VPC, Subnets, Routing tables, Route 53 Zone Delegation
One time events

12. CD with Blue-Green Deployment
ci-svc.intuit.com qa-svc.intuit.com svc.intuit.com
PreProd
Account
Prod
Account
Public ELB
Subnets
Private Web
Subnets
Private App
Subnets
100% 5%95% 100%0%
CI Web
Build 10
CI App
Build 10
CI Web
Build 12
CI App
Build 12
QA App
Build 10
QA Web
Build 10
QA App
Build 10
QA Web
Build 12
QA App
Build 12
Prod App
Build 10
Prod Web
Build 10
Prod App
Build 10
Prod Web
Build 12
Prod App
Build 12

13. Benefits
• Early feedback on changes flowing through the system
• Increase in quality
• Frequent releases to production
• Development productivity from day 1

14. Slingshot Demo

15. Recap
Challenges
• Security requirements
• Onboarding into CI/CD
• Highly Available
The Solution: Slingshot
• Build security in
• Automate onboarding to CI/CD
• Build HA/DR in

16. The Next Problem
• Automation tests failed due to unreliable dependency server
• Builds are not promoted

17. Our Solution

18. Overview of Wiremock
Wiremock is a library for stubbing and proxying web services
• Stubbing
• Fault Injection
• Easy Set up
• Easy onboarding

19. How does Wiremock work?
System
Under Test
Wiremock
Server
Dependency
Server
Automation
Test
• Configure the Wiremock server to be man-in-the-middle
• Increased integration test pass rate
• Increased code coverage
• Does not interrupt other team’s calling the SUT
= Stubs

20. Before Wiremock
Subnet
ELB for SUT
SUT 1
SUT 2
Dependency
Server
Subnet
Automation
Test

21. We tried this
Subnet
ELB for SUT
SUT 1
SUT 2
Subnet
Automation
Test
• Deploy Wiremock on SUT EC2 instance
• No consistent stub response!
= Stubs

22. Our Solution
ELB for WM
SUT 1
SUT 2
Automation
Test
Wiremock
(Stub
Dependency)
ELB for SUT
• Deploy Wiremock on dedicated EC2 and ELB
• Consistent Stub responses!
= Stubs

23. If no stubs…
ELB for WM
SUT 1
SUT 2
Automation
Test
Wiremock
ELB for SUT
Dependency
Server
WM will proxy the
request to
dependency
server

24. Wiremock Code Snipets

25. Starting Up Wiremock Server
java -jar wiremock-1.53-standalone.jar --verbose --
port 8080 --proxy-all=[Dependency Server DNS Name]

26. Stubbing the response
//This calls Wiremock API to stub the response
stubFor(get(urlEqualTo(“/from/where”))
.willReturn(aResponse().withStatus(200)
.withHeader("Cache-Control", "no-cache")
.withHeader("Content-Type", ”text/plain")
.withBody(“Taiwan” )));

27. Simulating Fault
//This calls Wiremock API for fault injection
stubFor(get(urlEqualTo(“/some/thing”))
.willReturn(aResponse()
.withFault(Fault.EMPTY_RESPONSE)));

28. Benefits
• We fixed the CI/CD pipeline
• No more unnecessary test failures debugging
• Less production escapes and firefighting

29. Recap
The Second Challenge
• Integration tests failures broke CICD pipeline
• Hard to do resiliency testing
The Solution : Wiremock

30. Next Step
Why don’t we combine them?

31. Slingshot With Wiremock

32. Slingshot with Wiremock
W
ASG
Web
ASG
App
ASG
System Under Test
Region US-WEST-2
Availability Zone #1
Wiremock
ASG
Wiremock
Internet Gateway
Dependency
Server
AWS Region X / Datacenter X

33. Automate WM Deployment

34. Automate Wiremock Deployment
Chef is an infrastructure automation code tool we use
• Code how you deploy and manage your infrastructure
• Allows version control
• Code can be reused

35. Automate Wiremock Deployment
We wrote a Wiremock Recipe
• Download the Wiremock jar
• Start up the Wiremock server

36. Automate Wiremock Deployment
Berkshelf is a dependency manager for chef
• Get the Java recipe to download Java
• Get the Wiremock recipe to deploy Wiremock server

37. Chef Snipet
#This will start the wiremock server with the
parameters passed in
function start { cd "${USER_DIRECTORY}" ;java -jar
wiremock-${WIREMOCK_VERSION}-standalone.jar --port
${PORT} --proxy-via ${PROXY_VIA} -–proxy-all=
${PROXY_ALL} --verbose > /var/log/wiremock.log 2>&1
& }

38. Automate Wiremock Stack Creation
Use AWS CloudFormation API
• Provision EC2 instances and ELB
• Create Auto Scaling Group
• Set up other AWS resources

39. Use WireMock in Slingshot
Call Chef from CloudFormation

40. How to Call Cookbook From CloudFormation
Write shell scripts In the InstanceLaunchConfig section
1. Download and install chef
2. Run Chef. In this case we created a Wiremock role to
execute the java and Wiremock cookbooks.

41. Call Chef from CloudFormation
"5_run_chef": {
"command": { "Fn::Join": [ "", [
"/usr/bin/chef-solo -c /var/chef/config/solo.rb -o
'role[", { "Ref": "Role" }, "]' -E '", { "Ref":
"Environment" },"'" ] ]
}

42. Benefits
• A simplified CI/CD pipeline onboarding
• A successful CI/CD pipeline with increased test pass rate
• Resiliency testing capability built in
• Security features built in

43. Deep Dives

44. Demo
SUT Wiremock
Server
YelpTest
Automation

45. Recap

46. Recap
Combined Solution: Slingshot with Wiremock
• CI/CD pipeline easy onboarding
• Builds are auto-promoted
• Less Engineers’ time spent on debugging
• Resiliency issue found before production
• Happy Engineers

47. What we learned

48. What We Learned
• The initial investment is worth it
• Try to be flexible
• Set up DNS

49. References
• http://www.pnsqc.org/the-journey-of-mocking-in-aws/
• http://wiremock.org/
• https://www.chef.io/
• http://docs.aws.amazon.com/AWSCloudFormation/latest
/APIReference/Welcome.html

50. Related Sessions
Breakout Session:
ARC344
How Intuit Improves Security and Productivity with
AWS Virtual Networking, identity, and Account
Services
Track: Architecture
Session Level: Advanced (300 level)
Session Time: Thursday, Oct 8, 2:45 PM – 3:45 PM–
Palazzo

51. Contact
Lucy Chang
Sr. Software Engineer in Quality , Intuit
https://www.linkedin.com/pub/lucy-chang/11/312/a83
Alfred Tan
Sr. DevOps Manager, Intuit
https://www.linkedin.com/pub/alfred-tan/1/938/9b

52. Thank you!

53. Remember to complete
your evaluations!