Fabian Lim, Govtech Singapore Practitioner stories from the trenches

1. Fabian Lim
GovTech Singapore

4. Defending Thyself
With
Blue
Green
All Day DevOps
2016

5. My Story
-
about.me/
Fabian.Lim
● Xxxxxxxxxx Engineer
○ http://tech.gov.sg
● Passionate about CICD
● Cultural Hacking
● Always Thinking about Red
Teaming Possibilities

8. Black Box Deployment?

9. Black Box Deployment?

11. Who has an
automated CICD
pipeline?

12. GitHub
Source Code Check in
Jenkins
Build Tools
S3
Artifact Repository
Code
Pipeline
Deployment Tools
Code
Deploy
Production
Environment
EC2
CICD Pipeline is awesome

13. GitHub
Source Code Check in
Jenkins
Build Tools
S3
Artifact Repository
Code
Pipeline
Deployment Tools
Code
Deploy
Production
Environment
EC2
Code change is introduced

14. GitHub
Source Code Check in
Jenkins
Build Tools
S3
Artifact Repository
Code
Pipeline
Deployment Tools
Code
Deploy
Production
Environment
EC2
Build starts, and passes!

15. GitHub
Source Code Check in
Jenkins
Build Tools
S3
Artifact Repository
Code
Pipeline
Deployment Tools
Code
Deploy
Production
Environment
EC2
Artifact is dropped into repository

16. GitHub
Source Code Check in
Jenkins
Build Tools
S3
Artifact Repository
Code
Pipeline
Deployment Tools
Code
Deploy
Production
Environment
EC2
CD Tool picks up new artifact for testing

17. GitHub
Source Code Check in
Jenkins
Build Tools
S3
Artifact Repository
Code
Pipeline
Deployment Tools
Code
Deploy
Production
Environment
EC2
CD Tool uses new artifact to deploy

18. GitHub
Source Code Check in
Jenkins
Build Tools
S3
Artifact Repository
Code
Pipeline
Deployment Tools
Code
Deploy
Production
Environment
EC2
New/latest code is deployed on an instance!

20. GitHub
Source Code Check in
Jenkins
Build Tools
S3
Artifact Repository
Code
Pipeline
Deployment Tools
Code
Deploy
Production
Environment
EC2
Shoot, I need to revert the previous code change!
The whole CICD process repeats...

21. GitHub
Source Code Check in
Jenkins
Build Tools
S3
Artifact Repository
Code
Pipeline
Deployment Tools
Code
Deploy
Production
Environment
EC2
Code change is introduced

22. GitHub
Source Code Check in
Jenkins
Build Tools
S3
Artifact Repository
Code
Pipeline
Deployment Tools
Code
Deploy
Production
Environment
EC2
Build starts, passes!

23. GitHub
Source Code Check in
Jenkins
Build Tools
S3
Artifact Repository
Code
Pipeline
Deployment Tools
Code
Deploy
Production
Environment
EC2
Artifact is dropped into repository

24. GitHub
Source Code Check in
Jenkins
Build Tools
S3
Artifact Repository
Code
Pipeline
Deployment Tools
Code
Deploy
Production
Environment
EC2
CD Tool picks up new artifact for testing

25. GitHub
Source Code Check in
Jenkins
Build Tools
S3
Artifact Repository
Code
Pipeline
Deployment Tools
Code
Deploy
Production
Environment
EC2
CD Tool uses new artifact to deploy

26. GitHub
Source Code Check in
Jenkins
Build Tools
S3
Artifact Repository
Code
Pipeline
Deployment Tools
Code
Deploy
Production
Environment
EC2
Changes in code is deployed on the same instance

27. Downsides
● Downtime (SLA)
● Previous State of
Deployment is overwritten
● Resistance to
Infrastructure Changes
● Relatively Sequential
(Traffic Jam Scenario)

28. Who has an
automated CICD
pipeline,
with blue-green
deployment?

29. Enough Talk
Show Me the Money

30. GitHub
Source Code
Check in
Jenkins
Continuous Integration
S3
Artifact
Repository
Code
Pipeline
Continuous Deployment
Code
Deploy
Production
Environment
Code
Pipeline Code
Deploy
EC2
EC2
Red outline indicates current CICD workflow
Blue-green to the rescue!
(LIVE)

31. GitHub
Source Code
Check in
Jenkins
Continuous Integration
S3
Artifact
Repository
Code
Pipeline
Continuous Deployment
Code
Deploy
Production
Environment
Code
Pipeline Code
Deploy
EC2
EC2
Red outline indicates current CICD workflow
Code change is introduced
(LIVE)

32. GitHub
Source Code
Check in
Jenkins
Continuous Integration
S3
Artifact
Repository
Code
Pipeline
Continuous Deployment
Code
Deploy
Production
Environment
Code
Pipeline Code
Deploy
EC2
EC2
Red outline indicates current CICD workflow
Build starts, and passes!
(LIVE)

33. GitHub
Source Code
Check in
Jenkins
Continuous Integration
S3
Artifact
Repository
Code
Pipeline
Continuous Deployment
Code
Deploy
Production
Environment
Code
Pipeline Code
Deploy
EC2
EC2
Red outline indicates current CICD workflow
Artifact is dropped into repository
(LIVE)

34. GitHub
Source Code
Check in
Jenkins
Continuous Integration
S3
Artifact
Repository
Code
Pipeline
Continuous Deployment
Code
Deploy
Production
Environment
Code
Pipeline Code
Deploy
EC2
EC2
Red outline indicates current CICD workflow
CD Tool picks up new artifact for testing
(LIVE)

35. GitHub
Source Code
Check in
Jenkins
Continuous Integration
S3
Artifact
Repository
Code
Pipeline
Continuous Deployment
Code
Deploy
Production
Environment
Code
Pipeline Code
Deploy
EC2
EC2
Red outline indicates current CICD workflow
CD Tool picks up new artifact for deployment
(LIVE)

36. GitHub
Source Code
Check in
Jenkins
Continuous Integration
S3
Artifact
Repository
Code
Pipeline
Continuous Deployment
Code
Deploy
Production
Environment
Code
Pipeline Code
Deploy
EC2
EC2
Red outline indicates current CICD workflow
Changes in code is deployed on green, going live
(LIVE)(LIVE)

37. GitHub
Source Code
Check in
Jenkins
Continuous Integration
S3
Artifact
Repository
Code
Pipeline
Continuous Deployment
Code
Deploy
Production
Environment
Code
Pipeline Code
Deploy
EC2
EC2
Red outline indicates current CICD workflow
Completely switch over to green, decommission blue
(LIVE)

40. GitHub
Source Code
Check in
Jenkins
Continuous Integration
S3
Artifact
Repository
Code
Pipeline
Continuous Deployment
Code
Deploy
Production
Environment
Code
Pipeline Code
Deploy
EC2
EC2
Red outline indicates current CICD workflow
Known good code / state is preserved on blue
(LIVE)

41. GitHub
Source Code
Check in
Jenkins
Continuous Integration
S3
Artifact
Repository
Code
Pipeline
Continuous Deployment
Code
Deploy
Production
Environment
Code
Pipeline Code
Deploy
EC2
EC2
Red outline indicates current CICD workflow
Completely switch over to blue, decommission green
(LIVE)

43. How does
blue-green
defend thy
stack?
● Minimise Downtime
● Preserved Last Known
Deployment
● Robust Infrastructure
● Parallel Pipelines

44. Minimise
Downtime
● Service and Data
Availability
● SLA
○ Business Critical

45. Preserved Last
Known
Deployment
● Rollback enabled
● Debugging
● Forensics
○ Take it offline and
isolate
○ Selfie* anyone?
*https://alldaydevops2016.sched.org/event/8614/taking-a-selfie
-just-try-to-resist-doing-forensics-the-devsecops-way

46. ● Resilient to Security Testing
and Fire Drills
● Restore to known good
state
○ “Refresh” stack
Robust
Infrastructure

47. Robust
Infrastructure
● Vulnerability Management
○ Quick to patch zero days
- app to infra layers
○ One New Zero-Day
Vulnerability Discovered
on Average Every
Week*
*https://www.symantec.com/en/aa/about/newsroom/press-releases/2016/s
ymantec_0413_01

48. Parallel
Pipelines
● Why stop at blue-green?
● Go RAINBOW!
● Scale
○ Restore to multiple
states on multiple
instances

49. Enough Talk
Show Me the RAINBOW

52. What makes it easier
to do blue-green?

53. Crawl?
● Virtualization
● Infrastructure and
Security as Code
● Build a CICD process
● Plan for Security
Testing
● Identify where
Blue-Green is relevant*
*https://d0.awsstatic.com/whitepapers/AWS_Blue_Green_Deployments.pdf
Walk
Run

54. Walk.
● Automate Existing
CICD Process
● Incorporate Security
Testing
● Manually implement
Blue-Green where
relevant
Crawl
Run

55. ● Automate Blue-Green!
● Automate Security
Testing
● Scale RAINBOW!
Walk
Crawl
Run!

56. GitHub
Source Code
Check in
Jenkins
Continuous Integration
S3
Artifact
Repository
Code
Pipeline
Continuous Deployment
Code
Deploy
Production
Environment
Code
Pipeline Code
Deploy
EC2
EC2
Just securing the end product isn’t enough, it’s the
WHOLE pipeline that’s the attack vector
(LIVE)
Red arrows indicate attack vectors

57. What do you
mean
secure CICD?

58. How would you feel if your
build tools and
deployment tools are
down / hacked?

59. (╯°□°）╯︵ ┻━┻

60. Yeah, me too.

61. GitHub
Source Code
Check in
Jenkins
Continuous Integration
S3
Artifact
Repository
Code
Pipeline
Continuous Deployment
Code
Deploy
Production
Environment
Code
Pipeline Code
Deploy
EC2
EC2
Emphasize on securing pipeline as much as
end-product
(LIVE)

62. Convincing your team
to go secure
blue-green?

63. Red team* it; to fix it.
GoalMethod
*https://alldaydevops2016.sched.org/event/861E/operationalizing-red-team-for-fun-and-profit

64. https://github.com/fabianlim1989/DefendThyselfBlueGreen

65. Time: 25 man-hours
Money: ~ $10
Engineer: 1
Software: Free
Security: Built-In, Priceless

66. Appreciate
Your
Kind
Attention
Thank you
Gracias
谢谢
Terima Kasih
Gam Xia