JULIAN BORREY Coinbase is a company that empowers its developers to deploy fresh code to production just minutes after writing it yet there are has massive security requirements. Cryptocurrency companies are constantly being attacked, and Coinbase, which stores billions of dollars of irreversible cryptocurrency, is one of the biggest bounties on the internet. One of the pillars that allows us to maintain security in a CICD engineering organization is automated security scanning. Such scanners are often configured on a per-repository bases and may look for CVEs in dependencies or common anti-patterns that lead to vulnerabilities. In order for the Coinbase security team keep up with our ever growing product space, we built a tool that helps us centrally orchestrate our scanning pipeline on every project simultaneously. This tool is called Salus and is now being released free and open source. It is not necessarily easy to integrate security scanners en masse. A security team will start by finding relevant scanners and then inserting them into a project’s test suite. At first, when Coinbase had just a few projects, custom configuration for each repository worked fine. Each time the security team wanted to use a new scanner, update scanner configuration or roll out new policies, we updated each repository. As Coinbase scaled and became more polyglot, the time it took to maintain our security scanners rose dramatically until it was untenable to maintain strong scanning on every repository. As David Wheeler said, “All problems in computer science can be solved by another level of indirection.” Salus is our level of indirection to solve this problem. It is a docker container equipped with security scanners for many commonly used languages and frameworks as well a small ruby application used to coordinate the scanners. A developer can now add the Salus scanner to their test suite and on each build, it will pull down the latest Salus container, volume in their source code and execute the relevant scanners. We ensure that Salus results are immediately communicated to the developer and metrics about each project are communicated to the logging pipeline. Salus became a single place for the security team to make changes to the scanning pipeline that would be instantly applied org wide. Metrics aggregation also allowed for immediate insight into possible dangers as new vulnerabilities are discovered or to keep a pulse on the aggregate security posture of the company. Today, Ruby, Node, Python, Go, Shell and arbitrary pattern searches are represented in Salus and this will expand in the future as the project evolves. This talk aims to explain how an engineering team can start using Salus to enable them to stay safe with as little friction and effort as possible.

1. LONDON 18-19 OCT 2018
Introducing Salus: How Coinbase
Scales Security Automation

2. LONDON 18-19 OCT 2018
Julian Borrey, Security @ Coinbase

3. LONDON 18-19 OCT 2018
A story of scaling security

4. LONDON 18-19 OCT 2018
A story of scaling security
“100% of services deployed to production must have a security scan.”

5. LONDON 18-19 OCT 2018
A story of scaling security
“100% of services deployed to production must have a security scan.”

6. LONDON 18-19 OCT 2018
A story of scaling security
“100% of services deployed to production must have a security scan.”
I join
&
Multi-service

7. LONDON 18-19 OCT 2018
Overview
● Review of security scanners

8. LONDON 18-19 OCT 2018
Overview
● Review of security scanners
● Problems with security scanners at scale

9. LONDON 18-19 OCT 2018
Overview
● Review of security scanners
● Problems with security scanners at scale
● What Salus does and how Salus works

10. LONDON 18-19 OCT 2018
Overview
● Review of security scanners
● Problems with security scanners at scale
● What Salus does and how Salus works → free, no vendors/accounts

11. LONDON 18-19 OCT 2018
Overview
● Review of security scanners
● Problems with security scanners at scale
● What Salus does and how Salus works → free, no vendors/accounts
● Salus details
○ Configuration
○ Custom scanners
○ Metrics & reporting

12. LONDON 18-19 OCT 2018
Overview
● Review of security scanners
● Problems with security scanners at scale
● What Salus does and how Salus works → free, no vendors/accounts
● Salus details
○ Configuration
○ Custom scanners
○ Metrics & reporting
● Pointers to source code & more resources

13. LONDON 18-19 OCT 2018
Software Scanners

14. LONDON 18-19 OCT 2018
Software Scanners
A scanner is software that analyses other software.

15. LONDON 18-19 OCT 2018
Software Scanners
A scanner is software that analyses other software.
Source Code

16. LONDON 18-19 OCT 2018
Software Scanners
A scanner is software that analyses other software.
ScannerSource Code

17. LONDON 18-19 OCT 2018
Software Scanners
A scanner is software that analyses other software.
Scanner
Pass
Fail
Source Code

18. LONDON 18-19 OCT 2018
Software Scanners
A scanner is software that analyses other software.
Scanner
Pass
Fail
Source Code

19. LONDON 18-19 OCT 2018
Software Scanners
Linter - checks syntax follows certain policies.

20. LONDON 18-19 OCT 2018
Software Scanners
Linter - checks syntax follows certain policies.

21. LONDON 18-19 OCT 2018
Software Scanners
Linter - checks syntax follows certain policies.
Rubocop
(Ruby Linter)

22. LONDON 18-19 OCT 2018
Software Scanners
Linter - checks syntax follows certain policies.
Rubocop
(Ruby Linter)

23. LONDON 18-19 OCT 2018
Software Scanners
Linter - checks syntax follows certain policies.
Rubocop
(Ruby Linter)

24. LONDON 18-19 OCT 2018
Security Scanners
CVE scanner - looks for known vulnerabilities in dependencies

25. LONDON 18-19 OCT 2018
Security Scanners
CVE scanner - looks for known vulnerabilities in dependencies
CVE = Common Vulnerability Enumeration -
some documented vulnerability

26. LONDON 18-19 OCT 2018
Security Scanners
CVE scanner - looks for known vulnerabilities in dependencies

27. LONDON 18-19 OCT 2018
Security Scanners
CVE scanner - looks for known vulnerabilities in dependencies

28. LONDON 18-19 OCT 2018
Security Scanners
CVE scanner - looks for known vulnerabilities in dependencies
BundlerAudit

29. LONDON 18-19 OCT 2018
Security Scanners
CVE scanner - looks for known vulnerabilities in dependencies
BundlerAudit

30. LONDON 18-19 OCT 2018
Security Scanners
CVE scanner - looks for known vulnerabilities in dependencies
BundlerAudit

31. LONDON 18-19 OCT 2018
Security Scanners
Scanners are important:

32. LONDON 18-19 OCT 2018
Security Scanners
Scanners are important:
● Powerful - can search huge CVE databases

33. LONDON 18-19 OCT 2018
Security Scanners
Scanners are important:
● Powerful - can search huge CVE databases
Verizon Data Breach Investigations Report, 2015

34. LONDON 18-19 OCT 2018
Security Scanners
Scanners are important:
● Powerful - can search huge CVE databases
Verizon Data Breach Investigations Report, 2018
Verizon Data Breach Investigations Report, 2015

35. LONDON 18-19 OCT 2018
Security Scanners
Scanners are important:
● Powerful - can search huge CVE databases
● Some anti-patterns are obvious and scanners can do
the job. E.g. using `eval()` on user controlled input.

36. LONDON 18-19 OCT 2018
Security Scanners
Scanners are important:
● Powerful - can search huge CVE databases
● Some anti-patterns are obvious and scanners can do
the job. E.g. using `eval()` on user controlled input.
● Not fatigued like humans

37. LONDON 18-19 OCT 2018
Security Scanners
Scanners are important:
● Powerful - can search huge CVE databases
● Some anti-patterns are obvious and scanners can do
the job. E.g. using `eval()` on user controlled input.
● Not fatigued like humans
● Can run on every build round the clock

38. LONDON 18-19 OCT 2018
Security Scanners
Scanners are important:
● Powerful - can search huge CVE databases
● Some anti-patterns are obvious and scanners can do
the job. E.g. using `eval()` on user controlled input.
● Not fatigued like humans
● Can run on every build round the clock
● Not silver bullets, use in tandem with human review

39. LONDON 18-19 OCT 2018
How might you deploy a scanner?

40. LONDON 18-19 OCT 2018
How might you deploy a scanner?
● Execute the scanner manually.
$ cd /path/to/repo
$ bundle-audit check

41. LONDON 18-19 OCT 2018
How might you deploy a scanner?
● Execute the scanner manually.
$ cd /path/to/repo
$ bundle-audit check
Name: activesupport
Version: 3.2.10
Advisory: CVE-2013-1856
Criticality: High
URL: http://www.osvdb.org/show/osvdb/91451
Title: XML Parsing Vulnerability affecting JRuby users
Solution: upgrade to ~> 3.1.12, >= 3.2.13

42. LONDON 18-19 OCT 2018
How might you deploy a scanner?
● Execute the scanner manually.
● Could be slightly better with pre-commit hook
$ cd /path/to/repo
$ bundle-audit check
Name: activesupport
Version: 3.2.10
Advisory: CVE-2013-1856
Criticality: High
URL: http://www.osvdb.org/show/osvdb/91451
Title: XML Parsing Vulnerability affecting JRuby users
Solution: upgrade to ~> 3.1.12, >= 3.2.13

43. LONDON 18-19 OCT 2018
How might you deploy a scanner?
● Run the scanner in the CI/CD pipeline via the repo’s test suite.

44. LONDON 18-19 OCT 2018
How might you deploy a scanner?
● Run the scanner in the CI/CD pipeline via the repo’s test suite.

45. LONDON 18-19 OCT 2018
● Run the scanner in the CI/CD pipeline via the repo’s test suite.
How might you deploy a scanner?
developer

46. LONDON 18-19 OCT 2018
● Run the scanner in the CI/CD pipeline via the repo’s test suite.
How might you deploy a scanner?
developer source control

47. LONDON 18-19 OCT 2018
● Run the scanner in the CI/CD pipeline via the repo’s test suite.
How might you deploy a scanner?
developer source control CI servers

48. LONDON 18-19 OCT 2018
● Run the scanner in the CI/CD pipeline via the repo’s test suite.
How might you deploy a scanner?
developer source control CI servers
AWS / GCP
/ etc
production servers

49. LONDON 18-19 OCT 2018
How might you deploy a scanner?
...
“100% of services deployed to production must have a security scan.”

50. LONDON 18-19 OCT 2018
Upgrading the fleet

51. LONDON 18-19 OCT 2018
Upgrading the fleet
● Have to make M x N code changes.

52. LONDON 18-19 OCT 2018
Upgrading the fleet
● Have to make M x N code changes.
● Want to avoid asking the service owners:
○ Lots of work to keep asking for this.
○ Requires a fair bit of context to understand
the tool and configure it correctly.

53. LONDON 18-19 OCT 2018
Upgrading the fleet
● Have to make M x N code changes.
● Want to avoid asking the service owners:
○ Lots of work to keep asking for this.
○ Requires a fair bit of context to understand
the tool and configure it correctly.
● So do it yourself?

54. LONDON 18-19 OCT 2018
Upgrading the fleet
● Have to make M x N code changes.
● Want to avoid asking the service owners:
○ Lots of work to keep asking for this.
○ Requires a fair bit of context to understand
the tool and configure it correctly.
● So do it yourself?

55. LONDON 18-19 OCT 2018
"All problems in computer science can be solved
by another level of indirection." - David Wheeler

56. LONDON 18-19 OCT 2018
Enter Salus
"All problems in computer science can be solved
by another level of indirection." - David Wheeler

57. LONDON 18-19 OCT 2018
How might you deploy a scanner?
Run Latest
Salus
Container
● Run the scanner in the CI/CD pipeline via the repo’s test suite.

58. LONDON 18-19 OCT 2018
What is Salus

59. LONDON 18-19 OCT 2018
What is Salus
$ docker run --rm -t -v $(pwd):/home/repo coinbase/salus

60. LONDON 18-19 OCT 2018
What is Salus
$ docker run --rm -t -v $(pwd):/home/repo coinbase/salus
● docker run
● --rm
● -t
● -v $(pwd):/home/repo
● coinbase/salus

61. LONDON 18-19 OCT 2018
What is Salus

62. LONDON 18-19 OCT 2018
Ruby app in container:
How Salus works

63. LONDON 18-19 OCT 2018
Ruby app in container:
● Initializes with configuration (more on this later)
How Salus works

64. LONDON 18-19 OCT 2018
Ruby app in container:
● Initializes with configuration (more on this later)
● Loops through each scanner
○ Ruby app? → run `bundle-audit check`
○ Rails app? → run `brakeman`
○ Node app? → run `npm audit`
How Salus works

65. LONDON 18-19 OCT 2018
Ruby app in container:
● Initializes with configuration (more on this later)
● Loops through each scanner
○ Ruby app? → run `bundle-audit check`
○ Rails app? → run `brakeman`
○ Node app? → run `npm audit`
● Compiles report, prints to STDOUT and HTTP post
How Salus works

66. LONDON 18-19 OCT 2018
Ruby app in container:
● Initializes with configuration (more on this later)
● Loops through each scanner
○ Ruby app? → run `bundle-audit check`
○ Rails app? → run `brakeman`
○ Node app? → run `npm audit`
● Compiles report, prints to STDOUT and HTTP post
● Exits !0 if issues are found (which could fail CI)
How Salus works

67. LONDON 18-19 OCT 2018
Why is Salus useful?
● Have to make M x N code changes.

68. LONDON 18-19 OCT 2018
Why is Salus useful?
● Have to make M x N code changes.
● Make 1 code change.

69. LONDON 18-19 OCT 2018
Why is Salus useful?
Latest Salus
Pass
Fail
Source Code
Container Registry or
Configuration Host

70. LONDON 18-19 OCT 2018
Why is Salus useful?
Latest Salus
Pass
Fail
Source Code
Container Registry or
Configuration Host

71. LONDON 18-19 OCT 2018
Why is Salus useful?

72. LONDON 18-19 OCT 2018
Why is Salus useful?

73. LONDON 18-19 OCT 2018
Salus Configuration

74. LONDON 18-19 OCT 2018
Salus Configuration
● Salus has a bunch of scanners:
○ BundlerAudit
○ Brakeman
○ npm audit
○ PatternSearch (grep)

75. LONDON 18-19 OCT 2018
Salus Configuration
● Salus has a bunch of scanners:
○ BundlerAudit
○ Brakeman
○ npm audit
○ PatternSearch (grep)
● You can choose what fails a Salus run → leads to !0 exit status
(useful for CI pipelines to fail)

76. LONDON 18-19 OCT 2018
Salus Configuration
● Salus has a bunch of scanners:
○ BundlerAudit
○ Brakeman
○ npm audit
○ PatternSearch (grep)
● You can choose what fails a Salus run → leads to !0 exit status
(useful for CI pipelines to fail)
● Each scanner could also be customized

77. LONDON 18-19 OCT 2018
Salus Configuration
● Salus has a bunch of scanners:
○ BundlerAudit
○ Brakeman
○ npm audit
○ PatternSearch (grep)
● You can choose what fails a Salus run → leads to !0 exit status
(useful for CI pipelines to fail)
● Each scanner could also be customized
● Salus has a --config flag

78. LONDON 18-19 OCT 2018
Salus Configuration
● Config can also be provided via:
○ A salus.yaml file in the repository’s root will be automatically parsed.
○ URI in the environment variable SALUS_CONFIGURATION
docker run
--rm
-v $(pwd):/home/repo
coinbase/salus --config file://tests/salus.yaml

79. LONDON 18-19 OCT 2018
Salus Configuration

80. LONDON 18-19 OCT 2018
Salus Configuration
● For global security policies that every repository should follow, use a remote URI.
docker run
--rm
-v $(pwd):/home/repo
coinbase/salus --config https://internal.net/salus.yaml

81. LONDON 18-19 OCT 2018
Salus Configuration
● For global security policies that every repository should follow, use a remote URI.
● Especially useful for testing out new security policies before enforcing them.
docker run
--rm
-v $(pwd):/home/repo
coinbase/salus --config https://internal.net/salus.yaml

82. LONDON 18-19 OCT 2018
Salus Configuration
● But what if we need to allow an exception to the global policy for just one repo?
docker run
--rm
-v $(pwd):/home/repo
coinbase/salus --config https://internal.net/salus.yaml

83. LONDON 18-19 OCT 2018
Salus Configuration
● But what if we need to allow an exception to the global policy for just one repo?
● You can concatenate configuration files to allow for local customization.
docker run
--rm
-v $(pwd):/home/repo
coinbase/salus --config “https://internal.net/salus.yaml
file://tests/salus.yaml”

84. LONDON 18-19 OCT 2018
Building a custom Salus

85. LONDON 18-19 OCT 2018
Building a custom Salus
Dockerfile

86. LONDON 18-19 OCT 2018
Building a custom Salus
Dockerfile
your_scanner.rb

87. LONDON 18-19 OCT 2018
Building a custom Salus - provide custom messages for devs

88. LONDON 18-19 OCT 2018
Building a custom Salus - provide custom messages for devs

89. LONDON 18-19 OCT 2018
Salus Reports

90. LONDON 18-19 OCT 2018
Salus Reports
Reports include:
● Which scanners pass/failed
● Reasons for failure
● Which dependencies are present
(name + verison + source)
● Which Salus configuration they it used

91. LONDON 18-19 OCT 2018
Salus Reports
Reports include:
● Which scanners pass/failed
● Reasons for failure
● Which dependencies are present
(name + verison + source)
● Which Salus configuration they it used

92. LONDON 18-19 OCT 2018
Salus Reports
Reports include:
● Which scanners pass/failed
● Reasons for failure
● Which dependencies are present
(name + verison + source)
● Which Salus configuration they it used
STDOUT (default)

93. LONDON 18-19 OCT 2018
Salus Reports
Reports include:
● Which scanners pass/failed
● Reasons for failure
● Which dependencies are present
(name + verison + source)
● Which Salus configuration they it used
STDOUT (default)

94. LONDON 18-19 OCT 2018
Salus Reports
Reports include:
● Which scanners pass/failed
● Reasons for failure
● Which dependencies are present
(name + verison + source)
● Which Salus configuration they it used
STDOUT (default)

95. LONDON 18-19 OCT 2018
Salus Reports
TXT format to STDOUT (developer) JSON format for consumer

96. LONDON 18-19 OCT 2018
Salus Reports
Screenshot of Kibana displaying the results of Salus scans

97. LONDON 18-19 OCT 2018
More resources
● Github: coinbase/salus
● Docker Hub: coinbase/salus
● Blog post: https://blog.coinbase.com/engineering/home

98. LONDON 18-19 OCT 2018
Why is Salus useful?
Latest Salus
Pass
Fail
Source Code
Container Registry or
Configuration Host

99. LONDON 18-19 OCT 2018
Why is Salus useful?
Latest Salus
Pass
Fail
Source Code
Container Registry or
Configuration Host
Security team can keep up
metrics

100. LONDON 18-19 OCT 2018
Why is Salus useful?
Latest Salus
Pass
Fail
Source Code
Container Registry or
Configuration Host
Quick developer
feedback loop
Security team can keep up
metrics

101. LONDON 18-19 OCT 2018
Thank you:
● Developers of open source scanners
● Ryan Sears, Adam Richardson, Slava
Kim - all contributors of Salus
● DevSecCon Organizers