3. 3
● What is a secure software supply chain?
● Grafeas approach
● How Microservices Platform is using Grafeas and Kritis to secure
microservices Continuous Delivery?
Agenda
4. 4
In general, a secured software supply chain means that only
authorized softwares are executed at each supply chain stage
resulting in only authorized software getting deployed in production
environment.
Definition of “authorized” varies from organization to organization
depending on security policies and their strictness.
What is a Secure Software Supply Chain?
5. 5
Some policy examples:
● Only vulnerability scanned and passed docker images are allowed
● Only QA verified software artifacts are allowed
● Only PM signed off software artifacts are allowed
● Only software artifacts which are passed during canary stage are
allowed
7. 7
In software supply chain stages, every stage consume some artifact
and produce a new one. “By authorized at each stage”, it means that
each stage is consuming a verified artifact based on organization
policy.
What does authorized at each stage means?
8. 8
● Only commits from authorized authors are allowed for build stage
● Only unit / integrations tests passed artifacts are allowed for QA or
end to end testing
● Only unit / integrations tests passed artifacts are allowed for security
scanning
Some examples:
9. 9
Traditionally, it is achieved by creating a process workflow or
orchestration where each stage will run only when previous stage is
passed. This can be done in various CI/CD tools or bots.
One of the problem with this approach is that it is hard to govern
whole supply chain from bird eye point of view. Metadata created by
one stage are lost after next stage has consumed it
How to achieve secure software supply chain?
12. 12
Governance in software supply chain means that all the software
artifacts deployed in production can be tracked. CTO/CIO has full
visibility and can make policy changes.
Software supply chain should be designed in such a way that new
security policies can be added easily in whole organisation
Security is something which should be forced from top-down IMO
What does governance means in software supply
chain?
13.
14.
15. 15
CTO / CIO need to make this big decision
● Do not make any change and just hope that we are running good
code 🤞
● Make the change and let performance degrade by 40%
Without data it is very difficult to make these kind of decisions.
Software supply chain should be designed in a way that we can do
these kind of analysis and make better action plans
Untrusted Code? 🤔
16. 16
Take Away
● Now we understand what is a secure software supply chain
● Why centralized governance is required in software supply chain
17. 17
● Growing and fragmented toolsets
● Microservices Architecture
● Open-Source software adoption
What are the current problems?
18. 18
Grafeas approach
“An open artifact metadata API to audit and govern your software
supply chain”
Basically, instead of just using supply chain result metadata for next
stage, store them in a metadata server so that, that information is not
lost and can be used anytime in any stage.
19.
20. 20
Grafeas Components
Metadata Server
metadata server which store all the metadata information generated during various
supply chain stages
Policy enforcement tool
tool which uses grafeas metadata using API and make decision based on configured
policy. It is not exactly a grafeas component but without this supply chain will not be
complete
21. build stage uses grafeas API
to check if commit is allowed
to build or not
deploy stage uses grafeas API
to check if artifact is QA
verified and have not
vulnerability before deploying
22. 22
Take away!
● Using grafeas, we can store metadata of all supply chain stages
in one centralised database
● Using policy enforcement tool, each stage can verify if incoming
artifact is authorized or not
23. How microservices platform team is using
Grafeas and Kritis to secure microservices
Continuous Delivery?
24. 24
Microservices Platform Architecture
● We use GCP
● Each microservice has its own dedicated GCP project and they
are free to choose any service such as cloudsql, pub-sub etc
● We have a centralised GCP project which is managed by platform
team and we run GKE there
● Each microservice has its own namespace and microservice
owner only have access to their namespace
● We use Spinnaker for Continuous Delivery
25. 25
GCP project for GKE
Centralized cluster
Namespace: Service A
Namespace: Service B
IAM: Platform Team IAM: Team A
IAM: Team B
Service A
Service B
RBAC: Team A
RBAC: Team B
26. PM will confirm if feature is
ready to release and ask
developers to create tag
QA will notify PM regarding
QA results
QA get notified and will do
QA if necessary in dev
environment
if image-tag has some
version, it will trigger deploy
image to production cluster
if image-tag has master-*
prefix, it will deploy image to
development cluster
spinnaker start deployment
pipeline based on image tag
docker image is pushed to
GCR
cloudbuild triggers if tests
passes and docker image is
built
developers push code to
github (branch, master, tag)
Software Supply Chain for Microservice
circleci is triggered and it runs
unit tests
27. PM will confirm if feature is
ready to release and ask
developers to create tag
QA will notify PM regarding
QA results
QA get notified and will do
QA if necessary in dev
environment
if image-tag has some
version, it will trigger deploy
image to production cluster
if image-tag has master-*
prefix, it will deploy image to
development cluster
spinnaker start deployment
pipeline based on image tag
docker image is pushed to
GCR
cloudbuild triggers if tests
passes and docker image is
built
developers push code to
github (branch, master, tag)
Software Supply Chain for Microservice
circleci is triggered and it runs
unit tests
28. For secure microservices Continuous Delivery, we want to make sure that
only authorized docker images get deployed to our production cluster
only authorized
docker images
29. 29
What is authorized docker image for us?
● Image is built by service owner
● Image has been verified by security team for vulnerabilities
● Image has been signed by QA
● Image has been signed by PM
● …
30. 30
● We use GCP Container Analysis API which is implementation of
Grafeas
● We use Kritis fork which is basically Policy Enforcement Tool for
Kubernetes
○ Kritis is basically a Kubernetes admission webhook which get triggered
whenever a new pod is created. Based on policy written in
`ImageSecurityPolicy` CRD kirtis make judgement of allowing or denying that
pod. Kritis uses container analysis API to get image metadata
How we do this?
31. Security Team
Signer Service
deploy new docker image in
echo-namespace with digest
(sha256:abcd…)
Cloud
Build
GCR
docker image
Container Analysis API
BUILD
(Occurrence)
VULNERABILITY
(Occurrence)
ATTESTATION
(Occurrence)
microservice GCP Project GKE Cluster GCP Project
Admission Controller
kritis-namespace echo-namespace
kritis validation
webhook
ImageSecurity
Policy
echo pod
OK to deploy
1
2
6
5
4
8
7
3
QA Team Signer
Service
32. 32
How it works?
1. Cloudbuild build docker image and push it to GCR. It also create a
build metadata with information such as which project was used
to build image
2. GCR triggers spinnaker
3. Spinnaker apply kubernetes manifest file with new docker image
tag. This goes to kubernetes admission controller
4. Admission controller call kritis validation webhook with new pod
information
5. Kritis admission controller check `ImageSecurityPolicy` CRD from
echo-namespace and get current policy
33. 33
6. Kritis calls container analysis API and get metadata for that docker
image
7. Depending on the policy, Kritis validate if image is allowed or not
and inform admission controller
8. Depending on Kritis results, admission controller allow to create
new pod or not
How it works?...
34. 34
Signer Service
Signer Services run independently and sign images based on their job.
For example, security signer service confirm all vulnerabilities or
check what base image is being used and sign it based on their policy.
Similarly, QA signer service can also be made which sign images once
their job is done.
35. 35
Why FORK?
● Official kritis is still at very early stage and does not have feature
which we wanted such as:
○ Validate GCPProjectId where docker image is built
○ Custom attestation check
○ Clusterwide image whitelist
https://github.com/mercari/kritis
36. 36
Ending
Using Grafeas and Kritis, we have introduced a new supply chain
governance strategy in our eco-system. Our experience is good so far.
We want to write more signer services and add grafeas in more
supply chain stages and add full observability for our software supply
chain
