Soft multi-tenancy can be hard to achieve and secure. Multiple tenants sharing the same cluster means there are global objects, like Custom Resource Definitions (CRDs), namespaces, and so on, that you don’t want tenants controlling. Platform admins, cluster admins, and tenants, should be separated, with dedicated namespaces, role bindings, node groups, taints and tolerations, etc. With Flux, tenant isolation is enforced by default, so you don’t have to worry about accidental tenant cross-over / cross-contamination. In this session, Priyanka “Pinky” Ravi, Developer Experience Engineer at Weaveworks, will walk you through how to set up multi-tenancy on an existing Kubernetes cluster and manage several tenants within the cluster. Take advantage of the benefits that come with infrastructure as code.

1. 1
1
Implementing Flux for Scale
with Soft Multi-tenancy
Russ Parmer
Senior Engineer, Weaveworks
Priyanka Ravi
Developer Experience Engineer, Weaveworks

2. 2
2
Weaveworks is founded on open source
● Flux & Flagger (CNCF): GitOps and Progressive Delivery for k8s
● Weave GitOps: A powerful extension & web UI for Flux
● Weave GitOps Terraform Controller: Flux controller for Terraform
resources
● GitOps Tools for Flux: VS Code extension
weave.works

3. 3
3
● Operating model for cloud native applications such as Kubernetes
● Utilizes a version controlled system (Commonly Git) as the “single
source of truth”
● Enables continuous delivery through automated deployment,
monitoring, and management by a version controlled system
● Managing your infrastructure and applications declaratively
What is GitOps

4. Source: GitOps Working Group
https://opengitops.dev/

5. 5
5
Individuals, teams, and organizations who implement GitOps experience
many benefits, including:
● Stronger Security Guarantees
● Increased Developer & Operational Productivity
● Enhanced Developer Experience
● Improved Stability
● Higher Reliability
● Consistency and Standardization
Why GitOps

6. 6
6
● A git centric package manager for your applications
● A set of continuous and progressive delivery solutions for Kubernetes
What is Flux
fluxcd.io

7. 7
7
🤝 Flux provides GitOps for both apps and infrastructure
🤖 Just push to Git and Flux does the rest
🔩 Flux works with your existing tools
☸ Flux works with any Kubernetes and all common Kubernetes tooling
🤹Flux does Multi-Tenancy (and “Multi-everything”)
📞 Flux alerts and notifies
👍 Users trust Flux
💖 Flux has a lovely community that is very easy to work with!
Flux in Short fluxcd.io

8. 8
8
● Reduces developer burden
● Extensible
● Comes with out of the box support for Kustomize and Helm
● Designed For Kubernetes
Benefits of Flux
fluxcd.io

9. 9
9
Overview of Flux
Source
controller
Notification
Controller
Image Reflector
& Automation
Controller
Flux
Flux is a set of
Kubernetes Controllers
fluxcd.io
Terraform
Controller
Helm
Controller
Kustomize
controller
VS Code
Extension

10. 10
10
What Flux’s Controllers do
Source Controller
- Fetch resources and store as
artifacts
Kustomize Controller
- Apply manifests, Run
manifest generation using
kustomize
Helm Controller
- Deployment of Helm Charts
Notification Controller
- Notification Dispatch
Image Reflector Controller
- Reflects Image metadata for
Automation Controller
Image Automation Controller
- Updates YAML when new
container images are
available
fluxcd.io

11. 11
11
● Helm
● Kustomize
● Prometheus
● Grafana
● Jenkins
● EKS
● AKS
● GCP
Flux Works with Other Tools
● Traefik
● Falco
● GitHub, GitLab, Bitbucket,
s3-compatible buckets
● Terraform
● …and more!!!
fluxcd.io

12. 12
12
● Makes life easier
● Multi-tenancy
● DependsOn
● Helm integration
● Notifications and Alerts
● Bootstrap
● Flux CLI
Reasons I and Others Love Flux fluxcd.io

13. 13
13
● There are two different forms of multi-tenancy
○ Hard multi-tenancy
■ Every tenant has their own cluster
○ Soft multi-tenancy
■ A cluster is shared across many different tenants
■ Tenants need to be isolated
What is Multi-Tenancy

14. 14
14
● Multi-tenancy lock down*
○ Ensure relevant controllers have cross namespace references disabled via
`--no-cross-namespace-refs=true`
● Resource Isolation
○ Ensure additional Flux instances are deployed when mission critical
tenants/workloads must be assured.
● Node Isolation
○ Ensure worker nodes are not being shared across tenants and the Flux
components.
● Network Isolation
○ Ensure the Container Network Interface (CNI) being used in the cluster
supports Network Policies.
Best Practices for Multi-Tenancy

15. 15
15
● Add `--no-cross-namespace-refs=true`
flag
○ Allow Flux to only reconcile Flux
resources that exist in the same
namespace
● Add `--default-service-account=default`
flag
○ Default Kustomization and HelmRelease
objects to not use the cluster scoped
service account
● Set the flux-system Kustomization
resource to use correct service account
○ This still needs cluster level access and
should not use the ‘default’ service
account
Enable Tenant Isolation for Flux

16. 16
Confidential do not distribute 16
Demo Time!

17. 17
17
Next Steps & Resources
● Try it yourself! Flux Docs: fluxcd.io/flux
○ Flux Multi-cluster setup:
■ https://fluxcd.io/flux/get-started/#multi-cluster-setup
■ https://github.com/fluxcd/flux2-kustomize-helm-example
○ Additional Best Practices for Shared Cluster Multi-tenancy:
https://fluxcd.io/flux/security/best-practices/#additional-best-practices-for
-shared-cluster-multi-tenancy
○ Flux Bootsrap Cheatsheet: https://fluxcd.io/flux/cheatsheets/bootstrap
● Kyverno Docs to Generate Flux Multi-tenant Resources
○ https://kyverno.io/policies/flux/generate-flux-multi-tenant-resources/gene
rate-flux-multi-tenant-resources/

18. 18
Confidential do not distribute 18
weave.works
Thank you