In the real-world there are 10000s of B2B companies. Their app-stack fits the multi-tenant model - each tenant(customer) deals with it’s own data. It is super critical to build scalable applications which gives the company leeway to grow as more customers get on-boarded. Let’s learn how to do that!
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
Scaling Multi-tenant Applications Using the Django ORM & Postgres | PyCon Canada 2018 | Sai Srirampur
1. Sai Srirampur | PyConCA 2018Sai Srirampur | PyConCA 2018
Scaling Multi-Tenant
Applications Using the
Django ORM & Postgres
Sai Srirampur
PyCon Canada | Toronto | Nov 2018
2. Sai Srirampur | PyConCA 2018
• Sai Srirampur a.k.a Sai
• Engineer at Citus Data
• Joined Citus to make it so
developers never have to
worry about scaling their
database
• Creator django multi-tenant
• Follow me @saisrirampur
@citusdata
6. Sai Srirampur | PyConCA 2018Sai Srirampur | PyConCA 2018
Why 💙 Postgres? TLDR;
Open source
Constraints
Extensions
PostGIS / Geospatial
HLL, TopN, Citus
Foreign data wrappers
Rich SQL
CTEs
Window functions
Full text search
Datatypes
JSONB
7. Sai Srirampur | PyConCA 2018
Today… Scaling Multi-Tenant Apps
Using the Django ORM & Postgres
3 architectures to build Multi-tenant apps
django_multitenant
Scaling multi-tenant apps with distributed
postgres
8. Sai Srirampur | PyConCA 2018
• Multiple customers
(“tenants”)
• Each with own data
• SaaS
• Shopify, Salesforce
Multi-Tenant Apps
Sai Srirampur | PyConCA 2018
9. Sai Srirampur | PyConCA 2018Sai Srirampur | PyConCA 2018
Why talk
about scalable
multi-tenant
applications?
tens
100’s
1000’s
10. Sai Srirampur | PyConCA 2018Sai Srirampur | PyConCA 2018
Architectures
To
Build
Multi-Tenant
Applications3
11. Sai Srirampur | PyConCA 2018
[1] One database per
tenant
Sai Srirampur | PyConCA 2018
12. Sai Srirampur | PyConCA 2018
What defines a Database?
• Organized collection of interrelated data
• Don’t share resources.
• Username and password
• Connections
• Memory
13. Sai Srirampur | PyConCA 2018
One database per tenant
Tenant 1251 Tenant 1252
Tenant 5
[1]
14. Sai Srirampur | PyConCA 2018Sai Srirampur | PyConCA 2018
CREATE DATABASE tenant_100;
./manage.py migrate --database=tenant_100;
One database per tenant / Onboarding
database_routing: db_for_read, db_for_write etc.
15. Sai Srirampur | PyConCA 2018
● Start quickly
● Isolate customer (tenant) data
● Compliance is a bit easier
● Time for DBA/developer to manage
● Maintain consistency (ex: create
index across all databases)
● Longer running migrations
● Performance degrades as #
customers (tenants) goes up
PROS CONS
[1] One Database Per Tenant
Sai Srirampur | PyConCA 2018
16. Sai Srirampur | PyConCA 2018
[2] One schema per
tenant
Sai Srirampur | PyConCA 2018
17. Sai Srirampur | PyConCA 2018
What defines a database Schema?
• Logical namespaces to hold a set of tables
• Share resources:
• Username and password
• Connections
• Memory
18. Sai Srirampur | PyConCA 2018
One schema per tenant
Tenant 5 Tenant 1251
Tenant 1252
Database
[2]
19. Sai Srirampur | PyConCA 2018Sai Srirampur | PyConCA 2018
CREATE schema tenant_100;
./manage.py migrate --schema=tenant_100;
app_code for schema routing based on tenant
One schema per tenant / Onboarding
20. Sai Srirampur | PyConCA 2018
● Better resource utilization vs. one
database per tenant
● Start quickly
● Logical isolation
● Hard to manage (ex: add column
across all schemas)
● Longer running migrations
● Performance degrades as #
customers (tenants) goes up
PROS CONS
[2] One Schema Per Tenant
Sai Srirampur | PyConCA 2018
21. Sai Srirampur | PyConCA 2018
[3] Shared table
architecture
Sai Srirampur | PyConCA 2018
24. Sai Srirampur | PyConCA 2018
● Easy maintenance
● Faster running migrations
● Best resource utilization
● Faster performance
● Scales to 1k-100k tenants
● Application code to guarantee
isolation
● Make sure ORM calls are always
scoped to a single tenant
PROS CONS
[3] Shared Table Architecture :-)
Sai Srirampur | PyConCA 2018
25. Sai Srirampur | PyConCA 2018
Comparing the 3 architectures for scaling
multi-tenant Django applications
ONE
DATABASE
PER TENANT
ONE
SCHEMA
PER TENANT
SHARED
TABLE
ARCHITECTURE
Database
Database
[1]
[2]
[3]
start quickly & isolation guarantees,
bad resource utilization & not scalable
better resource util, logical isolation
not scalable
Scales to over 100K tenants!
explicitly handle isolation
27. Sai Srirampur | PyConCA 2018
django_multitenant to the rescue
Sai Srirampur | PyConCA 2018
28. Sai Srirampur | PyConCA 2018Sai Srirampur | PyConCA 2018
django_multitenant
Automates all ORM calls
to be scoped to a single tenant
29. Sai Srirampur | PyConCA 2018
Today...
Purchase.objects.filter(id=1)
<=>
"SELECT* from purchase where id=1"
30. Sai Srirampur | PyConCA 2018Sai Srirampur | PyConCA 2018
With django_multitenant
Purchase.objects.filter(id=1)
<=>
"SELECT* from purchase where id=1 and
store_id=<current_tenant>"
31. Sai Srirampur | PyConCA 2018
Components of
django_multitenant
Sai Srirampur | PyConCA 2018
32. Sai Srirampur | PyConCA 2018
django_multitenant usage — 3 steps
1. Inherit all models with TenantModel
2. Change ForeignKey to TenantForeignKey
3. Define tenant scoping: set_current_tenant(t)
33. Sai Srirampur | PyConCA 2018
TenantModel defines new Manager
Sai Srirampur | PyConCA 2018
35. Sai Srirampur | PyConCA 2018
set_current_tenant(t)
• Specifies which tenant the APIs should be scoped to
• Set at authentication logic via middleware
• Set explicitly at top of function (ex. view, external
tasks/jobs)
36. Sai Srirampur | PyConCA 2018
Example Code
Sai Srirampur | PyConCA 2018
37. Sai Srirampur | PyConCA 2018
Models without django_multitenant
class Purchase(models.Model):
store = models.ForeignKey(Store)
product_purchased = models.ForeignKey(Product)
ordered_at = models.DateTimeField(default=timezone.now)
billing_address = models.TextField()
Sai Srirampur | PyConCA 2018
38. Sai Srirampur | PyConCA 2018
Post django_multitenant
class Purchase(TenantModel):
store = models.ForeignKey(Store)
product_purchased = TenantForeignKey(Product)
ordered_at = models.DateTimeField(default=timezone.now)
billing_address = models.TextField()
Sai Srirampur | PyConCA 2018
39. Sai Srirampur | PyConCA 2018
Setting the tenant at authentication
class SetCurrentTenantFromUser(object):
def process_request(self, request):
if not hasattr(self, 'authenticator'):
from rest_framework_jwt.authentication import
JSONWebTokenAuthentication
self.authenticator = JSONWebTokenAuthentication()
try:
user, _ = self.authenticator.authenticate(request)
except:
return
try:
#Assuming your app has a function to get the tenant associated
for a user
current_tenant = get_tenant_for_user(user)
except:
# TODO: handle failure
return
set_current_tenant(current_tenant)
Sai Srirampur | PyConCA 2018
41. Sai Srirampur | PyConCA 2018
Benefits of django_multitenant
Drop-in implementation of shared tables architecture
Guarantees isolation
Ready to scale with distributed Postgres (Citus)