1.
mrys@microsoft.com

2.


3.
MySpace: the Business Problem 223M users 900 Terabytes of data 450 SQL Servers Required (eventual) data consistency across databases

4.
MySpace’s Data Consistency Problem 1-1000 1001-2000 2001-3000 3001-4000 4001-5000 5001-6000

5.
MySpace’s Solution Propagate data changes from one DB to other DBs using reliable, async Message Service (Service Broker) And also used for

6.
MySpace’s Service Dispatcher Coordination point between all SQL Servers Load-balanced across 30 SQL Servers Enables multicast/broadcast functionality 18,000 ~2k msgs/sec per dispatcher SQL Server

7.
MySpace Architecture1-1000 3001-4000 My DB I change gets updated my status Service Service TX3 Broker TX2 TX1 Dispatcher Service userId=1024 Broker2001-3000 Service 1001-2000 Broker TX4 TX54001-5000 5001-6000 Web Tier Data Tier

8.
Many other customers using similarpatterns Online electronic stores (cannot give names ) Travel reservation systems (e.g. Choice International)

9.


10.
Social Social Networks Network Management s Ops Services Data Backend Data Backend Services Data Backend Services Services MSN MSN GamesGames l Web Portal Front Door GameBar Router WLM Services WLM Games HostGames Web Portal Bing Bing Auth Games STS STSGames l Web Portal Azure Data Centers

11.


12.
•••

13.
Partitioned over 100 SQL Azure DBs Find Friends’ Profiles Social Get my Profile Social Services User … DB Service Publish feed, read feed Find Friends’ Profiles Get Friends highscores Gamer Last Played Gamer Services Favorites STS Services Game Preferences STS Leaderb … Social Leaderboards DB oard Partitioned over 298 SQL Azure DBs Game Game Disable/Enable Ingestion Games from Front Door Write user specific game infos Ingestion accessing services Router Services Game Game binaries User … DB Catalog Game metadata 250 instances Partitioned over 100 SQL Azure DBs 250 instances

14.
• Fanout: Parallel calls to multiple database partitions• Quorum: Able to tolerate a percentage of request failures during Fanout• Retry: Retry on database requests error

15.
Statistics per game

16.


17.


18.


19.


20.


21.
Readable Replica Primary Shard Readable Replica Dynamic OLAP Workloads (Scale-out queries,OLTP Workloads often using(mostly touching 1 to eventual consistent Readable scale-outlow number of shards) Replica frameworks like Hadoop) Primary Shard Readable Replica Readable Replica Primary Shard Readable Replica

22.


23.
 Federation Azure DB with Federation Root Represents the data being sharded Federation Root Federation Directories, Federation Users, Database that logically houses federations, contains federation meta data Federation Distributions, … Federation Key Value that determines the routing of a piece of data (defines a Federation Distribution) Federation Member (aka Shard) Federation “Orders_Fed” A physical container for a set of federated (Federation Key: CustomerID) tables for a specific key range and reference tables Member: PK [min, 100) Federated Table Table that contains only atomic units for the AU AU AU member’s key range PK=5 PK=25 PK=35 Reference Table Non-sharded table Atomic Unit Member: PK [100, 488) All rows with the same federation key value: always together! AU AU AU Connection PK=105 PK=235 PK=365 Gateway Member: PK [488, max) AU AU AU Sharded PK=555 PK=2545 PK=3565 Application 25

24.
•••

25.


26.
 http://www.microsoft.com/casestudies/Case_Study_Detail.aspx?CaseSt udyID=4000008310 http://cacm.acm.org/magazines/2011/6/108663- scalable-sql http://download.microsoft.com/download/9/E/9/9E9F240D-0EB6-472E- B4DE- 6D9FCBB505DD/Windows%20Azure%20No%20SQL%20White%20Paper.p df http://blogs.msdn.com/b/cbiyikoglu/archive/2011/03/03/nosql-genes-in- sql-azure-federations.aspx mrys@microsoft.com http://sqlblog.com/blogs/michael_rys/default.aspx

27.
Connection: Server=az1cl321.db.windows.net; Database=MyDB; User=AppUser; Passwd=****; Gateway CREATE FEDERATION Existing Database CREATE FEDERATION sales (customer_id bigint RANGE)30

28.
Federation with a Single Shard Existing Database CREATE FEDERATION sales (customer_id bigint RANGE) Database root contains: sales • Federation root = DB level object containing federation scheme Gateway • Federation users • Federation metadata incl. federation Connection: mapServer=az1cl321.db.windows.net; Range: Min...MaxDatabase=MyDB;User=AppUser;Passwd=****; Federation Member 31

29.
Introducing Two Connection Modes • Filtered Connection – Guarantees that any queries or DML will produce the same results independent of changes to the physical layout of the federation members – Scoped to an “Atomic Unit” • Unfiltered Connection – Scoped to a Federation Member – Management Connection 32

30.
Create Schema on Member: Management Connection Existing Database sales Gateway USE FEDEDERATION sales (customer_id=0) Connection: WITH FILTERING=OFF, RESET;Server=az1cl321.db.windows.net; CREATE TABLE … Range: Min...Max Customer Order ProductDatabase=MyDB;User=AppUser;Passwd=****; Federation Member © 2011 Microsoft Corporation. Microsoft Materials - Confidential. All rights reserved. CITA # 33 MSFT101120_A

31.
DDL CREATE TABLE customer ( c_id bigint PRIMARY KEY, … ) FEDERATED ON (customer_id=c_id); CREATE TABLE order ( item_num int, customer_id bigint, date_sold datetime2, …, CONSTRAINT PK_Order PRIMARY KEY (item_num, customer_id, date_sold), CONSTRAINT FK_Cust FOREIGN KEY customer_id REFERENCES customer (customer_id) ) FEDERATED ON (customer_id=customer_id); CREATE TABLE product ( product_name varchar(100) NOT NULL, unit_price money, item_num int PRIMARY KEY, … ); 34

32.
More Detail • Supported data types for federation key : bigint, int, GUID, and varbinary (900) – Only range partitioning • Federation key must be part of unique index • Foreign key constraints only allowed between federated tables and from federated table to reference table • Not all Azure programmability features supported – Sequence, timestamp • Additional surface area restrictions – Indexed views, drop database (members) • Schemas are allowed to diverge over time – Furthermore, in v1, schema updates to existing members must be done in each member (where the change is needed) • USE FEDERATION “rewires a connection” – Connection is reestablished – All existing settings and context of the connection is lost (sp_reset_connection) – Must be in a batch by itself 35

33.
Connect to Atomic Unit: Filtered Existing Database When using into a specific key value, SELECT will only return sales records from federated tables that match that value. It will still return Gateway all records from non-federated tables. USE FEDEDERATION sales (customer_id=3) Connection: Inserts and UPDATES operating WITH FILTERING=ON, RESET; outside of the value will fail.Server=az1cl321.db.windows.net; Range: Min...Max customer order product 3Database=MyDB; SELECT * from customer 3User=AppUser;Passwd=****; SELECT * from product SELECT * from order Federation Member 36

34.
More on Connection Filtering • Most operations behave differently in filtered vs unfiltered connections • Connection filtering is a property of the session – Filter injected dynamically at runtime – Cannot inspect source code to determine how it behaves • E.g., running stored proc written for filtered mode on unfiltered connection could lead to unintended results • There are several operations that will not work in filtered connection in v1 – DDL, DML on reference tables, … • Fan-out, bulk operations not efficient in filtered mode – For now, filter=off is our best offer 37

35.
Support Matrix Connection Type Filtered Unfiltered Named Operation (unfiltered) Dynamic SELECT P P P DML* (federated tables) P P P DML* (reference tables) X P P DDL X P P Views (not indexed) P P P UDF - activate P P P Stored Proc - activate P P P Trigger (all modes) - activate P P P CREATE/UPDATE Stats X P P Bulk Ops openrowset bulk, bcp, bulk insert X P P * not including SELECT & modules ^ autostats will work on all connections System stored procs, intrinsics will be unaffected (run unfiltered) 38

36.
Splitting a Member Existing Database ALTER FEDERATION sales sales SPLIT AT (customer_id=50) Gateway USE FEDERATION ROOT Connection: WITH RESET Using to theServer=az1cl321.db.windows.net; federation ROOT will pop you out of a member back Range: Min...Max into the database that hosts the customer order product federation 3 3Database=MyDB;User=AppUser;Passwd=****; 40 58 58 58 Federation Member 39

37.
Two New Members Existing Database ALTER FEDERATION sales sales SPLIT AT (customer_id=50) Gateway USE FEDEDERATION ROOT WITH RESET Connection:Server=az1cl321.db.windows.net; Range: Min...50 Range: 51...Max customer order product customer order product 3Database=MyDB; 3User=AppUser; 58Passwd=****; 40 58 58 Federation Member Federation Member 40

38.
Two New Members Existing Database sales Gateway USE FEDEDERATION sales (customer_id=40) Connection: WITH FILTERING=ON, RESET;Server=az1cl321.db.windows.net; Range: Min...50 Range: 51...Max customer order product customer order productDatabase=MyDB; SELECT *User=AppUser; 58Passwd=****; from customer 40 40 58 58 SELECT * from order 41