Access structured data in Azure Table storage

Dr. Eduardo Castro Martínez
Microsoft MVP
ecastro@mswindowscr.org
http://comunidadwindows.org
http://ecastrom.blogspot.com

Cloud
Application
Cloud Platform

Users Developers

SQL Azure
Cloud
Applications
Windows Azure
Windows Azure AppFabric

On-Premises
Applications

Windows Others

Application

Compute Storage
Config Fabric

…

SQL Azure

Applications
Windows Azure
AppFabric
Windows Azure

Applications

Windows Others

VMs VMs
HTTP/
HTTPS
Web Worker
Load Role Role
Balancer
IIS Instance Instance

Application

Compute Storage

Fabric

…

Storage
Web Role Worker Role
Instance Instance

Fabric Fabric
Agent Agent

Fabric
Controller

HTTP/
HTTPS
Blobs Tables Queues

Application

Compute Storage

Fabric

…

Tables Provide structured storage. A Table is
a set of entities, which contain a set of
properties

Queues

Blobs

Drives
(new)
11

Table Table Table ...

Entity Entity Entity ...

Storage Property Property Property
Accounts

Name Type Value

1) Receive Web Role Worker Role
work
main() 4) Do
ASP.NET, work
WCF, etc. { … }

2) Put 3) Get
message in message
queue from queue
5) Delete
message
from queue
Queue

Accounts Tables Entities

Email =…
Name = …

Email =…
Name = …

Genre =…
Title = …

Genre =…
Title = …

21

PartitionKey RowKey Timestamp ReleaseDate
PartitionKey RowKey Timestamp ReleaseDate
(Category) (Title)
(Category) (Title)
Action Fast & Furious
Fast & Furious …
… 2009
2009

Action The Bourne Ultimatum … 2007

…
…
…
… …
… …
…
Animation Open Season 2 … 2009
Animation Open Season 2 … 2009
Animation The Ant Bully … 2006
Animation The Ant Bully … 2006

…
PartitionKey …
RowKey …Timestamp …
ReleaseDate
(Category) (Title)
Comedy
Comedy
Office Space
Office Space
…
…
1999
1999

…
… …
… …… …
…

SciFi X-Men Origins: …… 2009
Wolverine
… … …… …

War
War
Defiance
Defiance …… 2008
2008

25

[DataServiceKey("PartitionKey", "RowKey")]
public class Movie
{
/// <summary>
/// Category is the partition key
/// </summary>
public string PartitionKey { get; set; }

/// <summary>
/// Title is the row key
/// </summary>
public string RowKey { get; set; }

public DateTime Timestamp { get; set; }

public int ReleaseYear { get; set; }
public string Language { get; set; }
public string Cast { get; set; }
}

27

StorageCredentialsAccountAndKey credentials = new StorageCredentialsAccountAndKey(
“myaccount", “myKey");
string baseUri = "http://myaccount.table.core.windows.net";

CloudTableClient tableClient = new CloudTableClient(baseUri, credentials);

tableClient.CreateTable(“Movies");

TableServiceContext context = tableClient.GetDataServiceContext();
CloudTableQuery<Movie> q = (from movie in context.CreateQuery<Movie>(“Movies")
where movie.PartitionKey == “Action" && movie.RowKey == "The Bourne Ultimatum"
select movie).AsTableServiceQuery<Movie>();
Movie movieToUpdate = q.FirstOrDefault();

// Update movie
context.UpdateObject(movieToUpdate);
context.SaveChangesWithRetries();

// Add movie
context.AddObject(new Movie(“Action" , movieToAdd));
context.SaveChangesWithRetries();

28

>
>
>
>

>
>
>

> (new)
>
>

29

>
>
>

>

PartitionKey RowKey Category …
(Account name) (Title)
> moviesonline 12 Rounds Action …

moviesonline A Bug’s Life Animation …

100,000,000 more rows … … …

moviesonline Office Space Comedy …

moviesonline Platoon War …

50,000,000 more rows … … …

moviesonline WALL-E Animation …

31

>
>
>

PartitionKey RowKey Category …
(Account name) (Title)
Client
Client moviesonline 12 Rounds Action …

moviesonline A Bug’s Life Animation …

100,000,000 more rows … … …

moviesonline Office Space Comedy …

moviesonline Platoon War …

50,000,000 more rows … … …

moviesonline WALL-E Animation …

32

>
>
> PartitionKey RowKey (Title)
(Category) (Title)
> Action Fast & Furious
… 10000 more Action movies
10000 more Action movies

Action The Bourne Ultimatum
… 100000 more Action & & Animation
100000 more Action Animation movies
movies
Animation Open Season 2
Animation
… Open Season 2
100000 more Animation movies
Client
Client
…
Animation
100000 more Animation movies
The Ant Bully
Animation The Ant Bully
Comedy Office Space
… 1000000more Comedy & & SciFi movies
1000000 more Comedy SciFi movies

SciFi Star Trek
… 100000 more SciFi & War movies
… 100000 more War movies
War
War Defiance
Defiance

33

>
>
>
> PartitionKey Properties
(Timestamp)
> 2009-11-15 02:00:01 …

> 2009-11-15 02:00:11 …

> 100000 more rows …

2009-11-17 05:40:01 …

2009-11-17 05:40:01 …

80000 more rows …

Applications 2009-11-17 12:30:00 …
Client 2009-11-17 12:30:01 …

2009-11-17 12:30:01 …

2009-11-17 12:30:02 …

2009-11-17 12:30:03 …

35

>
>
PartitionKey Properties
> (ID_Timestamp)
> 01_2009-10-12 05:10:00 …
…
01_2009-11-17 12:30:01 …
>
100000 more rows …
> 09_2009-11-15 12:31:00 …
>
…
09_2009-11-17 12:30:22 …

10_2009-10-05 05:10:10 …

Applications
Client 5000000 more rows …
…
15_2009-11-17 12:30:01 …
…
19_2009-11-17 12:20:02 …
19_2009-11-17 12:30:10 …

36

>
>
> PartitionKey
(Category)
RowKey (Title)

Action Fast & Furious
… 10000 more Action movies

Action The Bourne Ultimatum
… 100000 more Action & Animation movies

Animation Open Season 2
… 100000 more Animation movies

Client Animation The Ant Bully

Comedy Office Space
… 1000000 more Comedy & SciFi movies

SciFi Star Trek
… 100000 more SciFi & War movies
… 100000 more War movies
War Defiance

38

>
> PartitionKey
(Category)
RowKey
(Title)
Rating

>
Action Fast & Furious 5
> …
Returns 1000 rated > 4
999 more movies
movies …
> … Action and Anim. movies here with
rating < 4
…

Animation A Bug’s life 2
Partition range boundary hit
… 100 more movies < 4 here …
Animation The Ant Bully 3

Comedy Are we there yet? 2

… Return continuation
More movies here …

Client Comedy Office Space 5

… 800000 more movies here …
Drama A Beautiful Mind 5
… 1200000 more movies here …
War Defiance 4

39

>
>
>
>
PartitionKey RowKey Rating
> (Category) (Title)

> Action Fast & Furious 5
… More movies here …
Comedy Office Space 5
Documentary Planet Earth 4
… More movies here
Drama Seven Pounds 4

Horror Saw 5 3
Client … More movies here …

Music 8 Mile 2
SciFi Star Trek 5

40

1. Fast
>
2. Medium
>
>
3. Slow
>
>
>

> 1

41

Faster
>
>
> Create and maintain own views that help
queries

> “Or” Predicates
> Execute individual query in parallel instead of
using “OR”

> User Interactive
> Cache the result to reduce scan frequency

42

>
>
>
>
>

>
>
>
>
>

45

>
>
>
>

>
>
>
>

PartitionKey RowKey Kind TotalRentals Name Address CheckOutOn Title DueOn
(AccountID) (Kind_*)

… … … … … … … … …
Sally A_Sally Account 8 Sally Field Ann Arbor,
MI
Sally R_Jaws Rental 2009/11/16 Jaws 2009/11/20
Sally R_Taxi Rental 2009/11/16 Taxi 2009/11/20
… … … … … … … … …

46

>
>
>
>

>

>

>

> Future Feature - Secondary Index

>

>
47

Accounts Queues Messages

49

Message now has dequeue count

50

CloudQueueClient queueClient = new CloudQueueClient(baseUri, credentials);
CloudQueue queue = queueClient.GetQueueReference("test1");

queue.CreateIfNotExist();

//MessageCount is populated via FetchAttributes
queue.FetchAttributes();

CloudQueueMessage message = new CloudQueueMessage("Some content");
queue.AddMessage(message);

message = queue.GetMessage(TimeSpan.FromMinutes(10) /*visibility timeout*/);

//Process the message here …

queue.DeleteMessage(message);

52

Producers Consumers

P2 C1 1. GetMessage(Q, 30 s)  msg 1

4 3 2 1
30 1 1
0 0 0

C2
P1
2. GetMessage(Q, 30 s)  msg 2

53

Producers Consumers
1 C1 1. GetMessage(Q, 30 s)  msg 1
P2 1
5. C1 crashed

4 3 2 1
3 6. msg1 visible 30 s after Dequeue
0 0 1 2
1

2
1 C2
P1
3. C2 consumed msg 2
4. DeleteMessage(Q, msg 2)

54

Producers Consumers
1. Dequeue(Q, 30 sec)  msg 1
P2 C1 5. C1 crashed
10. C1 restarted
4 33 1 12. DequeueCount > 2
0 0 3
2 13. Delete (Q, msg1)
1 C2
P1 2

6. msg1 visible 30s after Dequeue
2. Dequeue(Q, 30 sec)  msg 2 9. msg1 visible 30s after Dequeue
3. C2 consumed msg 2
4. Delete(Q, msg 2)
8. C2 crashed
55

>
>

>
>

>
>

>

>
>
>
>

56

>
>
>
> (new)
>
>
> (new)
>

> (new)
57

A relational DB in the cloud

SQL Azure Database

Data Hub

Others (Future)

.NET Services

Applications SQL Services
Relational database as a service
Highly available, automatically maintained
Live Services
Windows Azure

Extension of the SQL Server Data Platform
Applications

Windows Windows Windows Others
Server Vista/XP Mobile

Single Logical Multiple Physical
Database Replicas

Replic
a1

Replic
D a2
B

Replic
a3

Reference Data Sync
Data

Symmetric Programming Model Data Hub Aggregation

• Initial services – core RDBMS capabilities with SQL Azure Database,
Data Sync
• Future Offerings
• Additional data platform capabilities: Reporting, BI
• New services: Reference Data

Clear Feedback: “I want a database in the Cloud”

Focus on combining the best features of SQL Server
running at scale with low friction

Application Application Application
Browser Browser
ODBC, OLEDB,
Application Application
REST Client REST Client SQL Client* ADO.Net PHP,
Ruby, …

Cloud Cloud

Evolves

HTTP+REST
HTTP+REST

HTTP
HTTP

TDS
Windows Azure Windows Azure

Web App REST (Astoria) Web App
Data Center

Data Center
REST Client ADO.Net + EF SQL Client*

REST/SOAP + ACE Model TDS + TSQL Model

SDS Current SDS Next

* Client access enabled using TDS for ODBC,
ADO.Net, OLEDB, PHP-SQL, Ruby, …

SQL Azure Database

Others (Future)

SQL Azure

Applications
Windows Azure
AppFabric
Windows Azure

Applications

Windows Others

SQL Azure Database

TDS Database

Application
Database

Database

SQL Azure Database

Others (Future)

SQL Azure Database

Database
Application

Max database
Database size is 50 GB

Database
Application
Database

Applications use standard SQL client libraries:
ODBC, ADO.Net, PHP, …
Application

Internet
Azure
Cloud

TDS (tcp)
Security Boundary
Load balancer forwards „sticky‟ sessions to TDS
LB protocol tier

TDS (tcp)

Gateway Gateway Gateway Gateway Gateway Gateway

Gateway: TDS protocol gateway, enforces AUTHN/AUTHZ policy; proxy to CloudDB
TDS (tcp)

L SQL SQL SQL SQL

Scalability and Availability: Fabric, Failover, Replication, and Load balancing

Gateway Process

AdminSvc
TDS Endpoint Provisioning Endpoint
Endpoint

Protocol Parser

Business Logic Services

Connection Mgmt

L SQL SQL SQL SQL

Scalability and Availability: Fabric, Failover, Replication, and Load balancing

Live DNS Cluster
Customer
Browser Live DNS Svc

1

Datacenter (Sub-Region) 5
Portal LB Gateway LB
2
4
3 6

Front-end Node Front-end Node Front-end Node Front-end Node

Admin Portal Admin Portal Gateway Gateway

7

Backend Node Backend Node Backend Node

SQL Server SQL Server SQL Server
Mgmt. Mgmt. Mgmt.
Services Fabric Services Fabric Services
Fabric

1
TDS Gateway
Front-end Node
TDS Session Protocol Parser
2
3
Gateway Logic

Master Cluster

Master Node
4
Master Node
Components

7
5 8 6

Backend Node 1 Backend Node 2 Backend Node 3
SQL Instance SQL Instance SQL Instance
SQL DB SQL DB SQL DB

Scalability and and Availability: Fabric,Failover,Replication, and Load balancing
Scalability Availability: Fabric, Failover, Replication, and Load balancing

7
1
TDS Gateway
Front-end Node
TDS Session Protocol Parser
6 2
Gateway Logic

Global Partition Map

Master Node
8 3
Master Node
Components

4 5

Backend Node 1 Backend Node 2 Backend Node 3
SQL Instance SQL Instance SQL Instance
SQL DB SQL DB SQL DB

Scalability and and Availability: Fabric,Failover,Replication, and Load balancing
Scalability Availability: Fabric, Failover, Replication, and Load balancing

On Premises Windows Azure
Local Agent Data Sync Service
Sync Orchestrator Sync Orchestrator

HTTPS
SQL Server SQL Server SQL Server
Sync Provider Proxy Provider Provider

TDS

SQL
SQL
Server
Azure

SQL Azure

Now
Database

Data Sync
Service For Remote Offices

CTP2 – End of CY10
Retail Stores
SQL Azure

Sync

On-Premises (Headquarters)

Data Sync

Web Role

Reporting

Data Sync

Web Role
Data Sync
Reporting

Service Bus

Access Control ?

SQL Azure

Applications
Windows Azure
AppFabric
Windows Azure

Applications

Windows Others

Service Bus
3) Discover Service
Bus endpoint 1) Register service
Registry endpoint

2) Expose Service
Bus endpoint 5) Invoke operation
4) Invoke operation on service endpoint
on Service Bus
endpoint

Client
WCF Service
(On-premises
(On-premises)
or cloud)

Service Bus
Access Control

Blobs SQL Azure

On-Premises
or Hosted
Application

Tables

Web Role
Instance

Users

Tables Queues Blobs
Web Role Worker Role
Instance Instance

Users

SQL Azure
Web Role
Instance

Users

SQL Azure

Web Role
Service Bus
Instance

On-premises
Database

Users

Queues Blobs
Worker Role
Web Role Instance
Instance

User

http://comunidadwindows.org
http://www.slideshare.net/ecastrom
http://ecastrom.blogspot.com

http://go.microsoft.com/fwlink/?LinkId=158011

http://go.microsoft.com/?linkid=9682907

http://www.microsoft.com/sqlazure

http://blogs.msdn.com/sqlazure

http://msdn.microsoft.com/windowsazure/sqlaz
ure

http://msdn.microsoft.com/data

Access structured data in Azure Table storage

Access structured data in Azure Table storage

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (16)

Similar to Access structured data in Azure Table storage

Similar to Access structured data in Azure Table storage (20)

More from Eduardo Castro

More from Eduardo Castro (20)

Recently uploaded

Recently uploaded (20)

Access structured data in Azure Table storage