1. K.MohamedFaizal Lead Consultant @ NCS (P) Ltd. kmdfaizal@gmail.com http://faizal-comeacross.blogspot.com AzureUG.SG 26th Jan 2011. Exploring Windows Azure Cloud Storage

2. About Me 14

3. Agenda Cloud Storage Account Overview Tables Blobs Queues Storage Viewer / Explorer Tools

5. Tables – Hierarchical Data Tables

6. Queues – Asynchronous Messaging

7. Drives – NTFS file systems (Blob)Each item in storage is stored in triplicate.

8. Make a guess! What is the maximum size of data, a storage account can hold? 1 GB 100 GB 1 TB 100 TB Answer: d) 100 TB http://blogs.msdn.com/b/windowsazurestorage/archive/2010/05/10/windows-azure-storage-abstractions-and-their-scalability-targets.aspx

9. Windows Azure and SQL Azure

10. Creating a Storage Account

11. With client library Automatically referenced in a cloud project Supplies a nice .NET API on top of the REST API You will need the following: Account Name - movieconversion Account Shared Key – NOcqFlqUwJPLlm… Endpoint - blob.core.windows.net -or- UseDevelopmentStorage=true

12. Table Storage

13. Windows Azure Tables Provides Structured Storage Massively Scalable Tables Billions of entities (rows) and TBs of data Can use thousands of servers as traffic grows Highly Available & Durable Data is replicated several times Familiar and Easy to use API ADO.NET Data Services – .NET 3.5 SP1 .NET classes and LINQ REST – with any platform or language 10

14. Tables Is not relational Can Not- Create foreign key relationships between tables. Perform server side joins between tables. Create custom indexes on the tables. No server side Count(), for example. ADO.NET Data Service Creates and consumes data via REST-based API's The Table Service REST API implements the WCF Data Services API.

16. Table Data Model Table A storage account can create many tables Table name is scoped by account Set of entities (i.e. rows) Entity Set of properties (columns) Required properties PartitionKey, RowKey and Timestamp 14

17. Required Entity Properties PartitionKey & RowKey Uniquely identifies an entity Defines the sort order Use them to scale your application Timestamp Read only Optimistic Concurrency 15

18. PartitionKey And Partitions PartitionKey Used to group entities in the table into partitions A table partition All entities with same partition key value Unit of scale Control entity locality Row key provides uniqueness within a partition 16

19. Required Properties All entities must have the following properties: Timestamp PartitionKey RowKey PartitionKey + RowKey = “primary key”

20. Partitions and Partition Ranges Server A Table = Movies [Action - Comedy) Server A Table = Movies Server B Table = Movies [Comedy- Western) 18

21. Table Operations Table Create Query Delete Entities Insert Update Merge – Partial Update Replace – Update entire entity Delete Query Entity Group Transaction (new)

22. Creating a Table Tables can be created and destroyed in code. Wire up your storage in the on_start method. Use a CloudTableClient to access the table.

23. Entity: Inherit the Required Fields By inheriting from TableServiceEntity we can easily store our entity in an Azure Table.

24. Create a Context Class WCF Data Services needs a context class. This represents your connection to the table.

25. Adding an Entity Other operations are just like normal ADO.NET Data Services, such as delete, update, etc.

26. Deleting an Entity

27. Key Selection: Things to Consider Scalability Distribute load as much as possible PartitionKey is critical for scalability Hot partitions can be load balanced Query Efficiency & Speed Avoid frequent large scans Parallelize queries 25

28. Movie Listing – Solution 1 Why do I need multiple PartitionKeys? Account name as Partition Key Movie title as RowKey since movie names need to be sorted Category as a separate property Does this scale? 26

29. Movie Listing – Solution 1 Single partition - Entire table served by one server All requests served by that single server Does not scale Client Client Request Request Request Request Server A 27

30. Movie Listing – Solution 2 All movies partitioned by category Allows system to load balance hot partitions Load distributed Better than single partition Server A Client Client Request Request Request Request Request Request Request Request Server B 28

31. Key Selection: Case Study 2 Log every transaction into a table for diagnostics Scale Write Intensive Scenario Logs can be retrieved for a given time range 29

32. Logging - Solution 1 Timestamp as Partition Key Looks like an obvious choice It is not a single partition as time moves forward Append only Requests to single partition range Load balancingdoesnot help Server may throttle Server A Applications Client Server B Request Request Request Request 30

33. Logging Solution 2 - Distribute "Append Only” Prefix timestamp such that load is distributed Id of the node logging Hash into N buckets Write load is now distributed Better throughput To query logs in time range Parallelize it across prefix values Server A Applications Client Server B Request Request Request Request 31

34. Table Storage Demo demo

35. Overview Blob Storage

36. What is a BLOB Binary Large OBject

38. Job2.mpg

39. Header.png

40. Job 9.mpg

41. Index.docx

43. Windows Azure Storage Services New Features Blobs – Provide a simple interface for storing named files along with metadata for the file New Type of Blob Page Blob New Blob Operations Update Blob Copy Blob Snapshot Blob Lease Blob New Ways of Accessing and Serving Blob Content Custom Domain Names Root Blob Container Content Delivery Network access Shared Access Signatures Drives – Provides durable NTFS volumes for Windows Azure applications to use (Feb 2010)

44. Overview Queue Storage

45. What is a queue? Queues are used to store messages They are FIFO, and one way A queue name must be lower case name, and URL friendly Message Queue Account customer ID order ID http://… order processing cohowinery customer ID order ID http://…

46. Account, Queues and Messages An account can create many queues Queue Name is scoped by the account A Queue contains messages No limit on number of messages stored in a queue Set a limit for message expiration Messages Message size <= 8 KB To store larger data, store data in blob/entity storage, and the blob/entity name in the message Message now has dequeue count 40

48. What Next Our next UG Meeting on 30th March 2011 Microsoft Office Level 21, Singapore

49. Storage Viewer / Explorer Tools Azure Storage Explorer http://azurestorageexplorer.codeplex.com/ Myazurestorage.com https://myazurestorage.com/ Cloud Xplorer http://clumsyleaf.com/products/cloudxplorer Cerebrata Cloud Storage Studio http://www.cerebrata.com/products/cloudstoragestudio/