This is the Thomas Kejser and my 2010 SQL PASS Summit presentation on designing scalable and complex Analysis Services cubes.

1. SQLCAT: Designing scalable and
complex Analysis Service Cubes
Lessons learned from the Field
Denny Lee, Thomas Kejser
Microsoft SQL Customer Advisory Team

2. BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
Are you sitting in front of the correct
Denny? They look very much alike.
Designing scalable and
complex Analysis
Service Cubes: Lessons
learned from the Field –
Room 615-617
Getting SQL Service
Broker Up and Running
– Room 609

3. BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
SQL Server Customer Advisory Team (SQLCAT)
• Works on the largest, most complex SQL projects worldwide
• MySpace - 4.4 million concurrent users at peak time, 8 billion friend
relationships, 34 billion e-mails, 1 PetaByte store, scale-out using SSB
and SOA http://www.microsoft.com/casestudies/Case_Study_Detail.aspx?casestudyid=4000004532
• Bwin – Most popular European online gaming site – 30000 database
trans / second, motto: “Failure is not an option”; 100 TB total storage
http://www.microsoft.com/casestudies/Case_Study_Detail.aspx?casestudyid=4000004138
http://www.microsoft.com/casestudies/Case_Study_Detail.aspx?casestudyid=4000001470
• Korea Telecom - Largest telco in Korea serves 26 million customers;
3 TB Data Warehouse
http://www.microsoft.com/casestudies/Case_Study_Detail.aspx?casestudyid=4000001993
• Drives product requirements back into SQL Server from our
customers and ISVs
• Shares deep technical content with SQL Server community
• SQLCAT.com & http://blogs.msdn.com/mssqlisv

4. BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
SQL Server Design Win Program
• Target the most challenging and innovative SQL Server
applications
• 10+ TB DW, 3k/tran/s OLTP, Large 500GB+ Cubes, Competitive
migrations, Complex deployments, Server Consolidation (1000+)
• Invest in large scale, referenceable SQL Server projects
across the world
• Provide SQLCAT technical & project experience
• Conduct architecture and design reviews covering performance,
operation, scalability and availability
• Offer use of HW lab in Redmond with direct access to SQL Server
development team
• Work with Marketing Team Developing PR

5. BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
Upgrade Reliability Assurance
Regular status updates within one month of completing each
playback test cycle
Complimentary testing for qualified customer against every new
version of SQL Server
Opportunity to request improvements directly from the product
group for potential upgrade issues
Preparing for a smooth transition to the next version of SQL Server ...
Two-way NDA and Participation Agreement
to ensure security and privacy
For more information, contact sqlura@microsoft.com

6. BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
SQLCAT and SQL CSS Invite You…
ROOM 611
• To the SQL Server Clinic where the most experienced SQL Server experts in the
world will be waiting to talk with you.
• Bring your toughest Questions / Challenges to the experts who have seen it all
• Architect and Design your future applications with experts who have done it before
with some of the largest, most complex systems in the world
• Or just stop in to say hello!

7. Session Objectives and Takeaways
Session Objective(s):
• Provide the primary themes that revolve around
successful enterprise Analysis Services implementations
• Show the themes in relation to how SQLCAT customers
address these issues

8. Themes
• Get that hardware balance right (yes, we have to talk about IOPS)
• Repeat after me: partitioning, partitioning, partitioning!
• Multi-user query concurrency– how to handle it
• Keeping it simple
• Locking – how it works – and how to work around it
• What? Did you say ROLAP?

9. SQLCAT Case Studies
Generalized Learnings from the Field

10. BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
The Importance of I/O
• Enterprise = lots of data = lots of threads on the
storage engine
• Disk Partition Alignment Best Practices
• Scale Out Using Read-Only
• 64K vs. 32K block size
• Scale Out Query performance
• Key Point:
• Need to understand System Engineering concepts and
principles

11. BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
The Importance of I/O
Disk Partition Alignment
Disk Partition Alignment Best Practices
Latency (ms)
Duration (s)0
5
10
15
20
25
30
35
40
45
50
6 Disks - Not
Aligned 6 Disks - Aligned
8 Disks - Not
Aligned 8 Disks - Aligned
31
21 24
16
48.03
38.71
38.65
31.45
Latency (ms) Duration (s)
*Lower is better

12. BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
The Importance of I/O
SSAS I/O pattern
Typically Low block sizes on
scans – 32K
IOPS are wide and shallow
(lots of threads, short
queue)
Latency is key, low latency =
less context switching
Typical NAND device
speedup: 5-10x on SE
heavy queries 0
1
2
3
4
5
0
50000
100000
150000
200000
250000
0 32 64 96 128 160 192 224 256
Queue Depth (total outstanding kept constant at 256)
8K IOPS on FusionIO drive
IOPS
Latency (ms)

13. BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
Repeat after me: partitioning,
partitioning, partitioning!
• To get performance on multi-core boxes,
• Partition, partition, partition
• Query each partition, results in separate threads
• Distinct count
• Non-overlapping, continuous
• The importance of I/O is directly related to
storage engine queries

14. BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
Repeat after me: partitioning,…
Distinct Count performance
0
2
4
6
8
10
12
Physical Disk > Avg Disk Queue Length
Time x User (4) Time (23) Single
User Time x User (10) Time x Browser (4)
• Six different partitioning
schemes
• Server has 4 processors
• Lower / Shorter Duration Avg
Disk Queue Length
Type Ranking
Time x User (4) 2.13
Time 3.72
Single 4.64
User 2.25
Time x User (10) 2.97
Time x Browser 5.37

15. BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
Multi-user query concurrency
General rules
• Heap/Virtual Memory config settings
• MemoryHeapType = 2
• HeapTypeForObjects = 0
• Long running queries block processing
• It’s just time for a separation…
• Querying and processing servers
• Also importance of scale-out querying

16. BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
Multi-user query concurrency
Scale Out Analysis Services: Read Only
AS Processing Server
ASQueryLayer
AS database
AS Query 1
AS Query 2
AS Query 3
AS Query 4
AS database
Fast Disk (e.g. SAN)
Clients
Clients
Clients
NLB
SEQ
Synchronize
User Queries
User Queries
SE Query
1
2
3
4

17. BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
Multi-user query concurrency
Query performance
238.5634875
150.3148394
101.9169981
0
50
100
150
200
250
300
One Two Three Four
FE Heavy (80 cc users)
88.177996
25.12548352
102.0223114
28.35425396
0
20
40
60
80
100
120
One Two Three Four
SE Heavy
50 CC Users
60 CC Users

18. BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
Keeping it simple!
Careful with calculation scripts
• Don’t: One cube with calculation script thousands
of lines long
• Cell-by-cell mode WILL kill you
• Example from bank: speedup 1000x by rewriting
cell-by-cell to block style
When it’s large, careful with large features
• Keep Parent/Child small (tens of thousands)
• Keep M-M small (low millions max)

19. BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
What? Did you say ROLAP?!!
0 100000 200000 300000 400000 500000 600000 700000
-67158236
-2085251220
-458579892
-903461230
482449446
-1958890532
1101641660
-954727192
663180516
-808192897
2023856459
MOLAP
ROLAP
0 400000 800000 1200000
1757523554
-420451054
-1602608104
721132422
-889102219
990819614'
MOLAP
ROLAP

20. Case Study: AdCenter
How does AdCenter handle two 3.2TB (yes TB)
cubes?

21. BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
The Importance of I/O
EMC DMX V-Max to handle the I/O
• V-Max are dedicated to the application
• Hundreds of disks and spindles devoted to this process
• Short-stroking the disks to ensure fastest I/O
• Works closely with EMC directly (present at EMC World
regularly)
• Testing with EMC EFDs (Enterprise Flash Drives)
• Systems Engineering team dedicated to DW process
• Work closely with various vendors (EMC, HDS, etc.)
• References
• Accelerating Microsoft adCenter with Microsoft SQL Server 2008
Analysis Services.
• REAL PRACTICES: Performance Scaling Microsoft adCenter with
Microsoft SQL Server 2008 Analysis Services on EMC VMAX

22. BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
It’s all about I/O
View of adCenter Production
Storage Area Network
OLAP Processing Server
Windows Server 2003 x64 SP2
SQL Server Enterprise Edition
32 GB RAM, 8 Xeon procs (16 cores)
Staging Data Warehouse
Windows Server 2003 x64 SP2
SQL Server Integration Services
Network Load Balancing
Data Feeds
HBA BHBA A
Windows Server 2003 x64 SP2
SQL Server Analysis Services
64GB RAM, 8 Xeon procs (16 cores)
OLAP Standby Server
SAN
Fabric A
SAN
Fabric B
HBA BHBA A HBA BHBA A
Host Bus Adapters: 400 MB/sec each
HBA BHBA A HBA BHBA A HBA BHBA A
adCenter Production Environment
Windows Server 2003 x64 SP2
SQL Server Analysis Services
64GB RAM, 8 Xeon procs (16 cores)
OLAP LUNStandby OLAP LUN
19200 Max Reads
9600 Max Writes
DW LUN
180 300GB 10K Drives
RAID 1
19200 Max Reads
9600 Max Writes
180 300GB 10K Drives
RAID 1
2560 Max Reads
2560 Max Writes
32 300GB 10K Drives
RAID 1

23. BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
Repeat after me: partitioning, …
Facts Database
1 Partition per Day
31 Partitions, 1 Month of Data
ALTER PARTITION FUNCTION PerDay ()
SPLIT RANGE(CAST(CONVERT(varchar, GETDATE()+1, 112) AS int))
1 2 3 4 5 6 7 8 9 10 11
12 13 14 15 16 17 18 19 20 21 22 23
0
0 11 17 1 14 18 2 5 22 3 12 21
4 13 19 6 15 23 7 209 8 10 16
...
WHERE [date] = CAST(CONVERT(varchar, GETDATE(), 112) AS int)
AND [hour] IN (0, 11, 17)
8 Evenly Distributed Partitions per Day
3120 Partitions, 13 Months of Data
8 Parallel Partition
Processing Jobs
Current Day Partition Set
Current Day Partition
Cube

24. Case Study: E&D
Immense amounts of Xbox Live service
usage, purchase, and achievements data

25. BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
Partitioning Strategy
Evenly distributed, continuous, and non-overlapping
Rollups
June 2008
Mar 2010
...
Usage
-9223372036854775808 TO -8070015239897208167
-8070015239897208166 TO -6917692104582506052
8069877141170462663 TO 9223372036854775807
13 more...
• Monthly partitions
• Data further subdivided into 16
partitions each month (1 per core)
• Split based on SHA1 hash of userid
• Provides even distribution and
non-overlapping data spread

26. BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
The Importance of I/O
The effect of design and hardware
Day Week Month Quarter 7 months
Dev SSD 14 29 101 203 506
Dev HDD 14 29 104 610 1191
UAT SAN 9 73 445 1025 2800
V2 Cube, SSD 5 10 15 31 72
V2 Cube, HDD 5 7 30 244 540
0
500
1000
1500
2000
2500
3000
RunTme(seconds)
Amount of Data
Usage Cube Distinct Count Performance
select [Measures].[Online Unique Users] on 0,
[Date].[Date].[Date] on 1
from Usage
where
[Date].[Calendar Month].[Calendar Month].&[2009]&[1]

27. BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
Multi-User Query Concurrency
Using SSD to allow each server to handle more
simultaneous queries
0
500
1000
1500
2000
2500
3000
Run 6, One at a time Run 6, All at once Run 12, All at once
Runtime(seconds)
Access Pattern
Distinct Count Parallel Querying - Total Runtime
Dev HDD
Dev SSD
SQLCat SSD

28. Case Study: Yahoo!
Yes, I said 12TB cube

29. BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
Oracle 10gCDF
Ad Optimization Platform Architecture
SSAS Cube Builder
NAS
SSAS Query Servers
HW NLB
Users
Partition 1
Partition 2
Partition N
Partition 1
Partition 2
Partition N
1.2TB/day
File1
File2
FileN
50GB/hr
12TB
cube

30. Case Study – Large
Investment Banks

31. BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
Real time
Cube
Locking in Analysis Services – The Truth
you have to deal with
Process
Commit
Long Running Query
Perceived Runtime from User
TimeOut
New Perceived Runtime

32. BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
Relational locking
– same story, different engine
Important limitations in real time part of cube
• SWITCH/MERGE must take SCH-M lock (but held only a few ms)
• Can cause blocking queues
• If bulking into partitioned table – may cause a sort operations (use small batch sizes)
• Alternative Solution: partition real time part into a seperate table. Keep stale data in
partitioned tabled.
Table
2000
2001
2002LCK_S
LCK_SCH_M LCK_S
LCK_S
LCK_S
LCK_S
LCK_S
LCK_S
LCK_S
LCK_S
LCK_S
Long Running Query

33. BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
MOLAP Switching In Action
Basic idea:
• Use MOLAP for historical data
• Process latest MOLAP partitions often
Typical Latencies in minutes
Concerns:
• How to partition cube for fast, frequent MOLAP
processing
• Handling the processing lock when you need to
update the data

34. High Level Cube Partitioning
Partitions by BOTH time
and region
Full processing can be done
on ”active” partitions
• ”Roll with the daylight”
• Isolates changes and
reloads
Requirement: Must be able
to fully recalculate risk
2010-01-01
NY London Bombay Tokyo
2010-01-02
NY London Bombay Tokyo
2010-01-03
NY London Bombay Tokyo

35. Cube Flipping
Recall: Server level lock required to
process
Alternative Solution:
• Two copies of the cube, taking turns
processing
• ”Flip” between them
Two ways to flip:
• Use ASLB from CodePlex
• Write small Excel Plug-in
Excel Plug-in may be used when user
holds session state
• Session state automatically
refreshes when connectionstring is
changed
Process
Query
(Query)
(Process)

36. BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
ROLAP + MOLAP in Action
Basic ideas:
• Use MOLAP for historical/stale data
• Use ROLAP for frequently updated data
Typical latencies achievable are in seconds
Concerns/Challenges:
• Must maintain coherent ROLAP cache
• Use Real Time OLAP=true in connectionstring
• When MOLAP changes – must have minimum impact on others
• Consider archiving strategy
• Locking inside SQL Server must be handled (concurrent load). Consider
RCSI

37. BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
Real time Cube
Active Book
MOLAP
(Market
Snapshot)
ROLAP
Active Book
MOLAP
(Market
Snapshot)
ROLAP
Trader
Active Book
MOLAP
(Market
Snapshot)
ROLAP
Active Book
MOLAP
(Market
Snapshot)
ROLAP
x 25
Trader

38. | SQL Server Customer Advisory Team
Real Time
risk runs
(RCSI mode)
al time Cube
Active Book_X
MOLAP
(Market
Snapshot)
ROLAP
Market Snapshots (MOLAP Processing)
ROLAP_Book_X
Snapshot_X
x 25
x 25
TricklefeedETL

39. | SQL Server Customer Advisory Team
Real Time risk runs (RCSI mode)
Market Snapshots (MOLAP Processing)
ROLAP_Book_X
Snapshot_X
x 25
x 25
History
Day1
Empty
Day2
Day3
6 months
…
Merge
+
Agg
History
Cube

40. BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
Two Real Time Cube methods - Tradeoffs
Component MOLAP Switching ROLAP+MOLAP
Relation Tuning Low Must get right
AS Locking Must handle Not a big issues
Cache Usage Good Poor
Relational Concurrency N/A RCSI
Data Storage Best Compression ROLAP typically 2x MOLAP
Aggregate management Profiler + UBO Manual
Latency Minutes Seconds

41. Case Study: Sandvik
Finance
Designing scalable and complex Analysis Service Cubes

42. BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
Challenge: Financial Consolidation
Need: Writeback with 5 second latency
Need: 100% Uptime during month closing
Need: 300 truly concurrent users
Need: Dynamic scale as company grows

43. BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
Considerations
All about the hardware: Cubes a small, they will fit in
memory if we get enough (dont use corporate SAN
Multi User:
• Locking could be an issue, but all queries are small
(in control of the front end)
• Processing Online may work.. But see later
• Global company – what about latency?
KISS: Review calculation script – CHECK!
Security for auditing purpose

44. BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field

45. BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
Latency of the link
Observations:
• XMLA is flexible – but it is ”bloated” on bandwidth
• Excel tends to be chatty client
Results: User will not NOT like the latency added
by the speed of light
• The ms required really adds up
Solution: Citrix farms with remote users. Gives
great response times and good user experience

46. BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
Partitioning for Real Time Setup
Financial Data
Historial Records
Current Month
Real Time
Historial Records
(MOLAP / monthly)
Current Month
(MOLAP / 10 min)
Current Month
(ROLAP / ProActive)
Historial Records
(MOLAP / monthly)
Current Month
(MOLAP / 10 min)
Current Month
(ROLAP / ProActive)
Read
Read
Writeback

47. BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
Enryption techniques and overhead
Securing all lines and data:
• Crypt the XMLA traffic with SSL
• EFS on MOLAP data
• TDE on SQL Server data
• SQL Server TDS encryption
Overhead:
• Less than 10% on query performance
• Dimension processing times doubles!
• Fact Processing, only a few %

48. BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
ROLAP – yes ROLAP!
No way to bring in data to MOLAP fast
enough
• No – ProActive caching will not help.
• There is never a silent interval long
enough
Solution: ROLAP partition with recent data
• Keep ROLAP partition small. Don’t
want to touch the I/O system
• Avoids locking issue during
processing
Keep the ROLAP small..
Technique:
• Gradually shift data from ROLAP to MOLAP
”mid day” partition using ProActive cache
• Steps (every 10 min)
1. INSERT...SELECT from
ROLAP partitions to MOLAP
source table
2. Lock Cube
3. Process MOLAP (fast, only
intra day data)
4. Modify View on ROLAP to
exclude data
5. Unlock Cube
6. Delete Data (delay this, to
avoid holding lock)
• Necessary assumption: All queries are
short running

49. BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
Summarizing (1/2)
You HAVE to get the design right if you want to scale
Partitioning is absolute crucial
• Partition for processing speed
• Partition for data latency (real time vs. Stale)
• Partition for archival of old data
• Secondary concern to above: partition for query speed
Hardware really matters for large cubes
• NAND devices are match made in heaven.
• Tests show two core Nehalem CPU can often support hundreds of users
• Carefully balance IOPS vs. Memory, consider hot portion of cube

50. BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
Summarizing (2/2)
Locking is your enemy
• Consider processing server with sync vs. Individual servers processing
• Consider how to flip
• For super low latency, ROLAP is the way
Multi User Settings
• Just run them as per perf guide
• Consider adjusting CoordinatorBalancingFactor
Keep it Simple
• Remember the premise: AS is a ”send me any query” database.
• You really have to get this right at scale

51. BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
Call to Action!
Feedback really works – a story and “announcement”
Rules:
• Min 100GB cube and/or 100 concurrent users
• Max one pain / month
• Prefix mail subject: “Cube Feedback:”
Feedback, crucial questions:
• Who are you? (customer name / company profile)
• Measure/dimension size, users, cube project
• What are you trying to achieve?
• What does it cost you not to get there / lost opportunity?
My email: tkejser@microsoft.com

52. BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
Complete the Evaluation Form to Win!
Win a Dell Mini Netbook – every day – just for handing in
your completed form. Each session evaluation form
represents a chance to win.
Pick up your evaluation form:
• In each presentation room
• At the PASS Booth near registration
Drop off your completed form:
• Near the exit of each presentation room
• At the PASS Booth near registration
Sponsored by Dell

53. Thank you
for attending this session and the
2010 PASS Summit in Seattle