1. Denny Lee, Lukasz Pawlowski
SQL Customer Advisory Team
SQL Server Reporting Services
Building SSRS 2008 Large
Scale Solutions PASS Community Summit 2008
November 18 – 21, 2008 Seattle WA
2. SQL Server Customer Advisory Team
(SQLCAT)
Works on the largest, most complex SQL Server projects worldwide
– US: NASDAQ, Progressive, Premier Bankcard, Hilton Hotels
– Europe: Barclays Capital, Danske Bank, McLaren, Bwin
– Asia/Pacific: Korea Telecom, GMarket, Japan Railways East, China
Mobile
– LATAM: Banco Itau, Oi
– Strategic ISVs: SAP, Siebel, JDE, PeopleSoft, GE Healthcare, SunGard,
Siemens, Dynamics and more
Drives product requirements back into SQL Server from our customers
and ISVs
Shares deep technical content with SQL Server community
– SQLCAT.com
– http://blogs.msdn.com/sqlcat
– http://blogs.msdn.com/mssqlisv
– http://technet.microsoft.com/en-us/sqlserver/bb331794.aspx
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 2
3. SQL Server Design Win Program
Target the Most Challenging and Innovative
Applications on SQL Server
Investing in Large Scale, Referenceable SQL Server
Projects Across the World
– Provide SQLCAT technical & project experience
– Conduct architecture and design reviews covering performance,
operation, scalability and availability aspects
– Offer use of HW lab in Redmond with direct access to SQL
Server development team
Work with Marketing Team Developing Case Studies
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 3
4. Session Objectives and Takeaways
Session Objective(s):
– Provide guidance on how to scale out your Reporting Services
environment
– Provide RS best practices on RS catalogs, scale out deployment,
and performance optimizations
Agenda:
– Reporting Services Scale Out Architecture
– Report Catalog Best Practices
– Scale Out Deployment Best Practices
– Performance Optimization Configurations
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 4
5. Reporting Services Scale Out
Architecture
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 5
6. Scale Out Architecture:
Overall Architecture
Report Server
RS Scale Out Deployment
Clients
RS Server
Report Catalog
Reporting Data
Flat Files,
OLE DB,
ODBC
NLB
Clients
RS Server RSDB
SQL, AS,
DB2, Oracle,
RS Server Teradata, etc.
Clients
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 6
7. Scale Out Architecture
Read the manuals!
A lot of documentation on SSRS available online
Many mistakes in implementation could have been avoided
Read these:
– Planning for Scalability and Performance with Reporting Services
– Upgrading Reporting Services (SQL Books Online)
– Configuring a Report Server Scale-Out Deployment
On sqlcat.com
– Building and Deploying Large Scale SQL Server Reporting Services
Environments Series
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 7
8. Scale Out Architecture:
Enterprise Rent-A-Car Customer Scenario
Report Server
RS Server
RS Scale Out Deployment
AS Server
Report Catalog
Reporting Data
RS Server
Teradata
RSDB
RS Server
AS Server
RS Server
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 8
9. Scale Out Architecture:
Enterprise Rent-A-Car Customer Scenario
Build test system that accurately represented production
– Goal: 1800 concurrent users
using VS test
10s think time
Mean 33-36s txn time
– Testing allowed them to identify blocking issues
drop down parameter lists of thousands of rows for areas and branches
Developed accurate workload representation (e.g. Proclarity and SSRS
clients)
Currently in production
This presentation incorporates lessons learned from this
and other customers
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 9
10. Scale Out Architecture
Importance of Performance Testing
Need to understand your scenarios and reports
– Scenarios are defined by user personas & usage patterns
– Reports are either test reports or actual reports
– Tests should isolate Report Server from other systems
Need tools to automate the testing
– See white paper: Using Visual Studio 2005 to Perform Load Testing
on a SQL Server 2005 Reporting Services Report Server
– Make single incremental changes between tests
– Do not use SQL trace inside VSTE
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 10
11. Scale Out Architecture
Customer Performance Testing
RS User Mean Tx Mean
Servers s Time CPU%
Max # of Concurrent Users
1 Server 608 36.9 99
2500
2 servers 1218 36.8 96
2300
2000 4 servers 2300 30.5 80
1500
8GB RAM, 2 dual core RS
1218
1000
servers, Windows 2003
500 Graph is max # of users
608
reached for sustained time
0 period (>=15 min)
1 server 2 servers 4 servers
2x RAM and CPU core, only
Users
1/3 increase in load
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 11
12. Scale Out Architecture
RS 2008 vs. RS 2005: Lessons Learned
RS 2008 Front-End Server Scales Up Much Better than RS
2005
– Able to respond to 3–4 times the total number of users and their
requests without errors on the same hardware for all renderers
– RS 2008 consistently outperformed RS 2005 with the PDF and XLS
renderers on the four-processor, quad-core hardware platform
See: Scaling Up RS 2008 vs. RS 2005: Lessoned Learned
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 12
13. Scale Out Architecture
RS 2008 vs. RS 2005: Lessoned Learned
Avg. Response Time (lower is better)
250
4x4 2008 Mix
4x4 2005 Mix
200
4x2 2008 Mix
4x2 2005 Mix
2x2 2008 Mix
150
Avg. Response
Time (ms) 2x2 2005 Mix
100
50
0
User Load
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 13
14. Scale Out Architecture
Scaling Up and Scaling Out with RS 2008 (cont)
RS 2008
– Scale up front-end server to four-processor, quad-core servers for
performance
– Scale out to a two-node deployment for high availability
– Optimize disk I/O subsystem on all RS 2008 boxes for maximum
performance
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 14
15. Reporting Catalog Best
Practices
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 15
16. Report Catalog Best Practices
Report Server
RS Scale Out Deployment
Clients
RS Server
Report Catalog
Reporting Data
Flat Files,
OLE DB,
ODBC
NLB
Clients
RS Server RSDB
SQL, AS,
DB2, Oracle,
RS Server Teradata, etc.
Clients
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 16
17. Report Catalog Best Practices
Report Server Catalog Breakdown
Report Server Catalog (RSDB)
Stores all report metadata including report
Report Catalog
definitions, report / history snapshots,
scheduling, etc.
RSDB RS Temp DB
Stores temporary snapshots while running
reports
These databases can be a bottleneck
Optimize by applying standard SQL DB techniques
Catalog has a lot of I/O and transactions
– RS2005: Many inserts to ChunkData, SnapshotData, and SessionData tables
– RS2008: Many inserts Segment; takes majority of transactions of RSTempDB
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 17
18. Report Catalog Best Practices
Use a dedicated server
Common scenarios
Same server as SSRS Server
– Great for small environments
RSDB – In enterprise environments, too much resource contention
Same server as data source database
– SQL resource contention (TempDB, plan cache, memory
buffer pool) between data source and RS catalogs
– As load increases need to monitor CPU, I/O, network
resources, and buffer pool
Reduce resource contention by having a dedicated RS
catalog server you can tune.
Apply standard high availability and disaster recovery
(e.g. clustering, mirroring, log shipping) to protect the
RSDB
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 18
19. Report Catalog Best Practices
High Performance Disk
Check out Predeployment I/O Best Practices
Have more smaller size disks with faster rotation speeds
(>=15k RPM) vs. fewer larger disks with slower rotations
RSDB Maximize/balance I/O across ALL available spindles
Separate disks between RSDB and RSTempDB
– RSDB a lot of small transactions (report metadata)
– RSTempDB has more (not as many) larger transactions
Pre-grow your databases
Stripe dB files to number of cores (0.25 – 1.0)
– Minimize allocation contention
– Easier to rebalance database when new LUNs are available
Use RAID 10, not RAID 5
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 19
20. Report Catalog Best Practices
Operations Best Practices
Data in RSTempDB is highly volatile
– Report lifetime policy of data = SessionTimeout value (10min)
– CleanupCycleMinutes guides background cleanup thread
RSDB – Once session timeout reached, cleanup temporary snapshot from tempDB
– This is done every CleanupCycleMinutes
Data is RSDB is long lived; should be backed up
– Backing Up and Restore Databases in SQL Server
– Optimizing Backup and Restore Performance in SQL Server
– Backing Up and Restore Encryption Keys
Maintain your RS catalogs
– Remember, these are SQL databses
– E.g. Re-indexing catalog tables or updating stats may improve query
performance
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 20
21. Report Catalog Best Practices
Report Catalog Sizing
RSDB database size
– Varies by number of reports published and number of history snapshots
– General rule of thumb:
Moderate size report definition takes 100-200KB of disk space
RSDB This is larger than the actual RDL
SSRS persists both RDL and compiled binary
Assume 5:1 compression ratio (e.g. 10MB of data, snapshot is 2MB in size)
RSTempDB database size
– Varies by number of users whom are concurrently using the Report Servers
– Each live report execution generates report snapshot persisted in the
RSTempDB
– General rule of thumb:
10-20% concurrency of user base
E.g. 1000 users, then max 200 concurrent users.
If most users are accessing 10MB reports, then you will need 400MB of storage
– 200 users x 10MB reports / 5:1 compression ratio= 400MB
Want to calculate for the maximum number of concurrent users
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 21
22. Disaster Recovery
Primary Data Center Content Switch
Automatic Failover
SSRS SSRS
Manual Failover
Failover Cluster
RSDB Async RSDB
RSDB Mirroring
23. Scale Out Deployment Best
Practices
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 23
24. Scale Out Deployment Best Practices
Report Server
RS Scale Out Deployment
Clients
RS Server
Report Catalog
Reporting Data
Flat Files,
OLE DB,
ODBC
NLB
Clients
RS Server RSDB
SQL, AS,
DB2, Oracle,
RS Server Teradata, etc.
Clients
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 24
25. Scale Out Deployment Best Practices
RS 2005: File System Snapshots
RS TempDB has a lot of transactions to keep
report consistency (i.e. cached reports)
Reduce RS Catalog I/O with File System
RS Server Snapshots
– It will store data on file system
– Unlike RS/IIS setup, will require more disk space
RS Server To enable, update RSReportServer.config file:
– <Add Key="WebServiceUseFileShareStorage"
Value="true" />
RS Server – <WindowsServiceUseFileShareStorage>True</Wi
ndowsServiceUseFileShareStorage>
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 25
26. Scale Out Deployment Best Practices
RS 2005: File System Snapshots
RS TempDB has a lot of transactions to keep
report consistency (i.e. cached reports)
Reduce RS Catalog I/O with File System
RS Server Snapshots
– It will store data on file system
– Unlike RS/IIS setup, will require more disk space
RS Server To enable, update RSReportServer.config file:
– <Add Key="WebServiceUseFileShareStorage"
Value="true" />
RS Server – <WindowsServiceUseFileShareStorage>True</Wi
ndowsServiceUseFileShareStorage>
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 26
27. Scale Out Deployment Best Practices
RS 2008: Why not File System Snapshots?
SSRS 2005
– Advantage for SSRS 2005 because enabling feature
allowed less hits to RSTempDB
RS Server
– Entire report was calculated when requesting first page
SSRS 2008 caches a lot of this data into memory
– Data continually persisted in report catalogs
RS Server
– Local file system acts as a write-through cache
– Does not pre-calculate everything on initial request
– On-demand engine retrieves all of the data and places into
RSTempDB for consistency
RS Server
– But many calculations are done on-demand as needed vs.
pre-calculated and stored.
Still want to test in your environment
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 27
28. Scale Out Deployment Best Practices
Cache Execution
Recurring theme on effective user of memory and
minimal I/O
To help reduce I/O further, enable cache execution
RS Server on your reports.
By default, reports are live execution
Turn on cache execution for each report so the
RS Server report is stored in memory (thus reduced disk I/O)
E.g. Even if you cache report every 5 minutes,
potentially a 80% reduction in I/O
RS Server
– If report is hit every minute, now only I/O hit every 5
minutes, i.e. 20% of the time
No global setting for cache execution
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 28
29. Scale Out Deployment Best Practices
Load Balance your Network
Load balancing important for many
client connections to RS servers
RS Scale Out Deployment
Clients Recommend: Use cookie
RS Server persistence to preserve SSRS-to-
client connection
– IP affinity can work but may be
NLB overload for browser-based
Clients connections
RS Server
– Makes use of SSRS file cache
– Keep round-robin for initial
connections
RS Server Recommend: dual NIC for RS
Clients – Split browser and AS/DB traffic
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 29
30. Scale Out Deployment Best Practices
Isolate your workloads
Report Server
Interaction
RS Scale Out
Deployment
NLB
Clients
RS Server
Report Catalog
Reporting Data
Scheduling
Benefits:
RSDB
Predictable Workloads
RS Scale Out
Deployment
Helps with Security Model
Isolate Performance Issues
RS Server
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 30
31. Scale Out Deployment Best Practices
Report Data Performance Considerations
Scale out works for RS but may not work for
underlying Report Data (data source)
Reporting loads Report Data, limit impact of
large numbers of users
Reporting Data
Flat Files,
OLE DB, – Limit data set size using report filters
ODBC
– SSIS limited data from Operational data sources
– Do not let all users access all of the reports
– E.g. Report Builder against Analysis Services results in
many queries being executed.
SQL, AS,
DB2, Oracle,
Teradata, etc.
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 31
32. Scale Out Deployment Best Practices
Report Data Performance Considerations
Additional resources to scale out SQL and SSAS
Deploying a Scalable Shared Database
SQL Server Replication: Providing High
Availability using Database Mirroring
Reporting Data
Flat Files,
OLE DB,
ODBC Database Mirroring and Log Shipping
SQL Server Replication Features
Scale-Out Querying with Analysis Services
SQL, AS, Scale-Out Querying with Analysis Services Using
DB2, Oracle,
Teradata, etc. SAN Snapshots
Scaling out an Analysis Services Solution
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 32
33. Performance Optimization
Configurations
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 33
34. Performance Optimization
Handling Large Workloads
Control Size of reports
– Do you need them?
– Is this really a data feed?
RS Server
– Aggregate reports and remove unused columns
Recommendations
– Cache Execution
– Report Execution Timeouts
– Scheduled snapshots for large reports with data processing
bottlenecks
– Delivered Rendered reports for non-browser formats
– Pre-populate report cache using data driven subscriptions
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 34
35. Performance Optimization
Large Workload Tuning
Analyze your reports
– Use ExecutionLog2 View
Back to Report Catalogs
RS Server
– Increase size of your report catalogs to store more snapshot
data
Tune the web service
– SSRS 2005: Tune IIS
– SSRS 2008: Tune HTTP.sys
Windows 2003
Windows 2008
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 35
36. Performance Optimization
ExecutionLog2 Analysis Checklist
Sort by ElapsedSec or RowCount Sort by Instance
for long running reports – Determine if NLB is handling request in
– TimeDataRetrieval: If high, need to balanced fashion
optimize data source Sort by Report Path & Timestart
– High RowCount: A lot of data to determine report pattern
aggregated by SSRS, have SQL do
this – E.g. Expensive report (takes 5 minutes
to run) running every 10 minutes
Sort by Request Type
Sort by Status
– A lot of subscriptions, can determine
bottlenecks and stagger reports – Failures occur before (e.g. incorrect
RDL) or after (e.g. subscription delivery
Sort by source error) report is processed
– To determine if live data or snapshot – Outdated information or settings (e.g.
– If report can be snapshot (e.g. last expired passwords, missing
week’s report), create snapshot to subreports, etc.)
avoid query execution, report Data Driven Subscriptions
processing, and rendering
– Errors > 5%
– Continual Scale Mode
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 36
38. Performance Optimization
Should we use 64-bit?
Yes!
Report Server Catalog
– Standard Database techniques for optimization
RS Server
– Since SQL 2005, database written natively for 64-bit
Report Server Service
– Most reports memory intensive
– Note, some workloads (e.g. many small reports) 32-bit can
execute faster
– Handle more concurrent report users or more large reports
– Able to more effectively use memory in SSRS 2008
– Will spill to file system if hits memory pressure
– Exceptions:
Certain data provides not available for 64-bit
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 38
39. Performance Optimization
SSRS 2008 Memory Configurations
Uses memory more efficiently; under intensive workload pressure, it
uses the file system cache. E.g., small requests will continue to stay
in memory while long running request will go to disk
Therefore, before looking at the file system, check these memory
RS Server
configurations:
– WorkingSetMinimum / WorkingSetMaximum:
Minimum / Maximum amount of physical memory that RS will make available to
perform its task;
KB value within RSReportServer.config
Increase value to process more requests in memory
After WorkingSetMaximum is reached and exceeded for a period of time, recycle
app domains to reduce memory utlization
– MemorySafetyMargin:
Defines boundary between low/medium pressure scenarios
Default 80% value in RSReportServer.config
– MemoryThreshold:
Defines boundary between medium/high pressure scenarios
Default 90% value in RSReportServer.config
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 39
40. Performance Optimization
SSRS 2005 Memory Configurations
Recall, SSRS 2005 does not scale up as well as SSRS 2008
MemoryLimit Configuration
• Default 60% of physical memory
RS Server
• Increase help process more requests
• Once threshold hit, no new requests are accepted
MaximumMemoryLimit Configuration
• Default 80% of physical memory
• If this threshold is met, processing is aborted
Changing values may solve RS only to bring up other contentions
Recommendation: If constantly hitting memory thresholds, consider
scaling up and then scaling out
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 40
41. Thank you
for attending this session and the
PASS Community Summit 2008
PASS Community Summit 2008
November 18 – 21, 2008 Seattle WA
Notes de l'éditeur
Front ends connected to same cluster of databasesContent switch allows for automatic failover of SSRS servers (IP address remapping)Mirrored databases on disaster recovery site asynchronously (some metadata loss is okay)Manual failover from primary to disaster recovery siteDatabase Instance names are the same (e.g. REDMOND\\sql4, BAY\\sql4)
By default, RS uses Snapshot data stored in RSTempDB to render reportsTo be efficient, data is spread across over small logical divisions of dataBy default, server must query RSTempDB to get a snapshot chunkAs user load increases, perf degradationSolution: FS SnapshotsCreate file system chunks as cache for snapshot chunksi.e. hit the RS server file system for data instead of always hitting RSTempDBNote, recommended load balancing solution has affinity (e.g. cookie persistence) user sessions to RS node to access FS chunks