More Related Content
Similar to ISUG 113: File stream
Similar to ISUG 113: File stream (20)
More from sqlserver.co.il
More from sqlserver.co.il (20)
ISUG 113: File stream
- 2. Objectives
Familiarization with FILESTREAM
Overview
Storage
Security & Access
Implementation
Application Considerations
Combined with other SQL Server features
Cluster Configuration
RBS (Remote BLOB store)
©Spiral Solutions, Ltd. All rights reserved.
- 3. File Stream
Overview
For unstructured data docs, images, videos.
not new data type Storage attribute.
Varbinary (max) only Fully transactional.
BLOBs saved on storage (NTFS file system)
Needs coordination between DBA and System Administrator.
Can be accessed by T-SQL (DMLs) or Win32 API (stream access).
Use NT system cache:
Much faster Avoid double writing (Buffer cache + Disk).
Reduce load from DB engine.
SQL Server buffer pool isn’t used more free memory.
BLOB size limited only by the volume size (not limited to 2GB)
even in Express Edition.
©Spiral Solutions, Ltd. All rights reserved.
- 4. File Stream
Overview
Store BLOBs in table? On separate files? or in FILESTREAM?
Data size
Larger than 1 MB NTFS FILESTREAM.
Smaller than 256 KB inside the database.
Between 256 KB and 1MB depend on read vs. write ratio + overwrite rate.
Performance Fast read access is important NTFS FILESTREAM.
Transactional Consistency BLOB + structured data FILESTREAM.
Security sensitive data FILESTREAM SQL security over ACLs.
Manageability integrated solution less complex FILESTREAM.
Data fragmentation frequent updates easier to handle on NTFS.
Client access Using DB access protocols (ODBC, OLEDB) NTFS
FILESTREAM.
©Spiral Solutions, Ltd. All rights reserved.
- 5. File Stream
Overview
BLOB Storage Methods Comparison
Storage solution
Comparison point SQL Server (using
File server / file system FILESTREAM
varbinary(max))
Maximum BLOB size NTFS volume size 2 GB – 1 bytes NTFS volume size
Streaming performance of large
Excellent Poor Excellent
BLOBs
Security Manual ACLs Integrated Integrated + automatic ACLs
Cost per GB Low High Low
Manageability Difficult Integrated Integrated
Integration with structured data Difficult Data-level consistency Data-level consistency
Application development and
More complex More simple More simple
deployment
Recovery from data fragmentation Excellent Poor Excellent
Performance of frequent small
Excellent Moderate Poor
updates
©Spiral Solutions, Ltd. All rights reserved.
- 6. File Stream
Overview
Read Performance Comparison (Cold buffer pool)
©Spiral Solutions, Ltd. All rights reserved.
- 7. File Stream
Overview
Read Performance Comparison (Warm buffer pool)
©Spiral Solutions, Ltd. All rights reserved.
- 8. File Stream
Overview
Insert Performance Comparison (Cold buffer pool)
©Spiral Solutions, Ltd. All rights reserved.
- 9. File Stream
Overview
Update Performance Comparison (Cold buffer pool)
©Spiral Solutions, Ltd. All rights reserved.
- 10. File Stream
Storage
Preparing for FILESTREAM:
Set the proper Disk/RAID level resilience, performance, cost, load.
Use FSUTIL :
Turn off short (8.3) file names.
Disable last access time.
Format volume cluster size Use 64-KB NTFS clusters 4-KB for
compressed volumes.
Compress volume extra CPU.
Regularly defrag FILESTREAM computer systems.
Set antivirus scan of FILESTREAM volumes to avoid delete
infected files quarantine if it’s a must.
©Spiral Solutions, Ltd. All rights reserved.
- 11. File Stream
Storage
FILESTREAM stored in special file groups:
File system filter per each volume one for SQL Server
version Integrated security is a must.
File system directories (data containers)
instead of data files (mdf, ndf).
FILESTREAM file group = FILESTREAM data container.
Adding file to FILESTREAM file group = define data container:
Each logical file name is the root directory name.
Filestream.hdr FILESTREAM metadata describing data container.
$FSLOG FILESTREM equivalent to transaction log.
FILESTREAM operation mapped to LSN.
FILESTREAM garbage collection thread triggered by CHECKPOINT.
DBCC CHECKDB for validates link-level consistency.
©Spiral Solutions, Ltd. All rights reserved.
- 12. File Stream
Storage
File stream storage considerations:
Separate FILESTREAM data containers from:
OS volumes and paging file volumes.
Tempdb volumes.
Databases data & log files volumes.
Each data container on separate volume granular control.
Data containers cannot be nested.
All FILSTREAM columns data in a table must be stored in same
FILESTREAM file group FILESTREAM_ON better performance.
Each row must have non-null unique row id (uniqueidentifier
ROWGUID) and a unique/primary key index (clustered only for
non-random data to avoid index reordering on every insert/delete).
Can use partitioning on uniqueidentifier column for load balancing.
©Spiral Solutions, Ltd. All rights reserved.
- 13. File Stream
Security & Access
FILESTREAM is secured like other data:
GRANT permission on DB, schema, table or column.
NTFS permissions (ACLs) are ignored.
Only SQL Server service account is granted NTFS access to data
containers to avoid outside data editing.
Requires integrated security (Windows authentication)
Windows token pass client identity in file I/O operation.
Support snapshot isolation level (only in 2008 R2).
Remote access By Server Message Block (SMB) protocol.
Firewall considerations:
Enable both client & server to resolve DNS names.
Open file-sharing TCP ports: 139, 445 (for remote access).
©Spiral Solutions, Ltd. All rights reserved.
- 14. File Stream
Security & Access
Dual programing model to access BLOBs:
T-SQL for DMLs:
Insert can be used with null value, empty (‘’) value or short inline data.
Update if FILESTREAM field is set to null, the related BLOB is deleted.
Partial field update can’t be done with T-SQL (use UPDATE.Write()) since
update = create new copy of FILESTREAM data file + delete old file.
Partial update only by: FSCTL_SQL_FILESTREAM_FETCH_OLD_CONTENT
Delete or Truncate underlined BLOB data is deleted in file system.
File system streaming access (for read & write):
Statement model models T-SQL statements by using open & close
methods File handle is live in that scope open handle = open
transaction.
Storage namespace DB engine controls BLOB physical file new
internal function column_name.PathName(@option) used by API to provide
the logical UNC path of the BLOB to get the Win32 handle.
©Spiral Solutions, Ltd. All rights reserved.
- 15. File Stream
Security & Access
Transacted file system access New function provides token for current
session’s transaction GET_FILESTREAM_TRANSACTION_CONTEXT ()
binds file streaming operations with a started transaction.
Return NULL if there is no explicitly started transaction.
In C++/ODBC SQL Native Client 10 OpenSqlFilestream Win32 handle.
In C# .NET Framework 3.5 SP1 SqlFileStream Data Type path name +
transaction token handle can be passed to Win32 File API.
Write-through from remote client No write operation are cached on client
side better to make fewer writes with larger data size.
©Spiral Solutions, Ltd. All rights reserved.
- 16. File Stream
Demo
Demo
©Spiral Solutions, Ltd. All rights reserved.
- 17. File Stream
Application Considerations
Use NULL instead of 0x for non-initialized FILESTREAM
column to avoid file creation.
Avoid insert & update in tables with NOT NULL FILESTREAM
columns decrease application performance over time.
In replication use NEWSEQUENTIALID() instead of NEWID()
for better performance.
Avoid T-SQL for update, append, prepend data to FILESTREAM
BLOB will be spooled into tempdb.
Avoid appending small BLOB updates Causes FILESTREAM
files to be copied Write BLOB into varbinary(max) parameter
and do single write operation.
Avoid getting lots of BLOB files length in application use
DATALENGTH() in T-SQL does not open the BLOB.
©Spiral Solutions, Ltd. All rights reserved.
- 18. File Stream
Combined with other features
Snapshots Only by excluding FILESTREAM file groups
FILESTREAM file groups will be marked as offline.
Replication Supported by both transactional & merge
with many considerations SQL 2008 or later on all subscribers.
Log Shipping supports FILESTREAM.
Mirroring Does not support FILESTREAM.
Clustering fully supported. Must enable FILESTREAM on all nodes
Backup & Restore Fully supported on all recovery models.
Encryption is not supported even when TDE is enabled.
Indexes cannot be a part of: index key / include column / statistics.
Constraints (PK,FK,UQ) cannot be created on FILESTREAM column.
Temp tables & TVP cannot be used in local/global temp tables & TVP.
©Spiral Solutions, Ltd. All rights reserved.
- 19. File Stream
Cluster Configuration
Data containers must be on shared disk resource.
Separate LUN for FILESTREAM is recommended.
Steps for enabling FILESTREAM:
Enable file stream on service level on all nodes.
Windows Share Name must be the same on all nodes.
Allow remote clients access create a file-share cluster resource.
Enable file stream on instance level (facets) on all nodes.
Add dependency of FILESTREAM LUN to SQL Service.
Take SQL Service off & Bring SQL Service + dependencies on.
NetApp Snap Manager requirements:
Separate LUNs to data files, Log files & FILESTREAM files.
©Spiral Solutions, Ltd. All rights reserved.
- 20. File Stream
RBS
Complementary approach store BLOBs on remote storage.
Provided with SQL Server 2008 R2 Feature pack.
library API set moving BLOBs from MS SQL to external storage.
Set of SPs and SDK RBS provider for File stream.
Client-side DLL linked into a user application + stored procedures on
SQL Server.
©Spiral Solutions, Ltd. All rights reserved.
- 21. File Stream
Useful Links
FILESTREAM Storage in SQL Server 2008:
http://msdn.microsoft.com/en-us/library/cc949109(v=sql.100).aspx
To BLOB or Not To BLOB - Large Object Storage in a Database or
a File system:
http://research.microsoft.com/apps/pubs/default.aspx?id=64525
How to enable FILESTREAM from the command line:
http://sqlsrvengine.codeplex.com/wikipage?title=FileStreamEnable&referri
ngTitle=Home&ProjectName=sqlsrvengine
FSUTIL syntax:
http://technet.microsoft.com/en-us/library/cc785435(v=ws.10).aspx
SQL Server 2008 R2 Remote Blob Store (RBS) Download :
http://www.microsoft.com/download/en/confirmation.aspx?id=16978
©Spiral Solutions, Ltd. All rights reserved.
- 22. File Stream
End
Thanks
©Spiral Solutions, Ltd. All rights reserved.