SQL Server 2012 introduces new full-text search capabilities that allow rich semantic search over documents stored in SQL Server. The key features include: 1) Integrated full-text indexing and search over both structured and unstructured data stored in SQL Server tables. 2) Semantic search capabilities that understand relationships between concepts and terms in documents. 3) Support for filtering search results based on document properties and metadata.

2. Make SQL Server the preferred choice for managing
Unstructured Data and allow building Rich Application
Experience on top

3. Scale Up for storage and search to 100m to 500m documents
Easy use/access to Unstructured data from all applications
Rich insight into unstructured data to make better decisions

4. Transactional Access Streaming Win32 Access
Streaming Win32 Access??
Database Applications Windows Apps SQL Apps
Blobs SMB Share FileStream
Files/Folders API
Rich Services
Fulltext Search Database
Solutions
Scale-up
Semantic Similarity Disk Disk Disk
FileTable
1 2 3
FileStreams
Search
Multiple Containers
Integrated Administration?
Integrated Administration Remote BLOB Storage
Customer Application
SQL RBS API
D
D Centera SQL
B FileStre Azure lib lib FILESTREAM lib
B FileStreams
Integrated Azure Centera SQL DB
Backup/Replication/AlwaysOn

5. Machine Boundary
1 Write BLOB(Photo)
Application
2 Return Blob ID
2
RBS Client RBS 3 Write Blob ID to
Library Services: PhotoRef field
• Create
BLOB Store • Fetch
Provider Library • GC
• Delete
ClaimID ClaimDate PhotoRef
1 3 4390 6/5/2007 <Binary(20)>
BLOB Store SQL Server

6. // Store a new blob.
byte[] myBlobId;
SqlRemoteBlobContext blobContext = new SqlRemoteBlobContext(sqlConn);
using (SqlRemoteBlob newBlob = blobContext.CreateNewBlob()) {
// Write to a System.IO.Stream object.
newBlob.Write(…);
newBlob.Close();
myBlobId = newBlob.BlobId;
}
// Alternative way to write.
newBlob.WriteFromStream(inputStream);

7. // Add a new row including the blob ID to the database
// table.
// Fetch the blob.
using (SqlRemoteBlob existingBlob = blobContext.OpenBlob(myBlobId)) {
// Read from System.IO.Stream object.
existingBlob.Read(...);
}
// Alternative way to read.
existingBlob.ReadToStream(outputStream);

8. Store BLOBs in
DB + File System
Application
BLOB
DB

9. // New TSQL Function:
// Get_filestream_transaction_context()
//
SELECT Get_filestream_transaction_context()
// New TSQL Function :
// PathName()
//
SELECT ClaimImage.PathName()
FROM Insurancedb..Claims

10. // New SqlFileStream Class in VS08 SP1
//
SqlFileStream sfs = new SqlFileStream(path, txnId, System.IO.FileAccess.Read);
// output file to read into
System.IO.FileStream fs = new System.IO.FileStream ("c:output2.jpg", System.IO.FileMode.Create);
{
byte[] buffer = new byte[512 * 1024];
int cbBytesRead = buffer.Length;
while (cbBytesRead == buffer.Length)
{
cbBytesRead = sfs.Read(buffer, 0, buffer.Length);
fs.Write(buffer, 0, cbBytesRead);
}
}

11. sfs SqlFileStream
sfs.Write
// commit SQL transaction and close SQL connection.

13. FileTable Folder Hierarchy
FILESTREAM
Share
MSSQLSERVER
my_machine
Database MSSQLSERVEROffice
Directories DocsDocuments
Private Docs Office Docs
(Database1) (Database2)
FileTable
Directories
Media Documents LogFiles
(FileTable) (FileTable) (FileTable)
User-Defined
Directory
Structure

14. ALTER DATABASE Contoso SET FILESTREAM( non_transacted_access=FULL,
Directory_name = N’Contoso’)
CREATE TABLE Contoso..Documents AS FILETABLE
WITH (filetable_directory = N'Document Library')
<machine name><FILESTREAM share>ContosoDocument Library

15. FileTable Schema
File Attribute Name Type Purpose
Path_locator hierarchyid Represents position of this node in the hierarchical FileNamespace.
parent_path_locator hierarchyid Represents the hierarchyID of the parent directory
-- a computed column
stream_id uniqueidentifier UniqueId for Filestream Data
file_stream varbinary(max) filestream Filestream data
file_type nvarchar(255) Type of the file. Can be used for fulltext index creation
cached_file_size bigint Size of the filestream (cached value)
Name nvarchar(255) File/Folder Name (e.g foo.txt)
creation_time datetime2 Creation Time
last_write_time datetime2 LastWrite Time
last_access_time datetime2 LastAccess Time
is_directory bit TRUE for directories.
is_offline bit Offline attribute
is_hidden bit Hidden attribute
is_readonly bit Read Only attribute
is_archive bit Archive attribute
is_system bit System attribute
is_temporary bit Temporary attribute

16. ALTER TABLE Documents DISABLE FILETABLE_NAMESPACE

17. machine<FILESTREAMshare><Database_directory><FileTable_Directory>...

18. GetFileNamespacePath()
FileTableRootPath()
GetPathlocator()

19. DECLARE @path nvarchar(max)
// get FileNamespace path
SELECT @path=file_stream.GetFileNamespacePath()
FROM DocumentStore WHERE name='MySpec.doc';
// Open File handle
handle = CreateFile( @path, GENERIC_WRITE, 0, NULL, CREATE_ALWAYS,
FILE_ATTRIBUTE_NORMAL, NULL);

21. VNNSharedb

22. sys.dm_filestream_non_transact_handles
sp_kill_filestream_non_transacted_handles

25. Create/Alter Database
max_size
DBCC Shrinkfile Emptyfile

26. Use of multiple spindles for achieving better I/O Scalability

27. 2012 2012

28. File Stores /
SQL BLOBs Remote Blob
External Blob FILESTREAM FILETABLE
API
Stores (CAS)
Depends on Depends on
Streaming Performance
external store external store
Depends on Depends on
Win32 App Compat external store external store
Link Level Consistency
Data Level Consistency
Integrated Query &
Management
Non-local Windows File
n/a
Servers
External Blob Stores n/a

29. Features FileServer+DB SQL 2008– SQL 2012–
Solution FILESTREAM FileTable
Integrated Admin operations for Relational and File No Yes Yes
data
- Backup/Restore, HA/Mirroring
Integrated Services for Relational and File data No Yes Yes
- Tex/Semantic Search, Reports, Query etc
Integrated Security Model No Yes Yes
In-place update of Filestream data Yes No Yes
(non-transacted)
Fully Transacted update of Filestream data No Yes Yes
File/Directory hierarchy in DB No No Yes
Win32 App compatibility Yes No Yes
Relational access to File Attributes No No Yes

33. Queries over 350M documents database and random DMLs running in background.
Beating SQL Server 2005 with a scale factor more than 2x and with avg 60x times better throughput

34. 2005/8 vs 2012
2005/8
2012
Query avgExecTime (ms) under various number of connections (50 ~ 2000 users) for customer
playback benchmark

36. New Search Filter for Document Properties
CONTAINS (PROPERTY ( { column_name }, 'property_name' ), ‘contains_search_condition’ )

40. Source Table Keyphrases KeyphraseDocuments
--------------
Key Title Document -------------- ID Keyword ID DocID
D1 Annual Budget … --------------
--------------
-------------- T1 revenue T1 (revenue) D1 (Annual Budget)
D2 Corporate Earnings … --------------
--------------
--------------
-------------- T2 growth T2 (growth) D2 (Corporate Earnings)
D3 Marketing Reports … --------------
-------------- T3 Windows T3 (Windows) D3 (Marketing Reports)
--------------
--------------
… … … T4 Azure
-------------- … …
-------------- … … T1 (revenue) D7 (Finance Report)
1
… …
Full-Text and Semantic Processing T3 (Windows) D11 (Azure Strategy)
quarter, record,
T4 (Azure) D11 (Azure Strategy)
revenue…
3
DocumentSimilarity
2
a
Keyword Index (Full-Text) DocID MatchedDocID
ID Keyword Colid … compDocid CompOc CompPid D1 (Annual Budget) D2 (Corporate Earnings)
K1 revenue 1 … 10,23,123 (1,4),(5,8),(1,34) 2,5,6,8,4,3 D1 (Annual Budget) D7 (Finance Report)
K2 growth 1 … 10,23,123 (1,5),(5,9),(1,34) 2,5,6,8,5,4 D3 (Marketing Reports) D11 (Azure Strategy)
… … … … … … … …

41. CREATE FULLTEXT INDEX ON Production.Document ( ALTER FULLTEXT INDEX ON Production.Document
Title LANGUAGE 1033, ALTER COLUMN Document
Document ADD STATISTICAL_SEMANTICS
LANGUAGE 1033 WITH NO POPULATION;
TYPE COLUMN FileExtension
STATISTICAL_SEMANTICS …
) …
KEY INDEX PK_Document_DocumentID ALTER FULLTEXT INDEX ON Production.Document
ON documents_catalog START FULL POPULATION;
WITH CHANGE_TRACKING OFF, NO POPULATION;