Sql server – query optimization – remove bookmark lookup – remove rid lookup

SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID Lookup – Remove Key Lookup « SQL Server Journey with SQL Author...

Home All Articles SQL Interview Q & A Blog Stats Contact Tools SQL Books >>Search<<

SQL S ERVER J OURNEY WITH SQL Feeds: Posts Comments
A UTHORITY
Personal Notes of Pinal Dave

« SQL SERVER – Interesting Observation SQL SERVER – Query Optimization – COMMUNITY INITIATIVES
– Query Hint – FORCE ORDER Remove Bookmark Lookup – Remove RID
Lookup – Remove Key Lookup – Part 2 »

SQL SERVER – Query Optimization – Remove Bookmark
Lookup – Remove RID Lookup – Remove Key Lookup
October 7, 2009 by pinaldave

Today, I would like to share one very quick tip about how to remove bookmark
lookup or RID lookup. Let us first understand Bookmark lookup or RID lookup.
Please note that from SQL Server 2005 SP1 onwards, Bookmark look up is known
as Key look up.

When a small number of rows are requested by a query, the SQL Server optimizer
will try to use a non-clustered index on the column or columns contained in the
WHERE clause to retrieve the data requested by the query. If the query requests
data from columns not present in the non-clustered index, SQL Server must go
back to the data pages to get the data in those columns. Even if the table has a
clustered index or not, the query will still have to return to the table or clustered
index to retrieve the data.

In the above scenario, if table has clustered index, it is called bookmark lookup
(or key lookup); if the table does not have clustered index, but a non-clustered
index, it is called RID lookup. This operation is very expensive. To optimize any
query containing bookmark lookup or RID lookup, it should be removed from the
execution plan to improve performance. There are two different ways to remove
bookmark/RID lookup.

Before we understand these two methods, we will create sample table without
clustered index and simulate RID lookup. RID Lookup is a bookmark lookup on a
heap that uses a supplied row identifier (RID).

USE tempdb
GO

http://blog.sqlauthority.com/...ql-server-query-optimization-remove-bookmark-lookup-remove-rid-lookup-remove-key-lookup/[08/29/2012 5:01:34 PM]


-- Create Table OneIndex with few columns
CREATE TABLE OneIndex (ID INT,
FirstName VARCHAR(100),
LastName VARCHAR(100),
City VARCHAR(100))
GO
-- Insert One Hundred Thousand Records
INSERT INTO OneIndex (ID,FirstName,LastName,City)
SELECT TOP 100000 ROW_NUMBER() OVER (ORDER BY a.name) RowID,
'Bob',
CASE WHEN ROW_NUMBER() OVER (ORDER BY a.name)%2 = 1 THEN 'Smith'
ELSE 'Brown' END,
CASE
WHEN ROW_NUMBER() OVER (ORDER BY a.name)%1000 = 1 THEN 'Las Vegas'
WHEN ROW_NUMBER() OVER (ORDER BY a.name)%10 = 1 THEN 'New York'
WHEN ROW_NUMBER() OVER (ORDER BY a.name)%10 = 5 THEN 'San Marino'
WHEN ROW_NUMBER() OVER (ORDER BY a.name)%10 = 3 THEN 'Los Angeles'
ELSE 'Houston' END
ABOUT PINAL DAVE
FROM sys.all_objects a
Pinal Dave is a Technology
CROSS JOIN sys.all_objects b Evangelist. He has written over
GO 2200 articles on the subject on
his blog at
Now let us run following select statement and check the execution plan. http://blog.sqlauthority.com.

SELECT ID, FirstName Along with 8+ years of hands
on experience he holds a
FROM OneIndex
Masters of Science degree and
WHERE City = 'Las Vegas' a number of certifications,
GO including MCTS, MCDBA and
MCAD (.NET). He is co-author
of three SQL Server books -
SQL Server Programming, SQL
Wait Stats and SQL Server
Interview Questions and
Answers. His past work
experiences includes
Technology Evangelist at
Microsoft and Sr. Consultant at
SolidQ. Prior to joining
Microsoft he was awarded
Microsoft MVP award for three
continuous years for his
contribution in community.


As there is no index on table, scan is performed over the table. We will create a
clustered index on the table and check the execution plan once again.

-- Create Clustered Index
CREATE CLUSTERED INDEX [IX_OneIndex_ID] ON [dbo].[OneIndex]
(
[ID] ASC
) ON [PRIMARY]
Follow @pinaldave
GO Send +Pinal Dave an email at
pinal@sqlauthority.com
Now, run following select on the table once again.

SELECT ID, FirstName
FROM OneIndex BLOG STATS
WHERE City = 'Las Vegas' 47,000,371 (46 Million+)
GO

EMAIL SUBSCRIPTION
Enter your email address to
subscribe to this blog and
receive notifications of new
posts by email.

Join 32,364 other followers

Sign me up!

SQL IN SIXTY SECONDS





It is clear from execution plan that as a clustered index is created on the table,
table scan is now converted to clustered index scan. In either case, base table is
completely scanned and there is no seek on the table. BOOKS I AUTHORED
Now, let us see the WHERE clause of our table. From our basic observation, if we
create an index on the column that contains the clause, a performance
improvement may be obtained. Let us create non-clustered index on the table and
then check the execution plan.

-- Create Index on Column City As that is used in where condition
CREATE NONCLUSTERED INDEX [IX_OneIndex_City] ON [dbo].[OneIndex]
(
[City] ASC
) ON [PRIMARY]

GO

After creating the non-clustered index, let us run our select statement again and
check the execution plan. .
Amazon | Kindle
Flipkart | Indiaplaza
FROM OneIndex
WHERE City = 'Las Vegas'
GO

.
Amazon | Kindle



As we have an index on the WHERE clause, the SQL Server query execution
engine uses the non-clustered index to retrieve data from the table. However, the
columns used in the SELECT clause are still not part of the index, and to display
those columns, the engine will have to go to the base table again and retrieve
those columns. This particular behavior is known as bookmark lookup or key Amazon | Kindle
lookup. Flipkart | Indiaplaza

There are two different methods to resolve this issue. I have demonstrated both
the methods together; however, it is recommended that you use any one of these
methods for removing key lookup. I prefer Method 2.

Method 1: Creating non-clustered cover index.


In this method, we will create non-clustered index containing the columns, which
are used in the SELECT statement, along with the column which is used in the
Amazon | 1 | 2 | 3 | 4 | 5
WHERE clause.
CREATE NONCLUSTERED INDEX [IX_OneIndex_Cover] ON [dbo].[OneIndex]
(
City, FirstName, ID FUNNY INDEX VIDEO
) ON [PRIMARY]
GO

Once the above non-clustered index, which covers all the columns in query, is
created, let us run the following SELECT statement and check our execution plan.

FROM OneIndex
WHERE City = 'Las Vegas'
GO

From the execution plan, we can confirm that key lookup is removed the only Login

index seek is happening. As there is no key lookup, the SQL Server query engine
does not have to go to retrieve the data from data pages and it obtains all the
necessary data from index itself. SQLAuthori
on Facebook

Like
Con

21,525 people
like
SQLAuthority.co
21,524 people
like
SQLAuthority.co
.

Method 2: Creating an included column non-clustered index. Naresh
Facebook social plugin
Here, we will create non-clustered index that also includes the columns, which are
used in the SELECT statement, along with the column used in the WHERE clause.
In this method, we will use new syntax introduced in SQL Server 2005. An index
with included nonkey columns can significantly improve query performance when SQLAUTHORITY LINKS
Subscribe to Newsletter
all columns in the query are included in the index.
My Homepage
CREATE NONCLUSTERED INDEX [IX_OneIndex_Include] ON [dbo].[OneIndex] Windows Live Blog
( --------------------
City Top Downloads



) INCLUDE (FirstName,ID) ON [PRIMARY] PDF Downloads

GO Script Downloads

From the execution plan, it is very clear that this method also removes the key Script Bank
lookup as well. Favorite Scripts
All Scripts - 1
All Scripts - 2
All Scripts - 3

Top Articles
Best Articles
Favorite Articles - 1
Favorite Articles - 2
--------------------
> SQL Interview Q & A <
SQL Coding Standards
SQL FAQ Download
--------------------
Jobs @ SQLAuthority

In summary, Key lookup, Bookmark lookup or RID lookup reduces the
performance of query, and we can improve the performance of query by using
included column index or cover index.
ABOUT NUPUR DAVE
I will cover few additional concepts related to the optimal method in another Nupur Dave loves technology
article. simply because it makes life
more convenient. She is
Reference : Pinal Dave (http://blog.sqlauthority.com)
devoted to technology because
Related Post: it touches our heart makes our
daily lives easier. Among the
SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID
many technological programs
Lookup – Remove Key Lookup she uses and embraces

SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID Windows Live most because
she can do lots of things with
Lookup – Remove Key Lookup – Part 2
ease – from photo management
SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID to movies; business emails to
Lookup – Remove Key Lookup – Part 3 personal social media
connections.

DISCLAIMER
This is a personal weblog. The
opinions expressed here
represent my own and not
those of my employer. For
accuracy and official reference
refer to MSDN/ TechNet/ BOL.
My employer do not endorse
any tools, applications, books,
or concepts mentioned on the
blog. I have documented my
personal experience on this
blog.



Share:

TOP 3 COMMENTERS
2397 - Madhivanan
Like this: �Like Be the first to like this.
474 - Imran Mohammed
Posted in Pinal Dave, SQL, SQL Authority, SQL Index, SQL Optimization, SQL Performance, 301 - Ramdas Jaya
SQL Query, SQL Scripts, SQL Server, SQL Tips and Tricks, SQLServer, T SQL, Technology
| 28 Comments

28 Responses
Brad Schulz on October 7, 2009 at 11:13 am | Reply

No need to add ID to your nonclustered index… that’s redundant. Since
you have a clustered index on ID, then ID will automatically be included
in any nonclustered index you create (so that it could do the bookmark
lookup).

So your nonclustered index examples would be to either create an index on (City,
FirstName) or an index on (City) INCLUDE (FirstName).

–Brad

Pradip on October 7, 2009 at 11:29 am | Reply

I have a table customer in which cstid of BIGINT data type with identity.
As table have identity column I do not want any type of index on it. Being
cstid as primary key of table, is it possible without any index?

Michael on October 7, 2009 at 12:23 pm | Reply

Dear Pinal,

good post, very clear sample, but I would like to see a remark, that Key-
Lookups could be in some situations, special with very wide tables, and the SELECT
includes a lot or all of the columns, the much much better way. Think about a
table containing 11.000.000 customers with 200 columns an the avg size of a row
of 30kb. The SELECT need to return the whole row-data. Seach is done within the
lastname, no leading placeholder to be able to use an Index. So an additional
small index only containing the lastname and the key and then doing an Key
Lookup is much much faster instead of using the index, even clustered index
where every row nees severals data blocks. In case of IO we see differences in the
factor of 500 times more IO.

Regards,
Michael

DBTEAM on November 17, 2009 at 7:21 pm | Reply

Hi pinal,

We have a requirement to Merge six 12millions data tables into single
Table.All the tables are having duplicate Emails.So We put the Primary key for
email column using EnterPrise Manager and We r trying to merge by writing



the following insert query, But it’s taking more than 20hrs…:-(

Insert into Total(
Email,FirstName,LastName,Address,City,State,Zipcode,Gender,DOB ,
Phone,WebAddress1,Webaddress2,IPAddress,DateTime,Interest )
Select Email,FirstName,LastName,Address,City,State,Zipcode,Gender,DOB ,
Phone,WebAddress1,Webaddress2,IPAddress,DateTime,Interest
From [Set4]
Where
(not exists(select email from Total
where Total.email=[Set4].email))

How can we do this most efficiently??? Please Help Us……

Thanks in advance…

pinaldave on October 7, 2009 at 1:03 pm | Reply

Brad Schulz,

You are correct, however, I am going to cover that particular concept in
different blog post.

This post is written to show the concept of covering Index. In next blog post, I am
going to show as there is clustered index, your non clustered index will not require
that key and it can be still covering Index.

Kind Regards,
Pinal

stanly on October 21, 2011 at 2:24 pm | Reply

how can i remove duplicate records in a table and after deleting
duplicate records i need one distinct record

madhivanan on October 24, 2011 at 9:22 pm | Reply

Search for “delete duplicate values” in this site. You will get
related articles

Ramdas on October 7, 2009 at 7:37 pm | Reply

Hi Pinal,
Nice explanation of the covering index concept. What is your take on the
comment posted by Michael about the customer table situation, where
one needs to return a lot of columns.

Thank you



dave p on October 7, 2009 at 9:04 pm | Reply

Pinal,

Why not just remove, from the query, the column that’s causing the key
lookup. This is really much simpler I’m sure you’ll agree. You’ve written about best
practices for database design in the past, and taught me that fewer columns per
table are always better. So if you’re getting lots of key lookups, it follows that we
should split up our tables and make them smaller. This is the query optimizer’s
way of telling us that we have messed up our design!

Thank you for continuing to educate us on how best to use our database.


Hi,

We can not remove the columns which are needed in query from SELECT
statement.

That would not be best practices.

dave p on October 7, 2009 at 11:58 pm | Reply

But what if you don’t need those columns. Then you can remove them.
This example proves that you shouldn’t add columns to a query that you
don’t need. That’s never a best practice (at least in my experience). You
revealed in another post that we should keep indexes as narrow as possible, and I
have been following this rule religiously! Since starting down this path I have
managed to get almost every index in our database down to a single column. So
far this has worked very nicely as far as I can tell, but I’m not really sure how to
collect query performance numbers. Would you mind helping me so that I can
send you a report and you can show readers what great progress can be made if
we follow your methods.

pinaldave on October 8, 2009 at 6:28 am | Reply

Hi,

If we do not need those columns in query for SURE they need to be
removed. I agree with you there.

I will send you other details soon.

Kind Regards,
Pinal

SQL SERVER – Query Optimization – on October 8, 2009 at 7:01 am | Reply
Remove Bookmark Lookup – Remove RID Lookup – Remove Key
Lookup – Part 2 Journey to SQL Authority with Pinal Dave

[...] 8, 2009 by pinaldave This article is follow up of my previous article



Lookup &#821…. Please do read my previous article before continuing [...]

SQL SERVER – Query Optimization – on October 12, 2009 at 7:01 am | Reply
Remove Bookmark Lookup – Remove RID Lookup – Remove Key
Lookup – Part 3 Journey to SQL Authority with Pinal Dave

[...] SQL SERVER – Query Optimization – Remove Bookmark Lookup –
Remove RID Lookup – Remove Key Lo… [...]

jfbergeron on October 13, 2009 at 4:41 pm | Reply

Hi Pinal,

I’m curious to know why you have included the ID column in the covering
index, as well as in the nc index with the includes. I thought all columns in the
Clustered index were already included in the non clustered index.

Is there a reason why you did that?

Thanks,


Hi jfbergeron,

Please read my follow up post here :
Lookup – Remove Key Lookup – Part 3
http://blog.sqlauthority.com/2009/10/12/sql-server-query-optimization-remove-
bookmark-lookup-remove-rid-lookup-remove-key-lookup-part-3

Kind Regards,
Pinal

Barbara37 on October 23, 2009 at 4:24 pm | Reply

And these new notes were not just plain bank notes. ,

SQL SERVER – Removing Key Lookup – on November 9, 2009 at 7:01 am | Reply
Seek Predicate – Predicate – An Interesting Observation Related
to Datatypes Journey to SQL Authority with Pinal Dave


SQL SERVER – Remove Bookmark Key on November 22, 2009 at 7:02 am | Reply
Lookup – 4 Different Ideas Journey to SQL Authority with Pinal
Dave





Mike Chapman on November 24, 2009 at 1:19 am | Reply

Awesome. Managed to knock down a query from 5-6 minutes down to 5
seconds or so.

Ronald Lee on January 10, 2010 at 12:40 pm | Reply

Excellent Post!!! Thank you for the information. This helps me improve my
SP from 5 seconds to 0.xx second. Thanks alot!

SQL SERVER – Four Posts on Removing the on May 13, 2010 at 7:01 am | Reply
Bookmark Lookup – Key Lookup Journey to SQL Authority with
Pinal Dave


Sridhar Nalluri on June 15, 2010 at 2:13 pm | Reply

Hi Pinal,

Nice post, very impressive and understandable…Thank You.

Nick Duckstein on November 2, 2010 at 5:23 am | Reply

Dave,

I’m confused why the best recommendation isn’t to create a clustered
index. In our situation we have a 5 Billion row table without a clustered index. The
RID lookups are killing us. The need for a lot of extra non-clustered indexes is
also killing performance because this is on a replication server.

Finally the writes to a clustered index are dramatically less than to a non-
clustered index with a HEAP.

See: http://msdn.microsoft.com/en-us/library/cc917672.aspx#EFAA

How can anyone justify a large table without a clustered index?

Nick

Nidhi on March 28, 2011 at 8:36 pm | Reply

HI Pinal,

Thanks for this post.Its very easy to understand.Good post.

SQL SERVER – Interview Questions and on July 19, 2011 at 7:00 am | Reply
Answers – Frequently Asked Questions – Day 19 of 31 Journey to



SQLAuthority

[...] In the above scenario, if table has clustered index, it is called
bookmark lookup (or key lookup); if the table does not have clustered index, but a
non-clustered index, it is called RID lookup. (Read more here) [...]

SQL SERVER – An Important Part of Most on January 5, 2012 at 7:02 am | Reply
SELECT statement – WHERE clause – Quiz – Puzzle – 4 of 31 «
SQL Server Journey with SQL Authority

[...] Questions and Answers ISBN: 1466405643 Page#38-40 Does Order
of Column in WHERE clause Matter? Query Optimization – Remove Bookmark
Lookup – Remove RID Lookup – Remove Key Lookup Logical Query Processing
Phases – Order of Statement [...]

Hemant K on January 9, 2012 at 11:31 am | Reply

Hi,
If you are saying, if table has clustered index, it is called bookmark
lookup (or key lookup); if the table does not have clustered index, but a
non-clustered index, it is called RID lookup then you again wrote that as we have
an index on the WHERE clause, the SQL Server query execution engine uses the
non-clustered index to retrieve data from the table. However, the columns used in
the SELECT clause (id, firstname) are still not part of the index that means
(neither cluster index nor non-cluster index has applied on it)and to display those
columns, the engine will have to go to the base table again and retrieve those
columns. so how we can say this particular behavior is known as bookmark lookup
or key lookup

Comments RSS

Leave a Reply

Email (required) (Address never made public)

Name (required)



Website

Notify me of follow-up comments via email. Post Comment

Notify me of new posts via email.

Blog at WordPress.com. Theme: Customized MistyLook by WPThemes.


Sql server – query optimization – remove bookmark lookup – remove rid lookup

Recommandé

Recommandé

Contenu connexe

Similaire à Sql server – query optimization – remove bookmark lookup – remove rid lookup

Similaire à Sql server – query optimization – remove bookmark lookup – remove rid lookup (20)

Plus de Kaing Menglieng

Plus de Kaing Menglieng (20)

Sql server – query optimization – remove bookmark lookup – remove rid lookup