1. query optimization - How to search millions of record in SQL table faster? - Stack Overflow
log in | careers | chat | meta | about | faq search
Questions Tags Users Badges Unanswered Ask Question
How to search millions of record in SQL table faster?
Hello World!
This is a collaboratively edited
question and answer site for
professional and
I have SQL table with millions of domain name. But now when I search for let's say enthusiast
programmers. It's 100%
4 SELECT * free, no registration required.
FROM tblDomainResults
about » faq »
WHERE domainName LIKE '%lifeis%'
It takes more than 10 minutes to get the results. I tried indexing but that didn't help. tagged
1
sql × 92916
What is the best way to store this millions of record and easily access these information in short period of
time? query-optimization × 1505
There are about 50 million records and 5 column so far. like × 1192
sql query-optimization like asked 1 year ago
viewed 2,427 times
share | improve this question edited May 3 '11 at 23:29 asked May 3 '11 at 23:23 active 1 month ago
OMG Ponies user737063
101k 10 84 167 21 2
Community Bulletin
1 What database platform are you using? – mrdenny May 3 '11 at 23:46 event Please read before
flagging as SPAM (a
reminder from your
moderators)
feedback – ends in 7 days
Visit Chat
8 Answers active oldest votes
Related
What is the best way to spot
Most likely, you tried a traditional index which cannot be used to optimize LIKE queries unless the the slowest block of a SQL
query?
8
pattern begins with a fixed string (e.g. 'lifeis%').
Performing an ASP.NET
What you need for your query is a full-text index. Most DBMS support it these days. database search with
StartsWith keyword
share | improve this answer answered May 3 '11 at Create optimum query to find
23:27 records that are in only one
Igor Nazarenko table
936 1 5 SQL: Optimization problem,
has rows?
feedback Speeding up a SQL query with
generic data information
PostgreSQL LIKE query
performance variations
Optimising SQL Query With
Full-text indexing is the far-and-away best option here - how this is accomplished will depend on the Multiple Joins
3
DBMS you're using. Is there a tool online to
optimize a stored procedure
Short of that, ensuring that you have an index on the column being matched with the pattern will help or inline sql
performance, but by the sounds of it, you've tried this and it didn't help a great deal. Speeding up inner joins
between a large table and a
share | improve this answer answered May 3 '11 at small table
23:25
which one is a faster/better
Will A sql practice?
http://stackoverflow.com/questions/5876861/how-to-search-millions-of-record-in-sql-table-faster[08/29/2012 5:03:40 PM]

2. query optimization - How to search millions of record in SQL table faster? - Stack Overflow
13.3k 11 29 Optimizing SQL query with
checking special rules set
feedback
SQL Statement using LIKE
How can I learn to optimize
sql queries
SQL Optimization Strategy
Stop using LIKE statement. You could use fulltext search, but it will require MyISAM table and isn't all How to search for “%” in sql
2
that good solution. table-column
SQL Query Optimisation
I would recommend for you to examine available 3rd party solutions - like Lucene and Sphinx. (Direction of Condition
They will be superior. Evaluation)
How to optimize a count SQL
share | improve this answer answered May 3 '11 at query on a big table
23:29
tereško Optimizing SQL query being
slowed by bookmark lookup
13.6k 6 19 35
Rewriting sql query with a
nested subquery
feedback
What kind of overhead is
associated with multiple
SELECT queries in SQL?
SQL Query Optimization: is
One thing you might want to consider is having a separate search engine for such lookups. For example, SELECT * FROM Persons
really that much slower than
1
you can use a SOLR (lucene) server to search on and retrieve the ids of entries that match your search, SELECT * FROM Persons
then retrieve the data from the database by id. Even having to make two different calls, its very likely it WHERE City='Sandnes'?
will wind up being faster. Optimizing a SQL query finding
entries that do not exist in
share | improve this answer answered May 3 '11 at table
23:27
SQL, how to check that the
RHSeeger result of a select excludes all
7,096 12 19 results where a string is
matched from another table
feedback Shortening the sentence to
find a search value
SQL Server statements with
IN clauses: how to make them
faster?
Assuming that your 50 million row table includes duplicates (perhaps that is part of the problem), and
1
assuming SQL Server (the syntax may change but the concept is similar on most RDBMSes), another
option is to store domains in a lookup table, e.g.
CREATE TABLE dbo.Domains
(
DomainID INT IDENTITY(1,1) PRIMARY KEY,
DomainName VARCHAR(255) NOT NULL
);
CREATE UNIQUE INDEX dn ON dbo.Domains(DomainName);
When you load new data, check if any of the domain names are new - and insert those into the Domains
table. Then in your big table, you just include the DomainID. Not only will this keep your 50 million row
table much smaller, it will also make lookups like this much more efficient.
SELECT * -- please specify column names
FROM dbo.tblDomainResults AS dr
INNER JOIN dbo.Domains AS d
ON dr.DomainID = d.DomainID
WHERE d.DomainName LIKE '%lifeis%';
Of course except on the tiniest of tables, it will always help to avoid LIKE clauses with a leading wildcard.
share | improve this answer answered May 4 '11 at 0:19
Aaron Bertrand
51.5k 7 27 54
feedback
http://stackoverflow.com/questions/5876861/how-to-search-millions-of-record-in-sql-table-faster[08/29/2012 5:03:40 PM]

3. query optimization - How to search millions of record in SQL table faster? - Stack Overflow
Indexes are slowed down whenever they have to go lookup ("bookmark lookup") data that the index itself
0
doesn't contain. For instance, if your index has 2 columns, ID, and NAME, but you're selecting * (which
is 5 columns total) the database has to read the index for the first two columns, then go lookup the other
3 columns in a less efficient data structure somewhere else.
In this case, your index can't be used because of the "like". This is similar to not putting any where filter
on the query, it will skip the index altogether since it has to read the whole table anyway it will do just
that ("table scan"). There is a threshold (i think around 35-50% where the engine normally flips over to
this).
In short, it seems unlikely that you need all 50 million rows from the DB for a production application, but
if you do... use a machine with more memory and try methods that keep that data in memory. Maybe a
No-SQL DB would be a better option - mongoDB, couch DB, tokyo cabinet. Things like this. Good luck!
share | improve this answer answered May 3 '11 at
23:30
Jody
1,076 5 16
feedback
You could try breaking up the domain into chunks and then searh the chunks themselves. I did some
0
thing like that years ago when I needed to search for words in sentences. I did not have full text
searching available so I broke up the sentences into a word list and searched the words. It was really
fast to find the results since the words were indexed.
share | improve this answer answered May 3 '11 at
23:35
Scott Bruns
1,057 3 5
feedback
You can use full-text searching for millions of records... http://msdn.microsoft.com/en-
0
us/library/ms142571.aspx
share | improve this answer answered Jul 4 at 10:22
Dilip
1
feedback
Your Answer
http://stackoverflow.com/questions/5876861/how-to-search-millions-of-record-in-sql-table-faster[08/29/2012 5:03:40 PM]

4. query optimization - How to search millions of record in SQL table faster? - Stack Overflow
Name
Email
log in or
Home Page
Post Your Answer
By posting your answer, you agree to the privacy policy and terms of service.
Not the answer you're looking for? Browse other questions tagged sql
query-optimization like or ask your own question.
question feed
about | faq | blog | chat | data | legal | privacy policy | jobs | advertising info | mobile | contact us | feedback
■ stackoverflow.com ■ api/apps ■ careers ■ serverfault.com ■ superuser.com ■ meta ■ area 51 ■ webapps ■ gaming ■ ubuntu
■ webmasters ■ cooking ■ game development ■ math ■ photography ■ stats ■ tex ■ english ■ theoretical cs ■ programmers
■ unix ■ apple ■ wordpress ■ physics ■ home improvement ■ gis ■ electrical engineering ■ android ■ security ■ bicycles rev 2012.8.28.3842
■ dba ■ drupal ■ sharepoint ■ scifi & fantasy ■ user experience ■ skeptics ■ rpg ■ judaism ■ mathematica
site design / logo © 2012 stack exchange inc; user contributions licensed under cc-wiki with attribution required
http://stackoverflow.com/questions/5876861/how-to-search-millions-of-record-in-sql-table-faster[08/29/2012 5:03:40 PM]