Education for all..

1. DWH-Ahsan AbdullahDWH-Ahsan Abdullah
11
Data WarehousingData Warehousing
Lecture-26Lecture-26
Need for Speed:Need for Speed:
Conventional Indexing TechniquesConventional Indexing Techniques
Virtual University of PakistanVirtual University of Pakistan
Ahsan Abdullah
Assoc. Prof. & Head
Center for Agro-Informatics Research
www.nu.edu.pk/cairindex.asp
National University of Computers & Emerging Sciences, Islamabad
Email: ahsan1010@yahoo.com

2. DWH-Ahsan Abdullah
2
Need For Indexing: SpeedNeed For Indexing: Speed
Consider searching your hard disk using the Windows
SEARCH command.
 Search goes into directory hierarchies.
 Takes about a minute, and there are only a few thousand files.
Assume a fast processor and (even more importantly) a fast
hard disk.
 Assume file size to be 5 KB.
 Assume hard disk scan rate of a million files per second.
 Resulting in scan rate of 5 GB per second.
Largest search engine indexes more than 8 billion pages
 At above scan rate 1,600 seconds required to scan ALL pages.
 This is just for one user!
 No one is going to wait for 26 minutes, not even 26 seconds.
Hence, a sequential scan is simply not feasible.
No text goes to graphics

3. DWH-Ahsan Abdullah
3
Need For Indexing: Query ComplexityNeed For Indexing: Query Complexity
 How many customers do I have in Karachi?
 How many customers in Karachi made calls during
April?
 How many customers in Karachi made calls to
Multan during April?
 How many customers in Karachi made calls to
Multan during April using a particular calling
package?

4. DWH-Ahsan Abdullah
4
Need For Indexing: I/O BottleneckNeed For Indexing: I/O Bottleneck
 Throwing hardware just speeds up the CPU
intensive tasks.
 The problem is of I/O, which does not scales up
easily.
 Putting the entire table in RAM is very very
expensive.
 Therefore, index!
No text goes to graphics

5. DWH-Ahsan Abdullah
5
Indexing ConceptIndexing Concept
 Purely physical concept, nothing to do with logical model.Purely physical concept, nothing to do with logical model.
 Invisible to the end user (programmer), optimizer choosesInvisible to the end user (programmer), optimizer chooses
it, effects only the speed, not the answer.it, effects only the speed, not the answer.
 With the library analogy, the time complexity to find aWith the library analogy, the time complexity to find a
book? The average time takenbook? The average time taken
 Using a card catalog organized in many different ways i.e.Using a card catalog organized in many different ways i.e.
author, topic, title etc and is sorted.author, topic, title etc and is sorted.
 A little bit of extra time to first check the catalog, but itA little bit of extra time to first check the catalog, but it
“gives” a pointer to the shelf and the row where book is“gives” a pointer to the shelf and the row where book is
located.located.
 The catalog has no data about the book, just an efficientThe catalog has no data about the book, just an efficient
way of searching.way of searching.
No text goes to graphics

6. DWH-Ahsan Abdullah
6
Indexing GoalIndexing Goal
Look at as few blocks as
possible to find the
matching record(s)

7. DWH-Ahsan Abdullah
7
Conventional indexing TechniquesConventional indexing Techniques
 DenseDense
 SparseSparse
 Multi-level (or B-Tree)Multi-level (or B-Tree)
 Primary Index vs. Secondary IndexesPrimary Index vs. Secondary Indexes

8. DWH-Ahsan Abdullah
8
Dense Index
10
20
30
40
50
60
70
80
90
100
110
120
Data File
20
10
40
30
60
50
80
70
100
90
Every key in the data
file is represented in
the index file
Dense Index: ConceptDense Index: Concept

9. DWH-Ahsan Abdullah
9
Dense Index: Adv & Dis AdvDense Index: Adv & Dis Adv
 Advantage:Advantage:
 A dense index, if fits in the memory, is veryA dense index, if fits in the memory, is very
efficient in locating a record given a keyefficient in locating a record given a key
 Disadvantage:Disadvantage:
 A dense index, if too big and doesn’t fit into theA dense index, if too big and doesn’t fit into the
memory, will be expensive when used to find amemory, will be expensive when used to find a
record given its keyrecord given its key
No text goes to graphics

10. DWH-Ahsan Abdullah
10
Sparse Index
10
30
50
70
90
110
130
150
170
190
210
230
Data File
20
10
40
30
60
50
80
70
100
90
Normally keepsNormally keeps
only one key peronly one key per
data blockdata block
Some keys in theSome keys in the
data file will notdata file will not
have an entry inhave an entry in
the index filethe index file
Sparse Index: ConceptSparse Index: Concept

11. DWH-Ahsan Abdullah
11
Sparse Index: Adv & Dis AdvSparse Index: Adv & Dis Adv
 Advantage:Advantage:
 A sparse index uses less space at the expense ofA sparse index uses less space at the expense of
somewhat more time to find a record given itssomewhat more time to find a record given its
keykey
 Support multi-level indexing structureSupport multi-level indexing structure
 Disadvantage:Disadvantage:
 Locating a record given a key has differentLocating a record given a key has different
performance for different key valuesperformance for different key values
No text goes to graphics

12. DWH-Ahsan Abdullah
12
Sparse 2nd level
10
90
170
250
330
410
490
570
Data File
20
10
40
30
60
50
80
70
100
90
10
30
50
70
90
110
130
150
170
190
210
230
Sparse Index: Multi levelSparse Index: Multi level

13. DWH-Ahsan Abdullah
13
B-tree Indexing: ConceptB-tree Indexing: Concept
 Can be seen as a general form of multi-levelCan be seen as a general form of multi-level
indexes.indexes.
 Generalize usual (binary) search trees (BST).Generalize usual (binary) search trees (BST).
 Allow efficient and fast exploration at the expense ofAllow efficient and fast exploration at the expense of
using slightly more space.using slightly more space.
 Popular variant: BPopular variant: B++
-tree-tree
 Support more efficiently queries like:Support more efficiently queries like:
SELECT * FROM R WHERE a = 11SELECT * FROM R WHERE a = 11
 SELECT * FROM R WHERE 0<= b and b<42SELECT * FROM R WHERE 0<= b and b<42

14. DWH-Ahsan Abdullah
14
200
220
250
280
130
B-tree Indexing: ExampleB-tree Indexing: Example
Each node stored in one disk block
RIDlist
9
20
100
140
145
200
210
215
220
230
250
256
279
280
300
Looking for Empno 250

15. DWH-Ahsan Abdullah
15
B-tree Indexing: LimitationsB-tree Indexing: Limitations
 If a table is large and there are fewer unique values.
 Capitalization is not programmatically enforced
(meaning case-sensitivity does matter and
“FLASHMAN" is different from “Flashman").
 Outcome varies with inter-character spaces.
 A noun spelled differently will result in different
results.
 Insertion can be very expensive.
Nothing will go to graphics

16. DWH-Ahsan Abdullah
16
B-tree Indexing: Limitations ExampleB-tree Indexing: Limitations Example
Given that MOHAMMED is the most common first name in Pakistan,
a 5-million row Customers table would produce many screens of
matching rows for MOHAMMED AHMAD, yet would skip potential
matching values such as the following:
VALUE MISSED REASON MISSED
Mohammed Ahmad Case sensitive
MOHAMMED AHMED AHMED versus AHMAD
MOHAMMED AHMAD Extra space between names
MOHAMMED AHMAD DR DR after AHMAD
MOHAMMAD AHMAD Alternative spelling of MOHAMMAD

17. DWH-Ahsan Abdullah
17
Hash Based IndexingHash Based Indexing
 You may recall that in internal memory, hashingYou may recall that in internal memory, hashing
can be used to quickly locate a specific key.can be used to quickly locate a specific key.
 The same technique can be used on externalThe same technique can be used on external
memory.memory.
 However, advantage over search trees is smaller inHowever, advantage over search trees is smaller in
external search than internal.external search than internal. WHY?WHY?
 Because part of search tree can be brought intoBecause part of search tree can be brought into
the main memory.the main memory.

18. DWH-Ahsan Abdullah
18
Hash Based Indexing: ConceptHash Based Indexing: Concept
In contrast to B-tree indexing, hash based indexes do notIn contrast to B-tree indexing, hash based indexes do not
(typically) keep index values in sorted order.(typically) keep index values in sorted order.
 Index entry is found by hashing on index value requiringIndex entry is found by hashing on index value requiring
exact match.exact match.
SELECT * FROM Customers WHERE AccttNo= 110240SELECT * FROM Customers WHERE AccttNo= 110240
 Index entries kept in hash organized tables rather than B-Index entries kept in hash organized tables rather than B-
tree structures.tree structures.
 Index entry contains ROWID values for each rowIndex entry contains ROWID values for each row
corresponding to the index value.corresponding to the index value.
 Remember few numbers in real-life to be useful for hashing.Remember few numbers in real-life to be useful for hashing.

19. DWH-Ahsan Abdullah
19
.
records
.
key → h(key) disk block
Note on terminology:
The word "indexing" is often used
synonymously with "B-tree indexing".
Hashing as Primary IndexHashing as Primary Index

20. DWH-Ahsan Abdullah
20
key → h(key)
Index
recordkey
Can always be transformed to a secondary index using
indirection, as above.
Indexing the Index
Hashing as Secondary IndexHashing as Secondary Index

21. DWH-Ahsan Abdullah
21
 Indexing (using B-trees) good for rangeIndexing (using B-trees) good for range
searches, e.g.:searches, e.g.:
SELECT * FROM R WHERE A > 5SELECT * FROM R WHERE A > 5
 Hashing good for match based searches,Hashing good for match based searches,
e.g.:e.g.:
SELECT * FROM R WHERE A = 5SELECT * FROM R WHERE A = 5
B-tree vs. Hash IndexesB-tree vs. Hash Indexes

22. DWH-Ahsan Abdullah
22
Primary Key vs. Primary IndexPrimary Key vs. Primary Index
Relation Students
Name ID dept
AHMAD 123 CS
Akram 567 EE
Numan 999 CS
 Primary Key & Primary Index:Primary Key & Primary Index:
 PK is ALWAYS unique.PK is ALWAYS unique.
 PI can be unique, but does not have to be.PI can be unique, but does not have to be.
 In DSS environment, very few queries are PK based.In DSS environment, very few queries are PK based.

23. DWH-Ahsan Abdullah
23
 Primary index selection criteria:Primary index selection criteria:
 Common join and retrieval key.Common join and retrieval key.
 Can be unique UPI or non-unique NUPI.Can be unique UPI or non-unique NUPI.
 Limits on NUPI.Limits on NUPI.
 Only one primary index per table (for hash-basedOnly one primary index per table (for hash-based
file system).file system).
Primary Indexing: CriterionPrimary Indexing: Criterion

24. DWH-Ahsan Abdullah
24
Primary Indexing Criteria: ExamplePrimary Indexing Criteria: Example
What should be the primary index of the call table for aWhat should be the primary index of the call table for a
large telecom company?large telecom company?
call_id decimal (15,0) NOT NULL
caller_no decimal (10,0) NOT NULL
call_duration decimal (15,2) NOT NULL
call_dt date NOT NULL
called_no decimal (15,0) NOT NULL
Call TableCall Table
No simple answer!!No simple answer!!

25. DWH-Ahsan Abdullah
25
 Almost all joins and retrievals will occur throughAlmost all joins and retrievals will occur through
the caller _no foreign key.the caller _no foreign key.
 Use caller_no as a NUPI.Use caller_no as a NUPI.
 In case of non uniform distribution on caller_noIn case of non uniform distribution on caller_no
oror
 if phone number have very large number ofif phone number have very large number of
outgoing calls (e.g., an institutional number couldoutgoing calls (e.g., an institutional number could
easily have several thousand calls).easily have several thousand calls).
 Use call_id as UPI for good data distribution.Use call_id as UPI for good data distribution.
Primary IndexingPrimary Indexing

26. DWH-Ahsan Abdullah
26
For a hash-based file system, primary index isFor a hash-based file system, primary index is
free!free!
 No storage cost.No storage cost.
 No index build required.No index build required.
OLTP databases use a page-based file systemOLTP databases use a page-based file system
and therefore do not deliver this performanceand therefore do not deliver this performance
advantage.advantage.
Primary IndexingPrimary Indexing