This document discusses the major work required to build a unified discovery index from start to being ready for sale within one year. It notes that building such an index involves acquiring content from thousands of publishers, normalizing and deduplicating over 75 million records stored across 75 terabytes of data, and developing complex indexing models to deliver relevant search results. The process of acquiring, cleaning, and indexing vast amounts of content from various sources into a single searchable index is described as a very messy job that requires strong partnerships, funding, experienced teams, and ongoing maintenance.

1. WEB-SCALE DISCOVERY FROM START TO FOR-SALE IN ONE YEAR

2. It ain’t easy “We will so feel their pain, but I hope technology and content provider engagement have improved to make it a bit easier for them!” Miriam Blake, Los Alamos National Laboratory Research Library when talking about Summon and other Discovery tools in reference to their own experience in building a unified index on Code4Lib on Jun 30, 2010. 2

3. She goes on to say… “Aside from the contracts, I can also attest to the major amount of work it has been. We have 95M bibliographic records, stored in > 75TB of disk, and counting. Its all running on SOLR, with a local interface and the distributed aDORe repository on backend. ~ 2 FTE keep it running in production now.” 3

4. Did she say 75 Million Records? That’s a drop in the bucket for what a “Unified Discovery Index” needs to provide to be successful. 4

5. Requirements Strong Publisher Relations Plenty of funding Fresh team with lots of experience 5

6. Content Acquisitions Met with hundreds of content providers from all over the globe Nearly 7000 publishers represented in Summon index 6

7. Content Acquisition Methods 7

8. Merged Deduplication 8

9. Data Normalization Cleanup is important Dates Does October 15, 2010 = Fall 2010? Author Names Publication Title Is it really a journal article or a book review? Is an obituary really a newspaper article? 9

10. Complex Indexing Models Planning Maintenance Hardware Planning Analysis Planning 10

11. Better example Let’s pretend Data is Crude Oil 11

12. Content Provider 12

13. Content Acquisition 13

14. Content Acquisition and Cleanup 14

15. 15

16. Relevancy 16

17. Very Messy Job 17