Optimising Xapian
•Télécharger en tant que ODP, PDF•
0 j'aime•1,087 vues
Talk given to UKUUG on 9th August 2009 about the Xapian search engine, and some of the experiences I've had trying to optimise its design and implementation.
Recommandé
Contenu connexe
Similaire à Optimising Xapian
Similaire à Optimising Xapian (20)
Plus de Richard Boulton
Dernier
Dernier (20)
Optimising Xapian
- 1. Optimising Xapian Richard Boulton UKUUG, Birmingham 9 th August 2009
- 2. Xapian is a search engine
- 3. Xapian is a search engine an information retrieval toolkit
- 5. Find things
- 6. … quickly … quickly
- 7. Arbitrary Boolean restrictions Correct spelling mistaks Suggest search completions Find similar things Browse facets of things Find nearby things Get diverse results Find images which look similar Different sort orders Arbitrary weight influences
- 12. Limited memory
- 13. Database on a slow disk
- 18. Calculate all single-term queries
- 21. Remember the best Felt 1 6 8 Pens 3 6 7 9
- 23. Hold it in memory:
- 24. Read next list
- 25. Merge it in:
- 26. Select the best 1 6 8 1 3 6 7 8 9
- 28. Problem – no way to avoid reading all of the list
- 30. Start with the shortest
- 31. Jump forward in second list to keep up with first
- 32. Keep only the best N items
- 34. Start with the shortest
- 35. Jump forward in second list to keep up with first
- 36. Keep only the best N items
- 38. But, unlike AND, we can't skip items
- 39. So … make it into an AND
- 40. How?
- 42. Track only those
- 43. Keep track of the lowest weight of those
- 44. Also, calculate upper bound on weight of each term
- 45. When both upper bounds < lowest weight, we need both, so become an AND
- 47. Can introduce other operators – AND_MAYBE
- 49. Or, store pairs of adjacent terms, and then check positions
- 50. Or, store certain pairs...
- 51. Does it work?
- 52. YES 18
- 56. Datastructures
- 60. Disks are slow
- 61. But faster when reading in chunks
- 64. Work out what statistics to store, where
- 65. Get tighter bounds on possible weights, so we can skip better
- 67. Store it in each posting list – duplicated, but no side lookup
- 68. Or store it only once?
- 69. Currently, we store it in all posting lists
- 70. New backend stores it only once -> 40% smaller!
- 71. But, currently 10 times slower :(
- 72. Measurements 25
- 77. 500M = A huge collection 10 years ago, now only medium
- 78. 10M = A large collection 10 years ago, now small – will often fit fully in memory
- 79. => IO less of a bottleneck – optimise CPU
- 81. => Have to calculate all the result set!
- 82. Or – approximate
- 83. Or – precalculate the facet values somehow
- 85. Page rank / product rank
- 86. Fixed weights – so store documents in decreasing weight order – lets us finish early
- 87. But – harder to update dynamically
- 91. Combine candidates with dynamically calculated weight
- 93. Queries with hundreds of terms
- 95. … but distribution of frequencies and weights is less amenable to early termination.
- 97. Order based on combination of novelty and relevance
- 98. Score depends on earlier documents
- 99. => our early termination doesn't work
- 100. http://searchevent.org/ “ A day of informal presentations, open discussion and hacking on open source search technologies.” Tuesday 29 th September 2009 Friends meeting house, Cambridge, UK Learn more Free!
- 101. Questions Xapian: http://xapian.org/ Me: [email_address] Photo credits: http://www.flickr.com/photos/striatic/729822/ http://www.flickr.com/photos/stephmcg/1592886057/ http://www.flickr.com/photos/dullhunk/3389581452/ http://www.flickr.com/photos/katielips/3367600309/