Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

Search@flipkart

1 037 vues

Publié le

Slides from 2nd Solr/Lucene Meetup in Bangalore
http://www.meetup.com/Bangalore-Apache-Solr-Lucene-Group/events/129617772/

Publié dans : Ingénierie
  • Soyez le premier à commenter

Search@flipkart

  1. 1. Search @ Flipkart Umesh Prasad Thejus VM Empowering Consumers discover and find products Solr/ Lucene Meetup 2 @ Bangalore Date : July 27, 2013
  2. 2. Outline ● Search Architecture @ Flipkart ● Challenges for E-commerce ○ Diverse Catalogue ○ Availability, Uptime and performance ○ High frequency updates ● Solutions ○ Caching and warm up ○ External Source Fields (Sort, Facet, Filter) ○ Relevance optimizations
  3. 3. Flipkart Search Architecture
  4. 4. Technologies Used
  5. 5. The E-commerce Search Challenge ● Diverse catalogue ○ ~13 million products, ~900 categories ○ What fields to Search ○ How to rank (within category/across categories). Ranking Facets ? ○ tf-idf and vector space model doesn't help ● Performance ○ 99.99 % availability ○ ~1000 qps ○ ~75 ms for Search, ~5 ms for Autosuggest ○ Prefetching data (Conflicts with liveliness) ● High rate of updates ○ Multiple data sources (aggregate, index, commit, replicate) ○ Temporal fields (Price/Availability/SLAs/Offers) ○ Lucene doesn't support partial updates
  6. 6. Addressing - Performance / Latency ● Make Search Faster ○ Use Filters, score only if needed, lazy field loads, smaller indexes aka sharding ● Caching ○ Solr caches (Type/Sizing/Tuning/Warming) ○ Custom caches ○ Cache warmup on replication and startup
  7. 7. Solr Search Flow And High Latency Cache Cache hit is 10X - 50X faster.
  8. 8. Solr Caches ● QueryCache ○ Key = <Lucene Query, Filters, SortFields> ○ Value = Docset(Bitset) / DocList (bitset with score) ○ Caching only a results Window ○ Use : Pagination/repeat queries ● FilterCache ○ Key = Query ○ Value = Docset (maxDoc) ○ Matching / Faceting ● FieldValueCache ○ Key = FieldName ○ Value = <Term,DocSet> ○ Faceting ● DocumentCache ○ Key = docId ○ Value = Fields
  9. 9. Expensive Features ● Facet on Queries ○ Facet.queries ● Grouping ○ ngroups (counting number of groups ) ○ facet counting of groups (makes 2nd query) ○ No Cache for Group ● Solution : High Latency Cache ○ Key = All Request Params ○ Value = Full response object ○ Re-generate
  10. 10. How replication Impacts Caching ?
  11. 11. Challenge 3 : High Rate of Updates ● Two Solutions ○ Near real time Indexing / Searching ○ External Fields ● NRT Indexing and searching ○ Softcommits => solr caches invalidated ○ Lot of churn : Document deleted and re-added. ○ No autowarm for document cache ● External Fields ○ Resonates with Horizontal partition (Document level partitioning) ○ Great for Ephemeral fields (Price/availability/slas) ○ Supports faceting / filter / sorting
  12. 12. External Fields and Relevance Tuning
  13. 13. Sorting on 500 plus Dynamic Fields ● 10 million products * 4 bytes = 38.1 MB ● 38.1 MB * 500 fields = 17.0 GB of Heap Memory ● On replication : 17 * 2 = 34 GB Heap for just FieldCache BOOM
  14. 14. External Fields
  15. 15. Relevance and Scoring ● Search Page(Query based scoring) ○ Handcrafted boosts to capture retail specific signals ○ User feedback based ranking ○ Turn off - query norm, tf, idf on specific fields ● Browse Page(Non Query based Scoring) ○ Challenge - How do we rank in order to maximize diversity and still show relevant products
  16. 16. Query Classification ● Rank category for a given query ● Signals ○ Text Scoring ○ Retail signals ○ Click stream data ● Rules Specified over classifications for better customer experience
  17. 17. Q & A

×