Implementing search with solr at 7digital

•

3 j'aime•1,607 vues

lucenerevolution

Presented by James Atherton, Search Team Lead, 7digital A usage/case study, describing our journey as we implemented Lucene/Solr, the lessons we learned along the way and where we hope to go in the future.How we implemented our instant search/search suggest. How we handle trying to index 400 million tracks and metadata for over 40 countries, comprising over 300GB of data, and about 70GB of indexes. Finally where we hope to go in the future.

Formation Technologie Art & Photos

Implementing Search with Solr at 7digital
James Atherton
Content Discovery Team Lead

Implementing Search
with Solr
James Atherton
Content Discovery Lead
@mr_road

Who is 7digital?
Online digital content provider
Covering over 47 territories
Online music store: www.7digital.com
API: api.7digital.com
We power a number of music services:
Samsung
Blackberry
Turntable.fm
Pure

Where we came from...
SQL Searches
SELECT *
FROM <table>
WHERE name LIKE '<search_term>%';
This was SLOW and BAD!!

Wrapped Solr in an API

Old Architecture
API
DB

Domain Objects
Artist Documents
Release Documents (e.g. album or single)
Track Documents

First Attempt - 2011
• Artists and Releases
• Solr 1.4
• 17 stores
• ~40GB
• Dropped DIH as it had issues

2011 Architecture
HTTP
API
Search
API Solr
DB
Solr
Tracks
Artists
Releases

2012
• Added Tracks Core
• Solr 3.5
• 47 stores
• ~400GB
• More than 430 M docs
• Didn't revisit DIH

Current Architecture
HTTP
API
Search
API
Artist/Release
Solrs
Track Solrs
Track Solrs
Track Solrs
Track Solrs
Artist/Release
Solrs

Things Learnt
We should have split by <X>; for us Shops.

Beware Inflection Points
Data size: 400GB != 40GB * 10
Throughput: 600 rpm IS NOT 4 * 150 rpm

What we want in our servers?
RAM ?
Fast Disks?
CPUs?
Virtual?
Bare Metal?

Optimize really...?

Cache Warming/First search?

Testing
Test ingestion/data import, then test again
Your data is not as clean as you think
Load test early and often
We need to be better at this still

Logs
Logging is worth its weight in gold
But don't get weighed down

Monitoring
We use statsd/graphite and NewRelic:

Visualise Indexing
Which territory's data has been indexed?

Magic Deploys
We recently adopted CFEngine, it is awesome!!

The Future
HTTP
API
Search
API
Artist Solrs
Track
Solrs
Track
Solrs
Track
Solrs
Release
Solrs
Track Solrs
Solr Cloud, in
the Cloud??

Questions
?

Resources
https://github.com/etsy/statsd/
https://github.com/7digital
http://d3js.org/

James Atherton
@mr_road
@7digital

Contenu connexe

En vedette

South Big Data Hub: Text Data Analysis Panel

South Big Data Hub: Text Data Analysis Panel

South Big Data Hub: Text Data Analysis Panel

Presentation of the Semantic Knowledge Graph research paper at the 2016 IEEE 3rd International Conference on Data Science and Advanced Analytics (Montreal, Canada - October 18th, 2016) Abstract—This paper describes a new kind of knowledge representation and mining system which we are calling the Semantic Knowledge Graph. At its heart, the Semantic Knowledge Graph leverages an inverted index, along with a complementary uninverted index, to represent nodes (terms) and edges (the documents within intersecting postings lists for multiple terms/nodes). This provides a layer of indirection between each pair of nodes and their corresponding edge, enabling edges to materialize dynamically from underlying corpus statistics. As a result, any combination of nodes can have edges to any other nodes materialize and be scored to reveal latent relationships between the nodes. This provides numerous benefits: the knowledge graph can be built automatically from a real-world corpus of data, new nodes - along with their combined edges - can be instantly materialized from any arbitrary combination of preexisting nodes (using set operations), and a full model of the semantic relationships between all entities within a domain can be represented and dynamically traversed using a highly compact representation of the graph. Such a system has widespread applications in areas as diverse as knowledge modeling and reasoning, natural language processing, anomaly detection, data cleansing, semantic search, analytics, data classification, root cause analysis, and recommendations systems. The main contribution of this paper is the introduction of a novel system - the Semantic Knowledge Graph - which is able to dynamically discover and score interesting relationships between any arbitrary combination of entities (words, phrases, or extracted concepts) through dynamically materializing nodes and edges from a compact graphical representation built automatically from a corpus of data representative of a knowledge domain.

The Semantic Knowledge Graph

The Semantic Knowledge Graph

The Semantic Knowledge Graph

Text Mining using LDA with Context

Text Mining using LDA with Context

Text Mining using LDA with Context

Search engines, and Apache Solr in particular, are quickly shifting the focus away from “big data” systems storing massive amounts of raw (but largely unharnessed) content, to “smart data” systems where the most relevant and actionable content is quickly surfaced instead. Apache Solr is the blazing-fast and fault-tolerant distributed search engine leveraged by 90% of Fortune 500 companies. As a community-driven open source project, Solr brings in diverse contributions from many of the top companies in the world, particularly those for whom returning the most relevant results is mission critical. Out of the box, Solr includes advanced capabilities like learning to rank (machine-learned ranking), graph queries and distributed graph traversals, job scheduling for processing batch and streaming data workloads, the ability to build and deploy machine learning models, and a wide variety of query parsers and functions allowing you to very easily build highly relevant and domain-specific semantic search, recommendations, or personalized search experiences. These days, Solr even enables you to run SQL queries directly against it, mixing and matching the full power of Solr’s free-text, geospatial, and other search capabilities with the a prominent query language already known by most developers (and which many external systems can use to query Solr directly). Due to the community-oriented nature of Solr, the ecosystem of capabilities also spans well beyond just the core project. In this talk, we’ll also cover several other projects within the larger Apache Lucene/Solr ecosystem that further enhance Solr’s smart data capabilities: bi-directional integration of Apache Spark and Solr’s capabilities, large-scale entity extraction, semantic knowledge graphs for discovering, traversing, and scoring meaningful relationships within your data, auto-generation of domain-specific ontologies, running SPARQL queries against Solr on RDF triples, probabilistic identification of key phrases within a query or document, conceptual search leveraging Word2Vec, and even Lucidworks’ own Fusion project which extends Solr to provide an enterprise-ready smart data platform out of the box. We’ll dive into how all of these capabilities can fit within your data science toolbox, and you’ll come away with a really good feel for how to build highly relevant “smart data” applications leveraging these key technologies.

The Apache Solr Smart Data Ecosystem

The Apache Solr Smart Data Ecosystem

The Apache Solr Smart Data Ecosystem

What if your search engine could automatically tune its own domain-specific relevancy model? What if it could learn the important phrases and topics within your domain, automatically identify alternate spellings (synonyms, acronyms, and related phrases) and disambiguate multiple meanings of those phrases, learn the conceptual relationships embedded within your documents, and even use machine-learned ranking to discover the relative importance of different features and then automatically optimize its own ranking algorithms for your domain? In this presentation, you’ll learn you how to do just that - to evolving Lucene/Solr implementations into self-learning data systems which are able to accept user queries, deliver relevance-ranked results, and automatically learn from your users’ subsequent interactions to continually deliver a more relevant experience for each keyword, category, and group of users. Such a self-learning system leverages reflected intelligence to consistently improve its understanding of the content (documents and queries), the context of specific users, and the relevance signals present in the collective feedback from every prior user interaction with the system. Come learn how to move beyond manual relevancy tuning and toward a closed-loop system leveraging both the embedded meaning within your content and the wisdom of the crowds to automatically generate search relevancy algorithms optimized for your domain.

Reflected Intelligence: Lucene/Solr as a self-learning data system

Reflected Intelligence: Lucene/Solr as a self-learning data system

Reflected Intelligence: Lucene/Solr as a self-learning data system

Search engines frequently miss the mark when it comes to understanding user intent. This talk will walk through some of the key building blocks necessary to turn a search engine into a dynamically-learning "intent engine", able to interpret and search on meaning, not just keywords. We will walk through CareerBuilder's semantic search architecture, including semantic autocomplete, query and document interpretation, probabilistic query parsing, automatic taxonomy discovery, keyword disambiguation, and personalization based upon user context/behavior. We will also see how to leverage an inverted index (Lucene/Solr) as a knowledge graph that can be used as a dynamic ontology to extract phrases, understand and weight the semantic relationships between those phrases and known entities, and expand the query to include those additional conceptual relationships. As an example, most search engines completely miss the mark at parsing a query like (Senior Java Developer Portland, OR Hadoop). We will show how to dynamically understand that "senior" designates an experience level, that "java developer" is a job title related to "software engineering", that "portland, or" is a city with a specific geographical boundary (as opposed to a keyword followed by a boolean operator), and that "hadoop" is the skill "Apache Hadoop", which is also related to other terms like "hbase", "hive", and "map/reduce". We will discuss how to train the search engine to parse the query into this intended understanding and how to reflect this understanding to the end user to provide an insightful, augmented search experience. Topics: Semantic Search, Apache Solr, Finite State Transducers, Probabilistic Query Parsing, Bayes Theorem, Augmented Search, Recommendations, Query Disambiguation, NLP, Knowledge Graphs

Searching on Intent: Knowledge Graphs, Personalization, and Contextual Disamb...

Searching on Intent: Knowledge Graphs, Personalization, and Contextual Disamb...

Searching on Intent: Knowledge Graphs, Personalization, and Contextual Disamb...

En vedette (6)

South Big Data Hub: Text Data Analysis Panel

South Big Data Hub: Text Data Analysis Panel

South Big Data Hub: Text Data Analysis Panel

The Semantic Knowledge Graph

The Semantic Knowledge Graph

The Semantic Knowledge Graph

Text Mining using LDA with Context

Text Mining using LDA with Context

Text Mining using LDA with Context

The Apache Solr Smart Data Ecosystem

The Apache Solr Smart Data Ecosystem

The Apache Solr Smart Data Ecosystem

Reflected Intelligence: Lucene/Solr as a self-learning data system

Reflected Intelligence: Lucene/Solr as a self-learning data system

Reflected Intelligence: Lucene/Solr as a self-learning data system

Searching on Intent: Knowledge Graphs, Personalization, and Contextual Disamb...

Searching on Intent: Knowledge Graphs, Personalization, and Contextual Disamb...

Searching on Intent: Knowledge Graphs, Personalization, and Contextual Disamb...

Similaire à Implementing search with solr at 7digital

Hive at Last.fm

Hive at Last.fm

Hive at Last.fm

Storm at Spotify

Storm at Spotify

Storm at Spotify

Final Slide Deck.pptx

Final Slide Deck.pptx

Final Slide Deck.pptx

CaitlynMcDonald4

Last.fm API workshop - Stockholm

Last.fm API workshop - Stockholm

Last.fm API workshop - Stockholm

E commerce Search using Apache Solr

E commerce Search using Apache Solr

E commerce Search using Apache Solr

Elastic @Deezer

Elastic @Deezer

Elastic @Deezer

Aurelien Saint Requier

Apache Solr

Semih Hakkıoğlu

Introduction to Apache Lucene/Solr

Introduction to Apache Lucene/Solr

Introduction to Apache Lucene/Solr

Mon norton tut_publishing01

Mon norton tut_publishing01

Mon norton tut_publishing01

eswcsummerschool

State-of-the-Art Drupal Search with Apache Solr

State-of-the-Art Drupal Search with Apache Solr

State-of-the-Art Drupal Search with Apache Solr

State-of-the-Art Drupal Search with Apache Solr

State-of-the-Art Drupal Search with Apache Solr

State-of-the-Art Drupal Search with Apache Solr

Robert Douglass

Solr

Similaire à Implementing search with solr at 7digital (12)

Hive at Last.fm

Hive at Last.fm

Hive at Last.fm

Storm at Spotify

Storm at Spotify

Storm at Spotify

Final Slide Deck.pptx

Final Slide Deck.pptx

Final Slide Deck.pptx

Last.fm API workshop - Stockholm

Last.fm API workshop - Stockholm

Last.fm API workshop - Stockholm

E commerce Search using Apache Solr

E commerce Search using Apache Solr

E commerce Search using Apache Solr

Elastic @Deezer

Elastic @Deezer

Elastic @Deezer

Apache Solr

Introduction to Apache Lucene/Solr

Introduction to Apache Lucene/Solr

Introduction to Apache Lucene/Solr

Mon norton tut_publishing01

Mon norton tut_publishing01

Mon norton tut_publishing01

State-of-the-Art Drupal Search with Apache Solr

State-of-the-Art Drupal Search with Apache Solr

State-of-the-Art Drupal Search with Apache Solr

State-of-the-Art Drupal Search with Apache Solr

State-of-the-Art Drupal Search with Apache Solr

State-of-the-Art Drupal Search with Apache Solr

Solr

Plus de lucenerevolution

Presented by Isabel Drost-Fromm, Software Developer, Apache Software Foundation/Nokia Gate 5 GmbH at Lucene/Solr Revolution 2013 Dublin Text classification automates the task of filing documents into pre-defined categories based on a set of example documents. The first step in automating classification is to transform the documents to feature vectors. Though this step is highly domain specific Apache Mahout provides you with a lot of easy to use tooling to help you get started, most of which relies heavily on Apache Lucene for analysis, tokenisation and filtering. This session shows how to use facetting to quickly get an understanding of the fields in your document. It will walk you through the steps necessary to convert your text documents into feature vectors that Mahout classifiers can use including a few anecdotes on drafting domain specific features. Configure

Text Classification Powered by Apache Mahout and Lucene

Text Classification Powered by Apache Mahout and Lucene

Text Classification Powered by Apache Mahout and Lucene

lucenerevolution

Presented by Markus Klose, Search + Big Data Consultant SHI Elektronische Medien GmbH at Lucene/Solr Revolution 2013 Dublin Kibana4Solr is search-driven, scalable, browser based and extremely user friendly (also for non-technical users). Logs are everywhere. Any device, system or human can potentially produce a huge amount of information saved in logs. The amount of available logs and their semi-structured nature make a meaningful processing in real-time quite a difficult task. Thus, valuable business insights stored in logs might be not found. Kibana4Solr is a search-driven approach to handle that challenge. It offers user-friendly and browser-based dashboard which can be easily customized to particular needs. In the session the Kibana4Solr will be introduced. Some light will be shed on the architectural features of Kibana4Solr. Some ideas will be given in terms of possible business uses cases. And finally a live demo of Kibana4Solr will be shown. Configure

State of the Art Logging. Kibana4Solr is Here!

State of the Art Logging. Kibana4Solr is Here!

State of the Art Logging. Kibana4Solr is Here!

lucenerevolution

Search at Twitter

Search at Twitter

Search at Twitter

lucenerevolution

Presented by Daniel Beach, Search Application Developer, OpenSource Connections Solr is a powerful search engine, but creating a custom user interface can be daunting. In this fast paced session I will present an overview of how to implement a client-side search application using Solr. Using open-source frameworks like SpyGlass (to be released in September) can be a powerful way to jumpstart your development by giving you out-of-the box results views with support for faceting, autocomplete, and detail views. During this talk I will also demonstrate how we have built and deployed lightweight applications that are able to be performant under large user loads, with minimal server resources.

Building Client-side Search Applications with Solr

Building Client-side Search Applications with Solr

Building Client-side Search Applications with Solr

lucenerevolution

Presented by Timothy Potter, Founder, Text Centrix Storm is a real-time distributed computation system used to process massive streams of data. Many organizations are turning to technologies like Storm to complement batch-oriented big data technologies, such as Hadoop, to deliver time-sensitive analytics at scale. This talk introduces on an emerging architectural pattern of integrating Solr and Storm to process big data in real time. There are a number of natural integration points between Solr and Storm, such as populating a Solr index or supplying data to Storm using Solr’s real-time get support. In this session, Timothy will cover the basic concepts of Storm, such as spouts and bolts. He’ll then provide examples of how to integrate Solr into Storm to perform large-scale indexing in near real-time. In addition, we'll see how to embed Solr in a Storm bolt to match incoming tuples against pre-configured queries, commonly known as percolator. Attendees will come away from this presentation with a good introduction to stream processing technologies and several real-world use cases of how to integrate Solr with Storm.

Integrate Solr with real-time stream processing applications

Integrate Solr with real-time stream processing applications

Integrate Solr with real-time stream processing applications

lucenerevolution

Presented by Rafal Kuć, Consultant and Software engineer, , Sematext Group, Inc. Even though Solr can run without causing any troubles for long periods of time it is very important to monitor and understand what is happening in your cluster. In this session you will learn how to use various tools to monitor how Solr is behaving at a high level, but also on Lucene, JVM, and operating system level. You'll see how to react to what you see and how to make changes to configuration, index structure and shards layout using Solr API. We will also discuss different performance metrics to which you ought to pay extra attention. Finally, you'll learn what to do when things go awry - we will share a few examples of troubleshooting and then dissect what was wrong and what had to be done to make things work again.

Administering and Monitoring SolrCloud Clusters

Administering and Monitoring SolrCloud Clusters

Administering and Monitoring SolrCloud Clusters

lucenerevolution

In a recent project with the United States Patent and Trademark Office, Opensource Connections was asked to prototype the next generation of patent search - using Solr and Lucene. An important aspect of this project was the implementation of BRS, a specialized search syntax used by patent examiners during the examination process. In this fast paced session we will relate our experiences and describe how we used a combination of Parboiled (a Parser Expression Grammar [PEG] parser), Lucene Queries and SpanQueries, and an extension of Solr's QParserPlugin to build BRS search functionality in Solr. First we will characterize the patent search problem and then define the BRS syntax itself. We will then introduce the Parboiled parser and discuss various considerations that one must make when designing a syntax parser. Following this we will describe the methodology used to implement the search functionality in Lucene/Solr. Finally, we will include an overview our syntactic and semantic testing strategies. The audience will leave this session with an understanding of how Solr, Lucene, and Parboiled may be used to implement their own custom search parser.

Implementing a Custom Search Syntax using Solr, Lucene, and Parboiled

Implementing a Custom Search Syntax using Solr, Lucene, and Parboiled

Implementing a Custom Search Syntax using Solr, Lucene, and Parboiled

lucenerevolution

Many of us tend to hate or simply ignore logs, and rightfully so: they’re typically hard to find, difficult to handle, and are cryptic to the human eye. But can we make logs more valuable and more usable if we index them in Solr, so we can search and run real-time statistics on them? Indeed we can, and in this session you’ll learn how to make that happen. In the first part of the session we’ll explain why centralized logging is important, what valuable information one can extract from logs, and we’ll introduce the leading tools from the logging ecosystems everyone should be aware of - from syslog and log4j to LogStash and Flume. In the second part we’ll teach you how to use these tools in tandem with Solr. We’ll show how to use Solr in a SolrCloud setup to index large volumes of logs continuously and efficiently. Then, we'll look at how to scale the Solr cluster as your data volume grows. Finally, we'll see how you can parse your unstructured logs and convert them to nicely structured Solr documents suitable for analytical queries.

Using Solr to Search and Analyze Logs

Using Solr to Search and Analyze Logs

Using Solr to Search and Analyze Logs

lucenerevolution

Building real-time notification systems is often limited to basic filtering and pattern matching against incoming records. Allowing users to query incoming documents using Solr's full range of capabilities is much more powerful. In our environment we needed a way to allow for tens of thousands of such query subscriptions, meaning we needed to find a way to distribute the query processing in the cloud. By creating in-memory Lucene indices from our Solr configuration, we were able to parallelize our queries across our cluster. To achieve this distribution, we wrapped the processing in a Storm topology to provide a flexible way to scale and manage our infrastructure. This presentation will describe our experiences creating this distributed, real-time inverted search notification framework.

Real-time Inverted Search in the Cloud Using Lucene and Storm

Real-time Inverted Search in the Cloud Using Lucene and Storm

Real-time Inverted Search in the Cloud Using Lucene and Storm

lucenerevolution

Like many Web-Applications in the past, the Solr Admin UI up until 4.0 was entirely server based. It used separate code on the server to generate their Dashboards, Overviews and Statistics. All that code had to be maintained and still ... you weren't really able to use that kind of data for the things you needed it for. It was wrapped into HTML, most of the time difficult to extract and changed the structure from time to time w/o announcement. After a short look back, we're going to look into the current state of the Solr Admin UI - a client-side application, running completely in your browser. We'll see how it works, where it gets its data from and how you can get the very same data and wire that into your own custom applications, dashboards and/oder monitoring systems.

Solr's Admin UI - Where does the data come from?

Solr's Admin UI - Where does the data come from?

Solr's Admin UI - Where does the data come from?

lucenerevolution

Schemaless Solr and the Solr Schema REST API

Schemaless Solr and the Solr Schema REST API

Schemaless Solr and the Solr Schema REST API

lucenerevolution

In this session we will show how to build a text classifier using the Apache Lucene/Solr with libSVM libraries. We classify our corpus of job offers into a number of predefined categories. Each indexed document (a job offer) then belongs to zero, one or more categories. Known machine learning techniques for text classification include naïve bayes model, logistic regression, neural network, support vector machine (SVM), etc. We use Lucene/Solr to construct the features vector. Then we use the libsvm library known as the reference implementation of the SVM model to classify the document. We construct as many one-vs-all svm classifiers as there are classes in our setting, then using the Hadoop MapReduce Framework we reconcile the result of our classifiers. The end result is a scalable multi-class classifier. Finally we outline how the classifier is used to enrich basic solr keyword search.

Text Classification with Lucene/Solr, Apache Hadoop and LibSVM

Text Classification with Lucene/Solr, Apache Hadoop and LibSVM

Text Classification with Lucene/Solr, Apache Hadoop and LibSVM

lucenerevolution

Faceted search is a powerful technique to let users easily navigate the search results. It can also be used to develop rich user interfaces, which give an analyst quick insights about the documents space. In this session I will introduce the Facets module, how to use it, under-the-hood details as well as optimizations and best practices. I will also describe advanced faceted search capabilities with Lucene Facets.

Faceted Search with Lucene

Faceted Search with Lucene

Faceted Search with Lucene

lucenerevolution

Presented by Shai Erera, Researcher, IBM Lucene's arsenal has recently expanded to include two new modules: Index Sorting and Replication. Index sorting lets you keep an index consistently sorted based on some criteria (e.g. modification date). This allows for efficient search early-termination as well as achieve better index compression. Index replication lets you replicate a search index to achieve high-availability, fault tolerance as well as take hot index backups. In this talk we will introduce these modules, discuss implementation and design details as well as best practices.

Recent Additions to Lucene Arsenal

Recent Additions to Lucene Arsenal

Recent Additions to Lucene Arsenal

lucenerevolution

As part of their work with large media monitoring companies, Flax has developed a technique for applying tens of thousands of stored Lucene queries to a document in under a second. We'll talk about how we built intelligent filters to reduce the number of actual queries applied and how we extended Lucene to extract the exact hit positions of matches, the challenges of implementation, and how it can be used, including applications that monitor hundreds of thousands of news stories every day.

Turning search upside down

Turning search upside down

Turning search upside down

lucenerevolution

Presented by Xavier Sanchez Loro, Ph.D, Trovit Search SL This session aims to explain the implementation and use case for spellchecking in Trovit search engine. Trovit is a classified ads search engine supporting several different sites, one for each on country and vertical. Our search engine supports multiple indexes in multiple languages, each with several millions of indexed ads. Those indexes are segmented in several different sites depending on the type of ads (homes, cars, rentals, products, jobs and deals). We have developed a multi-language spellchecking system using solr and lucene in order to help our users to better find the desired ads and avoid the dreaded 0 results as much as possible. As such our goal is not pure orthographic correction, but also suggestion of correct searches for a certain site.

Spellchecking in Trovit: Implementing a Contextual Multi-language Spellchecke...

Spellchecking in Trovit: Implementing a Contextual Multi-language Spellchecke...

Spellchecking in Trovit: Implementing a Contextual Multi-language Spellchecke...

lucenerevolution

Shrinking the haystack wes caldwell - final

Shrinking the haystack wes caldwell - final

Shrinking the haystack wes caldwell - final

lucenerevolution

Presented by Mark Miller, Software Developer, Cloudera Apache Lucene/Solr committer Mark Miller talks about how Solr has been integrated into the Hadoop ecosystem to provide full text search at "Big Data" scale. This talk will give an overview of how Cloudera has tackled integrating Solr into the Hadoop ecosystem and highlights some of the design decisions and future plans. Learn how Solr is getting 'cozy' with Hadoop, which contributions are going to what project, and how you can take advantage of these integrations to use Solr efficiently at "Big Data" scale. Learn how you can run Solr directly on HDFS, build indexes with Map/Reduce, load Solr via Flume in 'Near Realtime' and much more.

The First Class Integration of Solr with Hadoop

The First Class Integration of Solr with Hadoop

The First Class Integration of Solr with Hadoop

lucenerevolution

Presented by Rajini Maski, Senior Software Engineer, Happiest Minds Technologies An important problem with document-search in any content management system (CMS) is the handling of permission-based search requests for each user. In this session, we present an algorithm and framework that allows the Search Engine to plainly index both public and privileged documents without any early binding overhead—thus enforcing document-level security policies only at the time of search. With our late-binding approach for ACL (access control lists) and some custom components, we have achieved reduction in search-time overhead. We will also discuss the order of complexity and execution time for the search overhead.

A Novel methodology for handling Document Level Security in Search Based Appl...

A Novel methodology for handling Document Level Security in Search Based Appl...

A Novel methodology for handling Document Level Security in Search Based Appl...

lucenerevolution

Presented by Hien Luu, Technical Lead, LinkedIn Rajasekaran Rangaswamy, LinkedIn For internet companies, marketing campaigns play an important role in acquiring new customers, retaining and engaging existing customers, and promoting new products. The LinkedIn segmentation and targeting platform helps marketing teams to easily and quickly create member segments based on member attributes using nested predicate expressions ranging from simple to complex. Once segments are created, then those qualified members are targeted with marketing campaigns. Lucene is a key piece of technology in this platform. This session will cover how we leverage Hadoop to efficiently build Lucene indexes for a large and growing member attribute data set of 225 million members, and how Lucene is used to create segments based on complex nested predicate expressions. This presentation will also share some of the lessons we learned and challenges we encountered from using Lucene to search over large data sets.

How Lucene Powers the LinkedIn Segmentation and Targeting Platform

How Lucene Powers the LinkedIn Segmentation and Targeting Platform

How Lucene Powers the LinkedIn Segmentation and Targeting Platform

lucenerevolution

Plus de lucenerevolution (20)

Text Classification Powered by Apache Mahout and Lucene

Text Classification Powered by Apache Mahout and Lucene

Text Classification Powered by Apache Mahout and Lucene

State of the Art Logging. Kibana4Solr is Here!

State of the Art Logging. Kibana4Solr is Here!

State of the Art Logging. Kibana4Solr is Here!

Search at Twitter

Search at Twitter

Search at Twitter

Building Client-side Search Applications with Solr

Building Client-side Search Applications with Solr

Building Client-side Search Applications with Solr

Integrate Solr with real-time stream processing applications

Integrate Solr with real-time stream processing applications

Integrate Solr with real-time stream processing applications

Administering and Monitoring SolrCloud Clusters

Administering and Monitoring SolrCloud Clusters

Administering and Monitoring SolrCloud Clusters

Implementing a Custom Search Syntax using Solr, Lucene, and Parboiled

Implementing a Custom Search Syntax using Solr, Lucene, and Parboiled

Implementing a Custom Search Syntax using Solr, Lucene, and Parboiled

Using Solr to Search and Analyze Logs

Using Solr to Search and Analyze Logs

Using Solr to Search and Analyze Logs

Real-time Inverted Search in the Cloud Using Lucene and Storm

Real-time Inverted Search in the Cloud Using Lucene and Storm

Real-time Inverted Search in the Cloud Using Lucene and Storm

Solr's Admin UI - Where does the data come from?

Solr's Admin UI - Where does the data come from?

Solr's Admin UI - Where does the data come from?

Schemaless Solr and the Solr Schema REST API

Schemaless Solr and the Solr Schema REST API

Schemaless Solr and the Solr Schema REST API

Text Classification with Lucene/Solr, Apache Hadoop and LibSVM

Text Classification with Lucene/Solr, Apache Hadoop and LibSVM

Text Classification with Lucene/Solr, Apache Hadoop and LibSVM

Faceted Search with Lucene

Faceted Search with Lucene

Faceted Search with Lucene

Recent Additions to Lucene Arsenal

Recent Additions to Lucene Arsenal

Recent Additions to Lucene Arsenal

Turning search upside down

Turning search upside down

Turning search upside down

Spellchecking in Trovit: Implementing a Contextual Multi-language Spellchecke...

Spellchecking in Trovit: Implementing a Contextual Multi-language Spellchecke...

Spellchecking in Trovit: Implementing a Contextual Multi-language Spellchecke...

Shrinking the haystack wes caldwell - final

Shrinking the haystack wes caldwell - final

Shrinking the haystack wes caldwell - final

The First Class Integration of Solr with Hadoop

The First Class Integration of Solr with Hadoop

The First Class Integration of Solr with Hadoop

A Novel methodology for handling Document Level Security in Search Based Appl...

A Novel methodology for handling Document Level Security in Search Based Appl...

A Novel methodology for handling Document Level Security in Search Based Appl...

How Lucene Powers the LinkedIn Segmentation and Targeting Platform

How Lucene Powers the LinkedIn Segmentation and Targeting Platform

How Lucene Powers the LinkedIn Segmentation and Targeting Platform

Dernier

1029 - Danh muc Sach Giao Khoa 10 . pdf

1029 - Danh muc Sach Giao Khoa 10 . pdf

1029 - Danh muc Sach Giao Khoa 10 . pdf

Nutritional Needs Presentation - HLTH 104

Nutritional Needs Presentation - HLTH 104

Nutritional Needs Presentation - HLTH 104

Activity 01 - Artificial Culture (1).pdf

Activity 01 - Artificial Culture (1).pdf

Activity 01 - Artificial Culture (1).pdf

Unit-IV; Professional Sales Representative (PSR).pptx

Unit-IV; Professional Sales Representative (PSR).pptx

Unit-IV; Professional Sales Representative (PSR).pptx

VishalSingh1417

Presentation by Andreas Schleicher Tackling the School Absenteeism Crisis 30 ...

Presentation by Andreas Schleicher Tackling the School Absenteeism Crisis 30 ...

Presentation by Andreas Schleicher Tackling the School Absenteeism Crisis 30 ...

Measures of Dispersion and Variability: Range, QD, AD and SD

Measures of Dispersion and Variability: Range, QD, AD and SD

Measures of Dispersion and Variability: Range, QD, AD and SD

Mattingly "AI & Prompt Design: Structured Data, Assistants, & RAG"

Mattingly "AI & Prompt Design: Structured Data, Assistants, & RAG"

Mattingly "AI & Prompt Design: Structured Data, Assistants, & RAG"

National Information Standards Organization (NISO)

Application orientated numerical on hev.ppt

Application orientated numerical on hev.ppt

Application orientated numerical on hev.ppt

RamjanShidvankar

Unit-V; Pricing (Pharma Marketing Management).pptx

Unit-V; Pricing (Pharma Marketing Management).pptx

Unit-V; Pricing (Pharma Marketing Management).pptx

VishalSingh1417

Mattingly "AI & Prompt Design: The Basics of Prompt Design"

Mattingly "AI & Prompt Design: The Basics of Prompt Design"

Mattingly "AI & Prompt Design: The Basics of Prompt Design"

National Information Standards Organization (NISO)

Basic Civil Engineering notes first year Notes Building notes Selection of site for Building Layout of a Building What is Burjis, Mutam Building Bye laws Basic Concept of sunlight ventilation in building National Building Code of India Set back or building line Types of Buildings Floor Space Index (F.S.I) Institutional Vs Educational Building Components & function Sills, Lintels, Cantilever Doors, Windows and Ventilators Types of Foundation AND THEIR USES Plinth Area Shallow and Deep Foundation Super Built-up & carpet area Floor Area Ratio (F.A.R) RCC Reinforced Cement Concrete RCC VS PCC

Basic Civil Engineering first year Notes- Chapter 4 Building.pptx

Basic Civil Engineering first year Notes- Chapter 4 Building.pptx

Basic Civil Engineering first year Notes- Chapter 4 Building.pptx

psychiatric nursing HISTORY COLLECTION .docx

psychiatric nursing HISTORY COLLECTION .docx

psychiatric nursing HISTORY COLLECTION .docx

Russian Escort Service in Delhi 11k Hotel Foreigner Russian Call Girls in Delhi Welcome to VIP Call Girl In Delhi Hello! Delhi Call Girls is one of the most popular cities in India. Girls who call in Delhi frequently Advertise their services in small promgons in magazines, as well as on the Internet but We do not act as a direct-promoter. We will do everything we can to make sure that you're safe to the max to the best of our abilities and making sure of our ability and ensuring that you're obtained to the best of our abilities and making sure that you get what you want. Sexuality of our females is recognized by our Business proposals. Top-of-the-line, fully-featured Delhi girl call number and we offer To be aware of it is a major reason in deciding to use our services to ensure that our customers realize the worth of their lives swiftly and in a pleasant manner by engaging with web series performers for a cost of 10000.Here you are able to be Relax knowing that personal information is stored in the business proposals, giving an appearance of like you're as if you are a full affirmation. Call Girls Service Now Delhi +91-9899900591 *********** N.M.************* 01/04/2024 ▬█⓿▀█▀ 𝐈𝐍𝐃𝐄𝐏𝐄𝐍𝐃𝐄𝐍𝐓 CALL 𝐆𝐈𝐑𝐋 𝐕𝐈𝐏 𝐄𝐒𝐂𝐎𝐑𝐓 SERVICE ✅ ❣️ ⭐➡️HOT & SEXY MODELS // COLLEGE GIRLS AVAILABLE FOR COMPLETE ENJOYMENT WITH HIGH PROFILE INDIAN MODEL AVAILABLE HOTEL & HOME ★ SAFE AND SECURE HIGH CLASS SERVICE AFFORDABLE RATE ★ SATISFACTION,UNLIMITED ENJOYMENT. ★ All Meetings are confidential and no information is provided to any one at any cost. ★ EXCLUSIVE PROFILes Are Safe and Consensual with Most Limits Respected ★ Service Available In: - HOME & HOTEL Star Hotel Service .In Call & Out call SeRvIcEs : ★ A-Level (star escort) ★ Strip-tease ★ BBBJ (Bareback Blowjob)Receive advanced sexual techniques in different mode make their life more pleasurable. ★ Spending time in hotel rooms ★ BJ (Blowjob Without a Condom) ★ Completion (Oral to completion) ★ Covered (Covered blowjob Without condom SAFE AND SECURE

Russian Escort Service in Delhi 11k Hotel Foreigner Russian Call Girls in Delhi

Russian Escort Service in Delhi 11k Hotel Foreigner Russian Call Girls in Delhi

Russian Escort Service in Delhi 11k Hotel Foreigner Russian Call Girls in Delhi

SECOND SEMESTER TOPIC COVERAGE SY 2023-2024 Trends, Networks, and Critical Th...

SECOND SEMESTER TOPIC COVERAGE SY 2023-2024 Trends, Networks, and Critical Th...

SECOND SEMESTER TOPIC COVERAGE SY 2023-2024 Trends, Networks, and Critical Th...

PROCESS RECORDING FORMAT.docx

PROCESS RECORDING FORMAT.docx

PROCESS RECORDING FORMAT.docx

microwave assisted reaction. General introduction

microwave assisted reaction. General introduction

microwave assisted reaction. General introduction

Grant Readiness 101 TechSoup and Remy Consulting

Grant Readiness 101 TechSoup and Remy Consulting

Grant Readiness 101 TechSoup and Remy Consulting

Seal of Good Local Governance (SGLG) 2024Final.pptx

Seal of Good Local Governance (SGLG) 2024Final.pptx

Seal of Good Local Governance (SGLG) 2024Final.pptx

Web & Social Media Analytics Previous Year Question Paper.pdf

Web & Social Media Analytics Previous Year Question Paper.pdf

Web & Social Media Analytics Previous Year Question Paper.pdf

Class 11th Physics NEET formula sheet pdf

Class 11th Physics NEET formula sheet pdf

Class 11th Physics NEET formula sheet pdf

AyushMahapatra5

Dernier (20)

1029 - Danh muc Sach Giao Khoa 10 . pdf

1029 - Danh muc Sach Giao Khoa 10 . pdf

1029 - Danh muc Sach Giao Khoa 10 . pdf

Nutritional Needs Presentation - HLTH 104

Nutritional Needs Presentation - HLTH 104

Nutritional Needs Presentation - HLTH 104

Activity 01 - Artificial Culture (1).pdf

Activity 01 - Artificial Culture (1).pdf

Activity 01 - Artificial Culture (1).pdf

Unit-IV; Professional Sales Representative (PSR).pptx

Unit-IV; Professional Sales Representative (PSR).pptx

Unit-IV; Professional Sales Representative (PSR).pptx

Presentation by Andreas Schleicher Tackling the School Absenteeism Crisis 30 ...

Presentation by Andreas Schleicher Tackling the School Absenteeism Crisis 30 ...

Presentation by Andreas Schleicher Tackling the School Absenteeism Crisis 30 ...

Measures of Dispersion and Variability: Range, QD, AD and SD

Measures of Dispersion and Variability: Range, QD, AD and SD

Measures of Dispersion and Variability: Range, QD, AD and SD

Mattingly "AI & Prompt Design: Structured Data, Assistants, & RAG"

Mattingly "AI & Prompt Design: Structured Data, Assistants, & RAG"

Mattingly "AI & Prompt Design: Structured Data, Assistants, & RAG"

Application orientated numerical on hev.ppt

Application orientated numerical on hev.ppt

Application orientated numerical on hev.ppt

Unit-V; Pricing (Pharma Marketing Management).pptx

Unit-V; Pricing (Pharma Marketing Management).pptx

Unit-V; Pricing (Pharma Marketing Management).pptx

Mattingly "AI & Prompt Design: The Basics of Prompt Design"

Mattingly "AI & Prompt Design: The Basics of Prompt Design"

Mattingly "AI & Prompt Design: The Basics of Prompt Design"

Basic Civil Engineering first year Notes- Chapter 4 Building.pptx

Basic Civil Engineering first year Notes- Chapter 4 Building.pptx

Basic Civil Engineering first year Notes- Chapter 4 Building.pptx

psychiatric nursing HISTORY COLLECTION .docx

psychiatric nursing HISTORY COLLECTION .docx

psychiatric nursing HISTORY COLLECTION .docx

Russian Escort Service in Delhi 11k Hotel Foreigner Russian Call Girls in Delhi

Russian Escort Service in Delhi 11k Hotel Foreigner Russian Call Girls in Delhi

Russian Escort Service in Delhi 11k Hotel Foreigner Russian Call Girls in Delhi

SECOND SEMESTER TOPIC COVERAGE SY 2023-2024 Trends, Networks, and Critical Th...

SECOND SEMESTER TOPIC COVERAGE SY 2023-2024 Trends, Networks, and Critical Th...

SECOND SEMESTER TOPIC COVERAGE SY 2023-2024 Trends, Networks, and Critical Th...

PROCESS RECORDING FORMAT.docx

PROCESS RECORDING FORMAT.docx

PROCESS RECORDING FORMAT.docx

microwave assisted reaction. General introduction

microwave assisted reaction. General introduction

microwave assisted reaction. General introduction

Grant Readiness 101 TechSoup and Remy Consulting

Grant Readiness 101 TechSoup and Remy Consulting

Grant Readiness 101 TechSoup and Remy Consulting

Seal of Good Local Governance (SGLG) 2024Final.pptx

Seal of Good Local Governance (SGLG) 2024Final.pptx

Seal of Good Local Governance (SGLG) 2024Final.pptx

Web & Social Media Analytics Previous Year Question Paper.pdf

Web & Social Media Analytics Previous Year Question Paper.pdf

Web & Social Media Analytics Previous Year Question Paper.pdf

Class 11th Physics NEET formula sheet pdf

Class 11th Physics NEET formula sheet pdf

Class 11th Physics NEET formula sheet pdf

Implementing search with solr at 7digital

1. Implementing Search with Solr at 7digital James Atherton Content Discovery Team Lead

2. Implementing Search with Solr James Atherton Content Discovery Lead @mr_road

3. Who is 7digital? Online digital content provider Covering over 47 territories Online music store: www.7digital.com API: api.7digital.com We power a number of music services: Samsung Blackberry Turntable.fm Pure

4. Where we came from... SQL Searches SELECT * FROM <table> WHERE name LIKE '<search_term>%'; This was SLOW and BAD!!

5. Wrapped Solr in an API

6. Old Architecture API DB

7. Domain Objects Artist Documents Release Documents (e.g. album or single) Track Documents

8. First Attempt - 2011 • Artists and Releases • Solr 1.4 • 17 stores • ~40GB • Dropped DIH as it had issues

9. 2011 Architecture HTTP API Search API Solr DB Solr Tracks Artists Releases

10. 2012 • Added Tracks Core • Solr 3.5 • 47 stores • ~400GB • More than 430 M docs • Didn't revisit DIH

11. Current Architecture HTTP API Search API Artist/Release Solrs Track Solrs Track Solrs Track Solrs Track Solrs Artist/Release Solrs

12. Things Learnt We should have split by <X>; for us Shops.

13. Beware Inflection Points Data size: 400GB != 40GB * 10 Throughput: 600 rpm IS NOT 4 * 150 rpm

14. What we want in our servers? RAM ? Fast Disks? CPUs? Virtual? Bare Metal?

15. Optimize really...?

16. Cache Warming/First search?

17. Testing Test ingestion/data import, then test again Your data is not as clean as you think Load test early and often We need to be better at this still

18. Logs Logging is worth its weight in gold But don't get weighed down

19. Monitoring We use statsd/graphite and NewRelic:

20. Visualise Indexing Which territory's data has been indexed?

21. Instant Search

22. Magic Deploys We recently adopted CFEngine, it is awesome!!

23. The Future HTTP API Search API Artist Solrs Track Solrs Track Solrs Track Solrs Release Solrs Track Solrs Solr Cloud, in the Cloud??

24. Questions ?

25. Resources https://github.com/etsy/statsd/ https://github.com/7digital http://d3js.org/

26. James Atherton @mr_road @7digital