Lucene at Yelp - By Sudarshan Gaikaiwari

•

1 j'aime•464 vues

lucenerevolution

See conference video - http://www.lucidimagination.com/devzone/events/conferences/revolution/2011

Technologie Divertissement et humour

Bio

1. Over a decade of experience in information retrieval
2. Used IR techniques at Symantec's DLP group
3. Search Engineer at Yelp

Outline

1. Overview of search services at Yelp
2. Federation Motivation
3. Lucy Indexing
4. Lucy Searching
5. Efficiently Retrieving top k hits

Hard Disk Seek Latency
Disk seek 10,000,000 ns

Source Software Engineering Advice from
Building Large-Scale Distributed Systems
Jeffery Dean

RAM read latency

Main memory
reference
100 ns

Pinning Index in RAM

● vmtouch
● mlock
● http://hoytech.com/vmtouch/

Problem

Index is too large fit in memory on a single machine

Geographical Sharding drawbacks

1. Cumbersome manual process to determine shard boundary
2. No guarantee that a boundary can be found.

Federation

1. Split index across multiple machines
2. Shard on business id
3. TF-IDF scores from different machines should be
comparable

Mapping businesses to shards

1. Assigning businesses to shards

shard = shardlist[hash(business_id) % len(shardlist)]

Problems
1. Involves re-indexing all the businesses if we want to add a
new shard

Advantages

1. Flexibility (move vbuckets from one shard to another)
2. Split hot spot shards

Lucy Master Slave Architecture

Separate indexing (masters)
A master for each shard of a service

Searching (slaves)
A slave for every replica of a service

Federator: Combining results across
shards
1. Once we distribute an index across shards we need a
component which will search all these shards and combine
their results.
2. Written in Python (runs inside a python web process).
3. Uses Tornado IO loop to send requests to all shards.
4. The transfer protocol for the requests in JSON RPC

Executing queries

1. Gather the top results for a query
2. Collect attribute statitics for attributes like places, categories

Lucene

1. Efficiently executes queries over the index
2. Provides how relevant the business is to the words in the
query (word score)
3. Upgrading lucene to 2.9/3.1 is WIP

Efficiently Retrieving top k hits

1. When user moves through multiple pages the number of
hits to be returned increases

num hits = start + count

2. So if we need to retrieve 500 hits the naive way would be to
retrieve 500 hits from each shard and then sort them

Binomial Distribution
Probability (r of top k hits) are in a particular shard

Mean

Variance

Simulation

Formula Hits selected from each Results Missed (%)
shard
k = 100
p = 0.2

24 0.017

32 0.0001407

44 0.00000

Results

1. ~ 50% savings over 100 hits (44 hits requested from each
shard)
2. 77% savings over 1000 hits (228 hits requested from each
shard)

Future work

1. In memory index
2. Move towards real time search

Contenu connexe

En vedette

Cpython embedded in solr - By Roman Chyla

lucenerevolution

Motion to allow discovery

J. Patrick Lucas

Presented by Patrick Beaucamp | Bpm-Conseil - See conference video - http://www.lucidimagination.com/devzone/events/conferences/lucene-revolution-2012 Vanilla, an Open Source business intelligence application by bpm-conseil.com, offers unique features such as report indexing through an embedded Lucene integration. Using Vanilla and Lucene, developers can manage both report indexing and external document indexing, which ultimately saves end users time when they search for specific keywords such as "product code," or "customer code." Vanilla can build upon an existing Solr/Lucene installation that takes care of all the indexing processes while Vanilla takes care of the Reporting/Dashboard creation. During this presentation, attendees will learn how we moved from embed Lucene Api to a Solr/Lucene platform and all the technical and business benefits from this architecture in terms of clustering, caching and access mode.

How to Gain Greater Business Intelligence from Lucene/Solr

lucenerevolution

NRI Residency Pari Chowk Gr. noida for confirmed allotment call 9540110008

Gaurav Arora

The Seven Deadly Sins of Solr - By Jay Hill

lucenerevolution

KEYNOTE: Lucene / Solr road map

lucenerevolution

En vedette (6)

Cpython embedded in solr - By Roman Chyla

Motion to allow discovery

How to Gain Greater Business Intelligence from Lucene/Solr

NRI Residency Pari Chowk Gr. noida for confirmed allotment call 9540110008

The Seven Deadly Sins of Solr - By Jay Hill

KEYNOTE: Lucene / Solr road map

Similaire à Lucene at Yelp - By Sudarshan Gaikaiwari

Deep learning is making news across the country as one of the most promising techniques in machine learning research. However, these methods are complex to implement, finicky to tune, and state-of-the-art accuracy is only achieved by a few experts in the field. In this session, we give a beginner-friendly explanation of deep learning using neural networks—what it is, what it does, and how; and introduce the concept of deep features, which allows you to obtain great performance with reduced running times and data set sizes. We then show how these methods can easily be deployed on GPU instances (G2) on Amazon EC2.

(CMP305) Deep Learning on AWS Made EasyCmp305

Amazon Web Services

Deep Learning architectures, such as deep neural networks, are currently the hottest emerging areas of data science, especially in Big Data. Deep Learning could be effectively exploited to address some major issues of Big Data, such as fast information retrieval, data classification, semantic indexing and so on. In this work, we designed and implemented a framework to train deep neural networks using Spark, fast and general data flow engine for large scale data processing, which can utilize cluster computing to train large scale deep networks. Training Deep Learning models requires extensive data and computation. Our proposed framework can accelerate the training time by distributing the model replicas, via stochastic gradient descent, among cluster nodes for data resided on HDFS.

Spark Based Distributed Deep Learning Framework For Big Data Applications

Humoyun Ahmedov

My Master's Thesis

Humoyun Ahmedov

Jeff Hajewski, Salesforce - There is a wealth of information on building deep learning models with PyTorch or TensorFlow. Anyone interested in building a deep learning model is only a quick search away from a number of clear and well written tutorials that will take them from zero knowledge to having a working image classifier. But what happens when you need to deploy these models in a production setting? At Salesforce, we use TensorFlow models to help us provide customers with insights into their data, and we do this as close to real-time as possible. Designing these systems in a scalable manner requires overcoming a number of design challenges, but the core component is Docker. Docker enables us to design highly scalable systems by allowing us to focus on service interactions, rather than how our services will interact with the hardware. Docker is also at the core of our test infrastructure, allowing developers and data scientists to build and test the system in an end to end manner on their local machines. While some of this may sound complex, the core message is simplicity - Docker allows us to focus on the aspects of the system that matter, greatly simplifying our lives.

Distributed Deep Learning with Docker at Salesforce

Docker, Inc.

Introduction to Azure DocumentDB

Denny Lee

Software systems are growing in size and complexity when the business is growing, and sometimes it is hard to figure out what is going on. Various teams make different changes for different business capabilities. Distributed Tracing is a useful way to look under the hood and see for yourself what operations are being performed, what services are used in a certain use case, and how performant are they. In this talk, I will present what Distributed Tracing is and how we introduced it into our software system with some tips and tricks on what you should focus on if you want to do the same.

"Introducing Distributed Tracing in a Large Software System", Kostiantyn Sha...

Fwdays

Machine learning has its challenges, and understanding the algorithms is not always easy. In this session, you’ll discover methods to make these challenges less daunting. Intended for software engineers who need to understand the requirements and constraints of data scientists, and data scientists who need to implement or help implement production systems, the session will begin with a quick introduction to data quality and a level-set on common vocabulary. You’ll then explore the formats that are required by Spark ML to run its algorithms, and see how to automate the build through user-defined functions and other techniques. Automation will make reproducibility easy, minimize errors and increase the efficiency of data scientists. Key takeaways will include: – How to build the required tool set in Java – Understanding the formats required by Spark ML (a new vocabulary) – Learning fundamentals about data quality and how to make sure the data is usable

The Key to Machine Learning is Prepping the Right Data with Jean Georges Perrin

Databricks

"Practical Machine Learning With Ruby" by Iqbal Farabi (ID Ruby Community)

Tech in Asia ID

This presentation is focused on the architecture, scalability concerns, performance bottlenecks, operational characteristics and lessons learned while designing and implementing Yammer distributed real-time search system. Yammer is an enterprise social network SaaS offering with over 100,000 networks (including 85% of the Fortune 100) and nearly 2 million users. The search system we developed scales well up to 1B messages and serves a foundation of knowledge base analysis services Yammer is developing.

Realtime search at Yammer

Boris Aleksandrovsky

See conference video - http://www.lucidimagination.com/devzone/events/conferences/revolution/2011 This talk will be focused on the architecture, scalability concerns, performance bottlenecks, operational characteristics and lessons learned while designing and implementing Yammer distributed real-time search system. Yammer is an enterprise social network SaaS offering with over 100,000 networks (including 85% of the Fortune 100) and nearly 2 million users. The search system we developed scales well up to 1B messages and serves a foundation of knowledge base analysis services Yammer is developing.

Real-time Search at Yammer - By Aleksandrovsky Boris

lucenerevolution

Real Time Search at Yammer

Lucidworks (Archived)

Puppetcamp Melbourne - puppetdb

m_richardson

Puppet Camp Melbourne 2014: Node Collaboration with PuppetDB

Puppet

Conf orm - explain

Louise Grandjonc

Functional solid

Matt Stine

This talk was presented in Startup Master Class 2017 - http://aaiitkblr.org/smc/ 2017 @ Christ College Bangalore. Hosted by IIT Kanpur Alumni Association and co-presented by IIT KGP Alumni Association, IITACB, PanIIT, IIMA and IIMB alumni. My co-presenter was Biswa Gourav Singh. And contributor was Navin Manaswi. http://dataconomy.com/2017/04/history-neural-networks/ - timeline for neural networks

AI and Deep Learning

Subrat Panda, PhD

Beyond php - it's not (just) about the code

Wim Godden

SenseiDB

Volodymyr Zhabiuk

Distributed deep learning_over_spark_20_nov_2014_ver_2.8

Vijay Srinivas Agneeswaran, Ph.D

Crm saturday madrid 2017 jordi montaña - test automation

Demian Raschkovan

Similaire à Lucene at Yelp - By Sudarshan Gaikaiwari (20)

(CMP305) Deep Learning on AWS Made EasyCmp305

Spark Based Distributed Deep Learning Framework For Big Data Applications

My Master's Thesis

Distributed Deep Learning with Docker at Salesforce

Introduction to Azure DocumentDB

"Introducing Distributed Tracing in a Large Software System", Kostiantyn Sha...

The Key to Machine Learning is Prepping the Right Data with Jean Georges Perrin

"Practical Machine Learning With Ruby" by Iqbal Farabi (ID Ruby Community)

Realtime search at Yammer

Real-time Search at Yammer - By Aleksandrovsky Boris

Real Time Search at Yammer

Puppetcamp Melbourne - puppetdb

Puppet Camp Melbourne 2014: Node Collaboration with PuppetDB

Conf orm - explain

Functional solid

AI and Deep Learning

Beyond php - it's not (just) about the code

SenseiDB

Distributed deep learning_over_spark_20_nov_2014_ver_2.8

Crm saturday madrid 2017 jordi montaña - test automation

Plus de lucenerevolution

Presented by Isabel Drost-Fromm, Software Developer, Apache Software Foundation/Nokia Gate 5 GmbH at Lucene/Solr Revolution 2013 Dublin Text classification automates the task of filing documents into pre-defined categories based on a set of example documents. The first step in automating classification is to transform the documents to feature vectors. Though this step is highly domain specific Apache Mahout provides you with a lot of easy to use tooling to help you get started, most of which relies heavily on Apache Lucene for analysis, tokenisation and filtering. This session shows how to use facetting to quickly get an understanding of the fields in your document. It will walk you through the steps necessary to convert your text documents into feature vectors that Mahout classifiers can use including a few anecdotes on drafting domain specific features. Configure

Text Classification Powered by Apache Mahout and Lucene

lucenerevolution

Presented by Markus Klose, Search + Big Data Consultant SHI Elektronische Medien GmbH at Lucene/Solr Revolution 2013 Dublin Kibana4Solr is search-driven, scalable, browser based and extremely user friendly (also for non-technical users). Logs are everywhere. Any device, system or human can potentially produce a huge amount of information saved in logs. The amount of available logs and their semi-structured nature make a meaningful processing in real-time quite a difficult task. Thus, valuable business insights stored in logs might be not found. Kibana4Solr is a search-driven approach to handle that challenge. It offers user-friendly and browser-based dashboard which can be easily customized to particular needs. In the session the Kibana4Solr will be introduced. Some light will be shed on the architectural features of Kibana4Solr. Some ideas will be given in terms of possible business uses cases. And finally a live demo of Kibana4Solr will be shown. Configure

State of the Art Logging. Kibana4Solr is Here!

lucenerevolution

Search at Twitter

lucenerevolution

Presented by Daniel Beach, Search Application Developer, OpenSource Connections Solr is a powerful search engine, but creating a custom user interface can be daunting. In this fast paced session I will present an overview of how to implement a client-side search application using Solr. Using open-source frameworks like SpyGlass (to be released in September) can be a powerful way to jumpstart your development by giving you out-of-the box results views with support for faceting, autocomplete, and detail views. During this talk I will also demonstrate how we have built and deployed lightweight applications that are able to be performant under large user loads, with minimal server resources.

Building Client-side Search Applications with Solr

lucenerevolution

Presented by Timothy Potter, Founder, Text Centrix Storm is a real-time distributed computation system used to process massive streams of data. Many organizations are turning to technologies like Storm to complement batch-oriented big data technologies, such as Hadoop, to deliver time-sensitive analytics at scale. This talk introduces on an emerging architectural pattern of integrating Solr and Storm to process big data in real time. There are a number of natural integration points between Solr and Storm, such as populating a Solr index or supplying data to Storm using Solr’s real-time get support. In this session, Timothy will cover the basic concepts of Storm, such as spouts and bolts. He’ll then provide examples of how to integrate Solr into Storm to perform large-scale indexing in near real-time. In addition, we'll see how to embed Solr in a Storm bolt to match incoming tuples against pre-configured queries, commonly known as percolator. Attendees will come away from this presentation with a good introduction to stream processing technologies and several real-world use cases of how to integrate Solr with Storm.

Integrate Solr with real-time stream processing applications

lucenerevolution

Configure your Solr cluster to handle hundreds of millions of documents without even noticing, handle queries in milliseconds, use Near Real Time indexing and searching with document versioning. Scale your cluster both horizontally and vertically by using shards and replicas. In this session you'll learn how to make your indexing process blazing fast and make your queries efficient even with large amounts of data in your collections. You'll also see how to optimize your queries to leverage caches as much as your deployment allows and how to observe your cluster with Solr administration panel, JMX, and third party tools. Finally, learn how to make changes to already deployed collections —split their shards and alter their schema by using Solr API.

Scaling Solr with SolrCloud

lucenerevolution

Presented by Rafal Kuć, Consultant and Software engineer, , Sematext Group, Inc. Even though Solr can run without causing any troubles for long periods of time it is very important to monitor and understand what is happening in your cluster. In this session you will learn how to use various tools to monitor how Solr is behaving at a high level, but also on Lucene, JVM, and operating system level. You'll see how to react to what you see and how to make changes to configuration, index structure and shards layout using Solr API. We will also discuss different performance metrics to which you ought to pay extra attention. Finally, you'll learn what to do when things go awry - we will share a few examples of troubleshooting and then dissect what was wrong and what had to be done to make things work again.

Administering and Monitoring SolrCloud Clusters

lucenerevolution

In a recent project with the United States Patent and Trademark Office, Opensource Connections was asked to prototype the next generation of patent search - using Solr and Lucene. An important aspect of this project was the implementation of BRS, a specialized search syntax used by patent examiners during the examination process. In this fast paced session we will relate our experiences and describe how we used a combination of Parboiled (a Parser Expression Grammar [PEG] parser), Lucene Queries and SpanQueries, and an extension of Solr's QParserPlugin to build BRS search functionality in Solr. First we will characterize the patent search problem and then define the BRS syntax itself. We will then introduce the Parboiled parser and discuss various considerations that one must make when designing a syntax parser. Following this we will describe the methodology used to implement the search functionality in Lucene/Solr. Finally, we will include an overview our syntactic and semantic testing strategies. The audience will leave this session with an understanding of how Solr, Lucene, and Parboiled may be used to implement their own custom search parser.

Implementing a Custom Search Syntax using Solr, Lucene, and Parboiled

lucenerevolution

Many of us tend to hate or simply ignore logs, and rightfully so: they’re typically hard to find, difficult to handle, and are cryptic to the human eye. But can we make logs more valuable and more usable if we index them in Solr, so we can search and run real-time statistics on them? Indeed we can, and in this session you’ll learn how to make that happen. In the first part of the session we’ll explain why centralized logging is important, what valuable information one can extract from logs, and we’ll introduce the leading tools from the logging ecosystems everyone should be aware of - from syslog and log4j to LogStash and Flume. In the second part we’ll teach you how to use these tools in tandem with Solr. We’ll show how to use Solr in a SolrCloud setup to index large volumes of logs continuously and efficiently. Then, we'll look at how to scale the Solr cluster as your data volume grows. Finally, we'll see how you can parse your unstructured logs and convert them to nicely structured Solr documents suitable for analytical queries.

Using Solr to Search and Analyze Logs

lucenerevolution

Enhancing relevancy through personalization & semantic search

lucenerevolution

Building real-time notification systems is often limited to basic filtering and pattern matching against incoming records. Allowing users to query incoming documents using Solr's full range of capabilities is much more powerful. In our environment we needed a way to allow for tens of thousands of such query subscriptions, meaning we needed to find a way to distribute the query processing in the cloud. By creating in-memory Lucene indices from our Solr configuration, we were able to parallelize our queries across our cluster. To achieve this distribution, we wrapped the processing in a Storm topology to provide a flexible way to scale and manage our infrastructure. This presentation will describe our experiences creating this distributed, real-time inverted search notification framework.

Real-time Inverted Search in the Cloud Using Lucene and Storm

lucenerevolution

Like many Web-Applications in the past, the Solr Admin UI up until 4.0 was entirely server based. It used separate code on the server to generate their Dashboards, Overviews and Statistics. All that code had to be maintained and still ... you weren't really able to use that kind of data for the things you needed it for. It was wrapped into HTML, most of the time difficult to extract and changed the structure from time to time w/o announcement. After a short look back, we're going to look into the current state of the Solr Admin UI - a client-side application, running completely in your browser. We'll see how it works, where it gets its data from and how you can get the very same data and wire that into your own custom applications, dashboards and/oder monitoring systems.

Solr's Admin UI - Where does the data come from?

lucenerevolution

Schemaless Solr and the Solr Schema REST API

lucenerevolution

Presented by Renaud Delbru, Co-Founder, SindiceTech In this presentation, we will discuss how Lucene and Solr can be used for very efficient search of tree-shaped schemaless document, e.g. JSON or XML, and can be then made to address both graph and relational data search. We will discuss the capabilities of SIREn, a Lucene/Solr plugin we have developed to deal with huge collections of tree-shaped schemaless documents, and how SIREn is built using Lucene extensibility capabilities (Analysis, Codec, Flexible Query Parser). We will compare it with Lucene's BlockJoin Query API in nested schemaless data intensive scenarios. We will then go through use cases that show how relational or graph data can be turned into JSON documents using Hadoop and Pig, and how this can be used in conjunction with SIREn to create relational faceting systems with unprecedented performance. Take-away lessons from this session will be awareness about using Lucene/Solr and Hadoop for relational and graph data search, as well as the awareness that it is now possible to have relational faceted browsers with sub-second response time on commodity hardware.

High Performance JSON Search and Relational Faceted Browsing with Lucene

lucenerevolution

In this session we will show how to build a text classifier using the Apache Lucene/Solr with libSVM libraries. We classify our corpus of job offers into a number of predefined categories. Each indexed document (a job offer) then belongs to zero, one or more categories. Known machine learning techniques for text classification include naïve bayes model, logistic regression, neural network, support vector machine (SVM), etc. We use Lucene/Solr to construct the features vector. Then we use the libsvm library known as the reference implementation of the SVM model to classify the document. We construct as many one-vs-all svm classifiers as there are classes in our setting, then using the Hadoop MapReduce Framework we reconcile the result of our classifiers. The end result is a scalable multi-class classifier. Finally we outline how the classifier is used to enrich basic solr keyword search.

Text Classification with Lucene/Solr, Apache Hadoop and LibSVM

lucenerevolution

Faceted search is a powerful technique to let users easily navigate the search results. It can also be used to develop rich user interfaces, which give an analyst quick insights about the documents space. In this session I will introduce the Facets module, how to use it, under-the-hood details as well as optimizations and best practices. I will also describe advanced faceted search capabilities with Lucene Facets.

Faceted Search with Lucene

lucenerevolution

Presented by Shai Erera, Researcher, IBM Lucene's arsenal has recently expanded to include two new modules: Index Sorting and Replication. Index sorting lets you keep an index consistently sorted based on some criteria (e.g. modification date). This allows for efficient search early-termination as well as achieve better index compression. Index replication lets you replicate a search index to achieve high-availability, fault tolerance as well as take hot index backups. In this talk we will introduce these modules, discuss implementation and design details as well as best practices.

Recent Additions to Lucene Arsenal

lucenerevolution

As part of their work with large media monitoring companies, Flax has developed a technique for applying tens of thousands of stored Lucene queries to a document in under a second. We'll talk about how we built intelligent filters to reduce the number of actual queries applied and how we extended Lucene to extract the exact hit positions of matches, the challenges of implementation, and how it can be used, including applications that monitor hundreds of thousands of news stories every day.

Turning search upside down

lucenerevolution

Presented by Xavier Sanchez Loro, Ph.D, Trovit Search SL This session aims to explain the implementation and use case for spellchecking in Trovit search engine. Trovit is a classified ads search engine supporting several different sites, one for each on country and vertical. Our search engine supports multiple indexes in multiple languages, each with several millions of indexed ads. Those indexes are segmented in several different sites depending on the type of ads (homes, cars, rentals, products, jobs and deals). We have developed a multi-language spellchecking system using solr and lucene in order to help our users to better find the desired ads and avoid the dreaded 0 results as much as possible. As such our goal is not pure orthographic correction, but also suggestion of correct searches for a certain site.

Spellchecking in Trovit: Implementing a Contextual Multi-language Spellchecke...

lucenerevolution

Shrinking the haystack wes caldwell - final

lucenerevolution

Plus de lucenerevolution (20)

Text Classification Powered by Apache Mahout and Lucene

State of the Art Logging. Kibana4Solr is Here!

Search at Twitter

Building Client-side Search Applications with Solr

Integrate Solr with real-time stream processing applications

Scaling Solr with SolrCloud

Administering and Monitoring SolrCloud Clusters

Implementing a Custom Search Syntax using Solr, Lucene, and Parboiled

Using Solr to Search and Analyze Logs

Enhancing relevancy through personalization & semantic search

Real-time Inverted Search in the Cloud Using Lucene and Storm

Solr's Admin UI - Where does the data come from?

Schemaless Solr and the Solr Schema REST API

High Performance JSON Search and Relational Faceted Browsing with Lucene

Text Classification with Lucene/Solr, Apache Hadoop and LibSVM

Faceted Search with Lucene

Recent Additions to Lucene Arsenal

Turning search upside down

Spellchecking in Trovit: Implementing a Contextual Multi-language Spellchecke...

Shrinking the haystack wes caldwell - final

Dernier

Architecting Cloud Native Applications

WSO2

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Edi Saputra

MINDCTI Revenue Release Quarter One 2024

MIND CTI

The Good, the Bad and the Governed - Why is governance a dirty word? David O'Neill, Chief Operating Officer - APIContext Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

apidays

Corporate and higher education. Two industries that, in the past, have had a clear divide with very little crossover. The difference in goals, learning styles and objectives paved the way for differing learning technologies platforms to evolve. Now, those stark lines are blurring as both sides are discovering they have content that’s relevant to the other. Join Tammy Rutherford as she walks through the pros and cons of corporate and higher ed collaborating. And the challenges of these different technology platforms working together for a brighter future.

Corporate and higher education May webinar.pptx

Rustici Software

The value of a flexible API Management solution for Open Banking Steve Melan, Manager for IT Innovation and Architecture - State's and Saving's Bank of Luxembourg Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The value of a flexible API Management solution for O...

apidays

Keynote 2: APIs in 2030: The Risk of Technological Sleepwalk Paolo Malinverno, Growth Advisor - The Business of Technology Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...

apidays

Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows. We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases. This video focuses on the deployment of external web forms using Jotform for Bonterra Impact Management. This solution can be customized to your organization’s needs and deployed to support the common use cases below: - Intake and consent - Assessments - Surveys - Applications - Program registration Interested in deploying web form automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.

Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...

Jeffrey Haguewood

"I see eyes in my soup": How Delivery Hero implemented the safety system for ...

Zilliz

Three things you will take away from the session: • How to run an effective tenant-to-tenant migration • Best practices for before, during, and after migration • Tips for using migration as a springboard to prepare for Copilot in Microsoft 365 Main ideas: Migration Overview: The presentation covers the current reality of cross-tenant migrations, the triggers, phases, best practices, and benefits of a successful tenant migration Considerations: When considering a migration, it is important to consider the migration scope, performance, customization, flexibility, user-friendly interface, automation, monitoring, support, training, scalability, data integrity, data security, cost, and licensing structure Next Wave: The next wave of change includes the launch of Copilot, which requires businesses to be prepared for upcoming changes related to Copilot and the cloud, and to consolidate data and tighten governance ShareGate: ShareGate can help with pre-migration analysis, configurable migration tool, and automated, end-user driven collaborative governance

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

sammart93

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Juan lago vázquez

Whatsapp Number Escorts Call girls 8617370543 Available 24x7 Mcleodganj Call Girls Service Offer Genuine VIP Model Escorts Call Girls in Your Budget. Mcleodganj Call Girls Service Provide Real Call Girls Number. Make Your Sexual Pleasure Memorable with Our Mcleodganj Call Girls at Affordable Price. Top VIP Escorts Call Girls, High Profile Independent Escorts Call Girls, Housewife Women Escorts Call Girl, College Girls Escorts Call Girls, Russian Escorts Call girls Service in Your Budget.

Mcleodganj Call Girls 🥰 8617370543 Service Offer VIP Hot Model

Deepika Singh

Dubai, often portrayed as a shimmering oasis in the desert, faces its own set of challenges, including the occasional threat of flooding. Despite its reputation for opulence and modernity, the emirate is not immune to the forces of nature. In recent years, Dubai has experienced sporadic but significant floods, testing the resilience of its infrastructure and communities. Among the critical lifelines in this bustling metropolis is the Dubai International Airport, a bustling hub that connects the city to the world. This article explores the intersection of Dubai flood events and the resilience demonstrated by the Dubai International Airport in the face of such challenges.

Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...

Orbitshub

Effective data discovery is crucial for maintaining compliance and mitigating risks in today's rapidly evolving privacy landscape. However, traditional manual approaches often struggle to keep pace with the growing volume and complexity of data. Join us for an insightful webinar where industry leaders from TrustArc and Privya will share their expertise on leveraging AI-powered solutions to revolutionize data discovery. You'll learn how to: - Effortlessly maintain a comprehensive, up-to-date data inventory - Harness code scanning insights to gain complete visibility into data flows leveraging the advantages of code scanning over DB scanning - Simplify compliance by leveraging Privya's integration with TrustArc - Implement proven strategies to mitigate third-party risks Our panel of experts will discuss real-world case studies and share practical strategies for overcoming common data discovery challenges. They'll also explore the latest trends and innovations in AI-driven data management, and how these technologies can help organizations stay ahead of the curve in an ever-changing privacy landscape.

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

TrustArc

💉💊+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHABI}}+971581248768 +971581248768 Mtp-Kit (500MG) Prices » Dubai [(+971581248768**)] Abortion Pills For Sale In Dubai, UAE, Mifepristone and Misoprostol Tablets Available In Dubai, UAE CONTACT DR.Maya Whatsapp +971581248768 We Have Abortion Pills / Cytotec Tablets /Mifegest Kit Available in Dubai, Sharjah, Abudhabi, Ajman, Alain, Fujairah, Ras Al Khaimah, Umm Al Quwain, UAE, Buy cytotec in Dubai +971581248768''''Abortion Pills near me DUBAI | ABU DHABI|UAE. Price of Misoprostol, Cytotec” +971581248768' Dr.DEEM ''BUY ABORTION PILLS MIFEGEST KIT, MISOPROTONE, CYTOTEC PILLS IN DUBAI, ABU DHABI,UAE'' Contact me now via What's App…… abortion Pills Cytotec also available Oman Qatar Doha Saudi Arabia Bahrain Above all, Cytotec Abortion Pills are Available In Dubai / UAE, you will be very happy to do abortion in Dubai we are providing cytotec 200mg abortion pill in Dubai, UAE. Medication abortion offers an alternative to Surgical Abortion for women in the early weeks of pregnancy. We only offer abortion pills from 1 week-6 Months. We then advise you to use surgery if its beyond 6 months. Our Abu Dhabi, Ajman, Al Ain, Dubai, Fujairah, Ras Al Khaimah (RAK), Sharjah, Umm Al Quwain (UAQ) United Arab Emirates Abortion Clinic provides the safest and most advanced techniques for providing non-surgical, medical and surgical abortion methods for early through late second trimester, including the Abortion By Pill Procedure (RU 486, Mifeprex, Mifepristone, early options French Abortion Pill), Tamoxifen, Methotrexate and Cytotec (Misoprostol). The Abu Dhabi, United Arab Emirates Abortion Clinic performs Same Day Abortion Procedure using medications that are taken on the first day of the office visit and will cause the abortion to occur generally within 4 to 6 hours (as early as 30 minutes) for patients who are 3 to 12 weeks pregnant. When Mifepristone and Misoprostol are used, 50% of patients complete in 4 to 6 hours; 75% to 80% in 12 hours; and 90% in 24 hours. We use a regimen that allows for completion without the need for surgery 99% of the time. All advanced second trimester and late term pregnancies at our Tampa clinic (17 to 24 weeks or greater) can be completed within 24 hours or less 99% of the time without the need surgery. The procedure is completed with minimal to no complications. Our Women's Health Center located in Abu Dhabi, United Arab Emirates, uses the latest medications for medical abortions (RU-486, Mifeprex, Mifegyne, Mifepristone, early options French abortion pill), Methotrexate and Cytotec (Misoprostol). The safety standards of our Abu Dhabi, United Arab Emirates Abortion Doctors remain unparalleled. They consistently maintain the lowest complication rates throughout the nation. Our Physicians and staff are always available to answer questions and care for women in one of the most difficult times in their lives. The decision to have an abortion at the Abortion Cl

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

?#DUbAI#??##{{(☎️+971_581248768%)**%*]'#abortion pills for sale in dubai@

Tracing the root cause of a performance issue requires a lot of patience, experience, and focus. It’s so hard that we sometimes attempt to guess by trying out tentative fixes, but that usually results in frustration, messy code, and a considerable waste of time and money. This talk explains how to correctly zoom in on a performance bottleneck using three levels of profiling: distributed tracing, metrics, and method profiling. After we learn to read the JVM profiler output as a flame graph, we explore a series of bottlenecks typical for backend systems, like connection/thread pool starvation, invisible aspects, blocking code, hot CPU methods, lock contention, and Virtual Thread pinning, and we learn to trace them even if they occur in library code you are not familiar with. Attend this talk and prepare for the performance issues that will eventually hit any successful system. About authorWith two decades of experience, Victor is a Java Champion working as a trainer for top companies in Europe. Five thousands developers in 120 companies attended his workshops, so he gets to debate every week the challenges that various projects struggle with. In return, Victor summarizes key points from these workshops in conference talks and online meetups for the European Software Crafters, the world’s largest developer community around architecture, refactoring, and testing. Discover how Victor can help you on victorrentea.ro : company training catalog, consultancy and YouTube playlists.

Finding Java's Hidden Performance Traps @ DevoxxUK 2024

Victor Rentea

When you’re building (micro)services, you have lots of framework options. Spring Boot is no doubt a popular choice. But there’s more! Take Quarkus, a framework that’s considered the rising star for Kubernetes-native Java. It always depends on what's best for your situation, but how to choose the best solution if you're comparing 2 frameworks? Both Spring Boot and Quarkus have their positives and negatives. Let us compare the two by live coding a couple of common use cases in Spring Boot and Quarkus. After this talk, you’ll be ready to get started with Quarkus yourself, and know when to select Quarkus or Spring Boot.

Spring Boot vs Quarkus the ultimate battle - DevoxxUK

Jago de Vreede

CNIC Information System with Pakdata Cf In Pakistan

danishmna97

The microservices honeymoon is over. When starting a new project or revamping a legacy monolith, teams started looking for alternatives to microservices. The Modular Monolith, or 'Modulith', is an architecture that reaps the benefits of (vertical) functional decoupling without the high costs associated with separate deployments. This talk will delve into the advantages and challenges of this progressive architecture, beginning with exploring the concept of a 'module', its internal structure, public API, and inter-module communication patterns. Supported by spring-modulith, the talk provides practical guidance on addressing the main challenges of a Modultith Architecture: finding and guarding module boundaries, data decoupling, and integration module-testing. You should not miss this talk if you are a software architect or tech lead seeking practical, scalable solutions. About the author With two decades of experience, Victor is a Java Champion working as a trainer for top companies in Europe. Five thousands developers in 120 companies attended his workshops, so he gets to debate every week the challenges that various projects struggle with. In return, Victor summarizes key points from these workshops in conference talks and online meetups for the European Software Crafters, the world’s largest developer community around architecture, refactoring, and testing. Discover how Victor can help you on victorrentea.ro : company training catalog, consultancy and YouTube playlists.

Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024

Victor Rentea

Strategies for Landing an Oracle DBA Job as a Fresher

Remote DBA Services

Dernier (20)

Architecting Cloud Native Applications

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

MINDCTI Revenue Release Quarter One 2024

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

Corporate and higher education May webinar.pptx

Apidays New York 2024 - The value of a flexible API Management solution for O...

Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...

Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...

"I see eyes in my soup": How Delivery Hero implemented the safety system for ...

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Mcleodganj Call Girls 🥰 8617370543 Service Offer VIP Hot Model

Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

Finding Java's Hidden Performance Traps @ DevoxxUK 2024

Spring Boot vs Quarkus the ultimate battle - DevoxxUK

CNIC Information System with Pakdata Cf In Pakistan

Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024

Strategies for Landing an Oracle DBA Job as a Fresher

Lucene at Yelp - By Sudarshan Gaikaiwari

1. Lucene @ Yelp Sudarshan Gaikaiwari

2. Bio 1. Over a decade of experience in information retrieval 2. Used IR techniques at Symantec's DLP group 3. Search Engineer at Yelp

3. Outline 1. Overview of search services at Yelp 2. Federation Motivation 3. Lucy Indexing 4. Lucy Searching 5. Efficiently Retrieving top k hits

4. The services we provide

5. Lucy: business search

6. Lucy also powers phone search

7. Cathy: she 'talks' a lot

8. Listsearch: it searches lists....

9. Reviewsearch: it searches reviews....

10. DYM: did you really mean that?

11. Suggest: auto completion

12. Federation Motivation

13. Problem Search is too slow

14. Hard Disk Seek Latency Disk seek 10,000,000 ns Source Software Engineering Advice from Building Large-Scale Distributed Systems Jeffery Dean

15. RAM read latency Main memory reference 100 ns

16. Pinning Index in RAM ● vmtouch ● mlock ● http://hoytech.com/vmtouch/

17. Problem Index is too large fit in memory on a single machine

18. Geographical sharding

19. Geographical Sharding drawbacks 1. Cumbersome manual process to determine shard boundary 2. No guarantee that a boundary can be found.

20. Federation 1. Split index across multiple machines 2. Shard on business id 3. TF-IDF scores from different machines should be comparable

21. Mapping businesses to shards 1. Assigning businesses to shards shard = shardlist[hash(business_id) % len(shardlist)] Problems 1. Involves re-indexing all the businesses if we want to add a new shard

22. Virtual Nodes

23. Advantages 1. Flexibility (move vbuckets from one shard to another) 2. Split hot spot shards

24. Lucy Master Slave Architecture Separate indexing (masters) A master for each shard of a service Searching (slaves) A slave for every replica of a service

25. Lucy Indexing

26.

27.

28.

29.

30. Lucy Searching

31.

32. Federator: Combining results across shards 1. Once we distribute an index across shards we need a component which will search all these shards and combine their results. 2. Written in Python (runs inside a python web process). 3. Uses Tornado IO loop to send requests to all shards. 4. The transfer protocol for the requests in JSON RPC

33. Lucy Server

34.

35.

36. Tokens to Business Attributes

37. Executing queries 1. Gather the top results for a query 2. Collect attribute statitics for attributes like places, categories

38. Lucene 1. Efficiently executes queries over the index 2. Provides how relevant the business is to the words in the query (word score) 3. Upgrading lucene to 2.9/3.1 is WIP

39.

40. Successive geobounds relaxation

41. Successive geobounds relaxation

42. Federation

43. Efficiently Retrieving top k hits 1. When user moves through multiple pages the number of hits to be returned increases num hits = start + count 2. So if we need to retrieve 500 hits the naive way would be to retrieve 500 hits from each shard and then sort them

44. Distribution of hits in shards

45.

46. Probability a hit is in a shard

47. Binomial Distribution Probability (r of top k hits) are in a particular shard Mean Variance

48. Formula Std Deviation Formula

49. Simulation Formula Hits selected from each Results Missed (%) shard k = 100 p = 0.2 24 0.017 32 0.0001407 44 0.00000

50. Simulation Graph

51. Results 1. ~ 50% savings over 100 hits (44 hits requested from each shard) 2. 77% savings over 1000 hits (228 hits requested from each shard)

52. Future work 1. In memory index 2. Move towards real time search

53. Come Join Us!

54. Thank You smg@yelp.com

Lucene at Yelp - By Sudarshan Gaikaiwari

Recommandé

Recommandé

Contenu connexe

En vedette

En vedette (6)

Similaire à Lucene at Yelp - By Sudarshan Gaikaiwari

Similaire à Lucene at Yelp - By Sudarshan Gaikaiwari (20)

Plus de lucenerevolution

Plus de lucenerevolution (20)

Dernier

Dernier (20)

Lucene at Yelp - By Sudarshan Gaikaiwari