Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
Prochain SlideShare
The Search Is Over: Integrating Solr and Hadoop in the Same Cluster to Simplify Big Data Analytics
Suivant
Télécharger pour lire hors ligne et voir en mode plein écran

1

Partager

Télécharger pour lire hors ligne

Integrating Hadoop & Solr

Télécharger pour lire hors ligne

Silicon Valley Code Camp 2014: presented by Yann Yu, Systems Engineer, Lucidworks.

Integrating Hadoop & Solr

  1. 1. Who am I? Yann Yu Systems Engineer @ Lucidworks
  2. 2. Lucidworks is Search. Technology Retail Financial Healthcare Services Industrial
  3. 3. Why would you integrate Hadoop and Solr? (and how would you do that?)
  4. 4. • Open-source • Enterprise support • Cheap, scalable storage • Distributed computation • Farm animals for extensibility • Open-source, Lucene based • Enterprise support • Real-time queries • Full-text search • NoSQL capabilities • Repeatedly proven in production environments at massive scales
  5. 5. I have Hadoop, why do I need Solr? Hadoop excels in storing and working with large amounts of data, but has difficulty with frequent, random access to it • NoSQL front-end to Hadoop: Enable fast, ad-hoc, search across structured and unstructured big data • Empower users of all technical ability to interact with, and derive value from, big data — all using a natural language search interface (no MapReduce, Pig, SQL, etc.) • Preliminary data exploration and analysis • Near real-time indexing and querying • Thousands of simultaneous, parallel requests • Share machine-learning insights created on Hadoop to a broad audience through an interactive medium
  6. 6. I have Solr, why do I need Hadoop? As Solr indexes grow in size, the size and number of the machines hosting Solr must also grow, increasing index time and complexity • Least expensive storage solution in market • Leverage Hadoop processing power (MapReduce) to build indexes or send document updates to Solr • Store Solr indexes and transaction logs within HDFS • Augment Solr data by storing additional information for last-second retrieval in Hadoop
  7. 7. So what does this actually look like? ?
  8. 8. The enterprise storage situation today ⚒
  9. 9. Enterprise data deployment Lucidworks HDFS connector processes documents and sends to SolrCloud Enterprise documents are stored in HDFS And retrieve source files directly from HDFS as necessary Users make ad-hoc, full-text queries across the full content of all documents in Solr Standard document storage and search
  10. 10. • Documents can be migrated from other file storage systems via Flume or other scripts • MapReduce allows for batch processing of documents (e.g. OCR, NER, clustering, etc.) Sink documents into HDFS
  11. 11. Index document contents into Solr • The Lucidworks Hadoop connector parses content from files using many different tools • Tika, GrokIngest, CSV mapping, Pig, etc. • Content and data are added to fields in a Solr document • The resulting document is sent to Solr for indexing
  12. 12. Enable users to search and access content • Users are empowered with ad-hoc, full-text search in Solr • Provides standard search tools such as autocomplete, more-like-this, spellchecking, faceting, etc. • Users only access HDFS as needed
  13. 13. Log record search Machine generated log records are sent to Flume. Flume forwards raw log record to Hadoop for archiving. Flume simultaneously parses out data in record into a Solr document, forwarding resulting document to Solr Lucidworks SiLK exposes real-time statistics and analytics to end-users, as well as full-text search High volume indexing of many small records
  14. 14. Flume archives data in HDFS • Flume performs minimal work on log files and sends them directly into HDFS for archival • Under optimal circumstances, the log files are sized to the block size of HDFS
  15. 15. Flume submits records to Solr • Flume processes records, extracting strings, ints, dates, times, and other information into Solr fields • Once the Solr document is created, it is submitted to Solr for indexing • This process happens in real-time, allowing for near real-time search
  16. 16. Real-time analytics dashboard • Lucidworks SiLK allows users to create simple dashboards through a GUI • The Banana dashboard will issue queries to Solr, rendering the received data in tables, graphs, and other plots • Users can also perform full-text search across the data, allowing for extremely fine granularity
  17. 17. End Find me at: yann.yu@lucidworks.com @yawnyou Any questions?
  • SpyrosAvramis

    Sep. 1, 2016

Silicon Valley Code Camp 2014: presented by Yann Yu, Systems Engineer, Lucidworks.

Vues

Nombre de vues

1 106

Sur Slideshare

0

À partir des intégrations

0

Nombre d'intégrations

3

Actions

Téléchargements

24

Partages

0

Commentaires

0

Mentions J'aime

1

×