Building Data Pipelines for Solr with Apache NiFiBryan Bende
This document provides an overview of using Apache NiFi to build data pipelines that index data into Apache Solr. It introduces NiFi and its capabilities for data routing, transformation and monitoring. It describes how Solr accepts data through different update handlers like XML, JSON and CSV. It demonstrates how NiFi processors can be used to stream data to Solr via these update handlers. Example use cases are presented for indexing tweets, commands, logs and databases into Solr collections. Future enhancements are discussed like parsing documents and distributing commands across a Solr cluster.
Vladimir Rodionov (Hortonworks)
Time-series applications (sensor data, application/system logging events, user interactions etc) present a new set of data storage challenges: very high velocity and very high volume of data. This talk will present the recent development in Apache HBase that make it a good fit for time-series applications.
Meta/Facebook's database serving social workloads is running on top of MyRocks (MySQL on RocksDB). This means our performance and reliability depends a lot on RocksDB. Not just MyRocks, but also we have other important systems running on top of RocksDB. We have learned many lessons from operating and debugging RocksDB at scale.
In this session, we will offer an overview of RocksDB, key differences from InnoDB, and share a few interesting lessons learned from production.
This document discusses using Apache Spark and Apache NiFi together for data lakes. It outlines the goals of a data lake including having a central data repository, reducing costs, enabling easier discovery and prototyping. It also discusses what is needed for a Hadoop data lake, including automation of pipelines, governance, and interactive data discovery. The document then provides an example ingestion project and describes using Apache Spark for functions like cleansing, validating, and profiling data. It outlines using Apache NiFi for the pipeline design with drag and drop functionality. Finally, it demonstrates ingesting and preparing data, data self-service and transformation, data discovery, and operational monitoring capabilities.
This document summarizes techniques for optimizing Hive queries, including recommendations around data layout, format, joins, and debugging. It discusses partitioning, bucketing, sort order, normalization, text format, sequence files, RCFiles, ORC format, compression, shuffle joins, map joins, sort merge bucket joins, count distinct queries, using explain plans, and dealing with skew.
Kudu is popularly referred to as "Fast Analytics on Fast Data" capable of performing both OLAP & OLTP operations. Understand right from essentials to deep-dive into Kudu internals and architecture for building applications based on Kudu and integrating with Hadoop ecosystem.
Read about Kudu clusters, architecture, operations, primary key design and column optimizations, partitioning and other performance considerations.
Building robust CDC pipeline with Apache Hudi and DebeziumTathastu.ai
We have covered the need for CDC and the benefits of building a CDC pipeline. We will compare various CDC streaming and reconciliation frameworks. We will also cover the architecture and the challenges we faced while running this system in the production. Finally, we will conclude the talk by covering Apache Hudi, Schema Registry and Debezium in detail and our contributions to the open-source community.
Lightweight locks (LWLocks) in PostgreSQL provide mutually exclusive access to shared memory structures. They support both shared and exclusive locking modes. The LWLocks framework uses wait queues, semaphores, and spinlocks to efficiently manage acquiring and releasing locks. Dynamic monitoring of LWLock events is possible through special builds that incorporate statistics collection.
Building Data Pipelines for Solr with Apache NiFiBryan Bende
This document provides an overview of using Apache NiFi to build data pipelines that index data into Apache Solr. It introduces NiFi and its capabilities for data routing, transformation and monitoring. It describes how Solr accepts data through different update handlers like XML, JSON and CSV. It demonstrates how NiFi processors can be used to stream data to Solr via these update handlers. Example use cases are presented for indexing tweets, commands, logs and databases into Solr collections. Future enhancements are discussed like parsing documents and distributing commands across a Solr cluster.
Vladimir Rodionov (Hortonworks)
Time-series applications (sensor data, application/system logging events, user interactions etc) present a new set of data storage challenges: very high velocity and very high volume of data. This talk will present the recent development in Apache HBase that make it a good fit for time-series applications.
Meta/Facebook's database serving social workloads is running on top of MyRocks (MySQL on RocksDB). This means our performance and reliability depends a lot on RocksDB. Not just MyRocks, but also we have other important systems running on top of RocksDB. We have learned many lessons from operating and debugging RocksDB at scale.
In this session, we will offer an overview of RocksDB, key differences from InnoDB, and share a few interesting lessons learned from production.
This document discusses using Apache Spark and Apache NiFi together for data lakes. It outlines the goals of a data lake including having a central data repository, reducing costs, enabling easier discovery and prototyping. It also discusses what is needed for a Hadoop data lake, including automation of pipelines, governance, and interactive data discovery. The document then provides an example ingestion project and describes using Apache Spark for functions like cleansing, validating, and profiling data. It outlines using Apache NiFi for the pipeline design with drag and drop functionality. Finally, it demonstrates ingesting and preparing data, data self-service and transformation, data discovery, and operational monitoring capabilities.
This document summarizes techniques for optimizing Hive queries, including recommendations around data layout, format, joins, and debugging. It discusses partitioning, bucketing, sort order, normalization, text format, sequence files, RCFiles, ORC format, compression, shuffle joins, map joins, sort merge bucket joins, count distinct queries, using explain plans, and dealing with skew.
Kudu is popularly referred to as "Fast Analytics on Fast Data" capable of performing both OLAP & OLTP operations. Understand right from essentials to deep-dive into Kudu internals and architecture for building applications based on Kudu and integrating with Hadoop ecosystem.
Read about Kudu clusters, architecture, operations, primary key design and column optimizations, partitioning and other performance considerations.
Building robust CDC pipeline with Apache Hudi and DebeziumTathastu.ai
We have covered the need for CDC and the benefits of building a CDC pipeline. We will compare various CDC streaming and reconciliation frameworks. We will also cover the architecture and the challenges we faced while running this system in the production. Finally, we will conclude the talk by covering Apache Hudi, Schema Registry and Debezium in detail and our contributions to the open-source community.
Lightweight locks (LWLocks) in PostgreSQL provide mutually exclusive access to shared memory structures. They support both shared and exclusive locking modes. The LWLocks framework uses wait queues, semaphores, and spinlocks to efficiently manage acquiring and releasing locks. Dynamic monitoring of LWLock events is possible through special builds that incorporate statistics collection.
A Thorough Comparison of Delta Lake, Iceberg and HudiDatabricks
Recently, a set of modern table formats such as Delta Lake, Hudi, Iceberg spring out. Along with Hive Metastore these table formats are trying to solve problems that stand in traditional data lake for a long time with their declared features like ACID, schema evolution, upsert, time travel, incremental consumption etc.
ORC files were originally introduced in Hive, but have now migrated to an independent Apache project. This has sped up the development of ORC and simplified integrating ORC into other projects, such as Hadoop, Spark, Presto, and Nifi. There are also many new tools that are built on top of ORC, such as Hive’s ACID transactions and LLAP, which provides incredibly fast reads for your hot data. LLAP also provides strong security guarantees that allow each user to only see the rows and columns that they have permission for.
This talk will discuss the details of the ORC and Parquet formats and what the relevant tradeoffs are. In particular, it will discuss how to format your data and the options to use to maximize your read performance. In particular, we’ll discuss when and how to use ORC’s schema evolution, bloom filters, and predicate push down. It will also show you how to use the tools to translate ORC files into human-readable formats, such as JSON, and display the rich metadata from the file including the type in the file and min, max, and count for each column.
The document discusses compaction in RocksDB, an embedded key-value storage engine. It describes the two compaction styles in RocksDB: level style compaction and universal style compaction. Level style compaction stores data in multiple levels and performs compactions by merging files from lower to higher levels. Universal style compaction keeps all files in level 0 and performs compactions by merging adjacent files in time order. The document provides details on the compaction process and configuration options for both styles.
MyRocks is an open source LSM based MySQL database, created by Facebook. This slides introduce MyRocks overview and how we deployed at Facebook, as of 2017.
This document discusses reindexing large repositories in Alfresco. It covers the Alfresco SOLR architecture, the indexing process, scenarios that require reindexing, alternatives for deployment during reindexing to minimize downtime, monitoring and profiling tools, and future improvements planned for Search Services 2.0 to optimize indexing performance. Benchmark results are presented showing improvements that reduced reindexing time for 1.2 billion documents from 21 days to 10 days.
Netflix’s Big Data Platform team manages data warehouse in Amazon S3 with over 60 petabytes of data and writes hundreds of terabytes of data every day. At this scale, output committers that create extra copies or can’t handle task failures are no longer practical. This talk will explain the problems that are caused by the available committers when writing to S3, and show how Netflix solved the committer problem.
In this session, you’ll learn:
– Some background about Spark at Netflix
– About output committers, and how both Spark and Hadoop handle failures
– How HDFS and S3 differ, and why HDFS committers don’t work well
– A new output committer that uses the S3 multi-part upload API
– How you can use this new committer in your Spark applications to avoid duplicating data
Apache Iceberg - A Table Format for Hige Analytic DatasetsAlluxio, Inc.
Data Orchestration Summit
www.alluxio.io/data-orchestration-summit-2019
November 7, 2019
Apache Iceberg - A Table Format for Hige Analytic Datasets
Speaker:
Ryan Blue, Netflix
For more Alluxio events: https://www.alluxio.io/events/
Maxim Fateev - Beyond the Watermark- On-Demand Backfilling in FlinkFlink Forward
This document discusses an on-demand backfilling solution for Flink pipelines that process incentive programs. It describes a pipeline that evaluates driver incentives based on their status changes. Some incentives are defined retroactively, so an on-demand source was created that can continuously backfill old data. This source emits state changes paired with the applicable incentives from the beginning of each period. It allows a single pipeline to process both new and retroactive incentives without needing separate pipelines. The document also suggests this approach could be generalized into a pipeline template.
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013mumrah
Apache Kafka is a distributed publish-subscribe messaging system that allows both publishing and subscribing to streams of records. It uses a distributed commit log that provides low latency and high throughput for handling real-time data feeds. Key features include persistence, replication, partitioning, and clustering.
1. Log structured merge trees store data in multiple levels with different storage speeds and costs, requiring data to periodically merge across levels.
2. This structure allows fast writes by storing new data in faster levels before merging to slower levels, and efficient reads by querying multiple levels and merging results.
3. The merging process involves loading, sorting, and rewriting levels to consolidate and propagate deletions and updates between levels.
This document provides a summary of improvements made to Hive's performance through the use of Apache Tez and other optimizations. Some key points include:
- Hive was improved to use Apache Tez as its execution engine instead of MapReduce, reducing latency for interactive queries and improving throughput for batch queries.
- Statistics collection was optimized to gather column-level statistics from ORC file footers, speeding up statistics gathering.
- The cost-based optimizer Optiq was added to Hive, allowing it to choose better execution plans.
- Vectorized query processing, broadcast joins, dynamic partitioning, and other optimizations improved individual query performance by over 100x in some cases.
Data in Hadoop is getting bigger every day, consumers of the data are growing, organizations are now looking at making their Hadoop cluster compliant to federal regulations and commercial demands. Apache Ranger simplifies the management of security policies across all components in Hadoop. Ranger provides granular access controls to data.
The deck describes what security tools are available in Hadoop and their purpose then it moves on to discuss in detail Apache Ranger.
ELK Elasticsearch Logstash and Kibana Stack for Log ManagementEl Mahdi Benzekri
Initiation to the powerful Elasticsearch Logstash and Kibana stack, it has many use cases, the popular one is the server and application log management.
Infrastructure, use cases and performance considerations for
an Enterprise Grade ECM implementation up to 1B documents on AWS (Amazon Web Services EC2 and Aurora) based on the Alfresco (http://www.alfresco.com) Platform, leading Open Source Enterprise Content Management system.
Introduction to memcached, a caching service designed for optimizing performance and scaling in the web stack, seen from perspective of MySQL/PHP users. Given for 2nd year students of professional bachelor in ICT at Kaho St. Lieven, Gent.
The document provides steps for debugging a local NiFi processor, including getting the NiFi source code from GitHub, setting up NiFi and an IDE, and launching the IDE in debug mode to trigger breakpoints when a processor starts. It recommends using a feature branch workflow and links to Apache NiFi contribution guides.
The document provides instructions for installing Solr on Windows by downloading and configuring Tomcat and Solr. It describes downloading Tomcat and Solr, configuring server.xml, extracting Solr to c:\web\solr, copying the Solr WAR file to Tomcat, and accessing the Solr admin page at http://localhost:8080/solr/admin to verify the installation.
Common issues with Apache Kafka® Producerconfluent
Badai Aqrandista, Confluent, Senior Technical Support Engineer
This session will be about a common issue in the Kafka Producer: producer batch expiry. We will be discussing the Kafka Producer internals, its common causes, such as a slow network or small batching, and how to overcome them. We will also be sharing some examples along the way!
https://www.meetup.com/apache-kafka-sydney/events/279651982/
Tech Talk: RocksDB Slides by Dhruba Borthakur & Haobo Xu of FacebookThe Hive
This presentation describes the reasons why Facebook decided to build yet another key-value store, the vision and architecture of RocksDB and how it differs from other open source key-value stores. Dhruba describes some of the salient features in RocksDB that are needed for supporting embedded-storage deployments. He explains typical workloads that could be the primary use-cases for RocksDB. He also lays out the roadmap to make RocksDB the key-value store of choice for highly-multi-core processors and RAM-speed storage devices.
Just the Job: Employing Solr for Recruitment Search -Charlie Hull lucenerevolution
See conference video - http://www.lucidimagination.com/devzone/events/conferences/ApacheLuceneEurocon2011
Using a case study on a major European executive recruitment company, we will show how we used Apache Lucene/Solr to build powerful, flexible, accurate and scalable search services over tens of millions of CVs and candidate records, allowing the company to completely restructure their IT provision for both local and national offices.
A Thorough Comparison of Delta Lake, Iceberg and HudiDatabricks
Recently, a set of modern table formats such as Delta Lake, Hudi, Iceberg spring out. Along with Hive Metastore these table formats are trying to solve problems that stand in traditional data lake for a long time with their declared features like ACID, schema evolution, upsert, time travel, incremental consumption etc.
ORC files were originally introduced in Hive, but have now migrated to an independent Apache project. This has sped up the development of ORC and simplified integrating ORC into other projects, such as Hadoop, Spark, Presto, and Nifi. There are also many new tools that are built on top of ORC, such as Hive’s ACID transactions and LLAP, which provides incredibly fast reads for your hot data. LLAP also provides strong security guarantees that allow each user to only see the rows and columns that they have permission for.
This talk will discuss the details of the ORC and Parquet formats and what the relevant tradeoffs are. In particular, it will discuss how to format your data and the options to use to maximize your read performance. In particular, we’ll discuss when and how to use ORC’s schema evolution, bloom filters, and predicate push down. It will also show you how to use the tools to translate ORC files into human-readable formats, such as JSON, and display the rich metadata from the file including the type in the file and min, max, and count for each column.
The document discusses compaction in RocksDB, an embedded key-value storage engine. It describes the two compaction styles in RocksDB: level style compaction and universal style compaction. Level style compaction stores data in multiple levels and performs compactions by merging files from lower to higher levels. Universal style compaction keeps all files in level 0 and performs compactions by merging adjacent files in time order. The document provides details on the compaction process and configuration options for both styles.
MyRocks is an open source LSM based MySQL database, created by Facebook. This slides introduce MyRocks overview and how we deployed at Facebook, as of 2017.
This document discusses reindexing large repositories in Alfresco. It covers the Alfresco SOLR architecture, the indexing process, scenarios that require reindexing, alternatives for deployment during reindexing to minimize downtime, monitoring and profiling tools, and future improvements planned for Search Services 2.0 to optimize indexing performance. Benchmark results are presented showing improvements that reduced reindexing time for 1.2 billion documents from 21 days to 10 days.
Netflix’s Big Data Platform team manages data warehouse in Amazon S3 with over 60 petabytes of data and writes hundreds of terabytes of data every day. At this scale, output committers that create extra copies or can’t handle task failures are no longer practical. This talk will explain the problems that are caused by the available committers when writing to S3, and show how Netflix solved the committer problem.
In this session, you’ll learn:
– Some background about Spark at Netflix
– About output committers, and how both Spark and Hadoop handle failures
– How HDFS and S3 differ, and why HDFS committers don’t work well
– A new output committer that uses the S3 multi-part upload API
– How you can use this new committer in your Spark applications to avoid duplicating data
Apache Iceberg - A Table Format for Hige Analytic DatasetsAlluxio, Inc.
Data Orchestration Summit
www.alluxio.io/data-orchestration-summit-2019
November 7, 2019
Apache Iceberg - A Table Format for Hige Analytic Datasets
Speaker:
Ryan Blue, Netflix
For more Alluxio events: https://www.alluxio.io/events/
Maxim Fateev - Beyond the Watermark- On-Demand Backfilling in FlinkFlink Forward
This document discusses an on-demand backfilling solution for Flink pipelines that process incentive programs. It describes a pipeline that evaluates driver incentives based on their status changes. Some incentives are defined retroactively, so an on-demand source was created that can continuously backfill old data. This source emits state changes paired with the applicable incentives from the beginning of each period. It allows a single pipeline to process both new and retroactive incentives without needing separate pipelines. The document also suggests this approach could be generalized into a pipeline template.
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013mumrah
Apache Kafka is a distributed publish-subscribe messaging system that allows both publishing and subscribing to streams of records. It uses a distributed commit log that provides low latency and high throughput for handling real-time data feeds. Key features include persistence, replication, partitioning, and clustering.
1. Log structured merge trees store data in multiple levels with different storage speeds and costs, requiring data to periodically merge across levels.
2. This structure allows fast writes by storing new data in faster levels before merging to slower levels, and efficient reads by querying multiple levels and merging results.
3. The merging process involves loading, sorting, and rewriting levels to consolidate and propagate deletions and updates between levels.
This document provides a summary of improvements made to Hive's performance through the use of Apache Tez and other optimizations. Some key points include:
- Hive was improved to use Apache Tez as its execution engine instead of MapReduce, reducing latency for interactive queries and improving throughput for batch queries.
- Statistics collection was optimized to gather column-level statistics from ORC file footers, speeding up statistics gathering.
- The cost-based optimizer Optiq was added to Hive, allowing it to choose better execution plans.
- Vectorized query processing, broadcast joins, dynamic partitioning, and other optimizations improved individual query performance by over 100x in some cases.
Data in Hadoop is getting bigger every day, consumers of the data are growing, organizations are now looking at making their Hadoop cluster compliant to federal regulations and commercial demands. Apache Ranger simplifies the management of security policies across all components in Hadoop. Ranger provides granular access controls to data.
The deck describes what security tools are available in Hadoop and their purpose then it moves on to discuss in detail Apache Ranger.
ELK Elasticsearch Logstash and Kibana Stack for Log ManagementEl Mahdi Benzekri
Initiation to the powerful Elasticsearch Logstash and Kibana stack, it has many use cases, the popular one is the server and application log management.
Infrastructure, use cases and performance considerations for
an Enterprise Grade ECM implementation up to 1B documents on AWS (Amazon Web Services EC2 and Aurora) based on the Alfresco (http://www.alfresco.com) Platform, leading Open Source Enterprise Content Management system.
Introduction to memcached, a caching service designed for optimizing performance and scaling in the web stack, seen from perspective of MySQL/PHP users. Given for 2nd year students of professional bachelor in ICT at Kaho St. Lieven, Gent.
The document provides steps for debugging a local NiFi processor, including getting the NiFi source code from GitHub, setting up NiFi and an IDE, and launching the IDE in debug mode to trigger breakpoints when a processor starts. It recommends using a feature branch workflow and links to Apache NiFi contribution guides.
The document provides instructions for installing Solr on Windows by downloading and configuring Tomcat and Solr. It describes downloading Tomcat and Solr, configuring server.xml, extracting Solr to c:\web\solr, copying the Solr WAR file to Tomcat, and accessing the Solr admin page at http://localhost:8080/solr/admin to verify the installation.
Common issues with Apache Kafka® Producerconfluent
Badai Aqrandista, Confluent, Senior Technical Support Engineer
This session will be about a common issue in the Kafka Producer: producer batch expiry. We will be discussing the Kafka Producer internals, its common causes, such as a slow network or small batching, and how to overcome them. We will also be sharing some examples along the way!
https://www.meetup.com/apache-kafka-sydney/events/279651982/
Tech Talk: RocksDB Slides by Dhruba Borthakur & Haobo Xu of FacebookThe Hive
This presentation describes the reasons why Facebook decided to build yet another key-value store, the vision and architecture of RocksDB and how it differs from other open source key-value stores. Dhruba describes some of the salient features in RocksDB that are needed for supporting embedded-storage deployments. He explains typical workloads that could be the primary use-cases for RocksDB. He also lays out the roadmap to make RocksDB the key-value store of choice for highly-multi-core processors and RAM-speed storage devices.
Just the Job: Employing Solr for Recruitment Search -Charlie Hull lucenerevolution
See conference video - http://www.lucidimagination.com/devzone/events/conferences/ApacheLuceneEurocon2011
Using a case study on a major European executive recruitment company, we will show how we used Apache Lucene/Solr to build powerful, flexible, accurate and scalable search services over tens of millions of CVs and candidate records, allowing the company to completely restructure their IT provision for both local and national offices.
This document provides an introduction to Apache Solr, an open-source enterprise search platform built on Apache Lucene. It discusses how Solr indexes content, processes search queries, and returns results with features like faceting, spellchecking, and scaling. The document also outlines how Solr works, how to configure and use it, and examples of large companies that employ Solr for search.
This document discusses strategies for implementing big search solutions using Apache Solr to index and search large datasets, referred to as "Big Data". It provides examples of prototyping searches over millions of documents, using Solr in novel ways like as a key-value store, and strategies for scaling indexing through techniques like sharding and distributed architectures. The key lessons emphasized are making systems easy to change and test, embracing failure, automating processes, and thinking about cloud-based architectures when dealing with massive amounts of data and search volume.
Distributed Logging Architecture in the Container EraGlenn Davis
Presentation given at LinuxCon Japan 2016 by Satoshi "Moris" Tagomori (@tagomoris), Treasure Data. Describes various strategies for aggregating log data in a microservices architecture using containers, e.g. Docker.
Distributed Logging Architecture in Container EraSATOSHI TAGOMORI
Distributed Logging Architecture in Container Era
The document discusses distributed logging architecture in the container era. It covers: 1) The difficulties of logging with microservices and containers due to their ephemeral and distributed nature, 2) The need to redesign logging to push logs from containers to destinations quickly without fixed addresses or mappings; 3) Common patterns for distributed logging architectures including source aggregation, destination aggregation, and scaling; and 4) A case study using Docker and Fluentd to implement source aggregation and scaling for logging. Open source solutions are important to keep the logging layer transparent, interoperable, and able to scale independently of applications and infrastructure.
Introduction to Lucene & Solr and UsecasesRahul Jain
Rahul Jain gave a presentation on Lucene and Solr. He began with an overview of information retrieval and the inverted index. He then discussed Lucene, describing it as an open source information retrieval library for indexing and searching. He discussed Solr, describing it as an enterprise search platform built on Lucene that provides distributed indexing, replication, and load balancing. He provided examples of how Solr is used for search, analytics, auto-suggest, and more by companies like eBay, Netflix, and Twitter.
The document discusses Orbit Patent Search, a tool for conducting patent searches. It describes how Orbit allows users to search patent databases, view search results and patent family information, collaborate with other users, and store and monitor patent records. The presentation outlines the key features of Orbit Patent Search including its extensive database, search capabilities, customizable display options, collaboration features, and abilities to manage search results.
Building a Large Scale SEO/SEM Application with Apache Solr: Presented by Rah...Lucidworks
The document discusses building a large scale SEO/SEM application using Apache Solr. It describes some of the key challenges faced in indexing and searching over 40 billion records in the application's database each month. It discusses techniques used to optimize the data import process, create a distributed index across multiple tables, address out of memory errors, and improve search performance through partitioning, index optimization, and external caching.
Sizing an alfresco infrastructure has always been an interesting topic with lots of unrevealed questions. There is no perfect formula that can accurately define what is the perfect sizing for your architecture considering your use case. However, we can provide you with valuable guidance on how to size your Alfresco solution, by asking the right questions, collecting the right numbers, and taking the right assumptions on a very interesting sizing exercise.
How many alfresco servers will you need on your alfresco cluster? How many CPUs/cores do you need on those servers to handle your estimated user concurrency? How do you estimate the sizing and growth of your storage? How much memory do you need on your Solr servers? How many Solr servers do you need to get the response times you require? What are the golden rules that can drive and maintain the success of an Alfresco project?
Apache Geode is an open source in-memory data grid that provides data distribution, replication and high availability. It can be used for caching, messaging and interactive queries. The presentation discusses Geode concepts like cache, region and member. It provides examples of how large companies use Geode for applications requiring real-time response, high concurrency and global data visibility. Geode's performance comes from minimizing data copying and contention through flexible consistency and partitioning. The project is now hosted by Apache and the community is encouraged to get involved through mailing lists, code contributions and example applications.
This document provides a summary of the Solr search platform. It begins with introductions from the presenter and about Lucid Imagination. It then discusses what Solr is, how it works, who uses it, and its main features. The rest of the document dives deeper into topics like how Solr is configured, how to index and search data, and how to debug and customize Solr implementations. It promotes downloading and experimenting with Solr to learn more.
The document discusses building a data platform for analytics in Azure. It outlines common issues with traditional data warehouse architectures and recommends building a data lake approach using Azure Synapse Analytics. The key elements include ingesting raw data from various sources into landing zones, creating a raw layer using file formats like Parquet, building star schemas in dedicated SQL pools or Spark tables, implementing alerting using Log Analytics, and loading data into Power BI. Building the platform with Python pipelines, notebooks, and GitHub integration is emphasized for flexibility, testability and collaboration.
Test driving Azure Search and DocumentDBAndrew Siemer
This document provides an overview and comparison of DocumentDB and Azure Search. It discusses what NoSQL and search are, when each service is better to use, how to set up and structure data in each, and examples of querying. DocumentDB is described as a NoSQL database that uses a flexible JSON document structure and scales easily. Azure Search is an elastic search service that indexes and scores search results. The document provides examples of setting up databases and indexes, adding and querying data, and considerations for different field types and scoring profiles. It also discusses where each service may fit in different parts of an application architecture.
Time Series Databases for IoT (On-premises and Azure)Ivo Andreev
This document discusses choosing the right time series database for IoT data. It compares InfluxDB to SQL Server and other databases.
Some key points made:
- InfluxDB outperforms SQL Server for writes by 40x and queries by 59x for time series data due to its optimized design.
- InfluxDB uses 19x-26x less disk storage than SQL Server for the same data.
- InfluxDB also outperforms MongoDB, Elasticsearch, OpenTSDB, and Cassandra for time series workloads.
- Azure Stream Insights is a managed service but has limited capabilities and can be pricey for high volumes of data.
- InfluxDB is open source, has no dependencies, and
Engage 2020 - Best Practices for analyzing Domino Applicationspanagenda
Let's take an in-depth look at why and how to best analyze your Domino applications from a design and code perspective. We will cover modernization, migration and optimization topics, look at Lotusscript, @Formulas and Java examples, and get a good understanding of the many design elements that make up your applications and how they make a difference. If you're an admin, bring some (Notes) development knowledge along, please. This session is vendor and solution independent and will help you better understand why you should analyze your Domino applications and what to watch out for.
In this On-Demand Webinar, Erik Hatcher, co-founder of Lucid Imagination, co-author of Lucene in Action, and Lucene/Solr PMC member and committer, presents and discusess key features and innovations of Apache Solr 1.4
This document provides an overview of Active Directory fundamentals. It covers key concepts such as domains, trees, forests and sites. It also discusses domain controllers, operations masters, and the role of DNS. The document agenda includes reviewing logical and physical concepts of Active Directory, a DNS overview, Active Directory replication, and operations masters. It aims to help users understand how to manage network resources using Active Directory.
The document provides an overview and agenda for an Apache Solr crash course. It discusses topics such as information retrieval, inverted indexes, metrics for evaluating IR systems, Apache Lucene, the Lucene and Solr APIs, indexing, searching, querying, filtering, faceting, highlighting, spellchecking, geospatial search, and Solr architectures including single core, multi-core, replication, and sharding. It also provides tips on performance tuning, using plugins, and developing a Solr-based search engine.
Similaire à A Practical Introduction to Apache Solr (20)
n this session, we'll simplify the complexities of configuring and troubleshooting mutual TLS (mTLS) within Alfresco environments. Attendees will gain practical insights into certificate management, trust validation, and common challenges encountered during configuration.
We'll showcase and provide custom tools for troubleshooting during the session. These tools can be used with ZIP, Ansible, Docker and Kubernetes deployments.
Event description available in https://hub.alfresco.com/t5/news-announcements/ttl-157-troubleshooting-made-easy-deciphering-alfresco-s-mtls/ba-p/319735/jump-to/first-unread-message
Using Generative AI and Content Service Platforms togetherAngel Borroy López
Slides for FOSDEM 2024 session: https://fosdem.org/2024/schedule/event/fosdem-2024-1858-using-generative-ai-and-content-service-platforms-together/
Describes a framework that provides GenAI operations for documents using a REST API. LLMs are stored locally, so no data is sent away.
It also includes a sample integration with a Content Service Platform (Alfresco), to enhance documents and pictures context information.
Session recording is available in https://ftp.fau.de/fosdem/2024/h2213/fosdem-2024-1858-using-generative-ai-and-content-service-platforms-together.av1.webm
Enhancing Document-Centric Features with On-Premise Generative AI for Alfresc...Angel Borroy López
Oractical guide on integrating Alfresco Community with On-Premise Generative AI.
This session outlines the steps to enhance both existing and new content, demonstrating features such as classification, summarization, translation, and prompting. But this framework allows you to include additional features.
Source code is available in https://github.com/aborroy/alfresco-genai
This presentation describes different methods to produce Alfresco Docker Assets for Docker Compose deployment.
From the previous methods (based in Python, Yeoman and Docker) to the Docker Init with Templates approach.
The recent launch of the Docker Init command has significantly simplified the process of generating Dockerfiles and Docker Compose templates for containerized applications. This presentation aims to explore the evolution of Docker deployment resources generation process, comparing its approach prior to the Docker Init command release and discussing the way forward. Before the introduction of the Docker Init command, I've been delivering some projects like the "alfresco-docker-installer"[1], which provides custom scripts and configurations to streamline the process of deploying Alfresco in Docker containers. These kinds of projects use tools like Yeoman or raw Python. There are some differences between a Docker Template for a technology (Go, Python, Node or Rust) and a Docker Template for a product (like Alfresco) that may be covered when generating automatic deployment resources. This presentation will delve into the methodologies employed before the Docker Init command:
Custom Dockerfile Extension
Compose Template for a complete product deployment, including a set of services like the database, content repository, search engine, or web application
Configuration Management, including techniques such as environment variable injection, externalized configuration files, and configuration overrides
Following the release of the Docker Init command, this presentation will provide insights into the possibilities and advantages it brings to complex products Docker deployment process. A PoC of a Docker Plugin, including this product-oriented approach for docker init, will be demoed live. >> Note that the Open Source Alfresco product is used only to explain the concepts of building a Docker Compose generator with a real example.
This deck includes a description of the Transform Service available for Alfresco 7.4.0.
Secure configuration sample, relying on mTLS, is also discussed.
How to migrate from Alfresco Search Services to Alfresco SearchEnterpriseAngel Borroy López
Presentation on how to move from the Alfresco Search Services product based in Apache Solr to the new Alfresco Search Enterprise integrated with Elasticsearch and Amazon Opensearch.
This presentation describes how to use Podman to replace Docker in the Alfresco 7.4.0 development process.
Alfresco platform is built using containerization technology. Alfresco can utilize containerization platforms like Podman, which provide the necessary tools and infrastructure to create, manage, and run containers.
Podman is presented as an alternative to Docker. Both Docker and Podman can be used effectively for Alfresco development. So consider your familiarity with the tools, preferred workflow, ecosystem support, security requirements, and any specific performance considerations to make the best choice for your Alfresco development needs.
CSP: Evolución de servicios de código abierto en un mundo Cloud NativeAngel Borroy López
Presentación realizada en Openexpo Europe 2023:
https://openexpoeurope.com/es/session/cuando-hyland-encontro-a-alfresco-evolucion-de-servicios-de-codigo-abierto-en-un-mundo-cloud-native/
Presenta una visión evolutiva de las plataformas de gestión documental: ECM, CSP y Cloud Native.
Incluye información relevante de los productos Alfresco, Nuxeo y Hyland Experience.
This presentation describes how to use the BPM Engine included with Alfresco ACS repository.
All the different APIs are covered: Workflow Console UI, REST API and Java API.
Support material for the blog post available in https://hub.alfresco.com/t5/alfresco-content-services-blog/alfresco-7-3-upgrading-to-transform-core-3-0-0/ba-p/315364
This presentation describes the differences between Alfresco Transform Engine and Alfresco Transform Core 3.0.0.
Deployment, configuration and extension topics for Transform Core are covered.
Practical information for Alfresco integration with AOS (Sharepoint Protocol), Google Drive, Microsoft 365, ONLYOFFICE and Collabora Online.
Additionally ADW support for ONLYOFFICE is provided by https://github.com/atolcd/adf-onlyoffice-extension#installation
Este documento proporciona recursos para aprender Docker, incluyendo documentación, libros, videos de YouTube y la comunidad Docker. Explica cómo instalar Docker en Windows, Mac y Linux, y cubre herramientas como Docker Desktop y Docker Hub. También describe los planes de suscripción disponibles para Docker.
This session will provide a guide to Alfresco truststores and keystores. Several live examples will be shown, including the replacement of existing cryptographic stores or certificates. Additionally, a troubleshooting configuration guide for mTLS communication will be provided.
What to do when you have a perfect model for your software but you are constrained by an imperfect business model?
This talk explores the challenges of bringing modelling rigour to the business and strategy levels, and talking to your non-technical counterparts in the process.
8 Best Automated Android App Testing Tool and Framework in 2024.pdfkalichargn70th171
Regarding mobile operating systems, two major players dominate our thoughts: Android and iPhone. With Android leading the market, software development companies are focused on delivering apps compatible with this OS. Ensuring an app's functionality across various Android devices, OS versions, and hardware specifications is critical, making Android app testing essential.
Artificia Intellicence and XPath Extension FunctionsOctavian Nadolu
The purpose of this presentation is to provide an overview of how you can use AI from XSLT, XQuery, Schematron, or XML Refactoring operations, the potential benefits of using AI, and some of the challenges we face.
Unveiling the Advantages of Agile Software Development.pdfbrainerhub1
Learn about Agile Software Development's advantages. Simplify your workflow to spur quicker innovation. Jump right in! We have also discussed the advantages.
WWDC 2024 Keynote Review: For CocoaCoders AustinPatrick Weigel
Overview of WWDC 2024 Keynote Address.
Covers: Apple Intelligence, iOS18, macOS Sequoia, iPadOS, watchOS, visionOS, and Apple TV+.
Understandable dialogue on Apple TV+
On-device app controlling AI.
Access to ChatGPT with a guest appearance by Chief Data Thief Sam Altman!
App Locking! iPhone Mirroring! And a Calculator!!
UI5con 2024 - Bring Your Own Design SystemPeter Muessig
How do you combine the OpenUI5/SAPUI5 programming model with a design system that makes its controls available as Web Components? Since OpenUI5/SAPUI5 1.120, the framework supports the integration of any Web Components. This makes it possible, for example, to natively embed own Web Components of your design system which are created with Stencil. The integration embeds the Web Components in a way that they can be used naturally in XMLViews, like with standard UI5 controls, and can be bound with data binding. Learn how you can also make use of the Web Components base class in OpenUI5/SAPUI5 to also integrate your Web Components and get inspired by the solution to generate a custom UI5 library providing the Web Components control wrappers for the native ones.
Need for Speed: Removing speed bumps from your Symfony projects ⚡️Łukasz Chruściel
No one wants their application to drag like a car stuck in the slow lane! Yet it’s all too common to encounter bumpy, pothole-filled solutions that slow the speed of any application. Symfony apps are not an exception.
In this talk, I will take you for a spin around the performance racetrack. We’ll explore common pitfalls - those hidden potholes on your application that can cause unexpected slowdowns. Learn how to spot these performance bumps early, and more importantly, how to navigate around them to keep your application running at top speed.
We will focus in particular on tuning your engine at the application level, making the right adjustments to ensure that your system responds like a well-oiled, high-performance race car.
Measures in SQL (SIGMOD 2024, Santiago, Chile)Julian Hyde
SQL has attained widespread adoption, but Business Intelligence tools still use their own higher level languages based upon a multidimensional paradigm. Composable calculations are what is missing from SQL, and we propose a new kind of column, called a measure, that attaches a calculation to a table. Like regular tables, tables with measures are composable and closed when used in queries.
SQL-with-measures has the power, conciseness and reusability of multidimensional languages but retains SQL semantics. Measure invocations can be expanded in place to simple, clear SQL.
To define the evaluation semantics for measures, we introduce context-sensitive expressions (a way to evaluate multidimensional expressions that is consistent with existing SQL semantics), a concept called evaluation context, and several operations for setting and modifying the evaluation context.
A talk at SIGMOD, June 9–15, 2024, Santiago, Chile
Authors: Julian Hyde (Google) and John Fremlin (Google)
https://doi.org/10.1145/3626246.3653374
Flutter is a popular open source, cross-platform framework developed by Google. In this webinar we'll explore Flutter and its architecture, delve into the Flutter Embedder and Flutter’s Dart language, discover how to leverage Flutter for embedded device development, learn about Automotive Grade Linux (AGL) and its consortium and understand the rationale behind AGL's choice of Flutter for next-gen IVI systems. Don’t miss this opportunity to discover whether Flutter is right for your project.
When it is all about ERP solutions, companies typically meet their needs with common ERP solutions like SAP, Oracle, and Microsoft Dynamics. These big players have demonstrated that ERP systems can be either simple or highly comprehensive. This remains true today, but there are new factors to consider, including a promising new contender in the market that’s Odoo. This blog compares Odoo ERP with traditional ERP systems and explains why many companies now see Odoo ERP as the best choice.
What are ERP Systems?
An ERP, or Enterprise Resource Planning, system provides your company with valuable information to help you make better decisions and boost your ROI. You should choose an ERP system based on your company’s specific needs. For instance, if you run a manufacturing or retail business, you will need an ERP system that efficiently manages inventory. A consulting firm, on the other hand, would benefit from an ERP system that enhances daily operations. Similarly, eCommerce stores would select an ERP system tailored to their needs.
Because different businesses have different requirements, ERP system functionalities can vary. Among the various ERP systems available, Odoo ERP is considered one of the best in the ERp market with more than 12 million global users today.
Odoo is an open-source ERP system initially designed for small to medium-sized businesses but now suitable for a wide range of companies. Odoo offers a scalable and configurable point-of-sale management solution and allows you to create customised modules for specific industries. Odoo is gaining more popularity because it is built in a way that allows easy customisation, has a user-friendly interface, and is affordable. Here, you will cover the main differences and get to know why Odoo is gaining attention despite the many other ERP systems available in the market.
5. 5
Why should you use Open Source?
• State of the Art Technologies
• Community Support
• Vast Documentation
• Code is accessible
• Customizable
• Mostly free licensing
6. 6
Why should you contribute to Open Source?
• Share Knowledge and Ideas
• Improve established Technologies
• Become part of a Community
• Not only code, all your skills are relevant
• Be useful to the World
8. 8
What is SOLR
• A Search Engine
• A REST-like API
• Built on Lucene
• Open Source
• Blazing-fast
• Scalable
• Fault tolerant
9. 9
Why SOLR
Scalable
Solr scales by distributing work (indexing and query processing) to multiple servers in a cluster.
Ready to deploy
Solr is open source, is easy to install and configure, and provides a preconfigured example to help you get
started.
Optimized for search
Solr is fast and can execute complex queries in subsecond speed, often only tens of milliseconds.
Large volumes of documents
Solr is designed to deal with indexes containing many millions of documents.
Text-centric
Solr is optimized for searching natural-language text, like emails, web pages, resumes, PDF documents,
and social messages such as tweets or blogs.
Results sorted by relevance
Solr returns documents in ranked order based on how relevant each document is to the user’s query.
11. 11
Features overview
• Pagination and sorting
• Faceting
• Autosuggest
• Spell-checking
• Highlighting
• Geospatial search
• More Like This
12. 12
Features overview
• Flexible query support
• Document clustering
• Import rich document formats (PDF, Office…)
• Import data from databases
• Multilingual support
DIH
Data Import Handler
16. 16
Lucene Document
• Documents are the unit of information for
indexing and search
• A Document is a set of fields
• Each field has a name and a value
• All field types must be defined, and all field
names (or dynamic field-naming patterns)
should be specified in Solr’s schema.xml
Seminars
Schema Configuration
• Per collection/index
• Xml file
• Define how the inverted Index will be built
• Fields/Field Types definition
Seminars
Schema Configuration
• Per collection/index
• Xml file
• Define how the inverted Index will be built
• Fields/Field Types definition
DOCUMENT
FIELD
17. 17
Lucene Document – Search problem
The Beginner’s Guide to Buying a House
How to Buy Your First House
Purchasing a Home
Becoming a New Home owner
Buying a New Home
Decorating Your Home
A Fun Guide to Cooking
How to Raise a Child
Buying a New Car
SELECT * FROM Books WHERE Name = 'buying a new home’;
0 results
SELECT * FROM Books
WHERE Name LIKE '%buying%’
AND Name LIKE '%a%’
AND Name LIKE '%home%’;
1 result
Buying a New Home
SELECT * FROM Books
WHERE Name LIKE '%buying%’
OR Name LIKE '%a%’
OR Name LIKE '%home%’;
8 results
A Fun Guide to Cooking, Decorating Your Home, How to Raise a Child, Buying a New Car,
Buying a New Home, The Beginner’s Guide to Buying a House, Purchasing a Home,
Becoming a New Home owner
Unimportant words
Synonyms
Linguistic variations
Ordering
18. 18
Lucene Document – Inverted Index
Doc # Content field Term Doc #
1 A Fun Guide to Cooking a 1,3,4,5,6,7,8
2 Decorating Your Home becoming 8
3 How to Raise a Child beginner’s 6
4 Buying a New Car buy 9
5 Buying a New Home buying 4,5,6
6 The Beginner’s Guide to Buying a House child 3
7 Purchasing a Home cooking 1
8 Becoming a New Home Owner decorating 2
9 How to Buy Your First House home 2,5,7,8
house 6,9
how 3,9
new 4,5,8
purchasing 7
your 2,9
INVERTED
INDEX
19. 19
Searching
TERM DOCS
buying 4,5,6,7,9
home 2,5,6,7,8,9
Unimportant word “a” is skipped
Synonyms purchasing ~ buying
Linguistic variations buy ~ buying
Synonyms house ~ home
(AND) = 5,6,7,9
Buying a New Home
The Beginner’s Guide to Buying a House
Purchasing a Home
How to Buy Your First House
20. 20
Searching operators
• Required terms
• Optional terms
• Negated terms
• Phrases
• Grouped expressions
• Fuzzy matching
• Wildcard
• Range
• Distance
• Proximity
buying AND home
buying OR home
buying NOT home
“buying a home”
(buying OR renting) AND home
offi* off*r off?r
yearsOld:[18-21]
administrator~
“chief officer”~1
21. 21
Relevancy till SOLR 4 (TF/IDF)
A relevancy score for each document is calculated and the search results are sorted from the highest score to the lowest.
Similarity
Term frequency
• A document is more relevant for a particular term if the term appears multiple times
Inverse document frequency
• Measure of how “rare” a search term is, is calculated by finding the document frequency (how many total documents
the search term appears within)
Boosting
• Multiplier in query time to adjust the weight of a field
• title:solr^2.5 description:solr
Normalization factors for fields, queries and coord
Ordering
22. 22
Relevancy from SOLR 6 (BM25)
BM25 improves upon TF/IDF
BM25 stands for “Best Match 25” (25th iteration on TF/IDF)
Includes different factors
• Frequency of a term in all Documents
• Term Frequency in a Document
• Document Length
BM25 limits influence of term frequency:
• less influence of commonwords
With TF/IDF: short fields (title,...) are automatically scored higher
BM25: Scales field length with average
• field length treatment does not automatically boost short fields
Ordering
23. 23
Precision and Recall
Precision is a measure of how “good” each of the results of a query is. A query that returns one single
correct document out of a million other correct documents is still considered perfectly precise.
Recall is a measure of how many of the correct documents are returned. A query that returns one
single correct document out of a million other correct documents is considered a very poor recall
scoring.
>> Precision and Recall balance will improve the quality of your search results.
20 correct documents
Search results containing 10 documents
(8 correct and 2 incorrect)
Precision = 80% (8 / 10)
Recall = 40% (8 / 20)
What is the precision and
recall for the
previous ”buying a home”
sample?
24. 24
Searching at Scale
Scaling SOLR
Solr is able to scale to handle
billions of documents and an
infinite number of queries
by adding servers.
Some limitations
• You can insert, delete, and update documents, but not single fields (easily)
• Solr is not optimized for processing quite long queries (thousands of terms) or returning quite
large result sets to users.
26. 26
Requirements
• Java Runtime Environment 1.8+
$ java -version
openjdk version "11.0.2" 2019-01-15
OpenJDK Runtime Environment 18.9 (build 11.0.2+9)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.2+9, mixed mode)
• Supported Operating Systems
• Linux
• MacOS
• Windows
https://lucene.apache.org/solr/downloads.html
27. 27
Directory layout
bin/
• solr | solr.cmd : start SOLR
• post : posting content to SOLR
• solr.in.sh | solr.in.cmd : configuration
contrib/
• add-ons plugins
dist/
• SOLR Jar files
docs/
• JavaDocs
example/
• CSV, XML and JSON
• DIH for databases
• Word and PDF files
licenses/
• 3rd party libraries
server/
• SOLR Admin UI
• Jetty Libraries
• Log files
• Sample configsets
28. 28
Starting SOLR
• Use the command line interface tool called bin/solr (Linux) or binsolr.cmd (Windows)
$ bin/solr start -p 8983
Waiting up to 180 seconds to see Solr running on port 8983 []
Started Solr server on port 8983 (pid=4521). Happy searching!
• Check if Solr is Running
$ bin/solr status
Found 1 Solr nodes:
Solr process 4521 running on port 8983
{
"solr_home":"/Users/aborroy/Downloads/solr-introduction-university/solr-8.4.1/server/solr",
"version":"8.4.1 832bf13dd9187095831caf69783179d41059d013 - ishan - 2020-01-10 13:40:28",
"startTime":"2020-03-08T08:13:49.969Z",
"uptime":"0 days, 0 hours, 17 minutes, 56 seconds",
"memory":"91.6 MB (%17.9) of 512 MB"}
30. 30
Creating a new Core
$ bin/solr create -c films
• -c indicates the collection name
Check default fields added by SOLR to the Schema >>>>>>>
Check JSON Data to be posted in example/films/films.json
{
"id": "/en/45_2006",
"directed_by": [
"Gary Lennon"
],
"initial_release_date": "2006-11-30",
"genre": [
"Black comedy",
"Thriller"
],
"name": ".45"
}
31. 31
Posting data
$ bin/post -c films example/films/films.json
Posting files to [base] url http://localhost:8983/solr/films/update...
POSTing file films.json (application/json) to [base]/json/docs
SimplePostTool: WARNING: Solr returned an error #400 (Bad Request) for url: http://localhost:8983/solr/films/update/json/docs
SimplePostTool: WARNING: Response: {
"responseHeader":{
"status":400,
"QTime":120},
"error":{
"metadata":[
"error-class","org.apache.solr.common.SolrException",
"root-error-class","java.lang.NumberFormatException"],
"msg":"ERROR: [doc=/en/quien_es_el_senor_lopez] Error adding field 'name'='¿Quién es el señor López?' msg=For input string:
"¿Quién es el señor López?"",
"code":400}}
SimplePostTool: WARNING: IOException while reading response: java.io.IOException: Server returned HTTP response code: 400 for
URL: http://localhost:8983/solr/films/update/json/docs
1 files indexed.
COMMITting Solr index changes to http://localhost:8983/solr/films/update...
Time spent: 0:00:00.323
32. 32
How many results were posted?
http://127.0.0.1:8983/solr/films/select?indent=on&q=*:*&wt=json
• q: query event
• fq: filter queries
• sort: asc or desc
• start, rows: offset and number of rows
• fl: list of fields to return
• wt: response in XML or JSON
33. 33
What was wrong?
Check carefully JSON Data to be posted in example/films/films.json
{
"id": "/en/quien_es_el_senor_lopez",
"directed_by": [
"Luis Mandoki"
],
"genre": [
"Documentary film"
],
"name": "u00bfQuiu00e9n es el seu00f1or Lu00f3pez?"
},
http://127.0.0.1:8983/solr/#/films/schema?field=name
34. 34
Auto-Generated SOLR Schema
http://127.0.0.1:8983/solr/#/films/files?file=managed-schema
A single document might
contain multiple values
for this field type
The value of the field
can be used in queries
to retrieve matching
documents (true by
default)
SOLR rejects any
attempts to add a
document which does
not have a value for this
field
The actual value of the
field can be retrieved by
queries
name can contain text!
35. 35
Re-Creating the Core
Deleting core “films”
$ bin/solr delete -c films
Deleting core 'films' using command:
http://localhost:8983/solr/admin/cores?action=UNLOAD&core=film
s&deleteIndex=true&deleteDataDir=true&deleteInstanceDir=true
Creating core “films”
$ bin/solr create -c films
Created new core 'films’
Creating the field “name” for the core “films”
http://127.0.0.1:8983/solr/#/films/schema
36. 36
Posting Data 2
$ bin/post -c films example/films/films.json
Posting files to [base] url http://localhost:8983/solr/films/update...
POSTing file films.json (application/json) to [base]/json/docs
1 files indexed.
COMMITting Solr index changes to http://localhost:8983/solr/films/update...
Time spent: 0:00:00.417
http://127.0.0.1:8983/solr/films/select?indent=on&q=*:*&wt=json
37. 37
Exploring SOLR Analyzers
• Solr analyzes both index content and query input before matching the results
• The live analysis can be observed by using “Analysis” option from Solr Admin UI
39. 39
Searching
q = genre:Fantasy directed_by:"Robert Zemeckis"
• This query is searching for both genre Fantasy and directed by Robert Zemeckis (OR is default operator)
40. 40
Filtering
q = genre:Fantasy
fq = initial_release_date:[NOW-12YEAR TO *]
• This query is searching for both genre Fantasy in the latest 12 years
41. 41
Sorting
q = *:*
sort = initial_release_date desc
• This query is ordering all the films by release date in descent order
44. 44
Faceting
Multiple fields for faceting
http://127.0.0.1:8983/solr/films/select?facet.field=directed_by_str&facet.field=genre&facet=on&indent=on&q=*:*&wt=js
on