Factors to Consider When Choosing Accounts Payable Services Providers.pptx
Elastic search and Symfony3 - A practical approach
1. Nadhiir Rosun / @nads_rosun
E l a s t i c s e a r c h a n d S y m f o n y
31/03/2017
A p r a c t i c a l a p p r o a c h
2. 2
1. Introduction
2. ES versus RDBMS
3. Talking to ES
4. Architecture
5. Querying ES
6. Case study
7. Conclusions
3. 3
Introduction
Highly scalable open-source full-text search and
analytics engine
Allows you to store, search, and analyze big volumes of
data quickly and in near real time.
Build on top of Apache Lucene
¬ Popular/powerfull full-text search engine library
Developed in Java
4. 4
Introduction / Projects using ES
Wikipedia
¬ Full-text search
¬ Highlighted search snippets
¬ Search-as-you-type
¬ Did-you-mean suggestions
The Guardian
¬ Real–time analytics of visitor logs
Stack Overflow
¬ Full-text search with geolocation queries
¬ Find related questions and answers
GitHub
¬ Query 130 billion lines of code
5. 5
Introduction / Use case 1
An online store
Customers can search for products
Big volume of products / large amount of transactions
Product catalog to be full-text search, autocomplete
suggestions, did-you-mean suggestions
6. 6
Introduction / Use case 2
Analytics/business intelligence needs
Collect log and transaction data
Analyze the data to look for trends, statistics &
summarizations
Build custom dashboard to visualize aspects of the data
7. 7
1. Introduction
2. ES versus RDBMS
3. Talking to ES
4. Architecture
5. Querying ES
6. Case study
7. Conclusions
8. 8
ES versus RDBMS
Document oriented (NoSQL)
Schema free
Easily distributed & scalable
All fields indexed
Comprehensive and powerfull RESTful API
Advanced search
¬ Full-text search
¬ Aggregations
9. 9
ES versus RDBMS / Teminology
Relational database ES
Database Index
Table Type
Row Document
SELECT GET
INSERT PUT/POST
UPDATE PUT/POST
DELETE DELETE
10. 10
1. Introduction
2. ES versus RDBMS
3. Talking to ES
4. Architecture
5. Querying ES
6. Case study
7. Conclusions
11. 11
Talking to ES / Installation
https://www.elastic.co/downloads/elasticsearch
Check if ES is running
¬ http://localhost:9200/
{
"name" : "Cp8oag6",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "AT69_T_DTp-1qgIJlatQqA",
"version" : {
"number" : "5.3.0",
"build_hash" : "f27399d",
"build_date" : "2016-03-30T09:51:41.449Z",
"build_snapshot" : false,
"lucene_version" : "6.4.1"
},
"tagline" : "You Know, for Search"
}
12. 12
Talking to ES
RESTful API with JSON over HTTP
¬ Curl
¬ Guzzle
¬ Kibana
PHP Client
¬ Official php client
- https://github.com/elastic/elasticsearch-php
¬ Elastica
- https://github.com/ruflin/Elastica
Framework specific
¬ Symfony : FOSElasticaBundle
- https://github.com/FriendsOfSymfony/FOSElasticaBundle
¬ Drupal 8 : Elasticsearch Connector
- https://www.drupal.org/project/elasticsearch_connector
14. 14
Talking to ES / Index a document
PUT /[index]/[type]/[doc_id]
Request
PUT /providers/accomodation/1
{
"name" : "Beaurivage Appartments",
"about" : "Room or studio accommodation in the centre of the village",
"city" : "Grand Baie",
"star" : 3,
"facility" : ["Internet", "Pool", “Television”, "Parking"]
}
15. 15
Talking to ES / Index a document
Response
{
"_index": "providers",
"_type": "accomodation",
"_id": "1",
"_version": 1,
"result": "created",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"created": true
}
16. 16
Talking to ES / Retrieve a document
GET /[index]/[type]/[doc_id]
Request
GET /providers/accomodation/1
17. 17
Talking to ES / Retrieve a document
Response
{
"_index": "providers",
"_type": "accomodation",
"_id": "1",
"_version": 1,
"found": true,
"_source": {
"name": "Beaurivage Appartments",
"about": "Room or studio accommodation in the centre of the village",
"city": "Grand Baie",
"star": 3,
"facility": ["Internet", "Pool", "Television", "Parking"]
}
}
18. 18
Talking to ES / Update a document
PUT /[index]/[type]/[doc_id]
Request
PUT /providers/accomodation/1
{
"name" : "Beaurivage Appartments",
"about" : "Room or studio accommodation in the centre of the village",
"city" : "Grand Baie",
"star" : 4,
"facility" : ["Internet", "Pool", “Television”, "Parking"]
}
22. 22
1. Introduction
2. ES versus RDBMS
3. Talking to ES
4. Architecture
5. Querying ES
6. Case study
7. Conclusions
23. 23
Architecture / Cluster
A collection of one or more nodes (servers)
¬ Sharing the same cluster name
¬ Working together to share data/workload
Perfectly fine to have a cluster with only a single node
24. 24
Architecture / Node
A single server that is part of your cluster
¬ A single instance of ES
Master node
¬ One node is elected as MASTER
¬ Manage cluster-wide changes
25. 25
Architecture / Shard
Think of shards as containers for data
¬ Allows to horizontally split/scale your content volume
¬ Allows to distribute and parallelize operations(Performance)
Primary shard :
¬ Document is first indexed on a single primary shard
Replica shard :
¬ Is just a copy of a primary shard
¬ Protect against hardware failure / share server load
26. 26
Architecture / Index
A collection of documents with similar characteristics
¬ An index for :
- Customer data
- Product catalog
- Order data
Create an index
PUT /providers
{
"settings" : {
"number_of_shards" : 3,
"number_of_replicas" : 1
}
}
27. 27
Architecture / Type
A logical category/partition of your index
Within an index, you can define one or more types
Has a list of fields mapping
A type is defined for documents that have a set of
common fields
¬ Product index with perishable type and non-perishable type √
¬ Blog index with article type and log type ×
28. 28
Architecture / Document
A basic unit of information that can be indexed
A JSON object stored in a single shard
Uniquely identified by its index, type and id
{
"_index": "providers",
"_type": "accomodation",
"_id": "1",
"_version": 1,
"_source": {
"name": "Beaurivage Appartments",
"about": "Room or studio accommodation in the centre of the village",
"city": "Grand Baie",
"star": 3,
"facility": ["Internet", "Pool", "Television", "Parking"]
}
}
32. 32
Querying ES / Query DSL
Match all
¬ Simply matches all documents
- SELECT * FROM providers.accommodation
GET /providers/accommodation/_search
{
"query": {
"match_all": {}
}
}
33. 33
Querying ES / Query DSL
Match
¬ Query a field
- SELECT * FROM providers.accommodation p WHERE p.name =
‘Beaurivage’
GET /providers/accommodation/_search
{
"query": {
"match": {
“name” : “Beaurivage”
}
}
}
34. 34
Querying ES / Query DSL
Range
¬ Find numbers or dates that fall into a specified range
- SELECT * FROM providers.accommodation p WHERE p.stars > 3
GET /providers/accommodation/_search
{
"query": {
"range" : {
“stars" : {
"gte" : 3,
}
}
}
}
35. 35
Querying ES / Query DSL
Term
¬ Finds documents that contain the exact term
- SELECT * FROM providers.accommodation p WHERE p.facilities =
‘Internet’
GET /providers/accommodation/_search
{
"query": {
"term" : { “facility" : “Internet" }
}
}
36. 36
Querying ES / Query DSL
Bool
¬ Combine query clauses
- Must :
› The clause must appear in matching documents
› Contribute to score
- Filter :
› The clause must appear in matching documents
› Scoring is ignored
GET /providers/accommodation/_search
{
"query": {
"bool": {
"must": [ … ],
"filter": [ … ]
}
}
}
37. 37
Querying ES / Query DSL
Pagination
¬ From
- Similar to offset in sql
¬ Size
- Similar to limit in sql
¬ SELECT * FROM providers.accommodation p LIMIT 10 OFFSET 15
GET /providers/accommodation/_search
{
"query": {
“match_all” : {}
},
“from” : 15,
“size” : 10
}
38. 38
Querying ES / Full-text search
Structured search
¬ Something either belongs in the set or it does not
¬ Simply includes or excludes documents
¬ Does not worry about document relevance or scoring
¬ Very fast and cachable
Full-text search
¬ Finds how well a documents match the search keywords
¬ Analysis/Inverted index
¬ Each document gets a score
¬ Return documents sorted by relevance
39. 39
Querying ES / Full-text search
Case study : 2 documents
{
"name" : "Beaurivage Appartments",
"about" : "Room or studio accommodation in the centre of the village",
"city" : "Grand Baie",
"star" : 3,
"facility" : ["Internet", "Pool", “Television”, "Parking"]
},
{
"name" : "Hotel Allamanda",
"about" : "Hotel room in front of the best lagoon of the island.",
"city" : "Tamarin",
"star" : 5,
"facility" : ["Internet", “Beach", “Telephone”, "Parking"]
},
40. 40
Querying ES / Full-text search
Case study : Full-text search query
GET /providers/accomodation/_search
{
"query": {
"match": {
"about": "Room accomodation"
}
}
}
42. 42
Querying ES / Query DSL
Inverted Index
¬ Designed to allow very fast full-text searches
¬ Consists of :
- a list of all the unique words that appear in any document
- For each word, a list of the documents in which it appears.
1.The quick brown fox jumped over the lazy dog
1.Quick brown foxes leap over lazy dogs in summer
43. 43
Querying ES / Query DSL
Split into separate tokens
List of unique tokens
List in which document each
term appears
Notice:
- Quick and quick appear as
separate terms
- fox and foxes share the same
root word
44. 44
Querying ES / Query DSL
Analyzers
¬ Character Filters
- Tidy up string
- Strip out HTML and ‘&’
¬ Tokenizer
- Split into tokens/terms
- Every whitespace/punctuation
¬ Token Filters
- Can change the tokens
› Lowercase
› Add synonyms
› Remove stopwords (a, and, the…)
Token Filters
Tokenizer
Character
filters
45. 45
1. Introduction
2. ES versus RDBMS
3. Talking to ES
4. Architecture
5. Querying ES
6. Case study
7. Conclusions
46. 46
Case study / SF3 resto search engine
Creating a simple restaurant search engine
¬ Full-text search
¬ Facets
¬ Pagination
Techno :
¬ Symfony 3
- FOSElasticaBundle
- PagerBundle
¬ Elasticsearch
48. 48
Case study / SF3 resto search engine
MySQLES
Restaurant
- Name
- About
- City
- facilities
Restaurant
- Name
- About
- City
- photo
- facilities
App 1. New/Update/Delete
Serializer
2. Doctrine object
3. JSON object
Indexing
49. 49
Case study / SF3 resto search engine
MySQLES
Restaurant
- Name
- About
- City
- facilities
Restaurant
- Name
- About
- City
- photo
- facilities
App
Finder
5.Doctrine object
2. JSON object
1. Search
3,4. Fetch from db
Searching
50. 50
Case study / SF3 resto search engine
Installing the bundles with composer
your_project/composer.json
"require": {
…
“friendsofsymfony/elastica-bundle": "^3.0",
"ruflin/elastica": "3.1.1 as 2.99.0",
"white-october/pagerfanta-bundle": "dev-master",
}
51. 51
Case study / SF3 resto search engine
Register the bundles
your_project/app/AppKernel.php
$bundles = [
…
new FOSElasticaBundleFOSElasticaBundle(),
new WhiteOctoberPagerfantaBundleWhiteOctoberPagerfantaBundle(),
];
52. 52
Case study / SF3 resto search engine
Configuration
¬ Add ES server address in ‘parameters.yml’
your_project/app/config/parameters.yml
parameters:
…
elastica_host: localhost
elastica_port: 9200
¬ Create/import new configuration file ‘fos_elastica.yml’
your_project/app/config/config.yml
imports:
…
- { resource: fos_elastica.yml }
54. 54
Case study / SF3 resto search engine
Persistence
¬ Defines how FOSElasticaBundle will index your docs depending
on your symfony entities
Driver
¬ Driver to use (ORM)
Model
¬ Allow to define an ES doc from a symfony entity
Finder
¬ Search interface (deserialize JSON objects into doctrine objects)
Provider
¬ Indexing interface (serialize doctrine objects to JSON objects)
Listener
¬ List of listeners for which the indexing is called
- Default : insert, update & delete
56. 56
Case study / SF3 resto search engine
Create Entity ‘Restaurant’
your_project/src/AppBundle/Entity/Restaurant.php
class Restaurant
{
protected $id;
protected $name;
protected $about;
protected $city;
protected $photo;
protected $facilities;
//getters and setters
}
57. 57
Case study / SF3 resto search engine
Index the doctrine objects to ES
¬ Uses the provider defined in fos_elastica.yml
¬ Serialize Doctrine objects to JSON using the Symfony Serializer
component
- Uses the predefined mapping in this process
From the command line
php bin/console fos:elastica:populate
58. 58
Case study / SF3 resto search engine
Create the form model
¬ Mapped to the FormType
¬ Contain two properties
- title
- facilities
your_project/src/AppBundle/Model/SearchModel.php
class SearchModel
{
protected $title; //string
protected $facilities; //array
//getters and setters
}
59. 59
Case study / SF3 resto search engine
Create the form type
¬ Contain two fields : title, facilities
your_project/src/AppBundle/Form/Type/SearchType.php
$builder
->add('title', null, array(
'required' => false,
))
->add('facilities', ChoiceType::class, array(
'choices' => array(
'Internet' => 'Internet',
'Parking' => 'Parking',
'Toilet' => 'Toilet',
'Beach' => 'Beach',
'Card Payment' => 'Card Payment',
),
'expanded' => true,
'multiple' => true,
))
->add('search', SubmitType::class);
60. 60
Case study / SF3 resto search engine
Bundle everything in your search action
your_project/src/AppBundle/Controller/DefaultController.php
$perPage = 8;
// Create form
$searchModel = new SearchModel();
$searchForm = $this->createForm(SearchType::class, $searchModel);
// Get form data
$searchForm->handleRequest($request);
$searchModel = $searchForm->getData();
// Perform query in ES
$elasticaManager = $this->container->get('fos_elastica.manager');
$results = $elasticaManager->getRepository('AppBundle:Restaurant')->search($searchModel);
// Pagination
$adapter = new ArrayAdapter($results);
$pager = new Pagerfanta($adapter);
$pager->setMaxPerPage($perPage);
$pager->setCurrentPage($page);
// return results, pager and form to your template
61. 61
Case study / SF3 resto search engine
ES search repository
your_project/src/AppBundle/Repository/SearchRepository.php
$boolQuery = new ElasticaQueryBoolQuery();
// Search Field name
$queryName = new ElasticaQueryMatch();
$queryName->setFieldQuery('name', $searchModel->getTitle());
$boolQuery->addShould($queryName);
// Search Field about
$queryAbout = new ElasticaQueryMatch();
$queryAbout->setFieldQuery('about', $searchModel->getTitle());
$queryAbout->setFieldParam('about', 'analyzer', 'english');
$boolQuery->addShould($queryAbout);
…
// Terms
if (count($searchModel->getFacilities()) > 0) {
$tagsQuery = new ElasticaQueryTerms();
$tagsQuery->setTerms('facilities', $searchModel->getFacilities());
$boolQuery->addMust($tagsQuery);
}
$query = ElasticaQuery::create($boolQuery);
62. 62
1. Introduction
2. ES versus RDBMS
3. Talking to ES
4. Architecture
5. Querying ES
6. Case study
7. Conclusions