Elasticsearch – mye mer enn søk! [JavaZone 2013]

Elasticsearch
Mye mer enn søk!
Alex Brasetvik
alex@found.no
@alexbrasetvik
Wednesday, September 11, 13

Hvem?
Co-founder av Found AS
7+ år søk, 2+ Elasticsearch
Håndterer hundrevis av Elasticsearch-clustre

Agenda
0. Elasticsearch
1. Bruksområder
2. Lingo
3. Datastrukturer
4. Tekstprosessering
5. Elasticsearch
6. NOSQL?

Elasticsearch
Open source
Real-time søk og analyse
Skjemafri
Basert på Lucene

$ curl localhost:9200/sample_index/sample_type -XPOST -d '
{
"user": {
"name": "DEVOPS_BORAT"
},
"followers": 42000,
"location": {
"lat": 56.78,
"lon": 12.34
},
"tags": [
"questionable",
"funny"
],
"message": "1+1=2 only in legacy system. In modern distributed database with eventual
consistent is 1+1=1.",
"retweets": 123
}
'
{"ok":true,"_index":"sample_index","_type":"sample_message",
"_id":"rjs9KSmPRnqhvs7QjgxJJw","_version":1}

$ curl localhost:9200/sample_index/sample_type/_search -XPOST -d '
{
"query":{
"match": {
"message": "consistent"
}
}
}
'

{
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 1,
"max_score" : 0.076713204,
"hits" : [ {
"_index" : "sample_index",
"_type" : "sample_message",
"_id" : "rjs9KSmPRnqhvs7QjgxJJw",
"_score" : 0.076713204,
"_source" : {
"user": {
"name": "DEVOPS_BORAT"
},
"message": "1+1=2 only in legacy system. In modern distributed database with
eventual consistent is 1+1=1.",
"retweets": 123,
...
}
} ]
}
}

{
"sample_index" : {
"sample_message" : {
"properties" : {
"followers" : {
"type" : "long"
},
"location" : {
"properties" : {
"lat" : {
"type" : "double"
},
"lon" : {
"type" : "double"
}
}
},
"message" : {
"type" : "string"
},
"retweets" : {
"type" : "long"
},
"tags" : {
"type" : "string"
},
"user" : {
"properties" : {
"name" : {
"type" : "string"
}
}
}
}
}Wednesday, September 11, 13

{"id"=>12296272736,
"text"=>
"An early look at Annotations:
http://groups.google.com/group/twitter-api-announce/browse_thread/thread/fa5da2608865453",
"created_at"=>"Fri Apr 16 17:55:46 +0000 2010",
"in_reply_to_user_id"=>nil,
"in_reply_to_screen_name"=>nil,
"in_reply_to_status_id"=>nil
"favorited"=>false,
"truncated"=>false,
"user"=>
{"id"=>6253282,
"screen_name"=>"twitterapi",
"name"=>"Twitter API",
"description"=>
"The Real Twitter API. I tweet about API changes, service issues and
happily answer questions about Twitter and our API. Don't get an answer? It's on my website.",
"url"=>"http://apiwiki.twitter.com",
"location"=>"San Francisco, CA",
"profile_background_color"=>"c1dfee",
"profile_background_image_url"=>
"http://a3.twimg.com/profile_background_images/59931895/twitterapi-background-new.png",
"profile_background_tile"=>false,
"profile_image_url"=>"http://a3.twimg.com/profile_images/689684365/api_normal.png",
"profile_link_color"=>"0000ff",
"profile_sidebar_border_color"=>"87bc44",
"profile_sidebar_fill_color"=>"e0ff92",
"profile_text_color"=>"000000",
"created_at"=>"Wed May 23 06:01:13 +0000 2007",
"contributors_enabled"=>true,
"favourites_count"=>1,
"statuses_count"=>1628,
"friends_count"=>13,
"time_zone"=>"Pacific Time (US & Canada)",
"utc_offset"=>-28800,
"lang"=>"en",
"protected"=>false,
"followers_count"=>100581,
"geo_enabled"=>true,
"notifications"=>false,
"following"=>true,
"verified"=>true},
"contributors"=>[3191321],
"geo"=>nil,
"coordinates"=>nil,
"place"=>
{"id"=>"2b6ff8c22edd9576",
"url"=>"http://api.twitter.com/1/geo/id/2b6ff8c22edd9576.json",
"name"=>"SoMa",
"full_name"=>"SoMa, San Francisco",
"place_type"=>"neighborhood",
"country_code"=>"US",
"country"=>"The United States of America",
"bounding_box"=>
{"coordinates"=>
[[[-122.42284884, 37.76893497],
[-122.3964, 37.76893497],
[-122.3964, 37.78752897],
[-122.42284884, 37.78752897]]],
"type"=>"Polygon"}},
"source"=>"web"}
The tweet's unique ID. These
IDs are roughly sorted &
developers should treat them
as opaque (http://bit.ly/dCkppc).
Text of the tweet.
Consecutive duplicate tweets
are rejected. 140 character
max (http://bit.ly/4ud3he).
Tweet's
creation
date.
DEPRECATED
The ID of an existing tweet that
this tweet is in reply to. Won't
be set unless the author of the
referenced tweet is mentioned.
The screen name &
user ID of replied to
tweet author.
Truncated to 140
characters. Only
possible from SMS.
Theauthorofthetweet.This
embeddedobjectcangetoutofsync.
Theauthor's
userID.
The author's
user name.
The author's
screen name.
The author's
biography.
The author's
URL.
The author's "location". This is a free-form text field, and
there are no guarantees on whether it can be geocoded.
Rendering information
for the author. Colors
are encoded in hex
values (RGB).
The creation date
for this account.
Whether this account has
contributors enabled
(http://bit.ly/50npuu). Number of
favorites this
user has.
Numberoftweets
thisuserhas.
Number of
users this user
is following.The timezone and offset
(in seconds) for this user.
The user's selected
language.
Whether this user is protected
or not. If the user is protected,
then this tweet is not visible
except to "friends".
Number of
followers for
this user.
Whetherthisuserhasgeo
enabled(http://bit.ly/4pFY77).
DEPRECATED
in this context
Whether this user
has a verified badge.
Thegeotagonthistweetin
GeoJSON(http://bit.ly/b8L1Cp).
The contributors' (if any) user
IDs (http://bit.ly/50npuu).
DEPRECATED
The place associated with this
Tweet (http://bit.ly/b8L1Cp).
The place ID
The URL to fetch a detailed
polygon for this placeThe printable names of this place
The type of this
place - can be a
"neighborhood"
or "city"
The country this place is in
The bounding
box for this
place
The application
that sent this
tweet
Map of a Twitter Status Object
Raffi Krikorian <raffi@twitter.com>
18 April 2010

user:
name: DEVOPS_BORAT
message: “1+1=2 only in legacy system. In modern
distributed database with eventual consistent is 1+1=1.”
location:
lon: 12.34
lat: 56.78
followers: 42000
retweets: 123
tags: [questionable, funny]

The quick brown fox had a day off
whitespace-tokenizer

Filter: boolean match
Query: match med score
Kan være satt sammen av andre queries
Filter / Query

“Søk”
Hele informasjonsbehovet
Query, ﬁltre, fasetter, paginering, ...

Invertert indeks
"If you don't ﬁnd it in the index, look very carefully
through the entire catalog."
–Sears, Roebuck, and Co., Consumers' Guide 1897

AbstractEnterpriseSingletonProxyFactoryBean

xkcd.com/292

AbstractSingletonProxyFactoryBean
camelCase-tokenizer
lowercase

Preﬁks-problemer!

Prefiks-problemer
*suffix xiffus*
(60.6384, 6.5017) u4u8gyykk
123 {1-hundreds, 12-tens, 123} (forenkla)

Elasticsearch
Distribuert
Cluster av noder
Selv-koordinerende

Mapping

Så langt
Inverterte indekser
Tekstprosessering
Indeks-termer
Mappings
Indeks-maler

xkcd.com/208

?q={!boost b=div(popularity,price) v=$qq}
&qq={!dismax qf=desc^2,review}cheap
&bq={!lucene df=keywords}lucene solr java
&fq={!geofilt sfield=location pt=10.312,-20.556 d=3.5}
&fq={!term f=$ff v=$vv}&ff=keywords&vv=solr
&sort=query(keywords:lame) asc, score desc

Filtre
Caches som bitmaps
Kompakte
Veldig raske

term:
className: "InternalFrameInternalFrameTitlePaneInternalFrameTitlePaneMaximizeButtonWindowNotFocusedState"

Filtre
Bruk ﬁltre når du kan …
… og queries når du trenger rangering.

Fasetter
Oppsummerer hele resultat-mengden
Filtre + fasetter grunnlag for analyse-bruk

Fasetterings-muligheter
Termer
Histogrammer
Tids-histogrammer
Geo-distanse
Statistisk fordeling
Filtre/Spørringer

Fasetter
Ressurskrevende
CPU + minne
Viktig å ha nok minne

Filter-cacher
Felt-cacher: fasetter, m.m.
Page-cache
Cacher
There are two hard things in computer science:
cache invalidation, naming things, and oﬀ-by-one errors.

Cacher
Now you are thinking with...
Per segment
Nye segmenter invaliderer
ikke gamle
Viktig for (near) real time

PostgreSQL
Veriﬁserer ressursbruk
Trygg >> rask
Bruker disk om den må

Elasticsearch stoler på deg
Bygd for fart
What could possibly go wrong?

OutOfMemoryError
Woah there
I ate all the memories
Your cluster may or may not work any more

NOSQL?
Kjapp, ikke robust
Dokumentdatabase
Skjema-ﬂeksibel
Ingen transaksjoner
Lett å skalere/distribuere
Naïv leader-election
Ingen auth/authz

?
Slides og relevante linker på
found.no/jz13
(Prøv hosted Elasticsearch i 6 mnd. gratis)
Solr-meetup i community-rommet
i morgen!

Image credits
Nails – Adam Rosenberg
Map of Westeros
Elephant, Roy Costello
Wingsuit, Richard Schneider

Elasticsearch – mye mer enn søk! [JavaZone 2013]

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (6)

Similaire à Elasticsearch – mye mer enn søk! [JavaZone 2013]

Similaire à Elasticsearch – mye mer enn søk! [JavaZone 2013] (20)

Dernier

Dernier (20)

Elasticsearch – mye mer enn søk! [JavaZone 2013]