introduction to elasticsearch
TRANSCRIPT
ElasticsearchTony MessiasMaceioDevMeetup #5
who am I?
➔ Tony Messias ~ @tony0x01
➔ Building web stuff since ~2010
search
what is “search”?
“A search is the organized pursuit of information (...) that you want to find, but you have no idea where it is.”
(NASA).
how do we add search to our apps?
➔ SQL queries (not just LIKE queries);➔ Elasticsearch;➔ Lucene;➔ Solr;➔ …
how do we add search to our apps?
➔ SQL queries (not just LIKE queries);➔ Elasticsearch;➔ Lucene;➔ Solr;➔ …
who uses it?
what is Elasticsearch?
➔ database server;➔ document based;➔ based on Apache Lucene;➔ schema-free*;
why Elasticsearch?
➔ speaks HTTP/JSON;➔ easy to setup;➔ data replication;➔ easily scalable;
what can I use it for?
but it can do more than that…
some analogy *Relational ES
database index
table type
row document
column field
schema mapping
index everything
SQL QueryDSL
REST(ish)
$ curl -XPOST http://es-server.com/index/type/id -d '{}'
REST(ish)
$ curl -XPOST http://es-server.com/index/type/id -d '{}'
REST(ish)
$ curl -XPOST http://es-server.com/index/type/id -d '{}'
REST(ish)
$ curl -XPOST http://es-server.com/index/type/id -d '{}'
REST(ish)
$ curl -XPOST http://es-server.com/index/type/id -d '{}'
enough with the talking… let’s do some searches!
but first, let’s talk about CRUD
indexing
{
"email": "[email protected]",
"first_name": "John",
"last_name": "Smith",
"info": {
"bio": "Eco-warrior and defender of the weak",
"age": 25,
"interests": [ "dolphins", "whales" ]
},
"join_date": "2014/05/01"
}
$ curl -XPOST localhost:9200/my-app/users -d ‘{
"email": "[email protected]",
"first_name": "John",
"last_name": "Smith",
"info": {
"bio": "Eco-warrior and defender of the weak",
"age": 25,
"interests": [ "dolphins", "whales" ]
},
"join_date": "2014/05/01"
}’
{
"_index": "my-app",
"_type": "users",
"_id": "AUxOBeJbue-jeR3jNcHj",
"_version": 1,
"created": true
}
$ curl -XPUT localhost:9200/my-app/users/AUxOBeJbue-jeR3jNcHj -d
‘{
"email": "[email protected]",
"first_name": "John"
}’
{
"_index": "my-app",
"_type": "users",
"_id": "AUxOBeJbue-jeR3jNcHj",
"_version": 2,
"created": false
}
$ curl -XGET localhost:9200/my-app/users/AUxOBeJbue-jeR3jNcHj
{
"_index": "my-app",
"_type": "users",
"_id": "AUxOJe-4ue-jeR3jNcHo",
"_version": 2,
"found": true,
"_source": {
"email": "[email protected]",
"first_name": "John"
}
}
$ curl -XDELETE localhost:9200/my-app/users/AUxOBeJbue-jeR3jNcHj
{
"found": true,
"_index": "my-app",
"_type": "users",
"_id": "AUxOBeJbue-jeR3jNcHj",
"_version": 3
}
searching
$ curl -XPOST localhost:9200/my-app/users/_search -d ‘{
"query": {
"match": {
"first_name": "John"
}
}
}’
{
...,
"hits": {
"total": 4,
"max_score": 0.30685282,
"hits": [
{
"_index": "my-app",
"_type": "users",
"_id": "AUxOJWlCue-jeR3jNcHn",
"_score": 0.30685282,
"_source": {
"email": "[email protected]",
"first_name": "John",
"last_name": "Doe",
"info": {
"bio": "Like rock music",
"age": 35,
"interests": [
"dolphins",
"whales"
]
},
"join_date": "2014/05/01"
}
},...
$ curl -XPOST localhost:9200/my-app/users/_search -d ‘{
"query": {
"filtered": {
"query": {
"match": {
"first_name": "John"
}
},
"filter": {
"range": {
"info.age": {
"gt": 34
}
}
}
}
}
}’
{
"took": 6,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 0.30685282,
"hits": [
{
"_index": "my-app",
"_type": "users",
"_id": "AUxOJWlCue-jeR3jNcHn",
"_score": 0.30685282,
"_source": {
"email": "[email protected]",
"first_name": "John",
"last_name": "Doe",
"info": {
"bio": "Like rock music",
"age": 35,
"interests": ["dolphins","whales"]
},
"join_date": "2014/05/01"
}
}
]
}
}
queries vs. filters
queries vs. filters
full-text search
$ curl -XPOST localhost:9200/my-app/users/_search -d ‘{
"query": {
"match": {
"info.bio": "rock climbing"
}
}
}’
{
"took": 9,
"timed_out": false,
"_shards": {...},
"hits": {
"total": 2,
"max_score": 0.2169777,
"hits": [
{
"_index": "my-app",
"_type": "users",
"_id": "AUxOJe-4ue-jeR3jNcHo",
"_score": 0.2169777,
"_source": {
"email": "[email protected]",
"first_name": "John",
"last_name": "Kennedy",
"info": {
"bio": "I love rock climbing!",
"age": 29,
"interests": ["dolphins","whales"]
},
"join_date": "2014/05/01"
}
},
{
"_index": "my-app",
"_type": "users",
"_id": "AUxOJWlCue-jeR3jNcHn",
"_score": 0.02250402,
"_source": {
"email": "[email protected]",
"first_name": "John",
"last_name": "Doe",
"info": {
"bio": "Like rock music",
"age": 35,
"interests": ["dolphins","whales"]
},
"join_date": "2014/05/01"
}
}
]
}
}
$ curl -XPOST localhost:9200/my-app/users/_search -d ‘{
"query": {
"match_phrase": {
"info.bio": "rock climbing"
}
}
}’
{…,
"hits": {
"total": 1,
"max_score": 0.30685282,
"hits": [
{
"_index": "my-app",
"_type": "users",
"_id": "AUxOJe-4ue-jeR3jNcHo",
"_score": 0.30685282,
"_source": {
"email": "[email protected]",
"first_name": "John",
"last_name": "Kennedy",
"info": {
"bio": "I love rock climbing!",
"age": 29,
"interests": ["dolphins","whales"]
},
"join_date": "2014/05/01"
}
}
]
}
}
$ curl -XPOST localhost:9200/my-app/users/_search -d ‘{
"query": {
"match_phrase": {
"info.bio": "rock climbing"
}
},
"highlight": {
"fields": {
"info.bio": {}
}
}
}’
...
{
"_index": "my-app",
"_type": "users",
"_id": "AUxOJe-4ue-jeR3jNcHo",
"_score": 0.30685282,
"_source": {
"email": "[email protected]",
"first_name": "John",
"last_name": "Kennedy",
"info": {
"bio": "I love rock climbing!",
"age": 29,
"interests": ["dolphins","whales"]
},
"join_date": "2014/05/01"
},
"highlight": {
"info.bio": ["I love <em>rock</em> <em>climbing</em>!"]
}
}
]
}
}
is this it?
analytics
aggregations
buckets metrics
$ curl -XPOST localhost:9200/my-app/users/_search?search_type=count -d'{
"aggs": {
"all_interests": {
"terms": {
"field": "info.interests"
}
}
}
}'
{ ...,
"hits": {"total": 4,"max_score": 0,"hits": []},
"aggregations": {
"all_interests": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": "whales",
"doc_count": 4
},
{
"key": "dolphins",
"doc_count": 3
}
]
}
}}
$ curl -XPOST localhost:9200/my-app/users/_search?search_type=count -d'{
"aggs": {
"all_interests": {
"terms": {
"field": "info.interests"
}
}
}
}'
$ curl -XPOST localhost:9200/my-app/users/_search?search_type=count -d'{
"aggs": {
"all_interests": {
"terms": {
"field": "info.interests"
},
"aggs": {
"avg_age": {
"avg": {
"field": "info.age"
}
}
}
}
}
}'
{
…,
"aggregations": {
"all_interests": {
…,
"buckets": [
{
"key": "whales",
"doc_count": 4,
"avg_age": { "value": 31.5 }
},
{
"key": "dolphins",
"doc_count": 3,
"avg_age": { "value": 32.333333333333336 }
}
]
}}}
$ curl -XPOST localhost:9200/my-app/users/_search?search_type=count -d'{
"query": {
"match_phrase": {
"info.bio": "rock climbing"
}
},
"aggs": {
"all_interests": {
"terms": { "field": "info.interests" },
"aggs": {
"avg_age": {
"avg": { "field": "info.age" }
}
}
}
}}'
{
…,
"aggregations": {
"all_interests": {
…,
"buckets": [
{
"key": "whales",
"doc_count": 1,
"avg_age": {
"value": 29
}
}
]
}}}
questions?
Resources
➔ http://www.elastic.
co/guide/en/elasticsearch/guide/current/index.
html
➔ https://leanpub.com/elasticsearch-quick-start
➔ https://www.elastic.co/use-cases
➔ https://speakerdeck.com/asm89/elasticsearch
Resources
➔ https://speakerdeck.
com/johnbeynon/elasticsearch-101
➔ http://blog.madewithlove.be/post/integrating-
elasticsearch-with-your-laravel-app/
➔ http://blog.madewithlove.be/post/elasticsearch-
aggregations/