anwendungsfälle für elasticsearch jax 2015
TRANSCRIPT
![Page 1: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/1.jpg)
Anwendungsfälle für
Florian Hopf - @fhopf
![Page 2: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/2.jpg)
Agenda
![Page 3: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/3.jpg)
Vorbereitung
![Page 4: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/4.jpg)
Installation
# download archivewget https://download.elasticsearch.org/
elasticsearch/elasticsearch/elasticsearch-1.4.4.zip
# zip is for windows and linuxunzip elasticsearch-1.4.4.zip
# on windows: elasticsearch.batelasticsearch-1.4.4/bin/elasticsearch
![Page 5: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/5.jpg)
Zugriff
curl -XGET http://localhost:9200{ "status" : 200, "name" : "Anomalito", "cluster_name" : "elasticsearch", "version" : { "number" : "1.4.4", "build_hash" :
"c88f77ffc81301dfa9dfd81ca2232f09588bd512", "build_timestamp" : "2015-02-19T13:05:36Z", "build_snapshot" : false, "lucene_version" : "4.10.3" }, "tagline" : "You Know, for Search"}
![Page 6: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/6.jpg)
Document Store
![Page 7: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/7.jpg)
Document
{ "title" : "Anwendungsfälle für Elasticsearch", "speaker" : "Florian Hopf", "date" : "2015-04-22T10:30:00.000Z", "tags" : ["Java", "Lucene"], "conference" : { "name" : "JAX", "city" : "Mainz" } }
![Page 8: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/8.jpg)
Speichern
curl -XPOST http://localhost:9200/conferences/talk/ --data-binary @talk-example.json
{"_index":"conferences","_type":"talk","_id":"GqjY7l8sTxa3jLaFx67_aw","_version":1,"created":true
}
![Page 9: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/9.jpg)
Speichern
curl -XPOST http://localhost:9200/conferences/talk/ --data-binary @talk-example.json
{"_index":"conferences","_type":"talk","_id":"GqjY7l8sTxa3jLaFx67_aw","_version":1,"created":true
}
Index
![Page 10: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/10.jpg)
Speichern
curl -XPOST http://localhost:9200/conferences/talk/ --data-binary @talk-example.json
{"_index":"conferences","_type":"talk","_id":"GqjY7l8sTxa3jLaFx67_aw","_version":1,"created":true
}
Type
![Page 11: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/11.jpg)
Lesen
curl -XGET http://localhost:9200/conferences/talk/GqjY7l8sTxa3jLaFx67_aw?pretty=true
{ "_index" : "conferences",
[...] "_source":{ "title" : "Anwendungsfälle für Elasticsearch", "speaker" : "Florian Hopf", "date" : "2015-04-22T10:30:00.000Z", "tags" : ["Java", "Lucene"], "conference" : { "name" : "JAX", "city" : "Mainz" } }}
![Page 12: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/12.jpg)
Sharding
● Aufteilen eines Index in mehrere Teile● Default: 5 Shards pro Elasticsearch-Index
● Mehrere ES-Instanzen bilden einen Cluster● Automatische Verteilung auf die Knoten
![Page 13: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/13.jpg)
Sharding
![Page 14: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/14.jpg)
Sharding
![Page 15: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/15.jpg)
Sharding
![Page 16: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/16.jpg)
Recap
● Einfache Speicherung von JSON-Dokumenten● Index und Type● Sharding für große Datenmengen● Verteilung ist First-Class-Citizen
![Page 17: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/17.jpg)
Nutzer
● Atlassian HipChat● http://highscalability.com/blog/2014/1/6/how-hipchat-stores-and-indexes-billions-of-messages-using-el.html
● OpenCrowd● https://www.elastic.co/use-cases/open-crowd
![Page 18: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/18.jpg)
Volltextsuche
![Page 19: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/19.jpg)
![Page 20: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/20.jpg)
![Page 21: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/21.jpg)
Suche per Parametercurl -XGET "http://localhost:9200/conferences/talk/_search
?q=elasticsearch&pretty=true"
{"took" : 73,
[…]
"hits" : {
[…]
"hits" : [ {
[…]
"_score" : 0.076713204,
"_source":{
"title" : "Anwendungsfälle für Elasticsearch",
"tags" : ["Java", "Lucene"],
[…]
} } ]
}
}
![Page 22: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/22.jpg)
Query-DSLcurl -XGET "http://localhost:9200/conference/talk/_search" -d'
{
"query": {
"filtered": {
"query": {
"match": {
"title": "elasticsearch"
}
},
"filter": {
"term": {
"tags": "java"
}
}
}
}
}'
![Page 23: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/23.jpg)
Query-DSL
client.prepareSearch("conferences")
.setQuery(filteredQuery(
matchQuery("title", "elasticsearch"),
termFilter("tag", "java")))
.execute()
.actionGet();
![Page 24: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/24.jpg)
Sprache
curl -XGET "http://localhost:9200/conferences/talk/_search?q=title:anwendungsfall&pretty=true"
{
[...]
},
"hits" : {
"total" : 0,
"max_score" : null,
"hits" : [ ]
}
}
![Page 25: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/25.jpg)
Mapping
curl -XDELETE "http://localhost:9200/conferences/"
curl -XPUT "http://localhost:9200/conferences/“
curl -XPUT "http://localhost:9200/conferences/talk/_mapping" -d'
{
"properties": {
"title": {
"type": "string",
"analyzer": "german"
}
}
}'
![Page 26: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/26.jpg)
Sprache
curl -XGET
"http://localhost:9200/conferences/talk/_search?q=title:anwendungsfall&pretty=true"
{
[...]
},
"hits" : {
"total" : 1,
[...]
}
}
![Page 27: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/27.jpg)
Was noch?
● Faceting/Aggregations● Suggestions● Highlighting● Sortierung● Pagination● ...
![Page 28: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/28.jpg)
Recap
● Ausdrucksstarke Suchen über Query DSL● Analyzing und Mapping● Lucene bietet viele Search-Features
![Page 29: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/29.jpg)
Users
● GitHub● http://exploringelasticsearch.com/github_interview.html● http://www.elasticsearch.org/case-study/github/
● StackOverflow● http://meta.stackexchange.com/questions/160100/a-new-
search-engine-for-stack-exchange● http://nickcraver.com/blog/2013/11/22/what-it-takes-to-run-
stack-overflow/● XING
● http://www.elasticsearch.org/case-study/xing/
![Page 30: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/30.jpg)
Flexibler Cache
![Page 31: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/31.jpg)
Setup Suche
Anwendung
DB
![Page 32: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/32.jpg)
Nur Suche?
![Page 33: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/33.jpg)
Queries
Anwendung
DB
![Page 34: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/34.jpg)
Ergebnislisting
curl -XPOST "http://localhost:9200/conferences/_search " -d'
{
"filter": {
"term": {
"tags": "java"
}
}
}'
![Page 35: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/35.jpg)
Nutzer
● Microsoft MSN● https://www.elastic.co/elasticon/2015/sf/powerin
g-real-time-search-at-microsoft
![Page 36: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/36.jpg)
Geo-SucheGeo-Suche
![Page 37: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/37.jpg)
Anwendungen
● Zeige nächste Filiale● Filialsuche● Sortierung Kleinanzeigen● Filterung auf Nähe● Social Media-Analysen
![Page 38: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/38.jpg)
Document{
"title" : "Anwendungsfälle für Elasticsearch",
"speaker" : "Florian Hopf",
"date" : "2014-07-17T15:35:00.000Z",
"tags" : ["Java", "Lucene"],
"conference" : {
"name" : "JAX",
"city" : "Mainz",
"coordinates": {
"lon": "8.275626",
"lat": "50.002090"
}
}
}
![Page 39: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/39.jpg)
Mappingcurl -XPUT "http://localhost:9200/conferences/talk/_mapping" -d'
{
"properties": {
[…],
"conference": {
"type": "object",
"properties": {
"coordinates": {
"type": "geo_point"
}
}
}
}
}'
![Page 40: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/40.jpg)
Sortierungcurl -XPOST "http://localhost:9200/conferences/_search " -d'
{
"sort" : [
{
"_geo_distance" : {
"conference.coordinates" : {
"lon": 8.403697,
"lat": 49.006616
},
"order" : "asc",
"unit" : "km"
}
}
]
}'
![Page 41: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/41.jpg)
Filterung
curl -XPOST "http://localhost:9200/conferences/_search" -d'
{
"filter": {
"geo_distance": {
"conference.coordinates": {
"lon": 8.403697,
"lat": 49.006616
},
"distance": "200km",
"distance_type": "arc"
}
}
}'
![Page 42: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/42.jpg)
Recap
● Elasticsearch kann mehr als Volltext● Ausgefeilte Geo-Algorithmen● Sortierung nach Distanz● Filterung nach Distanz oder Bereich● Berechnung von Distanz
![Page 43: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/43.jpg)
User
● FourSquare● http://engineering.foursquare.com/2012/08/09/fo
ursquare-now-uses-elastic-search-and-on-a-related-note-slashem-also-works-with-elastic-search/
● Gild● http://www.elasticsearch.org/case-study/gild/
![Page 44: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/44.jpg)
Logfile-AnalyseLog-Analyse
![Page 45: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/45.jpg)
Logfile-Analyse
● Zentralisierung Logs aus Anwendungen● Zentralisierung Logs über Maschinen
● Auch ohne Zugriff● Leichte Durchsuchbarkeit● Real-Time-Analysis / Visualisierung● Daten für alle!
![Page 46: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/46.jpg)
Logfile-Analyse
● Einlesen● Logstash
● Speicherung● Elasticsearch
● Auswertung● Kibana
![Page 47: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/47.jpg)
Logfile-Analyse
![Page 48: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/48.jpg)
Logstash-Config
input {
file {
path => "/var/log/apache2/access.log"
}
}
filter {
grok {
match => { message => "%{COMBINEDAPACHELOG}" }
}
}
output {
elasticsearch_http {
host => "localhost"
}
}
![Page 49: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/49.jpg)
Kibana
![Page 50: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/50.jpg)
Recap
● Einlesen, Anreichern, Speichern von Logevents● Zahlreiche Inputs in Logstash● Konsolidierung● Zentralisierung● Auswertung
![Page 51: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/51.jpg)
User
● CERN● https://medium.com/@ghoranyi/needle-in-a-
haystack-873c97a99983● Bloomberg
● http://www.elasticsearch.org/videos/using-elasticsearch-logstash-kibana-techologies-centralized-viewing-logs-bloomberg/
![Page 52: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/52.jpg)
AnalyticsAnalytics
![Page 53: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/53.jpg)
Analytics
● Auswertung auch großer Datenmengen● Social Media ● Data Warehouse
● Aggregationen auf Feldern● Datenkonsolidierung aus unterschiedlichen
Quellen● Visualisierung
![Page 54: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/54.jpg)
Aggregations
curl -XGET "http://localhost:9200/devoxx/tweet/_search" -d'
{
"aggs" : {
"hashtags" : {
"terms" : {
"field" : "hashtag.text"
}
}
}
}'
![Page 55: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/55.jpg)
Aggregations
"aggregations": {
"hashtags": {
"buckets": [
{
"key": "dartlang",
"doc_count": 229
},
{
"key": "java",
"doc_count": 216
},
[...]
![Page 56: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/56.jpg)
Aggregations
"aggs" : {
"hashtags" : {
"terms" : {
"field" : "hashtag.text"
},
"aggs" : {
"hashtagusers" : {
"terms" : {
"field" : "user.screen_name"
}
}
}
}
}
![Page 57: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/57.jpg)
Aggregations
"key": "scala",
"doc_count": 130,
"hashtagusers": {
"buckets": [
{
"key": "jaceklaskowski",
"doc_count": 74
},
{
"key": "ManningBooks",
"doc_count": 3
},
[...]
![Page 58: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/58.jpg)
Visualisierung
![Page 59: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/59.jpg)
Recap
● Auswertung großer Datenmengen● Visualisierung● Zahlreiche Aggregationen
● Bucket-Aggregations● Metric-Aggregations
![Page 60: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/60.jpg)
User
● Cogenta● https://www.elastic.co/use-cases/cogenta
● The Guardian● http://www.elasticsearch.org/blog/using-
elasticsearch-and-logstash-to-serve-billions-of-searchable-events-for-customers/
● http://www.infoq.com/presentations/elasticsearch-guardian
![Page 61: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/61.jpg)
Agenda
![Page 63: Anwendungsfälle für Elasticsearch JAX 2015](https://reader031.vdocument.in/reader031/viewer/2022032422/55a92d8b1a28ab3f578b465f/html5/thumbnails/63.jpg)
Images
● http://www.morguefile.com/archive/display/685952
● http://www.morguefile.com/archive/display/2359
● http://www.morguefile.com/archive/display/615356
● http://www.morguefile.com/archive/display/914733
● http://www.morguefile.com/archive/display/826258
● http://www.morguefile.com/archive/display/170605
● http://www.morguefile.com/archive/display/181488