elasticsearch for lua developers - the programming ... for lua developers pablo musa...
TRANSCRIPT
![Page 2: Elasticsearch for Lua Developers - The Programming ... for Lua Developers Pablo Musa pablo@elastic.co + + Me Pablo Musa Educational Engineer @ Elastic 4 • 5 interested students •](https://reader031.vdocument.in/reader031/viewer/2022021820/5adc85877f8b9a595f8b9f88/html5/thumbnails/2.jpg)
+ +
![Page 3: Elasticsearch for Lua Developers - The Programming ... for Lua Developers Pablo Musa pablo@elastic.co + + Me Pablo Musa Educational Engineer @ Elastic 4 • 5 interested students •](https://reader031.vdocument.in/reader031/viewer/2022021820/5adc85877f8b9a595f8b9f88/html5/thumbnails/3.jpg)
MePablo Musa
Educational Engineer @ Elastic
![Page 4: Elasticsearch for Lua Developers - The Programming ... for Lua Developers Pablo Musa pablo@elastic.co + + Me Pablo Musa Educational Engineer @ Elastic 4 • 5 interested students •](https://reader031.vdocument.in/reader031/viewer/2022021820/5adc85877f8b9a595f8b9f88/html5/thumbnails/4.jpg)
www.elastic.co4
• 5 interested students
• 3 very good proposals
• Key Points: - Background (Lua, Elasticsearch, Open Source) - Availability (how many hours per week) - Future (development likelihood after GSoC)
Which student?
![Page 5: Elasticsearch for Lua Developers - The Programming ... for Lua Developers Pablo Musa pablo@elastic.co + + Me Pablo Musa Educational Engineer @ Elastic 4 • 5 interested students •](https://reader031.vdocument.in/reader031/viewer/2022021820/5adc85877f8b9a595f8b9f88/html5/thumbnails/5.jpg)
Dhaval Kapil
3rd year computer science student at IIT Roorkee
Passionate about networking, security and development
Active participant in CTFs. Also, a developer and challenge setter of Backdoor CTF. (http://backdoor.sdslabs.co/)
Mostly self-taught himself by reading books and online articles.
https://dhavalkapil.com/
![Page 6: Elasticsearch for Lua Developers - The Programming ... for Lua Developers Pablo Musa pablo@elastic.co + + Me Pablo Musa Educational Engineer @ Elastic 4 • 5 interested students •](https://reader031.vdocument.in/reader031/viewer/2022021820/5adc85877f8b9a595f8b9f88/html5/thumbnails/6.jpg)
www.elastic.co6
• Elasticsearch
• Elasticsearch & Lua
• elasticsearch-lua
• WERpapers
• Elastic
Agenda
![Page 7: Elasticsearch for Lua Developers - The Programming ... for Lua Developers Pablo Musa pablo@elastic.co + + Me Pablo Musa Educational Engineer @ Elastic 4 • 5 interested students •](https://reader031.vdocument.in/reader031/viewer/2022021820/5adc85877f8b9a595f8b9f88/html5/thumbnails/7.jpg)
E l a s t i c s e a r c h i s . . .
![Page 8: Elasticsearch for Lua Developers - The Programming ... for Lua Developers Pablo Musa pablo@elastic.co + + Me Pablo Musa Educational Engineer @ Elastic 4 • 5 interested students •](https://reader031.vdocument.in/reader031/viewer/2022021820/5adc85877f8b9a595f8b9f88/html5/thumbnails/8.jpg)
www.elastic.co8
an open source, distributed, scalable,
highly available, document-oriented, RESTful
full text search engine
with soft real-time search and analytics capabilities
Elasticsearch is...
![Page 9: Elasticsearch for Lua Developers - The Programming ... for Lua Developers Pablo Musa pablo@elastic.co + + Me Pablo Musa Educational Engineer @ Elastic 4 • 5 interested students •](https://reader031.vdocument.in/reader031/viewer/2022021820/5adc85877f8b9a595f8b9f88/html5/thumbnails/9.jpg)
www.elastic.co9
an open source, distributed, scalable, highly available, document-oriented,
RESTful, full text search engine with real-time search and analytics capabilities
Elasticsearch is...
Apache 2.0 License https://www.apache.org/licenses/LICENSE-2.0
![Page 10: Elasticsearch for Lua Developers - The Programming ... for Lua Developers Pablo Musa pablo@elastic.co + + Me Pablo Musa Educational Engineer @ Elastic 4 • 5 interested students •](https://reader031.vdocument.in/reader031/viewer/2022021820/5adc85877f8b9a595f8b9f88/html5/thumbnails/10.jpg)
www.elastic.co10
an open source, distributed, scalable, highly available, document-oriented,
RESTful, full text search engine with real-time search and analytics capabilities
Elasticsearch is...
![Page 11: Elasticsearch for Lua Developers - The Programming ... for Lua Developers Pablo Musa pablo@elastic.co + + Me Pablo Musa Educational Engineer @ Elastic 4 • 5 interested students •](https://reader031.vdocument.in/reader031/viewer/2022021820/5adc85877f8b9a595f8b9f88/html5/thumbnails/11.jpg)
www.elastic.co11
an open source, distributed, scalable, highly available, document-oriented,
RESTful, full text search engine with real-time search and analytics capabilities
Elasticsearch is...
![Page 12: Elasticsearch for Lua Developers - The Programming ... for Lua Developers Pablo Musa pablo@elastic.co + + Me Pablo Musa Educational Engineer @ Elastic 4 • 5 interested students •](https://reader031.vdocument.in/reader031/viewer/2022021820/5adc85877f8b9a595f8b9f88/html5/thumbnails/12.jpg)
www.elastic.co12
an open source, distributed, scalable, highly available, document-oriented,
RESTful, full text search engine with real-time search and analytics capabilities
Elasticsearch is...
![Page 13: Elasticsearch for Lua Developers - The Programming ... for Lua Developers Pablo Musa pablo@elastic.co + + Me Pablo Musa Educational Engineer @ Elastic 4 • 5 interested students •](https://reader031.vdocument.in/reader031/viewer/2022021820/5adc85877f8b9a595f8b9f88/html5/thumbnails/13.jpg)
www.elastic.co13
an open source, distributed, scalable, highly available, document-oriented,
RESTful, full text search engine with real-time search and analytics capabilities
Elasticsearch is...
{ "name" : "Webinar" "geo" : { "city" : "Amsterdam", "lat" : 4.85, "lon" : 52.34 } }
Source: http://json.org/
![Page 14: Elasticsearch for Lua Developers - The Programming ... for Lua Developers Pablo Musa pablo@elastic.co + + Me Pablo Musa Educational Engineer @ Elastic 4 • 5 interested students •](https://reader031.vdocument.in/reader031/viewer/2022021820/5adc85877f8b9a595f8b9f88/html5/thumbnails/14.jpg)
www.elastic.co14
an open source, distributed, scalable, highly available, document-oriented,
RESTful, full text search engine with real-time search and analytics capabilities
Elasticsearch is...
Source: https://httpwg.github.io/asset/http.svg
![Page 15: Elasticsearch for Lua Developers - The Programming ... for Lua Developers Pablo Musa pablo@elastic.co + + Me Pablo Musa Educational Engineer @ Elastic 4 • 5 interested students •](https://reader031.vdocument.in/reader031/viewer/2022021820/5adc85877f8b9a595f8b9f88/html5/thumbnails/15.jpg)
www.elastic.co15
an open source, distributed, scalable, highly available, document-oriented,
RESTful, full text search engine with real-time search and analytics capabilities
Elasticsearch is...
![Page 16: Elasticsearch for Lua Developers - The Programming ... for Lua Developers Pablo Musa pablo@elastic.co + + Me Pablo Musa Educational Engineer @ Elastic 4 • 5 interested students •](https://reader031.vdocument.in/reader031/viewer/2022021820/5adc85877f8b9a595f8b9f88/html5/thumbnails/16.jpg)
www.elastic.co16
an open source, distributed, scalable, highly available, document-oriented,
RESTful, full text search engine with real-time search and analytics capabilities
Elasticsearch is...
![Page 17: Elasticsearch for Lua Developers - The Programming ... for Lua Developers Pablo Musa pablo@elastic.co + + Me Pablo Musa Educational Engineer @ Elastic 4 • 5 interested students •](https://reader031.vdocument.in/reader031/viewer/2022021820/5adc85877f8b9a595f8b9f88/html5/thumbnails/17.jpg)
www.elastic.co17
Getting up and running... is easy
$ wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.1.tar.gz
$ tar -zxf elasticsearch-1.7.1.tar.gz $ cd elasticsearch-1.7.1
$ ./bin/elasticsearch
http://localhost:9200
![Page 18: Elasticsearch for Lua Developers - The Programming ... for Lua Developers Pablo Musa pablo@elastic.co + + Me Pablo Musa Educational Engineer @ Elastic 4 • 5 interested students •](https://reader031.vdocument.in/reader031/viewer/2022021820/5adc85877f8b9a595f8b9f88/html5/thumbnails/18.jpg)
E l a s t i c s e a r c h & L u a
![Page 19: Elasticsearch for Lua Developers - The Programming ... for Lua Developers Pablo Musa pablo@elastic.co + + Me Pablo Musa Educational Engineer @ Elastic 4 • 5 interested students •](https://reader031.vdocument.in/reader031/viewer/2022021820/5adc85877f8b9a595f8b9f88/html5/thumbnails/19.jpg)
www.elastic.co19
About the Project
• 3-month project
• A "complete" Lua client to access the Elasticsearch REST API
• Influence of official clients
• Respect the student
![Page 20: Elasticsearch for Lua Developers - The Programming ... for Lua Developers Pablo Musa pablo@elastic.co + + Me Pablo Musa Educational Engineer @ Elastic 4 • 5 interested students •](https://reader031.vdocument.in/reader031/viewer/2022021820/5adc85877f8b9a595f8b9f88/html5/thumbnails/20.jpg)
"Nobody should have a reason not to use the client"
We have no opinions
Respect the language
Elasticsearch Clients Design Principles
![Page 21: Elasticsearch for Lua Developers - The Programming ... for Lua Developers Pablo Musa pablo@elastic.co + + Me Pablo Musa Educational Engineer @ Elastic 4 • 5 interested students •](https://reader031.vdocument.in/reader031/viewer/2022021820/5adc85877f8b9a595f8b9f88/html5/thumbnails/21.jpg)
www.elastic.co21
Much more than just a "JSON and HTTP wrapper"
• Distributed systems - sniffer - selector
• Add Features - reindex - common syntax
![Page 22: Elasticsearch for Lua Developers - The Programming ... for Lua Developers Pablo Musa pablo@elastic.co + + Me Pablo Musa Educational Engineer @ Elastic 4 • 5 interested students •](https://reader031.vdocument.in/reader031/viewer/2022021820/5adc85877f8b9a595f8b9f88/html5/thumbnails/22.jpg)
www.elastic.co22
High Level Architecture
Client
SnifferTransport
Connection Pool Selector
...Random
RoundRobin
ConnectionConnectionConnection
other http libluasocket
Parser
lua-‐yajlcjson
Helper
![Page 23: Elasticsearch for Lua Developers - The Programming ... for Lua Developers Pablo Musa pablo@elastic.co + + Me Pablo Musa Educational Engineer @ Elastic 4 • 5 interested students •](https://reader031.vdocument.in/reader031/viewer/2022021820/5adc85877f8b9a595f8b9f88/html5/thumbnails/23.jpg)
e l a s t i c s e a r c h - l u a
![Page 24: Elasticsearch for Lua Developers - The Programming ... for Lua Developers Pablo Musa pablo@elastic.co + + Me Pablo Musa Educational Engineer @ Elastic 4 • 5 interested students •](https://reader031.vdocument.in/reader031/viewer/2022021820/5adc85877f8b9a595f8b9f88/html5/thumbnails/24.jpg)
www.elastic.co24
Lua vs JSON
{ title = "A Requirements Elicitation Approach Based in Templates and Patterns", author = "A. Durán Toro", add_authors = { "B. Bernárdez Jiménez", "A. Ruiz Cortés", "M. Toro Bonilla" }, key_words = { "requirements engineering", "requirements elicitation" }, language = "en", conference = "WER99", }
{ "title" : "A Requirements Elicitation Approach Based in Templates and Patterns", "author" : "A. Durán Toro", "add_authors" : [ "B. Bernárdez Jiménez", "A. Ruiz Cortés", "M. Toro Bonilla" ], "key_words" : [ "requirements engineering", "requirements elicitation" ], "language" : "en", "conference" : "WER99", }
![Page 25: Elasticsearch for Lua Developers - The Programming ... for Lua Developers Pablo Musa pablo@elastic.co + + Me Pablo Musa Educational Engineer @ Elastic 4 • 5 interested students •](https://reader031.vdocument.in/reader031/viewer/2022021820/5adc85877f8b9a595f8b9f88/html5/thumbnails/25.jpg)
www.elastic.co25
CRUD
local paper = { title = "A Requirements Elicitation Approach Based in Templates and Patterns", ... }
local elasticsearch = require"elasticsearch" local client = elasticsearch.client() -- use default configs local r, e = client:index({ index = "papers", type = "wer", id = 1, body = paper })
local r, e = client:get( { index = "papers", type = "wer", id = 1 } )
local r, e = client:delete( { index = "papers", type = "wer", id = 1 } )
local r, e = client:update({ index = "papers", type = "wer", id = 1, body = { doc = { title = "Updated Title" } } })
![Page 26: Elasticsearch for Lua Developers - The Programming ... for Lua Developers Pablo Musa pablo@elastic.co + + Me Pablo Musa Educational Engineer @ Elastic 4 • 5 interested students •](https://reader031.vdocument.in/reader031/viewer/2022021820/5adc85877f8b9a595f8b9f88/html5/thumbnails/26.jpg)
www.elastic.co26
Searching
local r, e = client:search{ index = "papers", type = "wer", body = { query = { filtered = { query = { multi_match = { query = "requirements", fields = { "key_words^10", "title^5", "abstract" } } }, filter = { range = { year = { gte = "now-3y" } } } } } } }
local r, e = client:search( { index = "papers", type = "wer", q = "requirements" } )
local r, e = client:search( { index = "papers", type = "wer", q = "title:requirements"}){ took = 1.0, timed_out = false, _shards = { total = 5.0, successful = 5.0, failed = 0.0 }, hits = { total = 1.0, max_score = 0.17673586, hits = [ { _index = "papers", "_type": "wer", "_id": "1", _score = 0.17673586, _source = { title = "A Requirements Elicitation Approach Based in Templates and Patterns", key_words = { "requirements engineering", "requirements elicitation" }, year = "2013" } } ] } }
![Page 27: Elasticsearch for Lua Developers - The Programming ... for Lua Developers Pablo Musa pablo@elastic.co + + Me Pablo Musa Educational Engineer @ Elastic 4 • 5 interested students •](https://reader031.vdocument.in/reader031/viewer/2022021820/5adc85877f8b9a595f8b9f88/html5/thumbnails/27.jpg)
W h y E l a s t i c s e a r c h ?
![Page 28: Elasticsearch for Lua Developers - The Programming ... for Lua Developers Pablo Musa pablo@elastic.co + + Me Pablo Musa Educational Engineer @ Elastic 4 • 5 interested students •](https://reader031.vdocument.in/reader031/viewer/2022021820/5adc85877f8b9a595f8b9f88/html5/thumbnails/28.jpg)
www.elastic.co28
Workshop on Requirements Engineering
Ibero-american community of researchers on Requirements Engineering
![Page 29: Elasticsearch for Lua Developers - The Programming ... for Lua Developers Pablo Musa pablo@elastic.co + + Me Pablo Musa Educational Engineer @ Elastic 4 • 5 interested students •](https://reader031.vdocument.in/reader031/viewer/2022021820/5adc85877f8b9a595f8b9f88/html5/thumbnails/29.jpg)
www.elastic.co29
WERpapers
• Dedicated site to the WER publications
• Apache + CGILua + MySQL + Google Search
![Page 30: Elasticsearch for Lua Developers - The Programming ... for Lua Developers Pablo Musa pablo@elastic.co + + Me Pablo Musa Educational Engineer @ Elastic 4 • 5 interested students •](https://reader031.vdocument.in/reader031/viewer/2022021820/5adc85877f8b9a595f8b9f88/html5/thumbnails/30.jpg)
• Slow • No text analysis • No scoring • No auto-completion • ...
Not for full text search
![Page 31: Elasticsearch for Lua Developers - The Programming ... for Lua Developers Pablo Musa pablo@elastic.co + + Me Pablo Musa Educational Engineer @ Elastic 4 • 5 interested students •](https://reader031.vdocument.in/reader031/viewer/2022021820/5adc85877f8b9a595f8b9f88/html5/thumbnails/31.jpg)
![Page 32: Elasticsearch for Lua Developers - The Programming ... for Lua Developers Pablo Musa pablo@elastic.co + + Me Pablo Musa Educational Engineer @ Elastic 4 • 5 interested students •](https://reader031.vdocument.in/reader031/viewer/2022021820/5adc85877f8b9a595f8b9f88/html5/thumbnails/32.jpg)
www.elastic.co
• Implement your own search result page
• Relevancy based on downloads, title, abstract, etc.
• Auto-complete, Highlighting, Geo, etc.
• Everything in Lua
32
Elasticsearch
![Page 33: Elasticsearch for Lua Developers - The Programming ... for Lua Developers Pablo Musa pablo@elastic.co + + Me Pablo Musa Educational Engineer @ Elastic 4 • 5 interested students •](https://reader031.vdocument.in/reader031/viewer/2022021820/5adc85877f8b9a595f8b9f88/html5/thumbnails/33.jpg)
E l a s t i c s e a r c h i s n o t A l o n e
![Page 34: Elasticsearch for Lua Developers - The Programming ... for Lua Developers Pablo Musa pablo@elastic.co + + Me Pablo Musa Educational Engineer @ Elastic 4 • 5 interested students •](https://reader031.vdocument.in/reader031/viewer/2022021820/5adc85877f8b9a595f8b9f88/html5/thumbnails/34.jpg)
www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited
34
Elastic
2012 2013 2014 2015
Found Elasticsearch as a Service
ElasticsearchStore, search, analyze Logstash
Collect, parse and enrich data
Kibana Visualize and explore data
![Page 35: Elasticsearch for Lua Developers - The Programming ... for Lua Developers Pablo Musa pablo@elastic.co + + Me Pablo Musa Educational Engineer @ Elastic 4 • 5 interested students •](https://reader031.vdocument.in/reader031/viewer/2022021820/5adc85877f8b9a595f8b9f88/html5/thumbnails/35.jpg)
![Page 36: Elasticsearch for Lua Developers - The Programming ... for Lua Developers Pablo Musa pablo@elastic.co + + Me Pablo Musa Educational Engineer @ Elastic 4 • 5 interested students •](https://reader031.vdocument.in/reader031/viewer/2022021820/5adc85877f8b9a595f8b9f88/html5/thumbnails/36.jpg)
www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited
36
• Forums for all products
• Issue tracker
• User groups & Meetups
Community, community, community
![Page 37: Elasticsearch for Lua Developers - The Programming ... for Lua Developers Pablo Musa pablo@elastic.co + + Me Pablo Musa Educational Engineer @ Elastic 4 • 5 interested students •](https://reader031.vdocument.in/reader031/viewer/2022021820/5adc85877f8b9a595f8b9f88/html5/thumbnails/37.jpg)
www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited
37
• Forums for all products
• Issue tracker
• User groups & Meetups
Community, community, community
![Page 38: Elasticsearch for Lua Developers - The Programming ... for Lua Developers Pablo Musa pablo@elastic.co + + Me Pablo Musa Educational Engineer @ Elastic 4 • 5 interested students •](https://reader031.vdocument.in/reader031/viewer/2022021820/5adc85877f8b9a595f8b9f88/html5/thumbnails/38.jpg)
www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited
38
• Forums for all products
• Issue tracker
• User groups & Meetups
Community, community, community
![Page 39: Elasticsearch for Lua Developers - The Programming ... for Lua Developers Pablo Musa pablo@elastic.co + + Me Pablo Musa Educational Engineer @ Elastic 4 • 5 interested students •](https://reader031.vdocument.in/reader031/viewer/2022021820/5adc85877f8b9a595f8b9f88/html5/thumbnails/39.jpg)
www.elastic.co39
Next Steps
+Tests
+Real applications
+Use cases
Automate Integration with Elasticsearch
Make it an official client
![Page 41: Elasticsearch for Lua Developers - The Programming ... for Lua Developers Pablo Musa pablo@elastic.co + + Me Pablo Musa Educational Engineer @ Elastic 4 • 5 interested students •](https://reader031.vdocument.in/reader031/viewer/2022021820/5adc85877f8b9a595f8b9f88/html5/thumbnails/41.jpg)
www.elastic.co41
References
https://github.com/DhavalKapil/elasticsearch-lua https://dhavalkapil.com/blogs/Elasticsearch-Lua/ http://lua-users.org/wiki/JsonModules http://wer.inf.puc-rio.br/WERpapers/