mastering elasticsearch with ruby and tire

MasteringElasticSearch w/ Ruby (+ Tire)

RubyConf 2013

Luca Bonmassar, Gild

Tuesday, 5 November 13

Who am I?I’m Luca Bonmassar (@openmosix) # 31 # Italian living in San Francisco (and Stockholm)

I work at Gild

I love building products [for fun, profit and boredom]


Search - use case You’re building a product

User generated content

Let (other) users find or discover this content


Search is NOT easy It usually starts as

but then you want to support AND, OR, NOT, double quotes on multiple fields so

and then it goes like


And so it goes...


AgendaLet’s define a “pet project”

Boilerplate (download, install, scaffold, config, bla bla bla, yadda, yadda, yadda)

Build a website w/ simple search

Build a more advanced search

What next (homework)


Project! (for fun and no profit)


Project: build a full text-search for RubyGems.org


RubyGems Search Architecture

RubyGems.org

MongoDB

Rails App (Tire)

Elastic Search

Web Spider


RubyGems crawler

github.com/openmosix/rubygems-crawler

1. Download all ruby gem names from https://rubygems.org/gems2. Use gems API to download each gem info: read JSON - write JSON (MongoDB)


http://github.com/openmosix/rubygems-crawler

http://github.com/openmosix/rubygems-crawler

https://rubygems.org/gems

https://rubygems.org/gems

RubyGems Crawler II


RubyGems Search v1

Code: github.com/openmosix/rubygems-searchSpoiler: http://rubyconf.bonmassar.it


http://rubyconf.bonmassar.it


ElasticSearch


ElasticSearch II Open source search+analytics engine

Distributed[near] Realtime searchMulti tenantBuilt on Apache Lucene

REST APIsJSON documents


1. Download / set up a ES cluster

4. Query Index3. Index Data2. Define settings and data mapping (opt)

ElasticSearch III

Elastic Search MongoDB

Curl>

curl -‐X GET 'h.p://localhost:9200/ruby_gems/_search?from=0&size=25&pre.y'

{ "ok" : true, "status" : 200, "version" : { "number" : "0.90.5", "lucene_version" : "4.4" }, "tagline" : "You Know, for Search"}


http://localhost:9200/ruby_gems/_search?from=0&size=25&pretty'




ES download + setup

> wget http://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.6.tar.gz

> tar zxvf elasticsearch-0.90.6.tar.gz

> sudo mv elasticsearch-0.90.6 /usr/local/

Hint #1: you need JavaHint #2: you need Oracle Java


https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.6.tar.gz






ES config > ls elasticsearch-0.90.6/config/

Logging.yml: where to log, how much to logElasticsearch.yml: all server config. Defines:Name of the cluster (change it!!!)Node parameters (master/slave, store data/router)Sharing and # replicasPathsPluginsMemory (JVM, heap, memory locking)Network config“Gateway” (cluster backup)RecoveryDiscoverySlow log + GC log

Default options are good enough for dev env


ES boot + test Start: bin/elasticsearch

Test: curl http://localhost:9200/{ "ok" : true, "status" : 200, "name" : "Iron Fist", "version" : { "number" : "0.90.6", "lucene_version" : "4.5.1" }, "tagline" : "You Know, for Search"}

Stop: curl -XPOST 'http://localhost:9200/_shutdown'

{"cluster_name":"elasticsearch","nodes":{"jm2Z3J4dSzebjJ7Px2fAJg":{"name":"Iron Fist"}}}


http://localhost:9200

http://localhost:9200

http://localhost:9200/_shutdown'




Profit(you are now an ElasticSearch expert - go and tell the world)


ElasticSearch operations

Create a “RubyGem” IndexDefines a “RubyGem” Index data mapping

Index data (e.g. upload data from MongoDB to ES index = POST)

Query (= GET)


Tire now Re-Tire ;(

A ruby gem wrapping ElasticSearch REST APIs into a powerful ruby DSL

ActiveModel integration

Rake tasks and utilities to load and query ElasticSearch


Tire setupcat “gem ‘tire’” > Gemfile && bundle install

> cat config/initializers/elasticsearch.rb...Tire::Configuration.url('http://localhost:9200')Tire.configure { logger "#{Rails.root}/log/elasticsearch-queries.log" } if ENV['ES_LOG']


Define an ES index (with Tire DSL)


Indexing Get a recordConvert it to JSON format (to_indexed_json)Push it to Elastic Search (.update_index)

...under the hood...


Index (all data) Naive (POST on index for each record):

Use bulk updates:

...under the hood...


Search I


Search II


Simple Search


Highlight matches

Text

...add some CSS...


Advanced Search


Advanced Search II


Advanced Search III


Facets


ES facets


ES facets (running)


Facets - UI


Bonsai CoolI Search Suggesters (Did you mean... ?)


Bonsai Cool II

“Similar to this” (aka “More Like This” API)


Bonsai Cool III

Percolate API


DeploymentI Run your own cluster

Some learnings: at least 3 nodesmemory profiling / GCinstall very good monitoring (github.com/karmi/elasticsearch-paramedic)more RAM is (always) betterCheck IOPS (if on AWS)

Pros:Total controlCheaper (lot cheaper)

Cons:Can be a nightmare / Require dedicated devop


Deployment II ElasticSearch as a service

http://found.nohttp://searchly.comhttp://bonsai.io

Pros:Get cluster up & running in a minuteFocus on dev, not troubleshootingProfessional support

Cons:ExpensiveCan be in the wrong region / hosting providerExpensiveDid I say expensive?


http://found.no

http://found.no

http://www.searchly.com

http://www.searchly.com

http://bonsai.io

http://bonsai.io

Thanks!Code: github.com/openmosix/rubygems-searchgithub.com/openmosix/rubygems-crawler

Demo (will be down by the end of rubyconf): http://rubyconf.bonmassar.it

Say “hi”:Luca Bonmassar - [email protected]

twitter.com/openmosix

github.com/openmosix

linkedin.com/in/lucabonmassar




mailto:[email protected]

mailto:[email protected]

mastering elasticsearch with ruby and tire

Technology