practical search in the cloud - by marc krellenstein

Practical Searchin the Cloud

(sneak preview)

Martijn Koster, Lucid Imaginationmartijn.koster@lucidimagination.com

Wednesday, 25 May 2011

My Background Martijn Koster

Sr. Software Engineer at Lucid Imagination• Search in early Web – WebCrawler• Customer web services – Excite Mail• Proxies for Mobile – Danger

2

Wednesday, 25 May 2011

What I Will Cover A Solr based Cloud Service The need How Demo Wrap up

3

Wednesday, 25 May 2011

Why? – Getting Started is Hard Many new concepts

Complex software configuration

System Administration overhead

4

Wednesday, 25 May 2011

Search as a Service A few clicks to get started

Full Power of LucidWorks and Solr

Offload operations

Expertise, tools and processes are transferable to your own deployments.

5

Wednesday, 25 May 2011

Usage Model

6

Web UI

Direct Access

LWE JVMLWE JVM

Users

UserCode

Lucidworks

Solr

Wednesday, 25 May 2011

Architecture

7

Web UI(Rails)

Direct Access(Proxy with URL

rewriting)

Control(Chef)

LWE JVMLWE JVM

LWE JVM

Provisioner (Ruby)

Monitoring (Zabbix)

3rd party billingUsers

UserCode

Provisioning Service (Rails)

Heroku

Heroku

SQL(RDS)

Wednesday, 25 May 2011

Deployment Automation AMI

Chef

Custom

8

Wednesday, 25 May 2011

Scaling Heroku

Direct Access

Search Servers

Provisioning/DB

Monitoring and Control

9

Wednesday, 25 May 2011

Direct Access Example The Cloudy Direct Access Solr interface

• Simple submission• Simple search

First, copy SOLR access URL:

SOLR="https://access.cloudy.io/…/solr"

11

Wednesday, 25 May 2011

Example - Add$ cat > /tmp/doc.xml <<EOM<add><doc><field name="id">testdoc_LR2011</field><field name="text_en">LR rocks!</field></doc></add>EOMcurl $SOLR/my_news/update?commit=true \-H "Content-Type: text/xml" \--data-binary @/tmp/doc.xml

12

Wednesday, 25 May 2011

Example - Searchcurl $SOLR'/my_news/lucid?

q=news&role=DEFAULT&wt=json'

curl $SOLR'/my_news/select?wt=json&q=news&start=0&rows=10'

13

Wednesday, 25 May 2011

Example – Lucidworks API• The Cloudy Direct Access Lucid API interface

API="http://access.demo.cloudy.io/…/api”

curl $API/collections[

{"name":"lwelogs","instance_dir":"lwelogs”{"name":"collection1","instance_dir":"collection1_0"},{"name":"my_news","instance_dir":"my_news_1"}

]

14

Wednesday, 25 May 2011

Live Demo Sam Pierson, Director of Product Development

The Web User Interface• Administration UI improvements• Log indexing and graphing

API access• Ruby bindings for the LucidWorks REST API• http://rubygems.org/gems/lucid_works

10

Wednesday, 25 May 2011

Lessons Cloud technologies speed up development

Automate everything

Avoid lock-in

Dogfooding

16

Wednesday, 25 May 2011

Wrap Up

Search as a Service

Built in the Cloud, with open source

Full Lucidworks Enterprise and Solr

Powered by Lucid Imagination

17

Wednesday, 25 May 2011

When

Early Access Program: July• Interested? cloud@lucidimagination.com

18

Wednesday, 25 May 2011

Contact

Martijn Koster

• martijn.koster@lucidimagination.com

• http://www.lucidimagination.com/

• twitter.com/makuk66

19

Wednesday, 25 May 2011