the power of elasticsearch

What is Search?

Search is a feature that can make or break an application or service. The ability to do full-text search across a body of text, to narrow search queries by values or ranges of specific fields, to use advanced features like faceted search, geo queries, or similarity searching can add a wealth of functionality and act as a differentiator for your product. When search fails, either because of inflexible query parameters, irrelevant results, or an inability to scale to meet demands of volume or usage, users notice immedi-ately -- and they will be upset.

What is Elasticsearch?

Elasticsearch is a new database built to handle huge amounts of data volume with very high availability and to distribute itself across many machines to be fault-tolerant and scalable, all the while maintaining a simple but powerful API that allows applica-tions from any language or framework access to the database.

© 2012 Infochimps, Inc. All rights reserved. 1

The Power of Elasticsearch

A horizontally-scalable, distributed database built on Apache’s Lucene that delivers a full-featured search experience across terabytes of data with a simple yet powerful API.

Built on LuceneApache’s Lucene is an open-source Java library for text search. The Lucene project has been growing for more than a decade and has now become the standard reference for how to build a powerful yet easy to integrate, open-source search library. Its feature set includes but is not limited to:

Performance/Scalability Search Features Accessibility

Can index over 95 GB/hr/node

Ranked/relevance searching with fine-grained control

Completely open-source

Low RAM overhead (1MB heap)

Allows querying on phrases, wildcards, geographical proximity, variable ranges, &c.

Implemented in Java so inherently cross-platform


Wrapping Lucene for Big Data

Lucene, as a search library, must be wrapped with an interface to allow its features to be used by an application. Many such interfaces have been built for different platforms and use cases. One of the most popular is Apache’s own SOLR project, which creates an interface around Lucene tailored for something like a traditional web application.

An interface like SOLR, however, is designed for a world in which a single server can handle the full workload of indexing and querying the data. When the data volume begins to increase past this limit, SOLR (and similar interfaces to Lucene) become unwieldy to use: the same problems of sharding, replication, and query dispatching that occur in RDBMS systems begin occur again in this context. And just as various methods exist for dealing with these difficulties in the RDBMS world, various tools exist for shard creation and distribution around SOLR.

But just as the right solution to big data databases means moving away from RDBMS into NoSQL technologies, the right solution to scaling Lucene is to move away from tools like SOLR and use a tool built from the ground-up to work with terabytes of data in a horizontally scalable, distributed, and fault-tolerant way: Elasticsearch!

Elastic Search Features

Elasticsearch is best thought of as an interface to Lucene designed for big data from the ground up. The complex feature set that Lucene provides for searching data is directly available through Elas-ticsearch, as Lucene is ultimately the library that’s used for indexing and querying data. This also means that plugins that work with Lucene will work with Elasticsearch out of the box.

The features that Elasticsearch itself provides around Lucene are designed to make it the perfect tool for full-text search on big data:

Performance/Scalability Robustness East of Use

An 8-node cluster can provide sub-200ms response latency when performing complex searches on 10B+ records!

No single point of failure Simple, JSON-based REST API means any language can index or query records in an Elasticsearch cluster.

Add or subtract nodes on the fly to dynamically scale the cluster to the current load

Automatically backup all data in the cluster to local disk or permanent, remote storage (like AWS’ S3 service).

Java and Thrift APIs exist for finer-grained or more performant access.

Ability to independently scale the indexing and querying performance of the cluster to deal with different sorts of use cases

Tune the replication factor of data on a per-index level

Flexible schemas allow for complex treatments of types like dateswithout forcing all documents in a table to be identical.

Data will automatically be migrated through the cluster if a node fails to maintain performance and replication factor.

Multiple indices enable multi-tenancy out of the box.


Example Use Cases

There’s a lot you can do with Elasticsearch besides just searching for phrases. The following exam-ples act as a quick guide to just a few of the features Elasticsearch provides.

Powerful Query SyntaxThe simplest way to interface with Elasticsearch is also one of the most powerful: the query string. Elasticsearch exposes the full Lucene query syntax through query strings that can be passed from a user in an application directly to the database to be evaluated.

Feature Query String Notes

Boolean logic (coke OR pepsi) AND health

Wildcards apple AND ip*d Wildcards can be applied for a single character (?) or for groups of characters (*).

Specific search fields coffee AND author:Smith Can search on deeply nested fields like “author.lastName” as well.

Search within a range apple AND date:[20100101 TO 20100201]

Boost results in relevance taxicab AND (“New York”^2 or “San Francisco”)

Boosting can also be configured at index time.


Records can be Complex DocumentsA record in Elasticsearch doesn’t have to be flat like a record in a traditional RDBMS. Elasticsearch allows documents to be hierarchical, and for sub-fields within a document to themselves have hierar-chical structure. This makes data modeling very flexible. An example of how one might store a blog post:

{

“id”: 1001,

“author”: {

“name”: “Alexander Hamilton”,

“id”: 3874

},

“date”: “1787-10-07 12:31:00 -0600 CST”,

“title”: “The Federalist Papers”,

“subtitle”: “Paper #1”

“text”: “AFTER an unequivocal experience of the inefficiency...”

“similar_posts”: [ 1002, 1003, 1005]

“comments”: [

{ “author”: “John Adams”, “text”: “I must beg to differ...” },

…

]

}

We could query these records using “author.name” or even “comments.text”, giving us a great deal of flexibilty in how we choose to denormalize and access the data in the database.


Geo QueriesElasticsearch understands geography. Geolocations can be stored within records as (latitude, longi-tude) pairs or as geohashes. In either case, Elasticsearch provides the ability to query using a variety of geo-methods:

Geo queries defined with a bounding box

Geo queries defined by distance range from a given point


Time SeriesThings change. It’s important to see how. Elasticsearch understands dates and times and can return time series data which represent an aggregation of the search results binned by time interval.

Raw tweets stored in Elasticsearch can be binned into a time series on the fly at query time.


Application SupportElasticsearch isn’t just a search engine; it’s a full-fledged database, and you can build an entire fron-tend application on top of it.

Elasticsearch supports multiple indices (databases) and multiple mappings (tables) per index. This feature, combined with the complex document structure Elasticsearch allows, lets you build the com-plex data models that support applications.

And, in addition to being able to execute rich search queries across the data, Elasticsearch allows the more “traditional” operations that define an application database: listing records, creating records, updating records, and deleting records. These features give you what you need to build a traditional database-driven, read/write application on top of the same database that lets you do full-text search and complex queries, all with horizontal scalability built-in from the ground up.

Administration & MonitoringElasticsearch also exposes a complete administrative and monitoring interface over the same API that powers the indexing, retrieval, and search of data.

Creating indices, updating their indexing or storage properties, defining rules for dealing with specific fields in specific mappings, &c. can all be accomplished via this same API.

Getting detailed information about the cluster’s availability state, health, individual nodes’ memory footprint, &c. is also available through this API, making monitoring of Elasticsearch easy.



About Infochimps

Our mission is to make the world’s data more accessible. Infochimps helps companies understand their data. We provide tools and services that connect their internal data, leverage the power of cloud computing and new technologies such as Hadoop, and provide a wealth of external datasets, which organizations can connect to their own data.

Contact UsInfochimps, Inc.1214 W 6th St. Suite 202 Austin, TX 78703

1-855-DATA-FUN (1-855-328-2386)

[email protected]

Twitter: @infochimps

Get a free Big Data consultationLet’s talk Big Data in the enterprise!

Get a free conference with the leading big data experts regarding your enterprise big data project. Meet with leading data scientists Flip Kromer and/or Dhruv Bansal to talk shop about your project objectives, design, infrastructure, tools, etc. Find out how other compa-nies are solving similar problems. Learn best practices and get recommendations — free.

the power of elasticsearch

Technology