presentation meetup elasticsearch paris #10
TRANSCRIPT
WHERE DO WE COME FROM?
• ActivePivot by QuartetFS:
– InMemory ultra fast business
intelligence tool
– Mainly for traders and market risk
analysts
• One of the biggest french
success (& secret) story
• We are big data crunchers for a
long time now
HOW DID EVERYTHING START?
• Created in may 2012
• We wanted to build the perfect tool
to understand the social WEB
• We started with a very famous
NoSQL engine
But we quickly had problems
(performance, clustering, query/txns
overlaps, etc…)
ELASTICSEARCH 2 YEARS AGO
BigData ready
Easy to use/manage
Performance/Scalability
Analytical capabilities
Primary document storage?
IT DIDN’T MATCH OUR ANALYTICS REQUIREMENTS
• FieldData cache
– High memory consumption
– Memory is expensive on the cloud
• No multi-field & multi-metric aggregations
– We could not build the product of our dreams
• But ElasticSearch is modular
– We decided to implement our own analytics plugin
A SEARCH ENGINE WITH OLAP SKILLS
• Support lazy loading of fields
• Multi-Fields & Multi-Metric aggregations
• Ultra-Fast & Efficient
– Usage of columnar storage with primitive types
– Sub-seconds queries over tens of millions of elements
ELASTICSEARCH GETS EVEN BETTER
• Release 1.0.0 – February 2014– First version of the Aggregation Engine
– Introduction of doc_values
• Release 1.2.0 – May 2014– global_ordinals / Faster Aggregations
• Release 1.4.0 – November 2014– Improv. Circuit Breakers / Safer Aggregations
– Improv. doc_values
• Every releases since 1.0.0– More stability
– More aggregation capabilies
• We had more time to develop other things!
INTRODUCING LOGMATIC
LOG GOODNESS POWERED BY ELASTICSEARCH
• Introduced Logmatic.io in private beta this year
– Beginning of 2014: A lot of logs projects around us
– Our logs experiment: It was an eye opener! (30 VMs / ~6 apps)
– 2 friendly startups tried: they went live
– Market Study: 12 projects launched
– We faced new challenges and had to build a new product!!
OUR APPROACH
Customer’s applications Our cloud based
infrastructureCTO, devops,
developers
Tomorrow, we’ll even have
business people. We’ll tell
you more…
alerts, reports
queries
• Centralises & enriches all
data
• Fully hosted (SaaS)
• Advanced analytics
• Real-Time
• Beautiful dataviz
• Rapidfire answers
securedconnection
And more…
UDPTCP (SSL)
HTTP(S)
INCOMING FEATURES
• Integrated Grok parsing:
– Logs shipping should be
the only concern
– Logs structuration is done
totally in the cloud
– We extended Grok to
simplify issues like date
parsing
INCOMING FEATURES
• Integrated Grok parsing:
– Logs shipping should be
the only concern
– Logs structuration is done
totally in the cloud
– We extended Grok to
simplify issues like date
parsing
• And much more…
– Security and limited views
– Dimension contexts
– Complex metrics /
formulas
WE CONTINUE WITH ELASTICSEARCH
• Scalability
• Heterogeneity
• Query performance
• Great analytics
• Reactivity of the team
logmatic.io
@logmatic_
We’d love to hear from you and answer the
questions you might have