solr for analytics: metrics aggregations at sematext, presented by otis gospodnetic at lucene/solr...

31
+L

Upload: lucenerevolution

Post on 11-May-2015

968 views

Category:

Technology


1 download

DESCRIPTION

While Solr and Lucene were originally written for full-text search, they are capable and increasingly used for Analytics, as Key Value Stores, NoSQL databases, and more. In this session we'll describe our experience with Solr for Analytics. More specifically, we will describe a couple of different approaches we have taken with SolrCloud for aggregation of massive amounts of performance metrics, we'll share our findings, and compare SolrCloud with HBase for large-scale, write-intensive aggregations. We'll also visit several Solr new features that are in the works that will make Solr even more suitable for Analytics workloads.

TRANSCRIPT

Page 1: Solr for Analytics: Metrics Aggregations at Sematext, Presented by Otis Gospodnetic at Lucene/Solr Revolution Dublin 2013

+L���

Page 2: Solr for Analytics: Metrics Aggregations at Sematext, Presented by Otis Gospodnetic at Lucene/Solr Revolution Dublin 2013

���������������%LJ�GDWD�LV�OLNH�WHHQDJH�VH[������������������HYHU\RQH�WDONV�DERXW�LW������������������QRERG\�UHDOO\�NQRZV�KRZ�WR�GR�LW������������������HYHU\RQH�WKLQNV�HYHU\RQH�HOVH�LV�GRLQJ�LW������������������VR�HYHU\RQH�FODLPV�WKH\�DUH�GRLQJ�LW����

������������������'DQ�$ULHO\��'XNH�8QLYHUVLW\

Page 3: Solr for Analytics: Metrics Aggregations at Sematext, Presented by Otis Gospodnetic at Lucene/Solr Revolution Dublin 2013

6ROU�IRU�$QDO\WLFV0HWULFV�$JJUHJDWLRQV�DW�6HPDWH[W

2WLV�*RVSRGQHWLü�#RWLVJ

Page 4: Solr for Analytics: Metrics Aggregations at Sematext, Presented by Otis Gospodnetic at Lucene/Solr Revolution Dublin 2013

2*���3UR

«�VLQFH�FLUFD������

«��VLQFH�����

Page 5: Solr for Analytics: Metrics Aggregations at Sematext, Presented by Otis Gospodnetic at Lucene/Solr Revolution Dublin 2013

1HHGOH�$GGLFWLRQ

Page 6: Solr for Analytics: Metrics Aggregations at Sematext, Presented by Otis Gospodnetic at Lucene/Solr Revolution Dublin 2013

:DWHUPHORQ�$GGLFWLRQ

Page 7: Solr for Analytics: Metrics Aggregations at Sematext, Presented by Otis Gospodnetic at Lucene/Solr Revolution Dublin 2013

%UHDG�/RYH

Page 8: Solr for Analytics: Metrics Aggregations at Sematext, Presented by Otis Gospodnetic at Lucene/Solr Revolution Dublin 2013

1XWV�

Page 9: Solr for Analytics: Metrics Aggregations at Sematext, Presented by Otis Gospodnetic at Lucene/Solr Revolution Dublin 2013

$GGLFWLRQ�0DQLIHVWDWLRQ����

*DWKHULQJ�䘟

6HDUFK�����䘟

$QDO\WLFV��䘟

/I<gEP��[<YsjQEh

Page 10: Solr for Analytics: Metrics Aggregations at Sematext, Presented by Otis Gospodnetic at Lucene/Solr Revolution Dublin 2013

$GGLFWLRQ�0DQLIHVWDWLRQ���

*DWKHULQJ�䘟

6HDUFK�����䘟

$QDO\WLFV��䘟

/+!

Page 11: Solr for Analytics: Metrics Aggregations at Sematext, Presented by Otis Gospodnetic at Lucene/Solr Revolution Dublin 2013

$GGLFWLRQ�0DQLIHVWDWLRQ���

*DWKHULQJ�䘟

6HDUFK�����䘟

$QDO\WLFV��䘟

]OhI[I

Page 12: Solr for Analytics: Metrics Aggregations at Sematext, Presented by Otis Gospodnetic at Lucene/Solr Revolution Dublin 2013

7KH�6XUIDFH

Page 13: Solr for Analytics: Metrics Aggregations at Sematext, Presented by Otis Gospodnetic at Lucene/Solr Revolution Dublin 2013

8QGHU�WKH�+RRG

Page 15: Solr for Analytics: Metrics Aggregations at Sematext, Presented by Otis Gospodnetic at Lucene/Solr Revolution Dublin 2013

8QGHU�WKH�+RRG

Page 16: Solr for Analytics: Metrics Aggregations at Sematext, Presented by Otis Gospodnetic at Lucene/Solr Revolution Dublin 2013

6WULSSHG

�GLPHQVLRQ/LVW�QDPH �VROUBFDFKHVBPHWULFVBFXEH�IRUPXOD �V\VWHP �PRQWK�ZHHN�GD\�KRXU�ILYHBPLQXWHV�PLQXWH� VROUBFDFKHBQDPH KRVW VROUBFRUHBQDPH��!

Page 17: Solr for Analytics: Metrics Aggregations at Sematext, Presented by Otis Gospodnetic at Lucene/Solr Revolution Dublin 2013

6FKHPD

�ILHOG�QDPH �LG������������������������W\SH �VWULQJ���LQGH[HG �WUXH����VWRUHG �WUXH��UHTXLUHG �WUXH��PXOWL9DOXHG �IDOVH��!

�ILHOG�QDPH �EDVH5RZ.H\���W\SH �ELQDU\��LQGH[HG �IDOVH��VWRUHG �WUXH��UHTXLUHG �WUXH��PXOWL9DOXHG �IDOVH��!

�ILHOG�QDPH �GLP/LVW,G�����������W\SH �LQW���������LQGH[HG �WUXH����VWRUHG �WUXH��UHTXLUHG �WUXH��PXOWL9DOXHG �IDOVH��!

�ILHOG�QDPH �PHWULFV'DWD������W\SH �ELQDU\��LQGH[HG �IDOVH��VWRUHG �WUXH��UHTXLUHG �WUXH��PXOWL9DOXHG �IDOVH��!

�G\QDPLF)LHOG�QDPH �GLPBVWUB ������W\SH �VWULQJ��LQGH[HG �WUXH���VWRUHG �IDOVH���PXOWL9DOXHG �WUXH��GRF9DOXHV �WUXH��!

�G\QDPLF)LHOG�QDPH �GLPBQXPB ���W\SH �WORQJ���LQGH[HG �WUXH���VWRUHG �IDOVH���PXOWL9DOXHG �WUXH��GRF9DOXHV �WUXH��!

���GLPHQVLRQ/LVW�QDPH �VROUBFDFKHVBPHWULFVBFXEH�����IRUPXOD �V\VWHP �PRQWK�ZHHN�GD\�KRXU�ILYHBPLQXWHV�PLQXWH� VROUBFDFKHBQDPH KRVW VROUBFRUHBQDPH��!

Page 18: Solr for Analytics: Metrics Aggregations at Sematext, Presented by Otis Gospodnetic at Lucene/Solr Revolution Dublin 2013

$SSHQG�RQ�:ULWH��&RPSDFW�/DWHU"

Page 19: Solr for Analytics: Metrics Aggregations at Sematext, Presented by Otis Gospodnetic at Lucene/Solr Revolution Dublin 2013

&RPSDFWLRQ«�PD\EH"

Page 20: Solr for Analytics: Metrics Aggregations at Sematext, Presented by Otis Gospodnetic at Lucene/Solr Revolution Dublin 2013

,GHDOO\

JLWKXE�FRP

���VHPDWH[W���+%DVH+

87

Page 21: Solr for Analytics: Metrics Aggregations at Sematext, Presented by Otis Gospodnetic at Lucene/Solr Revolution Dublin 2013

0HUJH�RQ�5HDG

Page 22: Solr for Analytics: Metrics Aggregations at Sematext, Presented by Otis Gospodnetic at Lucene/Solr Revolution Dublin 2013

0HUJH�RQ�5HDG

Page 23: Solr for Analytics: Metrics Aggregations at Sematext, Presented by Otis Gospodnetic at Lucene/Solr Revolution Dublin 2013

0HWULF�5HWULHYDO

SDUDPV ^VKDUG�NH\V ����B����FROOHFWLRQ SDUWLDO3UHDJJ&ROOHFWLRQZW MDYDELQTW VHOHFWVWDUW �T GLP/LVW,G����IT GLPBQXPB��>��������������72��������������@IT GLPBVWUB��0M0�24��'��'IT GLPBVWUB����0'JZ<7OO1]JW17=N0\��2'KO/:(�1*8W=P0�2'(�1*1N1*1N�YHUVLRQ �URZV �����`KLWV ���VWDWXV ��47LPH ��

Page 24: Solr for Analytics: Metrics Aggregations at Sematext, Presented by Otis Gospodnetic at Lucene/Solr Revolution Dublin 2013

:ULWH�)ULHQGO\�3UHVHQW

62/5��������&ORXG6ROU6HUYHU62/5��������6WUHDPLQJ�XSGDWHV�EHWZHHQ�QRGHV62/5��������-DYDELQ�EHWZHHQ�QRGHV62/5�""""���$JJUHJDWLQJ�853

Page 25: Solr for Analytics: Metrics Aggregations at Sematext, Presented by Otis Gospodnetic at Lucene/Solr Revolution Dublin 2013

����YV�����

������QR�URXWLQJ�YV�

����URXWLQJ�YLD�&ORXG6ROU6HUYHU

)DVWHU�$)6�IOXVKHV)DVWHU�LQGH[LQJ������0�YV����0�GRFV�LQ����PLQ/RZHU�KHDS��FLUFD����

Page 26: Solr for Analytics: Metrics Aggregations at Sematext, Presented by Otis Gospodnetic at Lucene/Solr Revolution Dublin 2013

'RF9DOXHV

�ILHOG�QDPH �LG���������������������������������W\SH �VWULQJ��LQGH[HG �WUXH���VWRUHG �WUXH�����UHTXLUHG �WUXH�������GRF9DOXHV �WUXH��!

�G\QDPLF)LHOG�QDPH �GLPBVWUB ������W\SH �VWULQJ��LQGH[HG �WUXH���VWRUHG �IDOVH���PXOWL9DOXHG �WUXH��GRF9DOXHV �WUXH��!

�G\QDPLF)LHOG�QDPH �GLPBQXPB ���W\SH �WORQJ���LQGH[HG �WUXH���VWRUHG �IDOVH���PXOWL9DOXHG �WUXH��GRF9DOXHV �WUXH��!

)RUZDUG�LQGH[�GDWD�VWUXFWXUH«�LQ�PHPRU\�DQG�RU�RQ�GLVN«�FRPSUHVVLEOH«�RII�KHDS

6HHKWWSV���FZLNL�DSDFKH�RUJ�FRQIOXHQFH�GLVSOD\�VROU�'RF9DOXHVKWWS���VHDUFKKXE�RUJ������������IXQ�ZLWK�GRFYDOXHV�LQ�VROU�����

Page 27: Solr for Analytics: Metrics Aggregations at Sematext, Presented by Otis Gospodnetic at Lucene/Solr Revolution Dublin 2013

�'RF9DOXHV�YV��'RF9DOXHV

:LWK�'RF9DOXHV�«�VPDOOHU�KHDS«�PRUH�QXPHURXV��EXW�VPDOOHU�*&V«�URXJKO\�HTXDO�TXHU\�ODWHQF\«�IDVWHU�IDFHWLQJ��VRUWLQJ��DQG�JURXSLQJ��EXW�ZH�DUH�QRW�XVLQJ�WKDW�LQ�630�ZLWK�6ROU�«�GLGQ¶W�VHH�VORZ�GRZQ�LQ�LQGH[LQJ��KPPP�

Page 28: Solr for Analytics: Metrics Aggregations at Sematext, Presented by Otis Gospodnetic at Lucene/Solr Revolution Dublin 2013

�'RF9DOXHV�YV��'RF9DOXHV

Page 29: Solr for Analytics: Metrics Aggregations at Sematext, Presented by Otis Gospodnetic at Lucene/Solr Revolution Dublin 2013

$QDO\WLFV�)ULHQGO\�)XWXUH

/8&(1(��������1XPHULF�'RF9DOXHV�8SGDWHV62/5��������'LVWULEXWHG�3LYRW�)DFHWV62/5��������$QDO\WLFV�&RPSRQHQW62/5��������&RQILJXUDEOH�&ROOHFWRUV62/5��������3OXJJDEOH�$QDO\WLFV62/5��������)LHOG�&ROODSVLQJ�3RVW)LOWHU

Page 30: Solr for Analytics: Metrics Aggregations at Sematext, Presented by Otis Gospodnetic at Lucene/Solr Revolution Dublin 2013

),1���4XHVWLRQV�

������hIZ<jIrj�E]Z�<D]kj�W]Dh������ÃÁÚ�]NN�� .ÃÁÂÄ/+!ÃÁ���hdZ

���������������Y]OhI[I����������������������������������������hI<gEP�<[<YsjQEh

RWLV#VHPDWH[W�FRP³]jQhO��������������������������³hIZ<jIrj

Page 31: Solr for Analytics: Metrics Aggregations at Sematext, Presented by Otis Gospodnetic at Lucene/Solr Revolution Dublin 2013

)LOWHU�&ROOHFWLRQ