solr for analytics: metrics aggregations at sematext, presented by otis gospodnetic at lucene/solr...
DESCRIPTION
While Solr and Lucene were originally written for full-text search, they are capable and increasingly used for Analytics, as Key Value Stores, NoSQL databases, and more. In this session we'll describe our experience with Solr for Analytics. More specifically, we will describe a couple of different approaches we have taken with SolrCloud for aggregation of massive amounts of performance metrics, we'll share our findings, and compare SolrCloud with HBase for large-scale, write-intensive aggregations. We'll also visit several Solr new features that are in the works that will make Solr even more suitable for Analytics workloads.TRANSCRIPT
+L���
���������������%LJ�GDWD�LV�OLNH�WHHQDJH�VH[������������������HYHU\RQH�WDONV�DERXW�LW������������������QRERG\�UHDOO\�NQRZV�KRZ�WR�GR�LW������������������HYHU\RQH�WKLQNV�HYHU\RQH�HOVH�LV�GRLQJ�LW������������������VR�HYHU\RQH�FODLPV�WKH\�DUH�GRLQJ�LW����
������������������'DQ�$ULHO\��'XNH�8QLYHUVLW\
6ROU�IRU�$QDO\WLFV0HWULFV�$JJUHJDWLRQV�DW�6HPDWH[W
2WLV�*RVSRGQHWLü�#RWLVJ
2*���3UR
«�VLQFH�FLUFD������
«��VLQFH�����
1HHGOH�$GGLFWLRQ
�
:DWHUPHORQ�$GGLFWLRQ
�
%UHDG�/RYH
�
1XWV�
�
$GGLFWLRQ�0DQLIHVWDWLRQ����
*DWKHULQJ�䘟
6HDUFK�����䘟
$QDO\WLFV��䘟
/I<gEP��[<YsjQEh
$GGLFWLRQ�0DQLIHVWDWLRQ���
*DWKHULQJ�䘟
6HDUFK�����䘟
$QDO\WLFV��䘟
/+!
$GGLFWLRQ�0DQLIHVWDWLRQ���
*DWKHULQJ�䘟
6HDUFK�����䘟
$QDO\WLFV��䘟
]OhI[I
7KH�6XUIDFH
�
8QGHU�WKH�+RRG
�
8QGHU�WKH�+RRG
�
6WULSSHG
�
�GLPHQVLRQ/LVW�QDPH �VROUBFDFKHVBPHWULFVBFXEH�IRUPXOD �V\VWHP �PRQWK�ZHHN�GD\�KRXU�ILYHBPLQXWHV�PLQXWH� VROUBFDFKHBQDPH KRVW VROUBFRUHBQDPH��!
6FKHPD
�ILHOG�QDPH �LG������������������������W\SH �VWULQJ���LQGH[HG �WUXH����VWRUHG �WUXH��UHTXLUHG �WUXH��PXOWL9DOXHG �IDOVH��!
�ILHOG�QDPH �EDVH5RZ.H\���W\SH �ELQDU\��LQGH[HG �IDOVH��VWRUHG �WUXH��UHTXLUHG �WUXH��PXOWL9DOXHG �IDOVH��!
�ILHOG�QDPH �GLP/LVW,G�����������W\SH �LQW���������LQGH[HG �WUXH����VWRUHG �WUXH��UHTXLUHG �WUXH��PXOWL9DOXHG �IDOVH��!
�ILHOG�QDPH �PHWULFV'DWD������W\SH �ELQDU\��LQGH[HG �IDOVH��VWRUHG �WUXH��UHTXLUHG �WUXH��PXOWL9DOXHG �IDOVH��!
�G\QDPLF)LHOG�QDPH �GLPBVWUB ������W\SH �VWULQJ��LQGH[HG �WUXH���VWRUHG �IDOVH���PXOWL9DOXHG �WUXH��GRF9DOXHV �WUXH��!
�G\QDPLF)LHOG�QDPH �GLPBQXPB ���W\SH �WORQJ���LQGH[HG �WUXH���VWRUHG �IDOVH���PXOWL9DOXHG �WUXH��GRF9DOXHV �WUXH��!
���GLPHQVLRQ/LVW�QDPH �VROUBFDFKHVBPHWULFVBFXEH�����IRUPXOD �V\VWHP �PRQWK�ZHHN�GD\�KRXU�ILYHBPLQXWHV�PLQXWH� VROUBFDFKHBQDPH KRVW VROUBFRUHBQDPH��!
$SSHQG�RQ�:ULWH��&RPSDFW�/DWHU"
�
&RPSDFWLRQ«�PD\EH"
�
,GHDOO\
�
JLWKXE�FRP
���VHPDWH[W���+%DVH+
87
0HUJH�RQ�5HDG
�
0HUJH�RQ�5HDG
�
0HWULF�5HWULHYDO
SDUDPV ^VKDUG�NH\V ����B����FROOHFWLRQ SDUWLDO3UHDJJ&ROOHFWLRQZW MDYDELQTW VHOHFWVWDUW �T GLP/LVW,G����IT GLPBQXPB��>��������������72��������������@IT GLPBVWUB��0M0�24��'��'IT GLPBVWUB����0'JZ<7OO1]JW17=N0\��2'KO/:(�1*8W=P0�2'(�1*1N1*1N�YHUVLRQ �URZV �����`KLWV ���VWDWXV ��47LPH ��
:ULWH�)ULHQGO\�3UHVHQW
62/5��������&ORXG6ROU6HUYHU62/5��������6WUHDPLQJ�XSGDWHV�EHWZHHQ�QRGHV62/5��������-DYDELQ�EHWZHHQ�QRGHV62/5�""""���$JJUHJDWLQJ�853
����YV�����
������QR�URXWLQJ�YV�
����URXWLQJ�YLD�&ORXG6ROU6HUYHU
)DVWHU�$)6�IOXVKHV)DVWHU�LQGH[LQJ������0�YV����0�GRFV�LQ����PLQ/RZHU�KHDS��FLUFD����
'RF9DOXHV
�ILHOG�QDPH �LG���������������������������������W\SH �VWULQJ��LQGH[HG �WUXH���VWRUHG �WUXH�����UHTXLUHG �WUXH�������GRF9DOXHV �WUXH��!
�G\QDPLF)LHOG�QDPH �GLPBVWUB ������W\SH �VWULQJ��LQGH[HG �WUXH���VWRUHG �IDOVH���PXOWL9DOXHG �WUXH��GRF9DOXHV �WUXH��!
�G\QDPLF)LHOG�QDPH �GLPBQXPB ���W\SH �WORQJ���LQGH[HG �WUXH���VWRUHG �IDOVH���PXOWL9DOXHG �WUXH��GRF9DOXHV �WUXH��!
)RUZDUG�LQGH[�GDWD�VWUXFWXUH«�LQ�PHPRU\�DQG�RU�RQ�GLVN«�FRPSUHVVLEOH«�RII�KHDS
6HHKWWSV���FZLNL�DSDFKH�RUJ�FRQIOXHQFH�GLVSOD\�VROU�'RF9DOXHVKWWS���VHDUFKKXE�RUJ������������IXQ�ZLWK�GRFYDOXHV�LQ�VROU�����
�'RF9DOXHV�YV��'RF9DOXHV
:LWK�'RF9DOXHV�«�VPDOOHU�KHDS«�PRUH�QXPHURXV��EXW�VPDOOHU�*&V«�URXJKO\�HTXDO�TXHU\�ODWHQF\«�IDVWHU�IDFHWLQJ��VRUWLQJ��DQG�JURXSLQJ��EXW�ZH�DUH�QRW�XVLQJ�WKDW�LQ�630�ZLWK�6ROU�«�GLGQ¶W�VHH�VORZ�GRZQ�LQ�LQGH[LQJ��KPPP�
�'RF9DOXHV�YV��'RF9DOXHV
$QDO\WLFV�)ULHQGO\�)XWXUH
/8&(1(��������1XPHULF�'RF9DOXHV�8SGDWHV62/5��������'LVWULEXWHG�3LYRW�)DFHWV62/5��������$QDO\WLFV�&RPSRQHQW62/5��������&RQILJXUDEOH�&ROOHFWRUV62/5��������3OXJJDEOH�$QDO\WLFV62/5��������)LHOG�&ROODSVLQJ�3RVW)LOWHU
),1���4XHVWLRQV�
������hIZ<jIrj�E]Z�<D]kj�W]Dh������ÃÁÚ�]NN�� .ÃÁÂÄ/+!ÃÁ���hdZ
���������������Y]OhI[I����������������������������������������hI<gEP�<[<YsjQEh
RWLV#VHPDWH[W�FRP³]jQhO��������������������������³hIZ<jIrj
)LOWHU�&ROOHFWLRQ
�