better search engine testing for solr

BETTER SEARCH ENGINE TESTING

FOR SOLR

CVILLE MEETUP | EPUGH@O19S.COM | @DEP4B

1

WHY AM I QUALIFIED TO BE UP HERE?

• Principal of OpenSource Connections - Solr Search Consultancy

•Member of Apache Software Foundation

• Presenter at conferences (OSCON, ApacheCON, UberConf, ExpoQA, STPcon 2009/2011!)

• Fascinated by the art of software development

2

AUTHOR

3.X Coming

Soon

3

FATHER

4

AGENDA

Why Test Search?

Two General Approaches

Test Tools for Solr

Wrap Up

5

REDUCE HUMAN ERROR

6

photo credit to Jay Grieves @ www.bigswingingdeveloper.com

7

TWO APPROACHES TO TEST

Humans

Automated Tests

8

HUMAN TESTING

• Better deals with ambiguity

9

RELEVANCY IS LIKE PORNOGRAPHY....

10

I KNOW IT WHEN I SEE IT!

http://en.wikipedia.org/wiki/Les_Amants

11

AUTOMATED TESTING12

SWEET SPOT

• Repeat Testing for Relevancy Tuning

• Performance Testing: hitting SLA?

• Regression Testing (Typical use case for automated testing)

13

“ONE TRUE ANSWER”

• Absolute Truth / Matrix / Grid / TREC / Relevancy Assertions

• The correct answers for each search are known ahead of time

• Humans judges often decide these correct answers, stored as Relevancy Assertions

• Can be labor intensive to setup

• A “Numerical Grade” is produced for comparision

14

SO WHAT IS A JUDGEMENT LIST?

15

16

COMPARABLE RESULTSOriginal AjaxSolr WebSolr

gata 4HIF 1 alpha

HIF-1 alpha

HIF1 alpha

HIF1AiNOSIRE-1 alpha

T7 TagCollagen I

Collagen 1

Collagen-1

40.0 80.0 80.090.0 100.0 30

100.0 100.0 30

70.0 100.0 100.0

100.0 100.0 100.060.0 60.0 60.030 90.0 90.0

0.0 100.0 90.050.0 90.0 90.0

40.0 90.0 40.0

10.0 50.0 40.0

17

WRAP UP

18

User  Interface

Search  Engine

Informa3on

Search  Stack

19

20

21

A HYBRID APPROACH?

Humans

Automated Tests

22

A HYBRID APPROACH?

Humans

Automated Tests

Tool SupportedTesting

Judgements23

James Bach,satisfice.com

TESTING IS A HUMAN ACTIVITY

24

25

• twitter: dep4b

• speakerrate: http://www.speakerrate.com/epugh/

• email: epugh@opensourceconnections.com

THANK YOU!

25