better search engine testing for solr
DESCRIPTION
A refactoring of my earlier presentation targeted more towards Solr, and less on testing. I'm basing my presentation for Lucene EuroCon on this. Would love some more feedback!TRANSCRIPT
WHY AM I QUALIFIED TO BE UP HERE?
• Principal of OpenSource Connections - Solr Search Consultancy
•Member of Apache Software Foundation
• Presenter at conferences (OSCON, ApacheCON, UberConf, ExpoQA, STPcon 2009/2011!)
• Fascinated by the art of software development
2
AUTHOR
3.X Coming
Soon
3
FATHER
4
AGENDA
Why Test Search?
Two General Approaches
Test Tools for Solr
Wrap Up
5
REDUCE HUMAN ERROR
6
photo credit to Jay Grieves @ www.bigswingingdeveloper.com
7
TWO APPROACHES TO TEST
Humans
Automated Tests
8
HUMAN TESTING
• Better deals with ambiguity
9
RELEVANCY IS LIKE PORNOGRAPHY....
10
I KNOW IT WHEN I SEE IT!
http://en.wikipedia.org/wiki/Les_Amants
11
AUTOMATED TESTING12
SWEET SPOT
• Repeat Testing for Relevancy Tuning
• Performance Testing: hitting SLA?
• Regression Testing (Typical use case for automated testing)
13
“ONE TRUE ANSWER”
• Absolute Truth / Matrix / Grid / TREC / Relevancy Assertions
• The correct answers for each search are known ahead of time
• Humans judges often decide these correct answers, stored as Relevancy Assertions
• Can be labor intensive to setup
• A “Numerical Grade” is produced for comparision
14
SO WHAT IS A JUDGEMENT LIST?
15
16
COMPARABLE RESULTSOriginal AjaxSolr WebSolr
gata 4HIF 1 alpha
HIF-1 alpha
HIF1 alpha
HIF1AiNOSIRE-1 alpha
T7 TagCollagen I
Collagen 1
Collagen-1
40.0 80.0 80.090.0 100.0 30
100.0 100.0 30
70.0 100.0 100.0
100.0 100.0 100.060.0 60.0 60.030 90.0 90.0
0.0 100.0 90.050.0 90.0 90.0
40.0 90.0 40.0
10.0 50.0 40.0
17
WRAP UP
18
User Interface
Search Engine
Informa3on
Search Stack
19
20
21
A HYBRID APPROACH?
Humans
Automated Tests
22
A HYBRID APPROACH?
Humans
Automated Tests
Tool SupportedTesting
Judgements23
James Bach,satisfice.com
TESTING IS A HUMAN ACTIVITY
24
25
• twitter: dep4b
• speakerrate: http://www.speakerrate.com/epugh/
• email: [email protected]
THANK YOU!
25