gregg donovan @lucenerevolution 2011
DESCRIPTION
Solr & Lucene at EtsyTRANSCRIPT
![Page 2: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/2.jpg)
1.5 years Solr & Lucene at Etsy.com
3 years Solr & Lucene at TheLadders.com
![Page 3: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/3.jpg)
![Page 4: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/4.jpg)
8+ million members
![Page 5: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/5.jpg)
9.3 million items
![Page 6: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/6.jpg)
800k+ active sellers
![Page 7: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/7.jpg)
1+ billion pageviews / month
![Page 8: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/8.jpg)
![Page 9: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/9.jpg)
![Page 10: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/10.jpg)
![Page 11: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/11.jpg)
![Page 12: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/12.jpg)
![Page 13: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/13.jpg)
![Page 14: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/14.jpg)
Maximize Solr out-of-the-box
![Page 15: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/15.jpg)
Hack at a low-level
![Page 16: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/16.jpg)
Know when to do each
![Page 17: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/17.jpg)
![Page 18: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/18.jpg)
Or
![Page 19: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/19.jpg)
![Page 20: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/20.jpg)
Don’t fear trunk
![Page 21: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/21.jpg)
builds.apache.org/job/Solr-trunk/changes
![Page 22: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/22.jpg)
![Page 23: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/23.jpg)
![Page 24: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/24.jpg)
http://localhost:8393/solr/placesuggest/select?
q={!lucene}s*&sfield=latlong&pt=37.595804,-122.364521
&sort=div(geodist(),sqrt(sum(population,50)))%20asc
![Page 25: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/25.jpg)
{!lucene}
{!field}
{!func}
{!dismax}
{!edismax}
{!boost}
{!term}
![Page 26: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/26.jpg)
Cheap ranking awesomeness
![Page 27: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/27.jpg)
![Page 28: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/28.jpg)
ExternalFileField ftw!
![Page 29: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/29.jpg)
schema.xml: <fieldType name="file" keyField="treasury_id" defVal="0" stored="false" indexed="true" class="solr.ExternalFileField" valType="float"/> <field name="hotness" type="file"/>
/search/data/treasury/external_hotness.1306390802088:1=2.32=1.73=1.1
Solr query:sort={!func}hotness+desc
![Page 30: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/30.jpg)
ExternalFileField caveats
![Page 31: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/31.jpg)
More relevance: boost query
![Page 32: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/32.jpg)
http://localhost:8983/solr/listings/select?q={!boost b=$rel v=$qq}&rel=category:furniture^10+OR+((-material:acrylic)^5)&qq=desk
![Page 33: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/33.jpg)
Impression tracking
![Page 34: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/34.jpg)
etsy.com/search?q=desk&explain=1
![Page 35: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/35.jpg)
Side-by-Side testing
![Page 36: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/36.jpg)
![Page 37: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/37.jpg)
Cheap performance wins
![Page 38: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/38.jpg)
Put off sharding till you must
![Page 39: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/39.jpg)
cat ${indexDir}/* > /dev/null
![Page 40: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/40.jpg)
Return IDs, minimize stored fields
![Page 41: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/41.jpg)
RAM: $10-20 / GB
![Page 42: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/42.jpg)
SSD: 0.1ms vs 10ms seek
![Page 43: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/43.jpg)
Custom?
![Page 44: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/44.jpg)
solr-user
![Page 45: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/45.jpg)
Tools for low-level hacking
![Page 46: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/46.jpg)
Continuous deployment
![Page 47: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/47.jpg)
![Page 48: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/48.jpg)
One button. So easy a dog could do it.
![Page 49: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/49.jpg)
![Page 50: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/50.jpg)
![Page 51: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/51.jpg)
MTTR > MTBF
![Page 52: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/52.jpg)
![Page 53: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/53.jpg)
![Page 54: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/54.jpg)
github.com/etsy/logster
![Page 55: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/55.jpg)
Tracking GC
![Page 56: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/56.jpg)
export GC_DEBUG="-verbose:gc -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintAdaptiveSizePolicy -XX:AdaptiveSizePolicyOutputInterval=1 -XX:+PrintTenuringDistribution -XX:+PrintGCDetails -Xloggc:/var/log/search/gc.log"
![Page 57: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/57.jpg)
![Page 58: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/58.jpg)
![Page 59: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/59.jpg)
Alerting
![Page 60: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/60.jpg)
Testing
![Page 61: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/61.jpg)
![Page 62: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/62.jpg)
SaveAsFixture
![Page 63: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/63.jpg)
Profiling
![Page 64: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/64.jpg)
Java Primitive Library
fastutil
trove4j
![Page 65: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/65.jpg)
Know the hooks
QParserPlugin
SolrEventListener
SolrRequestHandler
SearchComponent
SolrCache
ValueSourceParser
![Page 66: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/66.jpg)
SolrIndexSearcher gotchasreference counting
using it as a cache key:
WeakHashMap<SolrIndexSearcher,MyValue> myCache...
![Page 67: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/67.jpg)
Example:personalized collections
![Page 68: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/68.jpg)
![Page 69: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/69.jpg)
fq={!term f=id}123 OR {!term f=id}456
![Page 70: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/70.jpg)
Need a map of PK to docId
![Page 71: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/71.jpg)
Use custom SolrCache plus SolrEventListener to fill it
![Page 72: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/72.jpg)
github.com/giokincade/FastTermFilter
![Page 73: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/73.jpg)
i18n currency sorting and filtering
![Page 74: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/74.jpg)
![Page 75: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/75.jpg)
currency.xml:
<currencyConfig version="1.0">! <currencies>! ! <currency name="United States Dollar" symbol="$" code="USD"/>! ! <currency name="Australian Dollar" symbol="$" code="AUD"/>! ! <currency name="Canadian Dollar" symbol="$" code="CAD"/>! ! <currency name="Czech Koruna" symbol="Kč" code="CZK"/>...! </currencies>! <rates>! ! <rate from="USD" to="AUD" rate="1.168750"/>! ! <rate from="USD" to="CAD" rate="1.085000"/>! ! <rate from="USD" to="CZK" rate="20.107500"/>! ! <rate from="USD" to="DKK" rate="5.323750"/>... </rates></currencyConfig>
![Page 76: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/76.jpg)
price:[10.00USD to 50.00USD]
price:20.00EUR
price:[$10.00 to $50.00]
![Page 77: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/77.jpg)
MoneyFieldType.java:
@Override public Query getRangeQuery(QParser parser, SchemaField field, String part1, String part2, final boolean minInclusive, final boolean maxInclusive) { final MoneyValue p1 = MoneyValue.parse(part1, defaultCurrency); final MoneyValue p2 = MoneyValue.parse(part2, defaultCurrency);
if (!p1.getCurrencyCode().equals(p2.getCurrencyCode())) { throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, new ParseException("Cannot parse range query " + part1 + " to " + part2 + ": range queries only supported when upper and lower bound have same currency.")); }
String currencyCode = p1.getCurrencyCode(); final MoneyValueSource vs = new MoneyValueSource(field, currencyCode, parser);
return new SolrConstantScoreQuery(new ValueSourceRangeFilter(vs, p1.getAmount() + "", p2.getAmount() + "", minInclusive, maxInclusive)); }
![Page 78: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/78.jpg)
Replication gotcha
![Page 79: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/79.jpg)
SOLR-2202
![Page 80: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/80.jpg)
Related Searches
![Page 81: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/81.jpg)
Autosuggest!
![Page 82: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/82.jpg)
bjewlery dewelry ejewelry ejwelry ewelery ewerly ewlery fewelry fewlery fjewelery fjewelry gewerly gewlery hewelery hewelry hewerly hewlery hjewelry iewelry ijewelry jawelery jawlery jeawlery jeelery jeelry jeewelery jeewelry jeewlery jeewlry jefwelry jejelry jelelry jelery jellery jelwelery jelwelry jelwlery jemelry jemerly jemwelry jeqwelry jerelery jerelry jerely jererly jerlery jerwelery jerwelry jerwely jerwerly jeselery jeselry jevelry jeverly jewalery jewdelry jewedlry jeweelrry jeweelry jeweely jeweer jeweery jeweilry jeweiry jewejery jewejlry jewejrly jewejry jewekey jewekry jewelary jeweldy jewele jewelee jewelelry jewelera jewelerey jewelerly jewelert jewelerty jeweleru jeweleruy jeweleryl jewelerys jeweleryy jewelet jewelety jeweleya jewelfry jewelfy jeweliy jewellryp jewelltry jewelly jewelory jewelra jewelray jewelre jewelree jewelreyy jewelrfy jewelrh jewelri jewelrky jewelrly jewelrr jewelrs jewelrsy jewelrt jewelrty jewelru jewelruy jewelrye jewelryh jewelryl jewelrym jewelryr jewelrys jewelryt jewelryu jewelryuk jewelryy jewelrz jewelsry jewelsy jeweltry jewelty jewelw jewelwery jewelwey jewelwy jewelya jewelyj jewelyr jewelyry jewelyu jewelyy jewelzry jeweory jewerey jeweriy jewerky jewerlary jewerley jewerli jewerlly jewerls jewerlt jewerlu jewerlyh jewerlyr jewerlys jewerlyu jewerry jeweryl jewetry jewewlry jewewly jewewrly jewewry jeweylry jewiery jewilary jewkery jewlary jewledy jewleery jewlelery jewlely
![Page 83: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/83.jpg)
The TermDictionary is not a whitelist
![Page 84: Gregg Donovan @Lucenerevolution 2011](https://reader033.vdocument.in/reader033/viewer/2022052605/568c52001a28ab4916b4ee57/html5/thumbnails/84.jpg)