full-text search: how it works and what it can do – couchbase connect 2016

©2016 Couchbase Inc.

Couchbase Full Text Search (FTS)

©2016 Couchbase Inc. 2©2016 Couchbase Inc.

about your speakers

Marty Schoch Steve Yen

©2016 Couchbase Inc.©2016 Couchbase Inc.

agenda

why?what is it? how does it work?how does it scale?demobest practicesstatus / roadmap / what’s next


agenda

why?


couchbase users need to search their documents


dedicated search solutions

✗ Provision✗ Install✗ Integrate✗ Transfer data✗ Learn✗ Manage✗ Troubleshoot

≠


why Full Text Search?

why Full Text Search?

simpleintegrated

80/20 of features


agenda

what is it?


what’s Full Text Search?


search results

Result Text Snippets


search results

Result Text Snippets

Highlighted Search Terms


agenda

how does it work?


how does it work?

• Inverted indexes• Language awareness• Scoring


inverted index

Terms

my: Doc 1, Doc 2, Doc 3dog: Doc 1, Doc 2, Doc 81has: Doc 1, Doc 2, Doc 3fleas: Doc 1, Doc 81…

Where found


language aware

Document contains…

Beauty

Indexed as…

beauti

stemmingstemming Text Analysis

✔Match!

User searches…

Beautiful

Searched as…

beauti


scoring


TF/IDF scoring

• TF = Term Frequency• How often does a term occur in a document?• More often yields a higher score

• IDF = Inverse Document Frequency• How many documents have this term?• More documents yields lower score • (because it means the term is more common)


index mapping


index mapping

•Exclude fields/sub-sections•Configure indexing behavior by type of document (beer vs brewery)•Configure indexing behavior per-field• Index Fields•Nested structures• Arrays


precision vs. recall

• Precision – ratio of document matches that are actually relevant• Recall – ratio of relevant documents that are actually matched• High quality results depend on performing the right analysis for your text• Beware: increasing precision may reduce recall (and vice versa)


agenda

how does it scale?


how does it scale?


how does it scale?

✔auto index partitioning? (hash partitioning))


how does it scale?

✔auto index partitioning (hash partitioning)

✔ (replicas promoted)


how does it scale?


✔to multiple FTS nodes? (auto-placement)



how does it scale?


✔to multiple FTS nodes (auto-placement)



how does it scale?



✔rebalance? (add/swap/remove)

(replicas promoted)


how does it scale?



✔rebalance (add/swap/remove)



how does it scale?




✔scatter/gather queries?)


how does it scale?




✔scatter/gather queries (partial results ok)



how does it scale?






how does it scale?





✔replicas? (only primaries queried)


how does it scale?





✔replicas (only primaries queried)


how does it scale?






✔failover? (replicas promoted)


how does it scale?






✔failover (replicas promoted)


agenda

demo


agenda

best practices


only use explicit field mappings in production

{ “type” : ”brewery”, “random_number” : 4, “edible” : false}

Dynamic mappings are great, until…


only use explicit field mappings in production

{ “type” : ”brewery”,

“comments”: 4k of text “random_number” : 4, “edible” : false}

Developer adds one small field


always use Index Aliases

Index Rebuilding



/users /usersV1



/users

/usersV1

/usersV2

Indexing 55%



/users

/usersV1

/usersV2

Atomic Switch to /usersV2



/users

/usersV2

Atomic Switch to /usersV2


go watch!

Dave Starling

seenit


agenda

status / roadmap / what’s next


project status

FTS is developer preview in 4.5, 4.6

planned GA in Spockplease help kick the tires

http://www.couchbase.com/download


links & Q+A

http://NICE-URL-TODO-HEREdownloads, getting started, tech docs

and, where you can ask questions

and share your feedback!

http://NICE/

http://NICE/


EXTRA SLIDES


FTS design

couchbase couchbase couchbase

FTS FTS FTS

cfg

DCP streamsfor incrementalindex updates

a cfg bucketholds metadata

about the indexes


Transition Slide TitleTransition Slide Subtitle Goes Here


Title of Slide Goes Here

• Heading 1• Heading 2



Title of Slide Goes Here






Speaker NameSpeakers TitleContact information

IMAGE GOES HERE


Thank You!


agenda

design


FTS design / index partitioning



bucket partitions: 0, 1, 2, 3, 4, … … ,1021, 1022, 1023 (1024 vbuckets)

FTS nodes:X Y Z




index partitions: (groups of vbuckets)

FTS nodes:X Y Z




index partitions: A B C(groups of vbuckets) 0-399 400-799 800-1023

FTS nodes:X Y Z





assign to FTS nodes:

FTS nodes:X Y Z





assign to FTS nodes: replicas, too:

FTS nodes:X Y Z


FTS design / indexing

couchbase couchbase couchbase

FTS FTS FTS

DCP streamsfor incrementalindex updates


FTS design / queries

a query sentto any FTSnode…

your application

REST

FTS FTS FTS


FTS design / queries

a query sentto any FTSnode…

…is scatter / gatheredto the other

FTS nodesRE

ST

your application

FTS FTS FTS

full-text search: how it works and what it can do – couchbase connect 2016

Software