enrich search user experience using amazon cloudsearch (svc302) | aws re:invent 2013

© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.

Enrich Search User Experience for Different Parts

of Your Application Using Amazon CloudSearch

Jon Handler, CloudSearch Solution Architect

November 15, 2013

Agenda

• Sourcing your documents

• Retrieval and ranking

• Search user interface

• Performance and scale

• Developer example:

Peter Simpkin, Solution Architect, Elsevier

Architecting with Amazon CloudSearch

Hands-Off Operation

SEARCH INSTANCE Index Partition n

Copy 1

SEARCH INSTANCE Index Partition 2

Copy 2

SEARCH INSTANCE Index Partition n

Copy 2


Copy n

SEARCH INSTANCE

Document Quantity and Size

Search Request Volume and Complexity

Index Partition n Copy n


Copy 1


Copy 1


Copy 2


Copy n

MovieMate Application

Multiple

Sources

Multiple

Functions

When wealthy industrialist Tony Stark is forced

to build an armored suit after a life-threatening

incident, he ultimately decides to use its

technology to fight against evil.

Iron Man (2008)

Tony Stark has declared himself Iron Man and

installed world peace... or so he thinks. He soon

realizes that not only is there a mad man...

Iron Man 2 (2010)

When Tony Stark's world is torn apart by a

formidable terrorist called the Mandarin, he

starts an odyssey of rebuilding and retribution.

Iron Man 3 (2013)

On the hunt for a fabled treasure of gold, a band

of warriors, assassins, and a rogue British soldier

descend upon a village in feudal China, where a

humble blacksmith...

The Man With The Iron Fists (2012)

Cancel Iron Man

Movies Search Social Account Nearby

Done Iron Man


Mobile Experience

Agenda







Sourcing your documents

Retrieval and ranking

Search user interface

Performance and scale

Developer example

Amazon CloudSearch Documents

• Unique identifier

• Version

• Fields – Indexed according to configuration

– Source of matches





Developer example

Amazon RDS

Application Content

Movie data

Theater data

User reviews,

lists etc.

DynamoDB

User actions

Amazon S3

Help files

Media (clips,

images)

Articles

Bootstrap Strategy

Source

System

Processing

Script

Queuing Batching

Amazon EC2

Amazon EC2

Amazon

CloudSearch

Amazon SQS





Developer example

Document Construction

• One source will be the master

for each record

determine doc id and version

create fields

for each auxiliary source

gather additional data

send or queue the document





Developer example





Developer example Relational DB

Movie

Title

Description

TheaterID

Theater

Name

AddressesID

ShowtimesID

Addresses

Street

City

State

Showtimes

Date

Time

State

Amazon S3

• Clips, images, reviews

• Apache Tika to extract content

• Amazon S3 Metadata for additional fields





Developer example

Amazon DynamoDB

DynamoDB CloudSearch

Table Domain

Item DocumentAttribute Field

Attribute

Attribute

Attribute

Field

Field

Field





Developer example

When wealthy industrialist Tony Stark is forced




Iron Man (2008)




Iron Man 2 (2010)




Iron Man 3 (2013)






Cancel Iron Man


Done Iron Man






Developer example

Searching Show Times

id title description t_name t_street date time

1 Iron

Man

... Galaxy Main 11/1

1

12:30pm

2 Iron

Man


1

1:15pm

3 Iron

Man


1

2:45pm

4 Iron

Man


1

6:00pm





Developer example





Developer example Heterogeneous Data

Multi Domain





Developer example

Updating CloudSearch

Amazon EC2 Amazon

CloudSearch

Amazon SQS Amazon EC2

Amazon S3 Amazon

DynamoDB

Amazon RDS

Web Server

Users

Update Processor





Developer example

Section Summary

• Multiple sources

• Bootstrap / Update

• Heterogeneous data





Developer example

Agenda











Developer example

Correct Matches When wealthy industrialist Tony Stark is forced




Iron Man (2008)




Iron Man 2 (2010)




Iron Man 3 (2013)






Cancel Iron Man


The Search Algorithm

• Locate documents that satisfy Boolean

constraints – Usually intersection

• Relevance rank those documents – Differentiated from databases by relevance





Developer example

Document Structure

Movie

title

description

user_rating

likes

release_date

latitude

longitude





Developer example

Configuring for Search

• Text fields for individual word search – User-generated and external text – titles, descriptions

• Literal fields for exact matches – Application-generated text like facets

• Integer fields for range searching and ranking





Developer example

Searching Text

http(s)://<endpoint>/2011-02-01/search?

• Simple searches – q=<text>

• Filtering – bq= (and title:'iron man' genre:'Action')

• Filtering with integer ranges – bq=(and 'iron man' year:..2010)

• Geo filtering – bq=(and 'iron man' latitude:12700..12900 longitude:5700..5800)





Developer example

Search Results

{"rank": "-text_relevance", "match-expr": "(label 'iron man')", "hits": { "found": 204, "start": 0, "hit": [ { "id": "sontsst12cf5f88b42" }, { "id": "sopvopr12ab017f082" }, { "id": "sorzrpw12ac468a13b" }, ] }, ... }





Developer example

Relevant Results When wealthy industrialist Tony Stark is forced




Iron Man (2008)




Iron Man 2 (2010)




Iron Man 3 (2013)






Cancel Iron Man


Customizing Ranking

• text_relevance and cs.text_relevance

• Rank expressions – Compute a score for each document

– &rank=<function>

• Defined in the console

• Defined at query-time – &q='iron-man'&rank-recency=text_relevance + year

&rank=recency





Developer example

Field Weighting

Field Weighting

• Adjust relative importance of fields

• &rank-title=

cs.text_relevance({"weights":{"title":4.0},

"default_weight":1})





Developer example

Popularity

Popularity

• Convert floating point to integer

• Weight by the number of ranks

• rank-pop=text_relevance +

(user-rating - 2) * log10(number-user-ranks) * 10

+ metascore * 3





Developer example

Freshness

Freshness

• Exponential decay function

• &rank-decay=text_relevance + 200*Math.exp(-

0.1*days_ago)

r = ce-lt





Developer example

Done Iron Man


Location Sort

Location Sort

• Latitude and longitude

expressed as integers

• Denormalized for particular

theaters with locations

Movie

title

description

user_rating

likes

release_date

latitude

longitude





Developer example

Location Sort

• Cartesian distance function

• &rank-geo=sqrt(pow(latitude - lat, 2) +

pow(longitude - lon, 2)

• &rank=-geo

(lat - latuser )2 + (lon- lonuser )

2





Developer example

Rank Expressions: Combined

• &rank-combined=text_relevance + 2.0 * geo +

0.5 * popularity + 0.3 * freshness

• &rank=combined





Developer example

Section Summary

• Search API basics

• Customizing ranking – Field weighting, popularity, freshness, GEO, combined

• Rank expression comparison tool





Developer example

Agenda











Developer example

Facets

Simple Faceting: Document

Movie

title

description

genre





Developer example

Simple Faceting: Configuration





Developer example

Simple Faceting: Query

q=iron+man&facet=genre

{"rank": "-text_relevance", "match-expr": "(label 'star wars')", "hits": {"found": 7, "start": 0, "hit": [] }, "facets": { "genre": { "constraints": [ {"value": "Family", "count": 62}, {"value": "Action/Adventure", "count": 21}, {"value": "Drama", "count": 5 },





Developer example

Simple Faceting: UI <div class='facet'>

<ul class='facet_list'>

<?php

$genres = $resultsObj->facets->genre->constraints;

for ($i = 0; $i < count($genres); $i++) {

$curGenre = $genres[$i]; $curCount = $thisGenre->count;

?>

<li class='facet_item'>

<div class='facet_name'><?=$curGenre?></div>

<div class='facet_count'><?=$curCount?></div>

</li>

<?php } ?>

</ul>

</div>





Developer example

Facets

Document

• title: Lincoln

• description: ...

• oscar1: Awards

• oscar2: Awards/Best Actor

• oscar3: Awards/Best

Actor/Daniel Day Lewis

Movie

title

description

oscar1

oscar2

oscar3





Developer example

Query

&q=lincoln&facet=oscar1,oscar2,oscar3 {"rank": "-text_relevance", "hits":{...}, "facets": { "oscar1": { "constraints": [ {"value": "Awards", "count": 23}, {"value": "Nominations", "count": 124}]}, "oscar2": { "constraints": [ {"value": "Awards/Best Actor", "count": 6}, {"value": "Awards/Best Actress", "count": 3}...]},

"oscar3": { "constraints": [ {"value": "Awards/Best Actor/Daniel Day Lewis", "count": 1}, {"value": "Awards/Best Actor/Denzel Washington", "count": 2}...]},





Developer example

Drilldown

• bq=oscar1:'Awards'

• bq=oscar2:'Awards/Best Actor'

• bq=oscar3:'Awards/Best Actor/Daniel Day Lewis'

• bq=(and 'star' oscar2:'Awards/Best Actor')





Developer example

Section Summary

• Simple faceting

• Hierarchical faceting

• Hierarchical data handling





Developer example

Agenda







The Search Algorithm

• Locate documents that satisfy Boolean

constraints – Usually intersection

• Relevance rank those documents – Differentiated from databases by relevance





Developer example

Performance Best Practices

• Match set size

• Text queries perform better than integer queries

• Complex relevance functions





Developer example

Optimizing Index Size

• Trade off literal and uint for cost/performance

• Result fields matter most

• Enabling faceting increases size





Developer example

Wrap Up

• Sourcing documents from various locations

• Building queries and ranking

• UI Components for faceting

• Getting the most out of your index

Peter Simpkin





Developer example

Solution Architect, Elsevier

Agenda

• Elsevier Intro

• Search Problem Statement

• Enterprise Content Search

• Hints and Tips

• Amazon CloudSearch Observations

• 7,000+ employees in 26 countries

• 2,200 journals / article market

share 25%

• $3B revenue

• Scientific, Technical & Medical

Customers Products Academic

Research

Institutions

Government &

Health

Corporate

Research

Labs

Individual

Researchers

http://images.google.com/imgres?imgurl=http://www.seattleu.edu/scieng/engpc/stdntprj/ME051/Boeing_logo_r.gif&imgrefurl=http://www.seattleu.edu/scieng/engpc/stdntprj/ME051/&h=148&w=200&sz=4&tbnid=zQPbyxsTqCoJ:&tbnh=73&tbnw=99&hl=en&start=3&prev=/images?q=boeing+logo&svnum=10&hl=en&lr=&safe=off&rls=GGLD,GGLD:2005-14,GGLD:en

http://images.google.com/imgres?imgurl=http://www.3rdapssam.com/images/pfizer logo.jpg&imgrefurl=http://www.3rdapssam.com/sponsors.php&h=908&w=1550&sz=356&tbnid=Fw2wI_whQI4J:&tbnh=87&tbnw=150&hl=en&start=2&prev=/images?q=pfizer+logo&svnum=10&hl=en&lr=&safe=off&rls=GGLD,GGLD:2005-14,GGLD:en&sa=N

http://images.google.com/imgres?imgurl=http://www.cluster.ncat.edu/GE_LOGO.gif&imgrefurl=http://www.cluster.ncat.edu/roster_of_members.htm&h=732&w=735&sz=23&tbnid=g9K8GGTpY3IJ:&tbnh=138&tbnw=139&prev=/images?q=ge+logo&hl=en&lr=&oi=imagesr&start=1

Content Systems

Content Challenges:

• No central place for consumers

to discover content

• Is not currently possible to

search and retrieve atomic

assets

• Assets are not reusable across

products Consumer Platforms

Enterprise Content Search Engine

Search Opportunities:

• Create a comprehensive

inventory to discover easily

content Elsevier owns

• Provide access to Granular /

Modular content they want at

will

• Assets must be uniquely

addressable

Empower our product development partners

Enterprise Content Search eco-system

Federated Content Warehouse Product Platform Data center

E.U Corporate Data center

U.S Corporate Data center

Amazon S3 Amazon

DynamoDB

Amazon SWF Amazon

CloudSearch SDF metadata

Simple Search UI

Elsevier Technical Drivers & Approach

• Fully-managed, full featured search service in

the cloud

• Automatically scales for data & traffic

• Easy to set up and use

• PoC created in days

• Search engine as a service

• Pay-as-you-go pricing model

Hints & Tips (and issn:'0022-1694'

(and type:'1.2'

(and (not action:'D')

(or (and pubstartdate:..2013176 pubenddate:2005002..)

(or (and pubstartdate:2005001

(and pubstarttime:0.. pubstarttime:..235959))

(or (and pubstartdate:2013177 pubstarttime:..235959)

(or (and pubenddate:2005001 pubendtime:0..)

(and pubenddate:2013177

(and pubendtime:..235959 pubendtime:0..)))))))))

• Query Response Time = 5 seconds

Optimising Nested Queries (and issn:'0022-1694' type:'1.2'

(not action:'D')

(or (and pubstartdate:..2013176 pubenddate:2005002..)

(and pubstartdate:2005001 pubstarttime:0..235959)

(and pubstartdate:2013177 pubstarttime:0..235959)

(and pubenddate:2005001 pubendtime:0..)

(and pubenddate:2013177 pubendtime:0..235959)))

• Response Time = 2.5 seconds

Optimised Nested Query ((not action:'D')

(or (and issn:'0022-1694' and type‘1.2'

and pubstartdate:..2013176 pubenddate:2005002..)

(and issn:'0022-1694' and type‘1.2'

and pubstartdate:2005001 pubstarttime:0..235959)


and pubstartdate:2013177 pubstarttime:0..235959)


and pubenddate:2005001 pubendtime:0..)


and pubenddate:2013177 pubendtime:0..235959)))

• Response Time = 0.17ms

Amazon CloudSearch Observations

facilitates knowledge sharing on content matters across Elsevier’s product platforms

ability to leverage content infrastructure and capabilities across Elsevier’s divisions

easy to integrate with existing on-premise content systems

speed to market, allows developers to focus building other core content strategy components

need to spend time optimising queries to maximise performance

Resources

• Amazon CloudSearch Overview Page http://aws.amazon.com/cloudsearch/

– Developer Guide

– FAQs, Articles

– Community Forum

– Tutorial

• Free 30-day trial

• Contact: [email protected]

http://aws.amazon.com/cloudsearch/

Please give us your feedback on this

presentation

As a thank you, we will select prize

winners daily for completed surveys!

SVC302

enrich search user experience using amazon cloudsearch (svc302) | aws re:invent 2013

Technology

actordaniel

formidable

installed

satisfy boolean

intersection

relevance

tony stark

scale developer