recsys 2016 talk: feature selection for human recommenders

Proprietary and confidential

Proprietary and confidential

Proprietary and confidential

Proprietary and confidential

Human Computation At Stitch Fix

Proprietary and confidential

Heavy and repetitive computation

Large-scale working memory

Large-scale long-term memory

Context sensitivity/nuance

Aesthetic judgements

Relationship building

Novel inferences

Unstructured data

Proprietary and confidential

Processes information to make recommendations

Can specify internal mechanisms

Can specify the data being used

Recommendations improve with better features (data)

Needs to be trained and tuned

Comes with internal mechanisms

Can consider the entire world

Proprietary and confidential

Processes information to make recommendations

Can specify internal mechanisms

Can specify the data being used

Recommendations improve with better features (data)

Needs to be trained and tuned

Comes with internal mechanisms

Can consider the entire world

Proprietary and confidential

Determine what they’re processing

Determine what they should be

processing

Change/shape what they’re processing

Proprietary and confidential

Determine what they’re processing

Determine what they should be

processing

Change/shape what they’re processing

Make more recommendations

Deliver those recommendationsReceive feedback

Proprietary and confidential

1: Determining what they’re processing

Proprietary and confidential

If someone isn’t attending to something, but you’re showing it anyways you might

■ Make your worker less efficient (slower)■ Fatigue them (unnecessary filtering)■ Lose opportunities for including something more useful

Figure out what your human workers are attending to while they make their recommendations

If they aren’t attending to a feature, then they’re not making recommendations off of it

Proprietary and confidential

Exploration

Proprietary and confidential

‘Online’ Observations

What you get

○ Ability to reduce the hypothesis space

○ Higher granularity observations ○ Time-dependent observations

(when is something considered)

Proprietary and confidential

Mouse Tracking

Cheap measure of attention

Non-invasive

Easy widespread deployment

Proprietary and confidential

Eye Tracking

Proprietary and confidential

[Visual] search patterns lie somewhere between random and systematic…. humans will attempt a more systematic search, but will still suffer from imperfect memory.

(Nickles et al., 2003)

Proprietary and confidential

Eye Tracking

Resistant to strategy

Deterministic

Higher accuracy

Proprietary and confidential

AREA OF INTEREST (AOI)

Eye Tracking

Resistant to strategy

Deterministic

Higher accuracy

Proprietary and confidential

Proprietary and confidential

Features You Want To Select!

Proprietary and confidential

2: Determining what they should be

processing

Proprietary and confidential

You’re interested in overall performance and can optimize for whatever is most important to you

■ True hits, false positives, false negatives■ Processing time

Given the features that they’re using, which ones produce the best recommendations?

Proprietary and confidential

The Logic:○ Workers may vary in what features they use○ Look for correlations between attention to features and positive

metrics

Allows you to learn the optimal features amongst your current candidates

Proprietary and confidential

Feature Drop Out Studies

Proprietary and confidential

Feature Drop Out Studies A/B

Proprietary and confidential

Feature Drop Out Studies

Logic

Show a feature to one cell, and remove it for another

If a positive difference in performance is observed, then that feature promotes better outcomes

Proprietary and confidential

Feature Drop Out Studies

Optimal Conditions

A highly controlled “offline” environment

○ Allows for true participant randomization

○ Allows for repeated measures○ Allows for high “internal validity”

Proprietary and confidential

Task-relevantbackground information(optional)

Ability to provide a response - track accuracy, RT, confidence, etc.

Trial-specific stimuli - use historic data with known outcomes

Proprietary and confidential

Correct ~ Condition + (1|participant_id)

Condition differences

Feature promotes better recommendations

Feature either isn’t considered or makes no difference to recommendations if it is

No condition differences

Proprietary and confidential

Further Use Of ‘Online’

Observations

What you get

○ Ability to determine whether there are certain times at which certain features are beneficial

○ Ability to figure out how information is searched for

Proprietary and confidential

-Status: loved-Department: top-Color: purple

-Status: loved-Department: dress-Color: green

-Status: hated-Department: pants-Color: orange

-Status: ...-Department: ...-Color: ...

Start with a study to determine correlations

Proprietary and confidential

Multiple metrics possible

■ Overall trajectories (http://www.eyetracking-r.com/)■ Saccade patterns■ Fixation times and locations

Correlate with success

Proprietary and confidential

correct ~ fixated_on_loves + fixated_on_color_matches + … + (1|participant_id) …

Factors predict success

Attention to features may promote better recommendations

Attention to features may make no difference to recommendations

Factors don’t predict success

Proprietary and confidential

correct ~ condition + … + (1|participant_id)

Follow up with a full experiment to determine whether the behavior

actually causes better recommendations

Manipulation congruent with ‘positive’ behaviors

Proprietary and confidential

3: Shaping What They’re

Processing

Proprietary and confidential

Controlled Lab Study Full A/B Test

Proprietary and confidential

Stitch Fix’s “Styling Lab”

Full A/B Test in the live styling

environment

Proprietary and confidential

Behavior Shaping : Humans :: Tuning : Computers Algorithms

Can be “in the moment”

● UX Changes● Directed Attention

Can be more sustained

● Training

Proprietary and confidential

Change how the information is displayed - exploit human perception (consult UX)

Proprietary and confidential

Testing

● Create questions relevant to what you want to train

● Have participants complete them

● Use IRT to determine question difficulty

Training

● Order questions by difficulty

● Have those being trained complete them in that order

● Given feedback on performance along the way

● Reinforce key concepts

Experimental Approach!

Proprietary and confidential

This approach is grounded in Cognitive research!

Progressive Alignment prescribes giving people tasks that they’re more likely to succeed at, then progressively making those tasks harder

.02

.08

Proprietary and confidential

Processes information to make recommendations

Can specify internal mechanisms

Can specify the data being used

Recommendations improve with better features (data)

Needs to be trained and tuned

Comes with internal mechanisms

Can consider the entire world

Proprietary and confidential

Questions?