1

Only Valuable Experts Can be

ValuedMoshe Babaioff, Microsoft Research, Silicon Valley

Liad Blumrosen, Hebrew U, Dept. of Economics

Nicolas Lambert, Stanford GSB

Omer Reingold, Microsoft Research, Silicon Valley and Weizmann.

2

Probabilities of Events

• Often, estimating probabilities of future events is important.

• Examples:– Weather: probability of rain

tomorrow

– Online advertising: what is the click probability of the next visitor on our web-site?

– What % of Toyota cars is defective?

– Many applications in financial markets.

3

Contracts and Screening

Uncertain about the probability of a future event.

Claims he knows this probability.

Averse to uncertainty, is willing to pay $$$ to reduce it.

May be uninformedand pretend to be informed to get $$$…

A decision maker (Alice)

An expert (Bob)

4

Contracts and Screening

• Goal: screen experts.

• That is, design contracts such that:

– Informed experts will:1. accept the contract2. reveal the true probability.

– Uninformed experts will reject the contract.

• Contracts can be based on outcomes only: True probabilities are never revealed.

5

This work

• We characterize settings where Alice can separate good experts from bad experts.

• We discuss what is a “valuable” expert, and its relation to screening of experts.

6

Outline

• Model

• With prior:– An easy impossibility result– A positive result

• No priors

• Extensions

7

Model (1/4)

• Ω - Finite set of outcomes.• p - A (true) distribution over Ω.

– Unknown to Alice• Φ - The set of possible distributions.

– Φ may be restricted, examples to come…

• Bayesian assumptions:

Prior f on Φ. • f is known to Alice,

Bob.

No Prior on Φ.

In the beginning of this talk Later….

8

Model (2/4)

• Contract: π(q,ω)

payment to Bob when reporting q when outcome is ω.– Bob is risk neutral.

• Bob’s expected payment depends on what he knows:

– Informed: π(q,p) = Eω~p [ π(q,ω) ]

– Uninformed: Ep~f Eω~p [ π(q,ω) ] = π(q,E[p])

Reported probability

Realized outcome

Notation: E[payment] upon reporting q when the true probability is p.

9

Model (3/4)

• Bob δ-accepts the contract if he has a report q with payment > δ– Otherwise, we say that Bob δ-rejects.

• For avoiding handling ties, we aim that for δa> δr :– an informed Bob will δa -accept

– an uninformed Bob will δr –rejectδa

δr

0

10

Model (4/4)

• We actually study a more general model: – experts are ε-informed.– Mixed strategies are allowed.

• This talk: perfectly informed, pure strategies.

11

Example• A binary event: Ω = { , }

Pr( ) = αp Φ = { (α , 1- α) | α [0,1] }

• Alice does not know p.– Knows, however, that α ~ U[0,1] .Sees an a-priori probability of ½.

• Bob claims he knows the realization of p.

• Contract: Bob reports α. Is paid according to or .

12

Main message

• The ability to screen experts closely relates to the structure of Φ– Roughly, on whether Φ is convex or not.

• If all possible experts are “valuable” to Alice, then screening is possible.

13

An Easy Impossibility

• Reason: when the true probability is E[p] a true expert accepts the contract.

Proof:o Informed experts always accept the contract π.That is, for all p, we have π(p,p) > δa.

o Then, an uninformed agent can get > δa by reporting E[p]:Ep~f Eω~p [π(E[p],ω)] =π(E[p],E[p]) > δa

Proposition: screening is impossible.

14

Valuable Experts

• So screening is impossible when E[p] Φ.

• But experts knowing that the true distribution is E[p] are not really valuable to Alice.

• In the binary-outcome example:−Alice’s prior is U[0,1], so she believes

that Pr( ) = ½.−An expert knowing that p=½ is not that

helpful…

What if all experts are “valuable”?

15

Possibility result• Theorem: when E[p] is not in Φ (and Φ is

closed), screening is possible.– When all experts are valuable, we can

screen...

• Immediate questions:−Is non-convex Φ natural?−Can we expect Φ not to contain E[p]?

E[p]

Φ

16

Non-convex Φ: examples

• Example 1: a coin which is either fair (p=1/2) or biased (p=3/4)– For any (non-trivial) prior, E[p] not in Φ.

• Example 2: many standard distributions are not closed under mixing.– E.g., uniform, normal, etc.

1/2 3/4

17

Non-convex Φ: examples

• Example 3: The binary outcome example. But now, Alice observes two samples.– For example, we wish to know the failure rate

in cars, and we thoroughly check 2 random Toyotas.

18

Non-convex Φ: examples

Ω = { ( , ), ( , ), ( , ) , ( , ) }Φ = { ( α2, α(1- α), (1- α)α, (1-α)2 ) } α [0,1]

• Example 3: The binary outcome example.

But now, Alice observes two samples.

For every prior, E[p] is not in Φ, and thus screening is possible.

Φ is not convex!− Moreover, for every p,p’,

the convex combination of the above vectors is not in Φ.

19

Possibility result• Theorem: when E[p] is not in Φ (and Φ is

closed), screening is possible.Φ

• One definition before proceeding to the proof…

20

Scoring rules

• Scoring rules:– Contracts that elicit distributions from

experts.• S(q,ω) = payment for an expert reporting q

when the realized outcome is ω.– A scoring rule is strictly proper if the expert is

always strictly better off by reporting the true distribution.

– Strictly proper scoring rules are known to exist [Brier ‘50, , Good ’52, Savage ‘71,…]

• We want, in addition, to screen good experts from bad.

21

Possibility result• Theorem: when E[p] is not in Φ (and Φ is

closed), screening is possible.Φ• Proof:

• Let s be some strictly proper scoring rule.− On the full probability space

• The following contracts screens experts:

rq

ra qpEsqqs

pEsqsq

)'],[()','(inf

)],[(),(2),(

'

We need to show:1. An informed expert δa-accepts.

2. An uninformed expert δr-rejects.

22

Possibility result• Theorem: when E[p] is not in Φ (and Φ is

closed), screening is possible.Φ• Proof:

• Let s be some strictly proper scoring rule.• On the full probability space

• The following contracts screens experts:

rq

ra qpEsqqs

pEsqsq

)'],[()','(inf

)],[(),(2),(

'

An informed expert reporting the truth p gains:

rrrapp 2),(

≥1

23

Possibility result• Theorem: when E[p] is not in Φ (and Φ is

closed), screening is possible.Φ• Proof:

• Let s be some strictly proper scoring rule.• The following contracts screens experts:

Since s is strictly proper, for every q:

An uninformed expert will gain: rrrapEq 02])[,(

0])[],[(])[,( pEpEspEqs

rq

ra qpEsqqs

pEsqsq

)'],[()','(inf

)],[(),(2),(

'

24

Outline

• Model

• With prior:– An easy impossibility result– A positive result

• No priors

• Extensions

25

Related work

• [Olszewski & Sandroni 2007] studied a similar model:– A binary event with unknown probability p.– No priors:

• An uninformed expert accepts a contract if it is good in the worst case.

• Theorem [O&S]:

– Use min-max theorems.– The probability space is convex.

All informed experts accept a contract

An uninformed expert also accepts it

26

Valuable Experts: No Prior

• We claim: invaluable agents are also behind this impossibility.– But what is a valuable agent without priors?

• What are Alice’s utility function and actions?– A(p) : Alice’s action when she knows p.– U( A(p),p ): utility maximizing actions.

Theorem:Φ is convex (and closed)

There exists p such thatU(A(p),p)=U(A(“reject”),p)no prior on Φ

• Interpretation: if Φ is convex then some informed expert is not valuable.

27

No prior: positive result• We have an analogues positive result for

the no-prior case:non-convex Φ screening is

possible.

• (we use the with-prior positive result in the proof)

28

Outline

• Model

• With prior:– An easy impossibility result– A positive result

• No priors

• Extensions

29

Extension: forecasting• A related line of research is forecasting:

– An unbounded sequence of events.– An expert provides a forecast before

each event occurs.– Goal: test the expert.

30

Forecasting: related work

• Negative results are known:– Informed experts pass the test uninformed

experts can do it too. [e.g., Foster & Vohra ‘97, Fudenberg & Levine ‘99]

– When forecasting is possible, decisions can be delayed arbitrarily. [Olszewski & Sandroni ‘09]

• Some works around this impossibility:– [Olszewski & Sandroni ‘09] show a counter-example

by constructing non-convex set of distributions. – [Al-Najjar & Sandroni & Smorodinsky & Weinstein ‘10]

Describe a class of distribution such that decisions can be made in time. The relevant class of distributions also admits non-convexities.

31

Extension: forecasting• We extend our approach to forecasting

settings.– In the works.

• We characterize conditions on the set of distributions that allow expert testing.

• Analysis is more involved, but the ideas are similar.– Results relate to the convexity of Φ.– For example: two samples at each period enable

testing.

32

Summary• A decision maker want to hire an expert.

– For learning the probability of some future event.– The expert may be a charlatan.

• Can the decision maker separate good experts from bad ones?

• We characterize the settings where such screening is possible.– With or without priors on Φ.

• We design screening contracts.

33

Thanks!