experts and accuracy

EXPERTS AND ACCURACY

Jim Joyce

Department of Philosophy

The University of Michigan

[email protected]

EXPERT NORMS

Some evidential norms are expressed as expert principles.

Principal Principle: Your credence in H should be your best

estimate of H’s current objective chance (provided you have

no “inadmissible” evidence).

c(H | CHANCEnow(H) = x) = x

Equivalently, c(H) = ∫ 𝒄1

0(CHANCEnow(H) = x)x dx

SOME NORMS OF EPISTEMIC DEFERENCE

Principal Principle. One should defer to the current CHANCES

as an expert (ceteris paribus)

Reflection Principle. One should defer to ONE’s FUTURE SELF

as an expert (ceteris paribus)

Alethic Principle. One should defer to TRUTH as an expert.

Note the ceteris paribus clauses!

TWO QUESTIONS

What is involved in treating a potential expert as an expert?

Are there potential experts that should be treated as experts?

I will focus mainly on the first question, but will have a little to

say about the second at the very end.

GOAL FOR TODAY’S TALK

I plan to explain the normative force of expert principles like PP

by showing that aligning one’s credences with those of an expert

promotes expected credal accuracy.

Expert credences can be used as proxies or surrogates for

Truth. Aligning one’s credences with those of an expert is as

an effective way of holding credences that approximate Truth.

Plan: (a) The basics of expert norms (brief)

(b) The basics of accuracy measurement (brief)

(c) Divergence from experts can be used as a as proxy

for divergence from Truth

THE SET-UP

W is a set of “possible worlds.”

H = {H1, H2,…, HN} is a partition of hypotheses (each a set of

possible worlds fromW).

Q, R will denote random variables that map elements of W to

coherent credence functions over H.

o These are potential experts (about the Hn).

THE SET-UP

c is a coherent ceredence function defined over propositions of

form [Hn & Q S], for Q a potential expert and S a set of

coherent credence functions defined over H.

o Q = q means that Q has q as its value.

o Q(Hm) = x means that Q(Hm) S where S is the set of all

coherent credence functions that assign credence x to Hm.

SIMPLIFYING ASSUMPTIONS (I think)

All potential experts are “data base” experts for c (as well as

“analyst” experts) so that c(E) = 1 implies q(E) = 1 for all q

with c(Q = q) > 0.

A potential expert Q knows everything c knows.

Each potential expert has only finitely many possible values,

so that c(Q = q) > 0 for only finitely many q.

Learn Q(H1) = 0.2, then Q = q1 or Q = q2

DEFERENCE TO AN EXPERT

Q is an expert for c if c (completely) defers to Q about the

credences over H.

c(H | Q = q) = q(H), H.

Equivalently, c(H | Q(H) = x) = x, x, H.

This entails that c’s credence for H is its expectation of Q’s

credence for H.

c(H) = q c(Q(H) = q)q(H)

EMULATION OF EXPERTS

If you defer to Q as an expert, you align your credences with

your expectations of Q’s credences, and your expectations are

your expectations of the Q’s expectations.

Key Fact: If f is any real-valued function of the Hn, then c’s

estimate of f is c’s estimate of Q’s estimate of f.

Expc(ExpQ(f )) = q c(Q = q)n q(Hn)f(Hn)

= q c(Q = q)n c(Hn | Q = q)f(Hn)

= n [q c(Hn & Q = q)]f(Hn)

= n c(Hn)f(Hn) = Expc(f )

AGREEMENT AMONG EXPERTS

If c defers to Q and R as experts, then c’s expectation of their

credences must agree for all H:

c(H) = q c(Q(H) = q)q(H) = r c(R(H) = r)r(H)

Even though your experts agree about H in expectation, you

recognize that they can disagree in their value assignments.

o You might even know they will disagree, and yet they

can still agree in expectation!

DISAGREEMENT AMONG EXPERTS

What should your updated credence be if you learn Q(H) = x

and R(H) = y where x y?

The answer depends on which expert carries most epistemic

weight with you.

Definition: Q overrides R about H for c just when

c(H | R(H) = r & Q(H) = q) = q(H), H, x, y

Why would one expert carry more weight than another?

Hold that thought!

INADMISSIBLE EVIDENCE

Any expert norm that tells you to align your credences with

your expectations of R’s expectations comes with a ceteris

paribus clause

provided you do not possess any ‘inadmissible’ evidence

about an expert Q of higher rank that is not be screened

off by knowledge of R’s value.

o I see this as a definition of admissibility.

SCREENING CONDITION

You have information about Q that undermines R’s expert status

when

x ∫ 𝒄1

0(Q(H) = y | R(H) = x)y dy, x, y.

Roughly, you take yourself to know more about Q’s values

than what is indicated about them by R.

o Example: Q is the reclusive Queen and R is her spokesman.

If all you know about Q’s opinions comes to you via R, then

R can be an expert for you despite ranking below the Queen.

THE PECKING ORDER

There is a “pecking order” among experts.

Truth overrides everything: c(H | R(H) = r & H) = 1

c(H | R(H) = r & ~H) = 0

Later chances override earlier chances.

o What else can be usefully said here?

This is a (very) partial order. Often neither Q nor R overrides

the other.

TWO QUESTIONS

When will c regard Q as an expert?

What accounts for the pecking order?

Considerations of credal accuracy answer both questions!

EPISTEMIC SCORING RULES

An epistemic scoring rule is a real-valued function Iv(c) that

expresses the inaccuracy of holding credences c in world where

the truth-values for the Hn are as described by v.

Iv(c) 0, with a perfect score of Iv(c) = 0 iff c = v on H.

Think of Iv(c) as a “penalty” for having inaccurate credences.

We will be interested in accuracy over H, and In gives the

inaccuracy of credence functions on H when Hn is true.

PROPERTIES OF SCORING RULES

TRUTH-DIRECTEDNESS. If all c’s values are closer to the truth

than b’s are, then c earns a strictly lower (= better) score.

EXTENSIONALITY. Iv(c) is a function solely of the credences

that c assigns and the truth-values that v assigns.

CONTINUITY. Iv(c) is a continuous function of c’s values.

STRICT PROPRIETY. Each coherent credence function c uniquely

minimizes expected inaccuracy when expectations are taken

relative to c itself:

Expc(I(b)) = v c(v)Iv(b) > v c(v)Iw(c) = Expc(I(c))

TWO PLEASING FACTS ABOUT EXPERTS AND ACCURACY

We regard Q as an expert only if we expect that learning its

value will increase our accuracy: Expc(|Q = q)(I(c) – I(q)) > 0

when q c.

If Q overrides R for c then c’s estimate of R’s inaccuracy

exceeds c’s estimate of Q’s inaccuracy.

o c’s estimate of R’s inaccuracy is

Expc(I(R)) = r c(R = r)n c(Hn | R = r)In(r)

= r c(R = r)n r(Hn)In(r)

= r c(R = r)M(r) r’s “modesty”

Note: The converses of these do not hold. Nor should they.

EXPERT CREDENCES AS PROXIES FOR TRUTH

We can show that aligning credences as closely as possible

with those of an expert serves a kind of proxy or surrogate for

aligning credences with Truth (or any higher-ranking expert).

If you cannot emulate God, emulate a saint!

DIVERGENCE FROM A CREDENCE FUNCTION

Given two probability functions b and q defined on H the I-

divergence of b from q is the amount by which q expects b to be

less accurate than itself:

Div(b, q) = Expq(I(b)) – Expq(I(q)) > 0 by SP.

Expq(I(q)) is q’s modesty M(q). higher = more modesty

DIVERGENCE FROM AN EXPERT

Div(b, Q) is b’s divergence from Q. This is a random variable

with values Div(b, q) = Expq(I(b)) – M(q), for variable q.

Div(b, Q) measures how well b does, according to Q, at

approximating Q.

TWO NICE FEATURES OF Div(b, Q)

1. If you know Q’s value is q, so that c(Q = q) = 1, you will be

certain that Div(b, Q) is minimized at b = q.

2. If Q = @ then Div(b, @) is a random variable with values

Exp@(I(b) – I(v)) = Iv(b)

For any truth-value assignment v, Div(b, v) = Iv(c), i.e.,

inaccuracy = divergence from truth.

EXPECTED DIVERGENCE FROM AN EXPERT

If Q is an expert for you, then your estimate of the degree to

which an arbitrary credal state b accurately approximates Q

should be your expectation of the divergence of b from Q.

ExpcDiv(b, Q)

To the extent that you are striving to emulate Q you should be

aiming to have credences that minimize this quantity.

We will see that ExpcDiv(b, Q) > ExpcDiv(c, Q) when b c, which

means a coherent believer who defers to Q always (immodestly)

expect herself to be doing better that anyone else at emulating Q.

THE MASTER FORMULA

Degree to which c expects b to diverge from Q

ExpcDivI(b, Q)

= Expc(I(b)) – q c(Q = q)M(q)

c’s estimate of b’s accuracy. c’s estimate of Q’s modesty.

Minimized at b = c, where it Less than c’s modesty.

equals c’s modesty. Not a function of b!

Not a function of Q’s value!

DIFFERENCE IN EXPECTED DIVERGENCE FROM AN EXPERT

= DIFFERENCE IN EXPECTED DIVERGENCE FROM TRUTH

ExpcDivI(b1, Q) – ExpcDivI(b2, Q) = Expc(I(b1)) – Expc(I(b2))

When Q is an expert for c, c expects b1 to be inferior to b2

as an approximation of Q to the same degree as c expects b1

to be inferior to b2 as an approximation of Truth.

When Q is an expert for you, seeking to align your credences

with Q’s values is tantamount, in your estimation, to seeking

to align them with Truth (or any expert that overrides Q).

o This explains why you should set your credence for H to

q(H) when you learn Q = q.

THE q c(Q = q)M(q) TERM

If Q and R are experts for c, and Q overrides R, then c’s

estimate of R’s modesty exceeds c’s estimate of Q’s

modesty (proof of “the 2nd pleasing fact”).

Expected modesty is a kind of measure of departure from

Truth for expert random variables: the higher Q’s modesty

the less well Q approximates Truth.

For Truth, v c(@ = v)M(v) = 0 since M(v) = 0 for every

truth-value assignment.

TAKE-HOME POINTS

Minimizing expected divergence from expert Q is tantamount

to minimizing expected inaccuracy!

Divergence from an expert can be used as a proxy or surrogate

for divergence from Truth.

Q-PROXY. If c expects b1 to diverge less from Q than b2 does,

then c also expects b1 to diverge less from Truth than b2 does,

and by the same amount.

So, the policy of minimizing expected divergence from an

expert will also minimize expected inaccuracy!

WHEN SHOULD WE DEFER?

What makes potential expert worthy of expert status?

I have no general answer, but there must be a tie to truth.

o There is such a tie with chance: barring complications (of

inadmissible evidence) chance manifests itself in frequency

of truth.

The Master Equation: ExpcDiv(b, Q) = Expc(I(b)) – Expc(M(Q))

ExpcDiv(b, Q) = q c(Q = q)Expq(I(b) – I(q)) (> 0 if b q)

= q c(Q = q)n q(Hn)(In(b) – In(q))

= q c(Q = q)n c(Hn | Q = q)(In(b) – In(q))

= q n c(Hn & Q = q)(In(b) – In(q))

= q n c(Hn & Q = q)In(b) – q n c(Hn & Q = q)In(q)

= n c(Hn)In(b) – q n c(Q = q)q(Hn) In(q)

= n c(Hn)In(b) – q c(Q = q)n q(Hn) In(q)

= Expc(I(b)) – q c(Q = q)M(q)

= Expc(I(b)) – Expc(M(Q))

2nd Pleasing Fact: If Q and R are both experts for c, then Q overrides R for

c only if c’s estimate of R’s inaccuracy exceeds c’s estimate of Q’s

inaccuracy: Expc(I(R)) > Expc(I(Q)).

Expc(I(R)) = n,r c(Hn & R = r)In(r)

= n,r,q c(Hn & R = r & Q = q)In(r)

= r,q c(R = r & Q = q)n c(Hn | R = r & Q = q)In(r)

= r,q c(R = r & Q = q)n q(Hn)In(r)

= r,q c(R = r & Q = q)n q(Hn)In(r)

Expc(I(Q)) = n,q c(Hn & Q = q)In(q)

= n,q,r c(Hn & Q = q & R = r)In(q)

= q,r c(Q = q & R = r)n c(Hn | Q = q & R = r)In(q)

= q,r c(Q = q & R = r)n q(Hn)In(q)

By Propriety, n q(Hn)In(r) > n q(Hn)In(q) for every q and r. From this the

desired result follows.

experts and accuracy

Documents