Equivariant Estimation t Singular valve Shrinkage①

Invariant Statistical Models

onside , the model Y - Nnxp I M,

o' OI ) )

µ )rank ( M ) so sp

,O

'>

O.

What us model for I = FYGT,

F c- On,

G c- Op ?

I = f ( M + C) GT = FMGT t FEGT

= Fr t E

check : rankin ) ⇐ r ⇐ ) rank C m ) er

E-Nco ,o

-

⇐OI ) ) ⇐ I E - N ( o,

o'

⇐OI ) )

so the Model to , Y is thesung as the model for I

.

we say that model txt ) is invariant u transformation

Y → FY Gi,

FE On,

GE Op

Invariant Estimation

Y - N ( M,

or2 LIGI ) )

,let in = SLY )

,an estimator of M

rxp

Now suppose you get

Y'

= f Y GT,

F c- On,

Goon known.

Then Y' rn ( m'

,

racial ) ),

su Th'= 8 ( Y

'

) is an est of M"

②So 8 ( FY GT ) estimates FMGT

SLY ) est mutes M

F SLY ) Gt est. nukes FMGT

Principle of Invariance require SIFYGT ) -

- FSH ) GT t foon GoopC- Rn xp

Characterization of equivariant estimates

wt Y -

- UOV'

be Svo of Y.

Then Ys ( UT ) too ) Vi utu -

- O.

it

n xp

Bp equivariant ,

8140118) rt ) = @at soo ) VT

Result An estimate , 8 is equivariant wet Y - FYGT.tt

8 ( Y ) = his f l o ) VT for some function f : Dp → Rn "

further result 8 is equivariant off f :

Dp - LET Je Dp

i.e,

S ( Y ) = U f ( O ) VT when f ! Dp → Dp

singular neuters of A = singular vector of Y.

⇒ only have to Agree out how to transformsay

Vals.

③Proof et O c- Dp

,

and wt 818 ) = I fo ) fear "

,

GenXP .

Now let S =

drug ( s, . . . sp ) Sj c- { - I

,ti }

C E Oh-

p

not : I : :X :p=

Is :p-

-

Is ;]-

-

I :]

so Sc : )=s( C so:] (9) s ),

by equivalence

. c ::] scoots .

. l so:3 LEIS

but soooo , .gg ,

" kiss ]

⇒ F ' SFS for drug . natives of ± I ⇒ F is diagonal←

Cbs-

- G for any octhoy nut.

C, days ⇒ 6=0

←

Summary SLY ) is an equivariant est.

of M iff

g ( Y ) = U Ev "

where E =

dig C DID, . .

idplrl).

Further results It 8 is admissible then d-,

? To ? . - - s Ip

( see,

for example,

Tsu Kuma C 20087 )

④

Orthogonally Equivariant Match Estimation

Model Mt OZ,

Z ~ Nn .pl 0,

IOI )

⇒ Yn Nap M,

o-

IOI ),

o-

so, rank ( M ) sp

Methods runcationlltacdthcesholdy )

SV Shrinkage C soft Thcesholdhy )

Bates C Bic,

full posterior approximation )

Truncation v were known then likelihood is

-2cg p ( YIM ,o' ) = np try o

'

+ MY - MH'

for

MLE = Minimize e of NY - MIP over can M -

r mutuals

in = Ur Devs"

,where Y = UDV

"

Is SVO of Y.

Problem thank r not known.

Solutions,

Li penalization,

Bic, Buyer model selection

,

CV 'new meter or M = FCT

.

Then model to , rank = 8 is

Yij= fit

gj to 2- ij , Zij ~ nd N Loi )

o

"

> u,

{ fi. ..fr/ciR

'

,{ g , . - . gp } cut

Loocv Fo . each valve of r urdu consideration,

do :

⑤

For each lis 's 3,

i.

estimate Ii ,^gj without tij Ii igi GIR"

z .constant Frig = titty;

Then Cuss.

= ?§ ( yij-

Fei;)'

Then pick rank with lowest cuss.

How to do item 2 : Note : data in column j car be writtenj

i

¥÷÷÷i÷÷÷÷÷:÷ti÷÷÷÷

= YI - i ,j ] = Fc - i,

I of j t O Z C . i ,j7

It "

working" valves of Face available

,the cord optimal gj

's

Jj = I if )"

Etf (F -

- Fein , I -. YE . i ,j7 )

Can repeat for each j-

. I . . p,

to get "

woehry Mattia G.

Then proceed to Update F similarly .

Iterate untilconvergence .

⑥

This as an

" alternating least squats"

CALSI algorithm .

The SS is reduced at each iteration .

Very useful fo , bi.

ar models' '

:

Linear Model' Y -

- X B t Z,

1. near in paeans B.

B. lineup Model . B'

t Z

,

1. near in B for fixed A

Ivrea in A for fixed B

⇒ " bilinear' '

Problems with CV

Computational eed to run an iterative agloafhn for

each Lij }

-

candidate rTe

n P' iterative

each algorithms .

Some shortcuts.

"bi - cross validation "

Gower how ) :

"

)←

I

Ynn

obtain low tanh appcux to Mai,

- n via two of YC . i,

- IT

+ Temp duvet te

but I, .

= IT . , Me.,

.is I. ins ( exercise : why does this

⇒ need to do a svo go ,each µjg , !

make sense ?)

Owen Percy l 2009 ) Generalized Bleu - leave out buyer# of cows look foe computational

efficiency .

Problems still. See demo -

generallywant to shrine

singular valves even it they ace teept .

Soft thus hotly

Recall variable selection via LI penalization :

y = Xp to 2-

,Blass

o

=

acy main Ily - Xp 11'

t a EIB; I

Note that lpj I = Cpj )' ' 2

Analogy to SV penalization

Y = M is o Z,

A =

my mainA Y - M 112 + all

. -

MH #

HN H.

= or ( Crim )" '

) = to I govt )" ' )

-

. to ( v OVi

)p

= to L O ) = j§, dj

⇒ A =

acymin HY - MN

'

t a §, ,

dj

⑧Nuclear room

- penalized estmntui ( Mazumder et at 2010,

Chi et at 2010 ) .

let Y -

. UOV'

be NO of Y.

Then

u E v'

,whew Itj = dj xmuxfo ,

I - Taj ).

di

(A w t:Yo%n

?

0 SURE

u 6 SURE ( Josse a Sandy C 20163 )

I° Asymptotic

an dj

Classical Empirical Bates Shrinkage ( Efron t Moeen 1972 ) ⑨

Y/fNnfp0 ,

Ip GI-

) simply model

0 - Nap ( O,

E On In ) pilot

Then 01 Y -

Nap ( Y ( I + E"

)"

,LITE

"

)"

ee I )

f- = Y ( Its" I

"

Matan Identity ( I"

= I - CITE )"

So G- = Y [ I - ⇐t E )"

]

why is this woeful ?suppose we dont have a poor helve of E

.

Considermany .

dust of Y :

Y n

n ( O,

⇐pts ) on In )

Estimate Ipt E with YtYn -

p- I

£eB= Y I I - Y )"

e-

p-y ]

How is this SV shrinkage? let Y = UPV

's

E-EB

= u OUT [ I - VO

-

Zvtcn - p - , ) ]

=U o [ I - o

-

Zen - p- i ) I VT

-

.

u ( o - o- '

( n - p-it TVT ⇒ ~dj= dj

- "dP

④Edm ( 1972 ) showed E.[ HEEB - 0 IT ] ? C- [ HY - 0-15 ]

-0

for all 0.

⇒ ⑤EB is a mining est

. with

Note ! Tj's could bemy

. typically use Tj = o-

( dj- "P )

Note ! 02 assumed Known, equal to 1 !

Fully Bages Model Selection a Shrinkage

f f l 2007 )E- NCO

,o' IGI )

IT - MLK ).

Given K,

Un Unit ( Vu, n

)f

di. . . dn - no Nlm ,

) UDVT t E

V - unit I Vn, p )

Inference via MCML .

Asymptotic opt

.nu/ityCshabuhntNohe1Czoi37

,lavish - Donoho ( 20143 )

Asnfp

→j E ( o

,IT

,an asf . opt est is UJVT while

Tj = ÷ ( ( dj- r-y

'-

4g )" t

x Ifj > C Itf ) )ya

shrink Tomcat

Note o'

assumed Known,

equal to I !-

!