Covariance and Correlation

CS 70, Summer 2019

Lecture 22, 7/31/19

1 / 27

Last Time...

I Variance measures deviation from mean

I Variance is additive for independent RVs

I Use linearity of expectation and indicatorvariables

Today:

I Proof that var. is additive for ind. RVs

I Talk about covariance and correlation

I Some RV practice (if time)2 / 27

Product of RVsLet X be a RV with values in A.Let Y be a RV with values in B.

What does the distribution of XY look like?

What if X , Y are independent?

3 / 27

xx -. Yawning

:#" " ' III"*aia

PEX = a ,Y=bI=P[ X -

- a) Pff b)will treat things like

f- 1,4=2 vs 7=1,

X -- 2 VAEA ,

BE B

as 2 separate cases .

For Independent RVs...If X and Y are independent, we can show that:

E[XY ] = E[X ] · E[Y ]

4 / 27

¢ Definition

I RHI Efx I EH ] = face,

A PCX -

- a ] )(ftp.b.PEF-bD

Distribute --

¥⇒.

Cable "

a

.by?bybyinde#=?afcab7.lPCX=a,Y=b3=fECXY

] tsee pre v slide ,

when

Xy 's dist.

defined .

Variance is Additive for Ind. RVsIf X and Y are independent, we can show that:

Var(X + Y ) = Var[X ] + Var[Y ]

5 / 27

⇐Efcxty )

2 ] - CECXTYIJZ all Efx ]

- Tin M2 ELY ]

E[X2t2XYtY2 ] - fifty , ]'

#

ECXZ ]t2E¥tECY2] - Mf-2¥ - UE

cancel b/c ECXYJ -- ECXJECY ]

Var tvarcy ) *for multiple RVs :

induction ! !

Can You Multiply?If X and Y are independent, then:

E[XY ] =

Is the converse true?

6 / 27

ECXTECY ]EAT

Nd . =o . Etc - DE -11147

410,17ELY ]=O .

st.is/!-.miY-" ""TEE÷÷÷÷:* ¥ :na÷

PEX = 0,4=0 ] # Paco ]p[y=o ]

CovarianceMeasures “how independent” two RVs are. LetE[X ] = µ1, and let E[Y ] = µ2.

Cov(X ,Y ) =

Alternate Form:

Cov(X ,Y ) =

7 / 27

fit.

COVCX.yf-OEGX-u.KY-u.IT

€xyf÷J"¥M,ECYI-M.tk

ECCX - it , Ky - up ] = IE EXT -④ Max t Milk ]

Lin .

→ = IECXYI - ¥- ii.Mita= ECXY ] - Milk

Example: Coin Flips II flip two fair coins.Let X count the number of heads, and let Y bean indicator for the first coin being a head.

First, what is XY ?

8 / 27

Wptz

X -- {& WPIFTT

XY={I ¥ IIF"

rain 't'¥*np¥¥¥" '

Hanzo.ie

'

I' i.III.

¥=3

y .. I : IPp¥tIII.In 4

ECXYIFECXIEEDIEEYI - I ⇒ x ,y not ind !!

( contrapositive )

Example: Coin Flips IWhat is Cov(X ,Y )?

9 / 27

EIXYI - Efx ) EH )-

If - ICE)

④

Properties of Covariance IIf X and Y are independent, then:

Cov(X ,Y ) =

Is the converse true?

10 / 27

O because

No .

ECXYI -- ECXIEED

=

Same counterexample

÷as " "

Properties of Covariance IIWhat happens when we take Cov(X ,X )?

Can use either definition of covariance!

11 / 27

if ECX ]

① Cov Cx , x K EKX - UK X - U ) ]= Efc X - u )2) = Var CX)

② covcx,

X ) -

- E CX Cx ) ) - ECXIECXI= Var CX)

Properties of Covariance IIICovariance is bilinear.

Cov(a1X1 + a2X2,Y ) =

Cov(X , b1Y1 + b2Y2) =

12 / 27

A ,CoV C Xi , Y ) t Az for ( Xz ,

Y )

b.for I X , Yi ) tbz COV ( X, Yz )

④ var C CX ) = Cov ( CX , CX ) = c Cove x. ex )= Cz Cov C X ,

X ).

Practice: Bilinearity!

Simplify Cov(3X + 4Y , 5X � 2Y ).

13 / 27

fist= z Cov ( x , SX - 2 'D -14 COVEY , 5 X - 2X )words

= 15 COV C xx ) - 6 Cov CX , Y )

1- 20 Cov CY ,X ) - 8 ANY

, 'D

=

15 Var Cx) -114COV CX , Y ) - 8 Var l Y )

Example: Coin Flips IISame setup: Two fair flips. X is number ofheads, and Y is indicator for the first coin heads.

Recall: Cov(X ,Y ) =

Now, let Y 0 be an indicator for the first coin beinga tail. How does the covariance change?

14 / 27

1-

4

Cov C x. Y ' I = COVCX , I - Y )Observe : Y' TY -

. I = covcx, 1) - COVCX , 'D

Y '-

-

t-Y.EE#fECxI.EGI-TT

=

Properties of Covariance IV

For any two RVs X , Y :

Var(X + Y ) =

15 / 27

Cov ( Xt Y ,Xty )

= Cov ( X , Xty ) + COVEY,

Xt Y)

= Cov ( X , x ) t Cov C Xi ) t COV CY,X ) TCOVCY ,'D

= vortex ,T

-12Eovcx, 'D t var #

Break

If you could eliminate one food so that no onewould eat it ever again, what would you pick todestroy?

16 / 27

Correlation

For any two RVs X , Y that are not constant:

Corr(X ,Y ) =

Sanity Check!

What is Corr(X ,X )?

What is Corr(X ,�X )?

What is Corr(X ,Y ) for X ,Y independent?

17 / 27

soamnion o

Cov ( X , Y )

¥Y)←

Std der

varixtarty )

( Farcz

=L

- 1.

O

Example: Coin Flips III flip two fair coins.Let X count the number of heads, and let Y bean indicator for the first coin being a head.

What is Corr(X ,Y )?

18 / 27

µ

↳ =

covcx.ygttxnBincz.EE#Y-ynBerCtzyvarCX7=npCi-pl

vary ) -

- pet - p )

=L = 49¥Ez 041=2

corrcx.tt#zI.= E

Size of Correlation?For any RVs X and Y that are not constants:

�1 Corr(X ,Y ) 1

Proof: Define new RVs using X and Y :

X =Y =

19 / 27

Size of Correlation?(Continued:)

20 / 27

RV Practice: Two RoadsThere are two paths from Soda to VLSB.I usually choose a path uniformly at random.# minutes spent on Path 1 is a Geometric(p1) RV.# minutes spent on Path 2 is a Geometric(p2) RV.

Today, it took me 6 minutes to walk from Sodato VLSB. Given this, what is the probability that Ichose Path 1?

21 / 27

- comin T ,-_ time path 1

path 't n Geomlpi )

¥ Tz -_ time path 2\ - comin .

rcaeomcpz )+ path 's

2 W ,-_ event I chose

path 't

RV Practice: Two RoadsContinued:

22 / 27

conditionalprob .

God,

Ipfw , 16min ] =

"XM=# d def .

IPC 6 min ]

=(E) ( I -pp,→ from

non .

PICT.= it - H - pimp

.fi#EEtiIDdtftEuie=

ICI - p , Isp ,

I- p , 75pct# - 17275 Pz

RV Practice: RandomSortI have cards labeled 1, 2, . . . , n. They are shu✏ed.

I want them in order. I sort them in a naive way.I start with all cards in an “unsorted” pile. I drawcards from the unsorted pile uniformly at randomuntil I get card 1. I place card 1 in a “sorted”pile, and continue, this time looking for card 2.

23 / 27

goaertd I 9,etard2IsT

,2

re a

What are these deists ?

RV Practice: RandomSortWhat is the expected number of draws I need?

What is the variance of the number of draws?

24 / 27

RV Practice: PacketsPackets arrive from sources A and B.I fix a time interval. Over this interval, the numberof packets from A and B have Poisson(�A) andPoisson(�B) distributions, and are independent.

What is the distribution of the total number ofpackets I receive in this time interval?

25 / 27

RV Practice: PacketsWhat is the probability that over this interval, Ireceive exactly 2 packets?

What is the expected number of packets I receiveover this interval?

What is the variance of this number?

26 / 27

Summary

I Covariance and correlation measure howindependent two RVs are.

I Variance can be expressed and manipulatedin terms of covariance.

I Independent RVs have zero covariance andzero correlation. However, the converse isnot true!

27 / 27