Upload File

principle components analysis (pca)rcs46/lectures_2017/10-unsupervise/... · 2017-10-19 ·...

  • Home
  • Documents
  • Principle Components Analysis (PCA)rcs46/lectures_2017/10-unsupervise/... · 2017-10-19 · Bi-plot-0.4 -0.3 -0.2 -0.1 0.0 0.1 0.2-0.4-0.3-0.2-0.1 0.0 0.1 0.2 PC1 PC2 1 2 3 4 5 6
26
Principle Components Analysis (PCA) Rebecca C. Steorts, Duke University STA 325, Chapter 10 ISL 1 / 26

Upload: others

Post on 05-Jul-2020

1 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: Principle Components Analysis (PCA)rcs46/lectures_2017/10-unsupervise/... · 2017-10-19 · Bi-plot-0.4 -0.3 -0.2 -0.1 0.0 0.1 0.2-0.4-0.3-0.2-0.1 0.0 0.1 0.2 PC1 PC2 1 2 3 4 5 6

Principle Components Analysis (PCA)

Rebecca C. Steorts, Duke University

STA 325, Chapter 10 ISL

1 / 26

Page 2: Principle Components Analysis (PCA)rcs46/lectures_2017/10-unsupervise/... · 2017-10-19 · Bi-plot-0.4 -0.3 -0.2 -0.1 0.0 0.1 0.2-0.4-0.3-0.2-0.1 0.0 0.1 0.2 PC1 PC2 1 2 3 4 5 6

Agenda

I Recap of information retrievalI Relation to PCAI Idea behind PCAI Application to NYTimes data setI Multidimensional scalingI Scree plots

2 / 26

Page 3: Principle Components Analysis (PCA)rcs46/lectures_2017/10-unsupervise/... · 2017-10-19 · Bi-plot-0.4 -0.3 -0.2 -0.1 0.0 0.1 0.2-0.4-0.3-0.2-0.1 0.0 0.1 0.2 PC1 PC2 1 2 3 4 5 6

Information retrieval

I Recall that information retrieval systems often representdocuments as what are called bags of words.

I Such documents are represented as vectorsI Each component counts how many times each word in the

dictionary appears in the text.I This throws away information about word order.I Part of the representation of one document might look like:

a abandoned abc ability able about above abroad absorbed absorbing abstract43 0 0 0 0 10 0 0 0 0 1

and so on through to zebra'',zoology’‘, “zygote”, etc. to the endof the dictionary.

3 / 26

Page 4: Principle Components Analysis (PCA)rcs46/lectures_2017/10-unsupervise/... · 2017-10-19 · Bi-plot-0.4 -0.3 -0.2 -0.1 0.0 0.1 0.2-0.4-0.3-0.2-0.1 0.0 0.1 0.2 PC1 PC2 1 2 3 4 5 6

Information retrieval

These bag-of-word vectors have three outstanding properties:

1. Most words do not appear in most documents; thebag-of-words vectors are very sparse (most entries are zero).

2. Small number of words appear many times in almost alldocuments.

I (Examples: “the”, “is”, “of”, “for”, “at”, “a”, “and”, “here”,“was”, etc.)

3. Many words’ counts are correlated with some but not all otherwords.

4 / 26

Page 5: Principle Components Analysis (PCA)rcs46/lectures_2017/10-unsupervise/... · 2017-10-19 · Bi-plot-0.4 -0.3 -0.2 -0.1 0.0 0.1 0.2-0.4-0.3-0.2-0.1 0.0 0.1 0.2 PC1 PC2 1 2 3 4 5 6

Takeaway

I Don’t get much value from keeping around all the words.I Better off projecting down to a smaller number of new

variables.I Project partially since the words mean slightly different things.I This is exactly what principal components analysis does and

it’s very useful as an exploratory data analysis exercise fordimension reduction.

5 / 26

Page 6: Principle Components Analysis (PCA)rcs46/lectures_2017/10-unsupervise/... · 2017-10-19 · Bi-plot-0.4 -0.3 -0.2 -0.1 0.0 0.1 0.2-0.4-0.3-0.2-0.1 0.0 0.1 0.2 PC1 PC2 1 2 3 4 5 6

Principal components analysis (PCA)

I PCA is a tool for exploratory data analaysis and dimensionreduction.

I Take large set of correlated variables and replace with smallernumber that collectively explain most of the variability.

I The PC directions are directions in the feature space alongwhich the original data are high variable.

I PCA is an unsupervised learning tool – since it involves thefeatures X and no response Y

6 / 26

Page 7: Principle Components Analysis (PCA)rcs46/lectures_2017/10-unsupervise/... · 2017-10-19 · Bi-plot-0.4 -0.3 -0.2 -0.1 0.0 0.1 0.2-0.4-0.3-0.2-0.1 0.0 0.1 0.2 PC1 PC2 1 2 3 4 5 6

Why is it useful?

Suppose that we wish to visualize n observations withmeasurements on a set of p features, X1, . . . ,Xp as part of anexploratory data analysis.

We could do this by examining two-dimensional scatterplots of thedata, each of which contains the n observations’ measurements ontwo of the features.

However, there are p = p(p − 1)/2 such scatterplots

For example, with p = 10 there are 45 plots!

PCA seeks to to find a low-dimensional representation of the datathat captures as much of the information as possible.

7 / 26

Page 8: Principle Components Analysis (PCA)rcs46/lectures_2017/10-unsupervise/... · 2017-10-19 · Bi-plot-0.4 -0.3 -0.2 -0.1 0.0 0.1 0.2-0.4-0.3-0.2-0.1 0.0 0.1 0.2 PC1 PC2 1 2 3 4 5 6

PCA

PCA seeks a small number of dimensions that are as interesting aspossible, where the concept of interesting is measured by theamount that the observations vary along each dimension.

Each of the dimensions found by PCA is a linear combination of thep features.

8 / 26

Page 9: Principle Components Analysis (PCA)rcs46/lectures_2017/10-unsupervise/... · 2017-10-19 · Bi-plot-0.4 -0.3 -0.2 -0.1 0.0 0.1 0.2-0.4-0.3-0.2-0.1 0.0 0.1 0.2 PC1 PC2 1 2 3 4 5 6

How many principal components are there?

There are a total number of min(n − 1, p) principal components.

For some reason R prints out the n principal component (and I’mnot sure why it does this)!

9 / 26

Page 10: Principle Components Analysis (PCA)rcs46/lectures_2017/10-unsupervise/... · 2017-10-19 · Bi-plot-0.4 -0.3 -0.2 -0.1 0.0 0.1 0.2-0.4-0.3-0.2-0.1 0.0 0.1 0.2 PC1 PC2 1 2 3 4 5 6

PCA

The first principal components of a set of features X1, . . . ,Xp is thenormalized linear combination of the features

zi1 = φ11X1 + φ21X2 + · · · + φp1Xp =p∑

j=1φi1Xi

that has the largest variance.

By normalized, we mean that∑p

i=1 φ2i1 = 1.

I φ11, . . . , φp1 are the loadings of the first principal component.I Together the loadings make up the the principle components

loading vector φ1 = (φ11, . . . , φp1)T

10 / 26

Page 11: Principle Components Analysis (PCA)rcs46/lectures_2017/10-unsupervise/... · 2017-10-19 · Bi-plot-0.4 -0.3 -0.2 -0.1 0.0 0.1 0.2-0.4-0.3-0.2-0.1 0.0 0.1 0.2 PC1 PC2 1 2 3 4 5 6

Finding the first PC

Given a data set Xn×p, how do we compute the first principlecomponent?

Since we’re only interested in the variance, let us assume that eachof the variables in Xn×p has been centered to have mean zero.

We then look for the linear combination of the sample feature valuesof the form

zi1 = φ11xi1 + φ21xi2 + . . .+ φp1xip =p∑

j=1φi1xi1. (1)

that has the largest sample variance subject to the constraint∑pj=1 φ

2j1 = 1.

11 / 26

Page 12: Principle Components Analysis (PCA)rcs46/lectures_2017/10-unsupervise/... · 2017-10-19 · Bi-plot-0.4 -0.3 -0.2 -0.1 0.0 0.1 0.2-0.4-0.3-0.2-0.1 0.0 0.1 0.2 PC1 PC2 1 2 3 4 5 6

Finding the first PCFinding the first principle component loading vector solves theoptimatization problem:

maxφ11,...,φp1

1n

n∑i=1

(p∑

j=1φj1xij)2

subject top∑

j=1φ2j1 = 1 (2)

Using equation 1, we can write the objective in equation 2 as

1n

n∑i=1

z2i1

Note: Since 1n

∑ni=1 xij = 0, the average of z11, . . . , zn1 is also zero.

Thus, the objective that we are maximizing in equation 2 is just thesample variance of the n values of zi1

where z11, . . . , zn1 are the scores of the first principal component.12 / 26

Page 13: Principle Components Analysis (PCA)rcs46/lectures_2017/10-unsupervise/... · 2017-10-19 · Bi-plot-0.4 -0.3 -0.2 -0.1 0.0 0.1 0.2-0.4-0.3-0.2-0.1 0.0 0.1 0.2 PC1 PC2 1 2 3 4 5 6

PC

The optimization problem in equation 2 is beyond the scope of thiscourse, but it can be solved by what is called aneigen-decomposition (it requires linear algebra).

13 / 26

Page 14: Principle Components Analysis (PCA)rcs46/lectures_2017/10-unsupervise/... · 2017-10-19 · Bi-plot-0.4 -0.3 -0.2 -0.1 0.0 0.1 0.2-0.4-0.3-0.2-0.1 0.0 0.1 0.2 PC1 PC2 1 2 3 4 5 6

Geometric Interpreation

There is a very nice geometric interpretation for the first principalcomponent.

The loading vector φ1 with elements φ11, . . . , φp1 defines a directionin the feature space along which the data vary the most.

If we project the n data points onto this direction, then theprojected values are the principle component scores z11, . . . , zn1.

14 / 26

Page 15: Principle Components Analysis (PCA)rcs46/lectures_2017/10-unsupervise/... · 2017-10-19 · Bi-plot-0.4 -0.3 -0.2 -0.1 0.0 0.1 0.2-0.4-0.3-0.2-0.1 0.0 0.1 0.2 PC1 PC2 1 2 3 4 5 6

New York Times Example

I Have news stories taken from the New York Times AnnotatedCorpus

I Consists of about 1.8 million stories from the Times, from 1987to 2007.

I Stories have been hand-annotated by humans with standardizedmachine-readable information about their contents.

I From this corpus, have randomly selected 57 stories about artand 45 stories about music.1

1Turned them into a bag-of-words data frame, one row per story, one columnper word; plus an indicator in the first column of whether the story is one aboutart or one about music.

15 / 26

Page 16: Principle Components Analysis (PCA)rcs46/lectures_2017/10-unsupervise/... · 2017-10-19 · Bi-plot-0.4 -0.3 -0.2 -0.1 0.0 0.1 0.2-0.4-0.3-0.2-0.1 0.0 0.1 0.2 PC1 PC2 1 2 3 4 5 6

PCA for NYTimesload("pca-examples.Rdata")

# The workspace now contains:# nyt.frame.raw: a data frame#with counts of words (columns) in stories (rows)# first column, "class.labels",# is a factor indicating "art"# or "music"# nyt.frame: the same, with word

# counts suitably normalized and weighted# art: vector where each row is itself a

# vector of words giving the# actual stories about art, with

# punctuation removed, etc.# music: ditto# Some miscellaneous functions used to

# create the data sets (see end of# this file for gory details)

16 / 26

Page 17: Principle Components Analysis (PCA)rcs46/lectures_2017/10-unsupervise/... · 2017-10-19 · Bi-plot-0.4 -0.3 -0.2 -0.1 0.0 0.1 0.2-0.4-0.3-0.2-0.1 0.0 0.1 0.2 PC1 PC2 1 2 3 4 5 6

PCA for NYTimes# How big is it?dim(nyt.frame)

## [1] 102 4432

# Remember: rows = stories, columns = words (except the first column, which# is the type of story)

# What are some typical words?colnames(nyt.frame)[sample(ncol(nyt.frame),30)]

## [1] "penchant" "brought" "structure" "willing" "yielding"## [6] "bare" "school" "halls" "challenge" "step"## [11] "largest" "lovers" "intense" "borders" "mall"## [16] "classic" "conducted" "mirrors" "hole" "location"## [21] "desperate" "published" "head" "paints" "another"## [26] "starts" "familiar" "window" "thats" "broker"

# A little bit of the datasignif(nyt.frame[sample(nrow(nyt.frame),5),sample(ncol(nyt.frame),10)],3)

## jacket patch tapes want ford failed condemn intentional confined## 24 0 0 0 0.0000 0.0000 0.0000 0 0 0## 2 0 0 0 0.0275 0.0704 0.0000 0 0 0## 85 0 0 0 0.0482 0.0000 0.0000 0 0 0## 59 0 0 0 0.0000 0.0000 0.0000 0 0 0## 76 0 0 0 0.0000 0.0000 0.0215 0 0 0## destroyed## 24 0## 2 0## 85 0## 59 0## 76 0

# What you should see: lots of zeroes! Most words do not appear in most stories.# Harder to see: strong correlations between words which do appear.

17 / 26

Page 18: Principle Components Analysis (PCA)rcs46/lectures_2017/10-unsupervise/... · 2017-10-19 · Bi-plot-0.4 -0.3 -0.2 -0.1 0.0 0.1 0.2-0.4-0.3-0.2-0.1 0.0 0.1 0.2 PC1 PC2 1 2 3 4 5 6

PCA for NYTimes

I Need to omit the first column in the first command because itcontains categorical variables, and PCA doesn’t apply to them.

I The second command just picks out the matrix of projectionsof the variables on to the components.

I Called rotation because it can be thought of as rotating thecoordinate axes in feature-vector space.

# Do PCAnyt.pca = prcomp(nyt.frame[,-1])

# Omit the first column of class labels# Extract the actual component directions/weights for ease of referencenyt.latent.sem = nyt.pca$rotation

18 / 26

Page 19: Principle Components Analysis (PCA)rcs46/lectures_2017/10-unsupervise/... · 2017-10-19 · Bi-plot-0.4 -0.3 -0.2 -0.1 0.0 0.1 0.2-0.4-0.3-0.2-0.1 0.0 0.1 0.2 PC1 PC2 1 2 3 4 5 6

Bi-plot

−0.4 −0.3 −0.2 −0.1 0.0 0.1 0.2

−0.

4−

0.3

−0.

2−

0.1

0.0

0.1

0.2

PC1

PC

2

1

23

4

5

6

7

8

910

111213

1415

1617

18 19

20

2122

23

24

252627

28

29 30

3132

33

34

35

36

3738

39

40

41

42

4344

45

4647

48

4950

51

52

53

54

55

56

57

5859

60

61

6263

64

6566

67

68

69

70

71

72

73

74

75

76

77

787980

81

82

83

84

85

86

87

88

89

90

9192

93

94

95

96

97

9899

100

101102

−0.2 −0.1 0.0 0.1

−0.

2−

0.1

0.0

0.1

X.X.dX.nd

X.s

X.thX.thisaabandonedabcabilityable

about

aboveabroadabsorbedabsorbingabstractabstractionabstractions

abundanceacademicacademyaccentsacceptaccessaccessible

acclaimedaccommodateaccompaniedaccompanyingaccordingaccordinglyaccountaccountedaccusedachievedachievement

acknowledgeacknowledgedacquiredacquisitionacquisitions

acreacrossacrylics

act

acted

actingaction

actionsactiveactivities

actoractors

actressacts

actuallyadamadamsadamssadaptationaddaddedadding

additionadditionaladdressaddressesaddsadheringadjacent

administrationadmiredadmissionadmitsadoptedadsadults

advanceadvancedadvantage

adventureadventurousadvertisementsadvertising

adviceadvisedadviseradvisingadvocatesaestheticaffair

affairsaffectaffectedaffectionaffordafraidafrica

africanafricasafter

afternoonafterwardagain

againstage

agentaggressiveagingagoagreeagreedagreementahead

aimair

airy

alain

alanalbeitalbert

albumalertalfredalicealienalienationalignedalive

allallen

alliancealliances

allowallowed

allowingallows

alludeallusionsalmost

alonealongalready

alsoalteralternate

alternatesalternativealthoughalwaysam

ambiguousambiguouslyambitionambitionsambitious

america

americanamericansamid

amongamountamsterdamamusementamyananalysisanchored

ancient

andandrea

andrew

andy

angeles

angelsangerangryanimalanimals

animatedannanneanniversary

announcedannual

anonymousanotheranselansweranthonyantiantonioanxietyanyanyoneanythinganywayapart

apartmentapparent

apparently

appealingappear

appearanceappearancesappearedappearing

appears

applause

appleappliesappreciateappreciationapprehensionapproachapproachedapproachesappropriate

approvalapproveapproved

april

archetypalarchitectarchitectsarchitecturalarchitecture

areareasarent

aria

armarmsarosearoundarranged

arrangementarrangementsarrestedarrivearrived

art

arteartfularthurarticle

artifactsartificial

artist

artistic

artists

arts

artworkartworks

asasideask

asked

aspectsassemblagesassembledassertionassistantsassociateassociated

association

associationsassumed

assuranceatatheneumatlanticatmosphereattemptattemptsattendedattendingattentionattitude

attitudesattorneyattractattractedauauctionaudacity

audience

audiencesauditoriumaugaugust

austin

austriaauthenticauthor

authoritiesauthorityautobiographicalautobiographyavailable

avantavenueavoid

award

awards

awareawarenessawayaweawkwardaxisbbabies

baby

backbackdropbackersbackgroundbackgroundsbackingbackstagebackwardbaconbad

badlybakerbalancebalancedballballadballet

balloonbamboobanalbanality

band

bands

bankbanks barbare

barefootbarely

baritone

baroquebarrelbarriersbased

basicbasisbassbathed

battlebaybebeachbear

beatbeating

beautifulbeautifully

beautybeauxbecamebecause

becomebecomesbed

beenbeethovensbefore

beganbegin

beginningbegins

begunbehaviorbehestbehind

beigebeingbelievebelievedbelievesbelieving

belowbenbeneathbergmanberlinbernard

bernstein

besidesbestbetterbetween

beverly

beyondbigbigger

biggestbill

billboardsbillionbillsbilly

biographerbirdbirds

birthbitbits

bizarreblackblackness

blacks

blakeblankblanketblastbleak

blendblended

blessedblockblocks

bloodbloomblow

blue

blues

bluntblurringboard

bob

bodiesbodily

body

bohemiabonusboogie

book

books

boom

boostbooth

borderborders

bored

born

bostonboth

bottombought

boulderboulevardbouncedboundboundariesbounds

bourgeoisbowl

box

boxesboy

brahmsbrancusibrandbrandywinebrant

brassbravadobravebread

break.breaking

breakthroughbreathedbreathing

brianbricebrickbridgebridgesbriefbriefingbrieflybrightbrilliantbringbringsbritainbritishbroad

broadcastbroadcasting

broadway

broke

brokerbronzebronzesbroodingbrookbrooklyn

brother

brothers

broughtbrownbruce

brushedbrushesbubblebuddingbudget

build

buildingbuildingsbuilt

bullbunchburiedburleighburnedburtonbusbushbushybusiestbusinessbusinessmanbustbusy

butbutchbutter

buybuyingbuzzbyc

cabaretcablecafecalif

californiacallcalledcallingcallscalmcalvin

cambridgecame

cameracampaigncampuscan

canadacanadiancanal

canceled

cannotcanscant

canvascanvases

capcapacitycapitalcaptaincaptivatedcapturecapturedcapturescarcare

career

careerscarlo

carnegiecarolcarouselcarriedcarries

carrycarryingcars

cartooncartoons

carvedcase

castcasting

castscasual

cat

catalogcatalogue

catastrophecatchcategoricallycategorycathedralcatherinecatholiccaughtcause

causedcautious

cbs

cd

ceaseceiling

celebratecelebratedcelebratescelebrationcelebratory

cellist

cementcenter

centerpiececenterscentralcenturies

century

cerebralceremony

certaincertainlycertificatechaddschairchairmanchallengechamberchampionshipschance

change

changedchangeschangingchannelchanting

chapinchaptercharactercharacteristiccharacterscharcoalchargecharlescharmcheapercheckcheckscheeredcheerfulchef

chelsea

chemicalchestchic

chicagochiefchild

childrenchildrens

childschinachinese

chipschoicechoiceschooses

choosingchoralchords

choreographedchoreographerchoreographychoruschoruses

chosechosen

chrischristianchristopher

churchchurchscinematic

circacirclecirclescitecitedcities

citizencitizenscitycityscivic

clarinetistclarityclark

classclassesclassic

classical

classics

clay

clearclearlyclichclientscliff

climateclimbclinical

clintonclips

close

closedcloselycloserclosesclosing

clothesclottedcloyingclubclubsclueco

coast

coat

cocacoincidecolacoldcolecollaboratedcollaborationcollagecolleaguecolleaguescollectingcollection

collections

collectivecollectorcollectors

college

color

coloredcolorfulcoloring

colors

columbiacolumncomcombinationcombined

combiningcome

comedy

comescomfortcomic

comingcommandcommentcommercecommercialcommercials

commissioncommissionedcommissionercommissionscommitmentcommittedcommitteecommoncommunity

compactcompagniecompanies

companycompanys

comparecomparedcomparisoncompellingcompetecompetition

complainedcompletecompleted

completely

complexcomplexitycomplicatedcomponentcomposed

composercomposers

compositioncompositions

computercomputers

conceivedconceptconceptual

concernconcernedconcerns

concertconcerts

concludedconcludesconclusionconcoction

concretecondemnconditionconductconductedconductor

conferenceconfidenceconfidentconfined

confirmscongressconjunction

connconnectedconnectionconnectionsconscious

consecutiveconservativeconsiderconsiderableconsideredconsists

constantly

constructedconstructionconstructionsconsumerconsumersconsumingcontainedcontemplativecontemporaries

contemporary

contentcontextcontinue

continuedcontinues

continuingcontourscontractcontractscontrastcontributed

contributescontributioncontributionscontrolcontrolledcontroversialconventionalconventionsconversationconveyedconvincedconvincingcool

cooper

coordinatedcopiedcopyrightcorcoran

corkcornercorporatecorrectioncorridor

costcostscostumecostumes

cotterartcottoncould

couldnt

councilcountcountercountless

countries

country

countryscountycouplecoupledcouplescoursecourtcourting

cousincovercoveredcoveringcovers

cowboycozycrabcrackscradlecraftcraftycramercrane

crawlcrazycream

creamycreatecreated

createscreatingcreationcreationscreativecreativelycreativitycreaturescreditcrewcriticcritical

criticizedcritics

crosscrowd

crowdedcrowdscrucialcrudecrycrystalcubist

culturalculture

cummingcummingscunningham

curatorcurators

curiouscuriouslycurrentcurrentlycurtaincustomers

cutcutscuttingcy

cyclecynical

ddaddadsdailydallasdamagedamaged

dancedanced

dancer

dancersdances

dancingdangerdarkdarkened

datedatesdaughter

daughtersdaumier

daviddavisdawndaydaysdaytime

dazzlingdedeaddeadlinedeadlydeadpandealdealer

dealers

dealing

deathdeborahdebt

debutdebutsdecdecade

decadesdecemberdecidedecideddecidedlydecidingdecisiondecorated

decorative

dedicateddeepdeeper

deeply

defiant

defineddefinesdefiningdefinitiondeftdegreedegrees

deldelayeddelicate

delightfuldemanddemanded

demandsdemonstratedenis

departdepartment

depictdepicteddepictingdepictionsdepictsdepressiondepthderiveddesdescendingdescribeddescription

designdesignateddesigned

designersdesignsdesiredesperatedespite

destroyeddestructiondetacheddetaildetailsdetermineddevelopdeveloped

developerdevelopers

developingdevelopment

deviseddevoteddi

diamond

dianadickdiddidnt

diedieddifference

differentdifferentlydifficult

digital

dimensiondinner

direct

directed

directingdirection

directly

director

directorsdirtydisappeardisasterdiscoverdiscovereddiscussdiscusseddiscussing

discussiondisk

displaydisplayed

dissolveddissonances

distantdistinctdistinctiondistinctivedistributiondistrict

divadive

diversedivideddivision

dodocumentary

doesdoesntdog

doingdollardollarsdolphin

domesticdominated

don

donald

donations

donedontdoordoorsdotdotsdoubledoublingdoubtdoug

douglasdown

dozendozensdrdrafteddrama

dramatic

drawdrawingdrawingsdrawndreamdreamily

dreamingdreamlike

dreamsdressdressed

dressesdrew

driftdrifted

drinkingdrivedrivendrivingdrummer

drydudueduetduke

dumpyduringdutch

dynamicdynastyeeach

eagereagerlyeagerness

earearlier

earlyearnedearnest

earth

earthly

easeleasily

easteasterneasyebullience

eccentricechoeseconomiced

edgeedges

editoreditorseducationedwards

effecteffectivelyeffectseffortefforts

eighteightheitherelelaborateelder

elderlyelectedelectronicelegant

elegantlyelementalelementaryelementselizabeth

else.elsewhereelvis

embeddedembody

emergingemilyemmyemotionally

emphasisemphasizedemphasizesempireempty

emulsionenable

encore

encounterencounteredencouragedend

endedendingendless

endsenergyengaged

engagement

engagingengineeringengineersengland

englishengravedenhancedenjoyenjoyedenjoymentenlargedenormous

enough

ensemble

enterenterpriseenthusiasmenthusiasticentireentirelyentrance

entrancesentrepreneurentries

entryenvelopingenvironmentenvisionedepisodesequallyequateequipped

eraericerroreruptedescapeespecially

essenceessentialessentiallyestablishestablished

establishmentestateestheticestheticallyestimate

estimated

et

etchings

europeeuropeaneuropeans

even

evening

eveningsevent

events

eventually

ever

everyeverydayeveryoneeverythingeverywhereevidenceevidentlyevokeevokedevokesevoking

evolutionexexactlyexaminationexampleexceptexceptionexceptional

exceptionallyexchangeexcitementexcitingexecutive

exhibitexhibitedexhibiting

exhibitionexhibitions

existexistenceexistingexitsexoticexpandexpansion

expectexpectedexperienceexperiencedexperiencesexperimentalexperimentingexpert

experts

explainexplained

explainingexplains

expletiveexploitsexplore

exploringexplosiveexposureexpressexpressedexpressingexpressionexpressionismexpressionistexpressionistic

exquisiteextendedextendsextensiveextentexterior

extraextraordinaryexultationeye

eyedeyesffabric

fabulous

facadefacefacesfact

fadefail

failedfailurefairfaithfallfalling

falls

falsefamefamiliarfamilies

familyfamilysfamous

fan

fancifulfantasyfar

farefascinating

fascination

fashion

fashionablefashionedfasterfatefatherfathersfavored

favoritefearfeaturefeaturesfeaturing

febfebruary

feelfeelingfeelingsfeels

feet

feldmanfeldmansfell

fellowfellowshipfelt

femalefeminist

festival

festivals

fewfewerfieldsfifthfightfightingfigurative

figure

figuresfiled

fillfilledfilmfilms

filter

finalfinallyfinancedfinances

financialfinancing

findfindingfinds

finefingerfingers

finishfinishedfirefiresfirm

firstfishfishingfitfitsfiveflagflairflamesflashesflat

flavoredfledflemish

flesh

flewflightflippedfloatfloating

floor

floorsflorence

flowflowerflowersfoam

focusfocusedfoldsfolk

followfollowedfollowersfollowingfollowsfood

foot

footballfootlightsfor.forbiddingforceforcedforcesfordforegroundforeignforest

foreverforget

forgotten

formformalformatformedformer

formsforwardfoundfoundationfoundationsfoundedfourfourth

fragmentationfragments

framefrancefrancescofrancis

franciscofrank

fredfrederic

frederickfree

freedom

frenchfrescoesfresh

fridayfriendfriends

friendshipfringesfromfront

frozenfrustrationfueledfull

fullyfultonfun

function.functionalfunctions

fund

fundamentallyfunny

furniturefurtherfusionfussfuturefuturisticg

gabrielgag

gagosiangaingained

gaining

galleries

gallery

gallerysgamegames

gapgaragegarbage

garde

garden

gardensgarygate

gathergatheringgave

gaygeldergeneralgenerallygeneratedgenerationgenerationsgenre

genresgentlegentlygeometric

george

geraldgerman

germanygertrudegesturegestures

get

getsgettinggiantgiantsgiftgildedgilman

girlgirls

giuliogiuseppe

givegivengivesgivingglamorous

glassglennglimpseglobalglobegloryglue

glueckart

gogoalgodgodsgoesgoghgoinggold

goldengonegoodgoodmangoods

gospelgotgothic

governmentgovernmentsgowngracegraduallygraffitigrahamgrammygrandgrandiosegranitegrantsgrasp

grass

graygraysgreatgreatergreatest

greekgreen

greenwichgrew

gridgridsgroom

ground

groundsgroupgroupsgrow

growinggrowsgrowthguarded

guards

guestguggenheim

guideguidelinesguilty

guitarguitarist

gun

guyguysh

had

hairhalf

hallhalls

hallucinatoryhallway

hand

handedhandlehandledhandlinghandshandsomehanghanging

hans

happenhappenedhappening

happenshappyharborhard

harderharlemharmonicharmonies

haroldharrisonsharshhartford

harvard

harvestedharvey

hashasnthat

hauntinghavehavenhavinghaydnhe

headheadache

headedheadingheads

healthy

hearheard

hearinghearingsheartheavenheavily

heavyheeledheldhelen

helphelpedhelpinghencehenry

her

hereheritageheroheroesheroic

herselfhes

hicksville

hidden

hide

high

higherhighest

highlyhillhills

himhimself

hintedhintshiphipshiredhishistorianshistoric

historicalhistory

hithitshockeyhoffman

holdholdingholdings

holdsholeholland

hollywood

homagehome

homelesshomerhomes

honghonor

honoringhop

hopehopeshopinghorsehospitalhosthot

hotel

hour

hourshouse

householdhouseshouston

how

howardhowever

howshugehugohuman

humanshumidityhumor

hundredshungryhunthunter

hurthusbandhustle

hybrid

i

icons

id

ideaidealisticidealizeidealsideasidenticalidentifiedidentifyidentityideological

if. ignatiusignoreignores

ii

illillegalilluminatedillusionsillustrateillustrational

im

image

images

imaginationimagine

imitation

immediate

immediatelyimpendingimplicationsimplied

impliesimportance

importantimpossibleimpresarioimpressimpression

impressionismimpressionist

impressiveimprovedimprovementimprovisationimprovisations

in.inchinclude

includedincludes

includingincomeincorporatedincorrectlyincrease

increasingincreasinglyindeedindependentindianindicatedindividualindoorsindustrialindustryinfamousinfatuation

inflate

inflated

influenceinfluencedinfluentialinformalinformationinformed

infraredinhabitedinherentinherited

initiallyinjuriesinner

innocenceinside

insistedinsists

inspirationinspiredinstallinstallationinstallationsinstalledinstanceinstantinstead

instituteinstitutionalinstitutionsinstrumentalinstrumentsinsuranceintact

intellectualintellectuallyintendedintenseintent

intentionalintentionallyinteractinterestinterested

interestinginterestsinteriorinteriorsinternational

internetinterpretationinterviewinterviewsintimacyintimateinto

introduced

introducingintroduction

invasioninventioninventionsinvestment

invisibleinvitationinvitedinvokeinvolve

involved

involvementinvolvesinvolvingiris

iron

ironyirvingisisland

islip

isnt

israeliissueissuedissuesit

italianitalians

italy

italysitemsits

itself

ive j

jackjacketjacksonjail

jamesjan

janejanisjanuaryjapan

japanesejasper

jaunty

jazzjeffjenniferjeremyjerry

jersey

jetsonsjewelryjimmyjobjoejohnjohnny

johnsjohnsonjohnsonart

johnssjoined

jointjokejokesjon

jonathan

jonesjosephjoyjrjuan

juliejuly

junejunglejunior

justjustice

juxtapositionkkane

keenkeepkeepingkenkennedy

kennethkeptkevin

key

kidskileykilled

killingkindkinds

kingkingskiosk

kisskneeknewknockedknowknowingknowledgeknown

knows

kong

kyl

la

lablabellabeled

labelslaborlacklackinglacks

land

landed

landscapelandscapes

lane

languagelarge

largelylargerlargest

larklarry

last

lastedlastinglatelaterlatestlatinolatter

laughlaughedlaughinglauralaurencelavish

law

lawnlawrence

lawyerslayers

leleadleaderleading

leagueleaguesleanleapslearn

learnedlearningleastleave

leavesleaving

ledleeleftleftover

legal

legendlegendarylegislatorslegitimatelegs

leighlendlengthlengthyleno

lentleonleonard

leslielessletlets

letterlettermanletters

lettinglevellevelsliamlibertylibrary

librettistlibrettolichtenstein

lieslifelifeslifetime

light

lighterlightinglights

like likedlikelylikes

limitedlimousinelincolnlindalinelined

lineslinklinks

lionliquid

lisa

listlisten

listenedlisteninglistsliterallyliterary

lithographslittle

livelived

liveslivinglloydloanloanslobbylocal

location

lockedloeilloftlondonlonelonelylonglonger

longtimelook

lookedlookinglookslooseloosely

lordloslosing

losslostlotlotsloudlouislounge

love

lovedlovelylover

loversloveslowlowerloyallucasluckluminous

lureslustluxurylyinglynnlynne

lyriclyricism

lyrics m

mamache

machine

machomacys

mademadisonmagazinemagazinesmagicmainmainstream

majesticmajor

makemakesmakingmalemallmanmanagedmanages

manet

manhattan

manholemanipulatemanipulatedmannermansmansion

mantegnamanuelmanymapmarble

marc

march

mariemarkmarkedmarket

marketingmarriagemarriedmarthamarvelousmary

maskmasksmasonmassmassesmassivemastermasterpiecemasterpiecesmastersmatchmatched

matchingmaterial

materials

matinee

matissemattermatters

mauricemax

maymaybemayormcdonald

me

meadows

mean

meaning

meansmeantmeanwhilemeasure

measuringmeat

media

medicalmeditationmediummeetmeeting

meetingsmeldsmelodramaticmelodymembermembersmemoirsmemory

menmenacing

menilmensmention

mentionedmentionsmerce

merely

merge

mermaidmess

messagemessages

met metalmethodmethodsmetropolismetropolitanmexicomezzanine

mezzo

michaelmicroscopicmid

middlemidnightmidtownmightmikemiles

militarymillenniummillermillionmillions

mindmindedmineminesminiatureminimalminimalistminimumministerminneapolisminor

minorityminuteminutesmiracle

mirrormirrorsmismatched

miss

missedmissingmissionmisspelledmisstated

mistressmitchellmixmixedmixesmixingmock

modemodelmodelsmodern

modernismmodernistmodestmogulmomentmomentarilymoments

momentummonday

mondriansmonetmoney

monochromesmonolithicmontages

monthmonthsmontrealmonumentalmood

moon

mooremoremoreovermorning

morris

moscowmostmostly

mother

motifmotifsmotionmountmountain

mountainsmountedmouthmove

movedmovementmovesmovie

moviesmoving

mozartsmr

mrs

ms

mtvmuchmugmultimediamultiple

mundanemuralmuralsmurray

museum

museums

musicmusical

musicalsmusicianmusiciansmusingsmust

mutedmutual

my

myers

mysteriousmysterymythological

nna

nails

naked

namenamednames

nancynarrativenarrativesnarrow

nashvillenassaunational

naturalnaturally

nature

nbcneal

nearnearbynearlyneatlynecessarilynecessarynecessity

neck nedneedneedednegotiatednegotiationsneighborneighborhoodneil

neithernelsonneo

nervousnetwork

networks

neverneverthelessnewnewestnewly

newman

newsnewspaper

next.

nicenicelynicholasnicknicole

night

nightclubnightsnine

no

nobodynoisenominatednominationsnondescript

nonenonethelessnonprofitnornormnormallynorthnorthernnostalgia

nostalgicnotnotablenotably

notenotednotes

nothingnoticenoticednotingnotion

novnovelnovembernow

nowherenudenumbernumbers

numerousnurtures

oobject

objects

obliterateobsessed

obstaclesobtainedobviousobviouslyoccasionoccasionaloccasionallyoccuroclock

octoctober

odds

ofoffoffensiveoffer

offeredofferingoffersofficeoffices

officialofficialsoftenoh

oiloilsold

olderoldestoleolsonominousononce

oneonesongoingonlineonly

onstage

onto opopenopened

openingopens

opera

operasoperatic

operatingoperationopportunitiesopportunityopposedoppositeoptimismoptimisticoptions

opulent

ororange

orchard

orchestra

orchestralorchestras

order

organorganization

organizations

organizedoriginaloriginality

originallyososcar

otherothersotherwise

ourourselvesout

outdooroutputoutrightoutsideoutsider

outstandingovaloveroverall

overcrowdedoverlappingoverlookingoversightowed

ownownedownerownersowns

ppapablopaces

pacific

packpackedpage

pagespaidpain

paint

painted

painter

painterlypainters

paintingpaintings

paintspairpairing

palace

palettepampanpandemonium

panel

panelspanicpantspaolopaper

papierpapp

parents

paris

parisianparkparks

parsonspartparticularparticularlypartiespartlypartnerpartnerspartnershippartnershipsparts

partypass

passedpassingpassion

passionatepast

pastichepatpatch

patentlypathpathspatrick

patronpatronspattern

patterns

pattipaulpavilion

pay

payingpeace

penchantpentagonpeopleperpercentpercentageperceptionperchedpercussionpercussionists

perfectperfectly

perform

performanceperformances

performedperformersperforming

perhapsperiodperiodspermanent

permissionperson

personal

personalitiespersonalityperspectivepeterphallusphase

phenomenonphiladelphia

philharmonic

philipphilippines

phillipsphilosophicalphone

photophotograph

photographedphotographerphotographersphotographicphotographsphotography

photos

physical

pianissimopianistpiano

picasso

pickpicked

picking

pictorialpicturepictures piecepiecespierre

pilepiles

pills

pinkpioneerpiratedpitplaceplaced

placesplagued

plainplanplanet

plannedplanning

plansplantedplants

plasterplastic

play

playedplayer

playersplayful

playingplaysplaywright

playwrightsplaywritingpleasepleasurepleasures

plentyplucky

plumplusplush

plywoodpoetpoetrypointpointedpoints

polaroidpolepolicepolished

politicalpoliticspolkepoolpoor

pop

popepopularpopularityportporter

portionportraitportraits

portrayspositionpositionedpositionspossesspossession

possibilitiespossibility

possiblepossiblypostposterpostersposturingpotentialpower

powerfulpracticalpracticepraisedprepreacherprecisely

precision

precocious

prejudice

premiere

premierespremisepreparationpreparatorypreparedpreparingpresence

presentpresentedpresenterspresentingpresentspreservationpreservepresidentpresspressingprettypreview

previews

previouspreviouslypricepricesprimalprimarilyprimaryprimeprince

principal

printprintedprintingprintmaking

printspristineprivateprivately

prizeprobably

problem

problemsprocessprocessesproduceproduced

producerproducers

producesproducingproduct

production

productions

productsprofession

professionalprofessionalsprofessor

profileprofitsprofoundly

programprogrammingprograms

progressprojectprojectionprojectionsprojectspromise

promisedpromisingpromotepromoting

prompted

proof

propaganda

propelledpropertyproportionproportionsproposals

proposedpropsprosceniumprospectprot

protectprotectedprotectionproud

proved

provesprovideprovidedprovider

providesproviding

provocativepruningpsychedelicpsychologicalpub

public

publicationspublished

puccinipulledpullspulsepunishment

punspurchase

purchasespurepurely

puritypurplepurposepurposespursued

pursuingpushputputsputtingpuzzlingqualitiesquality

quantitiesquarterquartet

queen

queens

questquestionquestionedquestioningquestionsquick

quietquietlyquilts quintetquite

r

rabbit

radicalradicallyradio

ragerailingsrainraiseraised

raisingralph

ranrandallrange

rangingraphael

rapidrapperrare

rarelyrarity

ratrate

ratherratingrauschenbergraw

rayrazed

rcarereachreachedreadreadingreadyreal

realismrealisticrealityrealizerealizedreally

realmreasonreasonsrebeccarecallrecalled

recalling

recapturereceive

receivedreceives

recentrecently

recitalrecitalsrecognitionreconstructs

recordrecordedrecordingrecordingsrecords

recoverrecoveredrecreaterectanglerectangularrecurring

red

reducedreductionreference

referencesreferred

referringreflectreflectedreflectingreflectsrefuserefused

regional

registerregistryregular

regulations

rehearsal

reinforcerejectedrejectionrelate

relatedrelationsrelationship

relativerelativelyrelease

releasedrelevant

reliefsreligion

religious

relyremainremainedremainingremainsremarkableremarksrembrandtrememberremembersremindreminder

removedremoving

renaissancerenderedrenderingrenderings

reneerenovatedrenovatingrenovationrent

rentedreopen

reopenedreopeningrepeatedrepeatedly

repertoryrepetitionreplacereplacedrepliedreport

reportedreportersreports

representrepresentationrepresentativerepresentativesrepresentedrepresents

repriserepublicanreputationreputationsrequirerequired

requiresresearchresemble

resembles

reservationsresidenceresidentresinresistance

resolvedresourcesrespectablerespectivelyresponseresponsible

restrestaurantrestaurantsrestingrestless

restorationrestoredresultresultedresultsretainedretiredretitledretreatretrospective

returnreturnedreturningreturnsrevealed

revealingrevealsrevenuereverencereversereview

revisionsrevisitrevival

revivals

revolutionrevolutionary

rewardrhythm

ricerichrichardrichardsonrichnessrid

riderifle

rightringingrippleriserisesrisingrisk

ritaritual

riverriversroadroadsrobrobertrobertarobesrobinson

robust

robustlyrochelle

rock

rocketrocks

rocky

roleroles

rollrollingromanromanceromanticromeronald

room

roomsrootroots

roserossrothenberg

rothkoroughroughlyround

routineroutinely

rowroyroyal

ruckusruderuinsrules

runrunning

runs

ruscharushrushingrussiarussianryder

s

sagasaidsaint

saints

salarysale

salessallesalutesamsamaras

samesamplingsamuelsan

sang

sarahsardonicsargentsatsatisfaction

saturday

saulsavedsawsawssaxophonesay

saying

says

scale

scandalscaryscattered

scenarioscenesceneryscenes

schedulescheduledscholarscholarsscholarship

schoolschoolsschwartz

science

scorescoresscottscramblescreaming

screen

screeningsscreens

sculptorsculptural

sculpturesculptures

seasearch

searching

season

seasonsseatseatingseatsseattlesecond

secretsection

sectionssecurityseductiveseeseeing

seekingseemseemed

seeminglyseems

seenseesselected

selectionselfsellsellingsemisenatesenatorsendsendingsenior

sensesensualsentsentimentsentimental

separate

septseptember

sequenceserene

seriesserious

seriousness

serveserved

service

servicesset setssetting

settingssettlesettledseven

severalsevereseverelysexsexual

shadesshadow

shakespeareshakespeares

shapeshapedshapesshapingsharesharedsharessharonsharp

sharply

she

sheershell

shes

shimmeringshineshinyshipsshirt

shirtsshockedshoesshootingshop

shoppingshops

short

shortlyshotshotsshould

shouldersshout

shouting

showshowedshowingshown

showsshrill

shutside

sidessightsightingssigmarsignsignaled

signaturesignedsignificantsigningsignssilhouettedsilhouettessilk

silver

silverysimilarsimilarlysimplesimplersimplicitysimply

simpsonssimultaneously

since

sing

singersingerssinging

single

singssinksinking

sirsistersitcomsitcomssite

sitessitssitting

sixsixthsize

sizedsizesskepticalsketch

sketchesskinskirtskyskylightskylineskyscrapers

slashsleeksleepslide

slightlyslow

slowly

small

smallersmartsmearingsmellsmith

smithtownsmokesmoothsnakesnakessnappy

snoutsnowso soapsocial

societiessociety

softsofter

softwaresoho

soldsoldiersolidsolosoloists

solomonsomber

somesomebodysomehowsomeonesomething

sometimessomewhat

sonsonatasondheim

song

songs

songwriterssonic

sonssoon

sophisticated

soprano

sortsought

soulsoundsoundedsounds

sourcesouth

southern

sovietspace

spacesspaceyspainspanishsparespatialspeakspeakingspecialspecializesspecialtyspecificspecificallyspectacularspeculationspendspendingspent

spherespinal

spirit

splashspokespokesmansponsoredsponsorssports

spotlight

spreadspring

spur

squaresqueezed

squid

st

stablestaff

stage

stagedstages

staging

stainedstaircasestandstandardstandardsstandingstands

stanley

star

starestarkstarredstarringstars

startstartedstarting

startlingstartsstatestatement

statementsstates

staticstationstatuary

statue

statuettesstatus

staysteadysteelsteepedstefanostein

steinbergsteinway

stentorian

stepstephen

steppedsteppingsteps

stevestevenstickstill

stillsstimulating

stint

stitched

stone

stonesstoodstopstopped

storestoresstoriesstory

straightstrangestreamstreetstreetsstrengthstrengthsstressesstretchstretchedstrictlystrikestriking

stringstringsstripstrippedstripsstrokesstrollstrongstrongeststronglystructurestructures

strugglestuckstudded

studentstudentsstudied

studiesstudio

studios

studystuffsturdystyle

stylesstylish

stylistically

subjectsubjectssubsequentsubstancesubstantialsubtlesubtletysuburb

suburbansubversivesucceed

succeededsuccesssuccessfulsuccessor

suchsuddensuddenly

sue

sufferedsufficientsuffolksuggestsuggestedsuggesting

suggestivesuggests

suitsuitssummer

summerssun

sundaysung

sunlightsunny

sunset

supersuperimposedsuperintendentsupport

supported

supportingsupposedsuresurely

surfacesurfaces

surgesurgeonssurprisesurprisessurprisingsurprisingly

surreal

surrealistsurroundedsurroundingsurroundingssurroundssurvey

survivesusan

suspectsuspended

sweetsweetlyswingswingingswirlingswisssymbolsymmetry

symphony

synonymoussystem

ttaaffetabletacticstailtaiwantake

takentakestaking

tale

talenttalestalk

talkingtalkstall

taptapestask

taught

tax

taylortchaikovskyteamteamsteartearstechtechnically

techniques

technology

teenstelephonetelevision

televisions

tell

tellingtellstemporarytendtendedtendencytender

tendstennessee

tenor

tentativelytenuretermtermsterritorytextexas

texttextstexturethanthankthat

thats

the

theater

theaters

theatricaltheirthemthematicthemethemesthemselvesthen

theretheres

these

they theyrethickthievesthinthingthingsthinkthinkingthinksthirdthis

thomasthosethough

thought

thoughtfulthoughts

thousandsthreatthreateningthreethrewthrift

throughthroughoutthrowthrown

thursday

thusticketticketmaster

tickets

tietiestimbre

time

timedtimelesstimestimingtimpanitintiny

title

titledtitlestotoccatatodaytodaystogethertold

tomtomkinss

tomorrowtonalitytonetoned

tonight

tonightstony

tootooktoolstoptopped

topstorsototaltotemtouchtouchestouchingtough

tourtouredtouring

tourists

tours

towardtowardstowertowers

towntownstoytracetraces

tracktractortradetradedtradition

traditionaltraditionallytraffic

tragedytrainingtransformtransformationstransformedtransformingtransgressionstransparenttrashtravel

traveledtravelerstraveling

treasurestreasurytreat

treatedtreatmenttreetreestriangle

tribute

tricktricky

triedtries trio

triptrompetroubletrucktruetrulytrust

trytryingtucked

tuesday

tuesdaystullytune

tunes

turn

turnedturner

turningturnstvtwice

twisttwotwomblytype

u

ultimateultimatelyunableuncannyuncertaintyunclearunderunderstanding

undoubtedlyundulatingunfinished

unfortunateuniformunionuniqueunited

universaluniversity

unknownunlikeunlikelyunnoticed

unrelated

untiluntitledunusual

unwelcomeup

uponupperupsideurbanurge

ususeused

useful

usesusingusualusuallyutterv

vavaguely

vainvalley

valuablevaluevaluesvanvaried

varietyvarious

varyingvastvaudeville

vevegasvenicevenue

venus

verdisvernonversesversionversionsveryveterans

vivibrant

vicevictimsvictorvictoriavictory

videovienna

vienneseviewviewed

viewerviewers

viewingviews

village

vincentvintageviolenceviolinviolinistviolins

violistvirtuallyvirtuosity

visiblevisiblyvisionvisit

visitedvisitingvisitor

visitors

vistavisualvivid

vocal

vogue

voice

voicesvolcanic

volumevonvulnerablew

wadingwadsworthwagners

waitingwaivedwalk

walkedwalker

walkingwalks

wallwalls

walterwaltzwantwantedwants

war

warholwarholswarm

warmthwarnerwarrenwars

was

washwashingtonwasntwatchwatcheswatchingwaterwatercolor

watercolorswatsonwavewaxwaywayswe wealthweaponwear

wearingwearsweather

webwebb

wednesday

week

weekendweeklyweeksweiweightweillwelcome

wellwentwere

westwesternwetwhat

whatswheelswhen

wheneverwherewhetherwhich

while.

white

whitney

who

wholewhomwhosewhywichita

widewidelywifewildwildly

will

williamwilliams

williamsburgwillingwilson

wind

windowwindows

wingwingswinning

winslowwinterwipedwirewisely

wish

witwitchwith

withinwithoutwitnesswitnesses

wittywolf

wolfgang

womanwomanswomen

womens

won

wonderwonderfulwonderingwont

woodwoodcutswooden

woogiewoosterwordwordswore

workworked

workersworking

works

workshopworkshopsworldworlds

worldwideworried

worryworseworstworthwouldwouldnt

wound

writerwriters

writes

writing

written

wrong

wrote

wrywww

xavieryyale

yangyard

yearyearsyellow

yesyesterday

yetyieldingyogiyorkyorkersyorks

you

youngyounger

youryoureyouth

youthfulyu

I This shows the two first PC for the NYtimes data.I Blue scores represent the scores for the first two PC’s.I The red arrows represent the first two PC loading vectors.I Here, a bi-plot is not useful at all since the data is too high dimensional. (See Figure 10.1 for where a

bi-plot is useful.)19 / 26

Page 20: Principle Components Analysis (PCA)rcs46/lectures_2017/10-unsupervise/... · 2017-10-19 · Bi-plot-0.4 -0.3 -0.2 -0.1 0.0 0.1 0.2-0.4-0.3-0.2-0.1 0.0 0.1 0.2 PC1 PC2 1 2 3 4 5 6

Look at the leading components# What are the components?# Show the 30 words with the biggest positive loading on PC1signif(sort(nyt.latent.sem[,1],decreasing=TRUE)[1:30],2)

## music trio theater orchestra composers opera## 0.110 0.084 0.083 0.067 0.059 0.058## theaters m festival east program y## 0.055 0.054 0.051 0.049 0.048 0.048## jersey players committee sunday june concert## 0.047 0.047 0.046 0.045 0.045 0.045## symphony organ matinee misstated instruments p## 0.044 0.044 0.043 0.042 0.041 0.041## X.d april samuel jazz pianist society## 0.041 0.040 0.040 0.039 0.038 0.038

# biggest negative loading on PC1, the other end of that scalesignif(sort(nyt.latent.sem[,1],decreasing=FALSE)[1:30],2)

## she her ms i said mother cooper## -0.260 -0.240 -0.200 -0.150 -0.130 -0.110 -0.100## my painting process paintings im he mrs## -0.094 -0.088 -0.071 -0.070 -0.068 -0.065 -0.065## me gagosian was picasso image sculpture baby## -0.063 -0.062 -0.058 -0.057 -0.056 -0.056 -0.055## artists work photos you nature studio out## -0.055 -0.054 -0.051 -0.051 -0.050 -0.050 -0.050## says like## -0.050 -0.049

# Ditto for PC 2signif(sort(nyt.latent.sem[,2],decreasing=TRUE)[1:30],2)

## art museum images artists donations museums## 0.150 0.120 0.095 0.092 0.075 0.073## painting tax paintings sculpture gallery sculptures## 0.073 0.070 0.065 0.060 0.055 0.051## painted white patterns artist nature service## 0.050 0.050 0.047 0.047 0.046 0.046## decorative feet digital statue color computer## 0.043 0.043 0.043 0.042 0.042 0.041## paris war collections diamond stone dealers## 0.041 0.041 0.041 0.041 0.041 0.040

signif(sort(nyt.latent.sem[,2],decreasing=FALSE)[1:30],2)

## her she theater opera ms## -0.220 -0.220 -0.160 -0.130 -0.130## i hour production sang festival## -0.083 -0.081 -0.075 -0.075 -0.074## music musical songs vocal orchestra## -0.070 -0.070 -0.068 -0.067 -0.067## la singing matinee performance band## -0.065 -0.065 -0.061 -0.061 -0.060## awards composers says my im## -0.058 -0.058 -0.058 -0.056 -0.056## play broadway singer cooper performances## -0.056 -0.055 -0.052 -0.051 -0.051

20 / 26

Page 21: Principle Components Analysis (PCA)rcs46/lectures_2017/10-unsupervise/... · 2017-10-19 · Bi-plot-0.4 -0.3 -0.2 -0.1 0.0 0.1 0.2-0.4-0.3-0.2-0.1 0.0 0.1 0.2 PC1 PC2 1 2 3 4 5 6

Look at the leading components

I These are the thirty words with the largest positive andnegative projections on to the first component2.

I Words with positive projections mostly associated with musicI Those with negative components with visual arts.I Why do we see women and mothers here?

2Which direction is positive and which is negative depend on internal choicesin the PCA algorithm

21 / 26

Page 22: Principle Components Analysis (PCA)rcs46/lectures_2017/10-unsupervise/... · 2017-10-19 · Bi-plot-0.4 -0.3 -0.2 -0.1 0.0 0.1 0.2-0.4-0.3-0.2-0.1 0.0 0.1 0.2 PC1 PC2 1 2 3 4 5 6

PCA for NYTimes# Plot the projection of the stories on to the first 2 components

# Establish the plot windowplot(nyt.pca$x[,1:2],type="n")

# Arts stories with red Aspoints(nyt.pca$x[nyt.frame[,"class.labels"]=="art",1:2],pch="A",col="red")

# Music stories with blue Mspoints(nyt.pca$x[nyt.frame[,"class.labels"]=="music",1:2],pch="M",col="blue")

−0.4 −0.3 −0.2 −0.1 0.0 0.1 0.2

−0.

3−

0.2

−0.

10.

00.

10.

2

PC1

PC

2

A

A

A

A

A

A

A

A

A

A

AAA

A

A

AA

AA

A

AA

A

A

AA

AA

AA

A

A

A

A

A

A

AA

A

A

A

A

A

AA

AA

A

AA

A

A

A

A

A

A

A

MM

M

M

MM

M

MM

M

M

M

M

M

M

M

M

M

M

M

MM

MM

M

M

M

M

M

M

M

M

M

MM

M

M

M

M

M

M

M

M

MM

# The separation is very good, even with only two components.

22 / 26

Page 23: Principle Components Analysis (PCA)rcs46/lectures_2017/10-unsupervise/... · 2017-10-19 · Bi-plot-0.4 -0.3 -0.2 -0.1 0.0 0.1 0.2-0.4-0.3-0.2-0.1 0.0 0.1 0.2 PC1 PC2 1 2 3 4 5 6

How well is PCA doing?

Even though we have gone from 4431 dimensions to 2, and thrownaway a lot of information, we could draw a line across this plot andhave most of the art stories on one side of it and all the musicstories on the other.

If we let ourselves use the first four or five principal components,we’d still have a thousand-fold savings in dimensions, but we’d beable to get almost-perfect separation between the two classes.

This is a sign that PCA is really doing a good job at summarizingthe information in the word-count vectors, and in turn that the bagsof words give us a lot of information about the meaning of thestories.

The figure also illustrates the idea of multidimensional scaling —finding low-dimensional points to represent high-dimensional data bypreserving the distances between the points.

23 / 26

Page 24: Principle Components Analysis (PCA)rcs46/lectures_2017/10-unsupervise/... · 2017-10-19 · Bi-plot-0.4 -0.3 -0.2 -0.1 0.0 0.1 0.2-0.4-0.3-0.2-0.1 0.0 0.1 0.2 PC1 PC2 1 2 3 4 5 6

Scree Plot

I We can figure out the number of principal components byfitting what’s called a scree plot.

I Choose the smallest number of principal components that arerequired such that an adequate amount of variability isexplained.

I We look for the point at which the proportion of varianceexplained by each subsequent principal drops off.

I This is called the elbow of the scree plot.I These plots are application specific and ad-hoc.

24 / 26

Page 25: Principle Components Analysis (PCA)rcs46/lectures_2017/10-unsupervise/... · 2017-10-19 · Bi-plot-0.4 -0.3 -0.2 -0.1 0.0 0.1 0.2-0.4-0.3-0.2-0.1 0.0 0.1 0.2 PC1 PC2 1 2 3 4 5 6

Scree Plot

plot(nyt.pca,type="l", main="")V

aria

nces

0.01

20.

014

0.01

60.

018

0.02

0

1 2 3 4 5 6 7 8 9 10

25 / 26

Page 26: Principle Components Analysis (PCA)rcs46/lectures_2017/10-unsupervise/... · 2017-10-19 · Bi-plot-0.4 -0.3 -0.2 -0.1 0.0 0.1 0.2-0.4-0.3-0.2-0.1 0.0 0.1 0.2 PC1 PC2 1 2 3 4 5 6

Summary

I Visual analysis of PCA is adhoc (such as the scree plot).I In fact, the question of how many PC to choose is very

ill-defined in practice and depends on a specific application.I In practice, we tend to look at the first few PC in to find

interesting patterns in the data.I If not interesting patterns are found, then further PC are

unlikely to be of interest.I We could use PC in the context of regression (see Section 6.3.1

for further details and this is more principled).

26 / 26


ExperimentalStudyonMechanicalBehaviorofHingedJointin … · -0.2 -0.1 0.0 0.1 0.2 0.3 0.4 0.5 0.6-0.025 0.000 0.025 0.050 0.075 0.100 0.125 0.150 0.175 0.200 0.225 height of hinge

Semantic Texture for Robust Dense Trackingjc8515/pubs/semantic_texture.pdf · pitch [rad] 0.2 0.1 0.0 0.1 0.2 yaw [rad] 0.2 0.1 0.0 0.1 0.2 5000 10000 15000 20000 25000 Figure 1

Modeling heteroskedasticity: GARCH modelinghedibert.org/wp-content/uploads/2018/05/garchmodeling.pdfDays Returns-0.2-0.1 0.0 0.1 0.2 Jan/3/05 May/08/08 Sep/12/11 Jan/16/15 May/23/18

 · 4/24/2020  · 0.1 3"ei 0.0 0.0 wood fence woo fence post & rail fence existing contour proposed contour proposed elevation lmu 0.0 0.1 0.7 0.7 0.1 0.0 0.0 0.0 mon. found 0.0

Missouri Student Survey 2018 Report - dmh.mo.govClub Drugs 0.9%-Methamphetamine 0.3% - 0.3 % 0.0 Heroin 0.2% - 0.1% 0.0% 6 The Youth Risk Behavioral Survey (YRBS, 2017) collects national

0.1 0.2 0 - Gotop

Supra Customer Session · Android 52.6% 52.8% 0.2% Apple 41.8% 41.7% -0.1% Microsoft 3.4% 3.5% 0.1% Blackberry 2.0% 1.8% -0.2% Symbian 0.1% 0.1% 0.0% Top Smartphone Platforms 3 Month

MARKET LENS€¦ · Dr. Reddy’s Lab (ADR) 33.6 0.1 0.2% -1.2% -10.5% Tata Motors (ADR) 26.8 (0.0) -0.1% -6.6% -19.0% Infosys (ADR) 18.2 0.1 0.7% 3.1% 12.0% ICICI Bank (ADR) 9.2

Using Vital Dye Staining - Lions VisionGiftvisiongift.org/wp-content/uploads/2019/02/ARVO-poster... · 2019-04-03 · 0.6% 0.6% 0.7% 0.4% 1.0% 0.9% 0.1% 0.1% 0.4% 0.0% 0.0% 0.5% 0.2%

Supplementary Information · 2 200 300 400 500 600 700 800 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 253 281 330 348 390 (a) Absorbance Wavelength / nm 200 300 400 500 600 700 800 0.0 0.1 0.2

2. Thackeray CLRF 2016 · -0.1-0.2-0.3-0.4-0.5-0.6-0.7 Trend (days/year) Trophic level Prim prod Prim cons Sec cons Mean change (days year-1)-0.6-0.5-0.4-0.3-0.2-0.1 0.0 Based upon

Comparison£of£methods£that£use£whole£genome£ data£to ... · CV MAF > 0.05 CV MAF < 0.0025 0.4 0.3 0.2 0.1 0.0 –0.1 Estimate of h 2 0.5 0.4 0.3 0.2 0.1 0.0 –0.1 Estimate

Årsrapport for TV-seing i Norge 2015...National Geographic 0,7 0,8 +0.1 VG 0,0 0,2 +0.2 Disney Channel 0,6 0,4 -0.2 Nickelodeon 0,2 0,2 0.0 Fox 0,9 0,8 -0.1 Øvrige kanaler 8,4 7,3

Regional Outlook Asia and the Pacific 2018 May · Bangladesh 0.1 Korea 0.1 New Zealand 0.1 Mongolia 0.0 Japan 0.2 Indonesia 0.1 China 0.1 India 0.1 Australia 0.0 Other Countries &

Performance Reports · Performance Reports June 2020 0 2 4 6 8 10 12 14 12.3 4.6 2.9 Passengers Per Hour 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 0.9 0.3 0.1 Passengers Per Mile 0.0%

Ferroic collinear multilayer magnon spin valve - arXiv · Ferroic collinear multilayer magnon spin valve ... 600 700 800 900 1000 H (Oe) 0.2 ANE 0.1 0.0 0.1 0.2 0.3 0.4 0.5 ... sc

IN5490 Advanced Topics in Artificial Intelligence for ...€¦ · 3-D plot of LDA of the binary silhouettes of different activities.-0.2-0.1 0 0.1 0.2-0.2-0.1 0 0.1 0.2-0.05 0 0.05

The Role of Analytics in the Development of Advanced ... · N287 0.4% 0.1% 0.2% 0.6% 1.2% 1.2% N316 2.8% 0.7% 2.7% 6.1% 7.2% 7.6% N362 0.1% 0.1% 0.1% 0.2% 0.2% 0.2% N385, N390 4.6%

CHINESE PROPERTY INVESTMENT FOCUS: CANADA · GUANGXI 1.0% HAINAN 0.2% XINJIANG 0.4% NINGXIA 0.1% GANSU 0.4% QINGHAI 0.1% TIBET 0.0% SEARCH ORIGIN 11% from over 155 countries AROUND

RSEN SERIES - TDK...0.1～2 0.1～2-----0.1～10 0.1～10 0.2～10 0.3～5 減衰量保証帯域(MHz) ディファレンシャルモード 25dB保証 0.2～30 0.2～30 0.2～30 0.2～30

118.7° - q-tran.com file0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.0 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 2200 k 2700 k 3000 k 3500 k 4000 k 5000 k 6000 k 7000 k 8000 k 9000 k 10000 k 450 460 470

CalPERS Trust Level Review - Risk Management …...Source: BarraOne I Ca/PERS Tracking Error(%) 0.2% 0.3% 0.0% 4.3% 0.2% 0.3% 0.3% 0.1% 2.1% 1.8% 0.5% CalPERS Trust Level Appendix

ICHARM WorkshopFlood Damages as Percent of GDP (Based on damages and GDP data in 2000 dollars) 0.0% 0.1% 0.1% 0.2% 0.2% 0.3% 0.3% 0.4% 0.4% 0.5% 0.5% 1900 1903 1906 1909 1912 1915

THE PHILANTHROPIC LANDSCAPE · 2 | NATIONAL COMMITTEE FOR RESPONSIVE PHILANTHROPY 0.0 0.1 0.2 0.3 0.4 West South Northeast Midwest 21% 23% 31% 20% 22% with Gates 0.0 0.1 0.2 0.3 0.4

plotROC: A Tool for Plotting ROC Curves · JournalofStatisticalSoftware–CodeSnippets 5 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 A 1-Speci city y 0.0 0.2 0.4 0.6 0.8 1.0 0.0

Fast Pattern Detection using Orbit Decomposition · rotation of 2D signal projected into its 3 highest eigen-vectors-0.2-0.15-0.1-0.05 0 0.05 0.1 0.15-0.2-0.1 0 0.1 0.2 0.3-0.2-0.15-0.1-0.05

ILLINOIS INTEGRATED WATER QUALITY REPORT AND …...74,709 74,536 90.1 93.0 9.9 7.0 0.0 0.0 0.2 0.2 0.0 0.0 Secondary Contact(2) 318,477 1,092 100 100.0 -- -- -- -- 99.7 99.7 0.0 0.0

Evidence Base for the Local London Growth Business Plan...Liverpool City Region 0.0 0.1 1.4 North East 0.1 0.0 1.4 Sheffield City Region 0.2 0.2 1.4 Tees Valley 0.0 0.0 1.2 West Midlands

AC-DC POWER SUPPLYSafety Tab (C4201HF-2 2*3) Circuit 14.2 ± 0.1 Circuit 2 Last Circuit A ± 0.2 4.2 ± 0.1 4.2 ± 0.1 B ± 0.259.6 ± 0.2 10.65 ± 0.2 9.6 ± 0.2 11.7 ± 0.2 10.8

2017 - 2019 HUMANITARIAN RESPONSE PLAN€¦ · 06/04/2018  · UGANDA RWANDA BURUNDI 0.4 0.1 0.1 0.3 0.6 0.8 1.4 0.7 0.9 0.5 0.2 0.4 0.5 0.2 0.1 0.1 0.2 0.1 0.3 1.8 1.1 0.9 0.5 0.2

4 CA VC 4viwr- - MIT OpenCourseWare...0.0 Start of Shear Peak Shear Stress 0.1 0.2 0.3 Normalized shear stress, q/ s ' vm 1 1 2 2 4 4 8 8 0.0 0.1 0.2 0.3 0 5 10 15 Norm. Excess Pore

petit.lib.yamaguchi-u.ac.jppetit.lib.yamaguchi-u.ac.jp/.../file/221/20101111150720/2007010069.pdf · 03 0.2 0.1 0.0 0.0 1800 1400 1200 1000 800 400 200 0.0 0.1 0.1 0.2 a. 0.4 20 10

Investment Monthly - AdviserNET · 2012. 5. 25. · FSF0973AU Acadian Quant Yield Fund (Class A) 0.9726 Distribution 0.0 1.4 2.6 5.9 6.3 5.7 - - 0% Growth 0.2 -0.4 -0.2 -0.4 -0.1

A210 EXTERIOR ELEVATIONS · oh 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.1 0.1 0.1 0.1

Gernot Hoffmann CIE Color Space - Laboratoire Bordelais … · 570 575 580 585 590 595 600 605 610 620 635 700 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 0.0 0.1 0.2 0.3 0.4 0.5