personality at three levels of analysis
TRANSCRIPT
Open Science Open statistics, open materials, open methodology, open data Part I: Open Statistics: R
Personality at three levels of analysis
William Revelle
Department of Psychology Northwestern UniversityEvanston, Illinois USA
Keynote Address to the 2nd World Conference on PersonalityBuzios, Brazil
April, 2016Slides at http://personality-project.org/sapa.html
Partially supported by a grant from the National Science Foundation: SMA-1419324
1 / 101
Open Science Open statistics, open materials, open methodology, open data Part I: Open Statistics: R
Outline
Two alternative titles:1. Personality at three levels of analysis.
1.1 Within individuals1.2 Between individuals1.3 Between groups of individuals
2. Personality research: an open and shared science
The importance of open science for personality research can beshown at each of these three levels of analysis.
1. Open source statistics, e.g.,The R project
2. Open source materials, e.g., IPIP and ICAR
3. Open source methodology, e.g., Synthetic AperturePersonality Assessment (SAPA)
4. Open source data, e.g., Journal of Open Psychology Dataand DataVerse
2 / 101
Open Science Open statistics, open materials, open methodology, open data Part I: Open Statistics: R
Open Science
• Science is an international collaborative endeavor thatbenefits when more people from more countries participate.
• Scientific societies were started (e.g, the Royal Society inLondon in 1660) as an “invisible college” to facilitatecommunication and the sharing of ideas.
• Traditionally we collaborate by publishing our results inscientific journals and by sharing our ideas at national andinternational conferences or giving guest lectures to ourcolleagues.
• More recently, there is a trend towards sharing our materials,our methods, and our results, even our data, on the web.
• This makes for better science.
3 / 101
Open Science Open statistics, open materials, open methodology, open data Part I: Open Statistics: R
Open Science and the problem of replication
• The last several years has seen a plethora of papers reportingfailures to replicate results. This has lead some to worry aboutthe strength of our findings and others to question what doesit mean to “replicate” or reproduce a result.
• Others have suggested that we should be more open in ourdesigns, publishing what we plan to do independent of whatwe actually find.
• This is an important problem that should not be ignored,although pre-registering might inhibit exploratory research.
• But, open science is much more than protecting us from type Ierrors. It is a philosophy of collaboration. That is what I wantto emphasize today.
4 / 101
Open Science Open statistics, open materials, open methodology, open data Part I: Open Statistics: R
A Short History of Science: Instrumentation and Modeling
New methods of data collection led to new models and theories.Instrumentation
1. Telescopes (Galileo)
2. Sailing ships (e.g., Beagle)
3. Depth sounders
4. CO2 measurement (Keeling)
5. The internet (Al Gore?)
6. WWW (Tim Berners-Lee)allows for remote assesment
7. Cellphones (e.g., SteveJobs) allow for repeatedmobile assessment
Theories and Models
1. Newton (Principa )
2. Darwin/Wallace
3. Plate Tectonics
4. Climate change
5. Open Science
6. Modeling the data acrossvery large sample
7. Modeling within subjectvariation over time (processmodels)
5 / 101
Open Science Open statistics, open materials, open methodology, open data Part I: Open Statistics: R
A Short History of Science: Part 2: Mathematics and Statistics
1. Calculus (Newton/Leibniz)2. Data visualization (Playfair to Tukey to Cleveland to Tufte)3. Probability theory (Fermat/Pascal) and the normal curve
(Gauss/Quetelet)4. Correlation (Galton/Pearson)5. Factor analysis (Spearman/Thurstone) and Principal
Components analysis (Pearson/Hotelling)6. Discrete (experimental) conditions and the t and F (seeds x
manure) distributions (Gossett and Fisher)7. Main frame computation (Ada Lovelace, John von Neumann,
Grace Hopper)8. Randomization and resampling (empirical distributions, not
idealized)9. Open statistical software for all of us, e.g., R.
We have new methods for collecting, analyzing and sharing ourdata, lets use them to improve our science.
6 / 101
Open Science Open statistics, open materials, open methodology, open data Part I: Open Statistics: R
Four types of openness:
1. Open source software: The R project (R Core Team, 2015)
2. Open source materials:• The International Personality Item Pool (IPIP) (Goldberg, 1999)
• The International Cognitive Ability Resource (ICAR) (Condon &
Revelle, 2014)
3. Open source methodology: The Synthetic AperturePersonality Assessment Project (Revelle, Wilt & Rosenthal, 2010; Revelle, Condon,
Wilt, French, Brown & Elleman, 2016)
4. Open source data:• Data from the ICAR project (Condon & Revelle, 2016, 2015a)
• Data from SAPA studies (Condon & Revelle, 2015c,b)
In the process of summarizing the last several years of mystudents and my research, I will show how we use open sourcesoftware, items, and methods and then share them with the world.
7 / 101
Open Science Open statistics, open materials, open methodology, open data Part I: Open Statistics: R
Four types of openness:
1. Open source software: The R project (R Core Team, 2015)
2. Open source materials:• The International Personality Item Pool (IPIP) (Goldberg, 1999)
• The International Cognitive Ability Resource (ICAR) (Condon &
Revelle, 2014)
3. Open source methodology: The Synthetic AperturePersonality Assessment Project (Revelle et al., 2010, 2016)
4. Open source data:
• Data from the ICAR project (Condon & Revelle, 2016, 2015a)
• Data from SAPA studies (Condon & Revelle, 2015c,b)
In the process of summarizing the last several years of research, Iwill show how we use open source software, items, and methodsand then share them with the world.
8 / 101
Part I: Open Statistics: R R What is R Use R for replications and extensions Getting and using R Part II: Open Materials
Part I Open Statistics: R
Part I: Open Statistics: R
R: open source statistical system
What is R
Use R for replications and extensions
Getting and using RUseful packages
Part II: Open Materials
9 / 101
Part I: Open Statistics: R R What is R Use R for replications and extensions Getting and using R Part II: Open Materials
10 / 101
Part I: Open Statistics: R R What is R Use R for replications and extensions Getting and using R Part II: Open Materials
R: What is it?
1. R: An international collaboration for applied statisticalresearch
• Originally developed in New Zealand in 1991-93• Comprehensive R Archive (CRAN) run out of Vienna• Core R members in Austria (2), Canada, Denmark, France,
Germany (2), India, New Zealand (3), Switzerland, US (6), UK
2. R: The open source - public domain version of S+
3. R: Written by statisticians (and some of us) for statisticians(and the rest of us)
4. R: Not just a statistics system, also an extensible language.• This means that as new statistics are developed they tend to
appear in R far sooner than elsewhere.• R facilitates asking questions that have not already been
asked.• “Easy” to learn, harder to master.
11 / 101
Part I: Open Statistics: R R What is R Use R for replications and extensions Getting and using R Part II: Open Materials
Statistical Programs for Psychologists
• General purpose programs• R• S+• SAS• SPSS• STATA• Systat
• Specialized programs• Mx• EQS• AMOS• LISREL• MPlus• Your favorite program
12 / 101
Part I: Open Statistics: R R What is R Use R for replications and extensions Getting and using R Part II: Open Materials
Statistical Programs for Psychologists
• General purpose programs• R• $+• $A$• $P$$• $TATA• $y$tat
• Specialized programs• Mx (OpenMx is part of R)• EQ$• AMO$• LI$REL• MPlu$• Your favorite program
13 / 101
Part I: Open Statistics: R R What is R Use R for replications and extensions Getting and using R Part II: Open Materials
R: A brief history
• 1991-93: Ross Dhaka and Robert Gentleman begin work onR project for Macs at U. Auckland (S for Macs).
• 1995: R available by ftp under the General Public License.• 2001-2016: Core team continues to improve base package
with a new release every 6 months (now more like yearly).• Many others contribute “packages” to supplement the
functionality for particular problems.• 2003-04-01: 250 packages• 2004-10-01: 500 packages• 2007-04-12: 1,000 packages• 2009-10-04: 2,000 packages• 2011-05-12: 3,000 packages• 2012-08-27: 4,000 packages• 2014-05-16: 5,547 packages (on CRAN) + 824 bioinformatic packages on
BioConductor• 2015-05-20 6,678 packages (on CRAN) + 1,024 bioinformatic packages +
?,000s on GitHub/R-Forge• 2016-03-31 8,182 packages (on CRAN) + 1,104 bioinformatic packages +
?,000s on GitHub/R-Forge (increased by 60 in last 10 days)
14 / 101
Part I: Open Statistics: R R What is R Use R for replications and extensions Getting and using R Part II: Open Materials
Rapid and consistent growth in packages contributed to R
15 / 101
Part I: Open Statistics: R R What is R Use R for replications and extensions Getting and using R Part II: Open Materials
R as a way of facilitating replicable science
1. R is not just for statisticians, it is for all research orientedpsychologists.
2. R scripts are published in psychology journals to show newmethods:
• Psychological Methods• Psychological Science• Journal of Research in Personality
3. R based data sets are now accompanying journal articles:• The Journal of Research in Personality now accepts R code
and data sets.• JRP special issue in R.• The replicability project has released its data and R scripts.
4. By sharing our code and data the field can increase thepossibility of doing replicable science.
16 / 101
Part I: Open Statistics: R R What is R Use R for replications and extensions Getting and using R Part II: Open Materials
Reproducible Research: Sweave and KnitR
Sweave is a tool that allows to embed the R code forcomplete data analyses in LATEXdocuments. The purposeis to create dynamic reports, which can be updatedautomatically if data or analysis change. Instead ofinserting a prefabricated graph or table into the report,the master document contains the R code necessary toobtain it. When run through R, all data analysis output(tables, graphs, etc.) is created on the fly and insertedinto a final LATEXdocument. The report can beautomatically updated if data or analysis change, whichallows for truly reproducible research.
Friedrich Leisch (2002). Sweave: Dynamic generation of statistical reports using literate data analysis. I
Supplementary material for journals can be written inSweave/KnitR so that others can redo or extend the analyses.
17 / 101
Part I: Open Statistics: R R What is R Use R for replications and extensions Getting and using R Part II: Open Materials
What is so great about reproducible research?
1. Allows us to share methods with our collaborators.
2. This can be other labs who want to know what you did. It canbe your students, it can even be you.
3. David Condon has suggested that your closest collaborator isyou, six months ago, but you don’t answer your emails.
4. That is, scripted analyses are for you.
5. The Reproducibility Project (https://osf.io/ezcuj/ hasreleased their 100 replication data set and the R code toanalyze it. If any one finds errors or needs more information,they are happy to provide it.
6. See, for instance Dan Gilbert et al. critique (Gilbert, King, Pettigrew & Wilson,
2016) and the response (Anderson, Bahnık, Barnett-Cowan, Bosco, Chandler, Chartier & Cheung,
2016). Also see the recent denial of the need to worry aboutreplication by Roy Baumeister.
18 / 101
Part I: Open Statistics: R R What is R Use R for replications and extensions Getting and using R Part II: Open Materials
Misconception: R is hard to use
1. R doesn’t have a GUI (Graphical User Interface)• Partly true, many use syntax.• Partly not true, GUIs exist (e.g., R Commander, R-Studio).• Quasi GUIs for Mac and PCs make syntax writing easier.
2. R syntax is hard to use• Not really, unless you think an iPhone is hard to use.• Easier to give instructions of 1-4 lines of syntax rather than
pictures of menu after menu to pull down.• Keep a copy of your syntax, modify it for the next analysis.
3. R is not user friendly: A personological description of R• R is Introverted: it will tell you what you want to know if you
ask, but not if you don’t ask.• R is Conscientious: it wants commands to be correct.• R is not Agreeable: its error messages are at best cryptic.• R is Stable: it does not break down under stress.• R is Open: new ideas about statistics are easily developed.
19 / 101
Part I: Open Statistics: R R What is R Use R for replications and extensions Getting and using R Part II: Open Materials
What makes R so powerful are the > 8, 100 contributed packages
psych A general purpose toolkit for psychological researchwith a particular emphasis upon• Basic descriptive statistics and basic graphical
tools.• Basic psychometric procedures including
functions for finding α (= λ3), ωh, and ωt• More advanced data reduction techniques using
factor analysis, principal components analysis,and cluster analysis.
• Introductory Item Response Theory andMulti-level modeling
lavaan Basic and advanced structural equation modeling(“The gateway package to R”).
sem Structural equation modelinglme4 Multilevel modeling.
20 / 101
Part I: Open Statistics: R R What is R Use R for replications and extensions Getting and using R Part II: Open Materials
Short courses and workshops emphasize training in basic andadvanced R
1. Symposia• International Society for Study of Individual Differences (2005)• 1st World Conference on Personality (2013)• Society for Personality and Social Psychology (2015)
2. Short courses• European Conference on Personality (2012, 2014)• Association for Research on Personality (2012)• Association for Psychological Science (2013, 2014, 2015,
2016)• 1st World Conference on Personality (2013)• STuP 2015 preconference: Confirmatory factor analysis in the
lavaan package (R)
3. Summer schools• Summer school sponsored by ISSID, EAPP and SMEP (2014)
21 / 101
Part II: Open Materials
Part II: Open Materials
Part II: Open Materials
22 / 101
Part II: Open Materials
Four types of openness:
1. Open source software: The R project (R Core Team, 2015)
2. Open source materials:• The International Personality Item Pool (IPIP) (Goldberg, 1999)
• The International Cognitive Ability Resource (ICAR) (Condon &
Revelle, 2014)
3. Open source methodology: The Synthetic AperturePersonality Assessment Project (Revelle et al., 2010, 2016)
4. Open source data:
• Data from the ICAR project (Condon & Revelle, 2016, 2015a)
• Data from SAPA studies (Condon & Revelle, 2015c,b)
In the process of summarizing the last several years of research, Iwill show how we use open source software, items, and methodsand then share them with the world.
23 / 101
Part II: Open Materials TAI IPIP ICAR Part III: Open Methods
Part II: Open Materials
Part II: Open Materials
Temperament, Abilities, and Interests: considering appetites andaptitudes
Temperament, Abilities, and Interests: TAI
IPIP: The International Personality Item PoolLew Goldberg and the development of the IPIPExtending the IPIP to include more domains
ICAR: International Cognitive Ability ResourceAn international collaboration to measure ability with opensource itemsAnalysis of ICAR items
Part III: Open Methods
24 / 101
Part II: Open Materials TAI IPIP ICAR Part III: Open Methods
Personality, prediction, and life outcomes
1. It has long been known that to predict real world outcomes weneed to study more than just ability (Kelly & Fiske, 1950, 1951; Deary, 2008;
Roberts, Kuncel, Shiner, Caspi & Goldberg, 2007).
2. Level of education and jobs differ in their intellectualrequirements (Gottfredson, 1997).
3. My colleagues and I have shown that there are alsotemperamental requirements for educational and job choice(Condon & Revelle, 2014; Revelle & Condon, 2012, 2015a; Revelle, Wilt & Condon, 2011; Wilt & Revelle, 2015)
4. We consider individual differences in Temperament, Ability,and Interests (TAI) as they relate to niche selection in choiceof college major and in occupational choice (Bouchard, 1997; Hayes, 1962;
Johnson, 2010). as well as to the second and third level ofpersonality analysis (between individuals and between groupsof individuals (Revelle & Condon, 2015b)).
25 / 101
Part II: Open Materials TAI IPIP ICAR Part III: Open Methods
Measuring Affect, Behavior, Cognition and Desire within subjects
1. Personality is not just how people differ from each other, it isalso how people differ in their Affect, Behavior, Cognition andDesire (ABCDs) over time and space.
2. This can be done using simple text messaging technology, ormore deligtfully elegant systems such as the Big EAR (Mehl,
Pennebaker, Crow, Dabbs & Price, 2001; Mehl, Vazire, Ramirez-Esparza, Slatcher & Pennebaker, 2007; Mehl &
Robbins, 2012) or Ryne Sherman’s “Big Eye”.
3. Most potential subjects carry around a very powerful datarecorder: their cell phone.
4. Apps have been developed for smart phones to measure mostanything
5. Even dumb phones can receive text messages.
26 / 101
Part II: Open Materials TAI IPIP ICAR Part III: Open Methods
Measuring the ABCDs with text messaging
1. In the past, if you wanted to measure the within personstructure of affect, you needed to use paper and pencil diaries(Bolger, Davis & Rafaeli, 2003).
• This was tedious for the participant• Difficult for the investigator.
2. Josh Wilt, Katie Funkhouser, Wiebke Bleidorn and I haveused cell phone text messaging to measure affect, behavior,and meaning over time (Wilt, Funkhouser & Revelle, 2011; Wilt, Bleidorn & Revelle, 2016).
3. The basic observation is that the structure of affect withinpeople is stable, is predicable, and differs across people. (seealso (Rafaeli, Rogers & Revelle, 2007).
4. While for some people, Energetic Arousal is positivelycorrelated with Tense Arousal, for others they are orthogonal,and for yet others they are negatively correlated. Thiscorrelation is, itself, predictable.
27 / 101
Part II: Open Materials TAI IPIP ICAR Part III: Open Methods
Structure of Affect within subjects varies by orientation to world
28 / 101
Part II: Open Materials TAI IPIP ICAR Part III: Open Methods
Measuring individual differences1. A basic problem in the study of individual differences is that
there are so many different constructs that interest us. Theseinclude constructs from at least four broad domains
• Temperament• Ability• Interests• Character
2. Each domain has many constructs:• Dimensions of Temperament 2-3-5-6-15?• Structure of Ability (g - gf , gc , V-P-R)?• Hierarchical structure of interests people-things, RIASEC .• Range of possible measures of character.
3. But many important measures are proprietary.4. In addition, showing the utility of TAIC measures requires
criterion variables, and should include demographics.5. Our solution: Use and/or develop open source temperament,
ability, and interest items.29 / 101
Part II: Open Materials TAI IPIP ICAR Part III: Open Methods
The International Personality Item Pool (Goldberg, 1999)
1. Perhaps one of the greatest contributions from Lew Goldberghas been his release of the International Personality Item Poolor IPIP (Goldberg, 1999) s http://ipip.ori.org.
2. The IPIP adapted a short stem item format developed in thedoctoral dissertation of Hendriks (1997) and items from theFive Factor Personality Inventory developed in Groningen(Hendriks, Hofstee & De Raad, 1999).
3. Goldberg (1999) used about 750 items from the Englishversion of the Groningen inventory, and has sincesupplemented them with many more new items in the sameformat.
4. The IPIP items have been translated into at least 39languages by at least 65 different research teams. Thisincludes Arabic, German, Farsi, Icelandic, Indonesian,Japanese, Korean, Mandarin, Polish, Portuguese, Russian,Serbian, Spanish, Turkish, Urdu, Slovenian and Swedish.
30 / 101
Part II: Open Materials TAI IPIP ICAR Part III: Open Methods
IPIP and other personality inventories
1. The IPIP was originally meant to be short stems to measurethe Abridged Five Factor Circumplex structure of adjectives(Hofstee, de Raad & Goldberg, 1992) but also includes items targeted at mostmajor personality tests.
2. Using a panel of roughly 1000 residents fromEugene-Springfield, Oregon, Goldberg administered hisoriginal IPIP items along with the NEO-PI-R (Costa & McCrae, 1992), theCPI (Gough & Bradley, 1996), the 16PF (Cattell & Stice, 1957), the MPQ (Tellegen &
Waller, 2008), the Hogan PI (Hogan & Hogan, 1995), the TCI (Cloninger, Przybeck & Svrakic,
1994), the JPI-R (Jackson, 1983), and the 6FPQ (Jackson, Paunonen & Tremblay, 2000).3. Goldberg then developed item stems that were highly
correlated to the commercial inventories and put these intothe public domain with the formation of the IPIP.
4. The items are available at http://ipip.ori.org and theEugene-Springfield data are available from Goldberg.
31 / 101
Part II: Open Materials TAI IPIP ICAR Part III: Open Methods
What are the “Big 5”?: Some representative itemsSemantic analysis of many (although primarily European)languages suggest 5 broad factors of the ways in which wedescribe others.
Conscientiousness Complete my duties as soon as possible. Dothings according to a plan. Like order.
Agreeableness Take advantage of others. (R) Am concernedabout others. Sympathize with others’ feelings.
Neuroticism Get upset easily. Get overwhelmed by emotions.Have frequent mood swings.
Openness/Intellect Am able to come up with new and differentideas. Am full of ideas. Have a rich vocabulary.
Extraversion Like mixing with people. Enjoy meeting new people.Am a talkative person. Am rather lively.
These are sometimes organized as the OCEAN of personality,alternatively, the CANOE of personality.
32 / 101
Part II: Open Materials TAI IPIP ICAR Part III: Open Methods
Extending the IPIP
1. In addition to the basic temperament items at the IPIP site,there are additional items to measure vocational interests (theORVIS) (Pozzebon, Visser, Ashton, Lee & Goldberg, 2010) as well as avocationalinterests (Goldberg, 2010).
2. David Condon has expanded 2500 IPIP items to include theoriginal IPIP items, the ORVIS, the ORAIS, as well as itemsfrom the EPQ (Eysenck, Eysenck & Barrett, 1985), the O*NET interest profilescales (Rounds, Su, Lewis & Rivkin, 2010). These, and other items make atotal set of 4,300 items.
3. These are available athttps://sapa-project.org/MasterItemList/.
4. Condon has also noted that although 18 different inventories(with 168 scales) have what appear to be 1,894 items, thereare actually just 696 unique items. In addition, those “magic696” cover between 57% to 85% of 10 additional inventorieswith 235 additional scales.
33 / 101
Part II: Open Materials TAI IPIP ICAR Part III: Open Methods
The International Cognitive Ability ResourceICAR: Extending the IPIP to ability: IPIP:Personality::ICAR:Ability
1. ICAR is an international collaboration to develop open sourcecognitive ability items.
2. Information at http://www.icar-project.com/3. News letter at http:
//www.icar-project.com/ICAR_News_Issue_One.pdf
4. Key organizers who are coordinating the project:Germany Phillip Doebler (Munster and Ulm) and Heinz
Holling (Munster)U.K. Luning Sun and John Rust (Cambridge)
U.S.A William Revelle and David Condon(Northwestern)
5. Everyone is welcome to join this international collaboration.6. Supported by Open Research Area (ORA) for the Social
Sciences which includes participation from national fundingagencies (Germany:DFG), (UK:ESRC), (US:NSF).
34 / 101
Part II: Open Materials TAI IPIP ICAR Part III: Open Methods
ICAR: Proof of concept1. About 60 items were developed as part of a honors thesis at
Northwestern by Melissa Liebert (Liebert, 2006) and meant to be“Google resistant” (answers are not available on the web).
• This set was reported at conference in Krakow and in asubsequent book chapter (Revelle et al., 2010).
2. Subsequently David Condon developed some 3 Dimensionalrotations items and did some extensive item analysis of thetotal set.
3. Condon & Revelle (2014) examined the first 60 publiclyavailable items and validated them against self reported SATexam scores as well as a small sample given the Shipley-2(Shipley, 2009).
4. The original data set has been released to DataVerse (Condon &
Revelle, 2015a) and has been published in the Journal of OpenPsychology Data (Condon & Revelle, 2015c).
5. An example data set of 16 items with N = 1, 525 is includedas the ability dataset in the psych package.
35 / 101
Part II: Open Materials TAI IPIP ICAR Part III: Open Methods
Sample ICAR itemsMatrix Reasoning Verbal Reasoning
What number is one fifth of one fourth of one ninth of 900?
(1) 2 (2) 3 (3) 4 (4) 5 (5) 6 (6) 7
If the day after tomorrow is two days before Thursday,
then what day is it today?
(1) Friday (2) Monday (3) Wednesday
(4) Saturday (5) Tuesday (6) Sunday
Letter and Number SeriesIn the following alphanumeric series, what letter comes next?
I J L O S
(1) T (2) U (3) V (4) X (5) Y (6) Z
In the following alphanumeric series, what letter comes next?
Q S N P L
(1) J (2) H (3) I (4) N (5) M (6) L
Three-Dimensional Rotation
36 / 101
Part II: Open Materials TAI IPIP ICAR Part III: Open Methods
Sample analysis of ICAR items
1. Using basic R functions in the psych package (Revelle, 2015) we canevaluate the factor structure of the ICAR items.
2. irt.fa will do a factor analysis of the items and report thestatistics in terms of those statistics more commonly used inItem Response Theory.
• The two parameters from factor analysis are item difficultytaken from the τ parameter from the tetrachoric correlationand the item factor loading λ of the matrix of tetrachoriccorrelations.
a =λ√
1− λ2δ =
τ√1− λ2
• The hierarchical structure of the ability items may be shown byfactoring the factor intercorrelations.
• Loadings on a general factor may then be found by using theomega function which applies a Schmid Leiman transformationto the resulting higher level solution.
37 / 101
Part II: Open Materials TAI IPIP ICAR Part III: Open Methods
Item information curves for the 16 ICAR sample set
-3 -2 -1 0 1 2 3
0.0
0.2
0.4
0.6
0.8
1.0
Item information from factor analysis
Latent Trait (normal scale)
Item
Info
rmat
ion
reason.4
reason.16
reason.17
reason.19
letter.7
letter.33
letter.34letter.58
matrix.45matrix.46
matrix.47
matrix.55
rotate.3
rotate.4
rotate.6
rotate.8
38 / 101
Part II: Open Materials TAI IPIP ICAR Part III: Open Methods
Structure of sample ICAR 16 items shows a clear 4 factor hierarchicalsolution ωh = .87
Omega Hierarchical for ICAR Sample Test
LN.07
LN.34
LN.33
LN.58
R3D.03
R3D.08
R3D.04
R3D.06
MR.45
MR.46
MR.55
MR.47
VR.17
VR.04
VR.16
VR.19
LetterNum
0.80.70.60.5
3D Rotate
0.8
0.8
0.7
0.5
Matrices 0.70.70.50.4
Verbal R0.8
0.7
0.6
0.5
g
0.9
0.7
0.9
0.9
39 / 101
Part II: Open Materials TAI IPIP ICAR Part III: Open Methods
Structure of ICAR 60 items shows a messier 4 factor hierarchicalsolution ωh = .76
Hierarchical structure of ICAR60 items
VR.18VR.39VR.14VR.17LN.01VR.42VR.19VR.09VR.04VR.16LN.03VR.26VR.36LN.34R3D.07R3D.02R3D.17R3D.24R3D.18R3D.14R3D.19R3D.13R3D.12R3D.11R3D.01R3D.03R3D.21R3D.04R3D.08R3D.15R3D.10R3D.23R3D.09R3D.16R3D.05R3D.22R3D.06R3D.20MR.48MR.46MR.45MR.56MR.53MR.55MR.44MR.47MR.43LN.05LN.33MR.50LN.06LN.07MR.54LN.58LN.35VR.23VR.13VR.31VR.32VR.11
Verbal R
10.90.80.60.60.50.50.50.50.50.40.40.40.3
0.40.30.3-0.3
3D Rotate
0.80.70.70.70.70.70.70.70.70.70.60.60.60.60.60.60.60.60.50.50.50.50.40.4Matrices
0.30.3
0.3
0.60.60.60.60.60.60.60.50.50.50.50.40.40.40.40.40.30.3LetterNum
0.30.30.3
0.3
0.3
0.3
0.3
g
0.8
0.6
1
0.4
40 / 101
Part II: Open Materials TAI IPIP ICAR Part III: Open Methods
Open materials
1. The International Personality Item Pool items (Goldberg, 1999) aswell as the extended IPIP are in the public domain and areavailable to anyone for free.
2. The items from the International Cognitive Ability resource arealso in the public domain and are available to registeredusers. (We are trying to keep the items relatively secure anddo not put all of the actual items up on the web.)
• We have a basic set of 60 ICAR items (Condon & Revelle, 2014) and theICAR group is developing and validating item generators toautomatically produce hundreds of each of a growing numberof item types.
• We encourage others to join us in this mission.• Go to http://www.icar-project.com/ to register and for
more information.
41 / 101
Part III: Open Methods SAPA SAPA theory SAPA: practice Part IV: Open Data
Part III: Open Methods
Method: Synthetic Aperture Personality Assessment (SAPA)Measuring individual differences: the tradeoff between breadthversus depthSynthetic Aperture Astronomy
SAPA theorySample items as well as peopleCovariance algebra
SAPA: practiceOpen source software comes to the rescue
Part IV: Open Data
42 / 101
Part III: Open Methods SAPA SAPA theory SAPA: practice Part IV: Open Data
Four types of openness:
1. Open source software: The R project (R Core Team, 2015)
2. Open source materials:• The International Personality Item Pool (IPIP) (Goldberg, 1999)
• The International Cognitive Ability Resource (ICAR) (Condon &
Revelle, 2014)
3. Open source methodology: The Synthetic AperturePersonality Assessment Project (Revelle et al., 2010, 2016)
4. Open source data:
• Data from the ICAR project (Condon & Revelle, 2016, 2015a)
• Data from SAPA studies (Condon & Revelle, 2015c,b)
In the process of summarizing the last several years of research, Iwill show how we use open source software, items, and methodsand then share them with the world.
43 / 101
Part III: Open Methods SAPA SAPA theory SAPA: practice Part IV: Open Data
Measuring individual differences
1. A basic problem in the study of individual differences is thatthere are so many different constructs that interest us. Theseinclude constructs from at least four broad domains
• Temperament• Ability• Interests• Character
2. Each domain has many constructs• Dimensions of Temperament 2-3-5-6-?• Structure of Ability: g - gf , gc , V-P-R ?• Hierarchical structure of interests people-things RIASEC• Range of possible measures of character
3. In addition, showing the utility of TAIC measures requirescriterion variables
44 / 101
Part III: Open Methods SAPA SAPA theory SAPA: practice Part IV: Open Data
Breadth vs. depth of measurement
1. Factor structure of domains needs multiple constructs todefine structure.
2. Each construct needs multiple items to measure reliably.
3. This leads to an explosion of potential items .
4. But, people are willing to only answer a limited number ofitems.
5. This leads to the use of short and shorter forms (theNEO-PI-R with 300, the IPIP big 5 with 100, the BFI with 44items, the TIPI with 10) to include as part of other surveys.
45 / 101
Part III: Open Methods SAPA SAPA theory SAPA: practice Part IV: Open Data
Many items versus many people
1. Not only do want many items, we also want many people.
2. Resolution (fidelity) goes up with sample size, N (standarderrors are a function of
√N)
σx =σx√
N − 1σr =
1− r2√
N − 2
3. Also increases as number of items, n, measuring eachconstruct (reliability as well as signal/noise ratio varies asnumber of items and average correlation of the items)
λ3 = α =nr
1 + (n − 1)rs/n =
nr(1− nr)
4. Thus, we need to increase N as well as n. But how?
46 / 101
Part III: Open Methods SAPA SAPA theory SAPA: practice Part IV: Open Data
A short diversion: the history of optical telescopes
Resolution varies by aperture diameter (bigger is better)
47 / 101
Part III: Open Methods SAPA SAPA theory SAPA: practice Part IV: Open Data
A short diversion: history of radio telescopesResolution varies by aperture diameter (bigger is still better)
Aperture can be synthetically increased across multiple telescopesor even multiple observatories
48 / 101
Part III: Open Methods SAPA SAPA theory SAPA: practice Part IV: Open Data
Can we increase N and n at the same time?
1. Frederic Lord (1955) introduced the concept of samplingpeople as well as items.
2. Apply basic sampling theory to include not just people (wellknown) but also to sample items within a domain (less wellknown).
3. Basic principle of Item Response Theory and tailored tests.
4. Used by Educational Testing Service (ETS) to pilot items.
5. Used by Programme for International Student Assessment(PISA) in incomplete block design (Anderson, Lin, Treagust,Ross & Yore, 2007).
6. Can we use this procedure for the study of individualdifferences without being a large company?
7. Yes, apply the techniques of radio astronomy to combinemeasures synthetically and take advantage of the web.
49 / 101
Part III: Open Methods SAPA SAPA theory SAPA: practice Part IV: Open Data
Subjects are expensive, so are items
1. In a survey such as Amazon’s Mechanical Turk (MTURK), weneed to pay by the person and by the item.
2. Why give each person the same items? Sample items, as wesample people.
3. Synthetically combine data across subjects and across items.This will imply a missing data structure which is
• Missing Completely At Random (MCAR), or even moredescriptively:
• Massively Missing Completely at Random (MMCAR)
4. This is the essence of Synthetic Aperture PersonalityAssessment (SAPA).
50 / 101
Part III: Open Methods SAPA SAPA theory SAPA: practice Part IV: Open Data
3 Methods of collecting 256 subject * items dataa) 8 x 32 complete4621363452114345344364533121241421243623166421516154432261516513516613511551654636222244356233441114134336233221561215213561452225353121264561433433232246526411613351545664241146126412253535162463434215153624242541351343511611554654453123111162423325516334
b) 32 x 8 complete4632311425443314433154232631414541435614422361536242134435234443345141666341515444441342135143216636566312264546314661353264551466151251144114416244363633316236633254251153112661155546332453615224165463212356244146636366141445555223143644332146141633232365
c) 32 x 32 MCAR p=.25..3..2..6.....4.55.......44................4..6..45..3.4..6....16..3.......6.1.....6.2.......5.6....3522......5.3...3......5........3.2.2.......3..2......65..5......51....324.........23......5....552............25...54.5.......44.4.5....3..6...6........3......61.523.2....2...........3...5.............42.4..6.5......61.....3....3.6..1.4...1..5......5.1....54..........2.4.33..6......4.....52..6.....44.3...........2..44...1........1..42....5..1.....1..3.......2..3.521.......6..........3.142.........22......12..4...2..........3..162...4.....4..4..6..3.4...1....5.33.........5..........243..5....41......1....5..3..4...4.4..5..1.........4......4.......3..5.2.....64.4..4....1.1.2...6....4......55....2.......3..2..53.....2..2.3.3............1...2..43...3.13........5....2.........4..54...2.3..62....22.......332..1.....5......6.......5..3.4.....3....5.241..............63.1.......6...5..4..2...5..2.4..5..........52.4.....44...2.55.....2.....6.....6.....55.....5..........4....6341.4..2.........55......5.......45....3..32.51 / 101
Part III: Open Methods SAPA SAPA theory SAPA: practice Part IV: Open Data
Synthetic Aperture Personality Assessment
1. Give each participant a random sample of pn items takenfrom a larger pool of n items.
2. Find covariances based upon “pairwise complete data”.3. Find scales based upon basic covariance algebra.
• Let the raw data be the matrix X with N observationsconverted to deviation scores.
• Then the item variance covariance matrix is C = XX ′N−1
• and scale scores, S are found by S = K ′X .• K is a keying matrix, with K ij = 1 if itemi is to be scored in the
positive direction for scale j, 0 if it is not to be scored, and -1 ifit is to be scored in the negative direction.
• In this case, the covariance between scales, Cs, is
Cs = K ′X (K ′X )′N−1 = K ′XX ′KN−1 = K ′CK . (1)
4. That is, we can find the correlations/covariances betweenscales from the item covariances, not the raw items.
52 / 101
Part III: Open Methods SAPA SAPA theory SAPA: practice Part IV: Open Data
SAPA standard errors and effective sample size
1. When forming synthetic scales from MMCAR based items, thestandard error of correlations decreases as a function of theTotal number of subjects (N), the percentage of items samples(p), and the number of items forming the scale (n).
2. Ashley Brown has shown this quite clearly by simulation (Brown,
2014).
3. A good way to visualize this is to examine the standard errorof correlations as a function of N, p, and n.
4. An even more dramatic way is to plot the Effective SampleSize (Neff ) which because
σr =1− r2√
N − 2is merely Neff =
(1− r2)2
σ2r
+ 2
53 / 101
Part III: Open Methods SAPA SAPA theory SAPA: practice Part IV: Open Data
Effective sample size varies by the size of the composite scale.Simulating N= 10,000 with probability of any item (Brown, 2014)
(p = .125, .25, .5, or 1) and items in the composite 1 , 2, 4, 8, 16.
Rw: 0.3
p=0.125
p=0.25
p=0.5
p=1
0
2500
5000
7500
1000010000
Rb: 0
1 2 4 8 16n (items per scale)
Effe
ctive
Sam
ple
Size
54 / 101
Part III: Open Methods SAPA SAPA theory SAPA: practice Part IV: Open Data
SAPA is not magic: We can obtain high accuracy at the structure levelbut accuracy is much lower at the single subject level
1. Reliability of composite scales is high when formed fromsynthetic matrices Cs = K ′CK because the number of itemsper scale/per subject is the nominal amount.
2. Reliability of single scores is much less because very fewitems measuring a single trait are given to a single subjectS = K ′X .
3. However, the precision of the estimate of subject means (x) ishigh because σx = σx√
Np−1 and Np is large.
4. SAPA technique is very powerful for research of structure, butless powerful for research based upon single subjects.
5. Particularly useful in web based surveys with many subjectswhen we are limited in the number of items we can administerand where we are trying to increase our domain validity.
55 / 101
Part III: Open Methods SAPA SAPA theory SAPA: practice Part IV: Open Data
How does it work?1. Give our basic belief in open science, we use public domain
items, open source software:• Apache webserver, MySQL data bases, PHP and HTML5 web
tools, R for statistics.• Extensive coding in PHP and MySQL to present item sets in
random fashion (Joshua Wilt, David Condon, Jason French)• Code written for psychometric measurement and scale
construction as implemented in the psych package (Revelle, 2015)
using R (R Core Team, 2015)
2. Domains measured and item sources• Temperament items taken from International Personality Item
Pool (IPIP) (Goldberg, 1999) (ipip.ori.org) and supplemented withother items.
• Ability items have been validated (Condon & Revelle, 2014) as part of theInternational Cognitive Ability Resource Project(ICAR-project.org). (ICAR:Ability::IPIP:Temperament)
• Interest items taken from Oregon Vocational Interest Survey(ORVIS) (Pozzebon et al., 2010)
56 / 101
Part III: Open Methods SAPA SAPA theory SAPA: practice Part IV: Open Data
SAPA overview
1. A “Personality Test” is included as a resource at thehttp://personality-project.org and gives feedback toall participants.
2. Some participants then link their feedback to their socialmedia sites which then appeals to yet more to take it.
3. Some professors assign it to their students in various classes.
4. About 120-180 people per day from around the world visit thepersonality-project.org or sapa-project.orgwebsites. This does not sound like much, but over a year, weget around 40,000-50,000 participants.
5. This is much less than the 6 ∗ 106 subjects that Gosling andothers are reporting, but we are talking about structure basedupon 700-1000 items instead of a limited 44 BFI items.
6. This the benefit of random sampling.
57 / 101
Part III: Open Methods SAPA SAPA theory SAPA: practice Part IV: Open Data
10/28/15, 12:43 PMThe SAPA-Project: Explore Your Personality
Page 1 of 2https://sapa-project.org/
FAQ about the testIs it long? (not really) Is it free? (yes)
We won't sort you into a house or match
you to a harmonious date. But we will
give you feedback based on modern
psychological theory. Learn more...
The research behind SAPAHow was the test developed?
Each customized report is generated on
the basis of participant's responses, but
each participant gets a slightly different
subset of all 3,800 items. Learn more...
Individual DifferencesLearn more about differential psychology.
Why do individuals differ in the ways they
think, feel, and act? How do people differ
in response to the same situations? Learn
why individual differences matter...
Temperament Ability
Another personality test?Well, yes, but we like to think ours is a little more sophisticated than most.
The SAPA Project is a collaborative data collection tool for assessingpsychological constructs across multiple dimensions of personality. Thesedimensions currently include temperament, cognitive abilities, and interestsbecause research suggests that these three dimensions cover a large percentageof the variability among people without too much redundacy. Other broaddimensions like motivation and character may be included later depending onhow much incremental benefit they provide in terms of predicting outcomes.
The SAPA ProjectTake the test.Explore your personality.Advance the study of individualdifferences.
Start the test
More info
58 / 101
Part III: Open Methods SAPA SAPA theory SAPA: practice Part IV: Open Data
How does it work?: part II
1. Participants find us by searching web for “personality tests”,etc. and find personality-project.org orsapa-project.org
2. Each participant is given a number of web pagesConsent Form Basic description of project and question
whether they have taken test before.Demographics Age, sex, height, weight, education, parental
education, country, state, ZipCode (if US), ...TAIC questions Temperament/Ability/Interest questions (25
per page, 21 T/I, 4 Ability per pageContinuation pages After each page, told that feedback will
be more accurate if they keep going.Optional modules Creativity, Peer ratings, interests, ...Feedback Personality feedback based upon scores on
temperament items.3. Results are stored (page by page) on the MySQL server.
59 / 101
Part III: Open Methods SAPA SAPA theory SAPA: practice Part IV: Open Data
How does it work: part III
1. Various data cleaning scripts run using the SAPA-toolspackage (French & Condon, 2015) in R.
• Screen for duplicate responses based upon a RandomIdentification Number issued when subjects start the page. Wedrop all subsequent pages.
• Screen for subjects < 14 or > 90.
2. Subsequent analyses are done primarily using functions inthe psych package (Revelle, 2015) for R.
3. Analyses are done at multiple levels:3.1 At the item and scale covariance level to examine the structure
of items3.2 At the multiple levels of aggregation: zip code, state, college
major, occupation. This requires finding individual level scoresand then examining the structure of group means throughbasic multi-level techniques.
60 / 101
Part IV: Open Data Demographics Structure TAI and niche selection Summary: Open Science
Part IV: Open Data
Part IV: Open Data
Demographics of the SAPA dataSample characteristicsWeb data are not random samples
Personality StructureAre the “Big Five” really big?Temperament, Ability and Interests,
Temperament, Ability and Interests: Occupational ChoicePooled correlations 6= within group or between groupcorrelationsOccupational Choice as niche selection
Summary: Open Science
61 / 101
Part IV: Open Data Demographics Structure TAI and niche selection Summary: Open Science
Four types of openness:
1. Open source software: The R project (R Core Team, 2015)
2. Open source materials:• The International Personality Item Pool (IPIP) (Goldberg, 1999)
• The International Cognitive Ability Resource (ICAR) (Condon &
Revelle, 2014)
3. Open source methodology: The Synthetic AperturePersonality Assessment Project (Revelle et al., 2010, 2016)
4. Open source data:
• Data from the ICAR project (Condon & Revelle, 2016, 2015a)
• Data from SAPA studies (Condon & Revelle, 2015c,b)
In the process of summarizing the last several years of research, Iwill show how we use open source software, items, and methodsand then share them with the world.
62 / 101
Part IV: Open Data Demographics Structure TAI and niche selection Summary: Open Science
Our data
Time Frame Data collected at personality-project.org andsapa-project.org from August 18, 2010 toDecember 22, 2015
Subjects N = 207,002 (77,550 males, 129,451 females)Materials 953 items (696 temperament, 60 ability, 152
interests, 45 demographic)Scales used 15 Temperament, 4 Ability, 6 InterestsN in workforce N =80,317N students N = 86,348
Occupations 973 separate occupations, following a Paretodistribution with ≈ 80% represented by the top 20%of occupations
N ≥ 75 195 occupations for 55,902 participantsN ≥ 100 116 college majors for 130,584 participants
63 / 101
Part IV: Open Data Demographics Structure TAI and niche selection Summary: Open Science
The top 35 countries account for 89% of the sample
USACANGBRAUSINDDEUPHLMYSSWENLDNORMEXNZLSGPCHNBRAZAFFINFRAIRLPOLROUIDNHKGSRBPAKNGADNKITAKORTURCHEBELESPRUS
150000 160000 170000 180000
The top 35 countries account for 89% of the sample
1418398895
55774024
22421986198119261362959869856853794765668654625610547506500498488472440434433419346330329325316283
64 / 101
Part IV: Open Data Demographics Structure TAI and niche selection Summary: Open Science
Where do they come from? US SAPA data by zipcodes
65 / 101
Part IV: Open Data Demographics Structure TAI and niche selection Summary: Open Science
US lights (from NASA)
66 / 101
Part IV: Open Data Demographics Structure TAI and niche selection Summary: Open Science
Not a random sample of either education or gender (62% female)
Participants Education by Gender
count
Participants Education by Gender
60000
40000
20000
0
20000
40000
60000
<12 High Sch. In Coll <16 BA/BS In grad Grad/Pro
Female
Male
13% 7% 43% 5% 13% 5% 7%
67 / 101
Part IV: Open Data Demographics Structure TAI and niche selection Summary: Open Science
Not a random sample of either age or gender (62% female)
Participants' Age by Gender
count
Age
Participants' Age by GenderAge
020
4060
80
10000 5000 0 5000 10000
Male Female
68 / 101
Part IV: Open Data Demographics Structure TAI and niche selection Summary: Open Science
86,348 students, 80,317 employed)
Employment by Gendercount
Employment by Gender
40000
20000
0
20000
40000
Student Seeking work Employed Retired69 / 101
Part IV: Open Data Demographics Structure TAI and niche selection Summary: Open Science
Are the “Big Five” really big?
1. There is a “consensus” about the proper number offactors/components of personality (Goldberg, 1990, 1992; Hofstee et al., 1992).
2. This seems to match life challenges of Getting Along andGetting AheadConscientiousness WorkAgreeableness LoveNeuroticism Effective functioning in many domainsOpenness/Intellect PlayExtraversion Leadership
3. Additional work has been done on the same 800-1000 personEugene-Springfield sample and suggests a hierarchicalstructure (e.g., the BFAS) (DeYoung, Quilty & Peterson, 2007).
4. But what happens with a larger and more diverse sample?
70 / 101
Part IV: Open Data Demographics Structure TAI and niche selection Summary: Open Science
Multiple solutions to the dimensionality of temperament
• 2) Digman “alpha and beta” (Digman, 1997), DeYoung “stability andplasticity”(DeYoung, Peterson & Higgins, 2002)
• 3) Eysenck “Giant 3” (Eysenck, 1994) PEN• 5)The “Big 5” (Digman, 1990; Goldberg, 1990) CANOE/OCEAN/ I ... V (Norman,
1963; Tupes & Christal, 1961)
• 6)The HEXACO 6 (Lee & Ashton, 2004; Ashton, Lee & Goldberg, 2007)
• 7-9)Tellegen 7-9 (Tellegen & Waller, 2008)
• 8-9) Comrey 8-9 (Comrey, 2008)
• 16) Cattell 16 Personality Factors (Cattell, 1957)
1. Condon (2014, 2015) examined 696 non-overlapping itemsfrom IPIP:100, IPIP:NEO, IPIP:MSQ, BFAS, EPQ, etc. (Goldberg,
1999; DeYoung et al., 2007; Eysenck et al., 1985)
Found meaningful 3, 5, 15, (and now) 27 factor solutions.2. The Condon 3/5/15/27 form a heterarchical and non
hierarchical structure (i.e., lower levels are not cleanly nestedin higher levels.)
71 / 101
Part IV: Open Data Demographics Structure TAI and niche selection Summary: Open Science
Personality shows a heterarchical even fractal structure
1. David Condon (2014, 2015) and in prep has shown:
2. The structure of 696 personality items given to 100-200,000participants does not show a clean organization.
3. The number of factors problems (and its non-solutions) willbreak the heart of most investigators.
4. No clear structure at any level.
5. Equally compelling fits at 3/5/15/27 factors.
6. The larger the sample size, the greater the resolution andthere is always something more there.
7. Articles showing “incremental validity beyond the Big Five” arejust as likely to show incremental validity beyond any numberof factors.
8. We think of this as showing a fractal structure: equal level ofcomplexity at any level of resolution.
72 / 101
Part IV: Open Data Demographics Structure TAI and niche selection Summary: Open Science
The best items from the 15 scale solution
Table : Sample items from the Short Personality Inventory 15 factorsolution
Each scale has 8-10 itemsSPI Item ItemFear Panic easily. Begin to panic when there is danger.Volatility Get irritated easily. Lose my temper.Outlook Dislike myself. Feel a sense of worthlessness or hopelessness.Compassion Sympathize with others feelings. Am sensitive to the needs of others.Trust Trust others Trust what people say.Easygoing Let things proceed at their own pace. Take things as they come.Industrious Start tasks right away. Get chores done right away.Mach Use others for my own ends. Cheat to get ahead.Impulsivity Act without thinking. Do things without thinking of the consequences.Sociability Am mostly quiet when with other people. Tend to keep in the background on social occasions.Boldness Love dangerous situations. Take risks.Serious Seldom joke around. Am not easily amused.Conventional Don’t like the idea of change. Prefer to stick with things that I know.Intellectual Am quick to understand things. Catch on to things quickly.Open Enjoy thinking about things. Love to reflect on things.
73 / 101
Part IV: Open Data Demographics Structure TAI and niche selection Summary: Open Science
The correlations of 696 personality items
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●
● ●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
1920
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38 39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
5960
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152153
154
155
156
157
158
159
160
161
162163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238239
240
241
242
243
244
245
246
247
248
249
250
251
252
253254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497498
499
500
501
502
503
504
505506
507
508
509
510 511
512
513
514
515
516
517
518
519520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566 567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583 584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607 608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
74 / 101
Part IV: Open Data Demographics Structure TAI and niche selection Summary: Open Science
Factoring the items on the first factor of the 696
●●●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
8081
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260261
262
263
264
75 / 101
Part IV: Open Data Demographics Structure TAI and niche selection Summary: Open Science
And doing it again
1
2
3
4
5 6
7
8
9
10
11
12
13
14
15
16
17
18
1920
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
5556
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
7576
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
76 / 101
Part IV: Open Data Demographics Structure TAI and niche selection Summary: Open Science
And again
1 2
3
4
5
6
7
8
9
10
11 12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
3233
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
77 / 101
Part IV: Open Data Demographics Structure TAI and niche selection Summary: Open Science
6 factors of interests
1. 6 factors from the O*NET interest profiler scales (60 items; Rounds
et al., 2010)
2. 8 factor Oregon Vocational Interest Scales (92 items; Pozzebon et al., 2010)
3. Oregon Avocational Interest Scales (199 items; Goldberg, 2010)
4. Formed into 6 scales fitting a “RIASEC” structure (60 items)Realistic “Like to work with tools and machinery.”
Investigative “Would like to do laboratory tests to identifydiseases.”
Artistic “Would like to write short stories or novels.”Social “Would like to help conduct a group therapy
session.”Enterprising “Would like to be the chief executive of a large
company.”Clerical “ Would like to keep inventory records”
78 / 101
Part IV: Open Data Demographics Structure TAI and niche selection Summary: Open Science
The correlational structure of scales are found from the SAPA itemcorrelations
1. Given the raw data matrix, we can find the covariances (usingpairwise complete data) and the find the scaleintercorrelations.
2. The correlations with scales with overlapping items can becorrected for overlap scoreOverlap using a correction byBashaw & Anderson Jr (1967); Cureton (1966)
3. We can do this for 696 Temperament items, 60 Ability items,and 152 Interest items.
4. Although we have 150 different scales, I am reporting just IPIPbased Big 5, 4 ICAR ability scales, and the 6 RIASEC scales.
5. We can also drill down to the “Aspects Level” (DeYoung et al., 2007), the“facet level” (Costa, McCrae & Dye, 1991), the “nuances” (Mottus, 2016; ottus, Kandler,
Bleidorn, Riemann & McCrae, 2016) or items.
79 / 101
Part IV: Open Data Demographics Structure TAI and niche selection Summary: Open Science
Personality at 3 levels of analysis (Revelle & Condon, 2015b)
Personality can be examined at three levels of analysis1. Personality as a unique temporal signature of one’s Affect,
Behavior, Cognition and Desires (ABCDs) as they changeover time and space within a single individual.
• Measuring within person patterning requires repeatedmeasures on single subjects over time. We do this with opensource text messaging procedures e.g., (Wilt et al., 2011; Wilt, 2014).
2. Personality is also how people differ in their patterning of theABCDs between people.
• This can be multilevel modeling of data collected withinsubjects showing that the correlational structure withinsubjects differs across subjects (Wilt et al., 2011; Revelle & Wilt, 2015).
• It is also the more conventional structure of personality itemsas collected from the SAPA project.
3. But people choose groups such as college major oroccupation based upon their unique aptitudes and appetites.
• We can analyze this niche selection in terms of the covarianceof the mean personality of the group.
80 / 101
Part IV: Open Data Demographics Structure TAI and niche selection Summary: Open Science
TAI for groups is not the same as TAI for individuals
1. How do occupational groups or college majors differ on TAI?• The mean scores for groups allow us to compare the groups• But it is the structure of these group means that are particularly
interesting for they allow us to examine niche selection.
2. Overall correlation is a function of within group correlationsand between group correlations.
3. Correlations of aggregate scores rxybg (between groups) 6=aggregate of correlations rxywg (within groups)
4. The overall correlation rxy is a function of the within and thebetween correlationsrxy = etaxwg ∗ etaywg ∗ rxywg + etaxbg ∗ etaybg ∗ rxybg
5. These multi level correlations sometimes lead to what isknown as the Yule-Simpson paradox (Kievit, Frankenhuis, Waldorp & Borsboom,
2013; Simpson, 1951; Yule, 1903)
• These are independent and useful information.
81 / 101
Part IV: Open Data Demographics Structure TAI and niche selection Summary: Open Science
Temperament, Ability, and Interests – within and between groups
1. Examined the factor structure of the TAI scales at the normal,between subjects (across groups) level.
• This produces the normal factor structure of temperament, ofability and of interests
• Can show these correlations as a “heatmap”
2. But when analyzing the structure of the mean scores for eachof 148 majors or 196 occupational groups (minimum size of75 members), the structure is drastically different.
• Several dimensions of temperament and interests are nownegatively correlated with ability, others are orthogonal
• Can also show these correlations as a “heatmap”.• In that some correlations change sign, this is not an artifact of
aggregation.
82 / 101
Part IV: Open Data Demographics Structure TAI and niche selection Summary: Open Science
Subject Level data of 8 criteria, 5 personality scales, 4 ability, 6interests,
Within group correlations of Criteria, Temperament, Ability, Interests
ConventEnterpriSocialArtisticInvesti
RealisticVerbal3Drotmatrix
Let-NumICAR60
Open/IntelExtraStabAgreeConscincomeBMI
p2edup1edu
educationexercise
agegender
gender
age
exercise
education
p1edu
p2edu
BMI
income
Consc
Agree
Stab
Extra
Open/Intel
ICAR60
Let-Num
matrix
3Drot
Verbal
Realistic
Investi
Artistic
Social
Enterpri
Convent
-0.01 0.01 -0.02 0.02 -0.02 -0.01 0.02 0 0.03 0.01 -0.01 -0.01 0.02 0.09 0.08 0.08 0.07 0.07 0.21 0.15 -0.01 0.04 0.21 1
-0.06 -0.03 0.01 0 0 0.01 0 0 0.01 0.01 0.01 0.05 0.04 0.07 0.07 0.06 0.05 0.06 0.19 0.12 0.03 0.02 1
0.03 0.02 0 0.01 0 0 0.01 -0.01 0.01 0.1 0 0.05 0.04 -0.01 0 -0.01 -0.02 -0.01 -0.02 0.05 0.09 1
-0.01 0.02 0 0.02 0.02 0.02 -0.01 0.01 -0.01 0.02 0 0.01 0.07 0.01 0.01 0.01 0.03 0 0.1 0.1 1
-0.07 0 0 0.04 0.01 0.01 -0.01 0.02 -0.01 -0.02 0 -0.01 0.08 0.12 0.1 0.1 0.12 0.09 0.27 1
-0.11 0 0.01 0.02 0 0.01 0.02 0.01 -0.01 -0.01 0.01 -0.01 0.06 0.13 0.11 0.11 0.12 0.1 1
-0.07 0.03 -0.11 0.11 -0.07 0 0.06 0.02 -0.02 0.08 -0.07 -0.03 0.14 0.84 0.67 0.62 0.49 1
-0.1 0.02 -0.06 0.11 0 0.04 0 0.04 -0.02 0.04 -0.04 -0.02 0.1 0.75 0.59 0.57 1
-0.07 0.02 -0.12 0.1 -0.08 -0.01 0.05 0.02 -0.01 0.08 -0.07 -0.02 0.13 0.87 0.74 1
-0.07 0.02 -0.12 0.11 -0.07 0 0.05 0.02 -0.01 0.08 -0.07 -0.02 0.13 0.9 1
-0.09 0.03 -0.12 0.13 -0.07 0.01 0.05 0.03 -0.02 0.09 -0.08 -0.03 0.15 1
-0.08 0.11 0 0.13 -0.02 0.01 0.05 0.05 0.13 0.2 0.15 0.24 1
0.01 0.03 0.04 0.01 0.02 0.02 -0.01 0.02 0.18 0.36 0.27 1
-0.13 0.08 0.05 0.02 0.03 0.02 -0.02 0.04 0.2 0.15 1
0.12 0.09 0 0.01 -0.03 0 0.04 0 0.21 1
0.06 0.1 0.04 0.02 -0.02 -0.02 -0.01 0.03 1
-0.04 0.22 0.03 0.26 0.02 0.02 0.02 1
-0.03 0.2 -0.1 0.04 -0.16 -0.13 1
-0.01 -0.12 0.07 0.03 0.51 1
0 -0.11 0.09 0.03 1
-0.04 0.45 0.03 1
-0.05 0.02 1
-0.01 1
1
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
83 / 101
Part IV: Open Data Demographics Structure TAI and niche selection Summary: Open Science
Group Level data of 8 criteria, 5 personality scales, 4 ability and 6interests”
Between group correlations of Criteria, Temperament, Ability, Interests
ConventEnterpriSocialArtisticInvesti
RealisticVerbal3Drotmatrix
Let-NumICAR60
Open/IntelExtraStabAgreeConscincomeBMI
p2edup1edu
educationexercise
agegender
gender
age
exercise
education
p1edu
p2edu
BMI
income
Consc
Agree
Stab
Extra
Open/Intel
ICAR60
Let-Num
matrix
3Drot
Verbal
Realistic
Investi
Artistic
Social
Enterpri
Convent
-0.72 0.31 -0.01 0.42 0.14 0.17 -0.14 0.75 -0.03 -0.65 0.52 -0.32 0.19 0.51 0.53 0.51 0.52 0.45 0.67 0.22 -0.1 -0.62 0.73 1
-0.64 0.21 0.23 0.36 0.18 0.2 -0.22 0.72 0.07 -0.54 0.61 0.1 0.11 0.31 0.33 0.3 0.3 0.29 0.42 -0.04 -0.14 -0.47 1
0.73 -0.09 0.11 -0.41 -0.27 -0.35 0.24 -0.67 0.27 0.83 -0.31 0.49 -0.35 -0.6 -0.6 -0.6 -0.61 -0.55 -0.72 -0.31 -0.08 1
-0.06 -0.05 -0.04 0.33 0.42 0.44 -0.36 -0.09 -0.72 -0.24 -0.49 -0.3 0.6 0.2 0.15 0.15 0.19 0.31 0.11 -0.16 1
-0.39 -0.37 0 0.14 0.38 0.41 -0.37 0.27 -0.26 -0.43 0.15 -0.56 0.38 0.66 0.66 0.68 0.69 0.6 0.54 1
-0.86 0.02 -0.08 0.46 0.36 0.44 -0.32 0.7 -0.4 -0.79 0.4 -0.58 0.48 0.79 0.79 0.81 0.81 0.73 1
-0.75 -0.14 0.01 0.69 0.71 0.79 -0.63 0.62 -0.73 -0.79 0.11 -0.73 0.82 0.98 0.96 0.95 0.95 1
-0.79 -0.16 -0.04 0.59 0.64 0.71 -0.57 0.65 -0.63 -0.8 0.22 -0.72 0.72 0.99 0.98 0.98 1
-0.79 -0.14 -0.08 0.58 0.58 0.67 -0.53 0.66 -0.59 -0.78 0.22 -0.72 0.7 0.99 0.99 1
-0.8 -0.13 -0.06 0.61 0.6 0.69 -0.53 0.68 -0.6 -0.79 0.23 -0.71 0.71 0.99 1
-0.79 -0.14 -0.04 0.62 0.64 0.72 -0.57 0.66 -0.65 -0.8 0.2 -0.73 0.75 1
-0.51 -0.07 0.01 0.61 0.66 0.75 -0.51 0.35 -0.8 -0.59 -0.16 -0.57 1
0.46 0.08 0.34 -0.41 -0.38 -0.43 0.33 -0.35 0.63 0.65 0.21 1
-0.58 0.25 0.27 0.16 -0.08 -0.05 0.09 0.54 0.36 -0.26 1
0.86 0.03 0 -0.59 -0.54 -0.59 0.51 -0.75 0.53 1
0.37 0.28 0.02 -0.53 -0.72 -0.76 0.65 -0.17 1
-0.84 0.27 -0.04 0.55 0.27 0.32 -0.24 1
0.33 0.54 -0.39 -0.42 -0.87 -0.84 1
-0.47 -0.37 0.35 0.59 0.95 1
-0.39 -0.41 0.4 0.56 1
-0.55 0.38 0.17 1
-0.02 -0.11 1
-0.09 1
1
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
84 / 101
Part IV: Open Data Demographics Structure TAI and niche selection Summary: Open Science
Compare the two correlation matrices–college majors8 criteria, 5 personality scales, 4 ability and 6 interests
Within group (below diagonal), Between groups (above diagonal
ConventEnterpriSocialArtisticInvesti
RealisticVerbal3Drotmatrix
Let-NumICAR60
Open/IntelExtraStabAgreeConscincomeBMI
p2edup1edu
educationexercise
agegender
gender
age
exercise
education
p1edu
p2edu
BMI
income
Consc
Agree
Stab
Extra
Open/Intel
ICAR60
Let-Num
matrix
3Drot
Verbal
Realistic
Investi
Artistic
Social
Enterpri
Convent
-0.01 0.01 -0.02 0.02 -0.02 -0.01 0.02 0 0.03 0.01 -0.01 -0.01 0.02 0.09 0.08 0.08 0.07 0.07 0.21 0.15 -0.01 0.04 0.21
-0.06 -0.03 0.01 0 0 0.01 0 0 0.01 0.01 0.01 0.05 0.04 0.07 0.07 0.06 0.05 0.06 0.19 0.12 0.03 0.02 0.73
0.03 0.02 0 0.01 0 0 0.01 -0.01 0.01 0.1 0 0.05 0.04 -0.01 0 -0.01 -0.02 -0.01 -0.02 0.05 0.09 -0.47 -0.62
-0.01 0.02 0 0.02 0.02 0.02 -0.01 0.01 -0.01 0.02 0 0.01 0.07 0.01 0.01 0.01 0.03 0 0.1 0.1 -0.08 -0.14 -0.1
-0.07 0 0 0.04 0.01 0.01 -0.01 0.02 -0.01 -0.02 0 -0.01 0.08 0.12 0.1 0.1 0.12 0.09 0.27 -0.16 -0.31 -0.04 0.22
-0.11 0 0.01 0.02 0 0.01 0.02 0.01 -0.01 -0.01 0.01 -0.01 0.06 0.13 0.11 0.11 0.12 0.1 0.54 0.11 -0.72 0.42 0.67
-0.07 0.03 -0.11 0.11 -0.07 0 0.06 0.02 -0.02 0.08 -0.07 -0.03 0.14 0.84 0.67 0.62 0.49 0.73 0.6 0.31 -0.55 0.29 0.45
-0.1 0.02 -0.06 0.11 0 0.04 0 0.04 -0.02 0.04 -0.04 -0.02 0.1 0.75 0.59 0.57 0.95 0.81 0.69 0.19 -0.61 0.3 0.52
-0.07 0.02 -0.12 0.1 -0.08 -0.01 0.05 0.02 -0.01 0.08 -0.07 -0.02 0.13 0.87 0.74 0.98 0.95 0.81 0.68 0.15 -0.6 0.3 0.51
-0.07 0.02 -0.12 0.11 -0.07 0 0.05 0.02 -0.01 0.08 -0.07 -0.02 0.13 0.9 0.99 0.98 0.96 0.79 0.66 0.15 -0.6 0.33 0.53
-0.09 0.03 -0.12 0.13 -0.07 0.01 0.05 0.03 -0.02 0.09 -0.08 -0.03 0.15 0.99 0.99 0.99 0.98 0.79 0.66 0.2 -0.6 0.31 0.51
-0.08 0.11 0 0.13 -0.02 0.01 0.05 0.05 0.13 0.2 0.15 0.24 0.75 0.71 0.7 0.72 0.82 0.48 0.38 0.6 -0.35 0.11 0.19
0.01 0.03 0.04 0.01 0.02 0.02 -0.01 0.02 0.18 0.36 0.27 -0.57 -0.73 -0.71 -0.72 -0.72 -0.73 -0.58 -0.56 -0.3 0.49 0.1 -0.32
-0.13 0.08 0.05 0.02 0.03 0.02 -0.02 0.04 0.2 0.15 0.21 -0.16 0.2 0.23 0.22 0.22 0.11 0.4 0.15 -0.49 -0.31 0.61 0.52
0.12 0.09 0 0.01 -0.03 0 0.04 0 0.21 -0.26 0.65 -0.59 -0.8 -0.79 -0.78 -0.8 -0.79 -0.79 -0.43 -0.24 0.83 -0.54 -0.65
0.06 0.1 0.04 0.02 -0.02 -0.02 -0.01 0.03 0.53 0.36 0.63 -0.8 -0.65 -0.6 -0.59 -0.63 -0.73 -0.4 -0.26 -0.72 0.27 0.07 -0.03
-0.04 0.22 0.03 0.26 0.02 0.02 0.02 -0.17 -0.75 0.54 -0.35 0.35 0.66 0.68 0.66 0.65 0.62 0.7 0.27 -0.09 -0.67 0.72 0.75
-0.03 0.2 -0.1 0.04 -0.16 -0.13 -0.24 0.65 0.51 0.09 0.33 -0.51 -0.57 -0.53 -0.53 -0.57 -0.63 -0.32 -0.37 -0.36 0.24 -0.22 -0.14
-0.01 -0.12 0.07 0.03 0.51 -0.84 0.32 -0.76 -0.59 -0.05 -0.43 0.75 0.72 0.69 0.67 0.71 0.79 0.44 0.41 0.44 -0.35 0.2 0.17
0 -0.11 0.09 0.03 0.95 -0.87 0.27 -0.72 -0.54 -0.08 -0.38 0.66 0.64 0.6 0.58 0.64 0.71 0.36 0.38 0.42 -0.27 0.18 0.14
-0.04 0.45 0.03 0.56 0.59 -0.42 0.55 -0.53 -0.59 0.16 -0.41 0.61 0.62 0.61 0.58 0.59 0.69 0.46 0.14 0.33 -0.41 0.36 0.42
-0.05 0.02 0.17 0.4 0.35 -0.39 -0.04 0.02 0 0.27 0.34 0.01 -0.04 -0.06 -0.08 -0.04 0.01 -0.08 0 -0.04 0.11 0.23 -0.01
-0.01 -0.11 0.38 -0.41 -0.37 0.54 0.27 0.28 0.03 0.25 0.08 -0.07 -0.14 -0.13 -0.14 -0.16 -0.14 0.02 -0.37 -0.05 -0.09 0.21 0.31
-0.09 -0.02 -0.55 -0.39 -0.47 0.33 -0.84 0.37 0.86 -0.58 0.46 -0.51 -0.79 -0.8 -0.79 -0.79 -0.75 -0.86 -0.39 -0.06 0.73 -0.64 -0.72
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
85 / 101
Part IV: Open Data Demographics Structure TAI and niche selection Summary: Open Science
Niche selection
1. College majors and occupations differ systematically in theintellectual Ability they require.
2. But they also differ in the Interests and Temperament theyrequire.
3. Examine the structure of the mean personality scale scoresfor college majors and for occupations.
4. The space is 2-3 dimensional.
5. A simple two factor solution shows that high ability can tradeoff for low Industry or Conscientiousness and that Boldness(low Anxiety) and Realistic interests differs from high Anxietyand Social interests.
6. We can examine the extent these two dimensions relate to avariety of criteria using factor extension (Dwyer, 1937; Horn, 1973)
86 / 101
Part IV: Open Data Demographics Structure TAI and niche selection Summary: Open Science
Biplot of a two factor solution to the group level major data
-3 -2 -1 0 1 2 3
-3-2
-10
12
3TAI factors of aggregated college majors
MR1
MR2
Industryorder
comp
polite
volatile
withdrawn
assert
enthus
intel
open
Let-Nummatrix3Drot
Verbal
Realistic
Investi
Artistic
Social
EnterpriConvent
-1.0 -0.5 0.0 0.5 1.0
-1.0
-0.5
0.0
0.5
1.0
87 / 101
Part IV: Open Data Demographics Structure TAI and niche selection Summary: Open Science
Add 8 criteria to the extended factor solution of the group data
-3 -2 -1 0 1 2 3
-3-2
-10
12
3TAI factors of aggregated majors extended to criteria variables
MR1
MR2
Industryorder
comp
polite
volatile
withdrawn
assert
enthus
intel
open
Let-Nummatrix3Drot
Verbal
Realistic
Investi
Artistic
Social
EnterpriConvent
gender
exercise
education
p1edup2edu
BMIage
income
-1.0 -0.5 0.0 0.5 1.0
-1.0
-0.5
0.0
0.5
1.0
88 / 101
Part IV: Open Data Demographics Structure TAI and niche selection Summary: Open Science
Occupational choice
1. Just as people assort themselves into college majors basedupon the appetites and their aptitudes, so they assortthemselves into occupations.
2. Some occupations require getting along, some emphasizegetting ahead, some require both
3. I show a 2 dimensional solution and then using factorextension, add in gender to the model
4. The final figure is not meant to be read, but rather seen as therange of occupations covered.
89 / 101
Part IV: Open Data Demographics Structure TAI and niche selection Summary: Open Science
Biplot of a two factor solution to the group level data
-3 -2 -1 0 1 2 3
-3-2
-10
12
3
Biplot of TAI scores at group level
MR1
MR2
Fear
Volatility
Enthus
Compassion
Trust
Easygoing
Industry
Mach
Impulsivity
Sociability
Bold
Serious
Habit
Int
Open LetNumMatrix3DRot
Verb
Real
Invest
Arti
Social
EnterClerical
-1.0 -0.5 0.0 0.5 1.0
-1.0
-0.5
0.0
0.5
1.0
90 / 101
Part IV: Open Data Demographics Structure TAI and niche selection Summary: Open Science
Biplot of a two factor solution to the group level data
-3 -2 -1 0 1 2 3
-3-2
-10
12
3
Biplot of TAI scores at group level
MR1
MR2
Fear
Volatility
Enthus
Compassion
Trust
Easygoing
Industry
Mach
Impulsivity
Sociability
Bold
Serious
Habit
Int
Open LetNumMatrix3DRot
Verb
Real
Invest
Arti
Social
EnterClerical
gender
-1.0 -0.5 0.0 0.5 1.0
-1.0
-0.5
0.0
0.5
1.0
91 / 101
Part IV: Open Data Demographics Structure TAI and niche selection Summary: Open Science
Biplot of a two factor solution to the group level data
-3 -2 -1 0 1 2 3
-3-2
-10
12
3
Biplot of TAI scores at group level
MR1
MR2 Actor
Art Director
Coach and/or Scout
Copy Writer
Editor
Graphic Designer
Musician and/or Singer
Photographer
Producer and/or DirectorPublic Relations Specialist
Writer
Other - ArtistOther - Designer
Other - Entertainer and/or PerformerOther - Media Related Worker
Janitor and/or Cleaner (except Maid and/or Housekeeping Cleaner)
Landscaping and/or Groundskeeping Worker
Maid and/or Housekeeping Cleaner
AccountantAuditor
Budget AnalystBusiness Operations Specialist
Credit Analyst
Financial Analyst
Human Resources and/or Training
Loan Officer
Management Analyst
Personal Financial AdvisorOther - Financial Specialist
Other - Business Worker
Clergy
Counselor - Educational, Vocational, and SchoolCounselor - Mental Health
Probation Officer and/or Correctional Treatment Specialist
Social and Human Service Assistant
Social Worker - Child, Family and/or SchoolSocial Worker - Mental Health and/or Substance Abuse
Other - Community and/or Social Service Specialist
Other - Counselor
Other - Religious WorkerOther - Social Worker
Business Intelligence Analyst
Computer and Information Scientist
Computer Programmer
Computer Security Specialist
Computer Software Engineer
Computer Specialist
Computer Support Specialist
Computer Systems Engineers/Architect
Database AdministratorInformation Technology Project Manager
Network and Computer Systems Administrator
Software Quality Assurance Engineer and/or TesterWeb Developer
Other - Computer Science Worker
Carpenter
Construction Laborer
ElectricianSupervisor/Manager of Construction and/or Extraction Workers
Other - Construction and Related Work
Adult Literacy, Remedial Education, and/or GED Teacher and/or Instructor
Elementary School Teacher (except Special Education)
Graduate Teaching Assistant
Instructional CoordinatorInstructional Designer and/or Technologist
Kindergarten Teacher (except Special Education)Librarian
Library Technician
Middle School Teacher (except Special and Vocational Education)
Postsecondary Teacher - Business
Postsecondary Teacher - Education
Postsecondary Teacher - English Language and/or Literature
Postsecondary Teacher - Psychology
Preschool Teacher (except Special Education)
Secondary School Teacher (except Special and Vocational Education)
Special Education Teacher - Preschool, Kindergarten, and Elementary School
Special Education Teacher - Secondary School
Teacher Assistant
Tutor
Other - Education, Training, or Library WorkerOther - Teacher and/or Instructor
Architect
Civil Engineer
Electrical EngineersMechanical Engineer
Other - Engineer
BaristaBartender
Chef and/or Head Cook
Cook - Fast Food
Cook - Institutional and Cafeteria
Cook - Other
Cook - Restaurant
Cook - Short Order
Dishwasher
Food Preparation Worker
Food Server
Food Service Attendant/Helper
Host/HostessSupervisor/Manager of Food Preparation and Serving Workers
Waiter/Waitress
Other - Food Preparation Worker
Acute Care Nurse
Dental Assistant
Emergency Medical Technician and/or Paramedic
Family and General Practitioner
Home Health Aide
Licensed Practical Nurse and/or Licensed Vocational Nurse
Massage Therapist
Medical and Clinical Laboratory Technician and/or Technologist
Medical Assistant
Medical Records and Health Information Technicians
Nurse Practitioner
Nursing Aid, Orderly and/or Attendant
Pharmacist
Pharmacy Technician
Physical Therapist
Psychiatrist
Radiologic Technologist and/or Technician
Registered Nurse
Surgical TechnologistOther - Health Technologist and/or TechnicianOther - Healthcare Support Worker
Other - Therapist
Automotive Mechanic and/or Service Technician
Electrical and Electronics Installer and/or RepairerOther - Installation, Maintenance, and Repair Worker
Law Clerk
Lawyer
Paralegal and/or Legal Assistant
Other - Legal Worker
Other - Legal Support WorkerBiologists
Psychologist - ClinicalPsychologist - Counseling
Psychologist - Other
Social Science Research Assistant
Other - Social Scientist
Administrative Services Manager
Chief Executive Officer
Computer and Information Systems ManagerFinancial Manager
Food Service Manager
General and Operations Manager
Human Resources Manager
Marketing Manager
Sales ManagerTraining and Development Manager
Other - Manager
Supervisor/Manager of Production Workers
Other - Production Worker
Air Crew Member
InfantrySupervisor/Manager of All Other Tactical Operations Specialists
Other - Military Enlisted Special and Tactical Operations Crew Member and/or Specialist
Other - Military Officer Special and Tactical Operations Leader/Manager
Billing Clerk
Bookkeeping, Accounting, and Auditing Clerks
Customer Service Representative
Data Entry KeyerExecutive Secretary and/or Administrative Assistant
Hotel, Motel, and Resort Desk Clerk
Human Resources AssistantInsurance Claims and/or Policy Processing Clerk
Medical Secretary
Office Clerk, General
Receptionist and/or Information ClerkSecretary
Supervisor/Manager of Office and Administrative Support Workers
Other - Office and Administrative Support Worker
Child Care Worker
Hairdresser, Hairstylist, and/or Cosmetologist
Nanny
Personal and Home Care Aide
Other - Personal Care and Service Worker
Correctional Offices and/or Jailer
Security Guard
Other - Protective Service Worker
Advertising Sales AgentCashierCounter and/or Rental ClerkFinancial Services Sales Agent
Insurance Sales Agent
Real Estate Sales AgentRetail Salesperson
Sales Representative - Services
Sales Representative - Wholesale and ManufacturingSupervisor/Manager of Retail Sales Workers
Supervisors/Manager of Non-Retail Sales Workers
Telemarketer
Other - Sales RepresentativeOther - Sales Worker
Driver/Sales Worker
Laborer - Freight, Stock, and/or Material Moving
Other - Material Moving Worker
Other - Transportation Workers
Fear
Volatility
Enthus
Compassion
Trust
Easygoing
Industry
Mach
Impulsivity
Sociability
Bold
Serious
Habit
Int
Open LetNumMatrix3DRot
Verb
Real
Invest
Arti
Social
EnterClerical
gender
-1.0 -0.5 0.0 0.5 1.0
-1.0
-0.5
0.0
0.5
1.0
92 / 101
Replication Studies Summary: Open Science References
Part VI
Open Scientific Research
93 / 101
Replication Studies Summary: Open Science References
Replicability of personality research
1. To what extent is personality research replicable?
2. By using open methods and open data, we allow ourselves totest for replicability
94 / 101
Replication Studies Summary: Open Science References
State level personality differences: Are they replicable?
1. Rentfrow and Gosling have reported data from several largeinternet surveys (much larger than ours).
2. Are they replicable?
3. With data shared from Gosling and Rentfrow, we were able toexamine how much the 5 studies they report match ourfindings.
4. State differences are small, but reliable for some measures,not all.
5. We have looked at the basic state differences in personalitymean scores
6. Also examined 18 different behavioral criterion (e.g., crimerates, state income, state well being, state liberalism, etc.)
95 / 101
Replication Studies Summary: Open Science References
Replicability of state differences depends upon the trait
−1 0 1
Sam
ple
1
Sam
ple
2
Sam
ple
3
Sam
ple
4
Sam
ple
5
SAPA
2010
SAPA
2015
Sample 1
Sample 2
Sample 3
Sample 4
Sample 5
SAPA2010
SAPA2015
Neuroticism
80 77
66
74
67
64
54
45
47
63
53
41
48
53
40
76
66
59
72
49
64
−1 0 1
Sam
ple
1
Sam
ple
2
Sam
ple
3
Sam
ple
4
Sam
ple
5
SAPA
2010
SAPA
2015
Sample 1
Sample 2
Sample 3
Sample 4
Sample 5
SAPA2010
SAPA2015
Openness
85 70
62
72
85
67
53
68
34
65
43
48
19
29
27
43
36
9
18
22
63
−1 0 1
Sam
ple
1
Sam
ple
2
Sam
ple
3
Sam
ple
4
Sam
ple
5
SAPA
2010
SAPA
2015
Sample 1
Sample 2
Sample 3
Sample 4
Sample 5
SAPA2010
SAPA2015
Conscientiousness
72 37
45
70
61
14
30
31
23
27
26
27
12
21
11
37
47
37
41
43
41
−1 0 1
Sam
ple
1
Sam
ple
2
Sam
ple
3
Sam
ple
4
Sam
ple
5
SAPA
2010
SAPA
2015
Sample 1
Sample 2
Sample 3
Sample 4
Sample 5
SAPA2010
SAPA2015
Extraversion
70 56
59
20
21
12
−6
−11
−20
46
27
20
36
35
17
28
25
21
37
9
65
−1 0 1
Sam
ple
1
Sam
ple
2
Sam
ple
3
Sam
ple
4
Sam
ple
5
SAPA
2010
SAPA
2015
Sample 1
Sample 2
Sample 3
Sample 4
Sample 5
SAPA2010
SAPA2015
Agreeableness
76 53
49
62
28
37
26
31
5
19
−2
19
15
−16
−23
11
41
22
−16
11
47
96 / 101
Replication Studies Summary: Open Science References
Replicability of personality by state demographics depends upon trait
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
−1.0 −0.5 0.0 0.5 1.0
−1.0
−0.5
0.0
0.5
1.0
Neuroticism
Original Partial Correlations
Rep
licat
ion
Parti
al C
orre
latio
ns r=0.73
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●●
●
●
●
●
●
●●
−1.0 −0.5 0.0 0.5 1.0−1
.0−0
.50.
00.
51.
0
Openness
Original Partial Correlations
Rep
licat
ion
Parti
al C
orre
latio
ns r=0.76
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
−1.0 −0.5 0.0 0.5 1.0
−1.0
−0.5
0.0
0.5
1.0
Conscientiousness
Original Partial Correlations
Rep
licat
ion
Parti
al C
orre
latio
ns r=0.84
●
●
●
●
●
●●
●
●
●
●
●●
●
● ●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●●
●
● ●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●●
●
● ●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●●
●
●● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●●
●●
●
●
●
●
●
●
●
−1.0 −0.5 0.0 0.5 1.0
−1.0
−0.5
0.0
0.5
1.0
Extraversion
Original Partial Correlations
Rep
licat
ion
Parti
al C
orre
latio
ns r=0.12
●●●
●
●
●
●
●●
●●
●●
●●
●
●
●
●
●
●
●
●
●
●
● ●● ●
●
●
●
●
●●
●●
●●
●●
●
●
●
●
●
●
●
●
●
●
● ●●●
●
●
●
●
●●
●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●● ●
●
●
●
●
●●
●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●●
●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
−1.0 −0.5 0.0 0.5 1.0
−1.0
−0.5
0.0
0.5
1.0
Agreeableness
Original Partial Correlations
Rep
licat
ion
Parti
al C
orre
latio
ns r=−0.05
97 / 101
Replication Studies Summary: Open Science References
State level: Income varies by iq (weighted r = .51)
0.15 0.20 0.25 0.30 0.35
40000
45000
50000
55000
60000
65000
State Income by State Ability -- weighted r = .51
ICAR60 estimate of state ability
Med
ian
inco
me
Alabama
Arizona
Arkansas
California
Colorado
Connecticut
Delaware
Florida
Georgia
Idaho
Illinois
Indiana
Iowa
Kansas
Kentucky
Louisiana
Maine
Maryland
Massachusetts
Michigan
Minnesota
Mississippi
Missouri
Montana
Nebraska
Nevada
New Hampshire New Jersey
New Mexico
New York
North Carolina
North Dakota
OhioOklahoma
OregonPennsylvania
Rhode Island
South Carolina
South Dakota
Tennessee
Texas
Utah
Vermont
Virginia
Washington
West Virginia
Wisconsin
Wyoming
98 / 101
Replication Studies Summary: Open Science References
State level: Well being varies by neuroticism (weighted r = -.59)
3.30 3.35 3.40 3.45 3.50 3.55
5960
6162
6364
Personality x Demographics -- Weighted r = -.59
State Neuroticism
Sta
te W
ellb
eing
Alabama
Arizona
Arkansas
California
Colorado
Connecticut
Delaware
Florida
Georgia
Idaho
Illinois
Indiana
Iowa
Kansas
Kentucky
Louisiana
Maine
Maryland
Massachusetts
Michigan
Minnesota
Mississippi
Missouri
Montana
Nebraska
Nevada
New Hampshire
New Jersey
New Mexico
New York
North Carolina
North Dakota
Ohio
Oklahoma
Oregon
Pennsylvania
Rhode Island
South Carolina
South Dakota
Tennessee
TexasUtah
Vermont
Virginia
Washington
West Virginia
Wisconsin
Wyoming
99 / 101
Replication Studies Summary: Open Science References
State level: Liberalism varies by Conscientiousness (r= -.42)
4.1 4.2 4.3 4.4
0.10
0.12
0.14
0.16
0.18
0.20
0.22
Personality x Demographics
State Conscientiousness
Sta
te L
iber
alis
m
Alabama
Arizona
Arkansas
California
Colorado
Connecticut
Delaware
Florida
Georgia
Idaho
Illinois
Indiana
Iowa
Kansas
Kentucky
Louisiana
Maine Maryland
Massachusetts
Michigan
Minnesota
Mississippi
Missouri
Montana
Nebraska
Nevada
New Hampshire
New Jersey
New Mexico
New York
North Carolina
North Dakota
Ohio
Oklahoma
Oregon
Pennsylvania
Rhode Island
South Carolina
South Dakota
Tennessee
Texas
Utah
Vermont
Virginia
Washington
West Virginia
Wisconsin
Wyoming
100 / 101
Replication Studies Summary: Open Science References
Summary and Conclusions
1. Ability, temperament and interests all provide usefulinformation about human personality.
2. Intellectual and personality development is the process ofexperiencing and choosing niches.
3. When we describe the intellectual requirements of aprofession or a college major, we should not ignore thatappropriate interests and temperaments guide occupationalchoice.
4. We need to consider appetites along with aptitudes.
5. The statistics, materials, methods, and data from all of thesestudies are done using Open Source Science.
6. Join us in this journey.
7. For more information and for these slides go tohttp://personality-project.org/sapa.html
101 / 101
Replication Studies Summary: Open Science References
Anderson, C. J., Bahnık, S., Barnett-Cowan, M., Bosco, F. A.,Chandler, J., Chartier, C. R., & Cheung, F. (2016). Response tocomment on “estimating the reproducibility of psychologicalscience”. Science, 351(6277), 1037–1037.
Anderson, J., Lin, H., Treagust, D., Ross, S., & Yore, L. (2007).Using large-scale assessment datasets for research in scienceand mathematics education: Programme for InternationalStudent Assessment (PISA). International Journal of Scienceand Mathematics Education, 5(4), 591–614.
Ashton, M. C., Lee, K., & Goldberg, L. R. (2007). TheIPIP-HEXACO scales: An alternative, public-domain measure ofthe personality constructs in the HEXACO model. Personalityand Individual Differences, 42(8), 1515–1526.
Bashaw, W. & Anderson Jr, H. E. (1967). A correction for replicatederror in correlation coefficients. Psychometrika, 32(4), 435–441.
Bolger, N., Davis, A., & Rafaeli, E. (2003). Diary methods:101 / 101
Replication Studies Summary: Open Science References
Capturing life as it is lived. Annual Review of Psychology, 54,579–616.
Bouchard, T. (1997). Experience producing drive theory: howgenes drive experience and shape personality. Acta Paediatrica,86(S422), 60–64.
Brown, A. D. (2014). Simulating the MMCAR method: Anexamination of precision and bias in synthetic correlations whendata are ‘massively missing completely at random’. Master’sthesis, Northwestern University, Evanston, Illinois.
Cattell, R. B. (1957). Personality and motivation structure andmeasurement. Oxford, England: World Book Co.
Cattell, R. B. & Stice, G. (1957). Handbook for the SixteenPersonality Factor Questionnaire. Champaign, Ill.: Institute forAbility and Personality Testing.
Cloninger, C. R., Przybeck, T. R., & Svrakic, D. M. (1994). TheTemperament and Character Inventory (TCI): A guide to its
101 / 101
Replication Studies Summary: Open Science References
development and use. center for psychobiology of personality,Washington University St. Louis, MO.
Comrey, A. L. (2008). The Comrey Personality Scales. In G. J.Boyle, G. Matthews, & D. H. Saklowfske (Eds.), Sage handbookof personality theory and testing: Personality measurement andassessment, volume II (pp. 113–134). London: Sage.
Condon, D. M. (2014). An organizational framework for thepsychological individual differences: Integrating the affective,cognitive, and conative domains. PhD thesis, NorthwesternUniversity.
Condon, D. M. (2015). The many little items of “big five” measures:Hierarchy, heterarchy, and predictive utility in personalitystructure. Long Beach, California. Society of Personality andSocial Psychology.
Condon, D. M. & Revelle, W. (2014). The International CognitiveAbility Resource: Development and initial validation of apublic-domain measure. Intelligence, 43, 52–64.
101 / 101
Replication Studies Summary: Open Science References
Condon, D. M. & Revelle, W. (2015a). Selected ICAR data fromthe SAPA-Project: Development and initial validation of apublic-domain measure. Harvard Dataverse.
Condon, D. M. & Revelle, W. (2015b). Selected personality datafrom the SAPA-Project: 08dec2013 to 26jul2014. HarvardDataverse.
Condon, D. M. & Revelle, W. (2015c). Selected personality datafrom the SAPA-Project: On the structure of phrased self-reportitems. Journal of Open Psychology Data, 3(1).
Condon, D. M. & Revelle, W. (2016). Selected ICAR data from theSAPA-Project: Development and initial validation of apublic-domain measure. Journal of Open Psychology Data.
Costa, P. T. & McCrae, R. R. (1992). NEO PI-R professionalmanual. Odessa, FL: Psychological Assessment Resources,Inc.
Costa, P. T., McCrae, R. R., & Dye, D. A. (1991). Facet scales foragreeableness and conscientiousness: A revision of the neo
101 / 101
Replication Studies Summary: Open Science References
personality inventory. Personality and Individual Differences,12(9), 887–898.
Cureton, E. (1966). Corrected item-test correlations.Psychometrika, 31(1), 93–96.
Deary, I. (2008). Why do intelligent people live longer? Nature,456(7219), 175–176.
DeYoung, C. G., Peterson, J. B., & Higgins, D. M. (2002).Higher-order factors of the big five predict conformity: Are thereneuroses of health? Personality and Individual Differences,33(4), 533–552.
DeYoung, C. G., Quilty, L. C., & Peterson, J. B. (2007). Betweenfacets and domains: 10 aspects of the big five. Journal ofPersonality and Social Psychology, 93(5), 880–896.
Digman, J. M. (1990). Personality structure: Emergence of thefive-factor model. Annual Review of Psychology, 41, 417–440.
Digman, J. M. (1997). Higher-order factors of the big five. Journalof Personality and Social Psychology, 73, 1246–1256.
101 / 101
Replication Studies Summary: Open Science References
Dwyer, P. S. (1937). The determination of the factor loadings of agiven test from the known factor loadings of other tests.Psychometrika, 2(3), 173–178.
Eysenck, H. J. (1994). The big five or the giant three: Criteria for aparadigm. In C. F. Halverson, G. A. Kohnstamm, & R. P. Martin(Eds.), The developing structure of temperament and personalityfrom infancy to adulthood (pp. 37–51). Hillsdale, N.J.: LawrenceErlbaum Associates.
Eysenck, S. B. G., Eysenck, H. J., & Barrett, P. (1985). A revisedversion of the psychoticism scale. Personality and IndividualDifferences, 6(1), 21 – 29.
French, J. A. & Condon, D. M. (2015). SAPA Tools: Tools toanalyze the SAPA Project. R package version 0.1.
Gilbert, D. T., King, G., Pettigrew, S., & Wilson, T. D. (2016).Comment on “estimating the reproducibility of psychologicalscience”. Science, 351(6277), 1037.
101 / 101
Replication Studies Summary: Open Science References
Goldberg, L. R. (1990). An alternative “description of personality”:The big-five factor structure. Journal of Personality and SocialPsychology, 59(6), 1216–1229.
Goldberg, L. R. (1992). The development of markers for thebig-five factor structure. Psychological Assessment, 4(1), 26–42.
Goldberg, L. R. (1999). A broad-bandwidth, public domain,personality inventory measuring the lower-level facets of severalfive-factor models. In I. Mervielde, I. Deary, F. De Fruyt, &F. Ostendorf (Eds.), Personality psychology in Europe, volume 7(pp. 7–28). Tilburg, The Netherlands: Tilburg University Press.
Goldberg, L. R. (2010). Personality, demographics and selfreported acts: the development of avocational interest scalesfrom estimates of the mount time spent in interest relatedactivities. In C. Agnew, D. Carlston, W. Graziano, & J. Kelly(Eds.), Then a miracle occurs: Focusing on the behavior insocial psychological theory and research (pp. 205–226). NewYork, NY: Oxford University Press.
101 / 101
Replication Studies Summary: Open Science References
Gottfredson, L. S. (1997). Why g matters: The complexity ofeveryday life. Intelligence, 24(1), 79 – 132.
Gough, H. G. & Bradley, P. (1996). Cpi manual . palo alto.
Hayes, K. (1962). Genes, drives, and intellect. PsychologicalReports (Monograph Supplement 2), 10, 299–342.
Hendriks, A. A. J. (1997). The construction of the five-factorpersonality inventory (FFPI). PhD thesis, RijksunivsiteitGroningen, Groningen, The Netherlands.
Hendriks, A. A. J., Hofstee, W. K., & De Raad, B. (1999). Thefive-factor personality inventory (FFPI). Personality andIndividual Differences, 27 (2), 307 – 325.
Hofstee, W. K., de Raad, B., & Goldberg, L. R. (1992). Integrationof the big five and circumplex approaches to trait structure.Journal of Personality and Social Psychology, 63(1), 146–163.
Hogan, R. & Hogan, J. (1995). The Hogan personality inventorymanual (2nd. ed.). Tulsa, OK: Hogan Assessment Systems.
101 / 101
Replication Studies Summary: Open Science References
Horn, J. L. (1973). On extension analysis and its relation tocorrelations between variables and factor scores. MultivariateBehavioral Research, 8(4), 477 – 489.
Jackson, D. N. (1983). JPI-R: Jackson Personality Inventory,Revised. Sigma Assesment Systems, Incorporated.
Jackson, D. N., Paunonen, S. V., & Tremblay, P. F. (2000). SixFactor Personality Questionnaire. Port Huron, MI: SigmaAssesment Systems.
Johnson, W. (2010). Extending and testing tom bouchard’sexperience producing drive theory. Personality and IndividualDifferences, 49(4), 296 – 301. Collected works from theFestschrift for Tom Bouchard, June 2009: A tribute to a vibrantscientific career.
Kelly, E. L. & Fiske, D. W. (1950). The prediction of success in theVA training program in clinical psychology. AmericanPsychologist, 5(8), 395 – 406.
101 / 101
Replication Studies Summary: Open Science References
Kelly, E. L. & Fiske, D. W. (1951). The prediction of performance inclinical psychology. Ann Arbor, Michigan: University of MichiganPress.
Kievit, R. A., Frankenhuis, W. E., Waldorp, L. J., & Borsboom, D.(2013). Simpson’s paradox in psychological science: a practicalguide. Frontiers in Psychology, 4(513), 1–14.
Lee, K. & Ashton, M. C. (2004). Psychometric properties of theHEXACO personality inventory. Multivariate BehavioralResearch, 39(2), 329–358.
Liebert, M. (2006). A public-domain assessment of musicpreferences as a function of personality and general intelligence.Honors Thesis. Department of Psychology, NorthwesternUniversity.
Lord, F. M. (1955). Estimating test reliability. Educational andPsychological Measurement, 15, 325–336.
Mehl, M. R., Pennebaker, J. W., Crow, D. M., Dabbs, J., & Price,J. H. (2001). The electronically activated recorder (ear): A
101 / 101
Replication Studies Summary: Open Science References
device for sampling naturalistic daily activities andconversations. Behavior Research Methods, Instruments, &Computers, 33(4), 517–523.
Mehl, M. R. & Robbins, M. L. (2012). Naturalistic observationsampling: The electronically activated recorder (ear). In M. R.Mehl & T. S. Conner (Eds.), Handbook of research methods forstudying daily life. New York, NY: Guilford Press.
Mehl, M. R., Vazire, S., Ramirez-Esparza, N., Slatcher, R. B., &Pennebaker, J. W. (2007). Are women really more talkative thanmen? Science, 317 (5834), 82.
Mottus, R. (2016). Towards more rigorous personalitytrait-outcome research. European Journal of Personality.
Norman, W. T. (1963). Toward an adequate taxonomy ofpersonality attributes: Replicated factors structure in peernomination personality ratings. Journal of Abnormal and SocialPsychology, 66(6), 574–583.
101 / 101
Replication Studies Summary: Open Science References
ottus, R. M., Kandler, C., Bleidorn, W., Riemann, R., & McCrae,R. R. (2016). Personality traits below facets: The consensualvalidity, longitudinal stability, heritability, and utility of personalitynuances. Journal of Personality and Social Psychology, (inpress).
Pozzebon, J. A., Visser, B. A., Ashton, M. C., Lee, K., & Goldberg,L. R. (2010). Psychometric characteristics of a public-domainself-report measure of vocational interests: The oregonvocational interest scales. Journal of Personality Assessment,92(2), 168–174.
R Core Team (2015). R: A Language and Environment forStatistical Computing. Vienna, Austria: R Foundation forStatistical Computing.
Rafaeli, E., Rogers, G. M., & Revelle, W. (2007). Affectivesynchrony: Individual differences in mixed emotions. Personalityand Social Psychology Bulletin, 33(7), 915–932.
Revelle, W. (2015). psych: Procedures for Personality and101 / 101
Replication Studies Summary: Open Science References
Psychological Research.http://cran.r-project.org/web/packages/psych/: NorthwesternUniversity, Evanston. R package version 1.5.8.
Revelle, W. & Condon, D. M. (2012). Multilevel analysis ofpersonality: Personality of college majors. Presented at theannual meeting of the Society of Multivariate ExperimentalPsychology.
Revelle, W. & Condon, D. M. (2015a). Ability, temperament, andinterests: their joint predictive power for job choice. InternationalSociety for the Study of Intelligence. Albuquerque, New Mexico.
Revelle, W. & Condon, D. M. (2015b). A model for personality atthree levels. Journal of Research in Personality, 56, 70–81.
Revelle, W., Condon, D. M., Wilt, J., French, J. A., Brown, A., &Elleman, L. G. (2016). Web and phone based data collectionusing planned missing designs. In N. G. Fielding, R. M. Lee, &G. Blank (Eds.), SAGE Handbook of Online Research Methodschapter 37: Mobile Methods. Sage Publications, Inc. (in press).
101 / 101
Replication Studies Summary: Open Science References
Revelle, W. & Wilt, J. (2015). The data box and within subjectanalyses: A comment on Nesselroade and Molenaar.Multivariate Behavioral Research (in press).
Revelle, W., Wilt, J., & Condon, D. (2011). Individual differencesand differential psychology: A brief history and prospect. InT. Chamorro-Premuzic, A. Furnham, & S. von Stumm (Eds.),Handbook of Individual Differences chapter 1, (pp. 3–38).Oxford: Wiley-Blackwell.
Revelle, W., Wilt, J., & Rosenthal, A. (2010). Individual differencesin cognition: New methods for examining thepersonality-cognition link. In A. Gruszka, G. Matthews, &B. Szymura (Eds.), Handbook of Individual Differences inCognition: Attention, Memory and Executive Control chapter 2,(pp. 27–49). New York, N.Y.: Springer.
Roberts, B. W., Kuncel, N. R., Shiner, R., Caspi, A., & Goldberg,L. R. (2007). The power of personality: The comparative validityof personality traits, socioeconomic status, and cognitive ability
101 / 101
Replication Studies Summary: Open Science References
for predicting important life outcomes. Perspectives onPsychological Science, 2(4), 313–345.
Rounds, J., Su, R., Lewis, P., & Rivkin, D. (2010). O* NET R©interest profiler short form psychometric characteristics:Summary.
Shipley, W. C. (2009). Shipley-2: manual. Western PsychologicalServices.
Simpson, E. H. (1951). The interpretation of interaction incontingency tables. Journal of the Royal Statistical Society.Series B (Methodological), 13(2), 238–241.
Tellegen, A. & Waller, N. G. (2008). Exploring personality throughtest construction: Development of the multidimensionalpersonality questionnaire. The Sage handbook of personalitytheory and assessment, 2, 261–292.
Tupes, E. C. & Christal, R. E. (1961). Recurrent personality factorsbased on trait ratings. Technical Report 61-97, USAF ASD
101 / 101
Replication Studies Summary: Open Science References
Technical Report, Lackland Air Force Base (reprinted in Journalof Personality (1992) 60: 225–251).
Wilt, J. (2014). A new form and function for personality. PhDthesis, Northwestern University.
Wilt, J., Bleidorn, W., & Revelle, W. (2016). Finding a life worthliving: Meaning in life and graduation from college. EuropeanJournal of Personality, (in press).
Wilt, J., Funkhouser, K., & Revelle, W. (2011). The dynamicrelationships of affective synchrony to perceptions of situations.Journal of Research in Personality, 45, 309–321.
Wilt, J. & Revelle, W. (2015). Affect, behaviour, cognition anddesire in the big five: An analysis of item content and structure.European Journal of Personality, 29(4), 478–497.
Yule, G. U. (1903). Notes on the theory of association of attributesin statistics. Biometrika, 2(2), 121–134.
101 / 101