Download - Tuesday ETS AIR Symposium Data Analysis
-
8/7/2019 Tuesday ETS AIR Symposium Data Analysis
1/44
An Overview of Data Analysis
Jonathan SteinbergSenior Research Data Analyst, Data Analysis Research
Bruce A. Kaplan
Director, Data Analysis Research
-
8/7/2019 Tuesday ETS AIR Symposium Data Analysis
2/44
Copyright 2008 by Educational Tes ting Service. All rights reserved. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of
Educational Testing Servic e (ETS). The AIR logo is a trademark of the Associati on for Institutional Res earch. 9116
Outline
Key Considerations in Data Analysis
Components of a Data Analysis Plan
Distinguishing Data Types Distinguishing Different Types of Analyses
Overview of Different Statistical Software
Case Study: ACESTM
-
8/7/2019 Tuesday ETS AIR Symposium Data Analysis
3/44
Copyright 2008 by Educational Tes ting Service. All rights reserved. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of
Educational Testing Servic e (ETS). The AIR logo is a trademark of the Associati on for Institutional Res earch. 9116
Key Considerations in
Data Analysis
Identify the purpose of the analysis or project
Understand the sample(s) under study
Understand the instruments being used to
collect data
Be cognizant of data layouts and formats
Establish a unique identifier if matching ormerging is necessary
Plan your work and work your plan!
-
8/7/2019 Tuesday ETS AIR Symposium Data Analysis
4/44
Copyright 2008 by Educational Tes ting Service. All rights reserved. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of
Educational Testing Servic e (ETS). The AIR logo is a trademark of the Associati on for Institutional Res earch. 9116
Components of a
Data Analysis Plan
Statement of research questions
Methods used to answer research questions
Timeline Budget
File restructuring procedures (syntax creation, adding new
variables as needed)
Algorithms for scoring, equating, etc. Data cleaning procedures (e.g. removing outliers)
Quality control procedures at every step in the project
-
8/7/2019 Tuesday ETS AIR Symposium Data Analysis
5/44
Copyright 2008 by Educational Tes ting Service. All rights reserved. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of
Educational Testing Servic e (ETS). The AIR logo is a trademark of the Associati on for Institutional Res earch. 9116
Examples of Analyses
Frequency Distributions and Cross -Tabulations
Descriptive Statistics (Means, Std. Deviations, Correlations)
T-tests and Analysis of Variance (ANOVA)
Regression
Principal Components/Factor Analysis (Data Reduction)
Cluster and Discriminant Analyses (Segmentation)
Latent Class Analysis (Classification)
Hierarchical Linear Modeling (HLM)
Differential Item Functioning (DIF)
-
8/7/2019 Tuesday ETS AIR Symposium Data Analysis
6/44
Copyright 2008 by Educational Tes ting Service. All rights reserved. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of
Educational Testing Servic e (ETS). The AIR logo is a trademark of the Associati on for Institutional Res earch. 9116
More Advanced the Analysis,
Greater the Amount of Preparation
Most analyses can be executed straight from a
working data file Some analyses may require transformations of
the raw data, subsets, or specific input data to
comply with statistical software
-
8/7/2019 Tuesday ETS AIR Symposium Data Analysis
7/44
Copyright 2008 by Educational Tes ting Service. All rights reserved. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of
Educational Testing Servic e (ETS). The AIR logo is a trademark of the Associati on for Institutional Res earch. 9116
Data Types & Representation
Variables may require special coding for
different data representation Numeric
String
Date & time
Monetary
-
8/7/2019 Tuesday ETS AIR Symposium Data Analysis
8/44
Copyright 2008 by Educational Tes ting Service. All rights reserved. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of
Educational Testing Servic e (ETS). The AIR logo is a trademark of the Associati on for Institutional Res earch. 9116
ASCII Text Files
Usually rectangular in structure One record per observation
Each data variable in same position on each record
Each record may have multiple instances of data Arrays
Repeating blocks (sets of variables)
File may have multiple records per observation Number of records per observation can be variable
Most government data files come in this format at aminimum
Every software package can handle this file type
-
8/7/2019 Tuesday ETS AIR Symposium Data Analysis
9/44
Copyright 2008 by Educational Tes ting Service. All rights reserved. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of
Educational Testing Servic e (ETS). The AIR logo is a trademark of the Associati on for Institutional Res earch. 9116
CSV Files
(Comma-Separated Values)
Individual data elements separated by commas
Usually rectangular structure One record (line) per observation
Fixed number of elements on each record
Problems if data elements contain delimiter or blank
spaces (i.e. text strings) Missing data must be represented by nulls
-
8/7/2019 Tuesday ETS AIR Symposium Data Analysis
10/44
Copyright 2008 by Educational Tes ting Service. All rights reserved. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of
Educational Testing Servic e (ETS). The AIR logo is a trademark of the Associati on for Institutional Res earch. 9116
System Files
(SAS
, SPSS
)
Data stored in binary (machine) format
Issues of portability across platforms
Structured as rectangular tables
SAS files can be indexed for direct access
Self-contained documentation
Data variable labels & formats
Data value labels
Most analysis packages provide facility for reading (but not writing!)system files from other packages (SPSS more than SAS)
Using default data formats can yield system files that are much largerthan source ASCII files
-
8/7/2019 Tuesday ETS AIR Symposium Data Analysis
11/44
Copyright 2008 by Educational Tes ting Service. All rights reserved. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of
Educational Testing Servic e (ETS). The AIR logo is a trademark of the Associati on for Institutional Res earch. 9116
Relational Databases
Accessible using Structured Query Language
(SQL)
Optimal for filtering & sorting
One or more rectangular tables contain data of
interest
This format is often at odds with statistical analysis
needs
-
8/7/2019 Tuesday ETS AIR Symposium Data Analysis
12/44
Copyright 2008 by Educational Tes ting Service. All rights reserved. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of
Educational Testing Servic e (ETS). The AIR logo is a trademark of the Associati on for Institutional Res earch. 9116
Relational Database
Example
Index Student School Gender Ethnicity
1 29101 291 F 3
2 29102 291 M 2
3 29201 292 F 6
Student
SAT
Index Student TestDate Measure Score
1 29101 10/13/2006 Verbal 650
2 29101 10/13/2006 Math 640
3 29102 5/21/1995 Verbal 370
4 29102 5/21/1995 Math 400
5 29201 2/3/2000 Verbal 520
6 29201 2/3/2000 Math 550
Index Student TestDate Test Measure Score
1 29101 5/6/2006 General Verbal 760
2 29101 5/6/2006 General Quantitative 540
3 29102 5/3/2006 General Verbal 340
4 29102 5/3/2006 General Quantitative 420
5 29102 5/3/2006 General Analytical 0
6 29102 Subject Subject1 540
7 29201 5/6/2006 General Quantitative 500
8 29201 5/6/2006 General Verbal 530
9 29201 Subject Subject1 490
10 29201 Subject Subject2 530
GRE
-
8/7/2019 Tuesday ETS AIR Symposium Data Analysis
13/44
Copyright 2008 by Educational Tes ting Service. All rights reserved. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of
Educational Testing Servic e (ETS). The AIR logo is a trademark of the Associati on for Institutional Res earch. 9116
Hierarchy of
Statistical Software
Excel
General Stats
Psychometric
Advanced Modeling
SAS, SPSS
IRT: PARSCALE, BILOG
Factor Analysis: LISREL, EQS
Nested Models: HLM
-
8/7/2019 Tuesday ETS AIR Symposium Data Analysis
14/44
Copyright 2008 by Educational Tes ting Service. All rights reserved. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of
Educational Testing Servic e (ETS). The AIR logo is a trademark of the Associati on for Institutional Res earch. 9116
Overview
SAS and SPSS are most commonly used and tend to focus onthe classic statistical routines: Descriptive statistics and non-parametric (distribution-free) tests
ANOVA / Regression Factor analysis
However, many psychometric procedures (e.g. IRT) and newerstatistical models are not as well supported by these programs Very specialized programs are used
Designed to do a specific task or validate a theory
Specialized programs may have issues Interface not very user-friendly
Additional data types or files required
Expense
-
8/7/2019 Tuesday ETS AIR Symposium Data Analysis
15/44
Copyright 2008 by Educational Tes ting Service. All rights reserved. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of
Educational Testing Servic e (ETS). The AIR logo is a trademark of the Associati on for Institutional Res earch. 9116
What is Excel?
Data are organized by worksheets, rows and columns
Worksheet limits are 256 columns and 65,536 total cells
C
ells contain data or formulas with relative or absolute references toother cells
Direct manipulation of data and flexibility to move data around (e.g.sorting, replacing, merging)
Opens many file types
Quite useful in prepping files for use in SPSS, SAS or other programs Conditional formatting
Also features macro capabilities, replicating user actions, allowingsimple automation of regular tasks
-
8/7/2019 Tuesday ETS AIR Symposium Data Analysis
16/44
Copyright 2008 by Educational Tes ting Service. All rights reserved. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of
Educational Testing Servic e (ETS). The AIR logo is a trademark of the Associati on for Institutional Res earch. 9116
Data Presentation
Options in Excel
Tables and graphs can be exported to a wide variety of software packages
Can tweak and perfect example graph or table and then replicate by
replacing only the data being used
Main advantage is ability to combine data from multiple sources not just
what is found in the data file
Two-for-One deal - table creation usually puts data into a format that leads
to easy graph creation
User has control over virtually all aspects of a graph - size, colors, fonts, titles,
legends, labels, etc. Can combine graphs with tables and use cell layout to produce more complex
presentations
Final graphs can be of publication quality
-
8/7/2019 Tuesday ETS AIR Symposium Data Analysis
17/44
Copyright 2008 by Educational Tes ting Service. All rights reserved. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of
Educational Testing Servic e (ETS). The AIR logo is a trademark of the Associati on for Institutional Res earch. 9116
Data Presentation
Options - Example
0%
10%
20%
0%
0%
0%
60%
0%
80%
90%
100%
00 and
belo
10 to
800
810 to
900
910 to
1000
1010 to
1100
1110 to
1200
1210 to
1 00
1 10 to
1
00
1
10 to
1 00
1 10 to
1600
C
mbi
(V
rb
l + M
t
)
r
i trib ti f Fr m G b C mbi r
A(
.90 to
.2!
)
A-(
.60 to
.89)
" +(
.
0 to
.!
9)
" (2.90 to
.29)
"
-
(2.60 to 2.89)
C+
(2.
0 to 2.!
9)
C(1.90 to 2.29)
C-
(1.60 to 1.89)D
(0.60 to 1.!
9)
#
("
elo$
0.!
9)
-
8/7/2019 Tuesday ETS AIR Symposium Data Analysis
18/44
Copyright 2008 by Educational Tes ting Service. All rights reserved. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of
Educational Testing Servic e (ETS). The AIR logo is a trademark of the Associati on for Institutional Res earch. 9116
What is SAS?
A general purpose statistical package with a basicprogramming capability utilizing scores of statistical andmathematical functions in numerous modules
Can readily access data from a wide variety of sources,perform data management, and present findings in avariety of report and graph formats
Provides powerful tools for both specialized andenterprise-wide analytical needs
-
8/7/2019 Tuesday ETS AIR Symposium Data Analysis
19/44
Copyright 2008 by Educational Tes ting Service. All rights reserved. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of
Educational Testing Servic e (ETS). The AIR logo is a trademark of the Associati on for Institutional Res earch. 9116
SAS - Strengths
Versatile data input and output formats
SAS provides both SQL and DATA steps tomanipulate data: SQL provides a way of carrying out relational algebra
on tables and views SAS data sets can be indexed for direct access or
processed sequentially, without reading all records
into memory, which is sometimes much more efficient
-
8/7/2019 Tuesday ETS AIR Symposium Data Analysis
20/44
Copyright 2008 by Educational Tes ting Service. All rights reserved. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of
Educational Testing Servic e (ETS). The AIR logo is a trademark of the Associati on for Institutional Res earch. 9116
SAS - Weaknesses
Steep learning curve; volume of functions, options anddocumentation can be overwhelming for the novice
Inconsistent syntax across different procedures ormodules
Not a good choice for applications that interact withexternal systems such as hardware devices orsoftware programs because of its inconvenient
interface
Difficult interaction with other programming languages
Expensive
-
8/7/2019 Tuesday ETS AIR Symposium Data Analysis
21/44
Copyright 2008 by Educational Tes ting Service. All rights reserved. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of
Educational Testing Servic e (ETS). The AIR logo is a trademark of the Associati on for Institutional Res earch. 9116
What is SPSS?
A commercially produced statistical software packagethat is widely used in the fields of Education andPsychology
Program functionality is broken into over a dozendifferent modules which are sold individually
Most commonly used are Base, Regression Models, andAdvanced Models
Other modules can be installed to run more complexanalyses
SPSS data files include both the data and also variableinformation (variable and value labels, formats andmissing values)
-
8/7/2019 Tuesday ETS AIR Symposium Data Analysis
22/44
Copyright 2008 by Educational Tes ting Service. All rights reserved. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of
Educational Testing Servic e (ETS). The AIR logo is a trademark of the Associati on for Institutional Res earch. 9116
SPSS - Strengths
Easily opens data from other programs such as Excel and SAS
Variable view screen allows for quick overview of file contents andallows for easy modifications of names, formats, labels, and variable
order Having all data information in a single file allows sharing files on a
project to be very easy
Point-and-click menus do not require memorizing syntax for majorityof procedures
Many procedures can be expanded beyond the menu options in syntax
Split-file command allows all output to be replicated for variousgroups through a single command
Journal file tracks all commands used for life of program, with goodresources to find code accidentally deleted
-
8/7/2019 Tuesday ETS AIR Symposium Data Analysis
23/44
Copyright 2008 by Educational Tes ting Service. All rights reserved. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of
Educational Testing Servic e (ETS). The AIR logo is a trademark of the Associati on for Institutional Res earch. 9116
SPSS - Weaknesses
Ease of doing data manipulation can sometimes lead tomistakes as the program does not preclude inappropriatemodifications to the data
Matching feature requires exact match Duplicate records generate warnings but can be marked in file
Error logs are hard to interpret at times
Incompleteness of menus means some options are onlyavailable via syntax
While the majority of output is saved as pivot tables allowinggreat flexibility in modifying tables
Output tables and graphs generally not done as well as Exceland are harder to manipulate
-
8/7/2019 Tuesday ETS AIR Symposium Data Analysis
24/44
Copyright 2008 by Educational Tes ting Service. All rights reserved. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of
Educational Testing Servic e (ETS). The AIR logo is a trademark of the Associati on for Institutional Res earch. 9116
LISREL
Ideal for discrete data types
Test data, Likert scale item data
Data can be imported in various types
ASCII, Access, Excel, SAS, SPSS, etc.
Variable names have length restrictions
Data files then stored as system files for later use
Basic statistics (e.g. means and correlations) are generated in an
underlying program called PRELIS LISREL itself is used to confirm the structural validity of a
measurement model for any assessment
Requires syntax and input matrices
-
8/7/2019 Tuesday ETS AIR Symposium Data Analysis
25/44
Copyright 2008 by Educational Tes ting Service. All rights reserved. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of
Educational Testing Servic e (ETS). The AIR logo is a trademark of the Associati on for Institutional Res earch. 9116
EQS
Ideal for continuous data types (test subscales)
Data can be imported in various types ASCII,
Access, Excel, SAS, SPSS, etc. but variable names
have length restrictions
Data files then stored as system files for later use
EQS itself is used to confirm the structural validity ofa measurement model for any assessment
Some model syntax can be built through the menus
-
8/7/2019 Tuesday ETS AIR Symposium Data Analysis
26/44
Copyright 2008 by Educational Tes ting Service. All rights reserved. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of
Educational Testing Servic e (ETS). The AIR logo is a trademark of the Associati on for Institutional Res earch. 9116
HLM
Hierarchical Linear Modeling (HLM) is becoming a more
popular type of analysis, namely in cohort trend modeling
Also allows you to look at variance component estimatesand regression models given a nested sample ofrespondents
Students within countries within global regions on personality
variables
More tedious to set up analysis with fewer available filetypes
Also requires more upfront work as multiple data files are
needed
-
8/7/2019 Tuesday ETS AIR Symposium Data Analysis
27/44
Copyright 2008 by Educational Tes ting Service. All rights reserved. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of
Educational Testing Servic e (ETS). The AIR logo is a trademark of the Associati on for Institutional Res earch. 9116
What Program
Should I Use?
Microsoft Excel is the most basic and accessible spreadsheet programavailable today
It is most ideal for general data exploration, histograms, scatter plots, etc.
Appearance of tables can be customized to meet APA standards Allows for easy transition to other programs to complete analyses and
write reports
However, its heritage is not as a statistical analysis program
Certain statistical programs are designed for specific analytic tasks
Balance the results and what will being presented Choose wisely in the interests of efficiency and accuracy of results
Some output is good forlookingat the data through basic exploration andto generate basic tables, but not to present the data
-
8/7/2019 Tuesday ETS AIR Symposium Data Analysis
28/44
Copyright 2008 by Educational Tes ting Service. All rights reserved. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of
Educational Testing Servic e (ETS). The AIR logo is a trademark of the Associati on for Institutional Res earch. 9116
Summary
Be very clear about the analysis objectives
Be very familiar with all aspects of what defines your data
Develop and stay true to your data analysis plans and
research questions
Be cognizant of which statistical software programs can best
answer your research questions and present your results
Be thorough in your analyses, express openness to additionalinvestigations, yet be mindful of limitations given the data and
the programs you are using
-
8/7/2019 Tuesday ETS AIR Symposium Data Analysis
29/44
Overview of the Admitted Class
Evaluation Service (AC
ES)Bruce A. Kaplan
Director, Data Analysis Research
-
8/7/2019 Tuesday ETS AIR Symposium Data Analysis
30/44
Copyright 2008 by Educational Tes ting Service. All rights reserved. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of
Educational Testing Servic e (ETS). The AIR logo is a trademark of the Associati on for Institutional Res earch. 9116
What is ACES?
ACES is a validity study service offered by theCollege Board that is operated and maintained
by ETS.
-
8/7/2019 Tuesday ETS AIR Symposium Data Analysis
31/44
Copyright 2008 by Educational Tes ting Service. All rights reserved. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of
Educational Testing Servic e (ETS). The AIR logo is a trademark of the Associati on for Institutional Res earch. 9116
Types of ACES Studies
There are two types of ACES studies:
Admission
Evaluate the validity of tests in predicting how well
students will do in an institution of higher education
(used for admitting students into an institution)
Placement Evaluate the validity of tests in placing students into
or out of a class
-
8/7/2019 Tuesday ETS AIR Symposium Data Analysis
32/44
Copyright 2008 by Educational Tes ting Service. All rights reserved. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of
Educational Testing Servic e (ETS). The AIR logo is a trademark of the Associati on for Institutional Res earch. 9116
ACES Admission -
Types of Data
Institutions have many variables to choose from
when requesting their models The required variables are:
First-Year Grade Point Average (FGPA), supplied by the
institution
A measure of high school success (either from the SATQuestionnaire or supplied by the institution)
A choice from a list of SAT Test scores
-
8/7/2019 Tuesday ETS AIR Symposium Data Analysis
33/44
Copyright 2008 by Educational Tes ting Service. All rights reserved. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of
Educational Testing Servic e (ETS). The AIR logo is a trademark of the Associati on for Institutional Res earch. 9116
ACES Admission -
Types of Data (continued)
The optional variables are:
Course data (name of course, grade received, and
number of credits awarded) for each student,supplied by the institution
A choice from a list of SAT Subject Test scores.
Additional predictors, either supplied by the
institution or chosen from a list of ACES predictors Additional subgroups to be analyzed, either
supplied by the institution or chosen from a list of
ACES subgroups
-
8/7/2019 Tuesday ETS AIR Symposium Data Analysis
34/44
Copyright 2008 by Educational Tes ting Service. All rights reserved. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of
Educational Testing Servic e (ETS). The AIR logo is a trademark of the Associati on for Institutional Res earch. 9116
Data Merging, Matching, -
and Cleaning
ACES data poses many challenges
Merging
Institutions have choices of format type, and sending two files orone
ACES merges all this information in a more or less standardized
format
Matching
Institution data is matched against a file of college seniors to pull
SAT scores, SAT subject scores and SAT QuestionnaireResponses
Cleaning
Data is checked for outliers, out of range, and implausible values
-
8/7/2019 Tuesday ETS AIR Symposium Data Analysis
35/44
Copyright 2008 by Educational Tes ting Service. All rights reserved. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of
Educational Testing Servic e (ETS). The AIR logo is a trademark of the Associati on for Institutional Res earch. 9116
ACES Admission -
Types of statistics used
Basic Statistics
Means and Standard Deviations for Total Group andby Gender
Correlations for various predictor variables with:
First-Year Grade Point Average (FGPA) for Total group andsubgroups
Adjusted First-Year Grade Point Average (AGPA)
Adjusted for course difficulty
Both are corrected for Restriction of Range
-
8/7/2019 Tuesday ETS AIR Symposium Data Analysis
36/44
Copyright 2008 by Educational Tes ting Service. All rights reserved. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of
Educational Testing Servic e (ETS). The AIR logo is a trademark of the Associati on for Institutional Res earch. 9116
ACES Admission -
Types of statistics used (continued)
Various Regressions with up to six classes of
predictor variables
SAT (SAT-r) variables only
SAT-r and high school measure (either HSGPA or HS rank)
SAT-r, HS measure and school requested variables
SAT-r and SAT Subject Tests (SAT-s)
SAT-r, SAT-s, and HS measure
SAT-r, SAT-s, HS measure and school requested variables
-
8/7/2019 Tuesday ETS AIR Symposium Data Analysis
37/44
Copyright 2008 by Educational Tes ting Service. All rights reserved. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of
Educational Testing Servic e (ETS). The AIR logo is a trademark of the Associati on for Institutional Res earch. 9116
ACES Admission -
Types of statistics used (continued)
At Risk Students are students whose Actual FGPA
fell 1.5 or more standard deviations below their
Predicted Grade Point Average (PGPA) PGPA for a student is computed from regression
model with the most data available for that student
Return to institutions the data files used so they can
conduct further research. This includes any data sentto ACES along with the matched SAT variables, and
calculated values like PGPA and AGPA
-
8/7/2019 Tuesday ETS AIR Symposium Data Analysis
38/44
Copyright 2008 by Educational Tes ting Service. All rights reserved. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of
Educational Testing Servic e (ETS). The AIR logo is a trademark of the Associati on for Institutional Res earch. 9116
ACES Admission -
Restriction of Range
What is Restriction of Range?
We want to measure the correlation between FGPAand SAT scores for the applicant pool for a given
institution
We only have the attending student pool for a giveninstitution
What does that do to our correlations?
Demonstration of Restriction of Range: http://cnx.org/content/m11196/latest/
Our solution: use the SAT senior cohort to helpestimate the correlations
-
8/7/2019 Tuesday ETS AIR Symposium Data Analysis
39/44
Copyright 2008 by Educational Tes ting Service. All rights reserved. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of
Educational Testing Servic e (ETS). The AIR logo is a trademark of the Associati on for Institutional Res earch. 9116
ACES Admission -
Course Adjustment
Are all FGPAs at an institution really equal?
Adjusted FGPA (AGPA) using institution-supplied coursegrades
For the institution, calculate each students predicted FGPA using alinear regression with SAT CR, SAT M, SAT W (if used), and HSvariable as independent variables and FGPA as a dependentvariable
Calculate an average residual per course
C
alculate an adjustment factor for each student, using the averageresiduals for the courses they took, weighted by the number ofcredits for that course
Apply the adjustment factor to the FGPA to obtain theAGPA
-
8/7/2019 Tuesday ETS AIR Symposium Data Analysis
40/44
Copyright 2008 by Educational Tes ting Service. All rights reserved. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of
Educational Testing Servic e (ETS). The AIR logo is a trademark of the Associati on for Institutional Res earch. 9116
ACES Placement -
Statistics Used
Distributions of Predictor variables (Accuplacer
scores, CLEP scores, SAT Scores, or school
supplied variables) by Different levels ofCourse
Grade
Logistic regression calculated for two dependent
variables
Pass = B or higher, else Fail
Pass = C or higher, else Fail
-
8/7/2019 Tuesday ETS AIR Symposium Data Analysis
41/44
Copyright 2008 by Educational Tes ting Service. All rights reserved. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of
Educational Testing Servic e (ETS). The AIR logo is a trademark of the Associati on for Institutional Res earch. 9116
ACES Placement -
Statistics Used (continued)
Models provided for single predictors, and multiple
predictors
For each Model
A measure of association between the dependent and
predictor variables is computed
Percent correctly placed
Cut Scores Associated with Predicted Probability of
Success
-
8/7/2019 Tuesday ETS AIR Symposium Data Analysis
42/44
Copyright 2008 by Educational Tes ting Service. All rights reserved. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of
Educational Testing Servic e (ETS). The AIR logo is a trademark of the Associati on for Institutional Res earch. 9116
You can submit a study
for your Institution
Examples of sample reports:
Admissions Report
http://www.collegeboard.com/prod_downloads/highered/apr/aces/sample_admiss.pdf
Placement Report
http://www.collegeboard.com/prod_downloads/highered/apr/aces/sample_place.pdf
ACES site to request a study
http://professionals.collegeboard.com/higher-
ed/validity
-
8/7/2019 Tuesday ETS AIR Symposium Data Analysis
43/44
Copyright 2008 by Educational Tes ting Service. All rights reserved. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of
Educational Testing Servic e (ETS). The AIR logo is a trademark of the Associati on for Institutional Res earch. 9116
Contact Information
Jonathan Steinberg:[email protected]
Bruce Kaplan: [email protected]
For additional ACES help and information:
[email protected] Or call (609) 921-9000 and ask for help with ACES
-
8/7/2019 Tuesday ETS AIR Symposium Data Analysis
44/44
Questions orComments