ets proficiency profile test pilot summary report 2012 · ets proficiency profile test pilot...
TRANSCRIPT
ETS Proficiency Profile Test Pilot Summary Report
Pima Community College, Planning and Institutional Research
Spring 2012
Page 1 of 12
Executive Summary
As part of the College’s ongoing efforts to directly measure its student’s performance in general education proficiency,
Pima has adopted the ETS Proficiency Profile Test to begin standardized testing across the district.
The ETS Proficiency Profile Test is an exam intended to measure general education outcomes at institutions of higher
education. The test is specifically intended to measure four skill areas: critical thinking, reading, writing and
mathematics. According to ETS, the Proficiency Profile Test allows a college or university to: a) gain a full perspective of
the effectiveness of your general education program to meet requirements for accreditation and performance funding,
b) inform teaching and learning with actionable score reports you can use to pinpoint strengths and areas of
improvement, c) take your institution to the next level by providing comparative data on more than 400 institutions and
over 500,000 students nationwide, and d) create greater flexibility in your testing program by adding 50 locally authored
questions and using both paper-and-pencil and online formats, as well as choosing to use proctored and non-proctored
versions. The Proficiency Profile Test is administered to different student cohorts as a means of institutional
effectiveness assessment and for funding and accreditation purposes1.
ETS at Pima Community College
After examining different ways of directly measuring student performance, Pima administrators decided to pilot the ETS
Proficiency Profile Test in spring 2012. Pima chose the ETS Proficiency Profile Test for several reasons. First, ETS is a well-
known and well-trusted testing service provider. Second, the specific test is well-suited for the needs of the College, the
faculty and the student body. Third, the College had previously administered the ETS Academic Profile (an older ETS test
which measured performance in general undergraduate core curricula) which will consequently allow the college to
compare the results from the ETS Proficiency Profile Test to the ETS Academic Profile Pima in order to render a
longitudinal representation of the College’s performance based on direct assessment measures. The Planning and
Institutional Research Department (PIR) of the College was/ is responsible for the implementation of this pilot project.
Pima elected to use the forty-minute (compared to the two-hour), abbreviated version of the test to mitigate impact on
normal classroom activities. ETS uses Criterion-referenced scores for the abbreviated version of the test. Criterion-
referenced scores measure proficiency levels associated with specific skill sets in mathematics, writing and reading,
creating proficiency classifications. These scores are generally used for measuring individual student progress as well as
curriculum and program improvement.
PIR created a random sample to test one-hundred courses at or above the two-hundred level. The courses chosen were
from multiple disciplines across all of the Pima campuses. The test was administered via two delivery methods. The first
method was a proctored, paper-and-pencil, face to face administration. The second method was an unproctored online
administration. The test administration period spanned from 02/10/2012-0310/2012.
Spring 2012 Test Details Number of students tested: 1121 Number of students included in these statistics: 1027 Number of students excluded: 94 (These students were excluded because they completed less than 75% of the test.)
1 http://www.ets.org/proficiencyprofile/about/
Page 2 of 12
Summary of Scaled Scores
Possible Range
Mean Score
95% Confidence Limits2 for Mean
Standard Deviation
25th Percentile
50th Percentile
75th Percentile
Total Score 400-500 438.94 438-440 18.83 425 435 450
Skills Subscores
Critical Thinking
100-130 110.83 110-112 6.07 106 110 115
Reading 100-130 116.49 116-117 6.78 112 116 121
Writing 100-130 112.79 112-114 4.81 109 113 115
Mathematics 100-130 112.09 111-113 6.05 108 111 116
Context-Based Subscores
Humanities 100-130 114.01 113-115 6.34 109 113 118
Social Sciences
100-130 112.60 112-113 6.06 109 112 116
Natural Sciences
100-130 114.29 114-115 5.69 111 114 118
2 The confidence limits are based on the assumption that the questions contributing to each scaled score are a sample from a much larger set of possible questions
that could have been used to measure those same skills. If the group of students taking the test is a sample from some larger population of students eligible to be tested, the confidence limits include both sampling of students and sampling of questions as factors that could cause the mean score to vary. The confidence limits indicate the precision of the mean score of the students actually tested, as an estimate of the "true population mean" - the mean score that would result if all the students in the population could somehow be tested with all possible questions. These confidence limits were computed by a procedure that has a 95 percent probability of producing upper and lower limits that will surround the true population mean. The population size used in the calculation of the confidence limits for the mean scores in this report is 1027.
Page 3 of 12
Scaled Score Distribution by Context (Subject Area)
Page 4 of 12
Total Scaled Score Distribution
Page 5 of 12
Summary of Proficiency Classifications3 (shows the percentage of students proficient at each level4)
Note: Criterion-referenced scores (proficiency classifications) measure the level of proficiency students obtained on a certain skill set. These scores have meaning of their own — without other scores to compare against — but can be used to make comparisons between students or groups of students. The ETS® Proficiency Profile provides criterion-referenced scores of proficient, marginal and not proficient for three skill areas: reading/critical thinking, writing and mathematics.
Demographic Analysis Reports Age:
3 The skills measured by the ETS® Proficiency Profile test are grouped into proficiency levels - three proficiency levels for writing, three for mathematics, and three for the combined set of skills involved in reading and critical thinking. The table and graph show the number and percentage of students who are proficient, marginal, and not proficient at each proficiency level in reading and critical thinking, writing, and mathematics. A student classified as marginal is one whose test results do not provide enough evidence to classify the student either as proficient or as not proficient. See the User's Guide for more information about these classifications, including a list of the specific skills associated with each proficiency level in each skill area. 4 Explanations describing skills by level can be found in Appendix 1
Page 6 of 12
Best Language:
Credit Hours:
Curriculum Completed:
Enrollment Status:
Page 7 of 12
Ethnicity:
Gender:
GPA:
Hours Working:
Page 8 of 12
Major:
Page 9 of 12
Program Enrolled:
Transfer Status:
Supplemental Questions (Created by Pima Community College) Question 1: What is your reason for attending PCC? 0. PCC degree or certificate 1. Transfer to a four year institution 2. Co-enrolled at UA 3. Job skills 4. Personal interest
0 1 2 3 4 5 6 7 8 9
302 (27%) 595 (53%) 93 (8%) 23 (2%) 35 (3%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%)
No Response: 73 (7%) Question 2: Do you anticipate achieving your educational goal this year? 0. Yes 1. No 2. Not sure
0 1 2 3 4 5 6 7 8 9
518 (46%) 423 (38%) 110 (10%) 1 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%)
No Response: 69 (6%)
Page 10 of 12
Question 3: How long have you been taking courses at PCC? 0. Less than 1 year 1. 1 year or longer, but less than 2 years 2. 2 years or longer, but less than 3 years 3. 3 years or longer, but less than 4 years 4. 4 years or longer, but less than 5 years 5. 5 years or longer, but less than 6 years 6. 6 years or longer 7. Not sure
0 1 2 3 4 5 6 7 8 9
263 (23%) 252 (22%) 244 (22%) 154 (14%) 52 (5%) 23 (2%) 46 (4%) 17 (2%) 0 (0%) 0 (0%)
No Response: 70 (6%) Question 4: Do you consider yourself to be Hispanic/Latino? 0. Yes 1. No 2. Prefer not to answer
0 1 2 3 4 5 6 7 8 9
411 (37%) 601 (54%) 38 (3%) 0 (0%) 2 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%)
No Response: 69 (6%) Question 5: In addition, select one or more of the following racial categories to describe yourself: 0. American Indian or Alaska Native 1. Asian 2. Black or African American 3. Native Hawaiian or Pacific Islander 4. White
0 1 2 3 4 5 6 7 8 9
47 (4%) 48 (4%) 58 (5%) 16 (1%) 692 (62%) 5 (0%) 1 (0%) 0 (0%) 0 (0%) 2 (0%)
No Response: 252 (22%) Question 6: Please select from the following age ranges: 0. 13-17 1. 18-19 2. 20-21 3. 22-24 4. 25-29 5. 30-39 6. 40-49 7. 50-59 8. 60-100 9. Prefer not to answer
0 1 2 3 4 5 6 7 8 9
3 (0%) 201 (18%) 286 (26%) 173 (15%) 154 (14%) 118 (11%) 53 (5%) 27 (2%) 7 (1%) 8 (1%)
No Response: 91 (8%)
Page 11 of 12
Appendix 1: ETS Explanation of Proficiency Levels
Skills tested at each level (ETS User’s Guide, pages 9-11) Reading and Critical Thinking To be considered proficient at Level 1, a student should be able to:
recognize factual material explicitly presented in a reading passage
understand the meaning of particular words or phrases in the context of a reading passage To be considered proficient at Level 2, a student should be able to:
synthesize material from different sections of a passage
recognize valid inferences derived from material in the passage
identify accurate summaries of a passage or of significant sections of the passage
understand and interpret figurative language
discern the main idea, purpose of focus of a passage or a significant portion of the passage To be considered proficient at Level 3, a student should be able to:
evaluate competing causal explanations
evaluate hypotheses for consistency with known facts
determine the relevance of information for evaluating an argument or conclusion
determine whether an artistic interpretation is supported by evidence contained in a work
recognize the salient features or themes in a work of art
evaluate the appropriateness of procedures for investigating a question of causation
evaluate data for consistency with known facts, hypotheses or methods Writing To be considered proficient at Level 1, a student should be able to:
recognize agreement among basic grammatical elements (e.g., nouns, verbs, pronouns and conjunctions)
recognize appropriate transition words
recognize incorrect word choice
order sentences in a paragraph
order elements in an outline To be considered proficient at Level 2, a student should be able to:
incorporate new material into a passage
recognize agreement among basic grammatical elements (e.g., nouns, verbs, pronouns and conjunctions) when these elements are complicated by intervening words or phrases
combine simple clauses into single, more complex combinations
recast existing sentences into new syntactic combinations To be considered proficient at Level 3, a student should be able to:
discriminate between appropriate and inappropriate use of parallelism
discriminate between appropriate and inappropriate use of idiomatic language
recognize redundancy
discriminate between correct and incorrect constructions
recognize the most effective revision of a sentence
Page 12 of 12
Mathematics To be considered proficient at Level 1, a student should be able to:
solve word problems that would most likely be solved by arithmetic and do not involve conversion of units or of proportionality (These problems can be multi-step if the steps are repeated rather than embedded.)
solve problems involving the informational properties of numbers and operations, often involving the Number Line, including positive and negative numbers, whole numbers and fractions (including conversions of common fractions to percent, such as converting ¼ to 25%)
solve problems requiring a general understanding of square roots and the squares of numbers
solve a simple equation or substitute numbers into a algebraic expression
find information from a graph (This task may involve finding a specified piece of information in a graph that also contains other information.)
To be considered proficient at Level 2, a student should be able to:
solve arithmetic problems with some complications, such as complex wording, maximizing or minimizing and embedded ratios (These problems include algebra problems that can be solved by arithmetic [the answer choices are numeric].)
simplify algebraic expressions, perform basic translations and draw conclusions from algebraic expressions and inequalities (These tasks are more complicated than solving a simple equation, though they may be approached arithmetically by substituting numbers.)
interpret a trend represented in a graph, or choose a graph that reflects a trend
solve problems involving sets (The problems would have numeric answer choices) To be considered proficient at Level 3, a student should be able to:
solve word problems that would be unlikely to be solved by arithmetic; the answer choices are either algebraic expressions or are numbers that do not lend themselves to back-solving
solve problems involving difficult arithmetic concepts such as exponents and roots other than squares and square roots and percent of increase or decrease
generalize about numbers, e.g., identify the values of (x) for which an expression increases as (x) increases
solve problems requiring an understanding of the properties of integers, rational numbers, etc.
interpret a graph in which the trends are to be expresses algebraically or in which one of the following is involved: exponents and roots other than squares and square roots, percent of increase or decrease
solve problems requiring insight or logical reasoning