test security test prep paullin_2005

How Big is the Test Preparation Industry

and What are Employers Doing to

Counter It?

April 16, 2005

Cheryl Paullin

Personnel Decisions Research Institutes, Inc.

In Paullin, C. (Chair). Maintaining test security in a “cheating” culture. Practitioner Forum conducted at the 20th Annual Conference of the Society for Industrial and Organizational Psychology, Los Angeles, CA.

April 16, 2005 Paullin: Test Prep Industry 2

Disclaimer

• I am not saying that test preparation and cheating are the same thing.

• I am not saying that it is unethical to provide test preparation or test coaching services. I do believe that there are unethical providers out there.

• Test prep is a good thing to the extent it:

– Improves relevant skills or knowledge.

– Improves general test-taking skills.

– Reduces text anxiety.


Method

• Searched Research literature – keywords: – [employment] test preparation

– test coaching

• Searched the Internet, libraries, and bookstores for information about organizations that provide test preparation services.

• Talked with colleagues and clients about what employers are doing to counter the test prep industry.


Research Literature

• Sackett, Schmitt, Ellingson, & Kabin (2001)*

– Reviewed test coaching programs, specifically to evaluate impact (if any) on subgroup score differences on standardized tests.

• Conclusions -- Test Coaching Programs:

– Benefit minority & nonminority examinees slightly, but do not reduce subgroup score differences.

– Are viewed very positively by examinees.

– Can have a large practical impact, e.g., when one or two points is the difference between passing & failing a certification exam.

*American Psychologist, Vol 56 (4), 302-318.


Type of Exams Covered

• Education Program Admission (SAT, GRE, GMAT, LSAT, MCAT)

• Proficiency/Placement (GED, High School Exit Exams, Advanced Placement, English & Foreign Language Skills)

• Employment

– Selection (ASVAB, USPS Exam 470, Police Officer, Civil Service, Fire Service).

– Certification/Licensure (Health Care Professions, Teacher, Accountant, Engineer, Real Estate, Financial Planning, Cosmetology, Information Technology).


Scope of Test Prep Industry

• Number of orgs offering test prep generally

corresponds to size of examinee population

(i.e., the market): – Dozens for large-scale admission exams and for

nationwide or regional certification, licensure, or

employment exams.

– At least one or two for virtually all location- or

employer-specific exams administered on a regular

basis to more than a few hundred people.


Scope of Test Prep Industry (con’t)

• Test prep is avail for > 100 different exams.

• For anything other than printed test guides,

almost impossible to evaluate quality prior

to buying the service/materials.

• One website compares services (more later).


Test Prep Activities

• Test Prep Activities – Test guide books (avail for > 100 different exams).

– Practice tests (CD-based or online avail for 75-100 exams).

– Prep courses (CD-based, online, or live avail for 20-30 exams).

– Personal tutoring (for several high-stakes exams, e.g., SAT,

GMAT, Medical Licensure, Bar Exam, CPA).

• Practice Tests – Vary in extent to which they simulate real testing conditions.

• P&P, scoring key & explanation provided.

• Real-time administration & scoring with feedback -- access to item

bank for X number of tests or X number of hours.


Test Prep for a Fee

• Providers covering the widest array of exams: – Kaplan

– Learning Express

– Thomson Learning

– Princeton Review

– Cliff’s Test Prep

• Supplemented by an uncountable number of “Dr. Paullin’s Best Test Prep Ever” websites.


Costs & Claims • Costs

– $10-$50 for Test Guide books.

– $100-$300 for CD-based course, or limited online access to practice tests.

– Several hundred dollars for a lengthy online course, live course, or several hours of personal tutoring.

– Costs are higher for exams related to high-income professions, e.g., medical licensure, dental licensure, bar exam, CPA.

• Claims – Many test prep orgs offer a money-back guarantee unless X

happens, e.g., earning a passing score.

– Most say little or nothing about qualifications of the test prep developer(s); a few say course is taught by or materials were developed by someone who scored high on the exam in question.


Comparing Test Prep Providers

www.test-preparation.net

• For each of 23 exams:

– Which orgs offer test prep services.

– Type and cost of prep activities.

• Doesn’t evaluate quality of materials or providers.


Test Prep as a Public Service

• Public Sector Jobs

– Several federal government websites

www.usajobs.opm.gov.

– Public Service Employees Network

www.pse-net.com.

• College entrance – www.number2.com

• Labor Unions – Not much on websites at national

level; may be more info at local branches.


Test Developers/Sponsors

• Several offer test prep, some of it free, e.g.,:

– College Board – SAT.

– ETS (Educational Testing Service) – many tests.

– ACT (American College Testing) – ACT.

– American Association of Medical Colleges – MCAT.

– National Association of Pharmacy Boards – PCAT.

– Project Management Institute – PMI Certification.

– Pearson VUE – Nursing Certification.

– Experior Online -- state-specific certification exams.


What are Employers Doing to

Counter Test Prep Industry?

• Relying on predictors other than multiple-choice

tests, e.g., interviews.

• Outsourcing the testing process – countering test

prep industry someone else’s problem.

• Changing test content frequently.

– Multiple forms Adaptive testing.

– New test for each mass administration.

– Easier to do with computer-based & online testing.


What are Employers Doing? (con’t)

• Providing their own test prep materials.

• Warning test-takers to be cautious when

buying test prep services.

• (rare) Legally pursuing test prep

orgs/individuals that make fraudulent

claims or provide access to controlled test

content.


Bogus Credentials

• Issue of increasing concern.

• Search term “Diploma Mill” 10 websites.

• What’s an employer to do?

– http://ope.ed.gov/accreditation -- US Office of Post-

Secondary Education.

– Hire an organization that does education verification.

– If possible, assess KSAs directly.

http://ope.ed.gov/accreditation


Test Security in Litigation

• Attorneys may request, or court may order, that

test items and scoring key be shared with parties

to litigation or read into the court record.

• What should the I/O Psychologist do? • Principles of Psychologists and Code of Conduct (American

Psychological Association, 1992) -- If psychologists’ ethical

responsibilities conflict with law, psychologists make known their

commitment to the Ethics Code and take steps to resolve the

conflict in a responsible manner. (Standard 1.02)


Test Security in Litigation (con’t.)

• Standards for Educational and Psychological Testing (1999) state: When sensitive test documents are challenged, it may be appropriate to employ a third party, using a closely supervised secure procedure to conduct a review of the relevant materials. Such secure procedures are generally preferable to placing tests, manuals, and an examinee’s test responses in the public record.

• American Psychologist editorial (1999): The psychologist should ask the court to allow delivery of the secure materials only to psychologists or professionals who are bound by the an ethical code to maintain their security. If this isn’t feasible or isn’t allowed, then request that the court issue a protective order prohibiting the parties from making copies of the materials, requiring that the materials be returned to the psychologist at the close of litigation, and ordering that the record be sealed if test questions or answers are admitted in court.

test security test prep paullin_2005

Business