measuring what matters; noncognitive skills - webinar hosted by nutn

Measuring What Matters

The Role of Non-Cognitive Factors in Student Success

NUTN Webinar – February,

Dr. Mac Adkins, President, SmarterServices

Provided by

Question 1?

• How do you determine who can be enrolled at your school?– Standardized test scores– Prior grade point averages– Admissions exams

Top Admissions Factors

• The National Association for College Admission Counseling rated these factors.

• CONSIDERABLY IMPORTANT– College prep course grades– Strength of high school curriculum– Standardized test scores– Overall GPA

• MODERATELY IMPORTANT– Admissions essay– Letters of recommendation– Demonstrated interest– Class rank– Extracurricular commitment

Question 2Why Do Students Drop Out?

A study funded by the Bill and Melinda Gates Foundation ranked these reasons:

1. Conflict with work schedule

2. Affordability of tuition

3. Lack of support from family – financial and practical support

4. Lack of belief that a college degree is valuable

5. Lack of discipline – too much socializing, not enough studying

http://www.publicagenda.org/pages/with-their-whole-lives-ahead-of-them

Do You See the Disconnect?

Is Your School Measuring

What Matters?

To Find Out What Matters Let’s Ask:

Employers

Colleges

Faculty

National Research Council

US Department of Education

Mothers

Skills that Employer’s

Want

National Association of Colleges and Employers

Survey of Employers

http://www.unl.edu/svcaa/documents/how_employers_see_candidates.pdf

Outcomes Schools WantElements of Mission Statements From 35 Universities

Michigan State University, 2004

1. Knowledge, learning, mastery of general principles2. Continuous learning, intellectual interest, curiosity3. Artistic cultural appreciation 4. Appreciation for diversity5. Leadership6. Interpersonal skills7. Social responsibility, citizenship and involvement8. Physical and psychosocial health9. Career preparation10.Adaptability and life skills11.Perseverance12.Ethics and integrity

Traits Online Faculty Want

WICHE Cooperative for Educational Technologies, 2013

2012 National Research Council

COGNITIVEProblem solvingCritical thinkingSystems thinkingStudy skillsAdaptabilityCreativityMeta-cognitive skills

INTERPERSONALCommunicationSocial IntelligenceTeamworkLeadershipCultural sensitivityTolerance for diversity

INTRAPERSONALAnxietySelf-efficacySelf-conceptAttributionsWork ethicPersistenceOrganizationTime managementIntegrityLife-long learning

US Department of Education

“The test score accountability movement and conventional educational approaches tend to focus on intellectual aspects of success, such as content knowledge. However, this is not sufficient. If students are to achieve their full potential, they must have opportunities to engage and develop a much richer set of skills. There is a growing movement to explore the potential of the “noncognitive” factors — attributes, dispositions, social skills, attitudes, and intrapersonal resources, independent of intellectual ability—that high-achieving individuals draw upon to accomplish success.”

Parents Teach It

Are You Beginning To See The Picture?

• Non-cognitive skills matter– Determine student retention– Determine employer satisfaction– Determine online course success– Federal agencies recognize their importance– They are the mission of many schools– Parents value them

“Years of schooling predicts labor market outcomes — cognitive skills account for only 20%; therefore 80% of the “years of schooling” benefit is due to noncognitive skills” (Bowles, Gintis, & Osborne, 2001)

http://www.umass.edu/preferen/gintis/jelpap.pdf

Types of Data Used To Predict Learner Success

APTITUDE ATTITUDE SITUATION

What Are Non-Cognitive Skills?

Can Non-Cognitive Skills Be Taught?

You can’t change a tiger’s stripes, but you can teach that tiger to hunt in a different environment.

Recommended Uses of Non-Cognitive Skills Measures

1. Optic – A lens through which students can view their strengths and opportunities for improvement

2. Student Service – A tool to guide students toward available resources for support

3. Placement – Developmental / remedial course placement

4. Talking Points – A collection of statements which academic advisors can use to advise their students

5. Early Alert – A list of students who are likely to be benefitted by the instructor reaching out to them early in the course.

6. Predictive Analytic - A set of data which can be analyzed at the individual and aggregate level to project student performance

Methods of Measurement

• Instructor ratings – Time and task intensive for the faculty• Observer records – Expensive and time consuming• Letters of recommendation – Rarely objective• Interviews – Time consuming to conduct and code• Socioeconomic data – Beneficial mostly at the aggregate level due

to exceptions and bias• Self assessment – Yes, there are limitations, but it is the preferred

method.

Construct Comparison Matrix

ACT Engage

ETS Success Navigator

Wonderlic Admissions Risk Profile

SmarterMeasure

Individual Attributes

X X X X

Life Factors X

Learning Styles X

Technical Skills X X

Reading Skills X

Keyboarding Skills X

Custom Questions X

SmarterMeasure Learning Readiness Indicator

• A 124-item online skills test and attributes inventory that measures a student’s level of readiness for studying online

• Used by over 500 Colleges and Universities• Since 2002 taken by over 2,600,000 students

What Does The Assessment Measure?INTERNAL

INDIVIDUAL ATTRIBUTES

MotivationProcrastination

Time ManagementHelp Seeking

Locus of Control

LEARNING STYLES

VisualVerbalSocial

SolitaryPhysical

AuralLogical

EXTERNAL

LIFE FACTORS

Availability of TimeDedicated Place

ReasonSupport from Family

SKILLS

TECHNICAL

Technology UsageLife Application

Tech VocabularyComputing Access

TYPING

RateAccuracy

ON-SCREEN READING

RateRecall

Adjusting Readiness Ranges

Adjusting the cut points can make the reporting a more accurate predictor of success.

How Do Schools Use It?

• Orientation Course• Enrollment Process• Information Webinar• Public Website• Class Participation• Facebook• 68% of client schools administer the

assessment to all students, not just eLearning students

Thermometer Analogy

• More important than taking your child’s temperature is taking appropriate action based on their temperature.

• More important than measuring student readiness is taking appropriate action based on the scores.

Predictive

Correlation

Comparison

Descriptive

Student Service

Progression of SmarterMeasure Data Utilization

Research Ideas on the Research Page of the Website

Results of Middlesex ResearchFailure rates reduced by as much as 10%

The Findings

• Statistically Significant RelationshipsAcademic Achievement

Engagement Retention

Individual Attributes

X X X

Technical Knowledge

X X X

Learning Styles

X X

Life Factors X X

Technical Competency

X

J. Sargeant Reynolds Community College

• Required as admissions assessment

• Integral part of their QEP• Computed correlations

with grades and SmarterMeasure sub-scales of over 4000 students.

• P

Grades

Attributes

Technical

Learning Styles

Life Factors

Findings• Statistically significant correlations:

Scores Grades

- Dedicated place, support from employers and family, access to study resources, and academic skills (Life Factors)

- Tech vocabulary (Technical Knowledge)

- Procrastination (Individual Attributes)

Academic Success Rates

Skills Resources Time0

10

20

30

40

50

60

70

High Score

Low Score

Less than 10% of students with low scores experienced academic success.

Five Schools

What is the relationship between measures of online student readiness and measures

of online student satisfaction?

Methodology

Data from 1,611 students who completed both the SmarterMeasure Learning Readiness Indicator and the Priority Survey for Online Learners were analyzed.

Incoming vs Outgoing

Findings• There were statistically significant

relationships between factors of readiness and satisfaction.

SmarterMeasure.com

How important do you consider non-cognitive skills?

How is your school measuring and using non-cognitive factors?

For More Info

SmarterServices.com(877) 499-SMARTER

info@SmarterServices.com