Katsumi Senyo Douglas GloagShigeru Asano Takao HashizumeKoji Fujiwara Jun-ichiro Yasuda

October 22, 2018

2018 Assessment Institute

Using a Smartphone-based Integrated Data Collection System to Measure Student Learning Gains

Go to Google and search “Yamagata OIRE”

Or type https://ir.yamagata-u.ac.jp/AI2018.pdfOr scan with QR Code Reader

1

Slides are available!

Here!

2

Agenda

1. About Us2. Key Characteristics of

Higher Education in Japan3. Data Collection via smartphone4. CBT Assessment for General Ed –

Development & Analysis5. FYE Program / Big 5 personalities6. Results (Grades / Attendance)7. Conclusions

Douglas

Douglas

Jun-ichiro

Takao

3

Where is Yamagata, Japan?

Direct Distance from Tokyo: 190 miles• 3 hours by bullet train• 45 minutes by airplane

4

Yamagata University (YU)

■Type: National University• multi-campus (4 campuses)

■Students: 8,800 • 7,500 undergraduates

• 1,700 new incoming/year• 1,300 graduates

■Full-time Faculty: 900

■Six Distinct Colleges:• Agriculture• Literature and Social Science• Education, Arts and Science• Sciences• Medicine• Engineering

• Yamagata City – Tsuruoka City: 60 miles• Yamagata City – Yonezawa City: 30 miles

5

Yamagata University (YU)

Degree Programs Undergraduate: 22 Master: 19 Doctorate: 10

2017-2018 FT Enrollment & FT Faculty/Staff Degree-seeking UG: 7,546

• Male: 62% Degree-seeking, first-time freshmen: 1,731

• In-state: 23% Degree-seeking Grad: 1,394

• Male: 75% Faculty & Staff: 800+ & 300+, respectively

Same Tuition$4,800/year

6

Key Characteristic of H.E. in Japan

The academic year begins in April and ends the following March.

Students must decide on their major when they apply for admission.• Can’t change the major after enrollment• No double or more majors• No pre- program (e.g., Pre-Med, Pre-Vet, etc.)• Almost impossible to

Transfer to another university Have flexible study plan

High expectation to graduate in four years

7

Data Collection via Smartphone

8

Assessment via Smartphone

FACT: 99+% of YU students own a smartphone with Bluetooth4.0.

We would like to…a. Collect assessment data more effectively.b. Provide the personalized info. more efficiently.

• At YU, the data/info. is not centralized.c. Track classroom attendance more cost-effectively.

• Previously:NFC-based reader & IC chip-embedded student ID card

• Ongoing:Bluetooth4.0(BLE) based beacon & students’ smartphone

Much info. collected & provided

9

Attendance Tracking at YU

Current MethodNFC-based reader

IC chip-embedded student ID card

✚

NFC = Near-field Communication

10

Smartphone Application: YU Portal

Important Info. is now centralized!!

11

Main Features

Test Survey Personalized Class Schedule Class Attendance Record Personalized Notification

• e.g., Class Cancellations

ClassSchedule

Attendance Record

Semester Course Schedule

Test

12

Our TEST: CBT Assessment for General Education

- Development and Analysis -

13

TEST Objective

• Assess learning outcome directly & effectiveness of education program Senior

Sophomore, Junior (Specialized Ed)

Freshman (General Ed)

Assessment 1

Assessment 2

Assessment 3

14

Schedule

Planning, Pre-survey

Improvement, Pre-test

TEST

to Freshman

TEST to Freshman, Sophomore

2015 2018Apr

2017Apr2016

• 1,700 students for each grade

15

TEST Development

Subjects Statistics, Mathematics, Physics, Chemistry,

Biology Created 30~45 questions for each subject

Purpose To assess not only memorization but also

understanding of scientific concepts

16

Resources

Our TEST

Past exam

Our original questions

Diagnostic

test

Physics Force Concept

Inventory

Chemistry Concept test

Biology Bio- literacy

etc.

17

Example of Question (Physics)

Q. Which is the correct force direction in a parabolic motion?

18

TEST Design

1. Assessment via smartphone application

2. Use of Computerized Adaptive Testing (CAT) Questions are selected based on a response

Computerized Adaptive Testing

Q.1

Q.2(Difficult)

Q.2(Easy)

correct

incorrect ・・・

・・・

19

TEST Design

1. Assessment via smartphone application

2. Use of Computerized Adaptive Testing (CAT) Questions are selected based on a response

3. Students answer 5 questions for each subject

20

Computerized Adaptive Testing

With 5 questions, students’ abilities can be graded in 32 levels

21

TEST Design

1. Assessment via smartphone application

2. Use of Computerized Adaptive Testing (CAT) Questions are selected based on a response

3. Students answer 5 questions for each subject4. Time limit is 3 minutes for each question Test time is about 30 minutes for 5 subjects

22

Item Response Theory

1. Estimate the difficulty parameters of each question in a pre-test

23

Schedule

Planning, Pre-survey

Improvement, Pre-test

TEST

to Freshman

TEST to Freshman, Sophomore

2015 2018Apr

2017Apr2016

• 1,700 students for each grade

• To ~ 60 students (various majors Freshmen in December)

24

Item Response Theory

1. Estimate the difficulty parameters of each question in a pre-test

2. Estimate the ability parameters (θ) of examinees in our TEST

3. We used 1 parameter (Rasch) model

25

TEST Examination Rate

Statistics Math Physics Chemistry Biology

Examinee All Only science course students

Class of 20171st time

99.3% 99.0% 99.0% 99.0% 99.3%

Class of 20181st time

99.2% 99.6% 99.6% 99.6% 99.5%

Class of 20172nd time

89.2% 90.5% 90.2% 90.2% 90.6%

26

Class of 2017: 1st -time vs 2nd ; Chemistryability θ 1st

ability θ 2nd

ability θ diff. p d

all -0.19 0.09 0.28 0.00 0.33A -0.14 0.31 0.45 0.00 0.50

B 0.38 0.44 0.06 0.55 0.08

C -0.48 -0.56 -0.09 0.39 -0.13

D -0.22 0.50 0.72 0.00 0.81

E -0.05 0.32 0.37 0.00 0.46

F -0.39 -0.43 -0.04 0.54 -0.06

G -0.16 0.18 0.34 0.00 0.40

H -0.36 -0.11 0.26 0.43 0.33

I -0.45 0.16 0.62 0.00 0.72J -0.45 -0.37 0.08 0.22 0.13

For Freshman in Dec.2016,θ ave. = 0θ sd. = 1

p < 0.05

Cohen’s d + －

small

middlelarge

27

ability θ 1st

ability θ 2nd

ability θ diff. p d

all -0.08 0.03 0.11 0.00 0.12A -0.13 0.12 0.25 0.00 0.27

B 0.42 0.71 0.29 0.00 0.32

C 0.61 1.00 0.39 0.00 0.42

D 0.09 0.21 0.12 0.24 0.17

E -0.40 -0.38 0.02 0.79 0.02

F -0.54 -0.43 0.11 0.59 0.09

G 0.00 0.01 0.01 0.91 0.01

H -0.56 -0.60 -0.04 0.65 -0.06

I -0.54 -0.52 0.02 0.94 0.01J -0.39 -0.19 0.20 0.36 0.34

Class of 2017: 1st -time vs 2nd ; Biology

For Freshman in Dec.2016,θ ave. = 0θ sd. = 1

p < 0.05

Cohen’s d + －

small

middlelarge

28

Class of 2017: 1st -time vs 2nd ; Mathability θ 1st

ability θ 2nd

ability θ diff. p d

all -0.10 -0.14 -0.05 0.12 -0.05A 0.02 0.03 0.01 0.94 0.01

B 0.48 0.06 -0.42 0.00 -0.57

C -0.96 -0.94 0.02 0.85 0.02

D -0.07 -0.13 -0.06 0.57 -0.07

E -0.25 -0.06 0.19 0.02 0.24

F 0.08 0.11 0.03 0.70 0.04

G -0.01 0.14 0.15 0.04 0.19

H -0.07 -0.27 -0.20 0.49 -0.23

I 0.03 -0.10 -0.13 0.48 -0.17J -0.48 -0.75 -0.27 0.00 -0.32

For Freshman in Dec.2016,θ ave. = 0θ sd. = 1

p < 0.05

Cohen’s d + －

small

middlelarge

29

Class of 2017: 1st -time vs 2nd ; Physicsability θ 1st

ability θ 2nd

ability θ diff. p d

all 0.13 0.19 0.07 0.03 0.07A 0.16 0.33 0.17 0.03 0.20

B 0.34 0.20 -0.14 0.23 -0.14

C -0.52 -0.55 -0.03 0.78 -0.04

D 0.12 0.28 0.16 0.07 0.19

E 0.04 0.11 0.08 0.32 0.09

F 0.43 0.39 -0.04 0.68 -0.05

G 0.53 0.68 0.15 0.11 0.19

H 0.43 -0.06 -0.49 0.05 -0.56

I 0.14 0.34 0.19 0.31 0.21J -0.46 -0.37 0.09 0.23 0.12

For Freshman in Dec.2016,θ ave. = 0θ sd. = 1

p < 0.05

Cohen’s d + －

small

middlelarge

30

Class of 2017: 1st -time vs 2nd ; Statisticsability θ 1st

ability θ 2nd

ability θ diff. p d

all -0.04 0.00 0.04 0.18 0.04A 0.15 0.07 -0.08 0.33 -0.09B 0.45 0.28 -0.18 0.08 -0.22C -0.30 -0.06 0.24 0.05 0.40D -0.17 0.02 0.19 0.08 0.22E -0.08 0.00 0.08 0.40 0.09F 0.00 0.12 0.12 0.22 0.13G -0.04 0.03 0.07 0.53 0.07H 0.35 0.40 0.05 0.87 0.06I -0.22 -0.17 0.06 0.73 0.06J -0.03 -0.12 -0.08 0.37 -0.09K -0.35 -0.07 0.28 0.05 0.32L -0.41 -0.48 -0.07 0.71 -0.08M -0.10 0.00 0.10 0.32 0.11N -0.07 -0.11 -0.04 0.76 -0.05O -0.27 -0.30 -0.04 0.77 -0.05

For Freshman in Dec.2016,θ ave. = 0θ sd. = 1

p < 0.05

Cohen’s d + －

small

middlelarge

31

Summary

• To assess students’ learning outcome directly,we developed computerized adaptive testsfor five scientific subjects

“Important message”• Students’ abilities are improved if YU’s

teaching and learning system works• Students’ level in a particular subject can not

be maintained if we do not provideopportunities to learn it

32

33

First-Year Experience (FYE) Program＆

Big Five Personality Traits

FYE Program at YU

Course Goals1. To promote further development of

student success skills2. To help students gain confidence at

new environment (learn from experience)

Characteristics• 2 semester credits (meet 15 times, 100 min./class)• 25 students per class• Mandatory for ALL new first-year students• Target Course Grade: B or above• (Unofficial) Attendance Requirement: 80%

34

YU Start-up Seminar (SUS)

Improve students’ necessary study skills*

Characteristics• Same syllabus / slides• Same assignments and grading• Class size ~25：

Mix of major and gender• 08:50~10:20 time slot

Changes to Class of 20181. Peer/Mentor program initiated2. Slight reduction in class size

Format is aligned

* Data gathering, Discussion, Presentation, Report writing 35

Big Five Personality Test

Localized version developed by Professors Chieko and Yoshihiro Murakami 70 Yes-No Questions

Big Five Personality Traits• Openness to Experience• Conscientiousness• Extraversion• Agreeableness• Emotional Stability

(Changed from Neuroticism)

Are they related to FYE class grade and/orattendance?

36

37

Results, Comparisonsand Conclusions

Comparing Big 5 Results Class of 2017：1st time (April 2017) vs. 2nd time (April 2018)

Stability Coefficient:1st time vs 2nd time Stability Coefficient（ρ）Value showing exactness of result of the same student taking the same test again and again.

ρ gives a value from 0-1 with a score closer to 1 having a higher stability coefficient.

Big 5 Traits Class of 20171st time* 2nd time**

Mean Std. Dev. Mean Std. Dev.E: Extraversion 46.0 10.5 46.0 10.1A: Agreeableness 52.0 8.7 52.4 9.0C: Conscientiousness 54.5 9.5 54.3 9.7N: Emotional Stability 47.6 9.7 48.4 9.3O: Openness 49.4 9.7 49.8 9.8* Students meeting 2017 entry requirements（N = 1,691）

** N = 1,461

Big 5 Traits Class of 2017（ρ）*E: Extraversion 0.80A: Agreeableness 0.60C: Conscientiousness 0.68N: Emotional Stability 0.67O: Openness 0.65* All stability coefficient values significant（p < 0.0001）

39

Analysis Method

Basic Question• Is there any relation between the Big 5 and

academic achievement?

Student Data used for analysis• Attendance and grades of 1st semester

“Start-up Seminar” (2 credits)• Student GPA scores of 1st semester – Not used due

to lack of input time

Analysis scope1. Big 5 vs. “Start-up Seminar” Grades2. Big 5 vs. “Start-up Seminar” Attendance

40

Results concerning SUS Grades （1）

5 grade evaluation：A, B, C, D or F SUS is designed to give students an ‘B’ or higher if they

meet these conditions:

1. Hand in weekly assignment by deadline2. Meet specified course criteria

→ A or B ≅ Outcome accomplishment

Grade comparison：2017 vs. 2018 Evaluation Bands

Class of 2017 Class of 2018Freq. Percent Freq. Percent

Good（A or B） 1,427 84.4 1,499 87.1Poor（C, D or F） 264 15.6 222 12.9

Total 1,691 100.0 1,721 100.0

41

94%

82%

A/B C/D/F

Class of 2018

93%

82%

A/B C/D/F

Class of 2017

Results concerning SUS Grades（2）

Attendance comparison with grades

***

*** p < 0.0001

***

42

SUS Attendance

Attendance：80% expected

Attendance comparison: 2017 vs. 2018

Attendance Levels Class of 2017 Class of 2018Freq. Percent Freq. Percent

Good Attendance（80%+） 1,515 90.0 1,524 88.6Poor Attendance（<80%) 176 10.0 197 11.4Total 1,691 100.0 1,721 100.0

43

SUS: Attendance vs. Grades

***

***

*** p < 0.0001

91%

54%

9%

46%

GOOD ATTENDANCE（80%+）

POOR ATTENDANCE（<80%)

Class of 2018

A/B C/D/F

88%

53%

12%

47%

GOOD ATTENDANCE（80%+）

POOR ATTENDANCE（<80%)

Class of 2017

A/B C/D/F

44

Big 5 vs. 2017 Student Grades

60%

70%

80%

90%

100%

Extraversion Agreeableness Conscientiousness Emotional

Stability

Openness

Grade scoresof ‘A’/’B’

Class of 2017Start-up Seminar

P = 0.97 P = 0.02 P = 0.000 P = 0.04 P = 0.2

Low High← Big5 Score →

45

Big 5 vs. 2018 Student Grades

60%

70%

80%

90%

100%

Extraversion Agreeableness Conscientiousness Emotional

Stability

Openness

Grade scoresof ‘A’/’B’

Class of 2018Start-up Seminar

P = 0.07 P = 0.12 P = 0.000 P = 0.72 P = 0.003

Low High← Big5 Score →

46

Summary: Big 5 vs. SUS Grades

Grades and related Big 5 traits

Implied ResultsClass of 2017 Class of 2018

Agreeableness N/AConscientiousness Groups with high

scores in this trait get higher grades

Groups with high scores in this trait get higher grades (same trend)

Emotional Stability Groups with high scores in this trait get lower grades

Openness to experience

Groups with high scores in this trait get lower grades

47

Big 5 vs. 2017 Student Attendance

60%

70%

80%

90%

100%

Extraversion Agreeableness Conscientiousness Emotional

Stability

Openness

80%+Attendance Class of 2017

Start-up Seminar

P = 0.72 P = 0.79 P = 0.000 P = 0.59 P = 0.32

Low High← Big5 Score →

48

Big 5 vs. 2018 Student Attendance

80%+Attendance Class of 2018

Start-up Seminar

60%

70%

80%

90%

100%

Extraversion Agreeableness Conscientiousness Emotional

Stability

Openness

P = 0.32 P = 0.08 P = 0.02 P = 0.26 P = 0.000

Low High← Big5 Score →

49

Summary: Big 5 vs. SUS Attendance

Attendance related to Big 5

Implied ResultsClass of 2017 Class of 2018

Conscientiousness Clear difference in attendance with high scores having high attendance and low scoring students having lower attendance

Groups with high scores in this trait have higher attendance (almost same trend)

Openness to experience

Groups with high scores in this trait have lower attendance

50

60%

70%

80%

90%

100%

Extraversion Agreeableness Conscientiousness Emotional

Stability

Openness

Combined Data: Big 5 vs. Grades

P = 0.34 P = 0.01 P = 0.000 P = 0.13 P = 0.001

Grade scoresof ‘A’/’B’

Low High← Big5 Score →

51

60%

70%

80%

90%

100%

Extraversion Agreeableness Conscientiousness Emotional

Stability

Openness

Combined Data: Big 5 vs. Attendance

80%+Attendance

P = 0.42 P = 0.31 P = 0.000 P = 0.39 P = 0.000

Low High← Big5 Score →

52

Summary of Results and Future Plans

Personality traits that may be connected to SUS data on grades and attendance

Future Plans Develop Early-Alert (Early-Warning) System for YU

• Implement effective intervention program• Instruct students as to what is required of them

Class of 2017 Class of 2018 CombinedGrades Agreeableness？

Conscientiousness↑Emotional Stability↓

Conscientiousness↑

Openness↓

Agreeableness？Conscientiousness↑

Openness↓

Attendance Conscientiousness↑ Conscientiousness↑

Openness↓

Conscientiousness↑

Openness↓

53

Conclusions (1)

What we have done Two years data collection Understanding students’ study

preferences

What we have found Students’ abilities are improved if YU’s

teaching and learning system works Students’ level in a particular subject can

not be maintained if we do not provide opportunities to learn it

54

Conclusions (2)

What we will do Widen findings from data analysis Introduce logical reasoning questions Improve FYE class content Get more data from other universities Apply our findings to improve YU’s

education continuously

55

Acknowledgements

Special Thanks (in alphabetical order) to:• Shinobu Araki• Takahiro Abe• Takahiro Iijima• Nobuhiro Matsuzaka• Koto Sato• Tatsuya Suzuki• Eriko Watanabe• Minako Yamamoto

This project issupported by MEXT, Japan.

andAll Students, Faculty, Staff & Administrationat Yamagata University

The smartphone App is developed with

56