compare and contrast of student course evaluation (sce) between waseda and uw

Compare and Contrast of Student Course Evaluation (SCE) between Waseda and UW

YAMAGISHI, Naoji Center for Higher Education Studies (CHES) Waseda University, Tokyo, Japan

Japan Comparative Education Society The 52nd Annual Conference

June 24-26, 2016 @ Osaka University

1

Outline 1. Accountability and the Backgrounds in Japanese H.Ed.

• Grand Transformation in Japanese Higher Education • Japan’s Policy Issues • H.Ed.’s Responses

2. Compare & Contrast between Waseda and UW • Who is in charge? • SCE and its features • Compare & Contrast

3. Issues：What makes the difference？

Yamagishi

１. Accountability: Grand Transformation in Japanese H.Ed.

2

Global competitions & Knowledge-based economy

• Ask me my three main priorities for government…education, education and education (Blair, Oct. 1, 1996)

H.Ed. has become the key for economic success for nations, organizations, & individuals.

Quality of H.Ed. has become EVERYONE’s stakes.

Japan’s demographic concern: declining in the # of 18 years old and increasing the # of institutions falling below their quota

• 18-year-old population was about 2million in 1992, but declined to about 1.2million in 2011.

• the # of private institutions was 372 in 1990 → 605 in 2012

• among private 4-year-colleges, about 45% falls below their quota

• seating capacity (# of enrolled/ # of applicants) = 92%

• about 46% of applicants were admitted without admission tests

Knock, and the (college) door will be opened to you (Matthew 7:7-8 ).

Larger investment on H.Ed., but ambiguous quality. Are college graduates sufficiently educated?

Yamagishi


3 Yamagishi


4 Yamagishi

The # of applicants (thousand)

100% & beyond (left axis)

The # of private institutions

• Applicant-to-seat ratio and the number of applicants (private inst.)

The # of applicants (right axis)

Above 50% below 100% (left axis)

Below 50% (left axis)

Source:https://www.jri.co.jp/MediaLibrary/file/seminar/110602_359/handout_110602_359_01.pdf

１. Accountability: Japan’s policy stance

5

Deregulation of University Act (1991) :

• allowed H.Ed. Inst. to have considerable discretion over curriculum design, degree-grating criteria, etc.

Quality Assurance through Accountability on Student Learning

• Self-assessment & evaluation was stipulated as task for each institution. (University Act, 1991)

• Third party evaluation became compulsory for all H.Ed. Inst. (2004)

Institutional as well as program recognition

• Central Council for Education’s report to the government

Focus on student learning (2008)

Establish institution-wide academic management and PDCA cycle (2012)

Annually open results of assessment of student learning to the public (2012)

• Enforcement Regulations for the School Education Law (2011)

Mandatorily disclose 9 indicators of institution-wide information

• Public financial support (by MEXT): total budget ￥20 billion for 300 institutions for improving education quality that includes establishing institution-wide academic management and utilizing student course evaluation)

Yamagishi

１. Accountability: H.Ed. Responses

6

National Survey 1 (MEXT, 2014) (N=766 institutions <response rate 100%>)

• carried out SCE: 80% institutions（2008）→95％（2012)

• opened the aggregated results to the public: 18%

• opened the aggregated results to students and faculty only: 39%

• not open at all (1%)

National Survey 2 (The U of Tokyo, 2014) (n=555 institutions <response rate 71%>)

• Who can access SCE data: university executives (20%), IRer (14%), each school/department (93%)

Widely spread, but what is in need? (Central Council for Education, 2012)

• The most commonly used tool for grasping student learning is SCE & portfolio : 36.5% for presidents (n=684), 40.1% for deans (n=1,929)

• Important thing for improving education program is to establish a supporting system: 91.6% presidents (most important + important) presidents (n=684) said

How to manage SCE and how to use SCE results becomes a significant issue.

Yamagishi

2. Compare & Contrast: Waseda & UW

7

The University of Washington as a pioneer of SCE

• UW opened its door in 1861.

• UW is publicly run, and is one of “Public Ivys.”

• 7 undergraduate schools and 16 graduate schools

• # of enrolled students: about 45,000

• SCE began in the U.S. during about 1920’s (Fujita & kawashima, 2014).

• UW was one of the three pioneers of SCE (Fujita & kawashima, 2014).

Waseda University

• Waseda established in 1882.

• Waaseda is a private institution.

• 13 ungraduated schools and 21 graduate schools

• # of enrolled students: about 53,000

• SCE at Waseda began in 2001.

• Waseda revised its SCE in 2015, by referring to UW’s

Yamagishi

2．Compare & Contrast: Waseda & UW

8 Yamagishi

Who is in charge of SCE: Waseda

Who is in charge of conducting SCE Center for Higher Education Studies (CHES)

Who develops/revises of SCE CTLT Faculty WG within CHES

Who analyzes the data an outside contractor

On organizational chart CTLT is a committee within CHES

The director of CHES is the Senior Executive Vice President for Academic Affairs &Provost

Expected roles

・executes course evaluation・reports the results (both school and course level) to each relevantinsturctor・reports the results (both school and course level) to the evalutioncommittee of each relevant school・reports to the aggregated results (insitutional level) to the committee of theheads of academic affairs of each school・opens the aggregated results (insitutional level) to the public on theuniversity HP

Staff・8 administrative staff, 2 of them (collateral office with Academic Affairs Division) incharge of SCE・will be replaced by others thourgh job rotation


9 Yamagishi

Who is in charge of SCE: UW

Who is in charge of conducting SCE Office of Educational Assessment (OEA)

Who develops/revises SCE OEA

Who analyzes the data OEA (IASystem)

On organizational chart is part of Undergraduate Academic Affairs

reports to the Vice Provost & Dean for Undergraduate Academic Affairs

Expected roles provides course evaluation service at UW & to other insts.

provides placement testing & classroom test scoring at UW Seatle Campus

provides different types of assessment at all levels at UW

Staff 16 staff members

Some are resarch specialits in quatitative & qualitative methods

(e.x.) the director with Ph.D. in Education (Measurement & Evaluation)(e.x.) research staff with Ph.D. or graduate degrees in Psychometrics

Service rates for departments at UW・free of charge for online・relative to cost of service for paper ($.26/form)

Evaluation forms $.10/form

Optical scannning $.16/from

Billed to departments

Service rates for outside UW (online) Based on the # of enrollments

(e.x.) Enrollment category 1 (total FTE <2,000)・$1.25 (annual licensing fee/ enrolled student)・$3,500 (initial setup fee)

(e.x.) Enrollment category 6 (total FTE >20,000)・$1.00 (annual licensing fee/ enrolled student)・$3,500 (initial setup fee)

(e.x.) 40,000 FTE $1.00*40,000+$3,500=$43,500


10 Yamagishi

SCE and its features: Waseda

History began in 2001

online evaluation was introduced in 2004

currently both paper and online types are used

# of courses evaluated about 12,000 courses annually

Functions assists faculty in instrucational development

helps schools to know and improve its quality of education

provides schools with reference information for teaching award

Types of forms4 forms ・Lecture, Seminar, Skill Acquisition & Lab, Distance Learning

Form structure 4 sets of items on a given evaluation form

1set. course registration requirement Compulsory/Required, Compulsory Electives, Electives

2set. students' self-evaluation(e.x.) Indicate the time you spent on preparation and review for each courseoutside of the class.

3set. students' reviw on course(e.x.) The instructor effectively facilitated your participation in the form ofquestions and constructive comments.

4set. open ended questionsPlease state below any ideas you can think of in order for this course to bemore beneficial for future students.

Access to the results Instructors

School Evaluation Committee at each school


11

Yamagishi

SCE and its features: UW

History began in 1920's

Instructional Assessment System (IASystem) was created in the early 1970's

online evaluation was introduced in autum 2013

currently both paper and online types are used

# of courses evaluated about 12,000 courses annually

more than 60 other post-secondary institutions use IASystem

Functions assists faculty in instrucational development

informs administrative decision-making in such as merit, promositon, and tenure

helps students to select courses

Types of forms 14 forms (e.x.) Small Lecture, Seminar, Lab, Distance Learning, Project, and such

Form structure 4 sets of items on a given evaluation form

1set. Summative items the first four items, common in all forms

(e.x.) The course as a whole was

2set. Formative items about 18 items related to particular instructional forms

(e.x.) Quality of questions or problems raised by instructor was

Instructors can add their own questions

3set. Relative to other courses the last nine items, common in all forms

student academic challenge and engagement

(e.x.) The amount of effort you put into this course was

4set. open ended questions (e.x.) What aspencts of this class detracted from your learnig?

Access to the results: Everyone with UW Net-ID

(e.x.)Students, Instructors, School deans and department chairs,Faculty Curriculum Committee, Tenure/Promostion Committee, Executives (Provost/President)


12 Yamagishi

Waseda UW

Ability to evaluate courses either online or onpaper

〇〇

Summative questions on all forms to allowcross-course and cross-instuructor

〇（1st set)

Formative questons to provide feedback onspecific aspects of each course

〇〇（2nd set)

Index of student academic challenge andengagement

〇（3rd set): Challenge & Engagement Index (CEI)

Ability to add instructor generated questions 〇〇

Integrated data archiving and reporting foronline and paper evaluations

〇

Multi-level reproting by course, department,and school

〇

Institutional norms for comparison 〇

Online interface for couse evaluationmanagement

〇（Waseda net Course＠Navi）

〇（faculty portal @ UW NetID)

Responses to be connected to an individualstudent

〇（Waseda net Course＠Navi）

〇（IASystem）

Reponders' anonimity for instructors 〇〇

Research application ○(without IRB: Institutional Review Board )

Specialized research staff 〇

Charge × for online,〇 for paper

Finance ・university budget・partially state-funded through university budget

・partially self-sustaining

3. Issues：What makes the difference?

13

Finance

• OEA is partially state-funded and partially self-sustaining.

• OEA can extend their business by selling more SCE service to other institutions.

Human Resource

• OEA’s HR includes those with strong graduate training in appropriate fields.

Research

• SCE data can be studied, and the research results can be published without any commission review.

Right to Know

• Everyone with UW Net-ID can view on-line Course Evaluation Catalogue (CEC) ; CEC displays SCE results of each course with its instructor(s)’ name on.

• SCE results are considered as institutional public goods; the members can share the information.

• The display of evaluation results was originally required by ASUW. ASUW has paid the cost.

• “Course evaluation data are considered public information, so the faculty can’t object to display.”

Yamagishi