five year self study training

1

FIVE YEAR SELF STUDY TRAININGDr. Mary Hazzard, Assessment FellowDr. Jacque Caesar, Assessment Fellow October 23, 2013(Notes taken at the training session and added here by SETM SAC Chair are in italic.)

2

OUTLINE Learning Outcomes Assessment at NU Sections of Program Self-Study Report (PSSR) Program Review Purpose of Assessment NU Assessment Plan Possible Reflective Over-Arching Questions Program Self-Study Committee (PSC) Role of the School Assessment Committee (SAC) Tips from a Program Lead Rubrics Components of Program Self-Study Report (PSSR)

Overarching Reflective Questions Self-Study Plan Section 1 – General Information Section 2 – Relevancy of the Program Section 3 – Currency of the Program

3

OUTLINE - CONTINUED Section 4 – Faculty Qualifications Section 5 – Preparedness and Academic Support of Students Section 6 – Student Achievement Section 7 – Academic Success of Students Section 8 – Program Vitality – Life Cycle Analysis Section 9 – Adequacy of Resources Section 10 – Other Information Needed for This Self-Study Section 11 – Summary and Recommendations Section 12 – External Reviewer(s) Report

External Review Process for Contract for External Reviewer Sample Contract 1 Sample Contract 2

AMS Memorandum of Agreement (MOA) Self-Study Plan: Summary Review Process Resources Provided by the Office of Institutional Research and

Assessment (OIRA) Summary

4

LEARNING OUTCOMESThe program lead will be able to: Identify overarching questions for the self

study Develop a self study plan with timeline Identify members for a self study committee Identify potential external reviewer(s) Identify questions that could guide data

collection and analysis for each section of the self study

Identify resources they could use as they complete the self study.

Complete a self study for their program.

5

ASSESSMENT AT NU Five Year Review

Based in Inquiry – what does the faculty need to know to improve the program

Evidence-based (relevant qualitative and quantitative evidence) decision making

Evidence of program quality Curriculum and Learning Environment Faculty Evidence of program viability and sustainability Information and technology resources Facilities Staff

6

SECTIONS OF SELF-STUDY (ALL IN AMS)

Self Study Plan Overarching Question(s) General Program Information Relevancy Currency Faculty qualifications Preparedness and Academic support of students Student achievement Academic success of students Program vitality Adequacy of resources Other Summary and recommendations External Review Memorandum of Agreement (MOA)

7

PROGRAM REVIEW Includes

Self-evaluation of program Based in Inquiry – what does the faculty need to know to improve

the program Comprehensive analysis of program quality Analysis of a wide variety of data about the program

Peer-evaluation by reviewers external to program, department, and university

Internal peer-evaluation Used to:

Inform follow up planning Inform budgeting process Improve quality of program Improve student learning Update multi year assessment plan

8

PROGRAM REVIEW The “program five-year review” refers to the program

self-study activities, such as planning, coordinating, data collection, and data analysis, etc., performed by the program lead with his/her Program Self-Study Committee.

The scope of data to be included is data from the past five academic years or since the last five-year review was done.

The deliverables of a five-year review include: A Five-Year Self-Study Plan A Program Self- Study Report (PSSR) which also includes one

report from each external reviewer A Memorandum of Agreement (MOA)

9

Ensure evidence-based decision making Provide accountability to the public and the

profession Adherence to adopted standards Ensure students are learning Ensure high quality programs Encourage programmatic improvement Ensure high quality graduates

PURPOSE OF ASSESSMENT

10

NU ASSESSMENT PLAN To assure that we know what are

students learn Provide alignment across the mission

and through the Institutional Learning Outcomes, Program Learning Outcomes, to the alignment of institutional resources

Activities designed to achieve the National University’s educational objectives

Transparent and collaborative process

12

POSSIBLE REFLECTIVE OVER-ARCHING QUESTIONS * Please choose one to three over-arching questions or compose your own over-arching questions relevant to your program and to be answered by the five-year review and in the PSSR: think of what need to be done to improve/maintain the academic program in the next five years?* Please include why the questions chosen are relevant to your program when you write the PSSR.* The chosen over-arching questions are to tie the whole review process together, not the sole questions to be addressed by the review. Does the program meet the discipline criteria and/or

national standards? Does the program meet the needs of potential employers? Is the curriculum current and relevant? What do we need to do to meet the needs of tomorrow?

13

QUESTIONS CONTINUED How can we improve retention and graduation rates? How can we meet the needs of a diverse student body? What do faculty need to improve pedagogy and

improve curricular cohesion? Do we need to refocus resources to better meet

student needs? Do we need to improve student support systems such

as library, advising, technology, co-curricular activities to improve the academic success of students?

Do we need to refine the program learning outcomes and measures for assessment?

Are online courses designed and taught effectively? Why has there been a decline in the number of

students enrolled?

14

SOURCES OF DATA AS EVIDENCE FOR ANSWERING CHOSEN OVER-ARCHING QUESTIONS AND SUPPORTING OTHER PARTS OF THE PSSR

Supporting data can be qualitative (from interviews or focus group discussions, etc.)

Supporting data can also be quantitative Employer surveys, alumni surveys, adjunct

surveys, student surveys (best done in their last classes before they become alumni and go elsewhere )

Data analysis can compare data collected against external benchmarks or against data from other comparable university academic programs, etc.

15

PROGRAM SELF-STUDY COMMITTEE (PSC)

Full time and adjunct faculty who teach in program Chair: Program Lead Function: Advisory and committee members can draft some

sections of the PSSR Provide input and comments related to self-study plan and self-study Members should be diverse as reflecting the class sections of the

program: full-time, part-time, and adjunct instructors teaching online/on-site; teaching in San Diego/regions/online; teaching classes in various concentrations, etc.

Whether adjuncts get paid to be committee members is to be determined by the Dean’s office

Program lead should ask OIRA to grant access to committee members if committee members need full access to upload written documents into AMS

Formed: October and official line of completion is December 31 Minimum: 3 members Maximum: 5 to 6 members

16

ROLE OF THE SAC Scott McClintock, Co-Chair SAC in COLS:

1. Tracking progress 2. Sharing best practices3. Advising and mentoring when bottlenecks are

encountered by program leads4. Communicating between program leads and

the University Academic Assessment Committee (UAAC), as necessary

5. Reviewing the PSSR in June 2013

17

TIPS FROM A PROGRAM LEAD Jacque, Co-Program Lead BAIS with

Credential (self study completed 2012)

18

WASC rubrics for Program Learning Outcomes and Program Review

Five Year rubrics – Councils approved 2011 Will be identified with each section.

RUBRICS

WASC RUBRIC FOR PROGRAM LEARNING OUTCOMES

WASC RUBRIC FOR PROGRAM REVIEW

21

COMPONENTS OF SELF-STUDY Overarching reflective questions

Develop 1-3 overarching questions to guide the self study

22

SELF STUDY PLAN Complete the self study plan with target

dates for completion and potential areas for review in each section.

Upload into AMS under the plan

23

SECTION 1 – GENERAL INFORMATION(Some of the content here is from the National University Catalog and CurriUNET) How current is the catalog How the program learning outcomes support the mission

of the university, the school, and the department Alignment of current program learning outcomes to

institutional learning outcomes Summarized history of changes to the program,

including rationale for change Description of professional development opportunities

for students such as internships, publication/presentation, and the like

Summary of professional success of graduates: job placement/promotion data if available, notable contributions to the profession by program alumni.

24

ADDITIONAL THOUGHTS General information provides the context and sets

the stage for the review Internal context External context

Might include any specializations or concentrations included

Goals of program (if applicable) (These are not the Program Learning Outcomes or PLOs)

Need for program (This may then be used to update the need in AMS PAR Standing requirements)

26

I. GENERAL PROGRAM INFORMATION No rubric needed

27

SECTION 2 – RELEVANCY OF THE PROGRAM

How well do program outcomes prepare graduates with the skills and knowledge expected by potential employers?

How well do program outcomes prepare graduates for professional exams?

How well do the program learning outcomes reflect the contemporary standards of the profession?

How relevant and aligned are the course learning outcomes to the program learning outcomes?

The library may be able to provide data for this section.

Interviewing employers and doing an alumni survey can be a source of such information.

30

II. RELEVANCYCriteria for Relevancy in Programs

Initial Emerging

Developed Highly Developed

Marketplace DataIndustry TrendsEmployabilityProfessional StandardsWorkplace Expectations

Includes relevant data

Relevant data with brief analysis

Relevant data represents key issues and trends with developed analysis

Comprehensive data set represents key issues and trends with thorough analysis and application of data

31

SECTION 3 – CURRENCY OF THE PROGRAM Do the program learning outcomes reflect

current practice? Does the program make use of current

teaching strategies, curriculum structure, technology, delivery modalities, and the like?

Are the teaching materials provided to instructors kept current ?

Is each of the courses in the program maintained and updated regularly by one or more faculty who teach the course?

32

ADDITIONAL THOUGHTS In either relevancy or currency section:

How does the curriculum compare with curricula from other comparable institutions (the library may be able to supply the information for this question)

Is the curriculum map reflective of the progression throughout the program

Does the curriculum map reflect sufficient time for students to acquire the knowledge, skills, and values of the program outcomes

34

III. CURRENCYCriteria for Currency in Programs

Initial Emerging

Developed

Highly Developed

Does the program reflect a systematic review of the current teaching strategies, curriculum structure, and delivery modalities?



Relevant data represents currentissues and trends with developed analysis

Comprehensive data set represents currentissues and trends with thorough analysis and application of data

35

SECTION 4 – FACULTY QUALIFICATIONS Are academic and professional qualifications of instructors specified

for each course in the program ? Do the instructors who teach the courses in the program have the

appropriate academic and professional qualifications ? Have any instructors been successful in obtaining funding for

significant grants ? Do they get adequate orientation and training, and if so, how do we

know that ? Are they engaged in continuous improvement of their teaching, for

example by participation in best practice seminars and other sources? Are they engaged in scholarly activities that enhance their knowledge

of the subject matter they teach and/or their teaching strategies and skills ?

Do they keep current in their profession, and if so, how do we know that ?

Are their course outlines complete and appropriate for the course? How effective are they as teachers? OIRA can supply summarized

/aggregated student evaluation information here

36

ADDITIONAL THOUGHTS How do faculty backgrounds, expertise, research,

and other professional work contribute to the quality of the program

Proportion of faculty with terminal degrees Distribution across ranks (data can be from OIRA) Faculty service Diversity of faculty (data can be from OIRA) Awards and recognition of faculty Record of professional practice Institutions from which faculty earned degrees Faculty specialties and how they align with program If there were class sections in the regions, work with

regional leads to analyze data by region as well.

42

IV. QUALITY OF FACULTYEvidence for Quality of Faculty

Initial Emerging

Developed

Highly Developed

Does the Self-Study reflect a systematic collection of data regarding appropriateness of qualifications for the faculty?



Relevant datawith analysis and application of data specific to program

Comprehensive data with thorough analysis and application of data specific to program

43

SECTION 5 – PREPAREDNESS AND ACADEMIC SUPPORT OF STUDENTS Are there program-specific admission

requirements that must be met, and if so how do we know that they are being met ?

Is there a means of enforcing prerequisites? If so, how do we know it is effective ?

Is there program-specific academic support for students in this program, and if so, what does it consist of? (example: organized faculty advisement)

44

ADDITIONAL THOUGHTS Academic and career advising programs and

resources Tutoring. Supplemental instruction Basic skill remediation Support for connecting general learning requirements

to discipline requirements Orientation and transition programs Support for engagement in campus community Support for non-cognitive variables of success, e.g.,

psychological support for students or other flexibility for students who are parents with a lot of dependent children, etc.

Financial support, such as grants and scholarships

46

V. PREPAREDNESS ACADEMIC SUCCESS(INTEGRATION OF INSTITUTIONAL SUPPORT AND SERVICES)

Criteria for Academic Success

Not Met Met

Does the program integrate institutional support into their entire program and into specific courses?

Does not include relevant data

Relevant data is included and effectivelyintegrated

47

SECTION 6 – STUDENT ACHIEVEMENT How do we know the degree to which

students are achieving the learning outcomes of the program ?

48

ADDITIONAL THOUGHTS Annual results of direct and indirect assessment measures via

reviewing and summarizing past Program Annual Reviews (PARs) in these five years

Ongoing efforts to close the loop by responding to assessment data and have the identified loops been closed?

Does curriculum offer sufficient breadth and depth of learning for the degree

Are the courses well sequenced and reliably available in sequence

How well does the curriculum align with the program outcomes How well does the curriculum align with discipline standards, if

appropriate How should PAR be done in the future? Here, this component of the PSSR emphasizes PLOs and

assessment results

50

VI. STUDENT ACHIEVEMENTSUMMARY AND ANALYSIS OF PLO’S Criteria for Summary and Analysis of Student Achievement

Initial Emerging Developed Highly Developed

Does the review provide the comprehensive summary of the PAR findings from the previous five years?Does the review build on the results of student achievement towards the future development of the program?

Summary of the PAR findings from the previous five years need more description or are absent

Projections for the program are included but need more description

Summary of the PAR findings from the previous five years are included

Projections for the program are developed

Summary of the PAR findings from the previous five years are included and synthesized

Projections for the program are well-developed

Comprehensive summary of the PAR findings from the previous four years

Projections for the multi-year plan are highly developed

51

SECTION 7 – ACADEMIC SUCCESS OF STUDENTS Retention rates onsite vs. online Graduation rates onsite vs. online Pass rate on external licensure or

certification exams, if appropriate

52

ADDITIONAL THOUGHTS Placement in graduate schools Job placements Graduating students satisfaction survey results Employer critiques of student performance Student/alumni achievement Rating of program within discipline Number of admissions and number of

graduations Are the quantitative data statistically

significant?

54

VII. ACADEMIC SUCCESS OF STUDENTSAnalysis of data(Data provided by OIRA)

Initial Emerging


Retention rates onsite and onlineGraduation rates onsite versus online

Included relevant data

Relevant data included but not thoroughly analyzed

Relevant data included and analyzed

Comprehensiveanalysis of data reflects future goals for program retention and graduation

55

SECTION 8 – PROGRAM VITALITY- LIFE CYCLE ANALYSIS New admissions 5-year Trendline SICs: 5-year Trendline Life cycle graph based on OIRA life cycle

criteria: 5-year trendline Average class size – onsite all regions vs.

online Student to fulltime faculty ratio, and/or

proportion of students (SICs) taught by fulltime faculty vs. those taught by part-time faculty

Diversity of faculty proportionate to diversity of students (WASC)

56

ADDITIONAL THOUGHTS Benchmark data as appropriate

School University

In these five years, were there remediation programs? If yes, why did the program try to retain those students? Howe well did these programs work?

58

VIII. PROGRAM VITALITY AND LIFE CYCLEAnalysis ofdata(Data provided by OIRA)

Initial Emerging


Five-Year Trend Line reflects new admissions, SIP, and Life Cycle Data of the program


Relevant data included but not thoroughly analyzed

Relevant data included and analyzed

Comprehensive analysis of data reflects future goals for program

59

SECTION 9 – ADEQUACY OF RESOURCES Are there adequate resources at each onsite

location and for online delivery, for example: library resources, lab facilities, hardware and software, and the like?

60

ADDITIONAL THOUGHTS Faculty

Student/faculty ratio Faculty workload Mentoring processes/program Sufficient time for course/program development

Student support Academic and career advising programs and resources Basic skill remediation Results of student services survey

Information and technology Library print and electronic holdings Information literacy outcomes for graduates Technology resources available to support pedagogy and

research Technology resources to support student needs

61

ADDITIONAL THOUGHTS Facilities

Classroom space Instructional laboratories Research laboratories Student study space Access to classrooms suited for instructional

technology Access to classrooms designed for alternative

learning styles/universal design Appropriate and current software

Staff Clerical and technical staff supporting

program/department operations Financial

64

IX. ADEQUACY OF RESOURCESAnalysis of the adequacy of resources(Onsite and Online)

Initial Emerging


Does the SS provide an analysis of the adequacy of resources and any needed improvement/upgrades for submission?



Relevant data with analysis and application of data specific to program

Comprehensive data with thorough analysis and application of data specific to program

65

SECTION 10 – OTHER INFORMATION NEEDED FOR THIS SELF-STUDY Add any additional information you think

should be included but has not been in other sections.

This section is optional and is to be used as appropriate.

66

SECTION 11 – SUMMARY AND RECOMMENDATIONS Summary of current program relevancy,

quality, and vitality Recommendations for future

additions/changes, including cost/benefit analysis if applicable

Additional monetary resources needed, if any, and priority – which ones need to get funded first

Recommendations for the Memorandum of Agreement (MOA). Consider, for all regions, planning considerations such as program continuation, growth, consolidation, contraction, needed changes in the program, etc.

70

XI. SUMMARY AND RECOMMENDATIONS

Evidence for Quality of Self-Study

Initial Emerging Developed Highly Developed

Doesthe summary of the Self-Study provide evidence for the recommendations to improve the program?

Data is summarized but does not relate to recommendations.

Data is summarized with adequate detail for a few substantive recommendations.

Data is analyzed and applied to specific recommendations for the improvement of the program.

Comprehensive summary and thorough analysis resulting in highly developed recommendations for overall program improvement.

71

SECTION 12 – EXTERNAL REVIEWER(S) REPORT Identify external reviewer(s) and execute

contract External Reviewers report

72

EXTERNAL REVIEW

Conducted once the self study has been completed Contract developed through Office of Dean

(expectations, stipend, travel, etc) About 30 days prior to self study visit reviewers are

provided access to AMS self study and PAR reviews and may be sent additional material

Review visit typically lasts 1- 2 days Meets with PSC, faculty, administrators, department

chair, advisors, students. Participates in an exit interview to share preliminary

findings Provides written report to program lead within several

weeks of visit

73

PROCESS FOR CONTRACT FOR EXTERNAL REVIEWER

All lead faculty should be utilizing the same contracts process available through academic services to create a Contractor Services Contract for the external reviewer they wish to use. A brief description of the process is below.

Complete and submit a Contract Preparation Request Form (CPRF) available through the Provost’s SharePoint site (SharePoint Home > Academics > Office of the Provost > Contract Preparation Request Form). Each dean’s assistant should be able to help them with this. Type of Contract - should be “Contractor: Contractor

Services” Please include contact information for the external reviewer

(contractor), effective and term dates for the contract, detailed description of services, due dates/measurable, and total Payment for Services. Choose anything for Method of Delivery (this field is not applicable for this type of contract and will not affect the creation of the contract).

74

PROCESS FOR CONTRACT FOR EXTERNAL REVIEWER

The Dean will review their submission. If approved, a contract will be created using the information submitted in the CPRF.

The contract will be approved by the Provost and the Executive VP, Business and Regional Operations.

The contract will be sent to the contractor for their signature, and must be returned to NU.

This process takes time. Please plan for enough time for contract approval.

Each external reviewer gets paid a lump sum for everything and does not get additional money for traveling and meals during campus visit, etc.

As for SETM, usually one academic reviewer and one non-academic (a major employer) reviewer are invited as external reviewers for a five-year review.

75

SAMPLE CONTRACT 1 [Contractor] will be the External Reviewer for the Five Year

Review of the [program]. Her services will include: Electronic review of the self study report to be sent to her no later

than [date] Preparation of questions and participation in interview schedule for

follow-up while in San Diego [dates]--Visit to San Diego for interviews, meeting with self study

team and debrief with department chair and program lead. Draft report due no later than [date] Final report due [date]

Expected Deliverables and Due Dates: [date]--phone

conference with [Program Lead] to review report format., arrange travel [date]--trip to San Diego [date]--draft report due [date]--phone conference with [Program Lead] to review draft [date]--final report due

76

SAMPLE CONTRACT 2 The consultant will complete an in-depth review of the [Program]

as part of National University’s Five-Year Program Review. The review will include data collection (both archival and new data); data analysis; and a report summarizing the data, identifying strengths and areas of improvements, and making recommendations, including but not limited to, on ways in which the learning experiences of candidates can be enhanced and enrollment can be increased.

The consultant will examine the program in-depth within the context of accreditation standards and best practices in the field as they relate to [specific] standards while staying in compliance with [other specific] standards.

A 2 day site visit will include the [location] and [other location] sites. Data collection will include but is not limited to (a) interviews of stakeholders (including faculty and students), (b) review of program assessment data, and (c) review of data such as demographics, student support services, student grade, program completion, and course evaluations.

77

CONTRACT 2 CONTINUED By [date], the consultant’s review will be summarized in a 10-15 page final

typed electronic report which addresses all relevant sections of the self-study. By [date], finalize meeting dates with the Program Lead a) reviewing all sections

of the Five-Year Review in the AMS system including student portfolio work and Program Learning Outcomes; and b) conducting two onsite visitation dates during the month of [month].

Artifact: Submit a draft of a one to two page typed reflective essay including preliminary recommendations for improvement by [date].

Artifact: Submit a draft of a one to two page reflective essay discussing alignment between Program Learning Outcomes and students' work by [date].

Conduct two onsite visitations, one at [here] and one at [there] campuses by [date].

Artifact: Submit a 10-15 page final typed report that includes findings and recommendations to the Program Lead and Department Chair by [date].

Note: Reflective essays could be revised as needed and included in the final typed report.

Payment of [amount] includes services (i.e. preliminary review meeting, program evaluation, onsite visits, two draft reflective essays, a 10-15 page final typed report) and all expenses, accommodations (i.e. transportation, hotel, and meals).

78

AMS Walk through of AMS for five year review. Remember to add content you will check out

the section and then once finished check the section back in.

Select you want to add “text & image” Reviewers find it helpful if you provide

information in the text section You can also select “attachments” or “links” Upload supporting evidence as attachments.

79

MEMORANDUM OF AGREEMENT Developed after the Council reviews and approves Obtain template for MOA from MOA section. Program lead develops an initial draft Meeting to discuss with Provost, Dean, Council

representative, and Lead Revisions to MOA. Signed by Provost, School Dean, Department

Chair, Program Lead, Council representative Program lead needs to follow-up (Closing the Loop)

in subsequent years PAR reports

80

MOA

81

SELF-STUDY PLAN: SUMMARY Lead prepares draft plan to be reviewed by PSC Modify the self study plan template to meet

needs of the program Include the overarching reflective questions Identify possible evidence Indicate who will place in AMS Set a target start date Set a target completion date You can track by completing the actual

completion date as you complete the self-study Identify the potential external reviewers and

rationale and qualifications

83

REVIEW PROCESS June 1 - 30 – School Assessment Committee

reviews July 1-15 – Department Chair reviews July 15-August 1 – Dean reviews August – Council subcommittees review September – Finalize review at Graduate and

Undergraduate Council meetings Council reviews submitted to Provost by

September 30 Memorandum of Agreement (MOA) completed by

December 15.

84

RESOURCES PROVIDED BY THE OFFICE OF INSTITUTIONAL RESEARCH AND ASSESSMENT (OIRA) AMS Data

Standard data for all programs uploaded to AMS by November 30 in 2012

Specialized data requests program specific (Be sure to allow sufficient time)

Can assist with alumni or other surveys that you may want. (Be sure to start on this early)

85

SUMMARY Develop overarching questions and include in AMS Create a self study committee Complete self study plan and upload into AMS Credentials for external reviewer included in section XII Collect data, analyze and reflect and make

recommendations for each section and include in AMS. Finalize self study External Reviewer site visit and report uploaded into

AMS section XII Finalize summary and recommendations including

external reviewer recommendations. (Section XI) After Council review completed develop draft of MOA.

The End

five year self study training

Documents

program reviewthe program

hisher program

year self study trainingdr

data analysis

assessment oirasummary

assessment fellowdr

self studydevelop

data collection