program annual assessment report - morningside college · the externship/internship/research...
TRANSCRIPT
2019 AAFS Programmatic Assessment - 1
Program Annual Assessment Report
Program: Applied Agricultural and Food Studies (BS and BA) Date: July 1, 2019
Please use a separate copy of the table and three prompts below for each outcome assessed this year.
Outcome: Program-Specific Outcome # 1
Students will demonstrate an understanding of
Midwestern agricultural systems.
Reviewers:
Dr. Thomas H. Paulsen
Dr. Annie Kinwa-Muzinga
Mr. Richard Crow
Mr. Dan Witten
Ms. Melissa Nelson
Ms. Dee McKenna
Number of Artifacts
Collected:
39 Written Papers
13 Oral Presentations
Number of Artifacts
Reviewed:
39 Written Papers
13 Oral Presentations
How were the artifacts reviewed selected:
A census of graduating senior Externship Supervisor
Written Evaluations were reviewed for baseline data in the
department’s second year of Programmatic Outcome
Assessment. The department garnered a 100% response
rate from graduating senior program completers.
Describe how the artifacts were used to assess student learning. (If a rubric was used please attach it to the
report.):
Introduction
Established in 2014 through a generous donation from Regina Roth, Siouxland philanthropist and cofounder/
executive at Beef Products, Inc., now empirical Foods Inc., the Applied Agricultural and Food Studies (AAFS)
curriculum was designed by an interdisciplinary cross-section of Morningside College faculty under the advisement
of agricultural industry leaders to meet the needs of the modern agricultural industry. A unique feature of the
curriculum is the full-time, academically integrated, working semester (the Externship) which typically occurs in the
fall of the senior year through collaboration with area, regional (and eventually national and international)
agricultural businesses for students desiring the BA and BS degree in AAFS.
As designed and approved in the program assessment plan, a culminating capstone paper is completed in AGRI 480,
the program’s capstone course, where students address the following Institutional Assessment requirements: Written
Communication/Writing Proficiency, Oral Communication, Integration/Synthesis in the program, Critical Thinking
in the program. Additionally the capstone paper addresses the following Essential Skills in the program: Writing in
the discipline Literature Review (Type 1) and Descriptive Writing (Type 2), Integration/Synthesis in the discipline,
and Critical Thinking in the program. Finally, the capstone paper is used as one of the artifacts for two AAFS
Program-specific Outcomes: 1) demonstrating an understanding of Midwestern agricultural systems, and 2)
formulating and critiquing business/management plans for real world situations. Externship Coordinator written
evaluations serve as the artifact for student modeling of “work habits and knowledge for common agriculture work
conditions”. Ethical reasoning in the program has previously been assessed through an artifact produced in PHIL
308: Applied Agricultural Ethics. The purpose of this report is to provide the baseline AAFS Program Annual
Assessment Report for Outcome #1: Students will demonstrate an understanding of Midwestern agricultural
systems.
2019 AAFS Programmatic Assessment - 2
Procedures
Artifacts
The AAFS Outcomes Assessment Plan calls for the Senior Capstone Paper and the recorded Senior Capstone
Presentation developed, submitted, and graded as part of AGRI 480: Capstone in Applied Agricultural and Food
Studies course to be used as the artifacts for measuring Programmatic Outcome #1.
The Senior Capstone Paper and recorded Senior Capstone Presentation artifacts are also used for the following
Institutional Outcomes (Essential Skills):
1. Writing in the Discipline (Type I) Literature Review
2. Writing in the Discipline (Type II) Descriptive
3. Integration/Synthesis in the Discipline
4. Critical Thinking in the Program
To measure the aforementioned Program-Specific Outcomes, the AAFS Outcomes Assessment Plan prescribes that
the Senior Capstone Paper contain several components listed as follows:
1. Review of the Externship Experience
2. Overview of a Research/Investigative Project implemented during the Externship Experience
3. Description of Midwestern agricultural systems
4. Formulated, analyzed, and critiqued Business/Management plan
5. Business Management Decision analysis and critique considering multiple factors and perceptions
Based upon recommendations from previous years programmatic assessment reports, the Capstone paper has been
divided into three specific assignments: 1) Externship/Internship/Research Reflection paper at the end of the
experiential learning activity, 2) Externship/Internship/Research Challenges paper developed as the first capstone
writing assignment, 3) Capstone Research paper, and 4) Capstone research presentation.
Each faculty and staff member assessed four of the thirteen senior student Capstone Artifacts (N = 52) using the
newly created rubric (attached) as none had been previously developed for this AAFS program-specific outcome.
Seven criterion were rated on a five-point scale to assess student attainment of the program-specific outcome. The
seven criteria included:
From the “Experiential Learning Component” Artifacts (Reflection Paper and Challenges Paper)
1. Background evidence of the experiential learning component of the program (Externship, Internship, or
Research)
2. Attainment of the learning objectives specified in the experiential learning component of the program
3. Challenges encountered through the experiential learning component of the program
From the “Capstone Research Component” Artifacts (Research Paper and Presentation)
1. Relevance of the research to the agricultural industry
2. Literature review
3. Research conclusions
4. Use of discipline-specific vocabulary
The five-point scale rubric used the following ordinal scale:
0 = Missing
1 = Ineffective
2 = Slightly Effective
3 = Somewhat Effective (Proficient level)
4 = Moderately Effective
5 = Effective (Capstone level)
2019 AAFS Programmatic Assessment - 3
Reliability
Since this is the first year of data collection to assess program completers (graduating Seniors) attainment of
Outcome #1, a rubric was developed requiring rater norming (Jonsson & Svingby, 2007). To determine inter-rater
reliability (IRR) of the newly developed program-specific outcome rubric, faculty initially discussed the newly
developed rubric, asked specific questions, and reviewed examples. AAFS faculty then assessed the Senior
Capstone artifacts submitted by the same student to determine Inter-rater Reliability (IRR) of responses on
their scaled responses individually. Responses that were not whole numbers were rounded up prior to determining
reliability. Percent agreement was then calculated between the raters. Ratings that varied between the following
dyads were considered different relative to reliability: 0-1, missing and ineffective; 2-3, slightly and somewhat
effective; and 4-5, moderately effective and capstone.
Percent agreement on the sample student was 85.71% and considered Acceptable (Norcini, 1999). Following initial
ratings using the newly developed program-specific outcome rubric, faculty further discussed each criterion, areas of
concern, etc. relative to the student’s attainment of the programmatic outcome. Once norming activities were
completed, faculty were placed into teams of two to review the remaining 12 students’ capstone artifacts. Faculty
teams were assigned four students each, rating each student on each criterion individually and then discussed
differences with their faculty peer. If faculty were in disagreement across dyads, a third rater was used. Table 1
displays results from the norming activity.
Conclusions of the review regarding student learning in the outcome with rationale supported by evidence
from the artifacts:
Findings
Individual student outcome attainment of the standard was determined by calculating the Grand Mean (GM) of the
faculty ratings for the seven aforementioned rubric criteria in the reviewed artifacts (reflection summary report from
the externship/internship/research project, experience challenges reflection paper, capstone research paper, and
capstone research paper/presentation) on a 5-point scale as follows (0 = Missing, 1 = Ineffective, 2 = Slightly
Effective, 3 = Somewhat Effective, 4 = Moderately Effective, 5 = Effective). Students who achieved a score GM ≥
3.00 on the 5.00 scale were considered Proficient. All thirteen students (N=100%) were considered Proficient. Table
2 displays the outcome attainment of AAFS seniors enrolled in the capstone agriculture course (AGRI 480) by
criteria and overall outcome attainment.
2019 AAFS Programmatic Assessment - 4
Table 2
Student Attainment of AAFS Programmatic Outcome #3: Students will demonstrate an understanding of Midwestern
agricultural systems
AAFS Departmental Outcome #1 Attainment (Met/Not Met) Reviewers:
Artifacts Reviewed:
Student Outcome #3 Criteria Faculty 1 Faculty 2 Mean Std. Dev. Grand Mean Grand Std. Dev. Outcome Met?
1 Background for Experience 3 3 3.000 0.000
1 Learning Objectives of Experience Attained 3 3 3.000 0.000
1 Challenges from Experience Identified 2.5 2.5 2.500 0.000
1 Research Topic Relevance 4 3.5 3.750 0.354
1 Literature Review 4 3.5 3.750 0.354
1 Research Conclusions 3.5 3 3.250 0.354
1 Discipline-specific Vocabulary 3.5 3.5 3.500 0.000 3.250 0.470 Yes
2 Background for Experience 4 3.5 3.750 0.354
2 Learning Objectives of Experience Attained 3.5 4 3.750 0.354
2 Challenges from Experience Identified 4 3 3.500 0.707
2 Research Topic Relevance 3.5 4 3.750 0.354
2 Literature Review 4 4 4.000 0.000
2 Research Conclusions 4 4 4.000 0.000
2 Discipline-specific Vocabulary 3 3 3.000 0.000 3.679 0.421 Yes
3 Background for Experience 4 4 4.000 0.000
3 Learning Objectives of Experience Attained 2 2 2.000 0.000
3 Challenges from Experience Identified 4 4 4.000 0.000
3 Research Topic Relevance 5 5 5.000 0.000
3 Literature Review 5 4.5 4.750 0.354
3 Research Conclusions 4 4 4.000 0.000
3 Discipline-specific Vocabulary 5 5 5.000 0.000 4.107 1.003 Yes
4 Background for Experience 2.5 2.5 2.500 0.000
4 Learning Objectives of Experience Attained 5 5 5.000 0.000
4 Challenges from Experience Identified 5 5 5.000 0.000
4 Research Topic Relevance 4 5 4.500 0.707
4 Literature Review 5 5 5.000 0.000
4 Research Conclusions 5 5 5.000 0.000
4 Discipline-specific Vocabulary 5 5 5.000 0.000 4.571 0.917 Yes
5 Background for Experience 3 4 3.500 0.707
5 Learning Objectives of Experience Attained 2 2 2.000 0.000
5 Challenges from Experience Identified 3 4 3.500 0.707
5 Research Topic Relevance 4 5 4.500 0.707
5 Literature Review 4 5 4.500 0.707
5 Research Conclusions 3 4 3.500 0.707
5 Discipline-specific Vocabulary 3 5 4.000 1.414 3.643 1.008 Yes
6 Background for Experience 5 5 5.000 0.000
6 Learning Objectives of Experience Attained 0 1 0.500 0.707
6 Challenges from Experience Identified 5 5 5.000 0.000
6 Research Topic Relevance 5 5 5.000 0.000
6 Literature Review 4 5 4.500 0.707
6 Research Conclusions 5 5 5.000 0.000
6 Discipline-specific Vocabulary 4 5 4.500 0.707 4.214 1.626 Yes
7 Background for Experience 2 2 2.000 0.000
7 Learning Objectives of Experience Attained 3 3 3.000 0.000
7 Challenges from Experience Identified 4 5 4.500 0.707
7 Research Topic Relevance 5 5 5.000 0.000
7 Literature Review 4 5 4.500 0.707
7 Research Conclusions 5 5 5.000 0.000
7 Discipline-specific Vocabulary 4 5 4.500 0.707 4.071 1.141 Yes
Fro
m t
he
Cap
sto
ne
Res
earc
h
Art
ifac
ts
Fro
m t
he
Cap
sto
ne
Res
earc
h
Art
ifac
ts
Fro
m t
he
Exp
erei
nce
Art
ifac
ts
Fro
m t
he
Cap
sto
ne
Res
earc
h
Art
ifac
ts
Fro
m t
he
Exp
erei
nce
Art
ifac
ts
Fro
m t
he
Cap
sto
ne
Res
earc
h
Art
ifac
ts
Fro
m t
he
Exp
erei
nce
Art
ifac
ts
Fro
m t
he
Cap
sto
ne
Res
earc
h
Art
ifac
ts
Fro
m t
he
Exp
erei
nce
Art
ifac
ts
Fro
m t
he
Cap
sto
ne
Res
earc
h
Art
ifac
ts
Fro
m t
he
Exp
erei
nce
Art
ifac
ts
Fro
m t
he
Cap
sto
ne
Res
earc
h
Art
ifac
ts
Fro
m t
he
Exp
erei
nce
Art
ifac
ts
Student will Demonstrate an Understanding of Midwestern Agricultural Systems
Experience (Research/Internship/Externship) Reflection Paper, Challenges Paper, Capstone Research Paper, Capstone Research Presentation
Ratings0 = Missing, 1 =
Ineffective, 2 =
Slightly Effective, 3
= Somewhat
Effective, 4 =
Moderately
Effective, 5 =
Capstone Level
≥ 3.00 = Proficient
Fro
m t
he
Exp
erei
nce
Art
ifac
ts
2019 AAFS Programmatic Assessment - 5
_______________________________________________________________________________________
Note: Student 13 served as the norming activity for the faculty.
8 Background for Experience 2 1 1.500 0.707
8 Learning Objectives of Experience Attained 4 3 3.500 0.707
8 Challenges from Experience Identified 4 5 4.500 0.707
8 Research Topic Relevance 5 5 5.000 0.000
8 Literature Review 5 5 5.000 0.000
8 Research Conclusions 5 5 5.000 0.000
8 Discipline-specific Vocabulary 4 5 4.500 0.707 4.143 1.292 Yes
9 Background for Experience 5 5 5.000 0.000
9 Learning Objectives of Experience Attained 5 4 4.500 0.707
9 Challenges from Experience Identified 5 5 5.000 0.000
9 Research Topic Relevance 5 5 5.000 0.000
9 Literature Review 4 4 4.000 0.000
9 Research Conclusions 4 4 4.000 0.000
9 Discipline-specific Vocabulary 5 5 5.000 0.000 4.643 0.497 Yes
10 Background for Experience 3 3 3.000 0.000
10 Learning Objectives of Experience Attained 3.5 4 3.750 0.354
10 Challenges from Experience Identified 4 4 4.000 0.000
10 Research Topic Relevance 3.5 4 3.750 0.354
10 Literature Review 4 4 4.000 0.000
10 Research Conclusions 3 3 3.000 0.000
10 Discipline-specific Vocabulary 3 3 3.000 0.000 3.500 0.480 Yes
11 Background for Experience 4 5 4.500 0.707
11 Learning Objectives of Experience Attained 5 5 5.000 0.000
11 Challenges from Experience Identified 4 4 4.000 0.000
11 Research Topic Relevance 5 5 5.000 0.000
11 Literature Review 5 4 4.500 0.707
11 Research Conclusions 5 5 5.000 0.000
11 Discipline-specific Vocabulary 5 5 5.000 0.000 4.714 0.469 Yes
12 Background for Experience 3 3 3.000 0.000
12 Learning Objectives of Experience Attained 3 3 3.000 0.000
12 Challenges from Experience Identified 3 3 3.000 0.000
12 Research Topic Relevance 5 3 4.000 1.414
12 Literature Review 4 3 3.500 0.707
12 Research Conclusions 3 3 3.000 0.000
12 Discipline-specific Vocabulary 3 3 3.000 0.000 3.214 0.579 Yes
Fro
m t
he
Ca
psto
ne
Re
se
arc
h
Arti
fac
ts
Fro
m t
he
Ex
pe
re
inc
e
Arti
fac
ts
Fro
m t
he
Ca
psto
ne
Re
se
arc
h
Arti
fac
ts
Fro
m t
he
Ex
pe
re
inc
e
Arti
fac
ts
Fro
m t
he
Ca
psto
ne
Re
se
arc
h
Arti
fac
ts
Fro
m t
he
Ex
pe
re
inc
e
Arti
fac
ts
Fro
m t
he
Ca
psto
ne
Re
se
arc
h
Arti
fac
ts
Fro
m t
he
Ex
pe
re
inc
e
Arti
fac
ts
Fro
m t
he
Ca
psto
ne
Re
se
arc
h
Arti
fac
ts
Fro
m t
he
Ex
pe
re
inc
e
Arti
fac
ts
2019 AAFS Programmatic Assessment - 6
Conclusions/Discussion
Findings of this assessment indicate that program completers of the AAFS curriculum in the 2018-2019
academic year (N=13) were able to appropriately demonstrate an Understanding of Midwestern Agricultural
Systems. Previous outcome assessments in the first three years of the program provided significant recommendations
to assist AAFS faculty in in improving the scope and sequence of the required coursework in the program—
especially the connection and flow between the Workforce Preparation course (fall of the junior year); the
experiential learning component (Externship/Internship/and Research) and the formal supervision and reflection
process integrated within it; and the culminating Capstone course and its required research project.
The following specific areas of improvement identified from the assessment of Programmatic Outcome #2 in the
2017-2018 academic year have been implemented and helped students internalize the outcome addressed in this
report:
1. Additional structure within AGRI 380: Preparation for Externship should be established to include a formal,
written externship proposal. The proposal should include a specific career area of interest, background
research on several companies of interest, which provide that career area, and a list of several projects that
could serve as the foundation for developing a business management plan, analysis, and decision relevant to
the student extern’s future career. – Fall 2018
2. Specific activities to develop awareness of and skills in the measured outcomes related to the outcome
assessment criteria for the externship should be implemented in AGRI 380. Activities could include a
formal employer career panel, mock interviews, and other appropriate activities. – Fall 2018
3. A formal protocol for acquiring, gaining approval, and engaging in an externship experience (AGRI
480) should be developed and include a collaborative process for developing student learning outcomes
relevant to the employer’s needs and the student extern’s long-term development. – Fall 2018
4. A formal protocol for student reflection in the externship (weekly or bi-weekly) should be developed and
implemented which covers at a minimum the following items: - Summer 2018
Overview of activities assigned and completed
Major challenges or obstacles encountered and how they were addressed
New learning opportunities to be implemented over the next one-two weeks
5. A formal protocol for faculty supervision of the externship should be developed and implemented which
should include: - Summer 2018
Adequate resources to complete supervisor visits
A process for scheduling supervisor visits to include timing in relation to bi-weekly reflection
completion
A checklist of items to discuss with the employer and student during each visit including discussing,
adjusting, and assessing progress on the collaboratively developed learning objectives
6. A formal protocol for a mid-term employer evaluation of the student extern to allow for adjustments or
corrections in the student extern’s workplace behavior. – Summer 2018
7. A formalized one- to two-day professional development session should be held on campus at the end of the
summer (beginning of fall) and at the end of the fall semesters to reflect, debrief, receive feedback, and
finalize the formal written externship reflection paper and present the formal oral presentation. This will set
the stage for the spring capstone course and the development of a follow-up research paper to garner further
information needed for continued career development in the student extern’s career field of choice. – Fall
2018
2019 AAFS Programmatic Assessment - 7
8. Formally split the Externship Reflection Paper and Presentation from the Senior Capstone Paper and
Presentation assignment currently used in AGRI 480: Capstone in Agriculture.
9. Utilize the Externship Reflection Paper and Presentation as the culminating reflection assignment in AGRI
460: Externship in Agriculture to measure Outcome #3. – Fall 2018
Suggestions (if any) to improve student learning relative to this outcome. If no suggestions explain why not:
Although positive results in student attainment of Programmatic Outcome #3: Students will demonstrate an
understanding of Midwestern agricultural systems, have been noted in this report, continuous improvement of the
program through the following recommendations can occur.
A brief overview of these next steps is listed below:
.
1. Develop experiential learning activities in the freshman and sophomore year which include job shadowing
and internship experiences that are in clear alignment with Preparation for Externship, Externship, and the
Capstone course. (Fall 2019)
2. Develop formative activities that align and build to an assessment-ready Senior Capstone Project in the
Capstone course. (Fall 2019 – ongoing)
3. Develop an assessment plan to which provides formative feedback for instructors developing and improving
the foundational courses in the department. (Fall 2019)
4. Continue to improve the experiential learning supervision of learning and reflection process through the
improvement of bi-weekly reflection writing prompts and an enhanced supervision protocol that
intentionally utilizes faculty and industry mentors. (Summer 2019 – ongoing)
5. Work with students in the co-curricular clubs to improve programming (Leadership seminars, Career
seminars, Service-learning opportunities) that supports overall AAFS programmatic outcomes. (Fall 2019)
These specific recommendations, when implemented, will assist faculty in the AAFS department in overall program
improvement as it seeks to meet the mission of Morningside College.
Works Cited
Bartko, J. J. (1976). On various intraclass correlation reliability coefficients. Psychological Bulletin, 83(5), 762-765.
doi:http://dx.doi.org/10.1037/0033-2909.83.5.762
Fleiss, J. L., & Cohen, J. (1973). The equivalence of weighted kappa and the intraclass correlation coefficient as
measures of reliability. Educational and psychological measurement, 33(3), 613-619.
Jonsson A., Svingby G. (2007). The use of scoring rubrics: reliability, validity and educational consequences.
Educational Research Review, 2(1) 130–144. doi: 10.1016/j.edurev.2007.05.002
Norcini, J. J. (1999). Standards and reliability in evaluation: when rules of thumb don't apply. Academic medicine:
Journal of the Association of American Medical Colleges, 74(10), 1088-1090.
Appendix
AAFS Program Outcome #1 Assessment Rubric.....................................................................................Page 8
2019 AAFS Programmatic Assessment - 8
Appendix 1: Programmatic Outcome #1 Assessment Rubric