2017 ERA Summer Assessment Group Report Group Members Valerie Hennings (chair) Ross Fuglsang Patrick McKinlay Dean Stevens Tangela Sylvester Kari Varner Purpose and Outcome Assessed CPAC charged a multidisciplinary group of faculty to conduct assessment of the essential skills outcome, Ethical Reasoning and Application (ERA). The 2015 curriculum revision was amended to include this specific outcome:

● Outcome 4: Apply principles of ethical reasoning to concrete situations. The group assessed artifacts submitted for institutional assessment from ERA courses as well as from several programs. The results of this assessment will serve as a baseline for future assessment of ethical reasoning and application. This assessment built upon previous institutional assessment processes and practices. Process Summer work group members met for three days, Monday, July 10, through Wednesday, July 12. Associate Dean Alden Stout joined the group Monday morning, providing background on ethical reasoning scholarship and Morningside’s ERA rubric. During the summer of 2015, an ERA assessment faculty workshop group developed the Applied Ethical Reasoning rubric (see Appendix A). Building on the Association of American Colleges and Universities (AAC&U) Applied Ethical Reasoning Rubric, the purpose of the rubric was to distinguish the diverse components of ethical reasoning as well as identify a scale for assessment of student learning. Stout provided examples of how the ERA rubric had been applied to artifacts in the past. He led the group through the process of reading a case study, then assessing student artifacts related to that case. Monday’s afternoon session began with norming exercises and a discussion of the scoring rubric. Based upon research protocol, a rating scale was developed to score ERA artifacts. The scale ranged from 0-5. The group continued to use the Benchmark and Capstone descriptors for scores 1 and 5. In the case where the component was absent, the group included the option for a 0. The group also amended the rubric to include an “Overall Assessment of ERA” for each

1

2017 ERA Assessment

artifact, which is consistent with similar practice for other Essential Skills Outcomes (see Appendix B). Group members were presented 60 artifacts, 19 from upper level courses, and 41 from lower level courses approved for the ERA distribution requirement. Assessment used a random sample stratified proportionally by classes across campus. The table below distinguishes between 100/200 level ERA artifacts and 300/400 level programmatic artifacts. After finalizing the rubric and scoring, members paired up to read and score artifacts, with brief discussions in between and a final submission of scores. The same process was used during morning and afternoon sessions Tuesday, and a concluding session Wednesday morning. Work pairs changed regularly. By the end of the third session each member/pair had read and coded 20 artifacts. The work group then came together, spending the balance of Wednesday morning and afternoon reviewing results, discussing implications and recommendations, and writing its report. Results

Self- Awareness

Ethical Principles*

Issue Recognition*

Application* Differing Perspectives*

Overall Effectiveness*

ERA/Lower Level 1.6 (a) 1.5 (b) 1 (c)

2.4 (a) 2 (b) 2 (c)

2.3 (a) 2 (b) 3 (c)

2.1 (a) 2 (b) 2 (c)

1.9 (a) 2 (b) 2 (c)

2.1 (a) 2 (b) 2 (c)

Non-ERA/Upper Level

1.2 (a) 1 (b) 1 (c)

1.9 (a) 2 (b) 2 (c)

1.6 (a) 2 (b) 2 (c)

1.5 (a) 2 (b) 2 (c)

1.1 (a) 1 (b) 1 (c)

1.5 (a) 2 (b) 2 (c)

All Students 1.5 (a) 1 (b) 1 (c)

2.2 (a) 2 (b) 2 (c)

2.09 (a) 2 (b) 2 (c)

1.9 (a) 2 (b) 2 (c)

1.7 (a) 2 (b) 2 (c)

1.9 (a) 2 (b) 2 (c)

Table 01: Mean (a), median (b), and mode (c) score comparisons between ERA/lower level, Non-ERA/upper level, and All Students for each rubric category. All scores were on a scale of 1-5. ERA N=41, Non-ERA N=19, All Students N=60. There is a significant difference in means between ERA and Non-ERA courses in the categories marked with an asterisk (*): ethical principles (p = .03), issue recognition (p = .003), application (p = .01), differing perspectives (p = .002), and overall effectiveness (p = .007).

2

2017 ERA Assessment

Figure 01: Overall Assessment of Ethical Self Awareness. N=60.

Figure 02: Overall Assessment of Understanding Different Ethical Principles. N=60.

3

2017 ERA Assessment

Figure 03: Overall Assessment of Ethical Issue Recognition. N=60.

Figure 04: Overall Assessment of Application of Ethical Principles. N=60.

4

2017 ERA Assessment

Figure 05: Overall Assessment of Application of Ethical Principles. N=60

Figure 06: Overall Assessment of ERA. N=60.

5

2017 ERA Assessment

Figure 07: Overall Assessment of ERA by Course Type/Level. ERA/Lower Level Course N=41. Non-ERA/Upper Level Course N=19.

6

2017 ERA Assessment

Figure 08: Each category for ERA classes. N=41.

7

2017 ERA Assessment

Figure 09: Each category by Non-ERA classes. N=19.

8

2017 ERA Assessment

Figure 10: Combination graph of all classes and categories. N=60. Discussion The scores for the lower level courses had a mean of 2 (except Ethical Awareness). Since these artifacts were taken from first or second year courses, it seems reasonable not to expect Capstone level scores consistently for the vast majority of students; group members agreed scores for the 100/200 level artifacts would rarely achieve a 4 or 5. That said, as this is the first iteration of institutional assessment, it is not clear whether this score indicates sufficient or insufficient student learning, or if there are other intervening variables. Most significantly, this assessment process may inform pedagogical changes, particularly variation on how assignment prompts are constructed, to better meet the components of the outcome. The assessment team expected to see improvement of aggregate scores from lower to upper level courses; however, this was not confirmed by results. Indeed, in some areas, the scores are lower on some components. The team was uncertain to what extent the prompts for these particular upper level artifacts were crafted with the specific components of the outcome in mind. Programs may determine that the prompts could be adjusted to more closely follow the

9

2017 ERA Assessment

outcome components OR, programs may consider recommending adjustments to the components of the rubric to better permit disciplinary variation. The lowest scores for both the lower and upper level course were in the “ethical self-awareness” category. Members of the work group suggested continued development and emphasis of reflection on personal values/self-awareness. Questions Raised

● Were assignments aligned with the rubric? As noted above, one possible intervening variable is the extent to which assignments aligned coherently with the rubric and its distinct components. Phrasing of prompts could potentially change how students organize and respond. While the group could not access specific course prompts, team members did perceive that students may have been able to answer a component element more fully but were not sufficiently prompted to do so in the assignment. One drawback of this approach might be the dilemma of meeting the assessment prompt components but undermine the disciplinary or skills development intended by the assignment.

● Are we seeing the reinforcement of the skills in lower level ERA courses as students progress through their program of study? Although this small sample of artifacts shows significant differences between upper and lower courses, the answer is unclear. The inclusion of a broader range of courses, including ERA–designated courses as well as programmatic assessment artifacts, would add clarity.

● Many of the current courses fulfilling ERA were originally Ethics and Personal Values (EPV) courses. Are we experiencing transition effects from how courses were organized for meeting the original course’s objectives to a more focused acquisition of the components of the outcome rubric? For instance, some courses in the past measured student learning with regard to reflection on personal values through an assignment separate from the one used for examining and applying specific ethical theories. The new outcome may require additional pedagogical adjustments to capture more fully the distinct dimensions of the ERA outcome as opposed to the former EPV outcome.

Recommendations

● Process of assessment ○ The data set from which assessment artifacts was drawn did not include all ERA

courses offered in the past three years. It also did not include ethical reasoning artifacts from all programs. Institutional assessment would be more representative if a broader and more inclusive data set were available.

10

2017 ERA Assessment

● Ethical reasoning pedagogy ○ The group expected to find higher assessment scores for upper level courses

from programs. Opportunities to reinforce ethical reasoning within programs of study, as well as within some Integrated Experience courses, may enhance student learning performance on ethical reasoning components.

● ERA faculty groups ○ The group recommends that faculty, particularly those who teach ERA classes,

gather to discuss the report and discuss current pedagogical practice in their ERA courses. This activity would be consistent with past EPV (Ethics and Personal Values) group activity to occasionally meet to discuss the outcome and pedagogical practice. This may prove especially helpful to new faculty assigned ERA courses and those transitioning former EPV classes to ERA.

11

2017 ERA Assessment

Appendix A APPLIED ETHICAL REASONING RUBRIC  

 Ethical reasoning is the process of creating well-formed judgments regarding what we ought to do in 

particular situations. Ethical reasoning generally relies on normative principles, that is, principles that specify what is generally right/wrong or good/bad. Such principles are abstract because they apply to a wide variety of circumstances. These contexts often involve personal, professional, and civic dimensions. The skills required for ethical reasoning include: (a) reflective self-awareness regarding one’s own assumptions and values; (b) describing multiple ethical principles; (c) identifying facts relevant to moral decision making;  (d )coming to a conclusion about a particular situation by using an ethical principle(s) and morally relevant facts of that situation (e) identifying the possible areas of disagreement with others regarding one’s own conclusion  Evaluators are encouraged to assign a zero to any work sample or collection of work that does not meet benchmark (cell one) level perform   Capstone 

4 

Milestones 

3 2 

Benchmark 

1 

Ethical Self-Awareness  

Student clearly discusses and evaluates their core assumptions/values and the origins of those assumptions/values. 

Student clearly discusses core assumptions/values and the origins of those assumptions/values, but does not evaluate them. 

Student states both core assumptions/values articulates the origin of those assumptions/values, but the discussion is not clear. 

Student states either their core assumptions/values or articulates the origins of those assumptions/values. 

Understanding Different Ethical Principles/ Concepts 

Student names and substantively explains multiple ethical principles/concepts and critically discusses the full relevance/limitations of each principle. 

Student names multiple ethical principles/concepts and clearly explains them, but does not discuss the possible relevance or limitations of each principle.   

Student can name multiple principles/concepts, but gives an unclear explanation of those principles. 

Student only names an ethical principle/concept, but does not name more than one. 

Ethical Issue Recognition 

Student identifies various facts relevant to moral decision making in a situation, identifies such facts that others may believe to be relevant, and thoroughly explains the difference between the two sets of facts.  

Student identifies various facts relevant to moral decision making in a particular situation and identifies such facts that others may believe to be relevant, but does not explain the difference between the two sets of facts.  

Student identifies various facts relevant to moral decision making in a particular situation, but does not identify such facts that others may believe to be relevant.  

Student only  identifies at least one fact relevant to moral decision making in a particular situation.  

Application of Ethical Principles/ Concepts  

Student makes an evaluation of a particular situation and supports it using moral principles/concepts and facts relevant to moral decision making, and critically discusses the possible limitations of their analysis.  

Student makes an evaluation of a particular situation and supports it using moral principles/concepts and facts relevant to moral decision making, but does not critically discuss the evaluation.  

Student makes an evaluation of a particular situation, but only supports it using moral principles/concepts or facts relevant to moral decision making, but not both.  

Student makes an evaluation of a particular situation without supporting it.  

Evaluation of Different Ethical Perspectives   

Student is able to identify multiple areas of moral disagreement regarding their position and is able to examine all the substantive reasons for the disagreement.  

Student is able to identify multiple areas of possible disagreement about their position, but is unable to explain the reasons for the disagreement.  

Student is able to identify at least one area of possible disagreement about their position.  

Student merely states that disagreement regarding their position is possible.  

12

2017 ERA Assessment

Appendix B

13

2017 ERA Assessment

14

2017 ERA Assessment

15