Liberal Arts and Sciences Outcomes Assessment: Closing the loop or spiraling in the right direction?

Dr. Jenn BergAssistant Professor of MathematicsDr. Christopher Cratsley Interim Director of AssessmentDr. Elizabeth GordonAssistant Professor of Geo/Physical SciencesDr. Joe MoserAssistant Professor of English

Closing the loop or spiraling in the right direction?

Assess Student Learning

Use data to improve instruction

Assess Student Learning

Use data to improve instruction & assessment

LA&S Assessment at FSUThe Liberal Arts and Sciences (LA&S)

assessment process carried out at Fitchburg State University from 2008-2010.

The actions taken by the Fitchburg State University LA&S council in 2011 to reframe the objectives, reinterpret the role of LA&S courses, and revise the assessment system.

Proposed Fitchburg State University LA&S course approval process that has emerged from the work of the LA&S council in 2011 including the rubrics utilized to assess the Fitchburg State University LA&S objectives.

LA&S Curriculum ObjectivesAesthetic Appreciation

Communication

Problem Solving and Synthesizing

Ethical Reasoning

Citizenship

The Process of LA&S Assessment• In the old days of 2008-10, we solicited

student work as needed to match learning objectives.

• Faculty on the LA&S committee (and whomever else we could corral) performed assessment of a small sample of student work that matched a particular objective.

• Two to three faculty members scored each piece of student work on a 1-3 (insufficient, sufficient, or proficient) scale.

• Final assessment data stayed with the LA&S committee.

Example: CommunicationCriterion Performance Rating

  Insufficient Sufficient Proficient

Controlling idea Student work demonstrates little or no evidence of a theme or controlling idea.

Student work demonstrates some evidence of a theme or controlling idea.

Student work demonstrates a original and clear theme or controlling idea, skillfully limited to the dimensions of the assignment.

Organization and Development

Student work does not demonstrate empathy with the audience, which must work to extract meaning and organize the ideas in the student's communication effort.

Students work evidences one or more supporting points related to the controlling idea and developed in a manner which anticipates and answers most audience questions concerning the theme or controlling idea.

Student work evidences multiple points supporting the theme or controlling idea and developed in a manner which not only anticipates likely audience questions but poses and provides satisfactory answers to new questions.

Standards of evidence

"Evidence," in this student work, is limited primarily to personal assertions and/or challenges and responses which are emotional rather than rational in character.

Student work is developed using typical reference sources which may be reliable and which seem to be represented fairly and in context. Documentation of sources appears to be substantially correct.

Student work is developed using scholarly and unimpeachably reliable sources which are scrupulously documented and presented fairly and in context.

Mechanics / Presentation

Student work contains major and distracting errors in language and/or lapses in presentation.

Student work has few distracting errors and/or lapses in presentation.

Student work skillfully uses language which is free from distracting errors and other communication skills in order to enhance communication.

Communication resultsSpring 2010 – Nine student artifacts evaluatedFall 2010 – Eleven student artifacts evaluated; Honors course

Example: Problem solving Work is correct Work is organized Work is complete Uses formulas properly, where and when appropriate Creates graphs, tables and/or statistics to summarize data Gives clear, precise and relevant explanations, identifies

causes and proposes solutions Uses appropriate methodology to collect data Integrates information from outside sources Uses numerical data to defend argument(s) Explains patterns or trends in observations, data, graphs

and/or tables Identifies pros and cons of argument(s), including biases

and/or limitations Analyzes outcomes from multiple perspectives Applies content knowledge, methods and/or results to new

situations

Problem solving results• Separation of math and science rubrics

Problem solving results• Separation of math and science rubrics• Many unused criteria

The Problems of LA&S AssessmentOur sample sizes tended to be small.In some cases, due to a dearth of available

student work and time constraints, the faculty performing the assessment did not have the assignments to which the pieces of student work were adhering.

Some of our rubrics were either too vague or too detailed and thus hard to apply.

Faculty buy-in was difficult to achieve, which undermined the assessment process, which led to further difficulty with faculty buy-in, which further undermined the process, which led to . . .

Questions?

Assess Student Learning

Use data to improve instruction & assessment

January 2011

The Vision Project Key OutcomesCOLLEGE PARTICIPATION

College-going rates of high school graduates COLLEGE COMPLETION

Graduation and success rates of the students we enroll

STUDENT LEARNINGAcademic achievements by our students on campus-level and national assessments of learning

WORKFORCE ALIGNMENTAlignment of our degree programs with key areas of workforce need in the state's economy

ELIMINATION OF DISPARITIESAchievement of comparable outcomes among different ethnic/racial, economic and gender groups

Implementation PlanSeek to become a LEAP State and establish a

Massachusetts LEAP Team.Develop a Resource Plan that outlines direct

and indirect resources needed to achieve the goals of the Vision Project.

Engage campus provosts to provide leadership for faculty engagement and campus assessment activity including LEAP-related analysis.

Establish a network of faculty and professional staff assessment leaders to engage in collaborative activities including determining system-wide learning outcome measures within LEAP framework.

Assessment Afternoons4/6  LEAP Learning Outcomes and the

Fitchburg State University LA&S Curriculum4/13 Assessing Art Appreciation4/20 Assessing Communication Skills4/27 Assessing Problem Solving5/4 Assessing Ethical Reasoning 5/11 Assessing Citizenship5/23 LA&S AND YOU: Linking Liberal Arts

and Sciences Assessment to what is happening in the departments and courses.

LEAP Essential OutcomesKnowledge of Human

Cultures and the Physical and Natural World

• Through study in the sciences and mathematics, social sciences, humanities, histories, languages, and the arts

Intellectual and Practical Skills, including

• Inquiry and analysis• Critical and creative thinking• Written and oral

communication• Quantitative literacy• Information literacy• Teamwork and problem solving

Personal and Social Responsibility, including

• Civic knowledge and engagement—local and global

• Intercultural knowledge and competence

• Ethical reasoning and action• Foundations and skills for

lifelong learning

Integrative and Applied Learning, including

• Synthesis and advanced accomplishment across general and specialized studies

Assessment afternoonsWho came?

What were our goals?

What actually happened?

LA&S Curriculum ObjectivesAesthetic Appreciation

◦Aesthetic Appreciation and Aesthetic ExpressionCommunication

◦Oral Communication◦Written Communication

Problem Solving and Synthesizing◦Problem Solving through Quantitative Literacy◦Problem Solving through Inquiry and Data

AnalysisEthical Reasoning

◦Ethical Reasoning when Making ChoicesCitizenship

◦Citizenship through Critical Analysis of Events

The Arts rubricDeficient Sufficient Proficient

Artistic analysis Student's work demonstrates a superficial, impressionistic analysis of artistic expression.

Student's work demonstrates a basic conceptual analysis of artistic expression.

Student's work demonstrates a sophisticated, multi-dimensional analysis of artistic expression.

Art in context Student's work fails to engage art's historical, cultural, psychological, conventional, or other contexts.

Student's work demonstrates some awareness of art's historical, cultural, psychological, conventional, or other contexts.

Student's work engages and synthesizes art's historical, cultural, psychological, conventional, or other contexts.

The Arts rubric revisionDeficient Sufficient Proficient

Artistic analysis Student's work demonstrates a superficial, impressionistic analysis of artistic expression.

Student's work demonstrates a basic conceptual analysis of artistic expression.

Student's work demonstrates a sophisticated, multi-dimensional analysis of artistic expression.

Art in context Student's work fails to engage art's historical, cultural, psychological, conventional, or other contexts.

Student's work demonstrates some awareness of art's historical, cultural, psychological, conventional, or other contexts.

Student's work engages and synthesizes art's historical, cultural, psychological, conventional, or other contexts.

Composition/Performance

Student’s work displays no coherence or elegance of composition or performance.

Student’s work displays basic coherence and elegance of composition or performance.

Student’s work displays sophisticated coherence and elegance of composition or performance.

Communication rubric revisionsStudents will speak, read, write and listen to create and understand meanings using a variety of media. They will recognize how to participate in or lead groups to accomplish goals.

Written communication Controlling idea Organization and Development

Sources and Evidence

Academic Discourse

Mechanics/Presentation

Oral communication Controlling idea Organization and Development

Supporting Material

Oral and Physical Delivery

Mechanics/Presentation

Problem-Solving rubric revisionsAmong these strategies,

students will analyze and interpret data as a means to evaluate arguments and make informed choices.

Topics Selection Information from sourcesAppropriate methodologyCreates graphs, tables, etc.Explains patterns or trendsNumerical data to defend Identifies pros and consApplies content knowledge

Form problem-solving strategies and evaluate their effectiveness.

Work correct and completeUses formulas properly

Creates graphs, tables, etc.Explains patterns or trendsGives clear explanations

Applies content knowledge

Criterion PROBLEM SOLVING RUBRIC: Performance Rating  Deficient Sufficient Proficient

Work is correct. Does not meet minimum expectation for the said criterion.

Meets minimum expectations for the said criterion.

Exceeds minimum expectations for the said criterion.

Work is organized. Does not meet minimum expectation for the said criterion.

Meets minimum expectations for the said criterion.

Exceeds minimum expectations for the said criterion.

Work is complete. Does not meet minimum expectation for the said criterion.

Meets minimum expectations for the said criterion.

Exceeds minimum expectations for the said criterion.

Uses formulas properly, where and when appropriate.

Does not meet minimum expectation for the said criterion.

Meets minimum expectations for the said criterion.

Exceeds minimum expectations for the said criterion.

Creates graphs, tables and/or statistics to summarize data.

Does not meet minimum expectation for the said criterion.

Meets minimum expectations for the said criterion.

Exceeds minimum expectations for the said criterion.

Gives clear, precise and relevant explanations, identifies causes and proposes solutions.

Does not meet minimum expectation for the said criterion.

Meets minimum expectations for the said criterion.

Exceeds minimum expectations for the said criterion.

Uses appropriate methodology to collect data.

Does not meet minimum expectation for the said criterion.

Meets minimum expectations for the said criterion.

Exceeds minimum expectations for the said criterion.

Integrates information from outside sources.

Does not meet minimum expectation for the said criterion.

Meets minimum expectations for the said criterion.

Exceeds minimum expectations for the said criterion.

Uses numerical data to defend argument(s).

Does not meet minimum expectation for the said criterion.

Meets minimum expectations for the said criterion.

Exceeds minimum expectations for the said criterion.

Explains patterns or trends in observations, data, graphs and/or tables.

Does not meet minimum expectation for the said criterion.

Meets minimum expectations for the said criterion.

Exceeds minimum expectations for the said criterion.

Identifies pros and cons of argument(s), including biases and/or limitations.

Does not meet minimum expectation for the said criterion.

Meets minimum expectations for the said criterion.

Exceeds minimum expectations for the said criterion.

Proficient3

Sufficient2

Deficient1

Work is correct and complete.

Calculations attempted are essentially all successful and sufficiently comprehensive to solve the problem. Calculations are also presented elegantly (clearly, concisely, etc.)

Calculations attempted are either unsuccessful orrepresent only a portion of the calculations required to comprehensively solve the problem.

Calculations are attempted but are both unsuccessful and are not comprehensive.

Uses formulas properly, where and when appropriate.

Uses formulas correctly and appropriately and with analysis/explanation.

Uses formulas correctly and appropriately but without analysis/ explanation.

Uses formulas incorrectly OR in inappropriate places.

Creates graphs, tables and/or statistics to summarize data.

Skillfully converts relevant information into an insightful mathematical portrayal in a way that contributes to a further or deeper understanding.

Completes conversion of information but resulting mathematical portrayal is only partially appropriate or accurate.

Completes conversion of information but resulting mathematical portrayal is inappropriate or inaccurate.

Explains patterns or trends in observations, data, graphs and/or tables.

Provides accurate explanations of information presented in mathematical forms. Makes appropriate inferences based on that information. For example, accurately explains the trend data shown in a graph and makes reasonable predictions regarding what the data suggest about future events.

Provides somewhat accurate explanations of information presented in mathematical forms, but occasionally makes minor errors related to computations or units. For instance, accurately explains trend data shown in a graph, but may miscalculate the slope of the trend line.

Attempts to explain information presented in mathematical forms, but draws incorrect conclusions about what the information means. For example, attempts to explain the trend data shown in a graph, but will frequently misinterpret the nature of that trend, perhaps by confusing positive and negative trends.

Gives clear, precise and relevant explanations

Uses the quantitative analysis of data as the basis for deep and thoughtful judgments, drawing insightful, carefully qualified conclusions from this work.

Uses the quantitative analysis of data as the basis for workmanlike (without inspiration or nuance, ordinary) judgments, drawing plausible conclusions from this work.

Uses the quantitative analysis of data as the basis for tentative, basic judgments, although is hesitant or uncertain about drawing conclusions from this work.

Applies content knowledge, methods and/or results to new situations.

Insightfully discusses in detail relevant and supported limitations and implications of content knowledge, methods and results.

Presents basic, relevant and supported limitations or implications of content knowledge, methods and/or results.

Any presented limitations and implications are possibly irrelevant and unsupported.

Ethical Reasoning rubric revisionStudents will recognize the ethical issues involved in human actions and be able to formulate a set of principles and virtues which can be brought to bear in personal and public decision making.Moral ReasoningStatement of PositionEthical Issue RecognitionApplication of Ethical Perspectives/ConceptsDevelopment of RationaleEthical Self-AwarenessEvaluation of Different Ethical

Perspectives/Concepts

Citizenship rubric revisionStudents will articulate the relationships among local, national and global concerns, interests and needs. They will recognize possibilities and opportunities to enact positive change on an individual or group level.• Explanation of Event

• Evidence

• Student’s Position

• Diversity of Communities and Cultures

•Analysis of Knowledge

Questions?

Assess Student Learning

Use data to improve instruction & assessment

NEEAN Summer InstituteThrough facilitated dialogue, participants will:

1. Explore options for institutionalizing assessment to improve student learning; 2. Learn how to improve the reporting process; 3. Consider ways to enhance faculty development and target resources to continuous improvement; 4. Become familiar with helpful tools and methods to assess and improve student learning; and 5. Discuss how to best analyze data.

Affecting change through governanceWhat are our goals:

◦Increase faculty awareness/involvement

Increase the variety of data we collect

Increase faculty input on the process

◦Make LA&S-ness primary in course design

Means of effecting change Alter the course approval process

to ask for:◦Which objective(s) the course

addresses

◦Description of an assignment (aligned with an LA&S rubric) whose student work could be submitted to the LA&S council

◦Option to indicate need for new rubric

Pros and cons of this ideaPros:

◦Achieves many of our goals Increase faculty awareness/involvement

Increase the variety of data we collect Increase faculty input on the process

Make LA&S-ness primary in course design

Cons◦Expect faculty resistance

LA&S Assessment Cover SheetsPlease indicate yes or no for each of these boxes and then

make a final yes/no judgment on whether scorers should

assess this criterion.Faculty member or department name

_________________Course #__________Course Name

_________________Assignment name __________# students in course

______Date ___ % of grade □< 3% □3-5% □6-10%

□11-20% □> 20%

Aesthetic Appreciation through Aesthetic Expression Criteria: See rubric on back for details

This criterion is . . . Taught as part of course instruction

Addressed in assignment prompt

Appropriate to be assessed

Aesthetic analysis

Work in context

Composition/Performance

Final Summary SlideAssessing student work has value

even when the process is imperfect because the data allows us to improve our assessment methods.

A meaningful assessment system has to take into account what faculty are actually doing with their students in the classroom.

Assessment may be more meaningful and effective if it is integrated into the process of course design and development.

Questions?

Assess Student Learning

Use data to improve instruction & assessment

Thanks to:The Fitchburg State University

LA&S CouncilThe Davis FoundationNew England Educational

Assessment NetworkDr. Ben Railton and Dr. Eric Budd

past chairs of the LA&S CouncilFitchburg State University Office

of Academic Affairs