Minutes of the Third AMCOA Meeting, August 18, 2011

Prepared by Kerry McNally

Host Campus: Holyoke Community College

I. Attendance

The third AMCOA meeting was hosted by Holyoke Community College from 10:00 a.m.-12:00 Noon on August 18, 2011. Representatives from 21 institutions attended the meeting (See list in Appendix A), as well as Richard M. Freeland, Commissioner of Higher Education; Jonathan Keller, Associate Commissioner for Research, Planning and Information Systems; Anne Perkins, Vision Project Research Associate, Research and Planning; and Peggy Maki, Consultant under the Davis Educational Foundation Grant awarded to the Department of Higher Education, who also chaired the meeting.

II. Holyoke Community College President William Messner Opening Remarks

President William Messner welcomed the AMCOA Team to Western Massachusetts and Holyoke Community College. He praised the Team for representing campuses all across the state and for getting together and acting “as a system.” “We don’t do enough of that.” He emphasized the importance of this collaborative work and credited Commissioner Freeland’s leadership for bringing campuses to these conversations. After having guided Holyoke Community College through a New England Association of Schools and Colleges (NEASC) accreditation review this past spring, he said that it is wonderful that the NEASC is reviewing schools with an eye for improvements in assessment

1

and that assessment is now a priority for the state and federal governments, as well as the schools.

III. New AMCOA Team Members

Peggy Maki introduced new Team members who were not introduced at the previous meeting: Benjamin Railton, Associate Professor of English, Coordinator of American Studies, Fitchburg State University; Ann Caso, Director of Institutional Research, and Susan Chang, Director of Assessment, at Framingham State University; Richard Parkin, Assistant Vice President/Academic Affairs, and Linda Meccouri, Professional Development Coordinator and Professor, Multi Media Technology, Springfield Technical Community College; and Yves Salomon-Fernandez, MassBay Community College.

IV. Update on Conferences (place, date, chairs)

Peggy Maki provided an update on the scheduled conferences and asked for a volunteer to Co-Chair the November 17th conference at Greenfield Community College to serve with Ellen Wentland. Judith Turcotte volunteered for this role.

Jim Gubbins of Salem State University volunteered to Chair the Fourth AMCOA Conference on April 23rd.

Peggy reminded AMCOA Team members to confirm their correct titles for the September 30th Conference Brochure as soon as possible. Their names will appear on the brochures for each of our conferences.

Kris Bendikas, chair of the September 30th statewide conference, distributed a handout that listed the Conference Planning Group’s recommended sessions for that conference. (See handout in Appendix B.) All of the presentations could not be included in the first conference,

2

but will be given at one of the other conferences. It was decided that after each session there will be a companion roundtable as well as additional roundtables focused on topics identified by the AMCOA group, such as the following:

a) How LEAP outcomes are integrated into campusesb) How to assess onlinec) How to facilitate alignment between NEASC expectations and

those of DHEd) How to collect, store, and represent assessment results

Jonathan Keller will present a session as well as offer a roundtable after his session and be available at the rest of the conference. At the meeting Peggy asked members if they could identify topics for Jonathan that they would find useful. Among those recommended was a focus on identifying “what not to do in preparing or reporting results.”

Discussion also focused on how we can make our four statewide conference presentations available to our public institutions. Specifically, we focused on discussing the possibility of videotaping, audio taping, recording PowerPoints, or uploading handouts to our social network--Yammer. Lori Dawson is currently looking into the possibility of videotaping and can confirm that method after she sees the final schedule. Several team members stated that faculty and others who will not be able to attend the conference have already stated that they would like access to materials. We also discussed how important session contents are to faculty development.

Peggy reminded the Team to limit the number of colleagues attending the September 30th Conference to 2-3 because of space considerations at the Conference location. However, she said that Kerry will be able to track registrations and let institutions know if they can bring more than 2-3 people.

3

V. Volunteers to introduce sessions on September 30, 2011

Peggy asked for volunteers to introduce sessions at the conference. The following individuals volunteered: Susan Taylor, Yves Salomon-Fernandez, Suzanne Van Wert, Ellen Zimmerman, Michael Vieira, Susan Keith, Martha Stassen, Judith Turcotte, and Carol Lerch.

VI. Discussion of assessment experiments. Peggy Maki, AMCOA Consultant; Commissioner Richard Freeland; and Jonathan Keller, Associate Commissioner for Research, Planning, and Information Systems

a. Parameters for Assessment Experiments

Peggy Maki distributed a draft of parameters for assessment experiments supported by the Davis Educational Foundation Grant. After she reviewed those parameters (see Appendix C), she responded to questions about them. In response to the question about faculty receiving stipends for participating in experiments, she stated that funds could be used in that way. Another question raised was whether including performance on remedial coursework could be included. Peggy said she thought that could be included in experiments. A question was raised about whether or not “useful results for transfer from two-year to four-year institutions” should be a criterion for the experiments to address. Charlotte Mandell said that four-year institutions can disaggregate their data to look at the two-year population and that this is an important focus. Peggy noted that a common thread in her conversations at the community colleges is their desire to make sure that transfer students are able to demonstrate the same level of learning as those students who do their first two years at a four-year institution. Overall, Peggy stated, the issue that these experiments will have to address is that of scalability.

It was agreed that AMCOA team members would vet this current draft on their campuses and submit proposed edits or changes by Sept. 15, 2011. Peggy will incorporate proposed changes into a final Request for Proposals that would go out to team members one more time for final

4

approval before being officially released. It was also agreed that to develop strong proposals the deadline for submission should be moved to October 31, 2011. The Commissioner’s Advisory Committee would then select proposals in November 2011 based on the ranking of proposals by volunteer AMCOA members. (The appended draft reflects new dates proposed by team members at the meeting.)

b. Richard M. Freeland, Commissioner of Higher Education

Commissioner Freeland expressed his gratitude for the work of the AMCOA team –especially participants’ commitment to meet over the summer months. He also expressed his excitement about the AMCOA project and the importance of our focus on documenting our students’ learning. He stated that he believes “We are on the edge of a real revolution in Higher Education, and Massachusetts has a chance to initiate the dialogue. There are many political aspects in public higher education, but the educational outcomes overshadow the political sides. We must emphasize the educational outcome of this process.” Further, he stated that if faculty and other educators at our institutions approach outcomes assessment in the right way, they can become more effective teachers, particularly for the range of students who come into public higher education. This commitment can be the most important development in higher education.

The Commissioner also described how the Davis grant supports campus-based work in assessment through the four statewide conferences, the AMCOA meetings and members’ leadership, Peggy Maki’s initial campus visits and her availability as a campus resource—all of which will lead to the design of an eventual statewide system of reporting students’ levels of achievement. Thus, the State’s approach to assessment is based on two levels: (1) campus-level and (2) statewide level. Focus on statewide reporting is aimed at learning how well students are doing so that we can identify ways to improve students’ learning. Even though political issues may emerge, he noted that we are capable of having discussions about

5

these levels as demonstrated by the AMCOA project itself. He also added that if a campus does not want to participate in the eventual reporting, that is O.K. What we do need to keep sight of is that “stronger education is our most important outcome.” Thus, he hopes that all campuses will want to join this statewide effort of reporting assessment results.

The Commissioner also reported on movement towards becoming a LEAP-affiliated State, a movement that was endorsed by the Board in June. Neal Bruss requested that the Board’s action on this be distributed to our campuses. And, the Commissioner reported that he intends to share this possible affiliation with NEASC as well as meet with NEASC representatives to discuss the ways in which an affiliation would align with NEASC reporting requirements.

The Commissioner reported on his July presentation at a national conference of SHEOs (State Higher Education Officers) during which he talked about learning outcomes assessment within the context of the Vision Project and the parameters of the AMCOA Project. Responses from the SHEOs and nationally recognized leaders such as Peter Ewell, Carol Schneider, and Paul Lingenfelter were that Massachusetts is way ahead on this focus. Specifically, he stated that conference participants were intrigued by the notion of a reporting system design that is not mandated. Other states expressed interest in working with us, so there is a lot of interest in our work. After the presentation there was a memo sent around lauding the work taking place in Massachusetts and asking other states to take note of our work. Peter Ewell stated that he thought our approach is “the best authentic assessment for accountability and improvement.”

Finally, the Commissioner reported that AAC&U has requested that we participate in a large Lumina grant they expect to receive with specific focus on developing better understanding about what students should be

6

learning at different levels (as represented in the Lumina Undergraduate Profile) to facilitate effective transfer. The Commissioner stated that because as a state we are well down the road with discussions about LEAP outcomes and VALUE rubrics, it would not be advisable for us to be diverted to another direction. Thus, he stated we need another year to see how we progress with our current direction and commitment.

c. Jonathan Keller, Associate Commissioner for Research, Planning, and Information Systems

Looking at the partnership focus of our AMCOA work, particularly in the design of assessment experiments that will inform a statewide system of reporting, Jonathan stated that he and his staff are happy to contribute their expertise and experience to the experiments--particularly since the scope of this commitment to design a system can seem overwhelming. He said that the State has resources that are available to assist team members as they and others design a reporting system. Specifically, he stated that he and his staff can assist in: (1) developing experiments; and down the road (2) developing something statewide. He also said that he can assist with analyzing and interpreting NSSE and CCSSE results because he has extensive experience with those instruments. He also can provide advocacy for looking at where institutions shine, such as identifying activities that show critical thinking. He has resources that can help with database development and definitions, demographics, and tracking students.

VII. Suggestion to develop a database of assessment instruments

Professor Benway has recommended that under our Davis Grant we develop a database of assessment instruments. Discussion focused on the value of this effort to all of our institutions as well as the significance of this database as a possible way to represent to external stakeholders how we assess our students—perhaps as part of the eventual statewide reporting system. Rather than simply list methods, there was a

7

recommendation that there be some kind of reporting or description of each method’s particular usefulness. Peggy will ask one of our team members to take a stab at writing a description of what we might include in this proposed database. This description will come forward to the group for further discussion.

VIII. Next AMCOA Meetings

The next AMCOA meeting will take place on September 14th from 10:00 a.m.-12:00 Noon at Massachusetts College of Liberal Arts. Peggy Maki will send an agenda and driving directions to the site one week before the meeting. Please let Kerry know as soon as possible if you plan to attend the meeting or cannot.

Mark Your Calendars: September 30, 2011: First Statewide Assessment Conference at Worcester State University, 9:00 a.m. – 3:00 p.m. (8:30 a.m. Registration)

8

Appendix A

Institutions Represented at the AMCOA August 18th Meeting:

Berkshire Community CollegeBristol Community CollegeBunker Hill Community CollegeCape Cod Community CollegeFitchburg State UniversityFramingham State UniversityHolyoke Community CollegeMassachusetts College of Liberal ArtsMassasoit Community CollegeMassBay Community CollegeMiddlesex Community CollegeNorthern Essex Community CollegeQuinsigamond Community CollegeRoxbury Community CollegeSalem State UniversitySpringfield Technical Community CollegeUniversity of Massachusetts AmherstUniversity of Massachusetts BostonUniversity of Massachusetts LowellWestfield State UniversityWorcester State University

9

Appendix B Session Proposals for September 30th AMCOA ConferenceTopic Presenters Institution“Teamwork: the Key to Faculty Engagement.” Discussion of the Title III funded program, The Connected College, administered at Bristol Community College, which involves collaborating with faculty to develop institution and program-level assessment strategies through the creation of a set of Course Design Toolkits.

Kevin Forgard, the project’s instructional designerMaureen Melvin Sowa, Professor of History

Bristol Community College

“The development and function of BHCC’s faculty-driven assessment initiative: the Student Learning Outcomes Assessment Project (SLOAP).” Based on a conceptual assessment framework that provides an institutional model for all assessment activities.

Judy Lindamood, Chair, Early Childhood Education & Human Services DepartmentTimothy McLaughlin, Chair, English DepartmentNatalie Oliveri, English DepartmentDavid Leavitt, Director of Institutional Research

Bunker Hill Community College

“Using Business Process Analysis to Facilitate Buy-In and Effective Assessment Information Flow.”

1) How to improve faculty buy-in by approaching VALUE Rubrics as a customizable tool to test in a low-pressure pilot project

2) How to get started on a Business Process Analysis

Ann Caso, Director of Institutional ResearchSusan Chang, Director of AssessmentCynthia Glickman, Business Systems AnalystPatricia Lynne, Associate Professor and Assessment Liaison, English DepartmentEllen Zimmerman, Associate Vice President for Academic Affairs

Framingham State University

“How Institutional Research Supports Middlesex Community College’s Program Review Process.” Improving access and advancing student success by strengthening evidence-based practices and resource allocation. Embedded in this program review process is the assessment of program effectiveness, including impact on students – student success and student learning – for both academic and co-curricular programs.

Lois Alves, Vice President of Institutional Research and Enrollment ServicesCynthia Lynch, Service Learning CoordinatorElise Martin, Associate Dean of Assessment

Middlesex Community College

10

Appendix B (continued)

Topic Presenters Institution“Assessment of Supplemental Instruction.” In the spring of 2011, NECC carried out an extensive assessment project, focused on classes designated for Supplemental Instruction (SI). Quantitative and qualitative data was collected through methods including surveys completed by students, faculty, and SI leaders. The findings provided important insights into SI at NECC, and suggested numerous strategies for improvement.”

Lynne Nadeau, Coordinator: Academic Resource and Tutoring CenterLinda Shea, Assistant Dean: Library and Academic Support ServicesEllen Wentland, Assistant Dean: Academic Program Review, Outcomes Assessment, and Educational Effectiveness

Northern Essex Community College

“Discussion of UMass Dartmouth’s Commitment to Student Learning statement (CSL).” Founded on broad campus discussion, using LEAP principles, and motivated by NEASC standard 4.16, it states aspirational goals for students upon graduation. Developed collaboratively, CSL has guided renovation of general education, improved advising, and will soon help refocus academic major learning goals.

Richard Panofsky, Assistant Chancellor for Institutional Research & Assessment

UMass Dartmouth

“Institutional Research.” Jonathan Keller, Assoc. Commissioner for Research, Planning and Information Systems

DHE

“Using NSSE/CCSSE Results.” A panel discussion to share strategies and best practices.

David Leavitt, Director of Institutional ResearchElise Martin, Associate Dean of Assessment TBD_________________________

TBD_________________________

Bunker Hill Community CollegeMiddlesex Community College

11

Appendix C

Draft for Review by AMCOA TEAM

Parameters for AMCOA Assessment Experiments

August 18, 2011

Funding for the AMCOA project under the Davis Educational Foundation grant awarded to DHE includes support for scalable assessment experiments developed among our public higher education institutions that have the potential to: (1) “provide a foundation for the system-wide plan connected to the Vision Project,” and (2) demonstrate that a “campus and system collaborative approach to assessment of student learning can be helpful to public campuses and to the public system as a whole” (Davis Educational Foundation Grant).

The following list provides parameters for developing a proposal to undertake the design of an assessment experiment. The experiment:

1. Builds on current campus-based systems that report results as well as plans to improve student learning, creating a statewide system that addresses both (a) accountability and (b) improvement.

2. Reports exiting students’ achievement levels in General Education based on scoring students’ authentic work, using agreed upon nationally informed scoring rubrics such as the LEAP rubrics and translating assessment results into scores or dashboards that are useful across campuses. Initially, GE reporting will begin with a focus on results of scoring student work that demonstrates critical thinking, writing, and quantitative reasoning.

3. Demonstrates or offers the ability to aggregate or to disaggregate assessment results based on institutional demographics

4. Experiments with a. web platforms (including current existing data systems at

DHE) to enable disaggregation and aggregation of assessment results across student populations or

b. portfolio assessment technologies that would provide cost efficiencies for campuses

5. Identifies plans to improve patterns of weakness in student work6. Provides useful results for transfer from two-year to four-year

institutions7. Experiments with the use of NSSE and CCSSE surveys and consortia

that enable comparisons between or among similar institutions, including consortia development of additional survey items for NSSE and CCSSE instruments

12

Appendix C (continued)

Funding:

The Davis Grant provides a total of $60,000 for this phase of the AMCOA Project. The maximum grant for a proposed experiment is $12,000.

Proposal:

Please submit to Peggy Maki by October 31, 2011 a one- to two-page proposal that includes the following information:

Initially, a description of your overall approach to your experiment based on the 7 parameters listed above

Identification of those involved in your experiment, including the person who will chair or convene the group

Approximate timeline you will follow to develop an experiment that you will demonstrate at the fourth statewide assessment conference

Budget that may include support for meetings, compensation for faculty time, compensation for new DHE staff demands focused on developing, pilot testing, or assisting the development of web-based reporting in an assessment experiment

Proposals that involve collaboration between or among two-year and four-year institutions are welcome, demonstrating that reported results are useful for our public institutions and provide valuable information about our students’ achievement levels across their education. Submitted proposals will be reviewed and ranked by a panel of AMCOA members and then submitted to the Commissioner and Advisory Board for final selection in November, 2011.

13