Minutes of the Tenth AMCOA Meeting, May 1st, 2012
Prepared by Kerry McNally
Host Campus: Framingham State University
I. Attendance
The tenth AMCOA meeting was hosted by Framingham State University (FSU)
from 10:00 a.m.-1:00 p.m. on May 1st, 2012. Representatives from 23
institutions attended the meeting (see list in Appendix A), and Peggy Maki,
Consultant under the Davis Educational Foundation Grant awarded to the
Department of Higher Education, opened and chaired the meeting.
Peggy thanked Ellen Zimmerman and FSU for hosting the meeting.
II. Welcome: Vice President for Academic Affairs Dr. Linda Vaden-Goad,
Framingham State University (FSU)
Vice President Linda Vaden-Goad welcomed AMCOA Team members to FSU
and this month’s meeting featuring an hour and a half panel discussion on
Quantitative Reasoning. She pointed to FSU’s new community garden, which
was visible from the meeting location, and compared it with educational
learning communities. “We’re ‘assessing’ the garden’s impact on the
community. Similarly, educational assessment is important, particularly how
we offer learning to the community. It is important to look at the
effectiveness of what we do. Sometimes we don’t know the impact of what
we do for years. It is essential to be doing outcome assessment as we go
- 1 -
along.” Dr. Vaden-Goad thanked the AMCOA Team for coming to FSU and for
all it is doing in the area of improving assessment methodologies.
Peggy Maki then welcomed the AMCOA Team to the final meeting of the
2011-2012 Academic Year. She told the Team that she would be sharing
responsibilities next year with the co-chairs because she will be reducing her
time on the project. “It has been my honor,” she said, “to work with you on
this project. It has been a highlight of my professional life.”
III. Introduction of Presenters, “Perspectives on and Experiences with Assessing
Quantitative Reasoning”: Peggy Maki
Peggy Maki said that it is important to hear from the people who are involved
in Quantitative Reasoning (QR) and learn about their experiences. She then
introduced the panelists:
Mark Pawlak, Director, Academic Support Department: Academic Support
Programs, University of Massachusetts Boston
Ellen Wentland, Associate Dean of Academic & Institutional Effectiveness,
Northern Essex Community College
John Donnellan, Professor of Business Administration, Holyoke Community
College
Richard Eells, Professor, Department of Math, Science & Technology, Roxbury
Community College
Alex Asare, Professor, Department of Math, Science & Technology, Roxbury
Community College
Christopher Cratsley, Director of Assessment, Fitchburg State University
Dawne Spangler, Director of the Center for Teaching, Learning and
Assessment, North Shore Community College
- 2 -
A. Mark Pawlak (MP), Director, Academic Support Department: Academic
Support Programs at UMass Boston opened the panel discussion. At the
outset he told the group that because of time restraints, he would
distribute his PowerPoint presentation at a later date, rather than showing
it at the meeting. A copy of Mark’s PowerPoint presentation is attached as
Appendix B.
He has worked for 14 years in QR assessment for an entry-level college-
level course. Initially, people were failing College Algebra and Calculus,
mostly those from the Liberal Arts majors. He and the University wanted to
give them a more authentic experience, so he developed a QR course that
incorporated statistics, numeracy, probability, etc., drawn from newspaper
articles and authentic problems. The students used technology in a robust
fashion. Students also learned why it is valuable to calculate percentages and
why a little Algebra can be useful.
Regarding the VALUE rubrics, UMass Boston is not there yet, but it is
incorporating some of them. Its QR program currently uses Blackboard, which
the Math faculty are accustomed to. Over the years UMB has migrated to a
portfolio, consisting of entries from a common final exam and a student
questionnaire. Faculty are allowed to add questions to the final exam. UMB
does a holistic evaluation of the exam to see if faculty are achieving agreed
upon assessment goals.
AMCOA Members asked the following questions:
What percentage of students is taking the course? We teach 11-12 sections
in the summer, and about 500 students take it per semester. At UMB QR is
required, and this course fulfills the requirement. The General Education QR
committee oversees this requirement. When new courses are approved, they
- 3 -
follow a set of guidelines for approval. The outcomes are understood by
everyone teaching the course. Mark Pawlak designs the outcomes and then
gives them to faculty for final review.
Do you follow up on the students later? We don’t do that formally, but we
do get anecdotal information. We have gotten input from faculty in
economics and social science statistics courses. We have also learned that
passing the placement test is not good enough to go right into Statistics.
Is the final exam the same each year? No. It changes every year.
Do you use Accuplacer for the first-time freshmen? Yes, but we also use in-
house placement tests based on the Mathematical Association of America.
B. Ellen Wentland (EW), Associate Dean of Academic & Institutional
Effectiveness, spoke on the QR assessment initiative at Northern Essex
Community College.
Several years ago, the college set six assessment goals. Last year, the
College established an interdisciplinary global awareness and QR learning
assessment for writing. We created an authentic-type scenario that was
fictional. In writing, it is difficult to assess products when the assignments
vary so much. So, we developed a fictional assignment that would be
uniform: Six countries needed to buy oil. We incorporated political
philosophies and relationships to the U.S. and asked students what they
would do to get oil from these countries. The solution requires the ability to
interpret charts, tables and graphs, and add and subtract. These are the
constants that you need, plus some basic math. Faculty had to give up 50
minutes of their class time to implement this scenario, considered to be a
moderate-level assessment. The faculty agreed and students gave their
permissions. There were detailed instructions on the vignette to students,
because motivation is lower if it is not graded. We used students with 49-59
- 4 -
credit hours and we accepted classes with, at least, 3 willing students, but
hoped to get 5 or 6 students. We collected the products, and raters who
taught math, government, and political science gathered for norming
sessions. The last page of the instructions to students contains the scoring
rubric (before we got into VALUE rubrics). We also looked at the VALUE rubric
to see if the interpretations were similar.
In general, faculty who rated were disappointed with the results of
students’ performance on global awareness even though the vignette did not
demand much of QR. We wondered if students’ performance resulted from
their lack of motivation; nevertheless, the scenario didn’t work, but
stimulated discussions on campus about the nature of assignments that
prompt students to demonstrate QR. If the VALUE QR rubric were used at the
College, we could become more intentional about aligning our assignments
with the criteria of the VALUE rubric. In actual courses faculty would be
intentional about their QR assignments.
AMCOA team members asked the following questions or made the following
comments:
Tom Curley (TC) said that Berkshire Community College has a variant
assignment – an out-of-class writing assignment that includes using graphs
and doing real research.
EW: NECC considered that, but questioned “what is more like real life?” On
the job, you are asked what you think about a situation, and you must
respond while at work. That is real life, so we thought using an in-class
project was more authentic.
TC: If you are evaluating depth of knowledge, an out-of-class assignment
would be a more comprehensive exam. You could have two options, one in
- 5 -
and one outside of class. You could also make it count for 10% of the class
grade to increase motivation.
Elise Martin (EM): Do you use the same methodology (a vignette) outside of
the class, or do you use products from the course?
EW: We get courses to define an intensive-level assessment assignment. We
don’t have QR covered yet. The question is: where is that going to appear?
Pharmacology has it down. We are following the students’ choices to see
how it can fit in other courses.
C. Next to speak was John Donnellan, Professor of Business Administration,
at Holyoke Community College (HCC).
John is a member of HCC’s Gen Ed Assessment Committee that is charged
to look at QR as a core competency in various courses. Initially, a focus group
was formed, and then the College looked at assessment in student work
samples first by asking students questions in focus groups (28 students in 3
groups). We bribed them to participate by offering them lunch, and it
worked. We got 4 pages of results from the focus group. Predictably,
students feared math and QR. Students said that some teachers made QR
easier to understand than others. Another common response was that
students wanted to know why they should take Calculus if they would never
use it. However, they seemed to understand that QR was an important skill
to have in daily living. After our focus groups, we thought through our
commitment to assess QR in the following ways:
We wanted to create an assignment.
We wanted to establish criteria, but not grade student work.
We assumed that QR is not just in the math department; it could
happen across courses.
- 6 -
We looked at a broad range of courses – chemistry, math, statistics,
psychology statistics, economics and music – to assess QR. We
didn’t assess music, because we didn’t know how.
Judy Turcotte knew a community college in Brooklyn, NY, that had
long experience in assessment using the AAC&U rubric in QR, and
had normalized and standardized it. At HCC the feeling was: why
re-invent the wheel? We had moderate success using the
AAC&U/Brooklyn-normalized rubric.
Some of the drawbacks of using their rubric?
o There was no “Does Not Apply” option.
o Standards were too high, but better than anything we could
come up with on our own.
We used it in a sample assignment in Economics to try to see how it
could be integrated into an assignment. The assignment asked
about a society’s food print – what people eat and where food
comes from. The mileage to get the food was flawed because it
showed finished product to people, rather than raw materials to
people, but it got students to look at data.
The Outcomes:
o Calculation is where they did best.
o A score of 2.5 was acceptable, which was closest with
calculations.
When students were told how to calculate and come up with
answers, it was higher. It was acceptable with communications.
- 7 -
The last page involved frequencies, but we were most interested in
the means.
AMCOA members asked the following questions:
Charlotte Mandell (CM), UMass Lowell: Could an assignment be designed to
more effectively get the outcomes? Yes, e.g., a vignette like Ellen Wentland
did.
Peggy Maki: Were you trying to get at the reasoning behind QR, not just
rate calculations? Yes, we found that the Chemistry group did not do so well
with this. There is an emphasis on rote memory in that discipline.
D. Then, Richard Eells (RE), Professor, Department of Math, Science &
Technology from Roxbury Community College (RCC) spoke.
Professor Eells said that he is not an assessment professional, but he has
been assessing QR for many years. RCC uses common sense mathematics.
Fifteen years ago, there was a first-level course in College Algebra and a
watered-down Statistics course, which was like QR. No inference was
covered, so it was not considered Statistics at UMass Boston, but it required
writing, thinking and analysis, so we had a QR course all along. However, we
had a problem: the statistics course that we had was more relevant to the
social sciences, so more people were taking the Algebra course, even though
it was irrelevant to most of them.
Finally, RCC gave in to UMass Boston and started a QR course this year. It
includes Fermi Problems – back-of-the-envelope-type questions, using rough
estimates. There is extensive use of Excel, and students feel empowered by
that. The outcome: the book we use is a web resource and is taught in the
computer lab. How do you write a test or quiz to gauge learning? One
- 8 -
colleague gives open-book tests, so the test must be much harder, and
students spend a great deal of the test time looking through the book to come
up with answers.
I’ve learned that Google is a great calculator. Some students just type in a
quantitative question and get an answer from Google without really knowing
the underpinnings of the mathematical process. I have taught charts and
graphs, like a pie chart, and students will Google the answer. I have not
developed a rubric, which is in two-dimensions. Facetiously, he said that he
wants to develop one in three dimensions – a rubric cubed.
At RCC the college math courses use number systems, set theory, Venn
diagrams, and/or, etc. They do not rely on Algebra because students find that
too abstract. There is now a 50% pass rate in the new QR course and, he said,
“We are trying to write better QR questions that are not Googleable.” He
concluded by saying that we should be asking ourselves what life skills we
want our students to take away and know in ten years?
E. Alex Asare, Professor of Math, Roxbury Community College. He teaches
the second section of the QR course.
Students find it challenging and difficult to learn to reason quantitatively.
A lot have dropped out of the course. However, the students have grown
accustomed to using Excel, and they are doing well with that. RCC has a
number of full-time Professors. We haven’t divided into groups, at least not
formally yet. There is a Carnegie Study that indicates that students do not
have enough experience with technology, so we would like to deal with that.
In response to a question raised about how successful group work can be
in solving QR-based problems, a faculty member visiting our meeting stated
that he uses problems generated from the university database and then pairs
- 9 -
students to solve them. Students are asked questions about affirmative
action and diversity, with groups opposing each other, arguing from different
perspectives. They select the data to inform their arguments, and they do a
PowerPoint presentation to back up the data. We also use high school
problem scenarios, and then ask students to solve the problems and write an
essay on how they solved those problems.
EW: It is similar to a debating society that takes opposing views and backs
them up.
F. Christopher Cratsley (CC), Director of Assessment, Fitchburg State
University
We looked at problem-solving QR and then asked: How can we assess
problem solving? Answering that question led to two rubrics at Fitchburg
State University:
1) Analyzing data. Students collect and analyze data and develop
conclusions.
2) Making Calculations in a meaningful way. We’d like to get to
applications. We hope to impart a facility with equations: using them to
make predictions, rising to a threshold, then expanding the analysis and
also generalizing. The ideal is an ability to use multiple representations
from symbolic equations and move to verbal descriptions. Also, students
should be able to make a calculation that others can use for different
purposes.
AMCOA team members made the following comments:
EW: So, you want students to understand the logic behind the equation.
- 10 -
Yves Salomon-Fernandez (YSF): It is also important to know the limitations of
the equation, how it can be knocked down. It is important to know the types
of analyses that can be done. You may forget the specific formula, but you
will remember the process of how to get the answer.
PM: The goal is integrated learning versus siloed learning.
TC: The goal is empowering community college students as opposed to
making them compliant.
CC: I am very involved in articulation of the K-12 Program and PARCC.
Twelfth grade math level is very aspirational, calculating residuals off linear
data to make inferences. There is a standard about model construction, a
model to plug in different data, reasoning and thinking about what they are
doing with their QR. They are constructing math models to be applied to
other questions.
G. Dawne Spangler, Director of the Center for Teaching, Learning and
Assessment, North Shore Community College
Dawne expressed her concerns about QR as follows: How do we define
QR? Does it mean taking a college Algebra course? We’ve come away from
that, but not far enough. We can’t get K-12 to do this unless the colleges
value it. It must come from the colleges, and then the lower grades will
comply.
The QR courses don’t transfer, so even if students need them, they don’t
take them, because they do not transfer to UMass. We need to look at
“holes” in understanding and fill in those holes. Where do you assess QR? If
you go to “sterilized” math classes, you won’t get good outcomes. Math must
be taught across the disciplines.
- 11 -
You have to decide where QR resides, so that it trickles down to K-12.
Who owns QR? Dawne has taught math for Elementary teachers. Math is a
study of structure. We need to get to quantitative understanding. According
to Grant Wiggins students who are given problems without contexts typically
fail. Students who are given context-based problems do well. Our
assessments will disappoint us if we don’t know how to put students in
contexts for solving them.
Concluding remarks:
PM: We need to continue having these discussions. Thank you to the panelists
for speaking and sharing your campuses’ experiences with QR assessment.
IV. Planning for AY 2012-2013
A. Overall Plans for Next Year: Pat Crosson
During the past year we have made great strides: we have formed the
AMCOA Team; held meetings and conferences with all the schools in the
Massachusetts public higher education system; shared best practices on
assessment methodologies; and become a LEAP state in February. The
Massachusetts initiative will be rooted in the WGSLOA goal of doing
assessment better than a single test; basing it on campus work and taking us
to a statewide reporting system. There have been a few modifications from
the WGSLOA to LEAP state, but they are minor. For example, we dropped the
composite working model, and focused on a basic statewide assessment plan.
Regardless of Davis Funding, we will continue this work next year. We will
name people to task forces to represent campuses to do this work. Three
campuses have done so already. The task force will provide overall guidance
for the work. Other groups doing thinking and planning are two smaller
teams: 1) the Massachusetts Team and 2) The State Partner Team that hopes
- 12 -
to find another set of states with similar goals to work with. A lot of states
are interested in partnering with Massachusetts. There will be co-chairs for
these task forces.
Pat stated that the AMCOA Team will go forward. We would like to see
this kind of collaboration continue across campuses. Campuses have said that
they still support AMCOA, so the work can continue.
AMCOA team members raised the following questions:
Will being on the task force exclude any of us from being on one of the
other two teams, other groups or committees? No. Pat said that she hoped
groups could straddle each other. There will be two teams peopled with
individuals who are willing to engage the challenging questions.
When will we know about the second Davis Grant? Pat stated we should
hear sometime by the end of May.
How many state partners might there be? Counting state partners, there will
be 14 states attending the conference at Boulder, Colorado. They will talk
about partnering with Massachusetts at that time. There is initial interest,
and we will know more about it at the end of May. The state partner work
will follow that.
Pat distributed a grid that describes how AMCOA will go forward in the
future. Phase II continues the good assessment work on campuses, sharing
ideas and collaborating. Phase I created a collaboration that is very
successful.
The following comments were made about AMCOA’s work this year:
EW: It worked well. We shared ideas and worked together.
PM: We still have the challenge of how to bring word back to campuses.
People in AMCOA and attendees learned a lot this year.
- 13 -
PC: We have an ongoing obligation to expand the involvement on the
campuses. AMCOA needs to bring word back to the schools.
PM: Next year we will be learning from each other. We will be moving to
working meetings and conferences — conferences that focus on the three
outcomes we eventually will report on statewide, including curricula design,
assignment design, and scoring of student work. The more we engage people
in these activities, the more comfortable people will become. Peggy asked
that people send her examples of exemplary assignments for QR, writing, and
CT so that we can use these at our conferences to help others develop
effective assignments. We also need models of student work that reflect their
levels of achievement in responding to assignments.
EM: I would like to brainstorm on what essential outcomes a work sample
exemplifies in its depth and breadth.
B. Days, Dates, and Times for Next Year’s Four AMCOA Meetings (two in the
fall and two in the spring) Based on Results of Survey Monkey and Plans
for Next Year
After some discussion, the Team agreed that the dates for next year’s four
AMCOA Meetings will be:
Thursday, September 20, 2012
Wednesday, November 7, 2012
Monday, March 11, 2013
Friday, May 3, 2013
- 14 -
C. Days, Dates, and Times of Next Year’s Three Statewide Conferences (one
in the fall and two in the spring) Based on Results of Survey Monkey and
Plans for Next Year
The Team agreed that the dates for next year’s three AMCOA conferences
will be:
Friday, October 19, 2012
Thursday, February 28, 2013 (now changed to March 1 because of a
scheduling conflict)
Wednesday, April 24, 2013
V. Summary of Results
A. Questionnaire about Topics/Foci for Assessment that AMCOA Team
Members Have Identified to Support Our Campuses: Bonnie Orcutt
Bonnie reported that there is interest in having an AMCOA Support Team
with on-call help teams. What would that commitment entail? There
would have to be a limit on the number of calls. Also, there would have to
be an application process demonstrating why institutions would bring you
onto their campuses to advise them. We would also have to consider the
institutional impact. Only four responses were sent in. To encourage
more volunteers, Bonnie pointed out that it doesn’t have to be a
permanent commitment. Also, some leads are looking for teammates to
work with them. Pat Crosson responded that she assumed there probably
wouldn’t be a huge demand from campuses, yet there should be a lead
person from AMCOA who understands the nature of the need and
matches it with talent and the campus needs.
- 15 -
Peggy stated that there will be several committees established under
AMCOA next year, such as a committee to determine how NSSE/CCSSE
might be used to contribute data about students’ learning of QR, CT and
writing; a committee to work with a DHE staff person to develop a
repository focused on assessment; and a committee to establish criteria
for a new round of assessment experiments focused on identifying or
designing a web-based reporting system for student outcomes.
B. March Scoring of Critical Thinking (CT): Peggy Maki
The scoring results of each group in our round of Critical Thinking (CT) are
posted on Yammer. There were issues about assignments in every single
group. “Does the assignment really ask students to demonstrate critical
thinking?” was a common group question. What are representative
assignments that ask students to exemplify writing and CT? We need to
gather examples of them to use in our work and in our conferences.
- 16 -
Appendix A: Institutions Represented at the AMCOA May 1st Meeting:
Berkshire Community College
Bunker Hill Community College
Cape Cod Community College
Fitchburg State University
Framingham State University
Greenfield Community College
Holyoke Community College
Massachusetts Bay Community College
Massachusetts Maritime Academy
Massasoit Community College
Middlesex Community College
Mount Wachusett Community College
North Shore Community College
Northern Essex Community College
Roxbury Community College
Salem State University
Springfield Technical Community College
University of Massachusetts Boston
University of Massachusetts Dartmouth
University of Massachusetts Lowell
University of Massachusetts President’s Office
Westfield State University
Worcester State University
- 17 -
Appendix B: Mark Pawlak’s PowerPoint Presentation on Quantitative
Reasoning Efforts at UMass Boston
[Please double click on the image below to open the presentation and then click once to
move from one slide to the next.]
- 18 -