ewp readings report fall 2005 report ay15 draft.doc · web viewthe reading guide asks readers to...

Electronic Writing Portfolio Readings Report AY 2015

Number of Readers: 17Number of Portfolios: 208Number of Papers: 616

Methodology

In Fall 2005, volunteers were solicited from the faculty and staff. Readers were required to be full-time at Eastern, to have completed a master’s degree or higher, and to teach at least one undergraduate course each year. Twenty-three readers were trained at a 1 ½-day workshop by Dr. Robert O’Brien-Hokanson, Associate Professor of English and Coordinator of the Communication Ability Department at Alverno College. Readers came from all four colleges, the School of Continuing Education, the Center for Academic Support and Achievement, and Booth Library. A second training was held in October 2008, a third in October 2010, and a fourth in January 2014; Dr. Karla Sanders delivered these trainings based on the work completed by Dr. O’Brien-Hokanson.

The following 2015 readers were trained at one of these sessions; they represent all four colleges:

Leann Akins, English, CAH, 2005Jay Bickford, Early Childhood, Elementary, and Middle Level Education, CEPS, 2010Diane Burns, Geology/Geography, COS, 2014Jonelle DePetro, Philosophy, CAH, 2005Crystal Duncan-Lane, Family & Consumer Sciences, LCBAS, 2014Bill Feltt, English, CAH, 2008Kathryn Fenton, Music, CAH, 2014Rose Gong, Secondary Education & Foundations, CEPS, 2005Angie Jacobs, Communication Studies, CAH, 2008Nicole Mulvey, Communication Disorders & Sciences, COS, 2014Mary Caroline Simpson, Art, CAH, 2014Jeff Snell, Management, LCBAS, 2005Tim Taylor, English, CAH, 2014Gordon Tucker, Biological Sciences, COS, 2005Traci Worby, Kinesiology & Sports Studies, CEPS, 2010Marjorie Worthington, English, CAH, 2014Diana Wyatt, School of Technology, LCBAS, 2008

Readers were urged to evaluate writing patterns across the portfolios rather than focusing on each individual document submitted to the portfolios. The reading guide asks readers to provide an assessment of writing ability for completed portfolios across seven aspects of writing: focus/purpose, organization, development, audience, style, mechanics, and use of sources. Readers were also asked to assess each portfolio overall. They complete a reading guide for each portfolio.

A sample of 10% of the completed portfolios are read each year; readers were given a month to read their portfolios, were given access to a secure web site containing the documents and the students’ account of the writing assignment, and were assigned to read 12 or 13 portfolios that were chosen at random from the completed portfolios. After reading their set of portfolios, readers were asked to complete a reader’s observation sheet and to attend one of three focus groups held at the beginning of March to discuss student writing as displayed in the portfolios.

The qualitative data that follow represent the discussions at the focus groups as well as information taken from the readers’ guides and observation sheets; all information has been collated and summarized by the

EWP Portfolio Readings Report, AY15, p. 1Prepared March 2015 by Karla Sanders, Center for Academic Support & Assessment

Executive Director of the Center for Academic Support and Assessment. A draft of this report was disseminated via email for readers’ comments. Changes to the report were made based on faculty feedback. The percentages given for each assessment of the various areas of writing are taken from a compilation of scores given by the readers for each portfolio, not each document.

The following data are divided into the categories assessed by the readers. Each section gives the readers’ impressions of the portfolios as a whole and the final section offers potential uses for the data in terms of improving the curriculum/pedagogy. Where appropriate, readers’ written and verbal comments have been quoted to support the general conclusions that have been drawn. Percentages refer to the percentage of portfolios that were rated in the categories described.

Portfolios Overall Strong Portfolios: 26%Adequate Portfolios: 56%Weak Portfolios: 17%

The increase in weak portfolios seen in the reading evaluations in AY14 was not the beginning of a new trend; this year’s portfolios had only 17% of portfolios rated as weak overall. Strong portfolios were about a quarter of those read while over half were deemed “adequate” by the faculty readers. Readers indicated that students’ writing communicated information but did not do so in a pleasing or elegant manner. Little evidence of revision and polishing was displayed in the samples read. “Most students are not putting the effort in to make the writing more interesting or engaging. It is just an assignment and that’s it.”

The weakest areas remain those aspects of writing related to critical thinking and developing an argument. Appropriate citation of sources remains an issue as well as asking students to think critically and analytically. One reader summed up these issues, which were described by many readers:

I found a couple of portfolios quite troubling because there were papers that showed research but did not properly paraphrase and/or cite sources—unintentional plagiarism. One trend that I noticed is that students appear to be assigned informative research papers, which make them work with sources. Often though, the papers were mostly data dumping (providing quotations and paraphrases) and very little of students’ writing. I think learning new information is important (part of what college is about); however, many of the papers I read didn’t seem to ask students to move to the next levels in terms of critical thinking—detailed analysis, sound critique, evaluation, and argument.

Organization and focus were primarily adequate with some portfolios displaying strong skills. While papers are only required to be 750 words in order to be submitted to the EWP, some readers found that shorter papers did not allow for development. “I had only 2 papers that were 22 and 25 pages long, and they were the best papers. They developed everything, had headings, and they stayed on focus. Then, my 3-page papers weren’t that good.” Some readers speculated that longer papers might have encouraged


students to spend more time on revision, so they would naturally be stronger papers because of the time and care that students would need to put into them. Some readers that no papers in their set of portfolios were longer than 5-6 pages.

The sections that follow discuss the strengths and weaknesses readers found in portfolios by writing trait followed by a section on ways to improve writing through pedagogy or curriculum changes.

Focus/Purpose

Strong (Consistently strong sense of focus/purpose throughout): 30%Adequate (Clear focus/purpose in most or all submissions): 46%Weak (Some evidence of ability to focus on a purpose): 21%Poor (Very little or no evidence of focus): 2%

The percentage of portfolios rated as “strong” in focus grew by 6% while the percentage deemed “adequate” grew by 1% and those rated “weak” dropped by 7%. The number of portfolios in the “poor” category is the most consistent. Readers indicated that staying on task (having a focus) was one of the stronger areas whereas creating a sense of purpose in one’s writing and working toward that purpose was more difficult and less apparent than simply staying with their topic.

Readers indicated that many portfolios were missing writing conventions that help guide the reader and maintain focus on a topic or argument or create a sense of purpose. These missing conventions include thesis statements, topics sentences, and transitions combined with weak aspects such as introductions and conclusions. “They had a general focus. But, if you wanted something specific like a thesis statement that they were trying to develop and really come to a point, most of mine weren’t getting to that level of focus.”

Several readers commented that students seemed to show the process of finding their purpose in the introduction rather than starting strong with a clear reason for writing to a particular audience. Assignments with specific genres with particular conventions were strongest in this category. “I had a number of proposals and business analysis and anthropology papers that were really good. They tended to have a really strong focus and purpose because of the genre they had to work within. The weaker ones tended to be, you get what you ask for, and a number of papers were just data dumping.” Many readers indicated that focus was fine (identification of the topic), but there were more significant problems with purpose (why I am writing this paper).

Organization

Strong (Consistent use of structure in ways that enhance presentation of ideas/information): 22%Adequate (Logical organization and/or clearly identifiable structure): 51%Weak (Inconsistent sense of structure and/or lapses in organization): 26%Poor (Very little or no sense of structure or organization): 1%


Little change was seen this year in the number of portfolios rated in each category related to organization. The percentage deemed “strong” rose by 3% while those evaluated at “adequate” increased by 2% and the “weak” portfolios dropped by 4%. As with the other categories, the number of portfolios rated “poor” remains very consistent.

As indicated by the percentages given in the chart above, readers felt that organization was a skill students had mastered at the “adequate” mark. Papers could be followed; students rarely went off on tangents, but very few provided well-developed, thoughtful introductions and conclusions.

In addition, transitions that could have guided the reader in terms of purpose and enabled a cogent organization were largely absent. “There were clear beginnings, middles, and ends, and I think they were able to somehow introduce their idea even though it wasn’t extraordinarily strong, and they were able to conclude that idea. Transitions were a little weaker sometimes; they were repetitive and sometimes they didn’t exist at all. It was just, ‘oh, here is another idea, and here is my second idea, and here’s my third idea.’ So, organization was very basic at best.” Several readers remarked on the lack of transitions and other markers to move pieces along or make comparisons and contrasts apparent. Most samples had simple organizational patterns, and many adopted an organization from an article or book being reported on. “The ones that did well [with organization] used subheadings to help with organization. Others did well in the beginning and end and sort of got lost in the middle. I had meandering.” Readers indicated that paragraphs did not build on each other. “So many of them you could cut the paragraphs out and re-arrange it and it wouldn’t have made a difference in their essay.” Students also included repetition of ideas and did not seem to revise for clarity and ease of following for the reader.

Development

Strong (Ideas consistently developed in depth and supported with rich and relevant details): 17%Adequate (Ideas developed in depth with appropriate supporting evidence/details): 42%Weak (Some development of ideas and use of supporting evidence/details): 31%Poor (Very little or no development of ideas or use of supporting evidence): 5%


Development remains one of the weakest traits in student writing as displayed by the portfolios although the percentage rated as strong did increase by 6% this year. Portfolios rated “adequate” in development dropped 7% from AY14 and 12% from AY13. Portfolios deemed weak rose to 38% showing a 7% increase from the previous year, while those rated “poor” portfolios remained the same.

With 42% of portfolios given poor or weak ratings, this writing skill is the most problematic as displayed in the papers submitted to the EWP. “They wrote with clarity but did not write with complexity. I think it had to do with tasks. Most of them were summarizing a book or writing about their feelings. There really wasn’t much depth,” one reader noted. Readers indicated that students either did this very well or had issues, and more seemed to struggle with development than not. Students did not provide integrated evidence, the readers found. Some of the strongest examples were those in which student followed the scientific method, which forces analysis.

Since many of the assignments asked students for their own opinions or reflections, few asked for an in-depth analysis or development of a side of an argument. One reader explained, most papers “were personal opinions. ‘I feel this way, I don’t feel this way, but I like this.’ Or, ‘Let me tell you my life story,’ and they weren’t really developing an idea. Many portfolios didn’t even use sources and the ones that did use sources, it was abstract. ‘This source said this, and this source said that.’ But, there was never a bringing of these things together. They weren’t developing ideas beyond just stating facts.”

Readers indicated that students tended to throw in sources and not explain who the quote was from or the sources relevance for the topic at hand. They did not guide the reader nor try to use evidence to prove a point and use of details was scarce. “There were a lot of quotes and paraphrasing but not a lot of their writing. With the weaker papers there was a lot of data and just informational research papers and not much of the writer there at all. Like a patchwork quilt of other peoples’ stuff.” Several readers concurred with this characterization. Another called it, “the inability to analyze the data and information.” Many concurred.

Some readers found students using data that did not support their points or ran contrary to their argument. Readers felt that few papers ran the gamut of Bloom’s taxonomy, and it was rare to encounter students attempting evaluation and not many had attempted analysis.

Audience

Strong (Sophisticated sense of audience—e.g., distinctive voice and/or appropriate tone): 23%Adequate (Some awareness of and/or attempt to communicate with audience): 58%Weak (Little or no awareness of audience): 16%Poor (No sense of writing for an audience): 3%

With each reading, faculty have indicated that it is often difficult to discern any reader other than the instructor for the course unless the genre of the paper demanded that an audience be identified, such as in a letter or report for a company. “If students are going to have to write in their professions and in their


civic lives, course could have assignments that mimic rhetorical situations in people’s possible professions and other situations,” suggests one reader.

Some writers, however, showed a good sense of audience by defining terminology while other writers seemed to be writing only for themselves. Readers speculated that an increase in reflective writing could have lead some writers to be more focused on themselves than on an external audience.

Readers indicated that most students seemed to have their professor as their target audience and seemed aware of this audience. Readers suggested that assignments could be created that “replicate professional jobs or realistic writing situations they might be in.” Some students seemed to address the professor or allude to a class discussion that the portfolio reader did not know about. “Some of the papers just picked up in mid-conversation, like they had talked about something in class. Then, they were referencing that thing without naming that thing.”

Overall, portfolio readers felt that students were writing to their faculty members because that is what they were asked to do in their assignments, so asking students to show a sense of writing to other audiences would require changes to the assignments being given.

Style

Strong (Sophisticated use of language (sentence structure, word choice) that enhances presentation of ideas/information): 16%Adequate (Appropriate use of language that effectively conveys ideas/information): 58%Weak (Use of language that is awkward, unnecessarily complex, and/or overly simplistic): 24%Poor (Use of language that is highly inconsistent or indeterminate): 2%

In the eight years of reading completed portfolios, few portfolios are ever rated “strong” in regards to style. This remains the weakest category in terms of the number of portfolios rated as strong. The large majority were “adequate” at 58%. A slight drop in the percentage rated as weak was seen with nearly a quarter of the portfolios falling into this category. The percentage deemed “poor” remains very consistent. “We are seeing 8th grade writing at a college level.”

Readers describe the majority of papers as using simple sentence construction and unsophisticated, simplistic diction. Students’ styles do not display a strong sense of vocabulary nor do they display strong revision skills. Many papers contained awkward phrasing or choppiness as well as a preponderance of filler words and phrases.

In addition, readers indicated that most of the student samples displayed an informal style rather than formal academic writing. Many noted that portfolios with strong styles also seemed to be well-edited, but informality was likely to lead to many errors in mechanics as well.


Mechanics

Strong (Few, if any, errors in mechanics relative to length and complexity of documents): 27%Adequate (Some errors in mechanics that do not interfere significantly with communication): 50%Weak (Patterns of errors in mechanics that affect clarity and/or credibility of writing): 20%Poor (Large numbers of errors in mechanics affecting almost all aspects of the writing): 2%

Readers indicated that most portfolios contained some mechanical errors, but the majority of papers were not incomprehensible due to these errors. The percentage of portfolios rated as “strong” in mechanics rose from 21% in 2014 to 27% in 2015 while the percentage described as “adequate” only dropped by 1%. Those assessed as “weak” and “poor” also remained relatively stable from the previous year.

All readers felt that students needed to do a better job at proofreading and revising rather than simply relying on spellcheck. Many readers felt that the submissions were first drafts and had not been significantly revised since many errors were careless mistakes that could have been fixed through careful proofreading and editing. Few student writings exhibited knowledge of when to use colons and semi-colons.

Readers indicated that few students seemed to improve in this area with mechanical problems repeating across the three papers in each portfolio. Comma usage was a common problem as was leaving periods off of the end of sentences. “We all need to focus on the basic grammatical and syntactical structure of writing professionally. Students are apparently not coming into college with these skills,” noted one reader. Another common problem was misuse of homonyms, such as “coarse” and “course.” Some of the choppiness discussed in the style section of this report indicated use of sentence fragments and ideas stopping short of being complete. Some students also struggled with knowing which verb tense to choose, so they went back and forth between tenses in a given paper.

Sources

Strong (Ability to integrate ideas/information from sources into own writing in meaningful and appropriate ways): 28%Adequate (Some effective integration of ideas/information from source): 55%Weak (Inappropriate/ineffective integration of ideas/information): 16%


Of the 208 portfolios read, 182 portfolios (88%) included at least one paper incorporating outside sources compared to 94% in AY12. The portfolios assessed as “strong” remained the same as 2014 while those rated as “adequate” increased by 12%, and the weak portfolios related to sources decreased by 11%.

Readers indicated that students did not seem to make discerning judgements related to their sources nor did they often explain to the reader the authority of their sources. Students seem to understand that evidence is needed to support what they are saying, but integrating that evidence or attesting to its efficacy was often lacking. “I saw a lot of hit and run quoting. ‘Here’s quote, see ya’ later.’”

Many readers suggested that students tended to use sources easily discovered on the internet rather than delving into the collection at Booth Library for the best evidence to support their ideas. One reader noted that “finding an old-fashioned research paper was kind of scarce.”

Using Data to Develop Curriculum & Improve Pedagogy

After reading all of their portfolios, readers were asked to indicate ways in which the information gleaned from the readings could be used to develop curriculum and/or improve pedagogy. Many readers are quoted in this section to preserve the intent of their suggestions. The predominant theme involved encouraging faculty to make changes in writing requirements/assignments and course curriculum through sharing EWP data from the readers. Readers felt that workshops guiding faculty on how to structure assignments would be useful to increase the critical thinking, development, and use of sources and to encourage revision. “I think a lot of faculty think, ‘Ok, I am a writer and I publish, so I understand writing,’ but they don’t teach writing, so they develop assignments that are all content-oriented, showing they have a certain mastery. The writing component is never really addressed in the instructions.” This theme appeared throughout the focus groups: the importance of being intentional about teaching writing and critical thinking across the curriculum and grading students based on writing elements as well as content. “I believe more focus (in the classroom) should be on teaching students how to develop an argument and then to effectively use sources to support claims they make within those arguments. This requires instructors to incorporate critical thinking skills into assignments.” Readers indicated that faculty need to give a certain number of points to the way in which papers are written in addition to content in order to impress upon students the value of revising and correcting mistakes. “If an instructor places weight in the area of grammatical structure by enlisting grammar as an area of evaluation for papers, students will channel their attention to their grammatical structure of their writing.”

Readers suggested that good student writing hinges on the assignments given as does the inclusion of critical thinking. Very few papers in the samples were making an argument and using critical thinking skills; the majority of samples were informative or reflective/personal narratives. “There were only maybe 1-3 argument-based papers. I find that trend troubling since the undergraduate learning goal of “critical thinking” coheres pretty well with argument-based writing. From my perspective, the papers that were stronger seemed to reflect clear guidelines for the assignment, such as IMRD papers and papers that critically evaluated others’ ideas.”


Since development was the weakest of the seven traits evaluated, many of the comments related to ways to improve these skills. “Students seem to think that they need to say everything they possibly can about a topic, without evaluating what they’re saying or prioritizing the information. Showing them how to reason through a set of facts and to draw reasonable conclusions from those facts in class and in small assignments might really help.” Several readers found correlations between lack of development and basic writing skills. “Evidence from these portfolios demonstrate that students need to be guided in basic writing skills, idea development, and how to better support their arguments. A majority of these portfolios lack sophisticated thought, as well as writing skill. It seems there is a focus on EITHER critical thinking OR strengthening writing skills.” Several readers across the colleges indicated that in their own classes they use examples of good papers and poor papers to show students their expectations and to discuss revision.

Paper length did seem to factor into development and the time students put into a paper. “Faculty need to be encouraged to have their students complete writing assignments and to complete longer writing assignments. The majority of my submissions were 3-5 pages, which does not always allow the students to fully develop their ideas.” While some readers did have some longer research papers, most indicated that longer, research-driven papers were rare in the portfolios. All readers agreed that evidence of drafting and revision were largely absent in the majority of portfolios.


ewp readings report fall 2005 report ay15 draft.doc · web viewthe reading guide asks readers to...

Documents