wasc arc conf final, iswi, april 8 2011.pptx
DESCRIPTION
TRANSCRIPT
![Page 1: Wasc arc conf final, iswi, april 8 2011.pptx](https://reader031.vdocument.in/reader031/viewer/2022022404/5463376ab1af9f936c8b536d/html5/thumbnails/1.jpg)
WASC ARC ConferenceApril 8, 2011, San Francisco, CA
Improving Student Writing and Strengthening Writing Programs – Technology and Techniques That Work
Ellen Junn, Associate Provost
Jennifer Ivie, Assistant Professor of PsychologyKim Morin, Professor of Theatre Arts
William Covino, ProvostCalifornia State University, Fresno
![Page 2: Wasc arc conf final, iswi, april 8 2011.pptx](https://reader031.vdocument.in/reader031/viewer/2022022404/5463376ab1af9f936c8b536d/html5/thumbnails/2.jpg)
California State University, Fresno (est. 1911)
•23-campus CSU system
•20,932 students
•1,197 faculty
2Copyright © April 2011
![Page 3: Wasc arc conf final, iswi, april 8 2011.pptx](https://reader031.vdocument.in/reader031/viewer/2022022404/5463376ab1af9f936c8b536d/html5/thumbnails/3.jpg)
Fresno State Demographics
3Copyright © April 2011
![Page 4: Wasc arc conf final, iswi, april 8 2011.pptx](https://reader031.vdocument.in/reader031/viewer/2022022404/5463376ab1af9f936c8b536d/html5/thumbnails/4.jpg)
Fresno State Demographics
§ First generation college: 68%
§ English proficiency:
63% freshmen require English remediation
4Copyright © April 2011
![Page 5: Wasc arc conf final, iswi, april 8 2011.pptx](https://reader031.vdocument.in/reader031/viewer/2022022404/5463376ab1af9f936c8b536d/html5/thumbnails/5.jpg)
Multi-pronged Approach to Teaching Writing on our Campus
¨ English Composition ¨ Upper Division Writing Exam
¡ Graduation Requirement¡ Substitution of Designated “W” courses
¨ Writing Requirement for ALL General Education Coursesú Requires “Iterative” writing assignments
5Copyright © April 2011
![Page 6: Wasc arc conf final, iswi, april 8 2011.pptx](https://reader031.vdocument.in/reader031/viewer/2022022404/5463376ab1af9f936c8b536d/html5/thumbnails/6.jpg)
The Dilemma
¨ High percentage of remedial students. ¨ Perception that surface errors distract
from content. ¨ Effective writing instruction requires
innovative pedagogies.
6Copyright © April 2011
![Page 7: Wasc arc conf final, iswi, april 8 2011.pptx](https://reader031.vdocument.in/reader031/viewer/2022022404/5463376ab1af9f936c8b536d/html5/thumbnails/7.jpg)
INSTRUCTORS KEEP ASKING…
“Where Is the Time to Respond To All of Those
Essays?”
![Page 8: Wasc arc conf final, iswi, april 8 2011.pptx](https://reader031.vdocument.in/reader031/viewer/2022022404/5463376ab1af9f936c8b536d/html5/thumbnails/8.jpg)
TECHNOLOGY AS ONE SOLUTION?
•Professor Kim Morin
•“E-scholar” Program
•Upper Division online GE courses
•Faculty interest across campus
![Page 9: Wasc arc conf final, iswi, april 8 2011.pptx](https://reader031.vdocument.in/reader031/viewer/2022022404/5463376ab1af9f936c8b536d/html5/thumbnails/9.jpg)
A DISCOVERY…Computer Essay Scoring Programs
Several College-level Essay Scoring and Writing programs available…
![Page 10: Wasc arc conf final, iswi, april 8 2011.pptx](https://reader031.vdocument.in/reader031/viewer/2022022404/5463376ab1af9f936c8b536d/html5/thumbnails/10.jpg)
COMPUTER ESSAY SCORING PROGRAMS
• ETS® Criterion®• http://www.ets.org/criterion/higher_ed/about • IEA Intelligent Essay Assessor
http://www.knowledge-technologies.com/prodIEA.shtml • SAGrader• https://www.sagrader.com/sgm/features• Pearson MyWriting Lab• http://www.mywritinglab.com/whatis.html
![Page 11: Wasc arc conf final, iswi, april 8 2011.pptx](https://reader031.vdocument.in/reader031/viewer/2022022404/5463376ab1af9f936c8b536d/html5/thumbnails/11.jpg)
ETS CRITERION® SELECTED ON A TRIAL BASIS…
First Trial (Spring 2009): 30 students – 1 essay assignment
First Pilot (Fall 2009)1 Faculty member100 students- 5 essay assignments
![Page 12: Wasc arc conf final, iswi, april 8 2011.pptx](https://reader031.vdocument.in/reader031/viewer/2022022404/5463376ab1af9f936c8b536d/html5/thumbnails/12.jpg)
About ETS Criterion®• Students write and revise essays
online• Diagnostic Feedback / Holistic Score
within 20 seconds.• Topics Library provides prompts.• Faculty can create topics/prompts. • Errors are highlighted but not
corrected.
12Copyright © April 2011
![Page 13: Wasc arc conf final, iswi, april 8 2011.pptx](https://reader031.vdocument.in/reader031/viewer/2022022404/5463376ab1af9f936c8b536d/html5/thumbnails/13.jpg)
ABOUT ETS CRITERION®
• No instructor Fees• Simple Registration
• Students purchase Access codes through Bookstore
• Approximately $11.00 per student • One fee provides student use in all classes per
term/semester• Technical Support from ETS®
![Page 14: Wasc arc conf final, iswi, april 8 2011.pptx](https://reader031.vdocument.in/reader031/viewer/2022022404/5463376ab1af9f936c8b536d/html5/thumbnails/14.jpg)
ABOUT ETS CRITERION®
• Bilingual feedback available • Spanish, Japanese, Simplified
Chinese, Korean, ELL
• Advanced levels available • College–1st & 2nd year, TOEFL, GRE
• No prior essays required
![Page 15: Wasc arc conf final, iswi, april 8 2011.pptx](https://reader031.vdocument.in/reader031/viewer/2022022404/5463376ab1af9f936c8b536d/html5/thumbnails/15.jpg)
ETS CRITERION® Online Tour
http://www.ets.org/Media/Products/Criterion/tour2/critloader.html
![Page 16: Wasc arc conf final, iswi, april 8 2011.pptx](https://reader031.vdocument.in/reader031/viewer/2022022404/5463376ab1af9f936c8b536d/html5/thumbnails/16.jpg)
Initial Presentation to Faculty
“Computer Essay Scoring has had a positive effect so far.
I spend more time assessing content, less on grammatical errors.
Students spend more time revising.However, it still misses errors and does not
grade for content.”
16Copyright © April 2011
![Page 17: Wasc arc conf final, iswi, april 8 2011.pptx](https://reader031.vdocument.in/reader031/viewer/2022022404/5463376ab1af9f936c8b536d/html5/thumbnails/17.jpg)
Early Student Comments
¡ “I like the instant feedback.” ¡ “I like that it goes into depth about why you
didn't get the max score or why you did well.” ¡ “It is easy to use - just copy and paste!”
¡ “I love the fact that I can revise my work for a
better score.” ¡ “It is very helpful and I can see my growth as a
writer.”
Copyright © April 2011 17
![Page 18: Wasc arc conf final, iswi, april 8 2011.pptx](https://reader031.vdocument.in/reader031/viewer/2022022404/5463376ab1af9f936c8b536d/html5/thumbnails/18.jpg)
Facts About College Student Writing
FACT # 1: Student improvements in writing increase with more
practice ¨ specific, constructive feedback from faculty ¨ opportunity for revisions. IMPLICATION: ü Students need to write or revise more frequentlyü Examine policy requiring “iterative” writing in GE or
W courses
18Copyright © April 2011
![Page 19: Wasc arc conf final, iswi, april 8 2011.pptx](https://reader031.vdocument.in/reader031/viewer/2022022404/5463376ab1af9f936c8b536d/html5/thumbnails/19.jpg)
FACT # 2: If students do not continue to practice, their writing performance may actually deteriorate. IMPLICATION: ü Examine all writing programs across campusü Identify key courses with writing requirementü Target specific faculty teaching those courses
19Copyright © April 2011
![Page 20: Wasc arc conf final, iswi, april 8 2011.pptx](https://reader031.vdocument.in/reader031/viewer/2022022404/5463376ab1af9f936c8b536d/html5/thumbnails/20.jpg)
FACT # 3: Teaching students to write effectively can be time-consuming and labor-intensive. IMPLICATION: ü Identify effective technology-related tools ü reduce faculty workloadü provide specific, timely feedback to students.
ü Target part-time faculty who teach writing intensive courses.
20Copyright © April 2011
![Page 21: Wasc arc conf final, iswi, april 8 2011.pptx](https://reader031.vdocument.in/reader031/viewer/2022022404/5463376ab1af9f936c8b536d/html5/thumbnails/21.jpg)
FACT # 4: Writing instruction involves faculty who are NOT
trained as writing teachers.
IMPLICATION: ü Implement Criterion training for faculty.ü Offer Writing Across the Curriculum workshops.ü Provide professional development funds for all
faculty during Year 1 Pilot. 21Copyright © April 2011
![Page 22: Wasc arc conf final, iswi, april 8 2011.pptx](https://reader031.vdocument.in/reader031/viewer/2022022404/5463376ab1af9f936c8b536d/html5/thumbnails/22.jpg)
FACT # 5: Assessing & documenting student learning outcomes in
writing performance are key elements to success. IMPLICATION: ü Ongoing data collected for past 3 semestersü Share results with faculty learning community to
determine Best Practices.ü Modify training based on feedback and assessment.
22Copyright © April 2011
![Page 23: Wasc arc conf final, iswi, april 8 2011.pptx](https://reader031.vdocument.in/reader031/viewer/2022022404/5463376ab1af9f936c8b536d/html5/thumbnails/23.jpg)
Next Came…
ISWI Pilot Launch
Spring 201023Copyright © April 2011
![Page 24: Wasc arc conf final, iswi, april 8 2011.pptx](https://reader031.vdocument.in/reader031/viewer/2022022404/5463376ab1af9f936c8b536d/html5/thumbnails/24.jpg)
ISWI Improving Student Writing
InitiativeNovember, 2009 - Campus email announcement
calling for faculty participants § Criterion® Training:§ Faculty ISWI Coordinator§ Faculty Learning Community (FLC)
§ Writing Across the Curriculum Workshops§ Assessment of Criterion®
24Copyright © April 2011
![Page 25: Wasc arc conf final, iswi, april 8 2011.pptx](https://reader031.vdocument.in/reader031/viewer/2022022404/5463376ab1af9f936c8b536d/html5/thumbnails/25.jpg)
Campus-Wide InitiativeBy The Numbers... Year 1 (2 Semesters)
Spring 2010 / Fall 2010
• 349 Classes Involved • 173 Instructors Used Criterion®
• 134 Spring 2010• 68 Fall 2010
• 5,920 Students (Spring 2010)• 3,756 Students (Fall 2010)
Numbers may include duplicates
25Copyright © April 2011
![Page 26: Wasc arc conf final, iswi, april 8 2011.pptx](https://reader031.vdocument.in/reader031/viewer/2022022404/5463376ab1af9f936c8b536d/html5/thumbnails/26.jpg)
Campus-Wide InitiativeBy The Numbers...
Year 2 (1 Semester) Spring 2011
• 231 Classes Involved • 93 Instructors Used Criterion® • 5442 Students (Spring 2011)• 44,080 Essays Submitted (by March 15)
Numbers may include duplicates
26Copyright © April 2011
![Page 27: Wasc arc conf final, iswi, april 8 2011.pptx](https://reader031.vdocument.in/reader031/viewer/2022022404/5463376ab1af9f936c8b536d/html5/thumbnails/27.jpg)
Faculty Participation- Year 2
Out of 93 instructors opting to use Criterion, only 20 received Professional Development funds as an incentive in Year 2.
27Copyright © April 2011
![Page 28: Wasc arc conf final, iswi, april 8 2011.pptx](https://reader031.vdocument.in/reader031/viewer/2022022404/5463376ab1af9f936c8b536d/html5/thumbnails/28.jpg)
Assessments from Spring 2010..
1. Analyzed Criterion® automatically generated data for trends.
2. CLA scores for students with & without
Criterion® 3. First and last papers submitted by students
on Criterion® scored by independent faculty panel
4. Collected student and faculty surveys
28Copyright © April 2011
![Page 29: Wasc arc conf final, iswi, april 8 2011.pptx](https://reader031.vdocument.in/reader031/viewer/2022022404/5463376ab1af9f936c8b536d/html5/thumbnails/29.jpg)
Automatically-Generated Data
¡ Students who used Criterion® for revision with more than one submission, scored better than those who did not.
¡ Students with a larger number of
assignments and more submissions on Criterion® increased their holistic score on average by approximately 1 level.
29Copyright © April 2011
![Page 30: Wasc arc conf final, iswi, april 8 2011.pptx](https://reader031.vdocument.in/reader031/viewer/2022022404/5463376ab1af9f936c8b536d/html5/thumbnails/30.jpg)
Collegiate Learning Assessment (CLA)
¡ A significant difference was found between Criterion® users and non-users on the CLA performance task.
¡ No significant differences were found
between the two groups on the analytic writing task.
30Copyright © April 2011
![Page 31: Wasc arc conf final, iswi, april 8 2011.pptx](https://reader031.vdocument.in/reader031/viewer/2022022404/5463376ab1af9f936c8b536d/html5/thumbnails/31.jpg)
Faculty Scoring Panel
First and last papers submitted by students on Criterion® scored by independent faculty panel.
¨ Overall, student writing improved in
classes that used ETS Criterion®.¡ The greatest improvements were made when
instructors engaged students in substantial discussions of writing and how to use Criterion®.
31Copyright © April 2011
![Page 32: Wasc arc conf final, iswi, april 8 2011.pptx](https://reader031.vdocument.in/reader031/viewer/2022022404/5463376ab1af9f936c8b536d/html5/thumbnails/32.jpg)
Faculty Feedback – Positive Comments
When using Criterion®, most faculty felt that:1. Papers were easier to grade.2. The program improved their students’
writing skills.3. Students spent more time revising written
assignments.4. Creating assignments was easy. 32Copyright © April 2011
![Page 33: Wasc arc conf final, iswi, april 8 2011.pptx](https://reader031.vdocument.in/reader031/viewer/2022022404/5463376ab1af9f936c8b536d/html5/thumbnails/33.jpg)
Faculty Feedback - Challenges
Many faculty felt that Criterion®
1. Did not reduce their workload.2. Did not do as much as they had
hoped.3. Was not worth the cost to the
students.
33Copyright © April 2011
![Page 34: Wasc arc conf final, iswi, april 8 2011.pptx](https://reader031.vdocument.in/reader031/viewer/2022022404/5463376ab1af9f936c8b536d/html5/thumbnails/34.jpg)
Student Survey Response
Copyright © April 2011 34
![Page 35: Wasc arc conf final, iswi, april 8 2011.pptx](https://reader031.vdocument.in/reader031/viewer/2022022404/5463376ab1af9f936c8b536d/html5/thumbnails/35.jpg)
Student Survey Highlights
What students liked about Criterion®: 1. It was available on-line 24 hours a day. 2. It allowed them to correct grammatical
or mechanical errors before turning in a paper.
3. It gave immediate feedback.
35Copyright © April 2011
![Page 36: Wasc arc conf final, iswi, april 8 2011.pptx](https://reader031.vdocument.in/reader031/viewer/2022022404/5463376ab1af9f936c8b536d/html5/thumbnails/36.jpg)
What students found challenging:
1. Criterion® identified technical terms or
citations as errors. 2. The program identified errors but did
not correct them. 1. The program did not grade for content.
36Copyright © April 2011
![Page 37: Wasc arc conf final, iswi, april 8 2011.pptx](https://reader031.vdocument.in/reader031/viewer/2022022404/5463376ab1af9f936c8b536d/html5/thumbnails/37.jpg)
Student Survey Response
Freshmen/sophomores were significantly more likely to agree:
- Criterion has helped me improve my writing.
- Criterion should suggest less and correct more.
- I was able to apply what I learned to other writing assignments.
Copyright © April 2011 37
![Page 38: Wasc arc conf final, iswi, april 8 2011.pptx](https://reader031.vdocument.in/reader031/viewer/2022022404/5463376ab1af9f936c8b536d/html5/thumbnails/38.jpg)
ESL Student Survey Response
Copyright © April 2011 38
![Page 39: Wasc arc conf final, iswi, april 8 2011.pptx](https://reader031.vdocument.in/reader031/viewer/2022022404/5463376ab1af9f936c8b536d/html5/thumbnails/39.jpg)
English as a Second Language Student Responses
ESL students were more likely to respond that Criterion® helped them improve their writingúPreferred Criterion® feedback to instructor feedback.úSubmitted their documents more frequently.
- I like the kind of feedback Criterion® gives. - I spent more time improving my writing with Criterion®. - I wish I could use Criterion® for other classes.
39Copyright © April 2011
![Page 40: Wasc arc conf final, iswi, april 8 2011.pptx](https://reader031.vdocument.in/reader031/viewer/2022022404/5463376ab1af9f936c8b536d/html5/thumbnails/40.jpg)
English as a Second Language Student Response
Copyright © April 2011
http://www.youtube.com/watch?v=pP6feJotLVM
40
![Page 41: Wasc arc conf final, iswi, april 8 2011.pptx](https://reader031.vdocument.in/reader031/viewer/2022022404/5463376ab1af9f936c8b536d/html5/thumbnails/41.jpg)
Positive Comments From Students
¨ “Awesome program. I find it really useful.”¨ “I like that it is really convenient and I
found myself using the program for all my classes.”
¨ “I started noticing themes in my writing that could use improvement.”
41Copyright © April 2011
![Page 42: Wasc arc conf final, iswi, april 8 2011.pptx](https://reader031.vdocument.in/reader031/viewer/2022022404/5463376ab1af9f936c8b536d/html5/thumbnails/42.jpg)
Ongoing Research
¨ Studies with Control Groups¨ Continue Collecting Data¨ Determine Best Practices / Uses
Copyright © April 2011 42
![Page 43: Wasc arc conf final, iswi, april 8 2011.pptx](https://reader031.vdocument.in/reader031/viewer/2022022404/5463376ab1af9f936c8b536d/html5/thumbnails/43.jpg)
Conclusion
So Far, Evidence suggests- ¨ Criterion® helps improve student writing
when combined with effective instructor practice.
¨ Criterion® appears useful for editing grammar and mechanics.
¨ Students who use Criterion® are more engaged with the writing process.
43Copyright © April 2011
![Page 44: Wasc arc conf final, iswi, april 8 2011.pptx](https://reader031.vdocument.in/reader031/viewer/2022022404/5463376ab1af9f936c8b536d/html5/thumbnails/44.jpg)
Next Steps
¨ Analyze and refine assessment data.¨ Provide online video tutorials.¨ Expand Professional Development &
training.¨ Establish ISWI committee
ú analyze writing instruction across campus.¨ Recognize and thank participating faculty.
44Copyright © April 2011
![Page 45: Wasc arc conf final, iswi, april 8 2011.pptx](https://reader031.vdocument.in/reader031/viewer/2022022404/5463376ab1af9f936c8b536d/html5/thumbnails/45.jpg)
What Have We Learned? ¨ Students generally found Criterion® to be easy
to use and cost effective.¨ ESL students responded more positively to
Criterion® feedback.¨ Faculty found Criterion® helpful when used to
complement instruction.¨ ISWI shows the high degree of faculty interest
in improving student writing across campus.
45Copyright © April 2011
![Page 46: Wasc arc conf final, iswi, april 8 2011.pptx](https://reader031.vdocument.in/reader031/viewer/2022022404/5463376ab1af9f936c8b536d/html5/thumbnails/46.jpg)
ISWI Broadly Endorsed & Supported By:¨ Writing Competency Subcommittee (subcommittee of Senate’s GE Committee)¨ Professional Development Subcommittee (subcommittee of Senate’s Personnel
Committee)¨ Chair of the GE Committee¨ Office of Undergraduate Studies¨ Division of Graduate Studies¨ Institutional Research & Assessment Planning Director leading faculty ISWI Assessment
Team¨ Provost’s Office as implemented by the Associate Provost through the Center for the
Scholarly Advancement of Learning and Teaching (CSALT) & Technology Innovations for Learning and Teaching (TILT)
46Copyright © April 2011
![Page 47: Wasc arc conf final, iswi, april 8 2011.pptx](https://reader031.vdocument.in/reader031/viewer/2022022404/5463376ab1af9f936c8b536d/html5/thumbnails/47.jpg)
Thank You!
¨ William Covino, Provost¨ [email protected]¨ 559-278-2636¨ ¨ Ellen Junn, Associate Provost¨ Interim Director, Center for the
Scholarly Advancement of Learning and Teaching (CSALT)
¨ Interim Senior Academic Technology Officer (SATO) and Technology Innovations for Learning and Teaching (TILT)
¨ [email protected]¨ 559-278-2636
¨ Kim Morin, Professor ¨ Department of Theatre Arts¨ Artistic Director, Theatre for
Young Audiences; English/Drama Credential Advisor
¨ ISWI Faculty Coordinator¨ [email protected] ¨ 559-278-4342¨ ¨ Jennifer Ivie, Assistant Professor ¨ Department of Psychology¨ Interim Director, Center for
the Scholarly Advancement of Learning and Teaching (CSALT)
¨ [email protected] ¨ 559-278-2842
Copyright © April 2011 47
![Page 48: Wasc arc conf final, iswi, april 8 2011.pptx](https://reader031.vdocument.in/reader031/viewer/2022022404/5463376ab1af9f936c8b536d/html5/thumbnails/48.jpg)
Contact Information- ETS®
ETS® Criterion®¨ http://www.ets.org/criterion
¨ Susan L. Yetman¨ Criterion Account Manager¨ Higher Education, ETS ¨ Rosedale Road MS 51-L¨ Princeton, NJ 08541¨ Direct line: 609.683.2675¨ Toll free: 866.717.1915¨ Fax: 609.683.204
0¨ Email: [email protected]
Copyright © April 2011 48
Arthur RuzzanoWestern Regional ManagerHigher Education Assessment SolutionsPhone: 310.944.4034Fax: 609.683.2040Email: [email protected] Bill WynneProduct Manager - ETS Proficiency ProfileETS Programs and Services DivisionTelephone (609) 683-2006E-mail [email protected]