tesol 2010: assessing academic bridge courses powerpoint presentation
DESCRIPTION
Assessing Academic Bridge Courses TESOL 2010--Boston, MA March 25, 2010 Jean M. Czaja & Brooke HaleyTRANSCRIPT
Assessing Academic Assessing Academic Bridge Bridge CoursesCourses
Jean M. Czaja & Brooke HaleyMarquette University
TESOL 2010—Boston, MAMarch 25, 2010
162B Boston Convention Center 3-3:45pm
Presentation OutlinePresentation Outline
Our Context Our Mandate & Timeline Marquette University’s General Assessment Process Steps for Designing Our Assessment Plan Examples of How We are Carrying Out Assessment in 2 of
Our Courses◦ How they are embedded into the courses
◦ sample assessment measures and rubrics
Data & Conclusions from Our First Assessment Cycle Actions We are Taking to Improve Students’ Learning Advice for Assessing Academic Bridge Courses
Our ContextOur Context
• Marquette University is a private Jesuit university located in Milwaukee, WI.
• We have a student body of approximately 8,000 undergraduate students and 3,500 post-graduate students.
• Of this total, about 500 (4.35%) are international. Approximately 150 (1.88%) are undergraduates, coming from 43 total countries.
• To support these undergraduates, we offer 4 advanced ESL academic bridge courses and an ESL section of Freshman English, which students take for academic credit:
• ESLP 82: Composition• ESLP 83: Reading• ESLP 84: Listening Comprehension• ESLP 182: Rhetoric and Composition for Non-Native Speakers• ENGL 1 (ESLP section): Rhetoric and Composition
• In AY 2009-2010, about 75% of the new international undergraduate students (54) took our English Placement Test and enrolled in our ESL academic bridge courses.
Our Mandate & TimelineOur Mandate & Timeline
• Marquette is accredited through the North Central Association of Colleges and Schools.
• As part of its accreditation, since Fall 2004 Marquette has been engaged in an initiative to improve:
(1) its overall assessment of students’ learning (2) the “culture of assessment” at Marquette
• The ESL Program first entered the assessment process in Fall 2006 when it created and listed specific course learning objectives on course syllabuses.
• Then in Spring 2008, the ESL Program wrote an assessment proposal & devised initial learning outcomes, performance indicators, and measures.
• In AY 2008-2009, the ESL Program carried out its first year of assessment, which we will be reporting on.
Three Levels of Marquette’s Assessment Three Levels of Marquette’s Assessment ProcessProcess
ESL ProgramInformation adapted from Marquette assessment literature
Marquette’s Assessment ProcessMarquette’s Assessment Process
At Marquette, student learning consists of 4 steps:
Feedback for continuous
improvement
Information adapted from the Marquette assessment website
Steps for Designing Our Assessment Steps for Designing Our Assessment PlanPlan
We devised our assessment proposal and plan for the university by:
1. familiarizing ourselves with the assessment terms & process used at Marquette
2. meeting with Foreign Language and Study Abroad representatives and examining their learning outcomes, indicators, and measures
3. speaking with ESL professionals, reviewing ESL assessment literature, and attending sessions at TESOL Assessment information is available for K-12, adult education,
and IEPs No guidelines for learning objectives & assessments of
university-level ESL courses
4. devising ESL Program learning outcomes, performance indicators, and measures for the university assessment committee
5. improving existing and creating new assessment specifications and course assessments, which are mapped to performance indicators
6. improving and creating rubrics for the course assessments
Our Assessment PlanOur Assessment Plan
MARQUETTE UNIVERSITY
UNDERGRADUATE CO-CURRICULAR
LEARNING OUTCOMES
English as a Second Language Program (ESLP)
At the completion of his/her ESL course(s), the student is able to:
1. Write using academic English
2. Read academic English texts
3. Listen to and speak using academic English
4. Use grammar, punctuation, and mechanics in academic writing
Program Learning Outcomes
Performance Indicators Measures Use of the Information
1. Write using academic English
Produce a research paper Employ components of the writing process Use strategies to address writing problems
Final research paper in ESLP 82 Pre and post diagnostic tests Pre and post self-assessment of
skills and use of learning strategies
ESLP faculty will collect, analyze, aggregate data, and present a report to the Director and staff of the Office of International Education. The information will be discussed and commented upon by the staff and faculty for the purpose of advancement of students’ proficiency and program evaluation.
2. Read academic English texts
Comprehend literal and implied meanings of academic texts
Determine meaning of new vocabulary Evaluate and respond to ideas in academic
texts Employ components of the reading process Read under time constraints Use strategies to address reading problems
Comprehensive final exam in ESLP 83, which directly measures learning outcomes
Pre and post diagnostic tests Pre and post self-assessment of
skills and use of learning strategies
Same as above
3. Listen to and speak using academic English
Employ note-taking techniques to take notes on and understand academic spoken texts
Participate in everyday interactions role-plays
Participate in small group discussions Deliver oral presentation on news broadcast Use strategies to address listening and
speaking problems
Comprehensive final exam in ESLP 84 which directly measures learning outcomes
Final (oral) presentation from last unit
Pre and post diagnostic tests Pre and post self-assessment of
skills and use of learning strategies
Same as above
4. Use grammar, punctuation, and mechanics in academic writing
Understand conventions of standard U.S. English grammar, punctuation, and mechanics
Produce a research paper using conventions of standard U.S. English grammar, punctuation, and mechanics
Use strategies to address grammar, punctuation, and mechanics problems
Revision of ENGL 1 Paper #1 at the end of ESLP 182
Pre and post diagnostic tests Pre and post self-assessment of
skills and use of learning strategies
Same as above
How We are Carrying Out Assessment in How We are Carrying Out Assessment in
ESLP 82: WritingESLP 82: Writing
Program Learning Outcome 1: Writing using academic English
Performance Indicators Measures
Produce a research paper
Employ components of the writing process
Use strategies to address writing problems
Final research paper in ESLP 82
Pre and post diagnostic tests
Pre and post self-assessment of skills and use of learning strategies
How We are Carrying Out Assessment in How We are Carrying Out Assessment in
ESLP 182: Rhet/Com for Non-Native ESLP 182: Rhet/Com for Non-Native SpeakersSpeakers
Program Learning Outcome 4: Use grammar, punctuation, and mechanics in academic writing
Performance Indicators Measures
Understand conventions of standard U.S. English grammar, punctuation, and mechanics
Produce a research paper using conventions of standard U.S. English grammar, punctuation, and mechanics
Use strategies to address grammar, punctuation, and mechanics problems
Revision of ENGL 1 Paper #1 at the end of ESLP 182
Pre and post diagnostic tests
Pre and post self-assessment of skills and use of learning strategies
Data from First Assessment Cycle Data from First Assessment Cycle
Program Learning Outcome 1: Write using academic English
Measures Used to Assess the Outcome: Final Research Paper in ESLP 82
Total Number of Students Assessed: 12
Program Learning Outcomes
Performance Indicators
Developing(CD, D, F)
Competent(B, BC, C)
Proficient(A, AB)
1. Write using academic English
Produce a research paper
0 2 10
Employ components of the writing process
2 7 3
Use strategies to address writing problems
0 7 5
Data from First Assessment Cycle Data from First Assessment Cycle (cntd.)(cntd.)
Program Learning Outcome 4: Use grammar, punctuation, and mechanics in academic writing
Measures Used to Assess the Outcome: Revision of ENGL 1 Paper #1 at the end of ESLP 182 and 100-point diagnostic test given at the end of the semester
Total Number of Students Assessed: 23 and 22 respectively
Program Learning Outcomes
Performance Indicators
Developing(CD, D, F)
Competent(B, BC, C)
Proficient(A, AB)
4. Use grammar, punctuation, and mechanics in academic writing
Understand conventions of standard U.S. English grammar, punctuation, and mechanics
5 13 5
Produce a research paper using conventions of standard U.S. English grammar, punctuation, and mechanics
1 6 15
Use strategies to address grammar, punctuation, and mechanics problems
No data collected
No data collected
No data collected
Conclusions from First Assessment CycleConclusions from First Assessment Cycle
When we discussed the academic year 2008-2009 assessment results, we reached the following conclusions regarding student learning:
• The majority of the assessed students were ranked as competent or proficient in most categories.
• Some students did not meet the expectations, which makes us wonder why they did not make adequate progress and what actions we could have taken to help them to become competent or proficient.
• We feel that in general that most results represent an accurate assessment of students' learning. This was reinforced by students' performance in class and on other major assessments.
Actions We are Taking to Improve Actions We are Taking to Improve Students’ LearningStudents’ Learning
Action #1: We are strengthening course content in areas where we want more students to move from developing to competent on the performance indicators.
Action #2: We are determining what measures are really needed for the university assessment. We are fairly certain that in the future that diagnostic tests and questionnaires will NOT be reported to the university but rather used for:
giving students feedback on their skills encouraging students to reflect on their learning & how they can learn better improving our course planning—helping us to decide areas to emphasize
Action #3: We are reexamining and revising our assessment specifications, rubrics, and assessments to ensure that they measure students' actual proficiency.
#Action 4: We have made some measures carry more value towards students’ final grades so that students are motivated to do well on them and take them seriously.
Action #5: We have worked with the International Student and Scholar staff to adjust admissions guidelines so we do not admit students with too low of English proficiency to adequately progress in our courses and to succeed at Marquette.
Action #6: We are meeting with students who show problem behaviors and/or who receive low grades on their first assessments to give them advice and refer them to additional university resources.
Advice for Assessing Academic Advice for Assessing Academic Bridge CoursesBridge Courses
1. Give yourself enough time to plan and implement your assessment initiative.
2. Understand how assessment is carried out within your university and be sure that your plan will fit with the university’s plan (if applicable).
3. Create standardized course learning objectives and make sure that your staff actually emphasizes these areas when they teach these courses.
4. Devise assessment specifications and assessments that directly measure the course learning outcomes.
5. Create rubrics for all assessments to provide a standard way of assessing students’ performance.
6. Devise a standard procedure for collecting the evidence from students.
7. Set aside a specific time for your staff to analyze and honestly reflect on the evidence. Keep in mind the limitations of standardized assessments—they don’t tell the entire story.
Advice for Assessing Academic Advice for Assessing Academic Bridge Courses Bridge Courses (cntd.)(cntd.)
8. As a staff, decide on specific actions you will take to help improve students’ learning.
9. Embed the learning objectives, assessments, and rubrics into your
courses so teachers and students know clearly what they are learning and expected to be able to do.
10. Try to make the assessment initiative as simple, direct, and manageable as possible, and only gather as much evidence as needed.
Thank You & Electronic CopiesThank You & Electronic Copies
Thank you for coming to our presentation.
Please contact us for more information about assessment at Marquette. Visit http://www.marquette.edu/oie/eslp.shtml for a link to electronic copies of the handouts or other resources. Click on the “TESOL Presentations” link.