assessment report: dcc islo 1: oral communication ... · islo #1: oral communication – assessment...

ISLO #1: ORAL COMMUNICATION – ASSESSMENT SUMMARY REPORT FORM

of 12

Assessment Report: DCC ISLO 1: Oral Communication

--- January 2016 ---

Semester and Year:

Fall 2015

State program and/or ISLO informed by information in this report:

DCC Oral Communication Institutional Learning Outcome

List any student learning outcome(s) and level (institution, program, course) addressed by this

report, state the program if PSLO:

This report summarizes the assessment of the DCC Oral Communication

Institutional Learning Outcome, as defined below:

Oral Communication: Students will demonstrate oral communication skills in a clear

and organized manner using appropriate verbal and nonverbal communication

techniques with regard to subject, purpose and audience.

List Courses from which Assessment Data is Gathered:

ARC103: Basic Architectural Drawing (3 sections, 1 instructor)

AVI110: Aviation Law (1 section, 1 instructor)

BUS107: Principles of Marketing (3 sections, 1 instructor)

BUS210: Business Communication (1 section, 1 instructor)

CIS212: Systems Analysis and Design (1 section, 1 instructor)

CRJ201: Criminal Justice Organization and Administration (2 sections, 2 instructors)

CRJ266: Contemporary Issues & Problems in Criminal Justice (1 section, 1 instructor)

ECH254: Diverse Early Childhood / Elementary Classrooms (2 sections, 1 instructor)

ELT218: Electronics II (1 section, 1 instructor)

EMB101: Emergency Medical Technician Clinical (1 section, 1 instructor)

ENR101: Introduction to Engineering (2 sections, 1 instructor)

FIR100: Fire Science Intro Seminar (1 section, 1 instructor)

NUR215: Parent-Child Nursing (2 sections 1 instructor)

PAL120: Legal Research (1 section, 1 instructor)

PFA100: Performing Arts Intro Seminar (2 sections, 2 instructors)

PSY102: Interviewing/Counseling Skills (5 sections, 5 instructors)

SCI100: Science Introductory Seminar (1 section, 1 instructor)

SPA201: Intermediate Spanish I (1 section, 1 instructor)

SPA202: Intermediate Spanish II (1 section, 1 instructor)

SPE101: Public Speaking (9 sections, 2 instructors)

THE105: Theatre History I (2 sections, 1 instructor)

THE120: Performing Skills for the Classroom (2 sections, 1 instructor)


of 12

List Participating Faculty and Academic Department:

Michael Adams, PVAC

Leah Akins, ENACT

Christopher Brellochs, PVAC

Thomas Costello, PVAC

Mark Courtney, ENACT

Sarita Green-Pesante, BHS*

Frank Kolarik, BHS

Margeaux Lippman-Hoskins, PVAC

Maryann Longhi, BUS

Erich Markert, AHBS

Ornella Mazzuca, ENG

Joan McFadden, BUS

James Palmatier, AHBS*

Paul Pilon, ENACT

Kathleen Rafferty, BHS*

Treesa Scaria, NUR

Christopher St. Germain, BHS*

Craig Stokes, ENG

John Troise, ENACT

Dan Valentine, BHS

Mareve VanVoorhis, BHS

Maritza Vasquez Reyes, BHS*

Dave Walsh, AHBS

Frank Whittle, ENACT

Scott Willmen, BUS

Margaret Woodcock, BHS

* indicates part-time faculty member

List number of sections:

45

List number of students:

859 students enrolled in the sample classes; 755 students with valid data


of 12

1. State the specific question or questions asked:

Can DCC employ a single oral communication rubric for use across all academic

disciplines? Can data from assessment activities using this rubric be used to make the

teaching of Oral Communication more effective at DCC?

What information can we gather about how proficient DCC students are in regards to

oral communication skills? How effectively do students who have taken at least one

course requiring formal oral communication meet the DCC ISLO standards for oral

communication in regards to organization, language, delivery, supporting material,

and central message?

What area(s) can we target or develop to help DCC students develop better Oral

Communication skills?

2. Describe the method used to answer the question(s).

Program chairs identified courses in their program that support the development of

competence in oral communication.

Where there was more than one course that could be assessed, program chairs

selected one of the courses for assessment in Fall 2015.

Departmental faculty responsible for the identified courses were consulted to make

sure that there was agreement about participating in a Fall 2015 assessment of this

ISLO in the identified course.

o In cases where departmental faculty provided valid reasons to not perform that

assessment in the identified course in Fall 2015, an alternative course was

identified.

Faculty teaching the courses in the finalized list of courses to be assessed in Fall 2015

for oral communication met for a 3.5 hour workshop to decide on the parameters for

the assessment. Workshop summary:

o Faculty chose to adopt the existing Association of American Colleges and

Universities (AACU) “Oral Communication VALUE Rubric” as the basis for

the Fall 2015 assessment of DCC’s Oral Communication ISLO.

o Faculty decided that for the purposes of assessing oral communication at

DCC, only prepared presentations or interviews would be assessed.

Casual participation in classroom discussion was deemed not sufficient

enough to warrant assessment in regards to DCC’s oral

communication ISLO.

o Faculty agreed that the rubric should be applied to specific presentation and/or

interview assignments, preferably later in the term.

o Faculty would assess students work with the agreed upon assessment rubric

independent of any pre-existing grading rubrics that were in use in classes.

Thus making it theoretically possible that a student could earn a high

grade, while earning low marks on the assessment, or vice versa.

o Participating faculty would then enter their rubric data into a consolidated

spreadsheet per course, and then send those spreadsheets, along with a

description of the assignment to Thomas Costello, who would collate the data.

o Faculty would then reconvened in early spring 2016 to examine the data and

draw conclusions about DCC’s effectiveness in regards to the ISLO.


of 12

3. Summarize the results.

Overall, the assessment data suggests that approximately 80% of the sampled

DCC students are meeting the goals of DCC’s Oral Communication ISLO,

according to the rubric.

Total # of Students: 755

Organization Language Delivery

Supporting

Material

Central

Message

% earn 4 32% 20% 25% 28% 29%

% earn 3 47% 59% 50% 52% 53%

% earn 2 19% 20% 24% 17% 16%

% earn 1 2% 1% 2% 3% 2%

o Within this data set, we can see that “Delivery” is the weakest area.

79% of students meet or exceed AACU goals for organization

79% of students meet or exceed AACU goals for language

74% of students meet or exceed AACU goals for delivery

80% of students meet or exceed AACU goals for supporting material

81% of students meet or exceed AACU goals for central message

There was an overall significant difference (p < .001) between Learning

Objectives regarding % of students meeting expectations (scores of 3 or 4). The

measures that differed significantly were as follows:

o Delivery (74.0%) was lower than Central Message (82.0%), p < .001

o Delivery (74.0%) was lower than Supporting Material (80.0%), p < .01

o Delivery (74.0%) was lower than Organization (79.1%), p < .05

There were significant relationships between Course Type and…

o Organization such that the % of students meeting expectations was lower in

100-level courses (76.6%) than in 200-level courses (86.4%), p < .01

o Supporting Material such that the % of students meeting expectations was

lower in 100-level courses (76.4%) than in 200-level courses (90.6%), p <

.001

o Central Message such that the % of students meeting expectations was lower

in 100-level courses (78.9%) than in 200-level courses (91.1%), p < .001

There were significant relationships between academic program and all the

Learning Objectives (ps < .05), indicating that % of students meeting

expectations differed by program.1

1 This data is will be available for program chairs to interpret and, if appropriate, act upon.


of 12

None of the relationships between FT/PT Status and the Learning Outcomes

were significant. (In other words, % students meeting expectations didn’t differ

significantly between FT students and PT students.)

FT/PT # of

Students

Organization

% earn 4 % earn 3 % earn 2 % earn 1

FT 611 30% 48% 20% 2%

PT 144 37% 47% 16% 1%

FT/PT # of

Students

Language


FT 611 19% 60% 20% 1%

PT 144 23% 56% 21% 1%

FT/PT # of

Students

Delivery


FT 611 24% 49% 25% 2%

PT 144 26% 51% 22% 1%

FT/PT # of

Students

Supporting Material


FT 611 27% 53% 18% 2%

PT 144 34% 49% 15% 3%

FT/PT # of

Students

Central Message


FT 611 27% 55% 16% 2%

PT 144 37% 46% 17% 1%


of 12

There was a significant relationship between Student Type (New, Transfer,

Continuing, High School) and Central Message (p = .01). The % of students

meeting expectations was lower among new students (74.9%) and transfer students

(72.0%) than among continuing students (84.6%) and high school students (100%). o NOTE: only 4 “early admit” students were sampled.

Student

Type

# of

Students

Organization


New 179 26% 51% 20% 3%

Continuing 547 33% 46% 19% 1%

Transfer 25 28% 52% 16% 4%

Early Admit 4 75% 25% 0% 0%

Student

Type

# of

Students

Language


New 179 16% 61% 22% 1%

Continuing 547 21% 58% 20% 1%

Transfer 25 20% 60% 20% 0%

Early Admit 4 25% 75% 0% 0%

Student

Type

# of

Students

Delivery


New 179 23% 48% 25% 4%

Continuing 547 25% 49% 24% 1%

Transfer 25 12% 64% 24% 0%

Early Admit 4 50% 50% 0% 0%

Student

Type

# of

Students

Supporting Material


New 179 24% 52% 21% 3%

Continuing 547 29% 52% 17% 2%

Transfer 25 32% 48% 16% 4%

Early Admit 4 25% 75% 0% 0%

Student

Type

# of

Students

Central Message


New 179 22% 53% 21% 4%

Continuing 547 31% 54% 14% 1%

Transfer 25 28% 44% 24% 4%

Early Admit 4 75% 25% 0% 0%


of 12

1. Summarize conclusions drawn and action plan for improvement.

a. Recommendations for improvement on an ISLO should be included in an action plan

managed and coordinated by the associate dean of academic affairs for assessment and

reported on in the six-year cycle for ISLOs.

b. Recommendations for improvement on a PSLO should be included in an action plan

managed and coordinated by the program chair and reported on in the program review.

c. Recommendations for improvement that pertains solely to a course are used to improve

the teaching and learning in that course and are the responsibility of faculty teaching

that course. These should be periodically discussed and considered by the department

head, discussed at the department level, and included in an action plan provided with the

department’s annual report.

Primary Conclusions

Can DCC employ a single oral communication rubric for use across all academic

disciplines? Can data from assessment activities using this rubric be used to make the

teaching of Oral Communication more effective at DCC?

Faculty were largely very happy with the Oral Communication rubric.

o A majority of the participating faculty chose to share the rubric with their

students, and many reported that the rubric helped to clarify for students what

it means to be an effective speaker.

o Many faculty members found that the rubric can or does dovetail nicely with

their own grading schema, and in future terms, their own grading rubrics may

become more refined and clear, thanks to the ISLO rubric.

Implementation of the Oral Communication rubric revealed while the rubric is

sound, inter-rater reliability remains a major concern.

o Only a small fraction of DCC faculty have specialized training, or advanced

degrees, in the field of Oral Communication. Consequently, the data

demonstrated noticeable subjectivity across ratings; for instance, Speech

faculty were more critical of their Public Speaking students, while those

students instruction was the most directly related to the ISLO.

Finally, what is Oral Communication at DCC?

o A wide variety of courses and assignments were included in this year’s ISLO

assessment for Oral Communication.

o Courses were identified only by individual faculty and program chairs, who

checked a box to indicate that the course includes elements of Oral

Communication, and should be assessed as such.

o At present there are no standards or requirements as to what “Oral

Communication” courses at DCC should include.


of 12

What information can we gather about how proficient DCC students are in regards to oral

communication skills? How effectively do students who have taken at least one course

requiring formal oral communication meet the DCC ISLO standards for oral

communication in regards to organization, language, delivery, supporting material, and

central message?

Approximately 80% of the DCC students sampled are meeting or exceeding

ISLO goals.

o This number is likely slightly inflated due to the non-inclusion of students

whose data was not captured due to absence, failure to do the assignment, etc.

A significant deficiency was discovered with regard to the Oral Communication

sub-category of “Delivery.” While students averaged 80% proficiency fairly evenly

across the categories of Organization, Language, Supporting Material, and Central

Message, students only averaged 74% proficient in Delivery. Upon analysis, this

deficit was deemed statistically highly significant, and faculty suggested that the

deficiency was likely due to a combination of factors:

o Training. Many non-Speech faculty are not trained to teach Oral

Communication delivery skills, nor do most teach it as a specific facet of Oral

Communication in their classes.

The other areas are more closely aligned with discipline-specific

student learning outcomes, while “delivery” often is not. (e.g. In a

given class, the organizational strategy and language might be dictated

by the subject, and the supporting material and central message are

often requirements of the assignment.)

Practice, Class Size, and Time. Success in delivery is directly

correlated with the amount of time a student spends delivering

speeches. Even in a relatively small class of 20 students, a basic 5-

minute presentation, with a 3 minutes for questions and 2 minutes for

turnover will use 3 hours and 20 minutes of class meeting time; that’s

more than a week of most classes.

Faculty expressed that time was a major known issue regarding

Oral Communication, saying that it can be difficult to teach the

target material while also allowing ample time for Oral

Communication skills to develop.

The larger the class size, the more difficult it is to incorporate

meaningful instruction and practice in Oral Communication.

Public Speaking is a skill that can be taught.

Several (non-Speech) faculty mentioned the clear advantage

exhibited in presentations by students who have taken a public

speaking course. A course dedicated to the craft of Oral


of 12

Communication invariably affords students with better Oral

Communication skills than a course with different, or many,

pedagogical aims.

Some faculty expressed concerns that their classes cannot

adequately cover both the required target material and Oral

Communication skills.

Technology. There was large agreement that in an increasingly

mediated society, students are and will be arriving at DCC with very

poor Oral Communication skills. Thus, this ISLO is increasingly

important, and will only become more vital to the success of our

students in the future.


of 12

Action Plan The following suggestions are based on the data gathered and the discussion that took place in

the January 2016 ISLO faculty workshop. These suggestions are organized broadly around the

third, and final, question that this report sought to answer:

What area(s) can we target or develop to help DCC students develop better Oral

Communication skills?

o Delivery

This assessment project brought out that DCC students are significantly

behind in the Oral Communication learning outcome of “delivery.”

Consequently, the suggestions below have been identified to both enhance

DCC’s Oral Communication ISLO, as well as to specifically help faculty and

students address delivery in Oral Communication.

o Time

Teaching Load

Reduce teaching load to allow faculty more time to work individually

with their students, as individual critique is vital to the development of

Oral Communication skills.

Create time annually, or each semester, for faculty across disciplines to

learn more about effective Oral Communication pedagogy, both as a

distinct area of study, and within their own discipline(s).

Class Size

Limit class size in Oral Communication courses. Teaching Oral

Communication effectively takes time, and class size has a direct

impact on how rigorously a course can include Oral Communication.

o Resources

Offer Oral Communication support for both DCC faculty and students.

Many colleges feature a Speaking Center on campus, similar to a

Writing Center or Math Center, where students (and faculty) could go

to develop their skills, work on class assignments, and receive direct

and specific instruction to advance their oral communication skills.

Training:

Create time annually, or each semester, for faculty across disciplines to

learn more about effective Oral Communication pedagogy. This is

especially important if faculty without formal education in Oral

Communication are expected to teach it.


of 12

In the Classroom

Address Oral Communication head-on, as a discipline to be taught,

even within classes focused on other target material.

Increase students exposure to effective orators: Bring in engaging

speakers, and/or travel to have students experience exceptional

examples of Oral Communication.

Consider integrating the Oral Communication rubric into course and

assignment development and/or grading practices, where appropriate.

Encourage use of effective speaking examples, such as TED talks, in

classwork across disciplines.

Employ technology such as video cameras and/or tablets to record

student presentations, and offer feedback using the video evidence.

o ISLO Assessment

What is Oral Communication at DCC?

Classes:

o Develop a clear set of guidelines that explain what is expected

of courses flagged as “Oral Communication” courses in

fulfilment of the DCC ISLO.

o Consider a tiered approach, where specific courses that teach

foundational concepts of Oral Communication are identified as

distinct from those that refine or practice Oral Communication

skills.

Assignments:

o Decide upon the level of rigor that is expected in Oral

Communication assignments.

Inter-Rater Reliability

Consider instituting a program that would train faculty teaching Oral

Communication courses how to reliably assess student presentations,

so that we can garner more significant data.

Other Suggestions for Future Assessment

Include/Harvest data on course order and previous courses each

student has taken, so that we can begin to assess longitudinal

development data.

Integrate statisticians into the assessment process from the start.

Program Level Assessment

The spreadsheet that accompanies this report will help program chairs

determine the extent to which their programs either do, or do not,

require courses that sufficiently teach Oral Communication skills.


of 12

Recommendations for resources needed to implement action plan.

The following recommendations are based directly on the above suggestions.

Support the creation of a Speaking Center on campus.

Reduce faculty teaching load.

Limit class size in Oral Communication classes.

Provide time and training annually for faculty required to teach Oral Communication.

Provide technology to help faculty teach Oral Communication.

Provide specific training for those tasked with assessing Oral Communication.

Support faculty in efforts to expose students to outstanding orators through guest

speakers/lecturers, interdisciplinary events, field trips, digital media, etc.

Foster a working relationship between Student Activities and faculty to bring exceptional

orators to campus to teach students about effective Oral Communication through first-

hand experience.

assessment report: dcc islo 1: oral communication ... · islo #1: oral communication – assessment...

Documents