An Assessment Study of the An Assessment Study of the Communication “A” Communication “A”

Requirement at UW-MadisonRequirement at UW-Madison 2006-07 2006-07

University General Education Research GroupUniversity General Education Research Group

Chuck Halaby, L&SChuck Halaby, L&SNancy Westphal-Johnson, L&SNancy Westphal-Johnson, L&S

Jim Wollock, T&E/SOEJim Wollock, T&E/SOEElaine Klein, L&SElaine Klein, L&S

The Communication “A” The Communication “A” RequirementRequirement

ObjectiveObjective: Enhance communication : Enhance communication proficiency by exposing students to proficiency by exposing students to courses that:courses that:

– ““develop student abilities in writing and develop student abilities in writing and public speaking, for both exposition and public speaking, for both exposition and argumentation.”argumentation.”

– ““advance key skills” in writing, advance key skills” in writing, communication, and information communication, and information literacy.literacy.

Skill Objectives for Comm-A Skill Objectives for Comm-A CoursesCourses

PlanningPlanning Selecting topics, analyzing audience Selecting topics, analyzing audience info needs; generating/organizing ideas; info needs; generating/organizing ideas; comprehending and analyzing texts. comprehending and analyzing texts.

DraftingDrafting Learning exposition/argument Learning exposition/argument structures; developing/organizing paragraphs, structures; developing/organizing paragraphs, papers, speeches; adapting writing/speeches for papers, speeches; adapting writing/speeches for intended audience. Mastering grammar, usage, intended audience. Mastering grammar, usage, style; prepare speeches; citing sources, avoiding style; prepare speeches; citing sources, avoiding plagiarismplagiarism

RevisingRevising Critical skills for revising and editing Critical skills for revising and editing essays/speeches for spelling, punctuation, essays/speeches for spelling, punctuation, grammar, style, organization, logic. grammar, style, organization, logic.

Information-SeekingInformation-Seeking Technical Technical skills/strategies for identifying/retrieving skills/strategies for identifying/retrieving materials from print/electronic sources.materials from print/electronic sources.

Key Features of Comm-A StudyKey Features of Comm-A Study

Research QuestionResearch Question Does Comm-A provide Does Comm-A provide students with opportunity and experience needed students with opportunity and experience needed to enhance their communication skills?to enhance their communication skills?

MethodMethod Compare students who had and had Compare students who had and had not satisfied Comm-A with respect to self-reported not satisfied Comm-A with respect to self-reported gains in communication skills in key areasgains in communication skills in key areas

Target populationTarget population Freshmen who had not Freshmen who had not satisfied Comm-A at start of fall 2006-07 semestersatisfied Comm-A at start of fall 2006-07 semester

Sampling designSampling design Stratified random samplingStratified random sampling

– Comm-A sample: Comm-A sample: 374/480 (78% response rate)374/480 (78% response rate)– Control sample: Control sample: 141/160 (88% response rate) 141/160 (88% response rate)

Communication ScaleCommunication Scale

PreamblePreamble

““Please think about the courses you Please think about the courses you took at UW during fall semester. This took at UW during fall semester. This includes lectures, readings, discussion includes lectures, readings, discussion sections, and assignments, as well as sections, and assignments, as well as any other experiences you had as part any other experiences you had as part of these courses.”of these courses.”

QuestionQuestion

““How much did any of these courses How much did any of these courses teach you to do each of the following?teach you to do each of the following?

Communication Skill Scale ItemsCommunication Skill Scale Items

Deliver speech or oral presentation?Deliver speech or oral presentation?

RetrieveRetrieve and analyze information from and analyze information from library databases?library databases?

Select and focus topics for paper or Select and focus topics for paper or speech/presentation?speech/presentation?

Recognize logically sound arguments?Recognize logically sound arguments?

Use language clearly and appropriately?Use language clearly and appropriately?

Comm-A Scale Items Comm-A Scale Items (continued)(continued)

Support ideas in a paper or speech?Support ideas in a paper or speech?

Produce papers or speeches/presentation for a Produce papers or speeches/presentation for a specific audience?specific audience?

Judge the credibility and soundness of evidence?Judge the credibility and soundness of evidence?

Organize ideas for a paper or Organize ideas for a paper or speech/presentation?speech/presentation?

Improve your grammar, punctuation, and style? Improve your grammar, punctuation, and style?

Comm-A Scale Items Comm-A Scale Items (continued)(continued)

Cite sources and avoid plagiarismCite sources and avoid plagiarism

Use library databases to locate research Use library databases to locate research materials ?materials ?

Write/revise drafts of paper or Write/revise drafts of paper or speech/presentation? speech/presentation?

Critique speeches/presentations or papers? Critique speeches/presentations or papers?

Cronbach’s alpha-reliability = .91Cronbach’s alpha-reliability = .91

FindingsFindings

Mean Gain in Communication Mean Gain in Communication Proficiency, by Comm-A Status, Proficiency, by Comm-A Status,

Freshmen, 2006-07Freshmen, 2006-07

t-ratiot-ratio = 14.2; p-value=.000= 14.2; p-value=.000

3.10

2.40

0.70

0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

Comm--A Control difference

over

all

scal

e sc

ores

Mean Gain in Communication Proficiency, by Mean Gain in Communication Proficiency, by Comm-A Status and Sex, Freshmen 2006-07Comm-A Status and Sex, Freshmen 2006-07

3.10 3.11

2.41 2.39

0.69 0.72

0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

m ales fem ales

over

al sc

ale sc

ore

Com m -A

Control

difference

Comm-AComm-A Students Almost Three Times More Likely to Students Almost Three Times More Likely to Report Communication Skills Grew by “fair amount/a Report Communication Skills Grew by “fair amount/a

lot”lot”

62.8

22.7

0

10

20

30

40

50

60

70

Comm -A Control

per

cen

tag

e

Comm -A

Control

Chi-square=66; p-value=.000Chi-square=66; p-value=.000

Odds of Communication Skills Increasing by Odds of Communication Skills Increasing by “fair amount/a lot” More than Five Times “fair amount/a lot” More than Five Times

Greater for Comm-A Students Greater for Comm-A Students

t-ratio= 7.69; p-value=.000t-ratio= 7.69; p-value=.000

0.29

5.76

1.69

0

1

2

3

4

5

6

7

Comm-A Control Ratio

odds

of 3

/4 s

cale

sco

re

Item-Specific AnalysisItem-Specific Analysis

Item-Specific Scale Means, by Comm-A Item-Specific Scale Means, by Comm-A StatusStatus

2.27

2.35

1.77

1.92

1.87

3.13

3.34

2.87

3.05

3.22

0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50 4.00

Papers/speeches forspecific audience

Cite sources/avoidplagiarism

Deliver a speech

Retrieve/analyze databaseinfo

Use library databases forspecific purpose

Com m unication Scale Means

Com m -A

Control

All differences significant at < .000All differences significant at < .000

Item-Specific Scale Means, by Comm-A Item-Specific Scale Means, by Comm-A StatusStatus

2.91

2.66

2.66

2.11

2.50

3.33

3.24

3.28

2.91

3.35

0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50 4.00

Support ideas inpaper/speech

Select/focus topics for paper

Organize ideas forpaper/speech

Critique speeches/papers

Write/revise drafts

Communication Scale Means

Comm-A

Control

All differences significant at < .000All differences significant at < .000

Item-Specific Scale Means, by Comm-A Item-Specific Scale Means, by Comm-A StatusStatus

2.85

2.97

2.22

2.49

2.93

3.10

2.53

2.88

0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50

Recognize logical arguments

Use language clearly/appropriately

Improve grammar, punctuation,style*

Judge credibility of evidence**

Scal

e Ite

ms

Communication Scale Means

Comm-A

Control

**Significant at <.000**Significant at <.000

* Significant at <.001* Significant at <.001

Percent Responding Skills Grew by “fair Percent Responding Skills Grew by “fair amount/a lot”, by Comm-A Statusamount/a lot”, by Comm-A Status

41

40

27

15

26

81

84

79

70

82

0 10 20 30 40 50 60 70 80 90

papers for specificaudience

cite sources/avoidplagiarism

retrieve/analyzedatabase info

deliver speech

use library database forspecific purpose

Percent "fair amount/alot"

Comm-A

Control

Percent Responding Skills Grew by Percent Responding Skills Grew by “fair amount/a lot”, by Comm-A Status“fair amount/a lot”, by Comm-A Status

65

62

54

34

47

89

86

87

68

84

0 10 20 30 40 50 60 70 80 90 100

support ideas inpaper/speech

select/focus topics forpaper

organize ideas forpaper/speech

critiquepapers/speeches

write/revise drafts ofpaper

Percent "fair amount/a lot"

Comm-A

Control

Percent Students Responding “fair Percent Students Responding “fair amount/a lot”, by Comm-A Statusamount/a lot”, by Comm-A Status

70

70

34

49

72

80

52

70

0 10 20 30 40 50 60 70 80 90

recognize logical arguments

use language clearly

improvegrammar,punctuation,style

judge credibility of evidence

percent "fair am ount/a lot"

Com m -A

Control

Comm-A Students More Likely to Report Comm-A Students More Likely to Report Skills Grew by “fair amount/a lot”.Skills Grew by “fair amount/a lot”.

2.00

2.09

2.92

3.13

4.67

0.00 1.00 2.00 3.00 4.00 5.00

critiquepapers/speeches

cite sources/avoidplagiarism

retrieve/analyzedatabase info

use library databasefor specific purpose

deliver speech

Number Times More Likely Than Controls

Comm-A Students Many Times More Comm-A Students Many Times More Likely to Report Skills Grew by “fair Likely to Report Skills Grew by “fair

amount/a lot”.amount/a lot”.

1.44

1.54

1.61

1.80

1.98

0.00 0.50 1.00 1.50 2.00 2.50

judge credibility of evidence

improvegrammar,punctuation,style

organize ideas forpaper/speech

write/revise drafts of paper

papers for specific audience

Number Times More Likely Than Controls

Comm-A Students More Likely to Comm-A Students More Likely to Report Skills Grew by “fair amount/a Report Skills Grew by “fair amount/a

lot”.lot”.

1 .0 3

1 .1 4

1 .3 6

1 .3 8

0 .0 0 0 .2 0 0 .4 0 0 .6 0 0 .8 0 1 .0 0 1 .2 0 1 .4 0 1 .6 0

s u p p o rt id e a s inp a p e r/s p e e ch

s e le c t/fo cu s to p icsfo r p a p e r

re co g n ize lo g ica la rg u m e n ts

u s e la n g u a g ec le a rly

Nu m b e r T im e s M o re L ike ly T h a n Co n tro ls

The ESL Comm-A ExperienceThe ESL Comm-A Experience

ESL Comm-A Compared to Regular Comm-A ESL Comm-A Compared to Regular Comm-A and Controland Control

3.10 3.10

2.40

0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

ESL Comm-A Regular Comm-A Control

Com

mun

icat

ion

Scal

e M

ean

ESL v Regular Comm-A: Item-Specific ESL v Regular Comm-A: Item-Specific ComparisonsComparisons

2.98

3.08

3.21

2.61

2.65

3.37

2.89

2.90

0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50 4.00

Deliver a speech or oral presentation?

Retrieve and analyze information from library databases?

Produce papers or speeches/presentation for a specificaudience?

Improve your grammar, punctuation, and style?

Communication Scale MeansESL Comm-A

Comm-A

All differences significant at < .000.All differences significant at < .000.

ESL v Regular Comm-A: Item-Specific ESL v Regular Comm-A: Item-Specific ComparisonsComparisons

3.38

3.35

3.22

3.08

3.50

3.24

3.30

3.01

2.70 2.80 2.90 3.00 3.10 3.20 3.30 3.40 3.50 3.60

Cite sources/avoidplagiarism?

Support ideas in a paperor speech?

Use library databases forspecific purpose

Use language clearly andappropriately?

Communication ScaleESL Comm-A

Comm-A

All differences statisticallyAll differences statistically insignificant.insignificant.

ESL v Regular Comm-A: Item-Specific ESL v Regular Comm-A: Item-Specific ComparisonsComparisons

3.24

2.90

2.98

3.28

2.92

3.29

3.18

2.84

3.02

3.33

2.95

3.28

2.50 2.60 2.70 2.80 2.90 3.00 3.10 3.20 3.30 3.40

Select/focus topics for a paper

Critique speeches/papers

Recognize logical arguments

Organize ideas for a paper/speech

Judge the credibility of evidence

Write/revise drafts of a paper/speech

Communication Scale MeansESL Comm-A

Comm-A

All differences statistically insignificantAll differences statistically insignificant..

The EndThe End