summarizing skills

8/6/2019 Summarizing Skills

http://slidepdf.com/reader/full/summarizing-skills 1/36

Summarizing with LSA-Based Feed back 1

Developin g Summ arization Skills through th e Use of LSA-Based Feedback*

Eileen Kin tsch, Dave Steinh art, Gerry Stahl, and

LSA Research Group**

University of Colorado

Cindy Matthews and Ronald Lamb

Platt Midd le School, Bould er Colorado

running head : Summarizing w ith LSA-Based Feedback

send correspondence to: Dr. Eileen Kintsch

Institu te of Cogn itive Science

Campus Box 0344

University of Colorad o

Boulder, CO 80309-0344

(303) 492-0736

email: [email protected] o.edu

fax: (303) 492-7177

(in pr ess). Interactive Learning Environments . [Special Issue, J. Psotka, gu esteditor].




Abstract

This paper d escribes a series of classroom tr ials du ring w hich we d eveloped

Summary Street , an edu cational software system th at uses Latent Seman tic

Analysis to sup p ort wr iting an d r evision activities. Summary Street provides

various kinds of feedback, primarily about wh ether a student su mm ary

ad equately covers imp ortant sou rce content and fulfills other requ irements, such

as length . The feedback allow s stud ents to engage in extensive, ind epen d ent

pr actice in wr iting an d r evising w ithout p lacing excessive dem and s on teachers

for feedback. We first discu ss the un d erlying edu cational rationale, then p resent

some resu lts of the trials cond ucted w ith the system. We describe the

collaborative process am ong researchers and teachers which enabled the

development of a viable and sup portive edu cational tool and its integration into

classroom instruction.

Keyword s: Latent Semantic Analysis (LSA), edu cational technology




Summary Street is an edu cational software system tha t uses Latent Seman tic

Analysis (LSA) to sup por t the reading an d w riting activities by wh ich stu d ents

develop and expan d th eir know ledge in new topic areas. Summary Street

determines the degree to wh ich a stud ent summ ary covers imp ortant source

content and conforms to requirem ents, su ch as length . It tells the stud ent w hat

information in the source is missing, provides comments on redu nd ancy,

extraneou s content and certain asp ects of mechanics. Its current op eration is

d escribed in m ore detail later. First, how ever, we d iscuss the u nd erlying

edu cational rationale and review the course of its research and d evelopm ent.

Text-based activities are ind ispu tably a major veh icle for acquiring basic

content know ledge in most school settings, across a ran ge of pedag ogical mod els,

from those that em ph asize trad itionally structured classroom s to those in w hich

stud ents direct their own p aths of inqu iry. One form of comp uter sup port for

comp rehension an d learning that ou r team has d eveloped uses LSA to prov ide

stud ents with imm ediate feedback on how well their su mm aries of informative,

expository texts cover the top ic they are w orking on . We intend for this tool to be

used by stud ents independ ently, though still within a classroom setting, so that

they can assess their own initial attemp ts to com pose and revise their sum mar ies.

We hope thereby to pr ovide students w ith more experience in extended writing

and revising, w hile leaving teachers m ore time for other kind s of edu cational

activities, such as coaching and m od eling w riting and su mm arization

techniqu es, providing ind ividu al help, planning an d delivering instruction,

evaluating final versions of stud ents’ wr iting an d other p rojects. Thu s, in n o

sense is the tool intend ed to r eplace the teachers’ role, for it is they w ho m ust




teach the skills and, at least in our im plem entation, evaluate the final pr od ucts of

stud ents’ writing. Even thou gh stu dents are able to use the sum marization tool

on their ow n, we w ant to emp hasize that it is a system that seeks to comp lement

classroom instru ction, rather th an existing as a stand -alone system. Its pu rp ose is

to reinforce wh at is being tau ght rath er than just p rovid e an adjunct learning

activity. Thu s, in d esigning our first p rototyp e, called State the Essence , we began

w ith the prem ise that this wou ld take place in collaboration w ith teachers w ho

were the intended users. The sum marization tool in its many transformations

and its integration into th e instructional cur riculu m r epresents a collaborative

effort of researchers and teachers.

The cur rent system evaluates only th e completeness of the conten t, for the

mo st part, leaving other imp ortan t aspects of w riting, such as sentence stru cture,

organ ization an d style, for traditional instru ctional metho d s. N onetheless, we

believe that in ad d ition to imp roving th eir writing skill, stud ents w ill benefit

metacognitively from w orking independ ently, guided by the imm ediate

feed back they receive. With frequen t p ractice in assessing an d revising the

content of their su mm aries, w e believe that stud ents w ill also become m ore

attun ed to their own thinking an d w riting processes; they w ill be mo re likely to

realize wh at they do and do n ot und erstand and better able to express wh at they

mean in w riting.

Imp ortance of Summ arization as a Learnin g Skill

Ou r initial discussions qu ickly conv erged on su mm arization as the kind of

learning activity that LSA technology could effectively su pp ort an d that

conformed well with th e teachers' instru ctional goals. The sixth-grade classroom s




in wh ich the tool is being tested em ploys a pro blem-based learning ap pr oach for

instructing the d istrict m and ated curriculum. Learning h ow to sum marize text is

emp hasized through out the school year as a crucial stud y skill that helps

stud ents acquire a basic un d erstand ing of d ifficult and n ovel subject matter

wh ich they can then app ly to solving p roblems or developing a p roject.

Sum marizing is more constrained than an open-ended writing task, with w hich

youn g stud ents often flound er, and it has a nu mber of advantages over simp ly

reading text and answering “comp rehension questions”, including the following:

• Sum mar izing n ot only prov ides practice in extended expository wr iting, it

also teaches imp ortan t stud y skills, such as identifying imp ortan t conten t

and separating m ain ideas from d etails. The fact that stu d ents at this age

tend to high light everything in a text – creating a “sea of yellow” – is

symp toma tic of their inability to d o this. This hap pen s especially w hen

stud ents are dealing w ith content that is comp letely new to them.

• Sum mar izing for a given pu rp ose (e.g., to wr ite a repor t on Mayan

religious beliefs) requires even d eeper th inking an d analysis to select the

relevant information.

• Sum marizing is a way to develop solid u nd erstand ing of comp lex

material and also to articulate one’s un d erstand ing so that it can be shar ed

with others. The teachers with w hom we w ork have noted clear

differences in d epth of u nd erstanding of topics that stud ents have

sum marized as opp osed to those they have only read about. Stud ents

app ear to retain app reciably m ore information over longer p eriods of time

if they hav e sum ma rized it, and in classroom d iscussion they d isplay an




ow nersh ip of those topics, w hich show s up in their ability to contribu te

detailed an d well reasoned ideas.

• H aving to expr ess content adequ ately yet concisely makes stud ents awar e

of the need to learn sum marization strategies that go beyond just ad ding

and d eleting single w ord s, ph rases or sentences. This aw areness becomes

a starting p oint for introd ucing stud ents to higher-level strategies, such as

how to reformu late text content by combining several ideas in a single

sentence and generalizing across details.

• Sum mar izing requ ires active meaning constru ction to a mu ch greater

d egree than choosing a respo nse on a m u ltiple-choice recognition test, or

even than w riting short answ ers to isolated q uestions. Thu s, not only is

sum ma ry wr iting an effective means to constru ct and integrate new

know ledge, it is also a m ore auth entic method for assessing w hat stud ents

do and do n ot und erstand than traditional comp rehension tests.

The Use of LSA to Provide Writers with Content Feedb ack

As the rationale as w ell as techn ical details about LSA can be foun d in variou s

other p ub lications, we w ill not review th em here (please see Land au er & Du ma is

(1997), Land au er (1998), Land au er, Foltz & Laham (1998), as well as the

introd uctory article by Land au er & Psotka in this volum e). Essentially, LSA is an

au tomatic statistical meth od for repr esenting the meaning of word s and text

pa ssages based on the analysis of a large amou nt of textual inpu t. A semantic

space is generated in w hich word s, sentences, and w hole texts can be

rep resented a s vectors. H ow closely related th ese vectors are to each other is




measu red b y the cosine betw een them . We use this cosine m easure to calculate

wh at feedback to provide w riters.

The most general LSA space available today is based on an inp ut of abou t

11M w ord s from carefully selected texts that form a rep resentative samp le of

wh at a single stud ent finishing h igh school might have read d uring h is or her

school year s. This sp ace is sufficient for ou r an alysis, except for technical topics.

Thus, for stud ents writing on the functioning of the pulmon ary and cardiac

systems, or stud ents w riting o n Meso-American civilizations, the general space

does n ot have enou gh information to m ake the fine d istinctions requ ired. It has

some basic inform ation abou t the Inca and Maya cultures, for instan ce, bu t not

enou gh to tell apart d etails of their religion or ag ricultu ral p ractices. Therefore, a

specialized space mu st be constru cted in or der to u se LSA. For instan ce, the

Heart space discussed below w as constructed from an inpu t of 830 docum ents

comp rising a bou t 17,688 wo rd s describing the fun ction of the h eart. The Meso-

Am erican space was based on 530 d ocum ents, com pr ising 46,951 w ord s dealing

with this topic. At the mom ent we do not yet have a good und erstand ing when

specialized spaces are required and w hen th e general space suffices. Thus, ad h oc

decisions m ust be m ade based on the p erformance of the system.

Becau se misspelled w ord s are not considered w ord s by LSA, we first have to

correct spelling. For this p ur pose, all misspelled wo rd s (or rather, all strings LSA

d oes not recognize) are flagged with asterisks, and the stud ent is asked to m ake

sur e that they are sp elled correctly. In p rinciple, althou gh th is is not don e in th e

present system, a stand ard spell checker can provide th e stud ent w ith

alternatives, and LSA can select the m ost pr om ising alternative(s) by looking at

the cosine betw een each alternative identified b y the sp ell checker and the




immed iate neighborhood of the word . Most likely, words w ith a higher cosine to

the context are th e right choice.

Content feedback is provid ed in th e following man ner. Sup pose stud ents

are asked to sum mar ize a text T containing the sections {T 1, T 2,…., T k }. The

teacher requires that each of these sections be covered in the stud ent’s summ ary.

What w e do is to comp ute th e cosine, C i, between the sum mary a stud ent wrote

and each of the sections T i. If C i ≤ t i, where t is an emp irically d etermined

thresh old value, the stud ent is told that section T i is not ad equ ately covered in

the sum mary. The student then h as the option to look at the app ropriate section

of the text on the comp uter screen and ad d som e material about this section to

the sum mary. If C i ≥ t for all sections, the stud ent is told that h e or she h as now

covered all par ts of the text.

Since the teachers require sum mar ies to be of a given w ord length to

avoid extensive cop ying (about on e quar ter of the source text), stud ents are told

how man y word s they have written so far and w hether this is within the allowed

limits. If the text is too long, the stu d ent is given tw o kind s of feedb ack to help

shorten it. One the one hand , irrelevant sentences in the su mm ary are identified.

The cosine is com pu ted betw een each sentence in the su mm ary and the text as a

w hole. If it is below som e lower th reshold, th e sentence is iden tified as (possibly)

irrelevan t. This relevancy check tend s to pick up sentences that are tru ly

irrelevan t (such as “I hop e you like the sum mar y I wrote”) or sentences that refer

to obscure d etails in th e text that are not ap prop riate for a sum mary. On the

other hand , redun dan t sentences are identified by comp uting the cosines among




all sentences in th e sum ma ry. If a cosine is greater than some u pp er limit, the

tw o sentences are highlighted in the text and th e stud ent is told to inspect them

for the pu rpose of combining them or d eleting one. Sixth-grade stud ents tend to

rep eat themselves, so this is a very u seful check. No te, how ever, that both th e

relevance and th e redu nd ancy check occasionally p ick u p false positive:

sentences, for examp le, w ith several overlapp ing wor d s, bu t distinct mean ings.

This has the positive result that stu d ents mu st critically evaluate the compu ter’s

adv ice and decide w hether they agree w ith it or n ot. Upper an d lower limits for

the relevance and red un d ancy checks are, once again, set emp irically. For

examp le, sentences with a cosine to the text that are below .30 migh t be termed

irrelevan t, and sentences with a cosine greater than .80 between them selves

might be termed redun dant.

The system itself is thu s quite simp le. H ow ever, what w as not simple was

to determine the best ways to provide this kind of feedback to stud ents and the

op timal sequencing of this feedback, as described below.

History of Trials Using Sta te t he Essence : Fall 1997 – Fall 1998

Instruction

Two team-tau ght classes participa ted in trials using an early version of the

sum ma rization tool called State the Essence d uring th e 1997-1998 school year, and

a su bsequen t trial took place in th e fall of the next academ ic year. The system

was d esigned to sup port stud ents’ sum mary w riting in three curricular un its,

each lasting abou t th ree-to-four w eeks: Energy Sour ces (Septem ber, 1997 and

Septem ber, 1998), Ancient Civilizations of th e Western H emisph ere (Janu ary,

1997) and The H um an Circulatory System (Ap ril, 1998). Stud ents first comp osed




their sum maries using a w ord p rocessor or pen and pap er in ad vance. They then

pasted or typed them into State the Essence in ord er to receive feedback on h ow to

revise them. For the trial on the circulatory system, w e collected su mm aries that

stud ents wrote u sing trad itional means as well as those written w ith State the

Essence , wh ich allowed u s to ma ke within-subject comp arisons. Ho w ever, our

ma in goal du ring this initial period w as to test the system rath er than to collect

learning and performance data.

1. Sou rces of Energy . In ad d ition to teaching stud ents abou t the new

content, du ring the first un it the teachers’ instru ction introd u ced stu d ents to the

concept of sum mar ization an d th e app rop riate strategies. The teachers’

instru ction inclu ded d irectly explaining the strategies and their pu rp ose, together

with m odeling the strategies and class discussion of good and poor examp les of

summ ary w riting.

Stud ents read 10 brief texts (two to tw o-and -a half pag es) abou t different

sour ces of energy (non renew able: coal, natu ral gas, nu clear, petroleum , pr opan e;

and renewable: biomass, geothermal, hydrop ower, solar, wind) and wr ote one

sum ma ry (75 - 200 w ord s) of each energ y typ e. Stud ents used this task as the

starting p oint for their projects, which involved becoming an expert in on e

energy sou rce, organ izing a science station and teaching th e subject to other

stud ents in small groups.

2. Ancient Civilizations. For th is unit stud ents w ere requ ired to

sum ma rize three texts (each about tw o-and -one-half to three pages) about the

Maya, Aztec and Inca civilizations, again to d evelop basic know ledge abou t the

cultures. The sum mar ies were to be betw een 200 to 300 wor ds long. Each class

then d ivided into thr ee group s, each focussing on one of the cu ltures, and each




mem ber of a group researched on e particular aspect of the culture (e.g., history,

religion, artistic or scientific contribu tions, social stru cture). Fina lly, each grou p

ma d e a joint presentation w ith visual prop s to the class as a who le, each memb er

filling in a p iece of the top ic in jigsaw fashion. The su mm arization instru ction

this time focused on h igher-level strategies, such as sentence combining a nd

constructing generalizations to achieve conciseness. Stud ents pr epared two of

their summ aries in the traditional mann er, using a w ord processor or p en and

pap er, and revised a third sum mary gu ided by feedback from the sum marization

software.

3. Circulatory System. Un like the pr eced ing u nits, the instru ctional focus

here was p rimarily on d eveloping a d eep un derstand ing of the content - a

challenging top ic w ith a great d eal of un familiar technical vocabulary an d

d ifficult concepts. Sum ma rization of tw o texts about the lun gs and the heart w as

used to help stud ents integrate this information and to assess their conceptu al

un derstand ing of the d ual-loop circulatory system. The sum maries were to be

150-250 w ord s in length, and stud ents used State the Essence to work on one of

these summ aries. They w rote the other sum mary using trad itional means.

Evolution of State the Essence

Initial trials with State the Essence wer e beset by techn ical prob lems from

overloading th e system with too m any simu ltaneous su bmissions. How ever,

these pr oblems w ere overcome in ou r later trials. In gen eral, the school trials

with th e sum marization software w ere a success in term s of student enthusiasm

and teacher satisfaction, at least to some d egree: the system w orked w ell, was

relatively easy to learn, an d using State the Essence did not interfere with

stud ents’ learning of the content (there wa s no significant d ifference between




sum ma rizing cond itions in scores received on a short-answ er test on th e unit of

stud y). How ever, as m entioned, the p urp ose of these school trials was not a

formal evaluation of the system bu t rather to further d evelop and refine it.

There are three classes of changes that w e explored:

1. H ow the stu d ent’s wr iting is to be evaluated by LSA: There are several

option s here; for examp le, a given essay can be ma tched aga inst a set of

pre-graded essays, or against an expert sum mary p repared by the teacher

or expert writer. In the end w e adop ted a m ore practical method th at

wou ld only require a teacher to subm it the text to be sum marized,

subd ivided into topic sections, a m ethod that has been incorporated into

the later versions of the system.

2. What feedback to give the stud ent, and in w hat ord er: It is easy to

overwh elm users and confuse them w ith the rich feedback the system is

able to provide. Over the course of the year w e experimented with several

d ifferent feedback formats before arriving at a system that is somew hat

constrained yet still flexible to u se. “Less is more” w as ou r take-hom e

message - less feedback and more su pp ort.

3. H ow to em bed ou r system into classroom instru ction: Use of the

sum ma rization tool as a stand -alone system is rather inefficient for

mid dle-school stud ents. Most stud ents at this level need explicit

instruction on how to sum marize, and how to revise. Furthermore,

available techn ology has m ad e it difficu lt to use th e system in a classroom

w ithout taking too m u ch time aw ay from other instru ctional activities.

Ou r trials therefore took place over on e or two sessions with th e entire




class – a pr actical necessity, thou gh n ot an op timal wa y to learn r evision

skills.

Evaluating the sum maries

Ou r initial problem in d elivering feedback to the stu d ents was to d ecide

w hat text to use as a basis for comp arison. Several different ap pr oaches to

evaluating college stud ents’ essays are described in Land au er, Foltz, and Laham

(1998), some of which w e also app lied to eva luating th e stud ents’ sum mar ies.

One app roach is to comp are a summ ary to a corpu s of previously graded

sum ma ries. The sum mar y w hich is the closest match in term s of the LSA cosine

becomes the basis for assigning a g rad e of A, B, or C, and so on. Since we had not

yet accumu lated a set of graded su mm aries to draw on, this option was n ot open

to u s. Hence, w e first tried m atching the sixth-graders’ summ aries against a set of

four or five sum mar ies w ritten by expert writers (teachers and r esearchers).

Given that even expert w riters d o not comp letely agree on wh at content to

includ e or exclud e, the stud ent’s overall score w as based on th e best fit (i.e., the

highest LSA cosine) to one of the exp ert texts. Section scores were based on a

comp arison of the sum mary to each section of a “golden” sum mary that

incorporated the m ain content in all the expert sum maries. Although this method

w orked q u ite w ell, pu tting together a set of expert sum mar ies for each novel text

proved too cum bersome in the long run.

An alternative basis of comp arison is to u se the sou rce text itself. A

holistic score can be obtained from th e cosine between th e stud ent’s sum mar y

and the original source text. In ad d ition, section scores ma y be derived by

d ividing th e text into d istinct topic sections, app roximately equal in length, and

comp aring th e entire sum mar y to each of these sections. As described earlier in




this pap er, a set of emp irically determ ined thr esholds is used as the basis for the

feed back given to the stud ent on h ow ad equ ately each section was covered. The

sum ma ry “p asses” wh en all sections h ave m et the criterion for each section

w ithin the given length constraints. This meth od u nd erlies all the versions of the

sum marizing software described here.

Presenting the Feedback

LSA-based feedback goes far beyond other form s of autom atic feed back, such as

spelling an d g ram mar checks, by evalu ating the seman tic content of a piece of

w riting. For essays and sum mar ies, it can tell the writer w hether or n ot all the

important su btopics have been covered and wh at kind of information is missing;

it can p oint out sentences that app ear to have too m uch overlap in content with

each other or w ith the original text; and it can su ggest sentences that seem to

hav e little relevance to th e top ic of the text.

In ad dition to th is conten t information, in ou r initial trial on Energy

Sources we pr ovided stu dents w ith feedback on the length of their sum maries.

Length constraints across all three trials varied between 100 to 300 word s for

texts that ran ged from abou t 800 to 1450 word s. Stud ents received a n overall

score weighted to reflect appr opr iateness of length, the ad equa cy of section

coverage and overall conten t coverage. In ad d ition, they could requ est checks for

(a) red un d ancy, (b) relevance (both based on a comp arison of sentences in th e

sum ma ry w ith those in the original text), and (c) rep etition (based on a

comp arison of all sentence pa irs in the sum mar y). Ou r sixth-grad e stud ents,

althou gh ap preciative and highly motivated, seemed confused an d floun dered in

their attemp ts to revise their sum ma ries. In ad dition to solving various technical

pr oblems, it was clear that w e needed to pr ovide better editing tools, a clearer




presentation, and m ore supp ort for sum marizing and revising both within the

system and through classroom instruction. We especially n eeded to p resent the

feedback in a w ay that w as easier to un derstand than th e set of nu merical scores

that w ere initially presented simu ltaneously.

In ou r second tr ial on An cient Civilizations th e feedba ck was given in

three stages, accessed by the u ser’s requ est first for gen eral feed back, then

successively m ore. The gen eral feedback includ ed length ( too long, too short ), an

overall score, and ad equa te/ inad equa te section coverage, as before. Requ ests for

mo re feedba ck first displayed irrelevant an d r elevan t sentences (the latter were

praised); then, at an adv anced level, feedback was pr ovided on redu nd ant

sentences (sum mary sentences with too mu ch overlapping content). In add ition,

we ad ded an overview of sum marization strategies to the Introd uction to State

the Essence and hyp erlinks to further hints and examp les. Links were also

pr ovided to the Maya, Aztec, and Inca sour ce texts and to add itional backgroun d

information.

The results of this classroom trial were both encou raging an d r evealing of

significant w eaknesses in the system. Again, the overall point score was a gr eat

motivator: stud ents were challenged to try to imp rove their scores and remained

focused on th e task. H ow ever, the scores were n ot always reliable, tend ing to be

inflated and too sensitive to sm all local variations. Sentence level feedba ck w as

especially prob lematic, w ith too many inap pr op riate flags (both good and bad ),

and d ifficult to use because pr oblematic sentences were p resented in a list, out of

context and on a sep arate screen from the w riter’s textbox. Presenting misspelled

w ord s as a list posed similar d ifficulties for m aking corrections. Even thou gh

pr esented in stages, or at different levels, stud ents w ere still overw helmed by the




am oun t of feed back they received and often dismayed at the mu ltiplicity of

problems to deal with. Furth er, many stud ents needed extensive and quite

explicit guidan ce on ho w to m ake mean ingful chan ges in revising their

sum maries; in p articular, they needed to be show n how to generalize across

sentences or how to combine ideas from several places into a single sentence in

the context of their ow n w ork. This need clearly goes beyond w hat LSA-based

feed back prov ides, but high lights an area w here the teacher’s classroom

intervention can be helpful.

Ou r next attemp t to imp rove the system consisted in greatly simplifying

the feedback, both w hat was p rovided an d h ow it was pr esented. Thu s, for the

un it on th e Circulatory System, feedback consisted on ly of a p oint score (0-100

po ints); length ( too short , too long , or about right ); an eva luation of the conten t of

each section ( good , ok , needs improving , or missing ); and listing th e w eakest section

w ith a hyp erlink to th at section of the source text. The same version of the system

w as used again w ith minor chan ges for the fall 1998 un it on Energy. A screenshot

showing the first feedback page for a summ ary on hyd ropow er is shown in

Figure 1.

Insert Figure 1 About H ere

This version w as easier to u se, althou gh th e overall conten t scores w ere

still not sufficiently reliable, and interactions w ere som etimes confusing. For

examp le, stud ents were frequen tly fru strated to see large d ecreases in their point

score w hen content w as cut to stay within th e length constraints. Indeed , the

d ifficulty of balancing the n eed to be comp lete and the need to be concise

revealed again a need for more exp licit instru ction of higher -level summ arization

strategies in th e classroom. It is still imp ossible to p rovid e au tomatically the k ind




of concrete help often need ed in the context of a stud ent’s own su mm ary

without, so to speak, giving the answ er away. In other word s, our comp uter tool

cannot yet interact w ith a stud ent abou t their writing like a hu man tutor , eliciting

an ap pr op riate respon se throug h carefu lly calibrated q uestions. How ever, it can

make both teachers and students aw are of where the gap s lie in skill and

un derstand ing, which can then be add ressed ind ividu ally or in general class

discussion.

To sum mar ize, the goals for ou r sum mar ization tool consisted of the

following:

• to prov ide sup por t for a challenging activity that fosters both d eep learning of

d ifficult new content and pr omo tes writing skills;

• to give students extended practice in w riting and revising sum maries wh ile

relieving teachers from the bu rd en of reviewing an d gr ad ing successive dr afts;

• to motivate stud ents to work hard and ind epend ently by providing immediate

and individualized feedback on h ow to revise their writing.

H ow w ell d id th e system fulfill these expectations? As stated p reviously,

we h ave a m odest amou nt of formal emp irical results at this point and quite a lot

in the way of informal observations and feedb ack from b oth teachers and

students.

Emp irical results w ith State the Essence

Du ring ou r classroom trials with State the Essence in the sp ring of 1998 we

collected d ata comp aring the sum mary scores award ed by th e system w ith those

of hu man scorers (Ancient Civilizations). In ad d ition, w e examined the effects on

learning of sum marizing with State the Essence vs. using trad itional mean s in a

w ithin-subjects design in th e un it on the Circulatory System. And in the fall of




1998 w e again comp ared teacher assigned scores w ith LSA scores for su mm aries

of two d ifferent energy top ics.

1. Comp arison of LSA scores with h u man grad ers: Ancient Civilizations.

For this un it stud ents sum mar ized texts on all three cultures, one using State the

Essence for feedback on rev ision. Stud ents w ere allow ed to choose the culture they

w ished to sp ecialize in. Most stud ents chose to sum mar ize the Inca text. For the

first comp arison w e derived the LSA cosine between the stud ent sum mary and the

text the stud ents had read. We then comp ared this cosine w ith the overall content

grad e assigned by an ou tside teacher (LK) to the stu den ts’ Inca and Az tec

sum ma ries. For these 50 sum ma ries, the correlation betw een the teacher grad e and

the LSA cosine was r = 0.64. The correlation betw een a second scorer (EK) and the

teacher w as r = 0.69. Therefore, LSA scores are qu ite comp arable to h ow an

experienced teacher rates these sum mar ies.

The second comp arison was intend ed to assess w hether LSA could m atch

a given senten ce to a p articular section of the source text as well as hu man

grad ers. Thus, the same two expert gr ad ers (LK & EK) were show n 119

ran d omly chosen sentences and asked to choose w hich of the five sections of the

Aztec text the sentence was d escribing. For each senten ce the LSA cosine w as

comp u ted against each of the five sections, and the section w ith the highest

cosine w as considered to be LSA’s “choice.” The tw o expert gr ad ers w ere in

agreement for 109 of the sentences (91.6%). LSA agreed w ith the first gra d er on

101 of the sentences (84.9%) and with the second grad er on 99 of the senten ces

(83.2%). Therefore, LSA d oes almost as w ell as hu man s at d etermining the sou rce

of know ledge for a given sentence, a fact that w e hop e can be u seful in d esigning

futu re versions of the system.




2. Posttest scores from su mm arizing w ith State the Essence versus with

traditional means: Hu man Circulatory System. During th e unit on the H um an

Circu latory System, 39 stud ents from tw o classes sum ma rized tw o texts, one on

how the lungs function and one on the h eart and th e circulation of blood in the

body. Each stud ent wr ote and revised a sum mary on one of the topics by

conventional means (using pen an d p aper or w ord p rocessor) and one using the

State the Essence software. To see if there were d ifferences in how well stud ents

had learned the material about each topic, we compared their scores on an end-

of-un it, short-answ er test on the h um an circulatory system w ith respect to these

two topics. We foun d no d ifference in students’ und erstanding of the tw o topics

related to h ow the sum maries were wr itten, although one class performed

consistently better than th e other on all questions. A com par ison of the average

grad es (0-10 points) given by two outsid e teachers (LK & AW) likewise show ed

no d ifference in qu ality of the stud ents’ sum mar ies related to cond ition. The

average gra d es were 6.80 for traditionally written sum mar ies and 6.74 for the

sum ma ries w ritten with LSA-based feedb ack. The agreement between the tw o

hu man grad ers was r = .59. Thu s, based on evidence from a single trial, the

sum marization software did n ot app ear either to benefit or to harm stud ents’

learning or writing.

3. Comp arison of LSA and hu ma n grad ers: Energy u nit. Fifty-six stud ents

wr ote their two required sum maries on chosen topics as homework an d u sed

State the Essence to revise them. The averag e correlation based on the grad es of

the tw o classroom teachers (CM & RL) with LSA scores for four of the texts

(biom ass, hyd rop ow er, petroleum, & pr opan e) w as quite high: r = 0.88.

H ow ever, the average correlation between teacher and LSA grad es was qu ite low




for the r emaining six topics: r = 0.32. In pa rt th is low correlation betw een LSA

and teacher grad es is du e to missing data (the summ ary topics were un equally

d istribu ted am ong th e two classes). Add itionally it results from the fact that State

the Essence used a single thresh old for all topics which, how ever, are not equal in

terms of their conceptua l difficulty. Hen ce, for som e topics stud ents’ sum ma ries

received h igher scores from LSA than th e teachers thou ght th ey deserved .

Evaluation of the system based on classroom o bservations

Despite overall encourag ing comm ents on State the Essence by both teachers at

the end of the year, our concern abo ut th e un reliability of the overall score

remained. In order to avoid frustrating the stud ents, we w anted to m ake it

po ssible to obtain 100 points. How ever, this often m ad e it too easy to reach a

high score, wh ich th en d iscouraged stud ents from continu ing to revise their

work. As m entioned earlier, minor changes often r esulted in un reasonable jum ps

in the score. The stud ents tend ed to regard their scores as an overall measu re of

w riting qua lity. Hence, once they had reached 100 points or w ere close enou gh,

they often did n ot review w hat they had written and consequently were up set

w hen th ey received poo r grad es from th e teachers d ue to lack of organ ization

and poor writing style. These observations led u s to qu estion w hether an overall

score was a good kind of feedback to pr ovide. Many stud ents treated th e score as

an en d in itself, trying to increase it by the cheap est means p ossible, rather than

focusing on imp roving their wr iting. These problems su ggested that the feed back

should be displayed in a form that w as more concrete and easier for students to

use tha n th e 100-point score comb ined w ith textu al pointers in State the Essence

(good, ok, needs more work , etc.).




History of Trials Using Summary Street : Sp rin g 1999

In 1999 we chan ged the n ame an d mod ified the interface to reflect a m ajor

revision in our ap proach to prov iding stud ents feedback on th eir sum maries.

After typing their sum mar ies into the textbox, stud ents now receive the

following k ind s of feedb ack:

1. Misspelled w ord s are highlighted an d can be corrected in th e textbox; the

stud ent’s sum mary is automatically saved by th is operation.

2. The request for feedback retur ns a grap hic d isplay ind icating the length of

the sum ma ry and how w ell the content of each section of the original text

has b een covered (see Fig. 2). The display for content coverage consists of

horizon tal green bars extending ou t to a vertical line symbolizing the

thresh old. The w eakest section is ind icated, and a hyp erlink is prov ided

that th e wr iter can u se to access that top ic section in th e original text.

Instead of an actual w ord coun t, length is shown by a vertical bar on the

left, with bisecting lines ind icating the p rescribed m inimu m a nd

maximum . A green bar is displayed if the sum mary length is within th ese

limits, while a red bar is show n if the summ ary is either too shor t or too

long. Praise is given on ce the su mm ary h as passed the criterion for

content coverage for all sections.

3. Furth er help for revising is available at this point, for examp le, if the

sum mary exceeds the p rescribed length, in the form of a red un dan cy

check an d a relevance check. These tools help stu d ents locate sentences

that have overlapp ing content or seem not very related to th e topic being

sum marized and that wou ld be possible cand idates for d eleting or

collapsing togeth er.




4. Finally, a “Format for Printing” bu tton allows the stu d ent to obtain a

dou ble-spaced version to p rint out, review an d hand in to the teacher.

Emp irical results w ith Summary Street

The new system functioned quite well du ring tw o classroom trials in spring

1999. Stud ents w ere better able to deal with th e feedback on th eir own an d th is

helped th em to stay focussed on their w riting for extend ed p eriods of time.

Although we still see ways to imp rove the system, futu re changes should be

fairly minor on es. With a stable system now in place w e have begu n m ore formal

testing of the system than w as possible un til now .

In trials that took p lace du ring sp ring 1999 two classes of the sixth-grad e

stud ents used Summary Street to comp ose or revise some of their sum mar ies on-

line, guided by the feedback, and other sum maries using a word processor or

pen an d p aper. Fifty-tw o sixth-grad e stud ents participa ted in both the trial on

Ancient American Civilizations and the trial on the Hu man Circulatory System.

Classroom instruction wh ich incorpora ted u se of the Summary Street software

followed the sam e procedure as in th e trials the p revious sp ring w ith State the

Essence . Thus, as part of their learning activities du ring the u nit on Ancient

Civilizations, stu den ts were required to w rite sum ma ries on three cultures -

Maya, Aztec and Inca – one u sing Summary St ree t, the others by hand or on a

word processor. The stud ents prepared rough drafts of their sum maries as

hom ewor k and revised them in class. For the Circu latory System u nit, stud ents

comp osed and revised both of their summ aries of texts about the heart and lungs

on-line, either u sing the sum marization software or a word processor, with the

experimental conditions (Summary Street or w ord processor) coun terbalanced.




1. Ancient Civilizations sum ma ries. The two classroom teachers each

scored half of the trad itionally written sum mar ies and half of the Summary Street

sum ma ries on a 10-point scale, bu t they w ere not blind to experim ental cond ition

since the m ain pu rp ose of this trial w as to try out th e new system . The results,

w hich are presented in Figu re 3, show for the first time a significant adv antage

for the Summary Street condition: Grad es assigned by the teachers are

significantly higher for the Inca sum mar ies w ritten with Summary Street than

those w ritten by h and or a w ord pr ocessor (t(50) = 2.47, p = .02). Interestingly,

both teachers and stud ents considered th e Inca text the m ore d ifficult of the

three, wh ich is confirmed by the lower m ean grad es these sum mar ies received

(8.02 for Inca sum maries vs. 8.94 for Aztec and 8.79 for Maya ).

Insert Fig. 3 about h ere

2. Circulatory System trial. The results obtained from th is trial consist of

stud ents’ Heart and Lungs sum maries and their scores on a posttest at the end of

the u nit. Sum ma ries of the tw o texts w ere scored on a 10-point scale by the tw o

classroom teachers w ho w ere not aw are of students’ identity and experimental

cond ition. Overall, the Lung s sum ma ries received lower gr ad es than those of the

H eart texts (mean grad e = 3.39 vs. 4.00, respectively), confirming ou r im pr ession

that th is text, althou gh sh orter, was the m ore difficult of the tw o. Quite stringen t

length constraints also ad d ed to th e difficulty of sum ma rizing this text. It is

noteworthy, therefore, that the Lungs su mm aries comp osed w ith Summary Street

received significantly higher grad es than th ose comp osed on a w ord p rocessor

(t(50) = 2.32, p = 0.02 ), thu s confirm ing th e “har d text effect” w e had foun d w ith

the Inca text in th e pr evious trial. There was n o d ifference in sum mar ies of the




H eart text, regardless of how they w ere written. These results are show n in

Figu re 4. The p osttest grad es did not d iffer significantly across text or cond ition.

Insert Figure 4 about h ere

The results of these two p relimin ary trials sug gest that the content

feed back delivered b y Summary Street is especially helpful wh en stud ents are

faced w ith mor e difficult task d eman d s or a hard er text. In a recent trial we hav e

furth er investigated the effectiveness of the Summary Street with hard text in

mo re tightly controlled circum stances. More sp ecifically, we comp ared the

revision process, and its time course, wh en stud ents are gu ided by content

feed back versus w hen th ey only receive feedback on length, such as they wo uld

get from a w ord pr ocessor. Preliminary resu lts are very encoura ging, but since

the analyses are not yet com plete, they m ust be repor ted elsewhere. In fu tur e

pr ojects, we h ope to replicate the resu lts we obtained w ith d ifficult text in an

older p opu lation of college stud ents. Finally, we w ou ld like to investigate the u se

of this software in collaborative wor k sessions as opp osed to ind ividu al work

sessions. Ou r informal observations d ur ing the Spring 1999 trials sug gest that

Summary Street could pr ovide an especially rich context for collaborative

learning.

Inform al observ ations of edu cational benefits: 1998-1999 trials

Based on both our observations and w hat the teachers have reported to us w e

believe that the n ew ver sion of the LSA-based su mm arization tool has the

po tential to effectively enh ance wh at teachers teach in a m eaningful w ay.

Providing students with man y opp ortunities to do extend ed w riting w ith

feed back abou t their w riting is one of the m ost effective tools for h elping th em

imp rove their writing. Such op por tun ities are necessarily red uced if stud ents




hav e to rely on teachers’ feed back for every p iece of w riting. Furth ermor e, as

some sixth grader s have pointed ou t in their evaluation of the softwa re, getting

feed back from th e teacher just takes too long. Therefore, if we can d evelop a

system that gives students imm ediate feedback on their w riting in a form they

can use, teachers w ill not hesitate to give more w riting assignmen ts.

It may b e d ifficult to attribute d irect benefits to stud ents’ learning from use of

the software by any form al means, given th e difficulty of controlling

instru ctional, teacher, and a nu mb er of other variables in a classroom setting and

w ith a fairly small samp le size. Yet w e can assu me some positive effects on

learning from the kind of behaviors we h ave witnessed in the course of

classroom trials, especially th ose that took p lace in sp ring 1999 with the n ew

system:

• Stud ents are able to assum e more respon sibility for writing and revising on

their own , using th e feedback on how w ell their sum mar y covers the essential

content. Moreover, they seem to enjoy the challenge.

• By focussing their attention on specific content th at is not ad equa tely covered

stud ents find it easier to iden tify imp ortan t information – certainly a good

beginning in learning how to summ arize!

• Stud ents who u sed the sum marization tool in the classroom w orked h ard an d

long to satisfy th e criteria set for them . Being able to tra ck their p rogress

motivated them to w ork throu gh m any cycles of revision gu ided by th e

imm ediate feedb ack: they kep t trying to get it right and in so doing they

interacted w ith the text content for a longer period of time and at a deep an d

analytic level. Withou t feedb ack mark ing their p rogress, stud ents typ ically




mak e few changes in th eir writing an d are likely to be satisfied w ith a first

draft, nicely formatted w ith a w ord processor.

• The sum ma rization tool helps stu d ents locate the specific problems w ith their

sum maries and makes them aware of the task demand s involved in summ ary

wr iting. It’s hard to balance the conflicting d eman d s of topic coverage and

conciseness. And it is frustrating to see the coverage ind icator go backw ard s

wh en you d elete material to satisfy the length constraint. Thus, in work ing

with th e system stud ents feel a need for the strategies being taug ht. They

eventually d iscover that ad ding or d eleting a word or a sentence is not

enou gh. And besides, w hich sentence? As they try to revise their sum mar ies

stud ents come to realize that they n eed better metho d s. These are valuable

metacognitive insights that m ake stud ents mor e receptive to instru ction.

Integrating Sum marization Software into Classroom Instru ction

It has become clear to us in observing stud ents work on th eir sum maries,

stud ents are not alwa ys able to apply the feedback they receive from either State

the Essence or Summary Street . For it is one thing to kn ow wh at the p roblems are

with you r wr iting and quite another to know how to fix them. Nor is it enough to

be told w hat th e processes are. Most young w riters still need mor e explicit

instruction on h ow to make ap prop riate changes, on h ow to app ly strategies like

combining sentences or find ing a generalizing term in their own su mm aries.

Taking these lessons back to th e classroom, th e teachers (CM & RL) revised th e

w ay they w ere teaching su mm arizing, d evoting mo re time to explicit instru ction

of the skill at a more concrete level than p reviously. Their new app roach involves

discussing all phases of planning and wr iting a sum mary u sing a comm on text:




• discussing the p urp ose for su mm arizing information: to d evelop the

ability to share kn owledge w ith others an d to come to a deep

un derstand ing of comp lex information;

• mod eling h ow to d ifferentiate main id eas from d etails in highlighting, to

counteract stud ents' tend ency to select all;

• developing together an ou tline of the most imp ortant information;

• developing an und erstanding of wh at makes a good summ ary: a focus on

main ideas, good organization, low redu nd ancy and making it interesting

for other read ers;

• showing m odels of good an d poor su mm aries and jointly identifying the

specific properties of each;

• mod eling strategies for iden tifying ap pr op riate content to includ e, for

collapsing and reorganizing information;

• providing more op portu nities for stud ents to collaborate in p lanning,

writing and revising their summ aries.

The goal of this instruction is to develop a sh ared vision am ong th e

classroom comm un ity of the steps involved in sum marizing an d a common

language to talk about and improve w riting. This shared k now ledge becomes the

basis for rich discussion of matters involving bo th content an d style, both in the

classroom and between individual stud ents. We find that the sum marization tool

pr ovides a natu ral setting for collaborative pr oblem solving by helpin g stud ents

identify specific problems w ith their w riting on th eir own . Teachers and

researchers observed many instances of students spontaneously discussing their

work - about wh ether a given idea really w as important enou gh to includ e, how

information could be combined and collapsed - evidence that stud ents were




thinking a bou t the content of their sum ma ries at a d eep and critical level. We

believe that such discussions came abou t throu gh the shared learning experience

that includ ed more p ractice in w riting, d iscussions and mod eling of

sum ma rization strategies in the classroom, collaboration, and the u se of the new

sum marization tool.

Factors affecting use of the system

The system m ay have p otential not only for giving stud ents more op portu nities

to w rite, bu t also for low ering class size for limited times. On ce stud ents are

familiar with the system, small grou ps, sup ervised by a parap rofessional

edu cator or adu lt volun teer, could work independ ently on their sum maries in

the comp uter lab, while the others work w ith the teacher in the classroom .

Rotating stu dents in the lab and classroom has the d ual ad vantage of giving

them m ore practice at w riting – w ith m ore feedback than th ey norm ally w ould

receive – and more op portu nity to interact with the teacher on an ind ividu al or

small group basis. H ow ever, this op timistic scenario d epen d s on factors that are

not en tirely u nd er the control of either teachers or th e researchers. Nam ely, a

per son w ith techn ical skills mu st be available at the school to deal with n ot

infrequent interrup tions in n etwork connections, through wh ich Summary Street

op erates. The lack of a techn ology expert in a school places the bu rd en of

maintaining the compu ters on teachers, whose highest pr iority m ust be on

planning an d executing a rich curriculum and mentoring stud ents. In fact this

ma y be one of the ma jor obstacles to ad opting th e sum mar ization tool as broadly

as intend ed. It is essential, how ever, that any comp onen ts of the system th at

ma ke it u nreliable or inconvenient to use (e.g., occasional p roblems w ith log-in

failures or losing stu den t work ) must be corrected.




Sug gestions for using th e system

The amoun t of time requ ired for teachers and students to learn the system is not

un reasonable, but the time comm itment mu st be mad e. Classroom instruction on

sum ma rization mu st precede introd uction of the software. It takes about 20-30

min. to demonstrate Summary Street to first-time u sers, w hich shou ld be follow ed

by ad equate time for students to try it out. Thus, teachers shou ld p lan one-to-two

class period s to familiarize stud ents w ith the software, because they w ill need

guidan ce initially to u nd erstand the software an d how to use the feedback.

How ever, once stud ents have this framework, they can u se the feedback from

Summary Street on their ow n. If the software is introd uced at the beginn ing of the

school year, midd le-school stud ents cou ld continue to u se it with m inimal

assistance from a p ara-edu cator or volunteer in all content areas. Ou r broad ly

stated goal for this summ arization tool was to p rovide stud ents with more

op por tun ities to write. Yet there are a variety of w ays it can be u sed to enh ance

the effectiveness of teachers’ limited resou rces. Writing a su mm ary n eed n ot be

viewed as an end in itself, but rath er as a step toward s another task, such as

pr epar ing a report or p resentation, or taking a test. The goal of a sum mar ization

assignm ent d oes not have to be to p rovide a finished prod uct every time; instead

teachers might u se a printout of Summary Street feed back as a q uick, yet

authentic way to check stud ents’ curren t u nd erstand ing of a topic.

In sum , although we aren’t quite there yet, we app ear to be on the right

track towar d m eeting ou r goals. Teachers cannot be th e sole dispen sers of

know ledge and feed back. Their job is to give stud ents tools that allow th em to

evaluate critically the content th ey learn, and to comm un icate pr oficiently w ith

others. Summary Street is one mor e tool that can help accomplish this goal.




References

Foltz, P. W. (Guest Ed.) (1998). Qu antitative ap pr oaches to sem antic know ledge

representations [Special Issue]. Discourse Processes , 25, 127-363.

Land au er, T. K. (1998). Learning and repr esenting verb al mean ing: The Latent

Seman tic analysis th eory. Current Directions in Psychological Science , 7 , 161-

164.

Land au er, T. K., and Du mais, S. T. (1997). A solu tion to P lato’s pr oblem: The

Latent Seman tic Analysis theory of th e acquisition, ind uction, and

representation of know ledge. Psychological Review , 104 , 211-240.

Land au er, T. K., Foltz, P. W., and Lah am , D. (1998). An In trod uction to Latent

Seman tic Analysis [Special Issue]. Discourse Processes , 25 , 259-284.

Land au er, T. K., and Psotka, J. (in p ress). Simu lating text u nd erstand ing for

edu cational ap plications w ith Latent Seman tic Analysis: Introd uction to

LSA. Interactive Learning Environments .




Author Note

*The autho rs wou ld like to acknow ledge the sup por t of the CSEP Program of the

McDonnell Foun d ation. We also thank the sixth-grad e stud ents from Platt

Midd le School of Bould er Colorado for th eir participation in this wor k.

We ded icate this pap er to the memory of Ann Brown , in gratitud e and profound

admiration.

** The mem bers of the LSA Research Grou p ar e Walter Kintsch, Thomas

Land au er, Rogerio De Pau la, Eileen Kintsch, Darr ell Laham , Maur een Schreiner,

Gerry Stahl, and Dave Steinhar t.




Figure Caption s

Figu re 1. Screen shot show ing a stud ent sum mar y and first-level feedback from

State the Essence : overall score, word length, sections with a d equate content

coverage, and sections with m issing information.

Figu re 2. Screen shot show ing a stud ent sum mar y and first level of conten t

feed back from Summary Street . Green h orizontal bars indicate to what extent the

content of each section w as adequ ately covered in th e sum mar y, with the

thresh old ind icated by a black v ertical line. The vertical bar o n th e left is a length

ind icator, shown in green if the sum mar y is w ithin the prescribed limits and red

if not .

Figu re 3. Mean gr ad es of Ancient Civilizations su mm aries by condition.

Figure 4. Mean grades of Heart and Lungs sum maries by cond ition.




6.5

7

7.5

8

8.5

9

G r a d e

Maya Aztec Inca

Traditional

Summary Street




2

2.5

3

3.5

4

4.5

G r a d e

Lungs Heart

WordProcessor

SummaryStreet

summarizing skills

Documents