1 matthieu hermet, stan szpakowicz automated analysis of students free-text answers for computer-...
TRANSCRIPT
1
Matthieu Hermet, Stan Szpakowicz
Automated Analysis of Students’ Free-text
Answers for Computer-Assisted Assessment
University of Ottawa, Canada
2
CAA for CALL
• To address the specificity of CALL…→ where student material contains
syntactic and orthographic errors• …with minimal pre-encoded material :
– Content validation : simple– Form validation : difficult
→ A good case for automating based on Natural Language Processing
3
Text comprehension
• The uOttawa project: CALL solutions for helping French-as-a-Second-Language students to enhance autonomous reading comprehension
→ master the structure of text in order to understand the author’s discursive intention
→ guess the meaning of unknown words
→ develop reformulation and synthesis capabilities
4
DidaLect
…is a FSL tool aimed at teaching autonomous reading skill (designed for intermediate- and advanced-level students)
• Is an instance of eLearning Intelligent Tutoring System:– adaptation to individual student’s skills and
agenda– access to external resources (dictionaries)– built to reflect the cognitive concerns such as
matching feedback to the student’s behaviour
5
Intelligence in DidaLect
• DidaLect begins its operation with a placement test to determine a student’s initial level: – varying order of questions to pick up the best of a
student’s skill– the implementation includes fuzzy logic methods
• A separate element of DidaLect is the processing of free-text answers:– need of a robust CAA component– a trade-off between symbolic processing and Machine
Learning techniques
6
Free-text answer assessment
• The problem is to know in advance what material to expect in student answers.
• Usually implemented as a classification problem: a student answer must match reference answer(s).
→ Size and form of reference material affects the process
• Here, a reference answer is the text itself→ A case for trying symbolic processing using
techniques of Computational Linguistics
7
Expected limitations
• No possibility of modifying the size and form of the reference material, except by automatic processing to control reformulation.
• Therefore, this only works for limited forms of questions.– Strong need to ground selection of
questions in a firm didactic theory.– Questions on texts for Text
Comprehension (didactics offers a classification of question types).
8
Question types 1
• Text-Explicit: based on a single sentence
« d'habitude, l'hermaphrodisme frappe surtout les mâles, qu'elle dote de simulacres d'appareils génitaux féminins. »
Q : Qu’arrive-t-il aux ours mâles lorsqu’ils sont frappés d’hermaphrodisme ?
Ex R : « Ils ont les génitaux féminins »
• Text-Implicit : based on two co-referenced sentences
« le détecteur de décélération situé à l'avant du véhicule génère instantanément un courant électrique, qui déclenche une amorce, qui elle-même enflamme un mélange allumeur. Ce dernier met finalement le feu à l'agent propulseur responsable du gonflement du coussin. »
Q : Quelle est la réaction en chaîne qui se produit lorsque survient un impact ?
9
Question types 2
• Identification, cause-effect, goal, comparison, definition, instrumental…
• These categories express linguistically through lexical connectors
• Goal : for, so that, in order to…• Cause–Effect : because, therefore…
• So, the control of reformulation can be automated
10
Processing
1. Find lexical differences between the student’s answer S and reference R
2. Parse S and R, produce dependency relations
3. Process different words (using a dictionary) to detect synonyms
4. Control of syntax in S
5. Control of reformulation in S wrt R
11
Tools
• A robust parser that enables partial recovery from errors in student’s answer
• A dictionary of synonyms
• A derivational dictionary
• Locally derived resources:– State and action verbs– Ensemble of typical errors, set of
syntactic and reformulation structures
12
Semantics and synonyms
• Examine word set differences and commonalities in search for:
– Common words– Reformulated words– Different words
• Detect synonyms accross parts of speech:
1. Derive forms for a word lemma2. Search synonyms for each form and
look for a match in Word Sets
13
Syntax and reformulation
• Correct syntactical structures to verify syntax of student’s answer
• Lexicalized reformulation structures to verify discourse conformity
Ex : pollution has increased with the rise of transportationQ : Why has pollution increased ?Ans : With the rise of transportation is partially wrong→ Because of the rise of transportationOR it has increased due to the rise of transportationETC.
14
Parsing and tree-building
S : Le cardio-vasculaire d'un rat s'approche à une personne humain.
SUBJ(<approche^approcher:53>, <cardio-vasculaire^cardio-vasculaire:48>)
OBJ(<approche^approcher:53>, <personne^personne:56>)
VMOD_POSIT1(<approche^approcher:53>, <une^un:55>)
NMOD_POSIT1(<cardio-vasculaire^cardio-vasculaire:48>, <rat^rat:51>)
PREPOBJ(<une^un:55>, <à^à:54>)
PREPOBJ(<rat^rat:51>, <d'^de:49>)
DETERM(<rat^rat:51>, <un^un:50>)
The above are incrementally recomposed, based on lexical selection that maximizes promise of discovering material which diverges from R. That material is processed in parallel, in a similar fashion:
SUBJ( <OBJ(<approcher>, <NMOD(<être>, <humain>)>)>,
<NMOD(<NMOD(<système>, <cardio-vasculaire>)>, <rat>)>)
15
Main types of heuristics
• To address syntactic correctness and/or equivalence between S and R: the same sense but different structures
→ bank of typical errors and correct structures• To address discursive variations, detected as
supplementary material→ bank of state and action verbs: action verbs
must be present, possibly reformulated• To address, partially, errors in S→ word replacement to relaunch parsing when
stopped due to lexical mistakes
16
Reformulation rules
• Examples :• Abstraction : incidence sur le temps de gestation →
incidence sur la possibilité d’avoir une gestation écourtée (words like fact, chance, etc.)
• Cause-Effect : Le plasma augmente et dilue les paramètres chimiques → L’augmentation du plasma dilue les paramètres chimiques
• Is-A : Le rat est un animal qui + S → Le rat + S• Attribute : Le rat possède un système cardio-
vasculaire → Le système C-V du rat
17
Assessment
• Must give student feedback on:– Agreement and orthography– Syntax: signal errors and
provides correction via display of a correct structure
– Semantics: signals error and provides admissible words
– Completeness of content with respect to R
18
Example
R: Et puisque le rat est un animal qui possède un système cardio-vasculaire très semblable à celui de l’humain, il est donc permis de tirer les mêmes conclusions pour l’humain.
Q: Pourquoi peut-on tirer les mêmes conclusions pour l'humain et pour le rat ?
S: Le cardio-vasculaire d’un rat s’approche à une personne humain.
Start by creating wordlists :
1. Words of S absent in R→ s’approcher, personne2. Words of R absent in S→ animal, posséder, système,
semblable3. Common words→ rat, cardio-vasculaire, humain
19
Parse (partial output)S:
SUBJ(<approche^approcher:53>, <cardio-vasculaire^cardio-vasculaire:48>)
OBJ(<approche^approcher:53>, <personne^personne:56>)
VMOD_POSIT1(<approche^approcher:53>, <une^un:55>)
NMOD_POSIT1(<cardio-vasculaire^cardio-vasculaire:48>, <rat^rat:51>)
PREPOBJ(<une^un:55>, <à^à:54>)
PREPOBJ(<rat^rat:51>, <d'^de:49>)
DETERM(<rat^rat:51>, <un^un:50>)
R:
SUBJ(<est^être:2>,<rat^rat:1>)
OBJ_SPRED(<est^être:2>, <animal^animal:4>)
OBJ(<possède^posséder:6>, <système^système:8>)
COREF_REL(<animal^animal:4>, <qui^qui:5>)
NMOD_POSIT1(<système^système:8>, <cardio-vasculaire^cardio-vasculaire:9>)
NMOD_POSIT1(<système^système:8>, <semblable^semblable:11>)
NMOD_POSIT1(<celui^celui:13>, <humain^humain:16>)
ADJMOD(<semblable^semblable:11>, <celui^celui:13>)
PREPOBJ(<humain^humain:16>, <de^de:14>)
PREPOBJ(<celui^celui:13>, <à^à:12>)
DETERM(<système^système:8>, <un^un:7>)
DETERM(<animal^animal:4>, <un^un:3>)
DETERM_DEF(<rat^rat:1>, <le^le:0>)
CONNECT_REL(<possède^posséder:6>, <qui^qui:5>)
20
Comparison (partial)
SUBJ(< OBJ(<approche^approcherapproche^approcher:53>,<personne^personnepersonne^personne:56>) >,<NMOD_POSIT1(<cardio-vasculaire^cardio-cardio-vasculaire^cardio-vasculairevasculaire:48>,<rat^ratrat^rat:51>)>)
SUBJ(<OBJ_SPRED(<est^être:2>,< COREF_REL(<animal^animal:4>,< CONNECT_REL(< OBJ(<possède^posséder:6>,< NMOD_POSIT1(< NMOD_POSIT1(<système^système:8>,<cardio-vasculaire^cardio-cardio-vasculaire^cardio-vasculairevasculaire:9>) >,<ADJMOD(<semblable^semblablesemblable^semblable:11>,< NMOD_POSIT1(<celui^celui:13>,<humain^humainhumain^humain:16>) >)>)>) >,<qui^qui:5>) >) >)>,<rat^ratrat^rat:1>)
• Consider structures to assess for syntactic correctness• Heuristics to put some structures into equivalence :→ Here, «Le SCV du rat» and «Le rat est un animal qui possède un
SCV» are equivalent, but expressed in different syntactic structures
21
Synonyms
• - <RESULTS>• - <DEF>• - <E L="fr">• <W>semblable</W> • <SW
ENC="n">00730065006D0062006C00610062006C0065</SW>
• <P>adj.</P> • </E>• - <DF N="1">• <W>qui ressemble à; comparable,
similaire.</W> • </DF>• </DEF>• - <DEF>…..• - <WD L="fr" W="2">• <W L="fr">approchant</W> • <SW
ENC="n">0061007000700072006F006300680061006E0074</SW>
• </WD>
To retrieve a synonymy relation :
1. Produce derivations for all words in List 1
2. Find matches in a synonymy basis under entries for words of List 2
3. The search process can be repeated at most once, using <DEF> lexemes
→ semblable = approchantOR→ ressembler à = s’approcher de4. In this way we can catch both
synonyms and attached prepositions.
22
Conclusions and future work
• Automation is possible, with 2 main restrictions :– « bad faith » answers– Lexical errors based on homonymy
→ as long as S contains elements of answer, S can be evaluated
• Future Work : to assemble the parts through software engineering !