on arabic stop-words: a comprehensive list and a dedicated ...madamira asl analyzer 90.17 97.85....
TRANSCRIPT
![Page 1: On Arabic Stop-words: A Comprehensive List and a Dedicated ...MADAMIRA ASL analyzer 90.17 97.85. ICALP 2019, 16-17 Octobre- Nancy 21 •grammatical category •clitized forms Default](https://reader035.vdocument.in/reader035/viewer/2022071006/5fc3e2e7172860376c7f7333/html5/thumbnails/1.jpg)
Driss Namly, Karim Bouzoubaa, Rachida Tajmout, Ali Laadimi
On Arabic Stop-words:A Comprehensive List and a
Dedicated Morphological Analyzer
ICALP 2019 16-17 Octobre- Nancy
ALELM Team
![Page 2: On Arabic Stop-words: A Comprehensive List and a Dedicated ...MADAMIRA ASL analyzer 90.17 97.85. ICALP 2019, 16-17 Octobre- Nancy 21 •grammatical category •clitized forms Default](https://reader035.vdocument.in/reader035/viewer/2022071006/5fc3e2e7172860376c7f7333/html5/thumbnails/2.jpg)
Stop-words
ICALP 2019, 16-17 Octobre- Nancy 2
• Common words that frequently appear
• Isolated
o no information
• In a sentence
o grammatical and syntactical function
Definition
On arabic stop-words: a comprehensive list and a dedicated morphological analyzer
![Page 3: On Arabic Stop-words: A Comprehensive List and a Dedicated ...MADAMIRA ASL analyzer 90.17 97.85. ICALP 2019, 16-17 Octobre- Nancy 21 •grammatical category •clitized forms Default](https://reader035.vdocument.in/reader035/viewer/2022071006/5fc3e2e7172860376c7f7333/html5/thumbnails/3.jpg)
• Text categorization
• Information extraction
• Text summarization
• Search engine indexing
• ...
IR +
• corpus size --
• Saves space in indexes
• Increases accuracy without damaging retrieval efficiency
Removal
Stop-words exploitation & removal
ICALP 2019, 16-17 Octobre- Nancy 3On arabic stop-words: a comprehensive list and a dedicated morphological analyzer
![Page 4: On Arabic Stop-words: A Comprehensive List and a Dedicated ...MADAMIRA ASL analyzer 90.17 97.85. ICALP 2019, 16-17 Octobre- Nancy 21 •grammatical category •clitized forms Default](https://reader035.vdocument.in/reader035/viewer/2022071006/5fc3e2e7172860376c7f7333/html5/thumbnails/4.jpg)
• Corpus-dependent
• Rely on the frequency feature
• Don’t consider all clitized forms
• The grammatical category default
• Don’t consider all clitized forms
Static stop-words list Morphological analysis
Analysis
List
Stop-words identification
To remove these words
we must identify them
ICALP 2019, 16-17 Octobre- Nancy 4On arabic stop-words: a comprehensive list and a dedicated morphological analyzer
contains هو
excludes clitized forms ،وهو، فهو...
detect هو (POS=particle)ignore الذي (POS=noun)
![Page 5: On Arabic Stop-words: A Comprehensive List and a Dedicated ...MADAMIRA ASL analyzer 90.17 97.85. ICALP 2019, 16-17 Octobre- Nancy 21 •grammatical category •clitized forms Default](https://reader035.vdocument.in/reader035/viewer/2022071006/5fc3e2e7172860376c7f7333/html5/thumbnails/5.jpg)
Objectives
ICALP 2019, 16-17 Octobre- Nancy 5
A comprehensive Arabic rule-based stop-words list01
02A contextual stop-words analyzer
On arabic stop-words: a comprehensive list and a dedicated morphological analyzer
![Page 6: On Arabic Stop-words: A Comprehensive List and a Dedicated ...MADAMIRA ASL analyzer 90.17 97.85. ICALP 2019, 16-17 Octobre- Nancy 21 •grammatical category •clitized forms Default](https://reader035.vdocument.in/reader035/viewer/2022071006/5fc3e2e7172860376c7f7333/html5/thumbnails/6.jpg)
ICALP 2019, 16-17 Octobre- Nancy 6
• Shereen Khoja (168)
• Ibrahim Abu El-Khair(1529)
• Walaa Medhat et al. (1061)
• Alajmi et al. (200)
• Stop words project(162)
• Ranks NL(102)
State of the art
• BAMA• ALKHALIL• MADAMIRA• CALIMA-star
Morphological analysisStop-word lists
On arabic stop-words: a comprehensive list and a dedicated morphological analyzer
![Page 7: On Arabic Stop-words: A Comprehensive List and a Dedicated ...MADAMIRA ASL analyzer 90.17 97.85. ICALP 2019, 16-17 Octobre- Nancy 21 •grammatical category •clitized forms Default](https://reader035.vdocument.in/reader035/viewer/2022071006/5fc3e2e7172860376c7f7333/html5/thumbnails/7.jpg)
02
01
Proposed approach
ICALP 2019, 16-17 Octobre- Nancy 7On arabic stop-words: a comprehensive list and a dedicated morphological analyzer
A
A
B
B
Simple Stop-words list
Complex Stop-words list
Context-free analysis
Contextual analysis
إلى
84 clitized formsلي يكما، ك، أفإليكم، إل أإليك، فإليك، ل
، وإليه، إل ...يه، إليها، فإلىأوإليكن
جاء الولد من المدرسة
المدرسة[من ، من ]جاء الولد
جاء الولد من المدرسة
المدرسة[من ، من ]جاء الولد
![Page 8: On Arabic Stop-words: A Comprehensive List and a Dedicated ...MADAMIRA ASL analyzer 90.17 97.85. ICALP 2019, 16-17 Octobre- Nancy 21 •grammatical category •clitized forms Default](https://reader035.vdocument.in/reader035/viewer/2022071006/5fc3e2e7172860376c7f7333/html5/thumbnails/8.jpg)
02
01
Proposed approach
ICALP 2019, 16-17 Octobre- Nancy 8On arabic stop-words: a comprehensive list and a dedicated morphological analyzer
A
A
B
B
Simple Stop-words list
Complex Stop-words list
Context-free analysis
Contextual analysis
![Page 9: On Arabic Stop-words: A Comprehensive List and a Dedicated ...MADAMIRA ASL analyzer 90.17 97.85. ICALP 2019, 16-17 Octobre- Nancy 21 •grammatical category •clitized forms Default](https://reader035.vdocument.in/reader035/viewer/2022071006/5fc3e2e7172860376c7f7333/html5/thumbnails/9.jpg)
ICALP 2019, 16-17 Octobre- Nancy 9
1.A- Simple Stop-words list design
Simple Arabic stop-words Lexicon
الكلمات المستبعدةمعجم البسيطةالعربية
Native particles
حروف خالصة
- Conjonction حروف العطف
- Conditional حروف الشرط
- …
- Interrogative particles حروف االستفهام
Special nouns
سماءألحروف بصفات ا
- Relative pronoun األسماء الموصولة
- Personal pronoun ضمائر الرفع المنفصلة
- …
- Interrogative pronouns أسماء االستفهام
Special verbs
فعالألحروف بصفات ا
- Certainty verbs أفعال اليقين
- Transposition verbs أفعال التحويل
- …
- Beginning verbs أفعال الشروع
...،أل،ىإل ،إل ،لكن
...، من، ما، التي، أنتن
...ليس، كان،
On arabic stop-words: a comprehensive list and a dedicated morphological analyzer
![Page 10: On Arabic Stop-words: A Comprehensive List and a Dedicated ...MADAMIRA ASL analyzer 90.17 97.85. ICALP 2019, 16-17 Octobre- Nancy 21 •grammatical category •clitized forms Default](https://reader035.vdocument.in/reader035/viewer/2022071006/5fc3e2e7172860376c7f7333/html5/thumbnails/10.jpg)
02
01
Proposed approach
ICALP 2019, 16-17 Octobre- Nancy 10On arabic stop-words: a comprehensive list and a dedicated morphological analyzer
A
A
B
B
Simple Stop-words list
Complex Stop-words list
Context-free analysis
Contextual analysis
![Page 11: On Arabic Stop-words: A Comprehensive List and a Dedicated ...MADAMIRA ASL analyzer 90.17 97.85. ICALP 2019, 16-17 Octobre- Nancy 21 •grammatical category •clitized forms Default](https://reader035.vdocument.in/reader035/viewer/2022071006/5fc3e2e7172860376c7f7333/html5/thumbnails/11.jpg)
ProClitic Simple Stop-Word EnClitic
Complex Stop-Word
ICALP 2019, 16-17 Octobre- Nancy 11
1.B- Complex Stop-words list design
فب غير هم
فبغيرهم
On arabic stop-words: a comprehensive list and a dedicated morphological analyzer
![Page 12: On Arabic Stop-words: A Comprehensive List and a Dedicated ...MADAMIRA ASL analyzer 90.17 97.85. ICALP 2019, 16-17 Octobre- Nancy 21 •grammatical category •clitized forms Default](https://reader035.vdocument.in/reader035/viewer/2022071006/5fc3e2e7172860376c7f7333/html5/thumbnails/12.jpg)
ICALP 2019, 16-17 Octobre- Nancy
1.B- Complex Stop-words list design
Arabic rules
ConjunctionPrepositionsAccusative ParticlesFuture particleDefinition particle…
حروف العطفحروف الجر
حروف النصبسين المستقبل
التعريف…
ProClitic
connected pronounspreventionAscribing…
ةالضمائر المتصلنون الوقايةياء النسبة
…
EnClitic
Word type
ParticleNounVerb
حرفاسمفعل
Arabic rules
حرف+ ال ضم+ اسم موصول
On arabic stop-words: a comprehensive list and a dedicated morphological analyzer 12
![Page 13: On Arabic Stop-words: A Comprehensive List and a Dedicated ...MADAMIRA ASL analyzer 90.17 97.85. ICALP 2019, 16-17 Octobre- Nancy 21 •grammatical category •clitized forms Default](https://reader035.vdocument.in/reader035/viewer/2022071006/5fc3e2e7172860376c7f7333/html5/thumbnails/13.jpg)
02
01
Proposed approach
ICALP 2019, 16-17 Octobre- Nancy 13On arabic stop-words: a comprehensive list and a dedicated morphological analyzer
A
A
B
B
Simple Stop-words list
Complex Stop-words list
Context-free analysis
Contextual analysis
![Page 14: On Arabic Stop-words: A Comprehensive List and a Dedicated ...MADAMIRA ASL analyzer 90.17 97.85. ICALP 2019, 16-17 Octobre- Nancy 21 •grammatical category •clitized forms Default](https://reader035.vdocument.in/reader035/viewer/2022071006/5fc3e2e7172860376c7f7333/html5/thumbnails/14.jpg)
diacritized stop-words list named ASL (Arabic Stop-words List) :
ICALP 2019, 16-17 Octobre- Nancy 14
1- Arabic Stop-words List (ASL)
Enclitic Simple stop-word Proclitic Complex stop-word Category
- maybe عل And And maybeول ولعل NP
- that ذلك like like thatك كذلك SN
- you are تكونين will we will beس ين ستكون SV
you no more thanك غير - no more than you غيرك SN
you maybeك عسى - Maybe you عساك SV
إلى they toهم And is And is to theyأو مأوإليه NP
Native Particles Special nouns Special verbs Total
Simple list 84 263 1,784 2,131
Complex list 1,590 9,627 27,936 39,153
On arabic stop-words: a comprehensive list and a dedicated morphological analyzer
Khoja (168)
Abu El-Khair(1529)
Alajmi (200)
، تما، كانت كنت، كنا، كن
...كانوا، نكون،
![Page 15: On Arabic Stop-words: A Comprehensive List and a Dedicated ...MADAMIRA ASL analyzer 90.17 97.85. ICALP 2019, 16-17 Octobre- Nancy 21 •grammatical category •clitized forms Default](https://reader035.vdocument.in/reader035/viewer/2022071006/5fc3e2e7172860376c7f7333/html5/thumbnails/15.jpg)
02
01
Proposed approach
ICALP 2019, 16-17 Octobre- Nancy 15On arabic stop-words: a comprehensive list and a dedicated morphological analyzer
A
A
B
B
Simple Stop-words list
Complex Stop-words list
Context-free analysis
Contextual analysis
![Page 16: On Arabic Stop-words: A Comprehensive List and a Dedicated ...MADAMIRA ASL analyzer 90.17 97.85. ICALP 2019, 16-17 Octobre- Nancy 21 •grammatical category •clitized forms Default](https://reader035.vdocument.in/reader035/viewer/2022071006/5fc3e2e7172860376c7f7333/html5/thumbnails/16.jpg)
ICALP 2019, 16-17 Octobre- Nancy 16
Contextual analysisContext-free analysis
Clitics Lexicon
ASL
Input text
Pre-
processing
Potential
clitics
Potential
stop-words
HMM
Smoothing
Viterbi
Analyzed
text
Model
2- Stop-words analyzer design
On arabic stop-words: a comprehensive list and a dedicated morphological analyzer
![Page 17: On Arabic Stop-words: A Comprehensive List and a Dedicated ...MADAMIRA ASL analyzer 90.17 97.85. ICALP 2019, 16-17 Octobre- Nancy 21 •grammatical category •clitized forms Default](https://reader035.vdocument.in/reader035/viewer/2022071006/5fc3e2e7172860376c7f7333/html5/thumbnails/17.jpg)
Contextual analysisContext-free analysis
ICALP 2019, 16-17 Octobre- Nancy 17
2- Stop-words analyzer design
On arabic stop-words: a comprehensive list and a dedicated morphological analyzer
![Page 18: On Arabic Stop-words: A Comprehensive List and a Dedicated ...MADAMIRA ASL analyzer 90.17 97.85. ICALP 2019, 16-17 Octobre- Nancy 21 •grammatical category •clitized forms Default](https://reader035.vdocument.in/reader035/viewer/2022071006/5fc3e2e7172860376c7f7333/html5/thumbnails/18.jpg)
ICALP 2019, 16-17 Octobre- Nancy 18
Stop-words list evaluation
78,477 tokens15,684 uniqueExist
هو 1
زيارة 0
… …
Total
68
On arabic stop-words: a comprehensive list and a dedicated morphological analyzer
Khoja Abu El-Khair List 1 Ranks NL ASL
144 552 98 68 1,247
Tagged evaluation corpus
Accuracy
Compute
![Page 19: On Arabic Stop-words: A Comprehensive List and a Dedicated ...MADAMIRA ASL analyzer 90.17 97.85. ICALP 2019, 16-17 Octobre- Nancy 21 •grammatical category •clitized forms Default](https://reader035.vdocument.in/reader035/viewer/2022071006/5fc3e2e7172860376c7f7333/html5/thumbnails/19.jpg)
ICALP 2019, 16-17 Octobre- Nancy 19
Context-free evaluation
Corpus(manually annotated )
1,628 words 419 stop-words
Analysis
ال تنفرد المدينة العتيقة بمؤهالت متنوعة و
نة محدودة، انصهرت فيها روافد ثقافية متباي
ميزتها عن غيرها من المدن
محدودة،[ الو ]تنفرد المدينة العتيقة بمؤهالت متنوعة
[ ع ن ، عن، ع ن ]روافد ثقافية متباينة ميزتها [ فيها]انصهرت
ها] ها، غ ير ، من، [ ]غي ر ن ن من م ن ، م المدن[ ، م
Accuracy
𝟗𝟒. 𝟏𝟏%
On arabic stop-words: a comprehensive list and a dedicated morphological analyzer
BAMA Alkhalil 1 Alkhalil 2 CALIMA-star ASL analyzer
91.09 88.70 93.98 93.12 99.94
![Page 20: On Arabic Stop-words: A Comprehensive List and a Dedicated ...MADAMIRA ASL analyzer 90.17 97.85. ICALP 2019, 16-17 Octobre- Nancy 21 •grammatical category •clitized forms Default](https://reader035.vdocument.in/reader035/viewer/2022071006/5fc3e2e7172860376c7f7333/html5/thumbnails/20.jpg)
ICALP 2019, 16-17 Octobre- Nancy 20
Contextual evaluation
Corpus(manually annotated )
1,628 words 419 stop-words
Analysis
ال تنفرد المدينة العتيقة بمؤهالت متنوعة و
نة محدودة، انصهرت فيها روافد ثقافية متباي
ميزتها عن غيرها من المدن
محدودة،[ الو ]تنفرد المدينة العتيقة بمؤهالت متنوعة
[ ، عن، ع ن ع ن ]روافد ثقافية متباينة ميزتها [ فيها]انصهرت
ها] ها،غي ر ن [ ]غ ير ن م ن ، من، من، م المدن[ ، م
Accuracy
𝟖𝟐. 𝟑𝟓%
On arabic stop-words: a comprehensive list and a dedicated morphological analyzer
MADAMIRA ASL analyzer
90.17 97.85
![Page 21: On Arabic Stop-words: A Comprehensive List and a Dedicated ...MADAMIRA ASL analyzer 90.17 97.85. ICALP 2019, 16-17 Octobre- Nancy 21 •grammatical category •clitized forms Default](https://reader035.vdocument.in/reader035/viewer/2022071006/5fc3e2e7172860376c7f7333/html5/thumbnails/21.jpg)
ICALP 2019, 16-17 Octobre- Nancy 21
• grammatical category
• clitized forms
Default
• comprehensive Arabic rule-based stop-words list
• contextual stop-words analyzer
Designed
• 1,247 Vs 552 (2nd list)
• context-free 99.94%
• contextual 97.85%
Evaluation
Conclusion
• Static list
• Morphological analysis
Identification
On arabic stop-words: a comprehensive list and a dedicated morphological analyzer
![Page 22: On Arabic Stop-words: A Comprehensive List and a Dedicated ...MADAMIRA ASL analyzer 90.17 97.85. ICALP 2019, 16-17 Octobre- Nancy 21 •grammatical category •clitized forms Default](https://reader035.vdocument.in/reader035/viewer/2022071006/5fc3e2e7172860376c7f7333/html5/thumbnails/22.jpg)
Driss Namly, Karim Bouzoubaa, Rachida Tajmout, Ali Laadimi
On Arabic Stop-words: A Comprehensive List and aDedicated Morphological Analyzer
ICALP 2019 16-17 Octobre- Nancy
ALELM Team
Thank you