a machine learning approach for identifying disease-treatment relations in short texts

A Machine Learning Approach for IdentifyingDisease-Treatment Relations in Short Texts

Oana Frunza, Diana Inkpen, and Thomas Tran, Member, IEEE

Abstract—The Machine Learning (ML) field has gained its momentum in almost any domain of research and just recently has become

a reliable tool in the medical domain. The empirical domain of automatic learning is used in tasks such as medical decision support,

medical imaging, protein-protein interaction, extraction of medical knowledge, and for overall patient management care. ML is

envisioned as a tool by which computer-based systems can be integrated in the healthcare field in order to get a better, more efficient

medical care. This paper describes a ML-based methodology for building an application that is capable of identifying and disseminating

healthcare information. It extracts sentences from published medical papers that mention diseases and treatments, and identifies

semantic relations that exist between diseases and treatments. Our evaluation results for these tasks show that the proposed

methodology obtains reliable outcomes that could be integrated in an application to be used in the medical care domain. The potential

value of this paper stands in the ML settings that we propose and in the fact that we outperform previous results on the same data set.

Index Terms—Healthcare, machine learning, natural language processing.

Ç

1 INTRODUCTION

PEOPLE care deeply about their health and want to be, nowmore than ever, in charge of their health and healthcare.

Life is more hectic than has ever been, the medicine that ispracticed today is an Evidence-Based Medicine (hereafter,EBM) in which medical expertise is not only based on years ofpractice but on the latest discoveries as well. Tools that canhelp us manage and better keep track of our health such asGoogle Health1 and Microsoft HealthVault2 are reasons and factsthat make people more powerful when it comes to healthcareknowledge and management. The traditional healthcaresystem is also becoming one that embraces the Internet andthe electronic world. Electronic Health Records (hereafter,EHR) are becoming the standard in the healthcare domain.Researches and studies show that the potential benefits ofhaving an EHR system are3:

Health information recording and clinical data reposi-

tories—immediate access to patient diagnoses, allergies,

and lab test results that enable better and time-efficient

medical decisions;Medication management—rapid access to information

regarding potential adverse drug reactions, immunizations,

supplies, etc;Decision support—the ability to capture and use quality

medical data for decisions in the workflow of healthcare; and

Obtain treatments that are tailored to specific healthneeds—rapid access to information that is focused oncertain topics.

In order to embrace the views that the EHR system has,we need better, faster, and more reliable access to informa-tion. In the medical domain, the richest and most usedsource of information is Medline,4 a database of extensivelife science published articles. All research discoveries comeand enter the repository at high rate (Hunter and Cohen[12]), making the process of identifying and disseminatingreliable information a very difficult task. The work that wepresent in this paper is focused on two tasks: automaticallyidentifying sentences published in medical abstracts (Med-line) as containing or not information about diseases andtreatments, and automatically identifying semantic relationsthat exist between diseases and treatments, as expressed inthese texts. The second task is focused on three semanticrelations: Cure, Prevent, and Side Effect.

The tasks that are addressed here are the foundation of aninformation technology framework that identifies anddisseminates healthcare information. People want fastaccess to reliable information and in a manner that issuitable to their habits and workflow. Medical care relatedinformation (e.g., published articles, clinical trials, news,etc.) is a source of power for both healthcare providers andlaypeople. Studies reveal that people are searching the weband read medical related information in order to beinformed about their health. Ginsberg et al. [10] show howa new outbreak of the influenza virus can be detected fromsearch engine query data.

Our objective for this work is to show what NaturalLanguage Processing (NLP) and Machine Learning (ML)techniques—what representation of information and whatclassification algorithms—are suitable to use for identifyingand classifying relevant medical information in short texts.We acknowledge the fact that tools capable of identifyingreliable information in the medical domain stand as

IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING, VOL. 23, NO. 6, JUNE 2011 801

. The authors are with the School of Information Technology andEngineering (SITE), University of Ottawa, 800 King Edward, Ottawa,Ontario K1N 6N5, Canada.E-mail: {ofrunza, diana, ttran}@site.uottawa.ca.

Manuscript received 13 Oct 2009; revised 20 Feb. 2010; accepted 21 Apr.2010; published online 27 Aug. 2010.Recommended for acceptance by E. Ferrari.For information on obtaining reprints of this article, please send e-mail to:[email protected], and reference IEEECS Log Number TKDE-2009-10-0718.Digital Object Identifier no. 10.1109/TKDE.2010.152.

1. https://www.google.com/health.2. http://healthvault.com/.3. http://healthcaretracker.wordpress.com/.

4. http://medline.cos.com/.

1041-4347/11/$26.00 � 2011 IEEE Published by the IEEE Computer Society

building blocks for a healthcare system that is up-to-datewith the latest discoveries. In this research, we focus ondiseases and treatment information, and the relation thatexists between these two entities. Our interests are inlinewith the tendency of having a personalized medicine, one inwhich each patient has its medical care tailored to its needs.It is not enough to read and know only about one study thatstates that a treatment is beneficial for a certain disease.Healthcare providers need to be up-to-date with all newdiscoveries about a certain treatment, in order to identify ifit might have side effects for certain types of patients.

We envision the potential and value of the findings ofour work as guidelines for the performance of a frameworkthat is capable to find relevant information about diseasesand treatments in a medical domain repository. The resultsthat we obtained show that it is a realistic scenario to useNLP and ML techniques to build a tool, similar to an RSSfeed, capable to identify and disseminate textual informa-tion related to diseases and treatments. Therefore, thisstudy is aimed at designing and examining variousrepresentation techniques in combination with variouslearning methods to identify and extract biomedicalrelations from literature.

The contributions that we bring with our work stand inthe fact that we present an extensive study of various MLalgorithms and textual representations for classifying shortmedical texts and identifying semantic relations betweentwo medical entities: diseases and treatments. From an MLpoint of view, we show that in short texts when identifyingsemantic relations between diseases and treatments asubstantial improvement in results is obtained when usinga hierarchical way of approaching the task (a pipeline oftwo tasks). It is better to identify and eliminate first thesentences that do not contain relevant information, and thenclassify the rest of the sentences by the relations of interest,instead of doing everything in one step by classifyingsentences into one of the relations of interest plus the extraclass of uninformative sentences.

The rest of the paper is organized as follows: Section 2discusses related work, Section 3 presents our proposedapproach to solve the task of identifying and disseminatinghealthcare information, Section 4 contains the evaluationand results obtained, Section 5 discussions, and Section 6conclusions and future work.

2 RELATED WORK

The most relevant related work is the work done by Rosarioand Hearst [25]. The authors of this paper are the ones whocreated and distributed the data set used in our research.The data set consists of sentences from Medline5 abstractsannotated with disease and treatment entities and witheight semantic relations between diseases and treatments.The main focus of their work is on entity recognition fordiseases and treatments. The authors use Hidden MarkovModels and maximum entropy models to perform both thetask of entity recognition and the relation discrimination.

Their representation techniques are based on words incontext, part of speech information, phrases, and a medicallexical ontology—Mesh6 terms. Compared to this work, ourresearch is focused on different representation techniques,different classification models, and most importantly gen-erates improved results with less annotated data.

The tasks addressed in our research are informationextraction and relation extraction. From the wealth ofresearch in these domains, we are going to mention somerepresentative works. The task of relation extraction orrelation identification is previously tackled in the medicalliterature, but with a focus on biomedical tasks: subcellular-location (Craven, [4]), gene-disorder association (Ray andCraven, [23]), and diseases and drugs (Srinivasan andRindflesch, [26]). Usually, the data sets used in biomedicalspecific tasks use short texts, often sentences. This is thecase of the first two related works mentioned above. Thetasks often entail identification of relations between entitiesthat co-occur in the same sentence.

There are three major approaches used in extractingrelations between entities: co-occurrences analysis, rule-based approaches, and statistical methods. The co-occur-rences methods are mostly based only on lexical knowledgeand words in context, and even though they tend to obtaingood levels of recall, their precision is low. Goodrepresentative examples of work on Medline abstractsinclude Jenssen et al. [14] and Stapley and Benoit [27].

In biomedical literature, rule-based approaches havebeen widely used for solving relation extraction tasks. Themain sources of information used by this technique areeither syntactic: part-of-speech (POS) and syntactic struc-tures; or semantic information in the form of fixed patternsthat contain words that trigger a certain relation. One of thedrawbacks of using methods based on rules is that theytend to require more human-expert effort than data-drivenmethods (though human effort is needed in data-drivenmethods too, to label the data). The best rule-based systemsare the ones that use rules constructed manually orsemiautomatically—extracted automatically and refinedmanually. A positive aspect of rule-based systems is thefact that they obtain good precision results, while the recalllevels tend to be low.

Syntactic rule-based relation extraction systems arecomplex systems based on additional tools used to assignPOS tags or to extract syntactic parse trees. It is known thatin the biomedical literature such tools are not yet at thestate-of-the-art level as they are for general English texts,and therefore their performance on sentences is not alwaysthe best (Bunescu et al. [2]). Representative works onsyntactic rule-based approaches for relation extraction inMedline abstracts and full-text articles are presented byThomas et al. [28], Yakushiji et al. [29], and Leroy et al. [16].Even though the syntactic information is the result of toolsthat are not 100 percent accurate, success stories with thesetypes of systems have been encountered in the biomedicaldomain. The winner of the BioCreative II.57 task was asyntactic rule-based system, OpenDMAP described inHunter et al. [13]. A good comparison of different syntactic

802 IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING, VOL. 23, NO. 6, JUNE 2011

5. The sentences were extracted from the first 100 titles and the first40 abstracts from each of the 59 files that are part of the Medline databasefrom 2001.

6. http://www.nlm.nih.gov/mesh/meshhome.html.7. http://www.biocreative.org/.

parsers and their contribution to extracting protein-proteininteractions can be found in Miyao et al. [19].

The semantic rule-based approaches suffer from the factthat the lexicon changes from domain to domain, and newrules need to be created each time. Certain rules are createdfor biological corpora, medical corpora, pharmaceuticalcorpora, etc. Systems based on semantic rules applied tofull-text articles are described by Friedman et al. [6], onsentences by Pustejovsky et al. [22], and on abstracts byRindflesch et al. [24]. Some researchers combined syntacticand semantic rules from Medline abstracts in order toobtain better systems with the flexibility of the syntacticinformation and the good precision of the semantic rules,e.g., Gaizauskas et al. [8] and Novichkova et al. [20].

Statistical methods tend to be used to solve various NLPtasks when annotated corpora are available. Rules areautomatically extracted by the learning algorithm whenusing statistical approaches to solve various tasks. Ingeneral, statistical techniques can perform well even withlittle training data. For extracting relations, the rules areused to determine if a textual input contains a relation ornot. Taking a statistical approach to solve the relationextraction problem from abstracts, the most used represen-tation technique is bag-of-words. It uses the words incontext to create a feature vector (Donaldson et al. [5]) and(Mitsumori et al. [18]). Other researchers combined the bag-of-words features, extracted from sentences, with othersources of information like POS (Bunescu and Mooney [1]).Giuliano et al. [9] used two sources of information:sentences in which the relation appears and the localcontext of the entities, and showed that simple representa-tion techniques bring good results.

Various learning algorithms have been used for thestatistical learning approach with kernel methods being thepopular ones applied to Medline abstracts (Li et al. [17]).

The task of identifying informative sentences is ad-dressed in the literature mostly for the tasks of summariza-tion and information extraction, and typically on suchdomains as newswire data, novels, medical, and biomedicaldomain. In the later mentioned domains, Goadrich et al.[11] used inductive logic techniques for informationextraction from abstracts, while Ould et al. [21] experi-mented with bag-of-word features on sentences. Our workdiffers from the ones mentioned in this section by the factthat we combine different textual representation techniquesfor various ML algorithms.

3 THE PROPOSED APPROACH

3.1 Tasks and Data Sets

The two tasks that are undertaken in this paper provide thebasis for the design of an information technology frame-work that is capable to identify and disseminate healthcareinformation. The first task identifies and extracts informa-tive sentences on diseases and treatments topics, while thesecond one performs a finer grained classification of thesesentences according to the semantic relations that existsbetween diseases and treatments.

The problems addressed in this paper form the buildingblocks of a framework that can be used by healthcare

providers (e.g., private clinics, hospitals, medical doctors,etc.), companies that build systematic reviews8 (hereafter,SR), or laypeople who want to be in charge of their health byreading the latest life science published articles related totheir interests. The final product can be envisioned as abrowser plug-in or a desktop application that will auto-matically find and extract the latest medical discoveriesrelated to disease-treatment relations and present them to theuser. The product can be developed and sold by companiesthat do research in Healthcare Informatics, Natural Lan-guage Processing, and Machine Learning, and companiesthat develop tools like Microsoft Health Vault. The value ofthe product from an e-commerce point of view stands in thefact that it can be used in marketing strategies to show thatthe information that is presented is trustful (Medline articles)and that the results are the latest discoveries. For any type ofbusiness, the trust and interest of customers are the keysuccess factors. Consumers are looking to buy or useproducts that satisfy their needs and gain their trust andconfidence. Healthcare products are probably the mostsensitive to the trust and confidence of consumers. Compa-nies that want to sell information technology healthcareframeworks need to build tools that allow them to extractand mine automatically the wealth of published research. Forexample, in frameworks that make recommendations fordrugs or treatments, these recommendations need to bebased on acknowledged discoveries and published results,in order to gain the consumers’ trust. The product value alsostands in the fact that it can provide a dynamic content to theconsumers, information tailored to a certain user (e.g., a set ofdiseases that the consumer is interested in).

The first task (task 1 or sentence selection) identifiessentences from Medline published abstracts that talkabout diseases and treatments. The task is similar to ascan of sentences contained in the abstract of an article inorder to present to the user-only sentences that areidentified as containing relevant information (disease-treatment information).

The second task (task 2 or relation identification) has adeeper semantic dimension and it is focused on identifyingdisease-treatment relations in the sentences already selectedas being informative (e.g., task 1 is applied first). We focuson three relations: Cure, Prevent, and Side Effect, a subset ofthe eight relations that the corpus is annotated with. Wedecided to focus on these three relations because these aremost represented in the corpus while for the other five, veryfew examples are available. Table 1 presents the originaldata set, the one used by Rosario and Hearst [25], that wealso use in our research. The numbers in parenthesesrepresent the training and test set size. For example, forCure relation, out of 810 sentences present in the data set,648 are used for training and 162 for testing.

The approach used to solve the two proposed tasks isbased on NLP and ML techniques. In a standardsupervised ML setting, a training set and a test set arerequired. The training set is used to train the ML algorithmand the test set to test its performance. The objectives are to

FRUNZA ET AL.: A MACHINE LEARNING APPROACH FOR IDENTIFYING DISEASE-TREATMENT RELATIONS IN SHORT TEXTS 803

8. Systematic reviews are summaries of research on a certain topic ofinterest. The topic can be a drug, disease, decision making step, etc.

build models that can later be deployed on other test setswith high performance.

For the work presented in this paper, the data setscontain sentences that are annotated with the appropriateinformation. Unlike in the work of Rosario and Hearst [25],in our research, the annotations of the data set are used tocreate a different task (task 1). It identifies informativesentences that contain information about diseases andtreatments and semantic relations between them, versusnoninformative sentences. This allows us to see how wellNLP and ML techniques can cope with the task ofidentifying informative sentences, or in other words, howwell they can weed out sentences that are not relevant tomedical diseases and treatments.

Extracting informative sentences is a task by itself inthe NLP and ML community. Research fields likesummarization and information extraction are disciplines

where the identification of informative text is a crucialtask. The contributions and research value that arebrought with this task stand in the usefulness of theresults and the insights about the experimental settings forthe task in the medical domain.

For the first task, the data sets are annotated with thefollowing information: a label indicating that the sentence isinformative, i.e., containing disease-treatment information,or a label indicating that the sentence is not informative.Table 2 gives an example of labeled sentences.

For the second task, the sentences have annotationinformation that states if the relation that exists in asentence between the disease and treatment is Cure, Prevent,or Side Effect. These are the relations that are morerepresented in the original data set and also needed forour future research. We would like to focus on a fewrelations of interest and try to identify what predictivemodel and representation technique bring the best results.The task of identifying the three semantic relations isaddressed in two ways:

Setting 1. Three models are built. Each model isfocused on one relation and can distinguish sentencesthat contain the relation from sentences that do not. Thissetting is similar to a two-class classification task inwhich instances are labeled either with the relation inquestion (Positive label) or with nonrelevant information(Negative label);

Setting 2. One model is built, to distinguish the threerelations in a three-class classification task where eachsentence is labeled with one of the semantic relations.

Tables 3 and 4 present the data sets that we used for ourtwo tasks.

In Table 4, the label “Positive” represents a sentence thatcontains the semantic relations (i.e., Cure, Prevent, or SideEffect), and “Negative” a sentence that does not containinformation about any of the semantic relations but containsinformation about either a disease or a treatment labelsTreatment_Only, Disease_Only in previous research byRosario and Hearst [25].

Up to this point, we presented the two tasks separatelyas being two self-defined tasks since they can be used forother more complex tasks as well. From a methodologicalpoint of view, and, more importantly, from a practical pointof view, they can be integrated together in a pipeline oftasks as a solution to a framework that is tailored to identifysemantic relations in short texts and sentences, when it isnot known a priori if the text contains useful information.The proposed pipeline solves task 1 first and then processes


TABLE 1Data Set Description, Taken from Rosario and Hearst (’04)

In brackets, are the numbers of instances used for training and fortesting, respectively.

TABLE 2Examples of Annotated Sentences for the Sentence Selection Task

the results in task 2, so that in the end, only informativesentences are classified into the three semantic relations.The logic behind choosing to experiment with and reportresults for the pipeline of tasks is that we have to identifythe best model that will get us closer to our main goal: beingable to identify and classify reliably medical information.Using the pipeline of tasks, we eliminate some errors thatcan be introduced due to the fact that we would consideruninformative sentences as potential data when classifyingsentences directly into semantic relations. We will showthat the pipeline achieves much better results than a morestraightforward approach of classifying in one step into oneof the three relations of interest plus an extra class foruninformative sentences.

The pipeline is similar to a hierarchy of tasks in whichthe results of one task is given as input to the other. Webelieve that this can be a solution for identifying anddisseminating relevant information tailored to a specificsemantic relation because the second task is trying a finergrained classification of the sentences that already containinformation about the relations of interest. This frameworkis appropriate for consumers that tend to be more interestedin an end result that is more specific, e.g., relevantinformation only for the class Cure, rather than identifyingsentences that have the potential to be informative for awider variety of disease-treatment semantic relations.

3.2 Classification Algorithms and DataRepresentations

In ML, as a field of empirical studies, the acquired expertiseand knowledge from previous research guide the way ofsolving new tasks. The models should be reliable atidentifying informative sentences and discriminating dis-ease-treatment semantic relations. The research experimentsneed to be guided such that high performance is obtained.The experimental settings are directed such that they areadapted to the domain of study (medical knowledge) and to

the type of data we deal with (short texts or sentences),allowing for the methods to bring improved performance.

There are at least two challenges that can be encounteredwhile working with ML techniques. One is to find the mostsuitable model for prediction. The ML field offers a suite ofpredictive models (algorithms) that can be used anddeployed. The task of finding the suitable one relies heavilyon empirical studies and knowledge expertise. The secondone is to find a good data representation and to do featureengineering because features strongly influence the perfor-mance of the models. Identifying the right and sufficientfeatures to represent the data for the predictive models,especially when the source of information is not large, as itis the case of sentences, is a crucial aspect that needs to betaken into consideration. These challenges are addressed bytrying various predictive algorithms, and by using varioustextual representation techniques that we consider suitablefor the task.

As classification algorithms, we use a set of sixrepresentative models: decision-based models (Decisiontrees), probabilistic models (Naıve Bayes (NB) and Comple-ment Naıve Bayes (CNB), which is adapted for text withimbalanced class distribution), adaptive learning (Ada-Boost), a linear classifier (support vector machine (SVM)with polynomial kernel), and a classifier that alwayspredicts the majority class in the training data (used as abaseline). We decided to use these classifiers because theyare representative for the learning algorithms in theliterature and were shown to work well on both short andlong texts. Decision trees are decision-based models similarto the rule-based models that are used in handcraftedsystems, and are suitable for short texts. Probabilisticmodels, especially the ones based on the Naıve Bayestheory, are the state of the art in text classification and inalmost any automatic text classification task. Adaptivelearning algorithms are the ones that focus on hard-to-learnconcepts, usually underrepresented in the data, a character-istic that appears in our short texts and imbalanced datasets. SVM-based models are acknowledged state-of-the-artclassification techniques on text. All classifiers are part of atool called Weka.9 One can imagine the steps of processingthe data (in our case textual information—sentences) for MLalgorithms as the steps required to obtain a database tablethat contains as many columns as the number of featuresselected to represent the data, and as many rows as thenumber of data points from the collection (sentences in ourcase). The most difficult and important step is to identify


TABLE 3Data Sets Used for the First Task

TABLE 4Data Sets Used for the Second Task

9. http://www.cs.waikato.ac.nz/ml/weka/.

which features should be selected to represent the data. Aspecial column in this table represents the label of eachinstance. An instance represents a row that contains valuesfor the selected features. The ML algorithms that are usingthis data representation to create predictive models shouldcapture correlations between features, feature values, andlabels, in order to obtain good prediction labels on futuretest data. The innovation and contribution that immergeform these experimental settings stand in identifying themost informative features for the task to feed the models,while using a suitable predictive algorithm in order toincrease the chance of predicting correct labels for new textsprocessed in the future. The following sections present thedata representation techniques.

3.2.1 Bag-of-Words Representation

The bag-of-words (BOW) representation is commonly usedfor text classification tasks. It is a representation in whichfeatures are chosen among the words that are present in thetraining data. Selection techniques are used in order toidentify the most suitable words as features. After thefeature space is identified, each training and test instance ismapped to this feature representation by giving values toeach feature for a certain instance. Two most commonfeature value representations for BOW representation are:binary feature values—the value of a feature can be either 0or 1, where 1 represents the fact that the feature is present inthe instance and 0 otherwise; or frequency featurevalues—the value of the feature is the number of times itappears in an instance, or 0 if it did not appear.

Because we deal with short texts with an average of20 words per sentence, the difference between a binaryvalue representation and a frequency value representationis not large. In our case, we chose a frequency valuerepresentation. This has the advantage that if a featureappears more than once in a sentence, this means that it isimportant and the frequency value representation willcapture this—the feature’s value will be greater than thatof other features. The selected features are words delimitedby spaces and simple punctuation marks such as (,) , [,] ,. ,’.We keep only the words that appeared at least three timesin the training collection, contain at least one alphanumericcharacter, are not part of an English list of stop words,10 andare longer than three characters. The frequency threshold ofthree is commonly used for text collections because itremoves noninformative features and also strings ofcharacters that might be the result of a wrong tokenizationwhen splitting the text into words. Words that have lengthof two or one character are not considered as featuresbecause of two other reasons: possible incorrect tokeniza-tion and problems with very short acronyms in the medicaldomain that could be highly ambiguous (could be anacronym or an abbreviation of a common word).

3.2.2 NLP and Biomedical Concepts Representation

The second type of representation is based on syntacticinformation: noun-phrases, verb-phrases, and biomedical

concepts identified in the sentences. In order to extract thistype of information, we used the Genia11 tagger tool. Thetagger analyzes English sentences and outputs the baseforms, part-of-speech tags, chunk tags, and named entitytags. The tagger is specifically tuned for biomedical text suchas Medline abstracts. Fig. 1 presents an example of the outputof the Genia tagger for the sentence: “Inhibition of NF-kappaBactivation reversed the anti-apoptotic effect of isochamaejasmin.”The noun and verb-phrases identified by the tagger arefeatures used for the second representation technique.

We ran the Genia tagger on the entire data set. Weextracted only noun-phrases, verb-phrases, and biomedicalconcepts as potential features from the output of eachsentence present in the data set.

The following preprocessing steps are applied in order toidentify the final set of features to be used for classification:removing features that contain only punctuation, removingstop words (using the same list of words as for our BOWrepresentation), and considering valid features only thelemma-based forms. We chose to use lemmas because thereare a lot of inflected forms (e.g., plural forms) for the sameword and the lemmatized form (the base form of a word)will give us the same base form for all of them. Anotherreason is to reduce the data sparseness problem. Dealingwith short texts, very few features are represented in eachinstance; using lemma forms alleviates this problem.Experiments are performed when using as features onlythe final set of identified noun-phrases, only verb-phrases,only biomedical entities, and with combinations of all thesefeatures. When combining the features, the feature vectorfor each instance is a concatenation of all features.

3.2.3 Medical Concepts (UMLS) Representation

In order to work with a representation that providesfeatures that are more general than the words in theabstracts (used in the BOW representation), we also usedthe Unified Medical Language system12 (hereafter, UMLS)concept representations. UMLS is a knowledge sourcedeveloped at the US National Library of Medicine(hereafter, NLM) and it contains a metathesaurus, asemantic network, and the specialist lexicon for biomedi-cal domain. The metathesaurus is organized around


Fig. 1. Example of Genia tagger output including for each word: its baseform, its part-of-speech, beginning (B), inside (I), outside (O) tags for theword, and the final tag for the phrase.

10. http://www.site.uottawa.ca/~diana/csi5180/StopWords. Stopwords are function words that appear in every document (e.g., the, it, of,and an) and therefore do not help in classification.

11. http://www-tsujii.is.s.u-tokyo.ac.jp/GENIA/tagger/.12. http://www.nlm.nih.gov/pubs/factsheets/umls.html.

concepts and meanings; it links alternative names andviews of the same concept and identifies useful relation-ships between different concepts.

UMLS contains over 1 million medical concepts, and over5 million concept names which are hierarchical organized.All concepts are assigned at least one semantic type from thesemantic network providing a generalization of the existingrelations between concepts. There are 135 semantic types inthe knowledge base that are linked through 54 relationships.

In addition to the UMLS knowledge base, NLM created aset of tools that allow easier access to the useful information.MetaMap13 is a tool created by NLM that maps free text tomedical concepts in the UMLS, or equivalently, it discoversmetathesaurus concepts in text. With this software, text isprocessed through a series of modules that in the end willgive a ranked list of all possible concept candidates for aparticular noun-phrase. For each of the noun-phrases thatthe system finds in the text, variant noun-phrases aregenerated. For each of the variant noun-phrases, candidateconcepts (concepts that contain the noun-phrase variant)from the UMLS metathesaurus are retrieved and evaluated.The retrieved concepts are compared to the actual phraseusing a fit function that measures the text overlap betweenthe actual phrase and the candidate concept (it returns anumerical value). The best of the candidates are thenorganized according to the decreasing value of the fitfunction. We used the top concept candidate for eachidentified phrase in an abstract as a feature.

Fig. 2 presents an example of the output of the MetaMapsystem for the phrase “to an increased risk.” The informationpresent in the brackets, “Qualitative Concept, QuantitativeConcept” for the candidate with the fit function value 861 isthe concept used as feature in the UMLS representation.Another reason to use a UMLS concept representation is theconcept drift phenomenon that can appear in a BOWrepresentation. Especially in the medical domain texts, thisis a frequent problem as stated by Cohen et al. [3]: newarticles that publish new research on a certain topic bringwith them new terms that might not match the ones thatwere seen in the training process in a certain moment oftime. The UMLS concepts also help with the data sparse-ness problem and give a better coverage of the features ineach sentence instance.

Experiments for the two tasks tackled in this researchwere performed with each individual above-mentionedrepresentations, plus their combinations. We combined theBOW, UMLS, NLP, and biomedical concepts, by putting thefeatures together to represent an instance.

4 EVALUATION AND RESULTS

This section discusses the evaluation measures and presentsthe results of the two tasks using the methodologydescribed above.

4.1 Evaluation Measures

The most common used evaluation measures in the MLsettings are: accuracy, precision, recall, and F-measure. Allthese measures are computed form a confusion matrix(Kohavi and Provost [15]) that contains information aboutthe actual classes, the true classes and the classes predictedby the classifier. The test set on which the models areevaluated contain the true classes and the evaluation triesto identify how many of the true classes were predicted bythe model classifier. In the ML settings, special attentionneeds to be directed to the evaluation measures that areused. For data sets that are highly imbalanced (one class isoverrepresented in comparison with another), standardevaluation measures like accuracy are not suitable. Becauseour data sets are imbalanced, we chose to report inaddition to accuracy, the macroaveraged F-measure. Wedecided to report macro and not microaveraged F-measurebecause the macromeasure is not influenced by themajority class, as the micromeasure is. The macromeasurebetter focuses on the performance the classifier has on theminority classes. The formulas for the evaluation measuresare: AccuracyAccuracy ¼ the total number of correctly classifiedinstances; RecallRecall ¼ the ratio of correctly classified positiveinstances to the total number of positives. This evaluationmeasure is known to the medical research community assensitivity. PrecisionPrecision ¼ the ratio of correctly classifiedpositive instances to the total number of classified aspositive. F -measureF -measure ¼ the harmonic mean between preci-sion and recall.

4.2 Results for the Task of Identifying InformativeSentences (Task 1)

This section presents the results for the first task, the one ofidentifying whether sentences are informative, i.e., contain-ing information about diseases and treatments, or not. TheML settings are created for a two-class classification taskand the representations are the ones mentioned in theprevious section, while the baseline on which we need toimprove is given by the results of a classifier that alwayspredicts the majority class.

Fig. 3 presents the results obtained when using asrepresentation features verb-phrases identified by the Geniatagger. When using this representation, the results are closeto baseline. The reason why this happens for all algorithmsthat we use is the fact that the texts are short and theselected features are not well represented in an instance. Wehave a data sparseness problem: it is the case when a lot offeatures have value 0 for a particular instance.

Fig. 4 presents the results obtained using as representa-tion features noun-phrases selected by the Genia tagger.Compared to previous results, we can observe a slightimprovement in both accuracy and F-measure. The bestresults are obtained by the CNB classifier. We believe thatthe slight improvement is due to a reduction of thesparseness problem: noun-phrases are more frequently


Fig. 2. Example of MetaMap system output.

13. http://mmtx.nlm.nih.gov/.

present in short texts than verb-phrases. Fig. 5 presents thebest results obtained so far. An increase of almost5 percentage points, for both accuracy and F-measure isobtained when using as representation features biomedicalentities extracted by the Genia tagger and CNB asclassifier. An increase in results for the other classifierscan be also observed.

This increase can be caused by the fact that, whenpresent in sentences, the biomedical entities have a strongerpredicting value. The entities identify better if a sentence isinformative or not and this is something that we wouldhope to happen. If a sentence contains a good proportion ofbiomedical entities this should trigger a higher chance oflabeling a sentence as informative.

Fig. 6 presents accuracy and F-measure results for allclassifiers when noun-phrases, identified by the Genia taggerand biomedical entities are used as representation features.Compared to previous representation techniques, the resultspresented in Fig. 6 follow what will become a trend, anincrease in results when more informative and diverserepresentation techniques are used. With a representationthat combines noun-phrases and biomedical entities andCNB as classifier, an increase of 2 percentage points isobtained compared to the results when only biomedicalconcepts are used (Fig. 5). An increase of 8 percentage points

is obtained compared to only a noun-phrase representation(Fig. 4).

In Fig. 7, we use as representation technique UMLS

concepts—medical domain-specific concepts identified by

the MetaMap tool. Compared to all previous results, this

representation technique for the CNB classifier obtains the

best results so far, with an increase of almost 4 percentage

points. We believe that this is because these concepts are


Fig. 3. Accuracy and F-measure results when using verb-phrases asfeatures for task 1.

Fig. 4. Accuracy and F-measure results when using noun-phrases asfeatures for task 1.

Fig. 5. Accuracy and F-measure results when using biomedical

concepts as features, task 1.

Fig. 6. Accuracy and F-measure results when using NLP and biomedicalconcepts as features, task 1.

Fig. 7. Accuracy and F-measure results when using UMLS concepts as

features for task 1.

more general than the biomedical concepts and the feature

space is not that sparse. Classifiers tend to obtain better

results when the feature space is well represented by an

instance. This observation is what we believe happened

with the results presented in Fig. 7.A representation based on noun-phrases, verb-phrases,

biomedical concepts, and UMLS concepts brings again an

increase in results compared to the ones presented in Fig. 7.

This increase of 2 percentage points can be observed in Fig. 8.

As stated before, the trend of increase in results is due to the

representation that captures more information.The bag-of-words representation technique is known in

the literature to be one that is hard to beat. Even though is not

a very sophisticated method—it contains only the words in

context; it is one that often obtains good results. In our

experiments, the BOW representation (Fig. 9) obtains the best

results between all the representation techniques mentioned

in this section so far. An increase of almost 2 percentage

points is obtained compared to the results in Fig. 8.In Fig. 10, we present the results for a BOW plus UMLS

concepts representation. Even though the BOW representa-

tion is one that gives good results, when used in

combinations with other types of representations, the

performance can be improved. This observation can be

drawn from Fig. 10 where a 2 percentage points improve-

ment is obtained for CNB, compared to the BOW

representation; an improvement of 10 percentage points is

obtained when comparing to the UMLS representation. For

all the other classifiers, an increase in results can be

observed as well. The results obtained by only a BOW

representation can be further improved when these features

are combined with the noun-phrases, verb-phrases, and

biomedical concepts. Fig. 11 presents results for this

representation technique with all the classification algo-

rithms. The best result is improved with 1 percentage point

compared to the one in Fig. 10.For this current task, identifying which sentences from the

abstracts of Medline articles that contain informative

sentences for diseases and treatments, the best results

obtained are the one presented in Fig. 12. The representation

technique that uses BOW features, UMLS concepts, noun

and verb-phrases, and biomedical concepts with the CNB

classifier obtain a 90.72 percent F-measure and 90.36 percent

accuracy. These increases in results are due to the fact that all

these various types of features create a rich and predictive

feature space for the classifiers.


Fig. 8. Accuracy and F-measure results when using NLP, biomedical,and UMLS concepts as features, task 1.

Fig. 9. Accuracy and F-measure results when using BOW features fortask 1.

Fig. 10. Accuracy and F-measure results when using BOW and UMLSconcepts as features for task 1.

Fig. 11. Accuracy and F-measure results when using BOW, NLP, andbiomedical features, task 1.

Even if some of the features alone are not the ones thatobtain good performance, e.g., verb-phrases, when com-bined with other types of features form a representationmodel capable to predict with 90 percent accuracy if asentence is informative or not. The fact that the best resultsfor this task are obtained when all features are put togethershows that a good representation technique is a crucialaspect of a classification task. In order to statistically supportour conclusions, we run t-tests at 95 percent confidencelevels for the CNB classifier with all the representationtechniques for both accuracy and F-measure. Even thoughsome of the differences are small, they are significantlydifferent, except the one between Figs. 8 and 9. Based on ourexperiments, we can conclude and suggest as future guide-lines for similar tasks that the richer and more informativethe representation technique is, the better the performanceresults. For a good performance level, we suggest acombination of all the features.

4.3 Results for the Task of Identifying SemanticRelations (Task 2)

The focus for the second task is to automatically identifywhich sentences contain information for the three semanticrelations: Cure, Prevent, and Side Effect. The reported resultsare based on similar settings to the ones used for theprevious task. Since imbalanced data sets are used for thistask, the evaluation measure that we are going to report isthe F-measure. Due to space issues, we are going to presentthe best results obtained for all settings. The best results arechosen from all the representation techniques and allclassification algorithms that we also used for the first task.The labels on the x-axis stand for the name of the semanticrelation, the representation technique, and the classificationalgorithm used.

In Fig. 13, we present the results when using Setting 1,described in Section 3.1, as the setup for the experiment. Onthe x-axis, we present for each relation the best F-measureresult, the representation technique, and the classifier thatobtained the result. For example, for the Cure relation, thecombination of BOW features, noun-phrases and verb-phrases, biomedical and UMLS concepts, with SVM as aclassifier, obtained the 87.10 percent result for F-measure.SVM and NB with rich feature representations are the

setups that obtained the best results. Fig. 14 presents thebest results that we obtain for the second task, a level ofalmost 100 percent F-measure for the Cure relation,100 percent F-measure for Prevent relation, and 75 percentF-measure for Side Effect. For this setting, we train a modelfor all three relations in the same time, and we distinguishsentences between these three relations.

For this setting, the NB classifier with combinations ofvarious representation features is the one that obtains thebest results for all relations. The improvement over theother settings can be due to the fact that the combination ofclassifier and features has a good predicting value for amodel trained on the three relations. Each of the relationscan be well-defined and predicted when using the modelthat we propose in Setting 2. The fact that we achieve closeto perfection prediction suggests that the choice of classifierand representation technique are key factors for a super-vised classification task, even for semantically charged taskslike ours. The good performance results that we obtain withthe second setting also suggest that a prior triage ofsentences, informative versus noninformative can be crucialfor a finer grained classification of relations betweenentities. Setting 1 uses all the sentences, including thosethat do not contain information about the three relations ofinterests, while in Setting 2, we used as training data onlysentences that we knew a priori to contain one of the threerelations. This observation for the results of the secondsetting also validates our choice of proposing the first task,identify which sentences are informative and which not. Forgood performance level in the relation classification task,we need to weed out noninformative sentences.


Fig. 13. Results for the second task, Setting 1.

Fig. 14. Results for the second task, Setting 2.

Fig. 12. Accuracy and F-measure results when using BOW, NLP,biomedical, and UMLS concepts features, task 1.

4.4 Results for the Pipeline—Task 1 Followed byTask 2

In this section, we present the evaluation results for thepipeline of the two tasks. When looking at the results thatwe obtain for the second task, the best setting was the one inwhich we classify sentences already known to be informa-tive (Setting 2). This observation let us believe that apipeline of the two tasks is a viable solution for our goal.

To show that a pipeline of results is better as a solutionfor identifying semantic relations in informative sentences,we need to compare its results to the results of a model thatclassifies sentences into four-classes directly: the threesemantic relations Cure, Prevent, SideEffect and the classfor sentences that are uninformative.

The results for the pipeline of tasks are obtained bymultiplying the evaluation measures acquired by the firsttask with the evaluation measure for the second task foreach semantic relation. To be consistent, we report the F-measure results. For the first task, the best F-measure resultof 90.72 percent is obtained by the CNB classifier using acombination of all types of features (Fig. 12). For the secondtask, the best F-measure results are obtained by the NBclassifier using a combination of all types of features for allthree semantic relations (Fig. 14). Table 5 presents theresults for the pipeline of tasks.

The results for a scenario in which we solve the task ofidentifying sentences relevant to one of the three semanticrelations by deploying a four-class trained classifier arepresented in Fig. 15. In this experiment, we used allclassifiers and all representation techniques that wepropose in this paper on a data set that consists of sentencesthat are either labeled as uninformative, sentences fromtask 1, or with one of the three semantic relations, the dataset used in task 2. To be consistent with all otherexperiments, we report F-measure results as well. Sincereporting the results of every setting would take a lot ofspace, we decided to report only the best ones. Fig. 15presents these results. The representation and classificationalgorithms mentioned in the legend correspond to the left-to-right results. The pipeline of tasks clearly outperformsthe four-class classification scenario for the Prevent andSideEffect class. An increase of 30 percentage points isobtained for the Prevent class and 18 percentage points forthe SideEffect class. For the class Cure, the four-classmethodology was superior with 4 percentage points. Thereason for this could be the fact that the Cure class is wellrepresented in the data set and in the end it has a higherchance to be correctly classified in almost any ML scenario.The important type of error that can occur in a four-wayclassification task is the high rate of false negatives for the

three classes of interest. Some sentences that belong to oneof the classes of interest get classified in the fourth class thatcontains uninformative sentences. In a pipeline of tasksscenario, if the result of the first task is high anduninformative sentences are removed, then informativesentences have higher changes to be classified into the rightclass, especially when the choices are reduced by one.

Usually the amount of data that is classified as beingnonrelevant is higher than the one relevant to a particularrelation. In a four-way classification task, the majority classoverwhelms the underrepresented ones while in a pipelineof tasks the balance between that relevant data andnonrelevant one is higher and the classifiers have betterchances of distinguish between them.

The fact that for the two underrepresented classes, weobtain a high increase in results suggests that a pipeline oftasks is superior in performance to a four-class classifica-tion task.

5 DISCUSSION

This section discusses the results we obtained for the twotasks in this study. For the first task, the one of identifyinginformative sentences, the results show that probabilisticmodels based on Naıve Bayes formula, obtain good results.The fact that the SVM classifier performs well shows thatthe current discoveries are in line with the literature. Thesetwo classifiers have always been shown to perform well ontext classification tasks. Even though the independence offeatures is violated when using Naıve Bayes classifiers, theystill perform very well. The AdaBoost classifier wasoutperformed by the other classifiers, which is a little


TABLE 5F-Measure Results for the Pipeline-Task 1 Followed by Task 2

Fig. 15. F-measure results for four-class classification.

surprising taking into account the fact that it is designed tofocus on hard-to-learn concepts. In our previous experience,it was shown to perform well on medical domain texts withimbalanced classes (Frunza and Inkpen [7]). One reasonwhy the AdaBoost classifier did not perform well might bethat fact that in previous experiments we used the entireabstract as source of information while in this current studywe use sentences.

In NLP and ML community, BOW is a representationtechnique that even though it is simplistic, most of the timesit is really hard to outperform. As shown in Fig. 9, theresults obtained with this representation are among the bestone, but for both tasks, we outperform it when we combineit with more structured information such as medical andbiomedical concepts.

One of the major contributions of this work is the fact thatthe current experiments show that additional information inthe representation settings brings improvements for the taskof identifying informative sentences. The task itself is aknowledge-charged task; the labeling process involves ahuman-intensive annotation process since relations betweenentities need to be manually identified. The experimentsdesigned for the automatic task aim to show that classifiersperform better when richer information is provided. In thefirst task, the CNB classifier using all the representationtechniques obtains the best result of 90 percent F-measurewhich is statistically significant. The classifier is speciallydesigned for imbalanced data, the fact that it proved to beone of the best in text classification tasks, even on short texts,was somewhat a foreseeable result.

The results obtained for the second task suggest thatwhen the focus of a task is to obtain good reliable results,extra analysis is required. The best results are obtainedwith Setting 2 when a model is built and trained on a dataset that contains all three data sets for the three relations.The representation and the classification algorithms wereable to make the distinction between the relations and toobtain the best results for this task. Similar observations asthe ones obtained for the first task are valid: probabilisticmodels combined with more informative feature repre-sentation bring the best results. The best results obtainedare: 98 percent F-measure for the class Cure, 100 percent F-measure for the class Prevent, and 75 percent F-measurefor the SideEffect class.

The fact that we obtain the best results when usingSetting 2 also validates our proposed methodology for apipeline of tasks in order to better classify relevantinformation in the three semantic relations. It is moreefficient to solve the second task when using data that areknown a priori to contain information about the relations inquestion, rather than identifying which sentences areuninformative as well.

In order to better validate the choices made in terms ofrepresentation and classification algorithms and to directlycompare with the previous work, additional experimentsfor all eight semantic relations originally annotated on thedata set were performed. These experiments are addressingexactly the same task as the previous work (Rosario andHearst [25]) and are evaluated with the same evaluationmeasure, accuracy. Due to space constrains, we will reportin Fig. 16 only the best results with the algorithms andrepresentations that we used for this task.

The first bars of results are obtained with the best modelfor each of the eight relations (e.g., for Cure, the representa-tion that obtains the best results is reported, a representationthat can be different from the one for another relation; thelabel of each set of bars describes the representation);the second bars of results report the model that obtains thebest accuracy over all relations (one representation and oneclassification algorithm are reported for all relations—CNBwith BOW+NLP+Biomed features), and the third bars ofresults represent the previous results obtained by Rosarioand Hearst [25].

The accuracy measure is reported since it is the measurethat was reported in the previous work. As depicted in thefigure, the results obtained in this study outperform theprevious ones. In one case, the same low results are obtained;for example, for the No_Cure class, the low results are due tothe fact that this class is underrepresented in the data set, byonly four examples in total.

The class Vague obtains similar results when one modelis used for all relations, but it outperforms previous resultswhen the best model is chosen for this class. For the otherrelation, our results are better with either the same modelfor all relations, or for the best one for each.

6 CONCLUSIONS AND FUTURE WORK

The conclusions of our study suggest that domain-specificknowledge improves the results. Probabilistic models arestable and reliable for tasks performed on short texts in themedical domain. The representation techniques influencethe results of the ML algorithms, but more informativerepresentations are the ones that consistently obtain thebest results.

The first task that we tackle in this paper is a task that hasapplications in information retrieval, information extraction,and text summarization. We identify potential improve-ments in results when more information is brought in therepresentation technique for the task of classifying shortmedical texts. We show that the simple BOW approach, wellknown to give reliable results on text classification tasks, canbe significantly outperformed when adding more complexand structured information from various ontologies.

The second task that we address can be viewed as atask that could benefit from solving the first task first. Inthis study, we have focused on three semantic relationsbetween diseases and treatments. Our work shows that thebest results are obtained when the classifier is notoverwhelmed by sentences that are not related to thetask. Also, to perform a triage of the sentences (task 1) fora relation classification task is an important step. InSetting 1, we included the sentences that did not containany of the three relations in question and the results werelower than the one when we used models trained only onsentences containing the three relations of interest. Thesediscoveries validate the fact that it is crucial to have thefirst step to weed out uninformative sentences, beforelooking deeper into classifying them. Similar findings andconclusions can be made for the representation andclassification techniques for task 2.

The above observations support the pipeline of tasks thatwe propose in this work. The improvement in results of 14and 18 percentage points that we obtain for two of theclasses in question shows that a framework in which tasks 1


and 2 are used in pipeline is superior to when the two tasksare solved in one step by a four-way classification.

Probabilistic models combined with a rich representation

technique bring the best results.As future work, we would like to extend the experi-

mental methodology when the first setting is applied for thesecond task, to use additional sources of information asrepresentation techniques, and to focus more on ways tointegrate the research discoveries in a framework to bedeployed to consumers. In addition to more methodologicalsettings in which we try to find the potential value of othertypes of representations, we would like to focus on sourcedata that comes from the web. Identifying and classifyingmedical-related information on the web is a challenge thatcan bring valuable information to the research communityand also to the end user. We also consider as potentialfuture work ways in which the framework’s capabilities canbe used in a commercial recommender system and inintegration in a new EHR system. Amazon representativeJeff Bezos said: “Our experience with user interfaces andhigh-performance computing are ideally suited to helphealthcare. We nudge people’s decision making andbehavior with the gentle push of data [. . . ]”.14

REFERENCES

[1] R. Bunescu and R. Mooney, “A Shortest Path Dependency Kernelfor Relation Extraction,” Proc. Conf. Human Language Technologyand Empirical Methods in Natural Language Processing (HLT/EMNLP), pp. 724-731, 2005.

[2] R. Bunescu, R. Mooney, Y. Weiss, B. Scholkopf, and J. Platt,“Subsequence Kernels for Relation Extraction,” Advances in NeuralInformation Processing Systems, vol. 18, pp. 171-178, 2006.

[3] A.M. Cohen and W.R. Hersh, and R.T. Bhupatiraju, “FeatureGeneration, Feature Selection, Classifiers, and Conceptual Driftfor Biomedical Document Triage,” Proc. 13th Text Retrieval Conf.(TREC), 2004.

[4] M. Craven, “Learning to Extract Relations from Medline,” Proc.Assoc. for the Advancement of Artificial Intelligence, 1999.

[5] I. Donaldson et al., “PreBIND and Textomy: Mining theBiomedical Literature for Protein-Protein Interactions Using aSupport Vector Machine,” BMC Bioinformatics, vol. 4, 2003.

[6] C. Friedman, P. Kra, H. Yu, M. Krauthammer, and A. Rzhetsky,“GENIES: A Natural Language Processing System for theExtraction of Molecular Pathways from Journal Articles,” Bioinfor-matics, vol. 17, pp. S74-S82, 2001.

[7] O. Frunza and D. Inkpen, “Textual Information in PredictingFunctional Properties of the Genes,” Proc. Workshop Current Trendsin Biomedical Natural Language Processing (BioNLP) in conjunctionwith Assoc. for Computational Linguistics (ACL ’08), 2008.

[8] R. Gaizauskas, G. Demetriou, P.J. Artymiuk, and P. Willett,“Protein Structures and Information Extraction from BiologicalTexts: The PASTA System,” Bioinformatics, vol. 19, no. 1, pp. 135-143, 2003.

[9] C. Giuliano, L. Alberto, and R. Lorenza, “Exploiting ShallowLinguistic Information for Relation Extraction from BiomedicalLiterature,” Proc. 11th Conf. European Chapter of the Assoc. forComputational Linguistics, 2006.

[10] J. Ginsberg, H. Mohebbi Matthew, S.P. Rajan, B. Lynnette, S.S.Mark, and L. Brilliant, “Detecting Influenza Epidemics UsingSearch Engine Query Data,” Nature, vol. 457, pp. 1012-1014, Feb.2009.

[11] M. Goadrich, L. Oliphant, and J. Shavlik, “Learning Ensembles ofFirst-Order Clauses for Recall-Precision Curves: A Case Study inBiomedical Information Extraction,” Proc. 14th Int’l Conf. InductiveLogic Programming, 2004.

[12] L. Hunter and K.B. Cohen, “Biomedical Language Processing:What’s beyond PubMed?” Molecular Cell, vol. 21-5, pp. 589-594,2006.


14. http://www.healthcare-informatics.com.

Fig. 16. Results for all annotated relations in the data set.

[13] L. Hunter, Z. Lu, J. Firby, W.A. Baumgartner Jr., H.L. Johnson,P.V. Ogren, and K.B. Cohen, “OpenDMAP: An Open Source,Ontology-Driven Concept Analysis Engine, with Applications toCapturing Knowledge Regarding Protein Transport, ProteinInteractions and Cell-Type-Specific Gene Expression,” BMCBioinformatics, vol. 9, article no. 78, Jan. 2008.

[14] T.K. Jenssen, A. Laegreid, J. Komorowski, and E. Hovig, “ALiterature Network of Human Genes for High-ThroughputAnalysis of Gene Expression,” Nature Genetics, vol. 28, no. 1,pp. 21-28, 2001.

[15] R. Kohavi and F. Provost, “Glossary of Terms,” Machine Learning,Editorial for the Special Issue on Applications of MachineLearning and the Knowledge Discovery Process, vol. 30,pp. 271-274, 1998.

[16] G. Leroy, H.C. Chen, and J.D. Martinez, “A Shallow ParserBased on Closed-Class Words to Capture Relations in Biomedi-cal Text,” J. Biomedical Informatics, vol. 36, no. 3, pp. 145-158,2003.

[17] J. Li, Z. Zhang, X. Li, and H. Chen, “Kernel-Based Learning forBiomedical Relation Extraction,” J. Am. Soc. Information Science andTechnology, vol. 59, no. 5, pp. 756-769, 2008.

[18] T. Mitsumori, M. Murata, Y. Fukuda, K. Doi, and H. Doi,“Extracting Protein-Protein Interaction Information from Biome-dical Text with SVM,” IEICE Trans. Information and Systems,vol. E89D, no. 8, pp. 2464-2466, 2006.

[19] M. Yusuke, S. Kenji, S. Rune, M. Takuya, and T. Jun’ichi,“Evaluating Contributions of Natural Language Parsers toProtein-Protein Interaction Extraction,” Bioinformatics, vol. 25,pp. 394-400, 2009.

[20] S. Novichkova, S. Egorov, and N. Daraselia, “MedScan, A NaturalLanguage Processing Engine for MEDLINE Abstracts,” Bioinfor-matics, vol. 19, no. 13, pp. 1699-1706, 2003.

[21] M. Ould Abdel Vetah, C. Nedellec, P. Bessieres, F. Caropreso,A.-P. Manine, and S. Matwin, “Sentence Categorization inGenomics Bibliography: A Naive Bayes Approach,” Actes de laJournee Informatique et Transcriptome, J.-F. Boulicaut andM. Gandrillon, eds., Mai 2003.

[22] J. Pustejovsky, J. Castano, J. Zhang, M. Kotecki, and B. Cochran,“Robust Relational Parsing over Biomedical Literature: ExtractingInhibit Relations,” Proc. Pacific Symp. Biocomputing, vol. 7, pp. 362-373, 2002.

[23] S. Ray and M. Craven, “Representing Sentence Structure inHidden Markov Models for Information Extraction,” Proc. Int’lJoint Conf. Artificial Intelligence (IJCAI ’01), 2001.

[24] T.C. Rindflesch, L. Tanabe, J.N. Weinstein, and L. Hunter,“EDGAR: Extraction of Drugs, Genes, and Relations from theBiomedical Literature,” Proc. Pacific Symp. Biocomputing, vol. 5,pp. 514-525, 2000.

[25] B. Rosario and M.A. Hearst, “Semantic Relations in BioscienceText,” Proc. 42nd Ann. Meeting on Assoc. for ComputationalLinguistics, vol. 430, 2004.

[26] P. Srinivasan and T. Rindflesch, “Exploring Text Mining fromMedline,” Proc. Am. Medical Informatics Assoc. (AMIA) Symp.,2002.

[27] B.J. Stapley and G. Benoit, “Bibliometrics: Information RetrievalVisualization from Co-Occurrences of Gene Names in MEDLINEAbstracts,” Proc. Pacific Symp. Biocomputing, vol. 5, pp. 526-537,2000.

[28] J. Thomas, D. Milward, C. Ouzounis, S. Pulman, and M. Carroll,“Automatic Extraction of Protein Interations from ScientificAbstracts,” Proc. Pacific Symp. Biocomputing, vol. 5, pp. 538-549,2000.

[29] A. Yakushiji, Y. Tateisi, Y. Miyao, and J. Tsujii, “Event Extractionfrom Biomedical Papers Using a Full Parser,” Proc. Pacific Symp.Biocomputing, vol. 6, pp. 408-419, 2001.

Oana Frunza received the BSc degree incomputer science from Babes-Bolyai University,Romania, in 2004 and the MSc degree incomputer science from the University of Ottawa,Canada, in 2006. Since 2004, when she joinedthe School of Information Technology andEngineering (SITE), University of Ottawa, shehas been working as a research and teachingassistant. Currently, she is enrolled in thecomputer science PhD program in SITE. She

has more than 10 publications in prestigious conferences and journals.She is also the author of the book “Cognates, False Friends, and PartialCognates.” Her research interests include artificial intelligence (AI),natural language processing (NLP), machine learning (ML), and textmining.

Diana Inkpen received the BEng from theDepartment of Computer Science, TechnicalUniversity of Cluj-Napoca, Romania, in 1994,the MSc degree from the Technical University ofCluj-Napoca, Romania, in 1995, and the PhDdegree from the Department of ComputerScience, University of Toronto, in 2003. Sheworked as a researcher at the Institute forComputing Techniques in Cluj-Napoca, from1996 to 1998. In 2003, after finishing the PhD

degree, she joined the School of Information Technology and Engineer-ing (SITE) at the University of Ottawa, as an assistant professor, and iscurrently an associate professor. She was an Erasmus Mundus visitingscholar, at the University of Wolverhampton, United Kingdom, duringApril-July 2009. She is a reviewer for several journals (ComputationalLinguistics, Natural Language Engineering, Language Resources andEvaluation, etc.) and a program committee member for many confer-ences (ACL, NAACL, EMNLP, RANLP, CICLing, TANL, AI, etc.). Shepublished 4 book chapters, 11 journal papers, and 54 conference andworkshop papers. She organized four international workshops. In 2005and 2007, together with one of her PhD students, she obtained the bestresults at the Cross-Language Evaluation Forum, the Cross-LanguageSpeech Retrieval track. Her research interests are in the areas ofcomputational linguistics and artificial intelligence, more specifically:natural language understanding, natural language generation, lexicalsemantics, and information retrieval. She is member of the Associationfor Computational Linguistics (ACL).

Thomas Tran received the BSc (Four-YearSpecialist) degree, double major in mathematicsand computer Science from Brandon Universityin 1999, and the PhD degree in computerscience from the University of Waterloo in2004. He was the recipient of the GovernorGeneral’s Gold Medal at the ConvocationCeremony. He worked as a postdoctoral re-searcher in the School of Computer Science,University of Waterloo from 2003 to 2004. He

joined the School of Information Technology and Engineering (SITE),University of Ottawa as an assistant professor in 2004 and he is currentlyan associate professor. He is also a member of the Institute of Electricaland Electronics Engineers (IEEE), the Association for the Advancementof Artificial Intelligence (AAAI), and the Canadian Artificial IntelligenceAssociation (CAIAC). He has had more than 40 publications in severalrefereed journals and conference proceedings, and he is the joint holderof the Outstanding Paper Award for a paper presented at IADIS E-Commerce-2008, the Best Paper Award Nominee at MCETECH-2009,and the Best Paper Award Nominee at UM-2003. His research interestsinclude artificial intelligence (AI), electronic commerce, intelligent agentsand multiagent systems, trust and reputation modeling, reinforcementlearning, recommender systems, knowledge-based systems, architec-ture for mobile e-business, and applications of AI.

. For more information on this or any other computing topic,please visit our Digital Library at www.computer.org/publications/dlib.


a machine learning approach for identifying disease-treatment relations in short texts

Documents

healthcare information

medical related information

reliable information

medical domain

medical care domain

health information recording

medical imaging

medical papers