Djangology: A Light-weight Web-based Tool for Distributed Collaborative TextAnnotation

Emilia Apostolova, Sean Neilan, Gary An*, Noriko Tomuro, Steven Lytinen

College of Computing and Digital Media, DePaul University,Chicago, IL 60604 U.S.A.*Department of Surgery, Northwestern University Feinberg School of Medicine, Chicago, IL 60611 U.S.A.

emilia.aposto@gmail.com, sean@seanneilan.com, docgca@gmail.com,tomuro@cs.depaul.edu, lytinen@cs.depaul.edu

AbstractManual text annotation is a resource-consuming endeavor necessary for NLP systems when they target new tasks or domains for whichthere are no existing annotated corpora. Distributing the annotation work across multiple contributors is a natural solution to reduce andmanage the effort required. Although there are a few publicly available tools which support distributed collaborative text annotation,most of them have complex user interfaces and require a significant amount of involvement from the annotators/contributors as wellas the project developers and administrators. We present a light-weight web application for highly distributed annotation projects -Djangology. The application takes advantage of the recent advances in web framework architecture that allow rapid development anddeployment of web applications thus minimizing development time for customization. The application’s web-based interface givesproject administrators the ability to easily upload data, define project schemas, assign annotators, monitor progress, and review inter-annotator agreement statistics. The intuitive web-based user interface encourages annotator participation as contributors are not burdenedby tool manuals, local installation, or configuration. The system has achieved a user response rate of 70% in two annotation projectsinvolving more than 250 medical experts from various geographic locations.

1. IntroductionText annotation is an inherent step in almost any NaturalLanguage Processing (NLP) task. Ever since the MessageUnderstanding Conference (MUC) was first held in 1987,the creation of annotated corpora and evaluation metricshas been a focus in the NLP community. Sharable com-mon resources such as gold standards and training corporarelated to various NLP tasks have been actively created forover 20 years. The ISO/TC 37/SC4 Language ResourcesManagement sub-committee 1 of the International Organi-zation for Standardization (ISO) is actively developing an-notation standards (Ide and Romary, 2004). Even thoughthe need for annotation standards has been recognized, his-torically annotated corpora and NLP frameworks employad-hoc annotation schemas. As a result, the task of the NLPdeveloper/researcher inevitably involves ad-hoc translationfrom one annotation coding scheme to another.The repository of existing publicly available annotated cor-pora has been growing (the catalogue of the Linguistic DataConsortium currently consists of 450 linguistic corpora2).However, most real-world NLP efforts tackle new domains,languages, or tasks and lack the support and convenienceof pre-existing annotated data. Manual annotation of tex-tual data is known to be a time- and resource-consumingtask. The task is further complicated by expectations of theinvolvement of multiple annotators for the purpose of de-veloping objective metrics based on inter-annotator agree-ment statistics. To address this problem, an idea of dis-tributing the effort across multiple groups/contributers hasemerged. This paper presents a light-weight web applica-tion for collaborative, distributed annotation of text docu-ments - Djangology. This web application provides easy

1http://www.tc37sc4.org/index.php2http://www.ldc.upenn.edu/Catalog/

integration with existing annotation schemas, as well as abasis for rapid development of customized web-based an-notation tools and information models.

2. Related WorkIt is not uncommon for NLP research groups to rely onmanual text annotation tools which are developed in-houseand for specific domains or tasks. However, the use ofgeneral-purpose text annotation tools (providing variousdegrees of customization capabilities) has become morecommon recently. A number of stand-alone annotation ap-plications have been made publicly available to the NLP re-search community. Callisto (Day et al., 2004) is a Java an-notation framework which provides a plug-in developmentenvironment for custom annotations. MMAX2 (Mullerand Strube, 2006) and Word-Freak (Morton and LaCivita,2003) are also Java-based stand-alone tools which supportannotation schema definitions and custom XML-based an-notation exports. The GATE NLP framework (Cunninghamet al., 2002) integrates an annotation interface to the frame-work’s Java GUI (Graphical User Interface). A couple ofannotation plug-ins have been developed for the Protege 3

Java framework - iAnnotateTab (Chintan, 2005) and Know-tator (Ogren, 2006). The Knowtator plug-in has gainedpopularity and allows complex annotation schema design,inter-annotator statistics, as well as the creation of a goldstandard based on annotations from multiple annotators.As stand-alone applications, all of the annotation toolsabove require significant involvement from the annotators- software installation and typically non-trivial configura-tion. Accumulating, evaluating, and consolidating annota-tions from various annotators also involves considerable ef-fort. Web-based distributed annotation tools have been de-

3http://protege.stanford.edu

3499

veloped to avoid the hassle of stand-alone applications andto streamline collaborative annotation efforts. The GATEframework (Cunningham et al., 2002) supports collabora-tive annotation through the OLLIE client-server applica-tion (Cunningham et al., 2003). The OLLIE client uses aJava-enabled web browser and communicates to a servervia Java Applets. The web-based application uses GATE’sdistributed database facilities to authenticate and store userdata as well as the language resources being edited. Onedrawback of the system is that communication between theweb-based client and the server is achieved via Java RMI(Remote Method Invocation), which has limited browsersupport and is known to have a number of deployment dif-ficulties that render the approach impractical. Serengeti(Stuhrenberg et al., 2007) is a project-specific Mozilla Fire-fox plug-in that allows web-based annotation of anaphoricrelations using a pre-defined schema. Annozilla 4 is an-other Mozilla Firefox plug-in that allows spans of text inhtml/text documents to be associated with annotation typesand free text. Annotations can be stored in a RDF-formatlocal datastore on the user’s machine, or posted/retrievedfrom a compliant Semantic Web annotation server.

3. Motivation and Case StudyThe Djangology5 annotation web application was origi-nally created to meet the needs of a collaborative annotationproject involving more than 250 international participants.The goal of the project was to create a gold standard corpuswhich is annotated with named entities of the domainof interest: medical studies of trauma, shock, and sepsisconditions. Abstracts from an annual conference dedicatedto the subject and hosted by the the North AmericanShock Society 6 were used to identify the domain-specificnamed entities via an automated process. The named entityannotations had to then be validated by domain experts -the contributors to the conference. The Djangology systemhas been in use for two consecutive years (2008 and 2009),and has achieved an average contributor response rate of70%.

The needs of the project led to a set of requirements com-mon to similar highly-distributed collaborative annotationprojects. An administration interface was needed to man-age documents and users, as well as for the definition ofannotation schemas. Annotations created via an automatedprocess needed to be loaded into the system. Participantswere notified via email and presented with a link to theweb-based interface. After logging in, annotators were ableto view a list of assigned documents. An intuitive web-based user interface was needed to allow participants to an-notate documents with minimal instructional text. Easy andquick annotation access was crucial to the success of theproject. As the time of domain experts is quite valuable,complicated installation or annotation instructions wouldbe prohibitive. The system also needed to display inter-annotator agreement statistics, as well as the evaluation

4http://annozilla.mozdev.org/5The system is named after the 1949 Django Reinhardt album.

The name also highlights the use of the Django web framework.6http://www.shocksociety.org/

statistics comparing a gold standard against the automatedannotation.

4. System Description4.1. Technical DetailsRecent years have introduced new technologies for rapidand easy development and deployment of web applications,most notably the Ruby on Rails7 framework (developedfor the Ruby language) and the comparable Django frame-work8 (based on the Python language). For our annotationproject, the Django web framework was selected becausethe framework has excellent documentation and design,which allows for the development of high-performing, ele-gant web applications quickly. In addition, the Python pro-gramming language has traditionally been in wider use inthe academic community (compared to Ruby) and a num-ber of Python NLP frameworks and tools could be easilyintegrated into the web application if necessary.In terms of deployment, Django requires almost no config-uration as it is based on the software design principle Con-vention over Configuration. It supports almost all populardatabase servers, including PostgreSQL, MySQL, Oracleand SQLite. An out-of-the-box administration web inter-face provides facilities for user and database record man-agement, thereby reducing development time significantly.Modifying the database schema requires minimal effort onthe developer through the use of the Active Record de-sign pattern. Similarly, the framework provides support forrapid development of custom pages or modifying existinginterfaces. In addition, Django supports agile developmentpractices through built-in automated testing and support forrapid unit test writing.Djangology can be deployed on any web-accessible serverand requires a Python installation, Django installation, andconnectivity to a database server 9. Source code and in-stallation instructions can be found at the project web-site http://djangology.sourceforge.net/. We estimate thatend-to-end installation and configuration time for a Pythonand Django-savvy developer is less than an hour. Oncedeployed, the application can be accessed from any webbrowser - no browser plug-ins, JVM installation, or customsecurity settings are necessary, as the client-server commu-nication is based on standard HTTP and Ajax requests.The application database schema (Figure 1(a)) and user in-terface can be rapidly extended and customized. For ex-ample, creating a new field to annotator accounts could beeffortlessly achieved by just adding a new attribute to thecorresponding Python model class. The corresponding webform and underlying database schema are transparently up-dated by the Django framework (Figures 1(b) and 1(c)).

4.2. User Interface and WorkflowThe Djangology 10 application presents administrators withan interface to create/modify annotation projects and man-age users (Figure 2). Administrators can import documents

7http://rubyonrails.org/8http://www.djangoproject.com/9Quick install guide can be found at

http://docs.djangoproject.com/en/dev/intro/install/10http://sourceforge.net/projects/djangology/

3500

(single document or batch mode) into a project, define theproject annotation schema, create annotator accounts, andassign annotators to specific projects and to a list of docu-ments. Existing annotations and documents could also beeasily loaded into the system through custom Python scripts(stand-alone Django scripts) or through direct connectionto the Djangology database. Djangology has been used toimport manually created annotations in the Knowtator for-mat and from the BioScope Corpus (Szarvas et al., 2008)as well as annotations created automatically by the Gateand UIMA (Ferrucci and Lally, 2004) frameworks and theMetamap c© system from the National Library of Medicine.In the workflow of the system, contributors are typicallyemailed their system authentication information and pre-sented with a link to the application (Figure 3). Oncelogged in, annotators can select one of their assigned docu-ments and proceed with the web-based annotation interface.An Ajax-based web page allows contributors to highlight afragment of text and assign it to one of the pre-defined an-notation types (based on the project annotation schema).The procedure for entering new annotations and modifyingexisting annotations is intuitive and based on user interfaceconventions - text selection/right-click menu selection. Thesystem is specifically designed to require minimum time-investment on the part of the involved annotators. No in-stallation, configuration, or reading user manuals is neces-sary on the part of the contributors. Annotations are savedto the backend database as they are entered, ensuring thatno work is lost. In order to save annotators’ effort, once aphrase is annotated, all occurrences of the phrase in the doc-ument are automatically annotated in the same type. Usersare also given a facility to override the automatically cre-ated annotations or change the system’s default behavior. Ifdesired, contributors could also mark documents as com-pleted to alert the project administrator of the annotationprogress.Once annotations are gathered from various contribu-tors, project administrators have the ability to view inter-annotator agreement statistics - a variety of pair-wiseproject-based and document-based metrics are computedand presented in the user interface (Figure 4(a)). As anal-ysis of inter-annotator disagreement is a common task, aninterface for a side-by-side comparison of document anno-tations is also provided (Figure 4(b)).

5. ConclusionsThis paper presented a light-weight open-source frameworksupporting distributed annotation projects. The frameworkwas built to satisfy project needs currently not supportedby existing publicly available annotation tools - minimaltime commitment on the part of distributed contributorsand minimal development and configuration effort on thepart of project administrators. We estimated that installingboth Django and Djangology takes less than one hour fora Python-savvy programmer. As contributors can accessthe annotation project and perform annotations online, noinstallation and configuration time-investment is requiredfrom annotators. An intuitive and standard user interfacefacilitates active project participation.The Django web framework allows rapid development

of extensions and customizations without limiting usersto a pre-defined set of configuration files or requiringinvestment in complex, heavy-weight applications. Byopen-sourcing the framework, we hope to receive valuablefeedback from the community and utilize it to prioritizethe features which we are planning to incorporate in futurereleases.

6. ReferencesP. Chintan. 2005. iAnnotate Tab.D.H. Cunningham, D.D. Maynard, D.K. Bontcheva, and

M.V. Tablan. 2002. GATE: A framework and graphi-cal development environment for robust NLP tools andapplications.

H. Cunningham, V. Tablan, K. Bontcheva, and M. Dim-itrov. 2003. Language engineering tools for collabora-tive corpus annotation. In Proceedings of Corpus Lin-guistics. Citeseer.

D. Day, C. McHenry, R. Kozierok, and L. Riek. 2004. Cal-listo: A configurable annotation workbench. In Interna-tional Conference on Language Resources and Evalua-tion. Citeseer.

D. Ferrucci and A. Lally. 2004. UIMA: an architecturalapproach to unstructured information processing in thecorporate research environment. Natural Language En-gineering, 10(3-4):327–348.

N. Ide and L. Romary. 2004. International standard for alinguistic annotation framework. Natural language en-gineering, 10(3-4):211–225.

T. Morton and J. LaCivita. 2003. Word-Freak: an opentool for linguistic annotation. In Proceedings of the 2003Conference of the North American Chapter of the As-sociation for Computational Linguistics on Human Lan-guage Technology: Demonstrations, pages 17–18.

C. Muller and M. Strube. 2006. Multi-level annotation oflinguistic data with MMAX2. Corpus Technology andLanguage Pedagogy: New Resources, New Tools, NewMethods. Peter Lang, Frankfurt aM, Germany.

P.V. Ogren. 2006. Knowtator: A Protege plug-in for an-notated corpus construction. In Proceedings of the 2006Conference of the North American Chapter of the As-sociation for Computational Linguistics on Human Lan-guage Technology: companion volume: demonstrations,page 275. Association for Computational Linguistics.

M. Stuhrenberg, D. Goecke, N. Diewald, A. Mehler, andI. Cramer. 2007. Web-based annotation of anaphoric re-lations and lexical chains. In Proceedings of the ACLLinguistic Annotation Workshop, pages 140–147.

G. Szarvas, V. Vincze, R. Farkas, and J. Csirik. 2008. TheBioScope corpus: annotation for negation, uncertaintyand their scope in biomedical texts. In Proceedings ofthe Workshop on Current Trends in Biomedical Natu-ral Language Processing, pages 38–45. Association forComputational Linguistics.

A Figures

3501

(a) The Djangology database schema is transparently created and managed by the application.

(b) Both the database schema and the web forms can be effortlessly extended by simplymodifying the underlying Django model objects. In this example, a new field is addedto annotator accounts.

(c) Model object modification are immediately reflected in the web inter-face.

Figure 1: Djangology allows rapid customizations.

3502

(a) A web interface allowing administrators to manage projects and users

(b) A web interface allowing administrators to define/modify projects

Figure 2: Sample pages demonstrating the Djangology administration interface

3503

(a) Annotators are sent authenticationinformation via email and a link to ac-cess the web application

(b) List of documents assigned to a project contributor

(c) An intuitive interface allowing annotation editing based on the project annotation schema as defined by the project administrator

Figure 3: Sample pages demonstrating the interface used by project annotators

3504

(a) Project administrators can view inter-annotator agreement statistics

(b) Side-by-side document comparison page allowing project administrator to analyze sources of annotation disagreement

Figure 4: Inter-annotator agreement and error analysis tools

3505