Proceedings of Workshop for NLP Open Source Software, pages 47–51Melbourne, Australia, July 20, 2018. c©2018 Association for Computational Linguistics

47

SUMMA: Integrating Multiple NLP Technologies into anOpen-source Platform for Multilingual Media Monitoring

Ulrich GermannUniversity of Edinburgh

ugermann@ed.ac.uk

Renars Liepin, šLETA

renars.liepins@leta.lv

Didzis GoskoLETA

didzis.gosko@leta.lv

Guntis BarzdinsUniversity of Latviaguntis.barzdins@lu.lv

Abstract

The open-source SUMMA Platform is ahighly scalable distributed architecture formonitoring a large number of media broad-casts in parallel, with a lag behind actualbroadcast time of at most a few minutes.It assembles numerous state-of-the-artNLP technologies into a fully automatedmedia ingestion pipeline that can recordlive broadcasts, detect and transcribe spo-ken content, translate from several lan-guages (original text or transcribed speech)into English,1 recognize Named Entities,detect topics, cluster and summarize doc-uments across language barriers, and ex-tract and store factual claims in these newsitems.This paper describes the intended use casesand discusses the system design decisionsthat allowed us to integrate state-of-the-art NLP modules into an effective workflowwith comparatively little effort.

1 IntroductionSUMMA (“Scalable Understanding of Multilin-gual Media”) is an EU-funded collaborative effortto combine state-of-the-art NLP components intoa functioning, scalable media content processingpipeline to support large news organizations intheir daily work. The project consortium com-prises seven acadamic / research partners — theUniversity of Edinburgh, the Lativan InformationAgency (LETA), Idiap Research Institute, Prib-eram Labs, Qatar Computing Research Institute,University College London, and Sheffield Univer-sity —, and BBC Monitoring and Deutsche Welleas use case partners.In this paper, we first describe the use cases that

the platform addresses, and then the design deci-1 The choice of English as the lingua franca within

the Platform is due to the working language of ouruse case partners; the highly modular design of thePlatform allows the easy integration of custom trans-lation engines, if required.

sions that allowed us to integrate existing state-of-the art NLP technologies into a highly scalable,coherent platform with comparatively little inte-gration effort.

2 Use CasesThree use cases drive the development of theSUMMA Platform.

2.1 External Media MonitoringBBC Monitoring is a business unit within theBritish Broadcasting Corporation (BBC). In con-tinuous operation since 1939, it provides mediamonitoring, analysis, and translations of foreignnews content to the BBC’s news rooms, the BritishGovernment, and other subscribers to its services.Each of its ca. 300 monitoring journalists usuallykeeps track up to 4 live sources in parallel (typicallyTV channels received via satellite), plus a numberof other sources of information such as social mediafeeds. Assuming work distributed around the clockin three shifts,2 BBC Monitoring thus currentlyhas, on average, the capacity to actively monitorabout 400 live broadcasts at any given time — justover a quarter of the ca. 1,500 TV stations that ithas access to, not to mention other sources such asradio broadcasts and streams on the internet. NLPtechnologies such as automatic speech recognition(ASR), machine translation (MT), and named en-tity (NE) tagging can alleviate the human mediamonitors from mundane monitoring tasks and letthem focus on media digestion and analysis.

2.2 Internal MonitoringDeutsche Welle is an international broadcaster op-erating world-wide in 30 different languages. Re-gional news rooms produce and broadcast contentindependently; journalistic and programming deci-sions are not made by a central authority withinDeutsche Welle. Therefore, it is difficult for theoverall organisation to maintain an accurate andup-to-date overview of what is being broadcast,and what stories have been covered.2 The actual distribution of staff allocation over the

course of the day may differ.

48

Transcribe*

Translate*

Recogniseand link

named entities

Extract factsand relations

Cluster similar

documents

Detecttopics

Record*

Summariseclusters

Database ofannotatednews items

* if applicable

Figure 1: The SUMMA Platform Workflow

The Platform’s cross-lingual story clustering andsummarization module with the corresponding on-line visualization tool addresses this need. It pro-vides an aggregate view of recent captured broad-casts, with easy access to individual broadcast seg-ments in each cluster.

2.3 Data JournalismThe third use case is the use of the Platform’sdatabase of news coverage for investigations involv-ing large amounts of news reports, for example, ex-ploring how certain persons or issues are portrayedin the media over time.

3 System OverviewFigure 1 shows the overall system architecture.The SUMMA Platform consists of three majorparts: a data ingestion pipeline built mostly uponexisting state-of-the-art NLP technology; a web-based user front-end specifically developed withthe intended use cases in mind; and a databaseat the center that is continuously updated by thedata ingestion pipeline and accessed by end usersthrough the web-based GUI, or through a RESTAPI by downstream applications.The user interfaces and the database structure

were custom developments for the SUMMA Plat-form. The grapical user interfaces are based oninput from potential users and wireframe designsprovided by the use case partners. They are im-plemented in the Aurelia JavaScript Client Frame-work3. For NLP processing, we mostly build onand integrate existing NLP technology. We de-scribe key components below.3 aurelia.io

3.1 Languages CoveredThe goal of the project to offer NLP capabilities forEnglish, German, Arabic, Russian, Spanish, Lat-vian, Farsi, Portuguese, and Ukrainian. We cur-rently cover the former 6; the latter 3 are work inprogress.Due to the large number of individual compo-

nents (number of languages covered times NLPtechnologies), it is not possible to go into detailsabout training data used and individual compo-nent performance here. Such details are coveredin the SUMMA project deliverables D3.1 (Garneret al., 2017), D4.1 (Obamuyide et al., 2017), andD5.1 (Mendes et al., 2017), which are availablefrom the project’s web site.4 We include applica-ble deliverable numbers in parentheses below, sothat the inclined reader can follow up on details.

3.2 Live Stream Recorder and ChunkerThe recorder and chunker monitors one or morelive streams via their respective URLs. Broad-cast signals received via satellite are converted intotransport streams suitable for streaming via theinternet and provided via a web interface. Thishappens outside of the platform since it requiresspecial hardware. The access point is a live streamprovided as an .m3u8 playlist via a web interfacethat is polled regularly by the respective data feedmodule.All data received by the Recorder-and-Chunker

is recorded to disk und chunked into 5-minutes seg-ments for further processing. Within the Plat-form infrastructure, the Recorder-and-Chunkeralso serves as the internal video server for recorded

4 www.summa-project.eu/deliverables

49

transitory material, i.e., material not obtainedfrom persistent sources.Once downloaded and chunked, a document stub

with the internal video URL is entered into thedata base, which then notifies the task schedulerabout the new arrival, which in turn schedules theitem for downstream processing.Video and audio files that are not transitory but

provided by the original sources in more persistentforms (i.e., served from a permanent location), arecurrently not recorded5 but retrieved from the orig-inal source when needed.

3.3 Other Data FeedsText-based data is retrieved by data feed modulesthat poll the providing source at regular intervalsfor new data. The data is downloaded and enteredinto the database, which then again notifies thetask scheduler, which in turn schedules the newarrivals for downstream processing.In addition to a generic RSS feed monitor, we

use custom data monitors that are tailored to spe-cific data sources, e.g. the specific APIs thatbroadcaster-specific news apps use for updates.The main task of these specialized modules is tomap between data fields of the source API’s spe-cific response (typically in JSON6 format), and thedata fields used within the Platform.

3.4 Automatic Speech Recognition (D3.1)The ASR modules within the Platform are built ontop of CloudASR (Klejch et al., 2015); the underly-ing speech recognition models are trained with theKaldi toolkit (Povey et al., 2011). Punctuation isadded using a neural MT engine that was trainedto translate from un-punctuated text to punctua-tion. The training data for the punctuation moduleis created by stripping punctuation from an exist-ing corpus of news texts. The MT engine usedfor punctuation insertion uses the same softwarecomponents as the MT engines used for languagetranslation.

3.5 Machine Translation (D3.1)The machine translation engines for languagetranslation currently use the Marian7 decoder(Junczys-Dowmunt et al., 2016) for translationwith neural MT models trained with the Nematustoolkit (Sennrich et al., 2017). We have recentlyswitched to the Marian toolkit for training.

3.6 Topic Classification (D3.1)The topic classifier uses a hierarchical attentionmodel for document classification (Yang et al.,5 On a marginal note, recording a single live stream

produces, depending on video resolution, up to 25GiB of data per day.

6 https://www.json.org7 www.github.com/marian-nmt

2016) trained on nearly 600K manually annotateddocuments in 8 languages.

3.7 Storyline Clustering (D3.1) andCluster Summarization (D5.1)

Incoming stories are clustered into storylines withAggarwal and Yu’s (2006) online clustering algo-rithm. The resulting storylines are summarizedwith the extractive summarization algorithm byAlmeida and Martins (2013).

3.8 Named Entity Recognition andLinking (D4.1)

For Named Entity Recognition, we use TurboEn-tityRecognizer, a component within TurboParser8(Martins et al., 2009). Recognized entities and re-lations between them (or propositions about them)about them are linked to a knowledge base offacts using techniques developed by Paikens et al.(2016).

3.9 DatabasesThe Platform currently relies on two databases.The central database in the NLP processingpipeline is an instance of RethinkDB9, a document-oriented database that allows clients to subscribeto a continuous stream of notifications aboutchanges in the database. This allows clients (e.g.the task scheduler) to be notified about the lastestincoming items as they are added to the database,without the need to poll the database periodically.Each document consists of several fields, such asthe URL of the original news item, a transcript foraudio sources, or the original text, a translationinto English if applicable, entities such as persons,organisations or locations mentioned in the newsitems, etc.For interaction with web-based user interfaces,

we are using a PostgreSQL10 database, which isperiodically updated with the latest arrivals fromthe data ingestion pipeline. This second databasewas not part of the original design; it was addedout of performance concerns, as we noticed at somepoint that RethinkDB’s responsiveness tended todeteriorate over time as the number of items in thedatabase grew. Ultimately, this turned out to bean error in the set-up of our RethinkDB instance:certain crucial fields weren’t indexed, so that Re-thinkDB resorted to a linear search for certain op-erations. The current split between two databasesis not ideal; however, it is operational and elimi-nating it is not a high priority on the current de-velopment agenda.

8 https://github.com/andre-martins/TurboParser9 www.rethinkdb.com10 www.postgresql.org

50

4 Platform Design andImplementation

As is obvious from the description above, the Plat-form utilizes numerous existing technologies. Indesigning and implementing the Platform, we hadtwo main objectives:

1. Minimize the effort necessary to integrate thevarious existing technologies.

2. Keep individual NLP components as indepen-dent as possible, so that they can be re-usedfor other purposes as well.

In order to meet these objectives, we designed theprocessing pipeline as a federation of microservicesthat communicate with each other via REST APIsand/or a message queue.For rapid prototyping, we defined REST APIs

for each NLP module within the OpenAPI Specifi-cation Framework11. The suite of Swagger tools12associated with OpenAPI allowed us to specifyREST APIs quickly and generate boilerplate codefor the back-end server of each microservice. Thisreduced integration efforts to implementing a fewback-end functions for each RESTful server — inwhatever programming language the contributingpartner felt most comfortable with.In the Platform prototype, we used separate pro-

cesses that act as dedicated intermediaries betweenthe message queue13 and each individual NLP com-ponent. As the Platform matures, we graduallymoving to implementing RabbitMQ interfaces di-rectly with the NLP processors, to eliminate theintermediaries and reduce the overall complexityof the system.One of the great advantages of using a message

queue is that it makes scaling and load balancingeasy. If we need more throughput, we add moreworkers (possibly on different machines) that talkto the same message queues. Each type of task hastwo queues: one for open requests, the other one forcompleted requests. Each worker loops over wait-ing for a message to appear in the queue, popping itof it, acknowledging it upon successful completion(so that it can be marked as done), and pushingthe response onto the respective queue. Workersneed not be aware of the overall architecture ofthe system; only the overall task scheduler has tobe aware of the actual work flow.Maintaining the RESTful APIs is nevertheless

worthwhile: it allows individual components to bedeployed easily as a service outside of the Plat-form’s context and workflow.

11 www.openapis.org; formerly Swagger12 www.swagger.io13 RabbitMQ (https://www.rabbitmq.com/) works

well for us.

The overall NLP processing workflow is designedas an icremental annotation process: each ser-vice augments incoming media items with addi-tional information: automatic speech recognition(ASR) services add transcriptions, machine trans-lation (MT) engines add translations, and so forth.Each component of the Platform runs indepen-

dently in a Docker14 application container. Similarto conventional virtual machines (VMs), Dockercontainers isolate applications from the host sys-tem by running them in a separate environment(“container”), so that each application can use itsown set of libraries, ports, etc. However, unlikeconventional VMs, which emulate a complete ma-chine including device and memory management,the Docker engine allows containers to share thehost’s kernel and resources, greatly reducing thevirtualisation / containerization overhead. Forsmall-scale single-host deployment (up to ten livestreams on a 32-core server), we use Docker Com-pose;15 for multi-host scaling, Docker Swarm16.Another great advantage of the Docker platform

is that many third-party components that we relyon (message queue, data bases) are available aspre-compiled Docker containers, so that their de-ployment within the Platform is trivial. No de-pendencies to manage, no compilation from scratchrequired, no conflicts with the OS on the host ma-chine.Our approach to the design of the Platform has

several advantages over tigher integration within asingle development framework in which contribut-ing partners would be required to provide softwarelibraries for one or more specific programming lan-guages.First, in line with our first design objective, it

minimizes the integration overhead.Second, it is agnostic to implementational

choices and software dependencies of individualcomponents. Each contributing partner can con-tinue to work within their preferred developmentenvironment.17Third, it provides for easy scalability of the sys-

tem, as the Platform can be easily distributed overmultiple hosts. With the message queue approach,multiple workers providing identical services canshare the processing load.Fourth, modules can be updated without having

to re-build the entire system. Even live continu-ous upgrades and server migration can be accom-plished easily by starting up a new instance of aspecific service and then shutting down the obso-

14 www.docker.com15 https://docs.docker.com/compose/16 https://docs.docker.com/engine/swarm17 In the case of Windows-based software, however, li-

censing issues have to be considered for deploymentin container environments such as Docker.

51

lete one.Fifth, meeting our second design objective, the

strong modularization of the Platform allows foreasy re-use of components. The back-end server foreach component can easily be integrated into otherapplications (albeit potentially requiring augmen-tations to the API).

5 Conclusion

We have reported on our experiences in imple-menting a high-performance, highly scalable natu-ral language processing pipeline from existing im-plementations of state-of-the art as an assembly ofcontainerized micro-services. We found that thisapproach greatly facilitated technology integrationand collaboration, as it eliminated many pointsof potential friction, such as having to agree ona joint software development framework, adaptinglibraries, and dealing with software dependencies.Using the Docker platform, we are able to deployand scale the Platform quickly.

Availability

The Platform infrastructure and most of its com-ponents are scheduled to be released as open-sourcesoftware by the end of August 2018 and will beavailable through the project web site at

www.summa-project.eu.

Acknowledgements

This work was conducted within the scopeof the Research and Innovation Action

SUMMA, which has received funding from the Eu-ropean Union’s Horizon 2020 research and innova-tion programme under grant agreement No 688139.

ReferencesAggarwal, Charu C and Philip S Yu. 2006. “Aframework for clustering massive text and cat-egorical data streams.” SIAM Int’l. Conf. onData Mining, 479–483.

Almeida, Miguel B and Andre FT Martins. 2013.“Fast and robust compressive summarizationwith dual decomposition and multi-task learn-ing.” ACL, 196–206.

Garner, Philip N., Alexandra Birch, AndreiPopescu-Belis, Peter Bell, Herve Bourlard, SteveRenals, Sebastião Miranda, and Ulrich Ger-mann. 2017. SUMMA Deliverable D3.1: InitialProgress Report on Shallow Stream Processing.Tech. rep., The SUMMA Consortium.

Junczys-Dowmunt, Marcin, Tomasz Dwojak, andHieu Hoang. 2016. “Is neural machine transla-tion ready for deployment? A case study on 30translation directions.” International Workshop

on Spoken Language Translation. Seattle, WA,USA.

Klejch, Ondřej, Ondřej Plátek, Lukáš Žilka, andFilip Jurčíček. 2015. “CloudASR: platform andservice.” Int’l. Conf. on Text, Speech, and Dia-logue, 334–341.

Martins, André FT, Noah A Smith, and Eric PXing. 2009. “Concise integer linear programmingformulations for dependency parsing.” ACL,342–350.

Mendes, Afonso, Pedro Balage, Mariana Almeida,Sebastião Miranda, Nikos Papasarantopou-los, Shashi Narayan, and Shay Cohen. 2017.SUMMA Deliverable 5.1: Initial Progress Reporton Natural Language Understanding. Tech. rep.,The SUMMA Consortium.

Obamuyide, Abiola, Andreas Vlachos, JeffMitchell, David Nogueira, Sebastian Riedel,Filipe Aleixo, Samuel Broscheit, Andre Mar-tins, Mariana Almeida, Sebastião Miranda,Afonso Mendes, and Andrei Popescu-Belis. 2017.SUMMA Deliverable D4.1: Initial Progress Re-port on Automatic Knowledge Base Creation.Tech. rep., The SUMMA Consortium.

Paikens, Peteris, Guntis Barzdins, Afonso Mendes,Daniel Ferreira, Samuel Broscheit, Mariana S. C.Almeida, Sebastião Miranda, David Nogueira,Pedro Balage, and André F. T. Martins. 2016.“SUMMA at TAC knowledge base populationtask 2016.” TAC. Gaithersburg, Maryland,USA.

Povey, Daniel, Arnab Ghoshal, Gilles Boulianne,Lukáš Burget, Ondrej Glembek, Nagendra Goel,Mirko Hannemann, Petr Motlíček, YanminQian, Petr Schwarz, Jan Silovský, Georg Stem-mer, and Karel Veselý. 2011. “The Kaldi speechrecognition toolkit.” ASRU.

Sennrich, Rico, Orhan Firat, Kyunghyun Cho,Alexandra Birch, Barry Haddow, JulianHitschler, Marcin Junczys-Dowmunt, SamuelLäubli, Antonio Valerio Miceli Barone, JozefMokry, and Maria Nadejde. 2017. “Nematus: atoolkit for neural machine translation.” EACLDemonstration Session. Valencia, Spain.

Yang, Zichao, Diyi Yang, Chris Dyer, XiaodongHe, Alexander J. Smola, and Eduard H. Hovy.2016. “Hierarchical attention networks for doc-ument classification.” NAACL. San Diego, CA,USA.