Mappa Mundi: An Interactive Artistic Mind Map Generator with ArtificialImagination

Ruixue Liu1 , Baoyang Chen2 , Meng Chen1 , Youzheng Wu1 , Zhijie Qiu2 and Xiaodong He11JD AI, Beijing, China

2Central Academy of Fine Arts, Beijing, China{liuruixue,chenmeng20,wuyouzheng1,xiaodong.he}@jd.com, {chenbaoyang, qiuzhijie}@cafa.edu.cn

AbstractWe present a novel real-time, collaborative, and in-teractive AI painting system, Mappa Mundi, forartistic Mind Map creation. The system consistsof a voice-based input interface, an automatic topicexpansion module, and an image projection mod-ule. The key innovation is to inject Artificial Imag-ination into painting creation by considering lexicaland phonological similarities of language, learningand inheriting artist’s original painting style, andapplying the principles of Dadaism and impossi-bility of improvisation. Our system indicates thatAI and artist can collaborate seamlessly to createimaginative artistic painting and Mappa Mundi hasbeen applied in art exhibition in UCCA, Beijing1.

1 IntroductionMind Map is a diagram that connects relevant words, termsor events around a keyword or an idea and it’s perceived asan artistic possibility representation of knowledge. Throughconstant interpretation and abstraction, the artistic creationof Mind Map is a process of subject decision making andinformation expansion. In this paper, we introduce MappaMundi, an interactive Mind Map generator featured with arti-ficial imagination.

The creation of Mind Map requires the interaction betweenAI and artist on the decision making of idea expansion. Whileprevious AI image generators, such as pix2pix [Isola et al.,2017] framework, depends only on the model performanceand less on taking artist’s interaction into account. Besides,models constructed with neural approaches barely extract fea-tures from big data, whereas features, such as imaginationand creativity, are far to reach. As Mind Map represents theuniversal mind of human thinking, the Kantian thinking offree play of imagination, which is the foundation of modernarts [Kant, 1987], plays an important role in expanding artist’sthought and connecting the flourishing information. Differentfrom the traditional framework, Mappa Mundi offers oppor-tunities to enable AI and artist together during art making.

Mappa Mundi enables voice input from human and ex-tracts the keywords for vivid expansion on myriad levels of

1http://ucca.org.cn/en/exhibition/qiu-zhijie-mappa-mundi/

domains, which explores the impossibility of improvisationvariation arising from standardisation. To improve the imag-inative feature of word expansion results, Mappa Mundi em-ploys not only semantic similarity and linguistic features butalso establishes a knowledge graph to imitate artist’s ideaand style. Besides, Mappa Mundi breaks conventions andexplores more possibilities for word’s connections partiallystem from the Dadaism art movement. Through interactionsparked in our system, the free play of human imaginationand machine intelligence is consolidated, and eventually con-stitutes a contemporary Mappa Mundi, which cultivates amore and more electrified arena for endless imagination ofboth human and machine.

2 Related Work and ChallengesNowadays, AI demonstrates stronger potential for art cre-ation. Many researches have been conducted to involveAI into poem generation [Zhang and Lapata, 2014; Chenget al., 2018], creation of classical or pop music [Manzelliet al., 2018; Hadjeres et al., 2017] and automatic imagesgeneration [van den Oord et al., 2016; Yan et al., 2016;Xu et al., 2018]. Whereas there are few researches exploringthe possibility of artificial imagination for artwork creation.

To generate an imaginative and creative Mind Map, thekey problem is automatic topic expansion. There are severalchallenges: first of all, language is an informative system.Thus many linguistic features should be considered in wordsand relation extension. Second, as an artwork, it should re-flect artist’s mind and individual experience. Third, for MindMap creation, artists usually pay less attention on definingaccurate relation between concepts, instead, they are definedwith imagination. So, to build artistic Mind Map, the sys-tem should find a way to break the convention rules for theconnection between literal representations.

3 System ArchitectureThe overall architecture of Mappa Mundi is illustrated in Fig-ure 1. It consists of three modules in total. The first one is avoice-based input interface, which translates user’s voice in-put into text and extracts keyword from it. The second is topicexpansion module, which expands the keyword into severalword candidates that share various connections with it. Thethird module is image projection, which takes the keyword

Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI-19)

6545

Figure 1: System architecture of Mappa Mundi

and its candidates as input and projects the concepts and re-lations into a Mind Map image. As an interactive system,Mappa Mundi presents artist’s speech as Mind Map imageand this imaginative picture in return provokes artist’s aspira-tion for further thoughts expansion.

3.1 Voice-based Input InterfaceAutomatic Speech Recognition (ASR) engine is designed forinteraction. Both English and Chinese languages are sup-ported. We also performed lexicon and language model adap-tion to enhance the recognition of words in art domain. Afterthat, we developed a keyword extraction function to extractmeaningful words from the converted text for further expan-sion. We adopted open-source software jieba2 for Chineseand Stanford parser [Toutanova and Manning, 2000] for En-glish POS tagging. Only informative words such as noun,verb and adjective words are kept and TFIDF weights are cal-culated for further filtering. The output of this module is thekeywords.

3.2 Topic ExpansionAs imagination is the soul for artistic Mind Map, MappaMundi employs several features to increase information va-riety [Liu et al., 2019] during topic expansion. It firstlyuses word embeddings to find candidates based on seman-tic similarity [Mikolov et al., 2013; Pennington et al., 2014;Peters et al., 2018]. To enrich linguistic information of ex-pansions, it also takes the morphological and phonologicalfeatures into account. Words sharing similar characters ormorphemes or phonetic syllables are selected as candidates.Moreover, we construct a knowledge graph (KG) by extract-ing concepts and relations from artist’s original Mind Mapmasterpieces. Mappa Mundi will adopt those concepts andrelations once a topic word is covered by the KG. By doingso, it can better imitate artists’ mind and thoughts. Further,Mappa Mundi breaks the restriction of domains and conven-tions and explores more possibilities for words’ connectionsby creating rules following the famous Dadaism principle inart movement [Elger, 2004]. Finally, with all above men-tioned methods, Mappa Mundi can create a branch of creativeand informative word candidates for a given keyword.

2https://github.com/fxsjy/jieba

Figure 2: Screen-shot of a generated Mind Map image with voiceinput of Beijing Bids for 2022 Winter Olympics

3.3 Image ProjectionTo visualize the abstract concepts, Mappa Mundi includesaround 3000 painting elements which are extracted fromthe representative Mind Map paintings of a famous artist3.These elements are sharing traditional Chinese Shan-shui(Mountain-river) painting style and are further analyzed into5 typical Shan-shui painting categories including architec-ture, mountain, river, grassland and lake. Mappa Mundi willclassify each word into one of the five categories above andthen connect it with the elements in the corresponding do-main. An additional type of painting element Route is alsoinvolved. The distance between the concept and it’s candidateis determined by their similarity score and then visualized asRoute in image. The more similar these two words are, thecloser distance will be.

Our Mappa Mundi is not a one-shot completion, it is aninteraction of live stimulations, which contains constant vo-cal inputs via ASR and outputs of generated words and theirrelations. It is a two-way street. The generated information,after being presented in our system, becomes the inspirationfor artist’s next vocal input. This artwork reflects both thedevelopment of artist’s thinking and the AI-enabled imagina-tion.

4 ConclusionIn this paper, we propose a Mind Map generator, MappaMundi, that can inject Artificial Imagination together withhuman interaction into an artwork. The way of enriching ar-tificial imagination includes considering semantic similarity,integrating informative linguistic features, inheriting artist’smind, and inserting Dadaism and impossibility of improvisa-tion principles. Mappa Mundi is presented to understand howAI can aid artistic practice. Its development also indicates AIhas more implications in art than a way of artistic creation,and it enriches the possibility of artistic practice.

AcknowledgmentsWe want to thank Yan Dai, Chuyan Xu, Haozhen Feng andXiaoyu Guo for their work and support to build this system.

3https://en.wikipedia.org/wiki/Qiu Zhijie

Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI-19)

6546

References[Cheng et al., 2018] Wen-Feng Cheng, Chao-Chung Wu,

Ruihua Song, Jianlong Fu, Xing Xie, and Jian-Yun Nie.Image inspired poetry generation in xiaoice. arXivpreprint arXiv:1808.03090, 2018.

[Elger, 2004] Dietmar Elger. Dadaism. Taschen, 2004.[Hadjeres et al., 2017] Gaetan Hadjeres, Francois Pachet,

and Frank Nielsen. Deepbach: a steerable model for bachchorales generation. In Proceedings of the 34th Interna-tional Conference on Machine Learning-Volume 70, pages1362–1371. JMLR. org, 2017.

[Isola et al., 2017] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou,and Alexei A Efros. Image-to-image translation with con-ditional adversarial networks. CVPR, 2017.

[Kant, 1987] Immanuel Kant. Critique of judgment. HackettPublishing, 1987.

[Liu et al., 2019] Ruixue Liu, Baoyang Chen, Xiaoyu Guo,Yan Dai, Meng Chen, Zhijie Qiu, and Xiaodong He. Fromknowledge map to mind map: Artificial imagination. Pro-ceedings of the Second International Conference on Mul-timedia Information Processing and Retrieval, pages 496–501, 2019.

[Manzelli et al., 2018] Rachel Manzelli, Vijay Thakkar, AliSiahkamari, and Brian Kulis. Conditioning deep gener-ative raw audio models for structured automatic music.arXiv preprint arXiv:1806.09905, 2018.

[Mikolov et al., 2013] Tomas Mikolov, Wen-tau Yih, andGeoffrey Zweig. Linguistic regularities in continuousspace word representations. In Proceedings of the 2013Conference of the North American Chapter of the Asso-ciation for Computational Linguistics: Human LanguageTechnologies, pages 746–751, 2013.

[Pennington et al., 2014] Jeffrey Pennington, RichardSocher, and Christopher Manning. Glove: Global vectorsfor word representation. In Proceedings of the 2014conference on empirical methods in natural languageprocessing (EMNLP), pages 1532–1543, 2014.

[Peters et al., 2018] Matthew E Peters, Mark Neumann, Mo-hit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee,and Luke Zettlemoyer. Deep contextualized word repre-sentations. arXiv preprint arXiv:1802.05365, 2018.

[Toutanova and Manning, 2000] Kristina Toutanova andChristopher D Manning. Enriching the knowledgesources used in a maximum entropy part-of-speech tagger.In Proceedings of the 2000 Joint SIGDAT conference onEmpirical methods in natural language processing andvery large corpora: held in conjunction with the 38thAnnual Meeting of the Association for ComputationalLinguistics-Volume 13, pages 63–70. Association forComputational Linguistics, 2000.

[van den Oord et al., 2016] Aaron van den Oord, Nal Kalch-brenner, Lasse Espeholt, Oriol Vinyals, Alex Graves,et al. Conditional image generation with pixelcnn de-coders. In Advances in Neural Information ProcessingSystems, pages 4790–4798, 2016.

[Xu et al., 2018] Tao Xu, Pengchuan Zhang, QiuyuanHuang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xi-aodong He. Attngan: Fine-grained text to image gener-ation with attentional generative adversarial networks. In2018 IEEE Conference on Computer Vision and PatternRecognition (CVPR). IEEE, 2018.

[Yan et al., 2016] Xinchen Yan, Jimei Yang, Kihyuk Sohn,and Honglak Lee. Attribute2image: Conditional imagegeneration from visual attributes. In European Conferenceon Computer Vision, pages 776–791. Springer, 2016.

[Zhang and Lapata, 2014] Xingxing Zhang and Mirella Lap-ata. Chinese poetry generation with recurrent neural net-works. In Proceedings of the 2014 Conference on Empir-ical Methods in Natural Language Processing (EMNLP),pages 670–680, 2014.

Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI-19)

6547