Data Mining Arts – Learning to Paint

RapidMiner Processes

Christian BockermannLehrstuhl fur kunstliche IntelligenzTechnische Universitat Dortmund

Deutschland

Abstract

Designing data mining processes is a creative task that requires ex-plorative, analytical thinking and often offers a large degree of freedomto the analyst. In this work we explore the composition of data min-ing process by means of drawing sketches with interactive devices liketablets or smart white-boards. Using machine learning operators builtinto RapidMiner itself we extend the user experience of the RapidMineruser interface using todays omnipresent touch pads trying to provide theanalyst with a new way of sketching her analytical thoughts.

1 Introduction

Mining data is an art by itself – designing business analytical processes re-quires a large portion of business understanding and a fine sense for mappingthe required actions into data analysis processes. Like a painter, the analystneeds to create a blueprint of such processes that, which then need to be re-fined to the implementation level. The designing requires either a high-levelmodelling language as provided by tools like R [10] and Matlab [8] or a toolsuite that comes with predefined operators such as RapidMiner [9] or librarieslike WEKA or MOA [4, 5]. With the RapidMinder tool suite, the user ex-perience for creating analytical processes has improved a lot over the pastyears. Its highlevel building blocks by means of operators and features suchas searching for operators, instant process validation using meta-data, simpledrag-and-drop behavior or abstract data repositories have opened the worldof data analysis to a wide range of business analysts and scientists.

While these improvements assist the analysist in implementing the analysisprocess within RapidMiner processes, they are still stuck to the traditional wayof providing user interfaces to end-users in the desktop application manner.

With the ubiquitous presence of touch pads or tablet computers, various newways of user interaction have become possible: Modern tablets support voiceinput or various gestures to interact with the applications. Interactive multi-touch tables or smart white-board solutions extend this concept to even largerphysical areas of interaction. A nice example for a cheap interactive white-board systems is the WIImote project, which provides an smart white-boardsolution based on the Nintendo Wii sensors [3, 7]. Other projects investigatethe use of Microsoft’s Kinect camera for user interaction. This led to thefollowing question:

How can we integrate these modern ways of interacting with the designingof analytical processes in RapidMiner?

In this paper we elaborate the use of gestures for designing data analytical pro-cesses. Our contributions are twofold: First, we provide an application calledRapidMiner Artist, which provides a sketch pad for drawing objects with amouse or by touch interaction. Within the RapidMiner Artist we derive ges-tures from the drawing actions and present a way to map these gestures toactions that build up a RapidMiner process. Second, we provide the Rapid-Miner Arts plugin that allows for gesture detection using the machine learningmethods provided by RapidMiner itself and exports remote editing of Rapid-Miner processes, e.g. by the use of tablets or other remote devices. With theclassifiers provided by RapidMiner, this allows for training of custom gesturesthat have been recorded with the RapidMiner Artists. This leads to a way ofadapting RapidMiner to the user gestures and for designing business analyticalprocesses using interactive devices.

The rest of this paper is structured as follows: In Section 2 we relate thedrawing of sketches to gestures performed with an interactive user interface andformalize the considered interactive environment. Here we also establish a linkbetween the notion of gestures and the symbolization of RapidMiner processelements. In Section 3 we investigate the use of machine learning for predictinggestures and mapping these to the editing actions within RapidMiner. Basedon this context and the prediction results, we outline the RapidMiner Artsplugin and its combination with the RapidMiner Artist application. Finallywe conclude the paper and give an outlook on future work in Section 5.

2 Deriving Processes from User Drawings

The designing of analytical processes is usually guided by dividing the task athand into small steps each of which can be solved by a RapidMiner operatoror another sub-process. As an illustrative example, we pursue the design of aprocess that reads data for training a model and validates it using some testdata set. Further we assume that the data requires the selection of attributes

as an additional preprocessing step. Figure 1 shows two slightly differenthand-drawn sketches of such a process. Such sketches can easily be drawnwith interactive touch pads or smart-boards as mentioned in the beginning.

Figure 1: Drawings of a simple RapidMiner process.

2.1 Drawing Sketches and Gestures

The challenge we are looking at is how to derive the exact RapidMiner pro-cesses from the interactions the user performs for creating such sketches ona tablet PC or a smart-board. As the basis for our work we regard the userinteractions to be performed on a 2-dimensional area, such as a touch pad, aninteractive white board or a screen. The area can be interacted with by theuse of a pointer, a finger or a pen. To formalize this, we define the notion ofa sketch pad as follows:

Definition 1: Let w, h ∈ R with w, h > 0. A sketch pad P = [0, w] × [0, h]is a 2-dimensional space with width w and height h. A user interaction withP is a sequence of points (xi, yi, ti) ∈ P ×T that result from the user touchingthe pad. Each point consists of coordinates xi and yi and a timestamp ti ofthe time at which the interaction took place.

The resolution of coordinate system of P and the time T is dependent onthe device that is used for user interaction. Usually a time resolution in theorder of milliseconds is an adequate granularity. If we have a closer look at thecreation of the sketches of Figure 1 it is obvious that they have been drawnwith a sequence of moves of a pen. If we leave away the labels of the sketches,then Figure 2 shows different stages of the drawing that eventually lead to thesketch of the process as shown in Figure 1.The dissection of the drawing into distinct steps is one of the key parts for theRapidMiner Artist application that we will present later on. A simple way tobreak down the drawing into a sequence of moves is define a move as a singlepen stroke. This leads to the following definition of a gesture:

Definition 2: A move is a sequence of points that starts with the touch downof a pen and ends with the lift up of the pen. For a given time interval τ , agesture is sequence of moves 〈m0, . . . ,ml〉 such that the time between the endof move mi and the start of move mi+1 is less or equal to τ .

Figure 2: Stepwise creation of a process sketch.

With that definition, a gesture is a connected set of moves that leads todistinct parts of the drawing. For example, each of the two boxes of the laststep of Figure 2 is the result of a single gesture. In case of the boxes, thesegestures consist of multiple moves1, whereas the line that connects the twoboxes is the result of a gesture of only a single move. Another consequenceof this definition is, that we can now regard a sketch simply as a sequence ofgestures, each of which relates to an element of the sketch. The task of hand-writing recognition, which is closely related to our gesture detection problem,inherits the problem of segmentation, i.e. a scanned writing needs to be dis-sected into single characters pictograms, which are then classified into letters.In the gesture detection, we solve this by splitting the interactive recording ofgestures over time using the above definition.

2.2 From Gestures to Process Elements

Based on the dissection of figures into gestures, we now need to define a map-ping of gestures to the elements of the processes that we want to design. Inthe context of RapidMiner, the elements we are “drawing” are operators andthe edges connecting the operators. We are however not limited to only mapgestures to process elements – we may also be interested in providing gesturesthat perform actions on the process we are drawing. As an example, we definea delete gesture, which provides a convenient way to remove elements from aprocess. Figure 3 shows the defined gestures that are part of the RapidMinerArtist tool.

Figure 3: Gestures defined within the RapidMiner Artist.

1We assume that the boxes have been drawn with multiple strokes. Of course it is alsopossible to draw the same boxes with a single stroke only.

This set of pre-defined gestures serves as the alphabet for writing up Rapid-Miner processes with the RapidMiner Artist tool. The next step required is todetect these gestures within the process of drawing a process for RapidMiner.

3 Gesture Prediction with RapidMiner

The gestures are derived from the user interactions based on the gesture def-inition in Section 2. Given a predefined set of known gestures, such as listedin Figure 3, the mapping of gestures to process elements can be viewed as amulti-class classification task: Input to the classification is a feature represen-tation of the gesture and the output (class label) will be the correct processelement or action that is associated with the gesture. In the following sectionwe will outline a basic feature representation for gesture prediction and extendthis with some extra information that can be derived from the gesture context.

3.1 Feature Representation for Gestures

The general idea for representing gestures as a feature vector reclines from theMNIST handwriting recognition [6]. This popular data set is related to thetask of identifying digits from handwritten graphics, e.g. ZIP codes that havebeen scanned from letters or post cards. Figure 4 shows sample digits of theMNIST data set.

Figure 4: Some digits of the MNIST handwritten dataset.

The digits are represented by a grayscale bitmap image of some squaredsize like 28×28 pixels. Each pixel holds a gray color value. The feature vectorextracted from the digit images is a sparse vector of length p = 28 · 28 = 784where each attribute represents a pixel of the bitmap image. As a slightmodification, each pixel can be weighted by its euclidean distance to the centerof the bitmap.

Recording features from Gestures

An important advantage over the handwriting recognition task as provided bythe MNIST data is the interactivity of the recorded gestures in the use caseat hand: In addition to the points a user touches when performing a gesture,we are able to obtain additional information such as the order in wich thegesture is drawn as well as speed and timing information of the gesture. By

recording pen-up events we are also provided with any interruptions of thegesture, which provides the number of strokes/moves a gesture is build upon.With this information we are able to extract much more fine grained featurevectors from each of the users gestures than only relying on the bitmap featuresas proposed in the MNIST dataset.

Based on the recordings of user interactions, we mapped each gesture ontoa square area of size 16× 16 and derived the following features for each of thegestures:

1. per-pixel feature: euclidean distance of the pixel to the center,

2. gesture duration: time in milliseconds of the user interaction,

3. height, width: original height, width of the gesture interaction (in pixels),

4. numberOfStrokes: the number of strokes the gesture consists of,

5. scale:x, scale:y: factors used to scale the gesture to the 16× 16 square.

With the gesture area size of 16 pixels and the additional features enlisted,we end up with 295 numerical base features for each of the recorded gestures(289 pixel-features plus the 6 additional features). The recordings themselvesare performed at the resolution of the sketch pad at hand. This allows forvariations of the feature extraction by means of different scalings (instead of16× 16 pixels). In addition, the weight of each pixel feature can be based ondifferent objectives, such as the center distance (see above), the time of thepixel recording or a simple binary value (e.g. 1.0 for pixels that belong to thegesture, and 0.0 for the other pixels). We explored different settings of thefeature extraction in Section 3.3.

3.2 Adding Context Features to Gesture Examples

While recording gestures during user interaction, a few additional features canbe derived from the interaction context. The interaction context is given by theset of elements that have been defined based upon previous gestures. A straightforward example are the start- and end-points of gestures. Figure 5 shows theConnection gesture (orange stroke) that starts within a source element andends in a target element. The start and end elements can be encoded intothe feature vector using binominal features and may not only refer to theexact element level, but may also result in the type of elements from whicha gesture starts or on which a gesture ends. As an example, the Connectiongesture in Figure 5 could be enriched with features source:type=Data Sourceand target:type=Operator. Other context based features may be derived fromprocess elements that are covered by the area of a gesture or elements thatappear near the area of a user gesture.

Figure 5: Gesture that starts in one element and ends in another one.

3.3 Evaluation of the Gesture Prediction

The RapidMiner Arts plugin, which we will introduce in Section 4 includes agesture trainer for recording labeled gestures. The trainer additionally allowsfor specifying additional context features for each of the recordings. For theprediction of gestures with RapidMiner, we recorded 60 gestures for each ofthe gestures listed in the pre-defined gesture alphabet in Section 2.2. Fromthese 300 gestures we extracted feature vectors as described above. We applieda simple Normalization operator to the example set resulting from the featureextraction and used this normalized data set as basis for training classifiers forgesture prediciton.

Testing the Feature Extraction

As we were especially interested in the required resolution of the per-pixelfeatures, we re-created the example set with different per-pixel resolutions byscaling the gestures to formats of 12 × 12, 16, 24 × 24 and 32 × 32 pixels.For estimating the quality of the features, we used a fixed learning scheme –a SVM classifier with a linear kernel (C = 1000.0). This classifier performedwell on a wide range of test cases. Table 1 shows the accuracy of the differentfeature extractions for the SVM classifier. The experiments in Table 1 wereperformed on the per-pixel features only. The additional features such as thecontext features, scaling and the number of strokes have been filtered out usingthe Select Attributes operator.

As in this multi-class problem we are especially interested in a high classrecall for each of the classes, we listed the class recall for the worst predictedclasses in an additional column. The accuracy is averaged by a cross validation.

Gesture Format � Accuracy Worst Recalls (Class)

12 × 12 99.00% (± 2.00%) 98.33% (connection, delete)

16 × 16 96.33% (± 4.99%) 91.67% (connection), 93.33% (delete)

24 × 24 93.00% (± 4.40%) 80.00% (connection), 95.00% (source)

32 × 32 94.67% (± 5.31%) 86.67% (delete), 98.33% (source)

Table 1: Linear SVM classifier on varying feature extractions.

Training a Gesture Classifier

With the training data available, we investigated different classifiers and pa-rameters for the gesture prediction. As some of the classifiers are binary clas-sifiers, they have been wrapped in a Polynominal by Binominal Classificationoperators with the default 1-against-all strategy. In this evaluation, we addedthe additional context features to the data set, i.e. we performed the train-ing/testing an all available features. The following Table 2 enlists all theclassifiers that have been tested on the data set using a 10-fold cross valida-tion. As feature format for the per-pixel attributes the 12 × 12 format hasbeen used. The per-pixel features for the shown results are binary features(encoded as numerical attributes 1.0 or 0.0) as these resulted in the highestaccuracy values.

Classifier � Accuracy (Variance) Worst Recall (Class)

NeuralNet, learn rate 0.3, 100 cycles 99.67% (± 1.00%) 98.33% (operator)

Linear SVM, C=1000.0 99.67% (± 1.25%) 98.33% (delete)

Perceptron, learn rate 0.05 98.00% (± 2.67%) 93.33% (source)

Naive Bayes 93.67% (± 4.58%) 90.00% (source)

Naive Bayes (Kernel) 92.24% (± 5.83%) 86.44% (source)

Table 2: Different classifiers tested for gesture prediction.

The evaluation results of Table 2 summarize the overall classification rate.With the multi-class classification problem at hand we are as well interestedin the exact prediction precisison for each of the classes. Figure 6 shows theconfusion matrix for the Neural Network classifier. As can be seen, all classesare predicted with high accuracy. The results show that a simple classifier suchas the Perceptron or the Naive Bayes algorithm achieve reasonable predictionrates for the gesture classification. All the experiments were conducted withthe builtin functionality of RapidMiner. We expect the prediction rate to beincreased with additional training examples and by excessive optimization ofthe parameters of the learning schemes at hand.

Figure 6: Results of the gesture prediction by a Neural Network.

4 The RapidMiner Artist

In the previous sections we introduced a way to derive gestures from interactivedevices and classification of these gestures to symbols that can build up aRapidMiner process. In this section we want to bring the classification togood use and introduce the RapidMiner Artist architecture.

The RapidMiner Artist is a way for interacting with RapidMiner basedon the gestures that have been outlined in Section 2.2. It consists of twocomponents:

1. the RapidMiner Arts plugin

2. the RapidMiner Artist application.

The RapidMiner Arts plugin is a small extension for RapidMiner which addsa remote service endpoint to the RapidMiner system. This allows for theRapidMiner Artist application to automatically discover RapidMiner instancesin the local network and use some of the services provided by the RapidMinerArts plugin. The discovery and service provision is implements using the[2, 1] framework, which includes a programmatic service level abstraction andautomatic multicast discover.

The RapidMiner Artist is a Java application that provides an alternativeprocess editor for RapidMiner. It is decoupled from the RapidMiner itself andcommunicates with the RapidMiner Arts plugin. In addition to the Rapid-Miner Artist, the plugin includes a web server that provides an interactivegesture trainer. This trainer can be used to add new labeled gestures to thedata set and by this provides a way to interactively add new gestures to Rapid-Miner. Figure 7 shows the architecture of the interplay of the two components.

Figure 7: RapidMiner Artist and the RapidMiner Arts plugin.

4.1 Painting Processes with the RapidMiner Artist

The RapidMiner Artist tool provides a sketch pad area for drawing RapidMinerprocesses using the gestures defined in Section 2.2. The sketch pad recordsgestures that are drawn by the user and maps these gestures to actions, such as

the addition of a new operator, the connection of two operators or the removalof an operator from the process. The gestures are classified into one of thepre-defined actions that been described in the previous section. This gestureclassification is provided by an SVM model within RapidMiner. For this, thegestures are transformed into a feature vector that is being sent to the remoteRapidMiner Arts plugin. This plugin then returns a class label for the gesturebased on a previously trained model and the appropriate action (e.g. additionof an operator) can be performed by the RapidMiner Artist. Any modificationof the process by means of a gesture is immediately performed within theconnected RapidMiner process via the exported remote editor service. As aresult, the modified process can directly be displayed on the sketch pad aswell. This provides an interactive editor for RapidMiner processes using thesketch pad application.

4.2 Training Gestures within RapidMiner

A base classifier for gesture prediction is included within the RapidMiner Artsplugin. In addition, the plugin includes operators for accessing the currentlyknown gestures and performing the feature extraction in the various flavoursoutlined in Section 2.2. To add new gestures, the RapidMiner Arts pluginprovides a gesture trainer that allows for extending the gesture classfication bytraining new custom gestures. In addition, the base classifier can be exchangedby a new RapidMiner prediction model. This allows for users to deploy theirown classifier, train custom gestures or adapt the classifier to their custom styleof painting processes. All that is required for the customization is reading thegestures, extracting the features and training a new classifier. For each ofthese steps the RapidMiner Arts plugin include all the required operators. Asa result, the gesture recognition can can solely be extended by re-running thetraining process using a RapidMiner process.

5 Conclusions and Future Work

In this work we presented a new approach to create RapidMiner processesby means of drawing sketches using interactive surfaces such as tablets orinteractive white boards. This allows for integrating RapidMiner usage directlyin smart devices like tablets or smartphones. The RapidMiner Artist softwareprovides a generic implementation of the interaction system that can easily beintegrated into an app for the android system. Future work will be providingan android app that allows designing RapidMiner processes with the gesturesystem proposed in this paper.

Our solution is based on the use of machine learning for classifying userinteractions and triggering editor actions based on these classifications. The

classification of gestures is itself performed using a RapidMiner process, whichallows for adapting the gesture detection directly to a user by re-training themodel using any of the learners included in RapidMiner itself.

Future work will explore further extensions of the user interaction suchas recommendations of operators (e.g. by additional context attributes – Atwhich position/Next to which operators has a gesture been drawn?) or theuse of handwriting recognition from gesture recordings (as opposed to tradi-tional handwriting detection from scanned images). Finally, building on theinteraction of the user and its impact on the training dataset, the use of activelearning methods to further adopt the interaction to a specific user will beof interest for extending the gesture trainer that comes with the RapidMinerArts plugin.

References

[1] Christian Bockermann. The streams framework (website). Website, 2013.http://www.jwall.org/streams/.

[2] Christian Bockermann and Hendrik Blom. The streams framework. Tech-nical Report 5, TU Dortmund University, 12 2012.

[3] Johnny Lee et. al. The WiiMote Project. http://www.wiimoteproject.com/.

[4] Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, PeterReutemann, and Ian H. Witten. The weka data mining software: anupdate. SIGKDD Explor. Newsl., 11(1):10–18, November 2009.

[5] Philipp Kranen, Hardy Kremer, Timm Jansen, Thomas Seidl, AlbertBifet, Geoff Holmes, Bernhard Pfahringer, and Jesse Read. Stream datamining using the moa framework. In Sang goo Lee, Zhiyong Peng, Xi-aofang Zhou, Yang-Sae Moon, Rainer Unland, and Jaesoo Yoo, editors,DASFAA (2), volume 7239 of Lecture Notes in Computer Science, pages309–313. Springer, 2012.

[6] Yann LeCun and Corinna Cortes. The MNIST Database. Website. http://yann.lecun.com/exdb/mnist/.

[7] Johnny Lee. Website. http://johnnylee.net/projects/wii/.

[8] MATLAB. version 7.10.0 (R2010a). The MathWorks Inc., Natick, Mas-sachusetts, 2010.

[9] Ingo Mierswa, Michael Wurst, Ralf Klinkenberg, Martin Scholz, andTimm Euler. Yale: Rapid prototyping for complex data mining tasks. In

Lyle Ungar, Mark Craven, Dimitrios Gunopulos, and Tina Eliassi-Rad,editors, KDD ’06: Proceedings of the 12th ACM SIGKDD internationalconference on Knowledge discovery and data mining, pages 935–940, NewYork, NY, USA, August 2006. ACM.

[10] R Core Team. R: A Language and Environment for Statistical Computing.R Foundation for Statistical Computing, Vienna, Austria, 2013.