1

Extended Abstract of Thesis:

“Workflow Extraction from UML Activity

Diagrams”

Thesis Summary – unaltered

http://vivliothmmy.ee.auth.gr/2262/

In the context of Software Engineering every system can have two aspects, static and

dynamic. During the investigation of the dynamic part of a system every engineer seeks to

automate the extraction of Workflow in a both human and machine processable form, as the

Workflow constitutes essentially all the available information of a system in the light of its

dynamic aspect.

One problem often faced by Software Engineers throughout the development of a new

software project is the fact that they can not easily find (online) the workflow for similar types of

systems. All they can find is .jpg images of UML Activity Diagrams by corresponding software

projects, which graphically represent system’s workflow.

As a part of this diploma thesis a potential solution to the above (stated) problem is

developed, a program named UADxTractor. This software – based tool receives images of

Activity Diagrams as input and subjects them to various levels of processing. First level aims to

enable the identification of the main entities of each diagram and the relationships between

them. The second processing level involves the detection and storage of text included in each

diagram as well as the detection of workflow's direction between its entities. In final level, the

proposed system, having already obtained all the available information which is included

(graphically) in the input image, stores it in a semantically aware structure (Ontology), named

Workflow_RDF. To verify the proper operation of UADxTractor several experiments were

performed, the results of which are presented along with the necessary conclusions at the end

of this paper.

Skoumpakis Spyridon,

Thessaloniki, August 2013

Problem Analysis

The fact of the software engineers’ inability in finding the workflow of similar systems in any form beyond images was the main motive of this dissertation. The

2

solution proposed in this work leads hopefully towards the establishment of Semantic Web.

In the past, the exchange of information between two people (software engineers or not) regarding the Workflow of their respective projects was being made, if not hand to hand, with exclusive contact of two parts.

At present, progress was made and any interested party can search the internet for information concerning the Workflow of similar projects but all he can find are images of UML Activity Diagrams.

The ultimate goal of the thesis is the creation of a practical engineering tool (not only) for better deconstruction and understanding of the dynamic part of each system but also for facilitating the sharing of information acquired.

The Unified Modeling Language (UML) is a general-purpose modeling language in the field of software engineering, which is designed to provide a standard way to visualize the design of a system. Activity Diagrams are a specific category of UML Diagrams and represent the Dynamic view of a system model. This Dynamic view emphasizes the dynamic behavior of the system by showing collaborations among objects and changes to the internal states of objects. Activity Diagrams are graphical representations of Workflows of stepwise activities and actions. Unsurprisingly, we focus on the study of this particular category of UML diagrams.

Figure 1 – Definition of Problem and Goal

Figure 2 – Chain of causality

3

Approach

It is more than clear (up until now) that any proposed solution will have to deal with Image Processing. The only data that every engineer has concerning the above stated problem are images (.jpg is the most common format). Consequently, those images will be inputs to our tool.

After analyzing the problem and finding the input the very next subject that we have to tackle is the Development Environment. In order to decide which programming tools and languages to use we need to clarify our field of interest. The field that includes methods for acquiring, processing analyzing and understanding images, in general, high-dimensional data from the real world in order to produce numerical or symbolic information is Computer Vision.

As soon as the acknowledgment of the field is completed we can proceed in choosing the programming tools. After an in-depth search of the existing alternatives we chose the library OpenCV and its interface in Java. The two main reasons which justify our choice are i) the unlimited capabilities of the library in question combined with the platform independency of java and the ii) compatibility of the API with an existing java-tool for use case diagrams. Code was developed in IDE Eclipse, an author’s personal preference.

In order to verify the proper function of our tool we had to realize instances of Activity Diagrams. Some of them were common and others not so common and could not be found online. We used two known modeling tools (StarUML and ArgoUML) to draw these instances and extract them in .jpg format. Therefore some input images came from the Internet and some from the two modeling tools.

The main objective of our software-based tool is the decomposition of Activity

Diagrams in a form that can be processed by both humans and machines. The steps of the decomposition are presented in detail later on.

Another main concern of our proposed solution is the attainment of easy sharing and reuse of all the information acquired from the decomposition. In this context, the last important choice that we had to make in our stepwise approach was the type of storage construct. There were many alternatives but for reasoning (semantically awareness) and compatibility purposes Ontology prevailed. The platform that was used in order to create, edit and view Ontologies was Protégé OWL and the API responsible for the transfer of the decomposed data to the storage construct was Jena.

A general scheme of the solution along with the exact version of each used program

is depicted in Figure 3.

4

Architecture & Outline of System

This section presents a brief outline of the system step by step. Our software – based tool was named UADxTractor from the sentence Uml Activity Diagram eXtractor which reflects its function. System receives images as input and subjects them to various levels of processing. These levels are shown in Figure 4.

Figure 3 – Tools and Languages used

Figure 4 – Levels of Processing

5

1. Image pre-processing: This level is placed at the point in time in which the user has already located the

input image and also precedes the main code. The functions of OpenCV Library (via its Java Interface known as JavaCV) which are used in our project insert some restrictions concerning the format of the input. The ultimate goal of our tool is the decomposition of any diagram. To accomplish that we use mainly OpenCV shape detection functions. These functions have some strict specifications about the images that can process. For example it is well known that most of the images which can found online are colored (RGB) with three input channels. One of the most common specifications of OpenCV is that the image has to be in greyscale and single – channel. The role of this level is to transform the input image in the proper format for the OpenCV functions of next levels. The procedure of pre-processing is depicted in Figure 5.

2. Shape & Entities Detection: In UML each shape of each diagram means something. Consequently, after the

preparation of the input image the task we have to do next is to detect every meaningful object depicted in the image. OpenCV has some ready-to-use functions for some of the known (geometric) shapes but there are also many objects that cannot be detected from the simple use of these functions. As a solution to the above stated problem we combined these functions with some customized algorithms (created ad-hoc). The visualization of the (arithmetic) outcome of each detection function is performed using the drawing functions of the Library. The basic entities in the context of Activity Diagrams are:

Figure 5 – Level of pre-processing explained

6

Initial Node (& Final Node)

After numerous experiments we concluded that the Contour and Circle Detection functions of OpenCV fail to systematic detect this entity so we performed the Template Matching Algorithm. Template Matching is a ready-to-use method of the library which searches the input image using as reference template images from a user-defined pool. Template images are usually small icons depicting entities. So if the algorithm finds in the input image a similar (nearly identical) entity as the one that is depicted in the template image returns its coordinates (and a colored rectangle is drawn around it).

Line/Transition

The largest and perhaps most important part of Entities Detection Level is Line Detection. Its importance rests in its complexity but also in the fact that practically all other entities of an Activity Diagram are related with lines and through them the essence of each diagram is depicted vividly (i.e. workflow).

In terms of programming for the detection of all lines in an Activity Diagram we used the cvHoughLines2() algorithm. This ready-to-use function of OpenCV is based on the Progressive Probabilistic Hough Transform Methodology. Usage example is shown below.

LinesSeq = cvHoughLines2(img_canny, store, CV_HOUGH_PROBABILISTIC, 1, Math.PI/360, 40, 40, 7); //(1)image: The input 8-bit single-channel binary image. In case of probabilistic method the image is modified by the function //(2)lineStorage: The storage for the lines detected //(3)method: The Hough transform variant //(4)rho: Distance resolution in pixel-related units //(5)theta – Angle resolution of the accumulator in radians //(6)threshold – Accumulator threshold parameter. Only those lines are returned that get enough votes (>threshold). //(7)param1: the minimum line length, //(8)param2: it is the maximum gap between line segments lying on the same line to treat them as a single line segment (that is, to join them)

However many times we change the arguments of the aforementioned method the

results will not be perfect. On the contrary, many imperfections appeared during the testing period such as the inability to detect some lines, the double detection of other lines, the fact that some oblique lines were detected as two or more different, the incorrect detection of lines out of parts of other entities and many other problems. In order to address all these problems we created some customized checks using mostly arithmetic boundaries and ifs.

7

Action

For the detection of Actions, we used cvFindContours() method combined with the cvBoundingRect() function. The first finds all the contours of the input image within some specified boundaries and the latter bounds the detected contour creating a rectangle around it. We then used the cvRectangle() method to draw and visualize the result. Corrective customized checks are needed again mostly because many objects of the image can be perceived as contours by the primary algorithm. CvMemStorage mem = CvMemStorage.create(); CvSeq contours = new CvSeq(); cvFindContours(img_canny, mem, contours, Loader.sizeof(CvContour.class) , CV_RETR_CCOMP, CV_CHAIN_APPROX_SIMPLE, cvPoint(0,0)); CvRect boundbox; boundbox = cvBoundingRect(contours, 0); cvRectangle( out_image , cvPoint( boundbox.x(), boundbox.y() ), cvPoint( boundbox.x() + boundbox.width(), boundbox.y() + boundbox.height()), CvScalar.MAGENTA, 1, 0, 0 );

Diamond (Decision & Merge Node)

The same methodology is applied here too except the part of the visualization. In order to draw the diamond we used four cvLines() functions because OpenCV lacks of a diamond drawing method. Due to the fact that Decision and Merge nodes are graphically identical we detect them as one entity but subsequently we make an explicit conceptual separation between them.

Synchronization Box (Fork & Join Node)

We constructed two equal solutions for the detection of Synchronizations Boxes. The first one uses the Multiple Template Matching method which we have already described and the second one uses the algorithm for detecting lines with altered arguments taking advantage of the graphical similarity between the two entities (a SynchBox is a very small rectangle, four lines).

A successful detection example is shown in Figure 7. The pink lines are the

visualization of the customized check which tackles the detection problem of broken lines. The two big lines due to the fact that they have a break point are detected as two or more separate lines from the cvHoughLines2() method and we have to merge these lines into one.

8

Figure 6 – Input Image

Figure 7 – Exported Image

9

3. Associations/Relationships of Entities Recognition:

Having all entities detected successfully we move to the next level of processing which is the recognition of all the relationships between them. We examine the associations between two entities at a time. According to the rules of UML and the assumptions that we have made we have eight cases:

1) Initial Node - Action 2) Final Node - Action 3) Final Node - Diamond 4) Final Node - Synchronization 5) Diamond - Synchronization 6) Action - Synchronization 7) Action - Diamond 8) Action – Action

Any relationship consists of 3 entities essentially if we take into account the line

joining the two basic. The algorithm that we have created it is quite similar to the customized checks mentioned before. We search at a rectangular area around each entity for line ends. If a line end is detected next to an entity we then search similarly around all the other entities for the second end of the line in question. Once the second line end is detected we have identified a relationship which we initially store in a table and later on in a vector (coordinates of two entities and line). Simultaneously, we perform some checks for the rejection of double-records. 4. Transitions/Lines Direction Detection:

Every line in any Activity Diagram is part of an association between two entities and has a specific direction towards one of them. The detection of this direction is the next processing level and it is very important because it contains information about the workflow depicted in each diagram.

This level was the most demanding task of the Thesis. Many solutions can be found but the complexity of the problem based on the number of the entities and the rules governing the relationships increases exponentially. After many tests, we created two solutions a primary and a complementary.

Our goal was to find a solution as general as possible. This solution has to obey the following rules which are derived from UML theory and some assumptions made by the author due to the limitations of the Thesis.

Initial Node is connected via only one line and only with an action/activity. Mandatory, this line is outgoing from the Initial Node.

Final Node can be connected with unlimited entities but in all cases the direction of the lines will be incoming/ingoing to the Final Node.

10

Every Action can have only one outgoing line and unlimited incoming.

Every Diamond is connected to at least three lines. o 1 ingoing, 2 outgoing = Decision Node o 2 ingoing, 1 outgoing = Merge Node

Every Synchronization Box is also connected with at least three lines. o 1 ingoing, 2 outgoing = Fork Node o 2 ingoing, 1 outgoing = Join Node

Very useful information that came out of these rules is that the first four of the

associations are unidirectional so we a priori know the direction of the arrow. The other four associations are bidirectional so we need to find the direction of the arrow every time.

Our primary solution is an algorithm that searches the neighbors of the two entities which comprise each relationship. In most cases a one-step neighbor search (search of the neighbors that are right next to each of the two entities of the association) is sufficient to extract all the information that is needed in order to find out the right direction.

During our (internet) research on candidate images that could be input to the tool in question some design patterns began to emerge.

The main pattern was about the development of the diagram. The vast majority of the depicted diagrams were following a top-down (vertical) development. Some other patterns that we came across were:

The fact that the first entity of the diagram is the Initial Node which is also located higher than any other entity. (The (0,0) point for OpenCV is the upper left corner of every image).

The last and the lowest entity of each diagram is the Final Node.

When a line is far from being horizontal (y1>>y2) then the arrow is directed to the entity which is located lower in the diagram (having greater y). We coined the term “Verticality” for these cases.

All of the above are closely related with the Top-Down development pattern. Taking

advantage of the frequency of occurrence of these patterns we simplified our algorithm.

Our complementary solution is less reliable and is applied rarely. In the context of this new algorithm in order to determine which end of each line has the direction arrow we examine a rectangular/square area around each end of each relationship for very small lines (using Line Detection Algorithm). Based on how many small lines we find next to each end we infer the direction of the arrow. Obviously, whichever end is detected to have a greater number of small lines around it is the one that has the arrow and thus defines the direction.

11

The result of the two direction detection algorithms is visualized by drawing colored dots at the right end of each line. This new image is given as output to the user. An example is shown in Figure 8 (the unidirectional transitions omitted). The black dots are from the primary algorithm and the pink from the complementary.

5. OCR Emulation:

Every Activity Diagram contains text and this text is crucial information for the complete deconstruction of the diagram. After an in-depth search on the internet for an Open Source OCR (optical character recognition) API we found only one suitable for our problem, Asprise OCR. This tool is available to the public in a free trial version. We tested this tool using many different diagrams but its performance was disappointing. The results were not worthy of showcasing.

The diagrams under study contain characters in only two places: the name of actions/activities and the guard conditions of decision nodes. This is all the text information that we want to extract from the image and insert to our program. We therefore decided to construct an alternative way of retrieving this information. The new path involves user intervention. We created two classes which undertake to guide the user to enter the required information from the keyboard. The interactivity is achieved through messages on the (Eclipse) console combined with explanatory colored images.

Figure 8 – Direction Detection Algorithms combined

12

Sorting Entities

This is a transitional intermediate level that allows the complete extraction of information depicted in the input image via classification of entities.

The largest part of the available information after its extraction from the input image consists of (entity) coordinates. Coordinates are a kind of data that cannot and does not make sense to store. Apart from the actions and the guard conditions we had to find a way to also impart to the remaining entities some characteristics in order to ultimately enable the overall storage of every Activity Diagram. Those characteristics are a way of distinguishing one entity from another.

The devised method was the serial naming of entities by combining their names with real numbers (e.g. Fork1, Fork2 etc.). In order to accomplish that in a consistent way the entities should be sorted. OpenCV detection algorithms automatically sort all entities (using their upper left point) except lines so all we had to do was to sort them “manually”.

At this level of processing we also performed the conceptual separation of Diamonds and Synchronization Boxes in Decision and Merge Nodes and in Fork and Join Nodes respectively.

6. Connection with Ontology

“Ontology is a formal, explicit and detailed definition of a domain” (general),

“Ontology is a common model for representing the basic concepts of Activity Diagrams” (specific)

For the development of the Ontology we used the tool Protégé OWL. Our

methodology is based on design rules found in the literature and the model Ontology OWL-S. The concepts ultimately selected to form the semantic model Workflow_RDF are depicted in Figure 9.

The last step before the insertion of the aggregate data in the Ontology is the grouping stage in order to achieve the complete conceptual reconstruction of the diagram and the representation of the Workflow.

We have developed an algorithm that searches all existing relationships of every diagram while detecting the direction of the arrows in order to record the flow of information.

In the context of our Thesis, the recognized relationships between all entities are eight. Four of them are unidirectional and the other four bidirectional. In addition, three of them (all the relationships of Decision Nodes) may contain binary conditions known as Guard Conditions. In short, we created eight functions for one direction of all relationships, plus another three for the other direction of the bidirectional (Action-Action can be used for both), plus another three for those with Guard Conditions. In total, we created fourteen functions separating the original information in groups of two

13

entities with a specified flow direction. These groups are called Sub-Activities as they constitute parts of the overall Activity which each Diagram represents.

For the connection of our main program with the constructed Ontology in order to transfer all the available information we used an Open Source Java Framework. This framework is called Jena API.

Implementation

Now it is time to verify the proper operation of UADxTractor through experiments. Every user of this tool will probably need to change only a limited number of variables, the name of the input image and the arguments of the detection algorithms. The values of those variables are externally defined from a text (.txt) file so the user will not need to come in contact with the main code.

Figure 9 - Hierarchical Structure of Ontology

14

We use a simple, yet representative diagram, found online, for our experiments. This diagram depicts the activity of displaying bowling score.

The pre-processing level is next and the results are shown in Figure 11.

Figure 10 - Display Bowling Score Diagram

Figure 11 – Example of pre-processing Level

15

Now the image is ready to be used from the OpenCV functions. The detection of the entities and their relationships follows. In the big diagram on the left we can see the overall results of the detection algorithms and the right four diagrams depict the detection of the relationships between the entities.

During the whole procedure explanatory messages are printed on the console of

the Eclipse. Examples of those messages are shown in Figure 13.

Figure 12 – Detection of Entities and Relationships

Figure 13 – Samples of Console Messages

16

The most complex level of processing is the direction detection. The outcome of our algorithm is presented in Figure 14.

The last step of our proposed solution is the connection of the main tool with the

Ontology which is the storage construct that we chose. There are two ways to ensure the unimpeded transfer of all the available information. The first is through console messages and the second is using Protégé. Both ways are shown below.

Method of Finding Sub - Activities has started... InitialNode->Line1->Calculate_numbers_of_knocked_out_pins Do you agree with this activity? Press y(Yes) or n(No) y So, the [Sub - Activities] of 2 entities are: --------------------------------------------- Sub-Act[0]:InitialNode->Line1->Calculate_numbers_of_knocked_out_pins Sub-Act[1]:Display_Score->Line7->FinalNode Sub-Act[2]:Decision0->Line6->Non_Strike->Display_Score Sub-Act[3]:Decision0->Line4->Strike->Show_Animation Sub-Act[4]:Calculate_the_score->Line3->Decision0 Sub-Act[5]:Show_Animation->Line5->Display_Score Sub-Act[6]:Calculate_numbers_of_knocked_out_pins->Line2->Calculate_the_score -------------------- Number of Sub - Activities = 7 THE END!

Figure 14 – Example of the Direction Detection Algorithm

17

Conclusions

Through this work I discovered some of my strengths and put myself into a test to see how I deal with adversity. The lessons I took from this process are invaluable. My supervisor Dr. Andreas Symeonidis stood by me and shared his insight when it was needed. We kept a close contact dividing my work into deliverables and discussing every issue.

One of the main goals of the current paper is to continue the effort started with the Diploma Thesis of Anastasia Mourka (Use Case Diagrams) automating the conceptual reconstruction of every UML diagram and thus prepare the ground for the future creation of a semantic (UML) search engine. Anastasia was an excellent advisor at the beginning of this effort providing valuable guidance as a former student of the same department.

This implementation was constructed based on the most common design patterns, but due to the vast number of existing alternative models was impossible to cover all eventualities. For example, the analysis of images depicting Activity Diagrams with horizontal development introduces some deficiencies. Nevertheless, the overall conclusion, in the context of analysis, is quite satisfactory, as the vertical development of these diagrams is the usual practice.

The images used in the experiments were chosen so as to cover the widest possible range of the online available diagrams. They have an average complexity and most of them came from the well-known tools StarUML and ArgoUML.

In conclusion, the main success of UADxTractor, in addition to saving time for Software Engineers, is that it provides direct access to the Workflow for similar types of Systems. This is something than can be useful for both humans (engineers) and other programs.

Figure 15 – Display Bowling Score Instance in Protégé