kuvatus project - jyväskylän...

Kuvatus ProjectPekka Iso-Ahola

Jussi PerttolaTommi Tuovinen

Application Report

PublicVersion 1.0.0

26.4.2012

University of Jyväskylä

Department of Mathematical Information Technology

Jyväskylä

Acceptor Date Signature Clarification

Project manager __.__.2012

Customer __.__.2012

Instructor __.__.2012

Kuvatus Project Application Report 1.0.0 Public

Document Info

Authors:• Pekka Iso-Ahola (PIA) [email protected] 040 842 8378• Jussi Perttola (JP) [email protected] 050 412 4420• Tommi Tuovinen (TT) [email protected] 040 751 8187

Document name: Kuvatus Project, Application ReportPage count: 36

Abstract: Kuvatus project developed a prototype of a web application for creatingand managing training sets, for starting an analysis which uses these training sets asan input and for previewing the analysis results. The document describes the back-grounds of the software as well as presents the user interface and the applicationstructure. Programming and testing practices are also presented. The realization ofthe functional requirements and other objectives and also advices for future devel-opers are described.Keywords: Application structure, future development, hyperspectral imaging,meeting the requirements, programming practices, Ruby on Rails, testing, trainingset, WWW interface.

i


Project Contact Information

Project group:Pekka Iso-Ahola [email protected] 040 8428 378Jussi Perttola [email protected] 050 412 4420Tommi Tuovinen [email protected] 040 751 8187

Customers:Paavo Nieminen [email protected] 040 576 8507Marko Peltola [email protected] 041 449 8622Ilkka Pölönen [email protected] 040 248 140Tero Tuovinen [email protected] 050 531 0791

Instructors:Tero Hänninen [email protected] 0400 240 468Jukka-Pekka Santanen [email protected] 050 550 4666

Contact information:Email lists [email protected]

[email protected]

Email archives https://korppi.jyu.fi/kotka/servlet/

list-archive/kuvatus/

https://korppi.jyu.fi/kotka/servlet/

list-archive/kuvatus-opetus/

ii


Version History

Version Date Modications Modifiers

0.0.1 6.4.2012 The report template was created. PIA0.0.2 8.4.2012 Completed initial drafts of Chapters 1 – 9. PIA0.0.3 9.4.2012 The initial draft of Summary was written and

grammar was fixed.PIA

0.1.0 11.4.2012 Grammar and some other errors were corrected. PIA0.1.1 15.4.2012 Grammar and other errors were corrected. TT0.1.2 16.4.2012 Added more development ideas and references.

Modified Section 4.4. Added Sections 4.7, 5.4and 5.5. Moved development ideas to Chapter9.

PIA

0.2.0 16.4.2012 Added realization of requirements. TT0.3.0 19.4.2012 Corrected spelling and grammar. Rearranged

some of the sections.PIA

0.4.0 23.4.2012 Corrected spelling and grammar. Added an-other mailing list. Added a reference.

PIA

0.5.0 25.4.2012 Added small corrections. PIA1.0.0 25.4.2012 Corrected version history. Added more gram-

matical corrections. Corrected references.PIA

iii


iv


Contents

1 Introduction 1

2 Terminology 2

2.1 Target Area and Application . . . . . . . . . . . . . . . . . . . . . . . . 2

2.2 Software and Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . 3

3 Background 4

4 User Interface 6

4.1 Page Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

4.2 Logging in . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

4.3 Creating and Managing Selection Groups . . . . . . . . . . . . . . . . 7

4.4 Selection Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

4.5 Miscellanous Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

4.6 Creating and Managing Training Sets . . . . . . . . . . . . . . . . . . 12

4.7 Creating and Managing Computations . . . . . . . . . . . . . . . . . . 14

5 Application Structure 15

5.1 Formats, Components and Software . . . . . . . . . . . . . . . . . . . 15

5.2 Class Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

5.3 File and Data Formats . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

5.4 YouData-YouHyper Integration . . . . . . . . . . . . . . . . . . . . . . 19

5.5 File Type Implementation and MATLAB . . . . . . . . . . . . . . . . . 20

6 Programming Practices 21

6.1 Formatting, Naming and Commenting Practices . . . . . . . . . . . . 21

6.2 Source Code Example . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

6.3 Grouping Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

6.4 Development Platform . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

7 Testing Practices and Results 25

7.1 Unit and Integration Testing Practices . . . . . . . . . . . . . . . . . . 25

7.2 Usability Testing Practices . . . . . . . . . . . . . . . . . . . . . . . . . 25

7.3 Testing Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

v


8 Realization of Objectives 288.1 Realization of Requirements . . . . . . . . . . . . . . . . . . . . . . . . 288.2 Unsatisfactory Solutions in the Implementation . . . . . . . . . . . . . 288.3 Challenges in the Implementation . . . . . . . . . . . . . . . . . . . . 298.4 Modifications During the Implementation . . . . . . . . . . . . . . . . 30

9 Guide for Future Developers 319.1 Essential Bugs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 319.2 Improvements to Existing Features . . . . . . . . . . . . . . . . . . . . 319.3 Further Development Ideas . . . . . . . . . . . . . . . . . . . . . . . . 329.4 Development Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

10 Summary 34

11 References 35

vi


1 Introduction

Researchers at Department of Mathematical Information Technology (MIT) in Uni-versity of Jyväskylä have been developing methods for analysis of hyperspectralimages. The process of the analysis starts out with the user marking and labelingthe areas of interest from one or several hyperspectral images. The set of data de-fined this way is called a training set, which is then used as a training input for aclassifier algoritm. The classifier is then used for automatic identification and clas-sification of similar areas of interest in other hyperspectral images.

Kuvatus project designed and developed a working prototype of an application forcreating and managing training sets. In addition, these training sets can be passedon to the certain third-party mathematical analysis software for processing and oncethe processing finishes, the results are loaded back into the application.

The application developed was named YouHyper to follow the naming conventionof earlier applications, YouData and YouSource. YouHyper is designed to be inte-grated into YouData, which is a data storage system developed in an earlier studentproject.

Kuvatus project wrote several documents describing the developed software andthe project. The document describes the implementation of the features describedin the requirements specification [2]. The realization of the goals and the practicesof the project are described in the project report [3]. Verso Application Report [1]was used in compiling this document.

In Chapter 2 the essential terminology used in the document is explained. Chapter 3describes backgrounds of the project. Chapter 4 presents the user interface of theapplication concentrating on the parts that were developed in the project. Chapter 5explains the inner structure of the application. Chapter 6 specifies the programmingpractices used during the project and Chapter 7 describes how testing was carriedout. In Chapter 8 it is considered how the objectives set to the application werereached. Finally, in Chapter 9, suggestions related to the further development of theapplication are presented.

1(36)


2 Terminology

The chapter explains the essential terms that appear in the document.

2.1 Target Area and Application

The terms below are specific to Kuvatus project and describe some of the key ele-ments in the developed software.

Bitmask is an array of bits describing which of the pixels in the givenarea are selected and which are not.

Class in the scope of mathematical classification is a collection ofobjects which can be unambiguously defined by propertiesthat all of the objects in the collection share. In YouHyper,each selection group represents a class.

Classifier is a component that is initialized by a process called train-ing. After being trained, a classifier can be used for classi-fication and identification of further inputs that share com-mon features with the training data.

Computation is a general term for the execution of any type of calculationprocess that is carried out outside of YouHyper. It includestraining, classification or clustering.

Hyperspectral image is an image taken by a hyperspectral camera. It is con-structed from several images, each of which represents asingle wavelength band of electromagnetic radiation.

Job is an entity describing a single computation process.

Project is a user created entity in YouData. A project consists ofusers who have the right to use common data materials andcalculation resources. A project is owned by a user or ateam.

Selection group contains a set of bitmasks. Each selection group is related toa particular hyperspectral image.

2(36)


Training set is a set of data whose classes are known and can thus beused as an input for training a classifier. In YouHyper, atraining set is an object that contains references to severalselection groups.

YouData is a file sharing system developed by Judo project.

YouHyper is the name of the application developed by Kuvatus project.

2.2 Software and Techniques

The following terms are related to the software that was developed in Kuvatusproject but they are non-specific and could apply to many other applications as well.

JavaScript is a prototype-based scripting language mostly used in dy-namic modification of content in WWW-based applications.

Metadata is information about data. It describes, among other things,the definition, structure and management of data files.

MVC architecture i.e. Model-View-Controller, is an architectural pattern usedin software engineering. The pattern isolates the applicationlogic from the input and the presentation, permitting inde-pendent development, testing and maintenance of each.

Ruby on Rails is an open source web application framework for Ruby pro-gramming language.

MATLAB is a third-party application used for mathematical process-ing of matrix format data.

3(36)


3 Background

Kuvatus project was started off from the need for an improved way of analyzingand processing of hyperspectral images in the field of machine learning. The chap-ter goes briefly through backgrounds and the initial needs that led to starting ofKuvatus project.

Hyperspectral imaging is used in the research of skin cancer, forestry and manyother fields. The research relies in the knowledge that there are certain wavelengthsin the spectrum of light in which these phenomena can be identified. For example,training sets can be created from skin areas that are known to have skin cancer andwith the analysis process it may be possible to identify skin cancer from samples ofhyperspectral images from other parts of skin.

Currently, at Department of Mathematical Information Technology in University ofJyväskylä, the researchers working at the field of machine learning have no appli-cation for creating and managing training sets. With the current tools, a researcherwill have to first extract hyperspectral images into a human-readable format, iden-tify areas of interest and label those areas by hand. After having done this to a setof images, the researcher will gather all the information together and pass the dataonto a second application, which will train a classifier. It is then used in analy-sis of further images with similar areas to the ones appearing in the training set.This analysis returns an image with several highlighted areas, a classification prob-ability estimate for each of the highlighted areas and a class to which each of theseareas belong to. This probability gives an estimate on the reliability the classificationresults are.

With the existing tools the process of forming a training set is laborious, especiallysince the researchers currently have no application for the identification and label-ing of interesting areas from a hyperspectral image. The researchers have to definescripts for looping through the pixel areas, instead of having a simple drawing ap-plication type user interface for this part of the process. Similarly, they will haveto store this data in their own workstations, instead of being able to automaticallyprovide the latest version of it to other researchers. The classifiers training withthese training sets is also done by hand by calling a function from a command lineinterface, instead of being able to select the algorithm and the target to which it isapplied from a convenient user interface.

Another problem with the current tools is that they do not support an easy direct

4(36)


way of sharing the results of their research to their colleagues. Currently the userscan only upload the results after the computation has finished, instead of being ableto track the progress of the analysis process somehow in a shared environment. Inshort, the challenge and aim of this group of researchers is to have a single applica-tion that supports all the phases in hyperspectral image analysis.

5(36)


4 User Interface

The user interface of YouHyper was developed using HTML5 elements, JavaScript,AJAX functionalities and a cascading style sheets (CSS) file. The page layout wasmade to resemble that of YouData by adding similar header and footer elements.

4.1 Page Structure

The general page structure in YouHyper consists of five elements: the header, tabbedlink bar, main content and the footer as shown in Figure 4.2. The navigation bar inthe header and the footer offer links to the views of the application. The header islocated at the top of each page and it currently only contains the YouHyper logo.

The main content presents the essential information and functionalities on the page,depending on which of the views the user is currently using. There are four differentviews: Selections, Training sets, Computations and Jobs. The tabbed navigation barindicates the page on which the user is currently at, and allows for easy access tothe other views (see Figure 4.2).

4.2 Logging in

Authentication and authorization in YouHyper is performed in YouData as You-Hyper is a part of YouData. YouData uses an LDAP authentication interface pro-vided by Korppi, which is one of the main information systems used in Universityof Jyväskylä. As seen in Figure 4.1, logging into YouData is performed directly byKorppi username and password. In addition, the user groups from Korppi can bedirectly copied over to YouData.

Once the user is in YouData, he or she has to join an existing project or the user hasto create a new one. After having added hyperspectral images to the project and theimages have been identified and extracted, the user can move on to YouHyper viathe link in the right hand side navigation menu.

6(36)


Figure 4.1: YouData login screen

4.3 Creating and Managing Selection Groups

Selection view in Figure 4.2 is the first view of YouHyper. It allows the users tomanage and create selection groups related to hyperspectral images. The view isautomatically populated by all the hyperspectral images in the YouData project fromwhich the user came.

The user can select a hyperspectral image from the list Images, after which the list Se-lections will be populated by the selection groups specified earlier to that particularimage. The user can add a new selection group by clicking on the Add new buttonand defining a name for the selection group. The name can be changed later on bydouble-clicking the name field on the selection group image.

Once the user has selected the hyperspectral image he or she wants to work on andhas created a selection group, he or she can use Wavelength selector for selecting thedesired wavelength. Once the user releases the mouse button, the correspondingimage of the wavelength will be downloaded and shown on the view below theslider. It should be noted, that regardless of the wavelength displayed, the selectionswill appear and apply to all the wavelengths of the image.

Section 4.4 presents the selection tools that can be used for creating selections. Sec-

7(36)


tion 4.5 deals with the other functional tools available in the Selections view.

Figure 4.2: The selection view

4.4 Selection Tools

There are four different selection tools available in Selection view for defining areasfor a selection group. The rectangle, ellipse, point-to-point polygon and magic wandtools can be activated from the toolbar on the left side of the screen (see Figure 4.2).Examples of each selection outlook are shown in Figure 4.3. The rectangle and el-lipse tools are tools for drawing primitive shapes.

The point-to-point polygon tool is a tool with which the user can draw polygons. Itworks in such a way that the first click draws the starting point with a circle on thepicture, and the consequent clicks each draw a line from the previous point to thenext one. The user can finish the area by clicking inside the circle drawn by the firstclick, after which a filled polygon is drawn on the picture.

8(36)


Figure 4.3: Examples of selections in an image made by the selection tools.

The magic wand tool can be used to select areas of pixels with the same color fromthe picture. It uses a flood fill algorithm with a threshold value for the pixel colorsimilarity. This threshold value is defined by a value from 0 to 100, and can beadjusted by the slider below the selection tool buttons. It should be noted that thisslider is only visible when the magic wand tool has been selected.

The specified selection area is sent immidiately to the server and stored into thedatabase. When the user comes back to a selection that was made earlier, the selec-tions that are downloaded from the server are colored blue and the selections of thecurrent session are colored green, as seen on Figure 4.4.

9(36)


Figure 4.4: The newly made selections (green, light) and the ones loaded from theserver (blue, dark).

4.5 Miscellanous Tools

In addition to the tools described in the previous section, there are five more func-tional buttons on the left toolbar of Selection view (see Figure 4.2). These are theerase mode, view all, zoom in, zoom out and undo. On the left toolbar, there is alsothe undo target state display.

When the erase tool is toggled, the selected areas that are made with any selectiontool (see Section 4.4) are substracted from the current selections instead of beingadded. This allows the user to easily, for instance, select the entire image and sub-stract the non-interesting parts. When activated, the areas being drawn appear red(as seen in Figure 4.5), until the mouse button is released.

When the view all button is toggled, all of the selection groups related to the currentimage are shown on the canvas at the same time. It can be useful if the user wants toperceive all the selection groups at the same time while working on a new selectiongroup. If the user only wants to peek one other selection group while working onanother, he can hover his mouse over the list element of the selection group and itwill be displayed on the canvas. Both the functionalities can be seen in Figure 4.6.It should be noted that even if the show all button is active, all selection operations

10(36)


Figure 4.5: The selected and the erased areas in the erase mode.

done by the tools still only affect the active selection group.

Figure 4.6: Displaying the current selection group, hovering over another with themouse or showing all selection groups.

The Zoom in and zoom out buttons effectively scale the image up or down. Eachclick either increases or decreases the level of the zooming by a factor of two. Oncezoomed in, horizontal and vertical bars will appear next to the image allowing theview to be moved. The difference between the normal view and the twice zoomedview can be seen in Figure 4.7.

11(36)


Figure 4.7: The no zoom and four-times zoomed (twice zoomed) image.

The undo button undoes the previous selection operation done to the currently ac-tive selection group. It should be noted that only the selections made in the presentsession (shown green) can be undone. There is also the undo target state display be-low the undo button, which shows the state to which the selection group will returnif undo is pressed. It should be noted that undo operations can not be undone.

When the user hovers the cursor over the image display, the coordinates of the cur-rent cursor position will be shown next to the cursor (see Figure 4.7. The coordinateorigin is located at the top left corner of the image.

4.6 Creating and Managing Training Sets

Training sets view displays the list of the generated training sets as seen in Figure 4.8,from which the user can either select an existing one or create a new one.

By selecting an existing training set, the hyperspectral images belonging to thattraining set will be loaded on the list Training set, and every other image in the You-Data project will be displayed in the list All images, as seen in Figure 4.9.

When the user clicks on an image that is listed in the Training set, all the se-lection groups belonging to that image will be listed on the double list structure,as presented in Figure 4.10. All the selection groups which are not selected for thetraining set are listed on the left hand side list, while the selected ones are listed onthe right hand side list.

User can add selection groups by clicking on the icon + on the corresponding selec-

12(36)


Figure 4.8: The view for selecting a training set.

Figure 4.9: The training set view when a training set has been selected.

tion groups, by dragging them to the right side or by clicking on Add all button. Theuser can also remove selection groups by clicking on the corresponding icon - or byclicking on Remove all button. If the user wishes to arrange the selected selectiongroups in some particular order, he can simply drag the list items to their desiredposition on the list. There is also a search field on the left list, with which the usercan filter the selections to be shown.

All the changes to the training set are updated to the server instantly to avoid loss

13(36)


Figure 4.10: The lists used for picking selection groups for a training set.

of information and for convenience. Once the user has finished creating a trainingset, he can move on to Computations tab for creation of computational jobs which isdescribed in the following section.

4.7 Creating and Managing Computations

Computations view was not implemented due to the limited work hours of the projectgroup. It should include a list of training sets from which the user can select one tobe used in computation. After having selected a training set, there would be a listfrom which the user can select an algorithm. The algorithm can have a list of pa-rameters, which may have default values, and the parameter list can also include ahyperspectral image to be analyzed.

When the user types in the parameters, he can then finalize defining the computa-tion by clicking Create button. This would create a job that can be viewed in Jobsview.

Jobs view should have two lists; one for the jobs that have been completed andanother one for the jobs that are being processed. There should be a progress barnext to each job as well as Cancel and Start buttons for aborting and starting of jobsthat have not been processed.

14(36)


5 Application Structure

The chapter describes the different components in YouHyper and their relations toeach other. YouHyper is mainly a server application due to its web service nature.The application uses Ruby on Rails libraries, MongoDB database in cooperationwith Mongoid, NetCDF-4 and HDF5 for computation-related data storage as wellas narray and ruby-netcdf Ruby libraries for reading and writing of the data.YouHyper interfaces with MATLAB to complete the heavy computational tasks.

The application itself is integrated to YouData, which is a data storage system. You-Hyper uses YouData for data storage and user authentication.

5.1 Formats, Components and Software

HDF5, or Hierarchical Data Format, is a data format that is suited for storing hier-archial numeridal data. It is used as an underlying storage format by NetCDF-4.

MATLAB is a numerical computing environment and a programming language de-veloped by MathWorks. It is used for many of the computational tasks done byYouHyper.

MongoDB is a document-oriented NoSQL database system, using BSON as it’s doc-ument storage format.

Mongoid is an Object-Document-Mapper for MongoDB written in Ruby. It allowsdeserialization and serialization of Ruby objects into the BSON document format.

NetCDF, or Network Common Data Form, is a set of software libraries and self-describing, machine-independent data formats that support the creation, access,and sharing of array-oriented scientific data. NetCDF version 4, or NetCDF-4, isused in YouHyper application.

Ruby libraries are used to perform some specific functionalities that Ruby on Railsdoesn’t provide. Particularly, ruby-netcdf and narray libraries were used forreading and writing NetCDF files.

Ruby on Rails libraries provide the basic functionalities for many classes in You-Hyper by inheritance. For instance, the controller classes are inherited from Ac-

15(36)


tionController class. Similarly, the model classes are inherited from Mongoid-

::Document.

YouData is a file sharing application developed by Judo project. It is being used asthe platform for YouHyper. It offers data storage, user authentication and autho-rization as well as user groups and project management.

5.2 Class Structure

A simplified class structure of YouHyper is presented in Figure 5.1 displaying therelations between the essential YouHyper model classes as well as the external in-terfaces. For a more thorough documentation on the server side implementation,please see YouHyper class documentation [5]. YouHyperProject model and themost essential models related to it are shown in the figure. YouHyperProject andRelatedFileSet models are discussed in Section 5.4.

YouHyperYouData

Selection

Project

Datafile

LayeredImage

YouHyperProject11

N

1

N

1

N

N

TrainingSetN1

N

Computation

1

N

External computational applications

Matlab

RelatedFileSetN 1

1

0..1

0..1

1N

Figure 5.1: A simplified class structure of YouHyper.

The Computations model, was not implemented due to the lack of time, but itwas drawn on the figure none the less. This was done to clarify it’s functionality forfurther developers.

The class structure in YouHyper is dictated by Ruby on Rails. Rails applications usethe Model-View-Controller (MVC) architecture, which means that all the classes

16(36)


are divided into application logic (model), input handling (controller) and datadisplay classes (view).

LayeredImagemodel represents one hyperspectral image and as seen in Figure 5.1,it’s instance can have any amount of Selection objects related to it. Selectionmodel represents a single selection group. In the figure, the relationship betweenTraining Set and Selection is described as a N-to-N relationship. However,in the Ruby code it is defined in such a way that Selection objects themselves donot know to which TrainingSet they belong to.

TrainingSet model also contains the method named export_netcdf, whichcan not be called from any of the currently implemented views. The method createsa NetCDF representation of the training set in question. The training set NetCDFformat is specified in Section 5.3.

In addition to the classes listed in Figure 5.1, there is also Pool class that works as athread pool. It is currently being used with a single thread to reduce the amount ofMATLAB licences used by the application and to avoid the usage of blocking oper-ations when extracting data from uploaded hyperspectral images. Another helperclass called BitMask was implemented to help store the string formatted bitmasksin the database. It offers transparent compression and decompression of the bit-masks related to Selection objects.

On the client side, the central JavaScript objects include LayeredImage, Selec-tion and Tool. Each of these forms the client side UI elements, functionalitiesand data storage objects required for drawing the dynamic parts of the user in-terface. They are defined in their corresponding JavaScript files located at app-/assets/javascripts/. In addition to these, there are several other JavaScriptfiles for the functionalities of the static parts of the page.

5.3 File and Data Formats

Hyperspectral images can be described in many different formats. YouHyper cur-rently supports the hyperspectral images stored in binary format txt file and theENVI type format. The latter uses two files: an hdr file contains metadata about theimage and a dat file holds the image payload.

Hyperspectral images are opened by the dedicated MATLAB scripts located in thedirectory app/assets/matlab/image_openers. The MATLAB scripts generate

17(36)


PNG images from each of the wavelengths to be viewed in the user interface aswell as generate a NetCDF image cube representation of the particular hyperspectralimage. The image cube format is described in Table 5.1. It is a three dimensionalarray of size n×m×k, where the resolution of the image is n×m and the wavelengthcount is k.

x1 · · · xn

y1 I(1, 1) · · · I(n, 1)...

... . . . ...ym I(1,m) · · · I(n,m)

I(n,m)

λ(n,m)1

...λ(n,m)k

Table 5.1: The image cube array format with wavelength range λ1, . . . , λk.

The image cube representation is used when a training set is being generated, asRuby can read the NetCDF files directly by using ruby-netcdf library. Further-more, using a three-dimensional array for storage allows for quick and easy accessto the wavelength vector I(x, y) at an arbitrary point (x, y).

Each training set is presented in the array format described in Table 5.2. The usageof a regular, two-dimensional array format for describing a training set was seenas a smart choice as it only holds relevant information that will have to be loopedthrough as a whole.

Light intensityImage label Class label x y λ1 · · · λn

1 2 43 234 12.33 · · · 130.2 Example data, first row... . . . . . . . . . . . . . . . ...4 5 22 93 17.45 · · · 4.52 Example data, last row

Table 5.2: A single training set in the array format.

The selection groups, or instances of Selection class, can be thought of as classesin the scope of mathematical classification. In the following paragraphs, the wordclass will be used to clarify this relationship.

There are four static columns in the array. These are image labe, class label, x and y.The x and y values determine the location of the pixel in the image (the origin beingat the top left corner). The image label is a number assigned for an image and theclass label is a number assigned for a class.

18(36)


In addition, there are n columns for light intensities, where n is the amount ofwavelengths in the particular hyperspectral image. This adds up to a total of n + 4

columns in the array. The light intensities are presented as double precision floatingpoint values. The amount of the rows is equal to the amount of the pixels in eachof the selection groups belonging to the training set.

As an example, a training set consisting of 10 images with 200 wavelength bands,each image holding 5 classes with each of the classes having 100000 pixels. Thearray would be sized as (4 + 200)× (10 · 5 · 100000) = 204× 5000000.

The image and class labels are assigned when the training set is created. Theyonly describe the relationship between the number and the image filename or theselection group name. These relationships are defined as strings stored in the sameNetCDF data container with the variable names classes and images. Both hold astring that contains a set of label–name pairs. These pairs are formatted as n:label_n|and are concatenated together. For example, the strings describing the labels of Mimages with N classes are formatted as follows:

classes = 1:Class_1_name|2:Class_2_name|. . .|N:Class_N_name

images = 1:Image_1_name|2:Image_2_name|. . .|M:Image_M_name

5.4 YouData-YouHyper Integration

As noted in Figure 5.1, there is RelatedFileSet model in YouHyper that has adirect relation to YouData’s Datafile model. RelatedFileSet is a model thatpoints to any number of files related to a hyperspectral image or a training set, in-cluding the PNG format images to be displayed in the user interface.

As seen in Figure 5.1, YouHyperProject acts as the integration interface towardsthe Project class defined in YouData. User should be able to go to YouHyper-

Project from a YouData’s Project and analyze the uploaded images there.

Marko Peltola implemented many of the changes needed for integration of You-Hyper to YouData. This included a Gem definition file, changes to some of thesource code paths and changes to the paths used for viewing images in the WWWinterface.

19(36)


5.5 File Type Implementation and MATLAB

Currently the MATLAB files to be run when opening an image are being storedin app/assets/matlab/. The implementation for adding a new supported filetype requires an image opener MATLAB script. These scripts should be stored inapp/assets/matlab/image_openers/.

The MATLAB script should take as arguments the files related to the hyperspectralimage as well as the path where the resulting PNG files should be saved. The MAT-LAB script should print to stdout a string that is prepended by ImageList: andappended by path/file| where file is the file generated by the script and | isused as a separator between each file path. As the file naming convention, the filesdescribing the PNG format wavelength images should be named image_n.png

where n is the index of the wavelength.

The script should also generate the NetCDF file datafile.nc that follows the for-matting described in Table 5.1. This array should be stored in the NetCDF file as avariable with the name data. For example, the following String could be returnedfrom the script, if the extracted image has 40 wavelengths.

/path/image_1.png|. . .|/path/image_40.png|/path/datafile.nc|

Note that the actual path was left out to fit the string in the page. In addition tothis, changes need to be made to some of the source code in Layered Image Rubymodel. Specifically, changes need to be made to the methods from_files anddetect_filetype. See the class documentation [5] for more accurate implemen-tation examples.

20(36)


6 Programming Practices

The programming in Kuvatus project begun almost immidiately after the WWWuser interface was specified. Early prototypes were made by JavaScript and thoseprototypes served as an early draft of how the application would function.

From the beginning of the project the goal was to develop a code that follows Rubyguidelines and is as easily readable as possible. Over the course of the project, thesource code was reviewed twice by the technical instructor Tero Hänninen. He gavefeedback on the questionable parts of the code, which were then refactored or rewrit-ten according to his comments.

Many problems faced in the development were overcome with the use of WWW-sources, such as the Ruby API documentation [10] and Ruby on Rails Guides [11].

6.1 Formatting, Naming and Commenting Practices

Readability was the main guiding principle while formatting, naming and com-menting the source code. Kuvatus group followed standard formatting used inRuby [8] and JavaScript programming [7]. Four whitespaces was used as the in-dentation between code blocks and longer lines of code were split to several lines toincrease readability at lower screen resolutions.

The naming of the methods and the variables was kept as self-explanatory as pos-sible. Due to the complex nature of the application, many methods and commandsstill needed to be commented to clarify their function. In JavaScript, the multipartnames of the methods and variables were formatted by the lower camel case. InRuby source codes, the snake case was used instead. All source code, including thecomments as well as the variable and method names, was written in english.

6.2 Source Code Example

The following example of source code is taken from the file related_file_set.rbillustrating formatting, naming and commenting practices used in Kuvatus project.

# This content is released under the MIT License.

21(36)


#

# Please see the full license in the attached LICENSE-file

# or at http://www.opensource.org/licenses/mit-license.php

#

# Copyright (c) 2012 Pekka Iso-Ahola

# Copyright (c) 2012 Jussi Perttola

# Copyright (c) 2012 Tommi Tuovinen

# A wrapper class that points to a number of Datafiles.

class RelatedFileSet

include Mongoid::Document

belongs_to :layered_image

belongs_to :training_set

references_many :datafiles

# Adds a file.

def add_file(file)

self.datafiles << file

self.save!

end

# Adds an array of files.

def add_files(files)

files.each do |file|

self.datafiles << file

end

self.save!

end

# Returns a file with the given name.

def get_file(name)

datafile = self.datafiles.select{|item|

item.name == name}

if datafile.length != 1

puts ’Error! Too many or no entries found.’

return nil

22(36)


end

return datafile.pop

end

# Returns files matching the given pattern.

def search_files(pattern)

self.datafiles.select {|file| file.name

=~ Regexp.new(pattern)}.uniq

end

end

6.3 Grouping Practices

The source code was divided into different files according to the MVC architectureand Ruby on Rails practices. For example, the amount of embedded Ruby code waskept minimal in the view files by processing the information as much as possible inthe corresponding controllers. Likewise, the controllers were kept clean by placingmost of the data processing in the model files. Also, the appearance of the webpages was developed with a separate style sheet (CSS) file so that the HTML codewould stay simple.

Due to the amount of JavaScript code required for the implementation of Selectionview, the code used in the view was split to several different files in order to increasereadability.

6.4 Development Platform

No single developing platform was used during the project. Each member of theproject used a text editor of his choice to edit the source code. Gedit and TextMatewere the mostly used editors.

The source code uses the UTF-8 character encoding. Version control was handledwith YouSource, which uses Git.

The browser Firefox 3.6.24 was used in testing of the WWW user interface and sup-port for other browsers was explicitly left out to save time. However, standard

23(36)


solutions were used to preserve as much browser compatibility as possible. Firebug1.7.3 plug-in was used to debug Javascript and CSS. The Ruby on Rails console wasused to debug new features.

24(36)


7 Testing Practices and Results

The developed YouHyper software was tested during the project by Kuvatus group,by the customer and by the instructors. The Kuvatus group constantly tested You-Hyper during the development.

In the project plan [4] the group was supposed to plan and carry out a system testingprocedure. Due to the lack of time and by the customer’s request, testing was movedto future developement in favor of more development hours.

7.1 Unit and Integration Testing Practices

When a new feature was developed, it was instantly tested by its developer on hiswork station by putting the feature on its place and testing it in practice. Afterthe first integration tests were carried out succesfully, the functional feature wasthen installed on the test server which was accessible for the members of the projectorganization.

The tests carried out by the project group were based on the functional require-ments specified. The functional requirements in turn were based on the needs andthe goals described by the customer. The complete list of the requirements can befound in Software Requirement Specification [2]. All testing was performed usingFirefox version 3.6.24 in Linux Fedora operating system.

No automatic unit testing was used, although the Ruby on Rails environment sup-ports it. It was considered too time consuming to learn the testing features sup-ported by Ruby on Rails.

7.2 Usability Testing Practices

YouHyper’s usability was tested by the developers, by the instructors and by thecustomer. YouHyper was available for use through most of the project in the ded-icated test server, where the project organization was allowed to use and try outthe features implemented in the current version. However, once a newer versionof MATLAB was required to be used in the application, the hosted YouHyper ver-

25(36)


sion was no longer updated due to unavailability of a new enough MATLAB for theserver.

New features were brought up in weekly project meetings, where the customercould give feedback on the most recently implemented features and provide in-sight as to how they would want them to be implemented. A single usability daywas also held, where a usability expert commented the current version and her userexperience.

7.3 Testing Results

Many minor bugs were discovered and fixed during the integration testing. Manyof these bugs had to do with the user interface elements not working as intended.The most common bugs our group faced and fixed are the following ones:

- The canvas elements used for displaying the drawn selections would be sizedincorrectly.

- The user could use the selection tools without having a selection group active.

- Sometimes, the bitmasks were being sent before decompression.

During the development and integration testing, the following minor bugs werefound that could not be fixed within the available time frame.

- Add all and Remove all buttons do not work on Training sets view.

- The selections that are one pixel wide do not get stored properly for somereason.

- The selection groups in Selections view are being rearranged once the page isrefreshed.

- Zooming and switching images repeatedly can lead to a strange bug wherethe displayed selection canvas area is smaller than the actual image.

- Activating a selection tool, moving the mouse outside of the drawing areaand releasing the mouse button does not register the correct mouseup-event,causing the particular drawing mode to stay active.

26(36)


- The HTML form fields are acting really strangely. For some reason the valuesinside other form fields are getting copied over to the other fields when thepage is refreshed.

Other bugs have been noted and listed in the e-mail archive of the project organiza-tion, many of them pointed out by instructor Jukka-Pekka Santanen.

27(36)


8 Realization of Objectives

The chapter describes the objectives set to YouHyper and how well they were met.The main features that were developed by Kuvatus group concentrated on Selectionand Training set views. YouData integration was done in cooperation with MarkoPeltola. The features that were not implemented included views for Computationsand Jobs as well as a properly implemented way of interfacing with external com-putational software such as MATLAB.

8.1 Realization of Requirements

Software Requirements Specification [2] includes 35 requirements. Due to the lim-ited amount of workhours Kuvatus group was able to implement 26 of these tothe application. For example, nearly all requirements concerning selections and Se-lection view were implemented. Requirements related to training sets were imple-mented right at the end of the project. Requirements concerning computations werenot implemented at all, yet computations are an important part of the analyzationprocess.

The requirements were given priorities as the integer values from one to five withone being the lowest and five the highest. Every requirement with the priorityhigher than three was implemented. Six out of 12 requirements with priority threewere not implemented due to lack of time. These requirements are important so theyand the other unimplemented requirements should be taken into consideration inthe future developement.

8.2 Unsatisfactory Solutions in the Implementation

The selection made to a selection group is uploaded to the server right after theselection has been made. They are, however, combined by the method merge

_selections when Selection view page gets reloaded. This is due to an issuein the HTML5 canvas method draw causing problems with nonzero opacity ar-eas when drawing overlapping shapes from a data table that has been created bycreateImageData method. A proper implementation would be to only merge the

28(36)


selections when creating training sets, but this would require better knowledge onhow to use draw in combination with createImageData.

Currently the hyperspectral image listing in all the views is being generated byJavaScript. A better way would be to have Rails render that part of the view, allow-ing a cleaner implementation of those views as well as the controllers. Once again,this problem was mainly caused by the lack of knowledge and mostly by the lack oftime to acquire the needed knowledge.

8.3 Challenges in the Implementation

Many of the problems faced in development of YouHyper were related to the lackof time reserved for development. The largest problems were encountered whenthe interfaces to the third-party software were specified too late and Kuvatus grouponly had left a small amount of hours reserved for development.

Some problems encountered by Kuvatus group were due to a MATLAB versionthat was too old. The version did not have support for the HDF5 type NetCDF-4container that was specified as the data format to be used in storaging of trainingsets. This was resolved quite quickly by requesting a newer version from IT support.

Some problems were encountered by the late specification of YouData integration.This lead to the group having to implement some YouData features independentlybefore the actual integration process. This caused loss of time from actual produc-tive development.

Problems were also caused by the fact that none of the project members had usedRuby, let alone Ruby on Rails. The application area was also somewhat obscure andnew to the project group. Also the lack of similar example programs was causingtrouble, as the user interface had to be made from nothing. As much as the grouphad problems, many of those were overcome with patience, with the help of goodprogramming tools and with the guidance of the technical instructor.

29(36)


8.4 Modifications During the Implementation

The serialization of selection groups was initially done in the most obvious wayby using an array containing arrays that describe points belonging to the selectiongroup. This was found to be a wrong way due to how JSON serialization workswhen serializing arrays. Each array element is prepended by the array name, objectname and index number, which caused an overhead of 18-to-1 due to the chosenobject format. This was resolved by changing the bit representation to String,which does not require JSON serialization, as it is sent as-is.

The storage and display of selection groups was initially done by simply upload-ing each selection operation and storing the individual selections in an array. How-ever, this was changed because the selection groups would be displayed incorrectlydue to a strange feature when the HTML5 canvas methods createImageData anddraw were used together. It caused a problem in the displaying selections thatwould overlap each other. This was resolved by adding the method merge_selec-tions that merges the existing selections in a selection group once the page is re-freshed.

30(36)


9 Guide for Future Developers

The customers from the Department of Mathematical Information Technology de-cided that the software developed in Kuvatus project is going to be further devel-oped after Kuvatus project ends. The chapter provides some tips to guide the futuredevelopers so that the most essential issues of the software would be addressed assoon as possible. For a more extensive look on the open issues, please see SoftwareRequirements Specification [2].

9.1 Essential Bugs

YouHyper currently has several known bugs and some of them limit the usabilityof the software considerably. The following bugs should be highly prioritized whenthe further development is started:

1. The selection groups are not locked from the other users once someone startsworking on one of them.

2. The first 2 or 3 wavelengths of a hyperspectral image can not be viewed forsome reason.

Lower priority bugs are listed in Section 7.3.

9.2 Improvements to Existing Features

Before starting to improve the existing features the developers should go throughthe unsatisfactory implementation solutions described in Section 8.2. In addition,the following features should be taken into account:

1. A better way of storing the selections should be found, instead of the currentmerge solution. The original way of storing each selection seperately is defi-nately better if a workaround is found for the display problem caused by it.

2. Selections view should be made better. Some of the features, such as the wave-length selector, were added so late in the development and in a hurry, that itdoesn’t really fit the rest of the view.

31(36)


9.3 Further Development Ideas

New ideas for features, usability improvements and bug corrections were suggestedby the customer, the instructors, the usability expert Meeri Mäntylä and by thegroup members. Some of the ideas presented by the instructor are described inthe archive of the e-mail list [6]. Many of these ideas were not developed as thecustomer explicitly stated that they wanted a working prototype supporting all thephases in the analysis process, whether the implementation is ugly or not. A partiallist of specifications can be found in the Software Requirements Specification [2].

Missing views such as Computation and Jobs should be implemented. These areessential for the actual use of the application.

The views should be changed in such a way that the hyperspectral image lists arepopulated by Ruby on Rails controllers instead of JavaScript. This would allow fora much cleaner code than the current JavaScript implementation.

Computation should be changed in such a way that it is used as a global object forall third-party program computational calls, including image opening procedure.This would offer easier tracking of all running processes via Jobs view.

YouData’s Project model should be changed in such a way that it allows the userto select which hyperspectral images he wishes to bring into YouHyperProject.

MATLAB scripts used for opening hyperspectral image files should be implementedby using a separate script for forming the NetCDF file. Having the image openerscall the script with a set of parameters would unify the opening procedure and re-duce code repetition.

Long file names of hyperspectral images should be dealt with gracefully. Currentlythe preview image of a hyperspectral image gets pushed out of it’s location if thefile name does not fit in the given space.

Implementation of the wavelength tool for displaying the intensities of the wave-lengths in a given point should be quite easy, as Ruby can read the NetCDF formatimagecube (see Section 5.3).

32(36)


9.4 Development Practices

Section 14.1 in the project report [3] describes some practices that were found inef-fective during the project or would have been useful if they had been used. Oneimportant thing that should have been done in Kuvatus project was found.

Make use of the branching of Git. Every little new feature and improvementshould have been implemented in its own branch and not in the master branchas stated in [9]. This practice would have helped to keep the version history clean.This feature was only used in the YouData integration process due to the lack ofknowledge about Git.

33(36)


10 Summary

Kuvatus project developed a software prototype for analyzation of hyperspectralimages for Department of Mathematical Information Technology in University ofJyväskylä. Kuvatus project did not complete all of the essential requirements setfor the prototype due to a very tight schedule chosen by the members of Kuvatusgroup. Many of the features were specified too late, at which point the group hada very limited amount of hours reserved for development. Moreover, the customerdecided to continue the development of YouHyper after the project.

The user interface has four different views: Selections, Training sets, Computations andJobs. The first two views were implemented by the project. In addition to this, theunderlying functionalities required for creating NetCDF files from training sets wereimplemented as well. The application is suitable for limited use but production usedemands further development.

The software, called YouHyper, is integrated to YouData application. It was de-veloped using Ruby on Rails and it relies heavily on JavaScript on the client-side.The user interface of YouHyper was developed using HTML5 elements, JavaScript,AJAX and CSS.

The application was once reviewed by a usability expert and the source code wasreviewed twice by the technical instructor of Kuvatus project. The application wastested using ad hoc testing. Planned and documented tests were agreed with thecustomer to be carried out in the future development because workhours wereneeded for other tasks.

34(36)


11 References

[1] Tero Hänninen, Juho Nieminen, Marko Peltola and Heikki Salo, "Verso Project,Application Report" University of Jyväskylä, Department of Mathematical In-formation Technology, 2010.

[2] Pekka Iso-Ahola, Jussi Perttola and Tommi Tuovinen, ”Kuvatus Project, Soft-ware Requirement Specification”, University of Jyväskylä, Department ofMathematical Information Technology, 2012.

[3] Pekka Iso-Ahola, Jussi Perttola and Tommi Tuovinen, ”Kuvatus Projekti, Pro-jektiraportti” (Kuvatus Project, Project Report), University of Jyväskylä, De-partment of Mathematical Information Technology, 2012.

[4] Pekka Iso-Ahola, Jussi Perttola and Tommi Tuovinen, ”Kuvatus Projekti, Pro-jektisuunnitelma” (Kuvatus Project, Project Plan), University of Jyväskylä, De-partment of Mathematical Information Technology, 2012.

[5] Pekka Iso-Ahola, Jussi Perttola and Tommi Tuovinen, ”YouHyper Class Docu-mentation”, Department of Mathematical Information Technology, 2012.

[6] Jukka-Pekka Santanen, ”Huomioita YouHyper sovelluksesta” (NotesAbout YouHyper Application), available at Kuvatus Mail Archive in URLhttps://korppi.jyu.fi/kotka/servlet/list-archive/kuvatus/

0196.html and https://korppi.jyu.fi/kotka/servlet/list-archive/kuvatus/0149.html, University of Jyväskylä, Department of MathematicalInformation Technology, 2012.

[7] Douglas Crockford, ”Code Conventions for theJavaScript Programming Language”, available at URLhttp://javascript.crockford.com/code.html, cited at 19.4.2012.

[8] Libin Pan, ”Ruby on Rails Code Conventions”, available at URLhttp://rails.nuvvo.com/lesson/5017-ruby-coding-convention,cited at 19.4.2012.

[9] Vincent Driessen, ”A Successful Git Branching Model” available at URLhttp://nvie.com/posts/a-successful-git-branching-model/,cited at 19.4.2012.

35(36)


[10] James Britt of Neurogami, ”Ruby-Doc: Documenting the Ruby Language”available at URL http://www.ruby-doc.org/, cited at 19.4.2012.

[11] Ruby on Rails Community, ”Ruby on Rails Guides”, available at URLhttp://guides.rubyonrails.org/, cited 19.4.2012.

36(36)

kuvatus project - jyväskylän...

Documents