Software Visualization and Deep Transfer Learningfor Effective Software Defect Prediction

Jinyin Chen1 Keke Hu1 Yue Yu2 Zhuangzhi Chen1 Qi Xuan3 Yi Liu4lowast Vladimir Filkov5

chenjinyinzjuteducn2111703352zjuteducnyuyuenudteducn2111703088zjuteducnxuanqizjuteducnyliuzjuzjuteducnfilkovcsucdavisedu

1College of Information Engineering Zhejiang University of Technology Hangzhou 310023 China2College of Computer National University of Defense Technology Hefei 230000 China

3Institute of Cyberspace Security Zhejiang University of Technology Hangzhou 310023 China4Institute of Process Equipment and Control Engineering Zhejiang University of Technology Hangzhou 310023 China

5Department of Computer Science University of California Davis CA 95616 USA

ABSTRACTSoftware defect prediction aims to automatically locate defectivecode modules to better focus testing resources and human effortTypically software defect prediction pipelines are comprised of twoparts the first extracts program features like abstract syntax treesby using external tools and the second applies machine learning-based classification models to those features in order to predictdefective modules Since such approaches depend on specific featureextraction tools machine learning classifiers have to be custom-tailored to effectively build most accurate models

To bridge the gap between deep learning and defect prediction wepropose an end-to-end framework which can directly get predictionresults for programs without utilizing feature-extraction tools To thatend we first visualize programs as images apply the self-attentionmechanism to extract image features use transfer learning to reducethe difference in sample distributions between projects and finallyfeed the image files into a pre-trained deep learning model fordefect prediction Experiments with 10 open source projects from thePROMISE dataset show that our method can improve cross-projectand within-project defect prediction Our code and data pointers areavailable at httpszenodoorgrecord3373409XV0Oy5Mza35

KEYWORDSCross-project defect prediction within-project defect predictiondeep transfer learning self-attention software visualization

ACM Reference FormatJinyin Chen1 Keke Hu1 Yue Yu2 Zhuangzhi Chen1 Qi Xuan3 Yi Liu4lowastVladimir Filkov5 2018 Software Visualization and Deep Transfer Learningfor Effective Software Defect Prediction In 42th International Conferenceon Software Engineering May 23-29 2020 Seoul South Korea ACM NewYork NY USA 12 pages httpsdoiorg10114511224451122456

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page Copyrights for components of this work owned by others than ACMmust be honored Abstracting with credit is permitted To copy otherwise or republishto post on servers or to redistribute to lists requires prior specific permission andor afee Request permissions from permissionsacmorgCorresponding author Yi LiuICSE 2020 May 23-29 2020 Seoul South Koreacopy 2018 Association for Computing MachineryACM ISBN 978-1-4503-9999-91806 $1500httpsdoiorg10114511224451122456

1 INTRODUCTIONSoftware defect prediction techniques can help software developerslocate defective code modules automatically to save human effortand material resources Most prediction methods build predictionmodels based on modules in the source code and historical devel-opment data at different levels of modeling eg commit changesmethods and files [1] In practice based on whether the historicaltraining data comes from the same project or not we distinguishbetween within-project defect prediction (WPDP) and cross-projectdefect prediction (CPDP) [2]

A large number of manually designed features including staticcode features and process features [3] have been adopted to predictwhether a module is defective or not To improve on those currentsoftware defect prediction studies mainly focus on two main direc-tions new feature extraction methods and classification methodslearned from large-scale datasets However defect prediction tech-niques which feed manually designed features into machine learningalgorithms for classification have some limitations [4] The requiredfeature engineering is time consuming and requires that special toolsbe used upstream such as code complexity analysis submissionlog mining and code structure analysis tools Consequently manyfeatures can be difficult to capture in some projects For examplesemantic code information such as the features hidden in abstractsyntax trees (ASTs) may not be effectively represented by existingtraditional features In addition to the inconvenience of feature engi-neering for traditional features as described by Wan et al in a recentreview of defect prediction [5] semantic information can be morecapable than syntax information to distinguish one code region fromanother Thus while AST-conveyed features can be useful for defectprediction such approaches are indirect requiring additional tools inorder to build and mine the ASTs Moreover in such approaches thesource code is most frequently not used once the ASTs are extracted

A number of machine learning methods eg support vector ma-chines (SVMs) [6] naive Bayes (NB) [7] decision trees (DTs) [8]and neural networks (NNs) [9] have been applied to defective mod-ule prediction In particular recent research has been conclusivethat deep learning networks are highly effective in image classifi-cation feature extraction and knowledge representation in manyareas [10ndash15] In defect prediction specifically to better generatesemantic features a state-of-the-art method [16] leveraged deepbelief network (DBN) for learning features from token vectors ex-tracted from programsrsquo ASTs On this basis Li et al [17] and Dam

1

ICSE 2020 May 23-29 2020 Seoul South Korea Chen et al

Figure 1 A motivating example

et al [18] used the structural information of programs and the seman-tic relations between keywords to improve defect prediction withconvolutional neural network (CNN) and long-short-term-memorynetworks (LSTMs) In those papers the required feature engineeringwas significant and required specific tools be used upstream

In this work we are motivated by the possibility of improving de-fect prediction by avoiding intermediary representations eg ASTsand instead obtaining code semantic information directly To thatend inspired by the power of existing deep learning platforms forimage classification in this paper we propose a more direct way touse programsrsquo semantic information to predict defects by represent-ing source code as images and training image classification modelson those images

Figure 1 shows a motivating example The two Java files on topFile1java and File2java are similar both containing 1 if state-ment 2 for statements and 4 function calls Nevertheless the filesrsquosemantic and structural features are different We wondered if avisualization can help to tell that those two programs are differentTo do so we turned code characters into pixels their colors based onthe charactersrsquo ASCII decimal value and then arranged those pixelsin a matrix thus obtaining code images By comparing those imageswe were able to visually recognize significant differences betweenthe corresponding programs as shown in the bottom of Figure 11Both semantic and structural differences can be recognized visuallyin those images That leads us to our driving thesis

Semantic and structural similarities between two programs can beeffectively identified by visually comparing program images

To test that thesis we start from a well known data set fromthe PROMISE repository of 10 projects with known defective filesOnce the files are converted to images we automatically extractfeatures and build a classification model with the popular AlexNetplatform [19] In addition we use deep transfer learning [20] and self-attention mechanism [21] (the details are in Figure 4 and Section 32)to further improve the defect prediction especially CPDP Finallywe propose an end-to-end framework ie deep transfer learning

1There were a number of technical details and choices we had to make when convertingprograms into images we discuss those in the Methods section

for defect prediction (DTL-DP) to automatically predict whether aprogram file contains defects or not

This paper makes the following contributions

bull We propose a novel approach for defect prediction based onvisualizing program files as images in order to capture theirsemantic and structural information

bull We propose an end-to-end deep-learning framework DTL-DP comprised of a deep transfer learning model combinedwith a self-attention mechanism which takes program fileimages as input and outputs labels either rsquodefectiversquo or rsquonotdefectiversquo

bull Our experiments on 10 OS Java projects show that our ap-proach improves both WPDP and CPDP The advantages ofDTL-DP in CPDP are much more obvious than in WPDPWe show that self-attention has the greatest contribution toWPDP and transfer learning contributes most to CPDP

The remainder of this paper is organized as follows First wereview related work in Section 2 Then we describe DTL-DP andpresent our experimental setup in Sections 3 and 4 respectivelyAfter that we discuss the performance of DTL-DP in Section 5 andthe threats to validity in Section 6 Finally we conclude the workand provide several potential future directions in Section 7

2 RELATED WORKDefect prediction (DP) As one of the primary areas of interest insoftware engineering research DP has been receiving significantattention [22ndash25] A number of studies have focused on manuallydesigning features or producing new combinations of existing fea-tures from labeled historical defects Those features are typicallyfed into a machine learning-based classifier to determine if a fileis defective Commonly used features can be divided into staticeg code size and code complexity (eg Halstead features [26]McCabe features [27] CK features [28]) and process features likethe behavioral differences of developers in the software developmentprocess Many studies have demonstrated that process features canpredict software quality [29] Moser et al [24] used authors pastfixes the number of revisions and ages of files as features to predictdefects Nagappan et al [30] indicated that code churn was effec-tive for DP Hassan et al [31] used entropy of changes to predictdefects Other process features are helpful too including individualdeveloper characteristics [22 32] and their collaborations [33ndash35]

Based on these features many machine learning models includ-ing SVM [6] NB [7] DT [8] NN [9] etc have been built for thetwo different DP tasks within-project WPDP and cross-projectCPDP For WPDP the training set and test set come from the sameproject while for CPDP they come from different projects WhileWPDP can give better results it is of limited use in practice as it isoften difficult to obtain enough training data for a new project Somestudies have instead used related projects to build prediction modelswith sufficient historical data and then used them to predict defectsin the target project [36ndash39] Panichella et al [40] proposed an ap-proach named CODEP which uses a classification model to combinethe results of 6 classification algorithms (including logistic regres-sion radial basis function network and multi-layer perceptrons)for CPDP Turhan et al [41] and Peters et al [42] used differentstrategies to select appropriate instances in a source project based on

2

Software Visualization and Deep Transfer Learningfor Effective Software Defect Prediction ICSE 2020 May 23-29 2020 Seoul South Korea

nearest neighbor methods to train the prediction models Addition-ally the application of transfer learning is receiving more attentionXia et al [43] proposed an approach named HYDRA to build classi-fiers using genetic algorithm and ensemble learning Ma et al [44]proposed a transfer naive Bayes (TNB) method to assign weightsto training instances and then used them to construct a predictionmodel Nam et al [45] proposed TCA+ which uses TCA [46] andoptimizes the normalization to improve CPDP

Recently some studies have used deep learning to directly get fea-tures from source code for DP Wang et al [16] deployed deep beliefnetworks (DBNs) to learn semantic features from token vectors ex-tracted from ASTs automatically and leveraged the learned semanticfeatures to build machine learning models for DP Li et al [17] usedconvolutional neural networks (CNNs) to generate features fromsource code and combined CNN learned features with traditionalfeatures to further improve upon the prediction Similarly Dam etal [18] proposed a prediction model that takes ASTs representingthe source file as input It used an LSTM architecture to capturelong-term dependencies which often exist between code elements

Our proposed approach differs from the above in that we donot use feature engineering followed by machine learning-basedclassifiers We also do not need ASTs to bridge the gap betweendefect prediction and deep learning Instead our approach is basedon feeding software visualizations into an automatic image basedfeature discovery pipeline

Software visualization (SV) for DP SV has long history in soft-ware engineering research and practice [47 48] It has been usedfor visualizing code structure and features [49] code execution [50]and evolution [51] of large codebases in particular [52] SV hasalso been applied to bug repositories where it has been helpfulin correlating bugs with code structure [53] In fact the way codewas visualized in a recent study on malware code visualization [54]has in part inspired us in this paper However to the best of ourknowledge there is a dearth of applications of SV to software defectprediction in the research literature perhaps because prior to DLapproaches no connection was seen between the two areas

Deep transfer learning (DTL) Transfer learning (TL) is an im-portant tool that can help when there is insufficient training datain machine learning It works by transferring information from asource domain to a target domain The key is to relax the assumptionthat the training and test data must be independent and identicallydistributed [55] This is helpful in areas where it is difficult to getenough training data In the field of DP studies have shown thatCPDP can achieve better performance with transfer learning [56]

Most TL studies are based on traditional machine learning meth-ods Recently deep learning-based transfer learning studies haveemerged called deep transfer learning DTL Based on the tech-niques used in them they can be divided into four categories network-based instance-based mapping-based and adversarial-based [57]

Instance-based DTL is implemented through a specific weightadjustment strategy The instances selected from the source domainare assigned weights that complement the training data in the targetdomain [58ndash61] Network-based DTL refers to the use of a partiallytrained network in the source domain for the deep neural networkof the target domain including its network structure and connecting

parameters [62 63] Adversarial-based DTL refers to introducing ad-versarial technology inspired by generative adversarial nets (GAN)to find a transferable representation that is applicable to the sourceand target domains [64 65] Mapping-based DTL refers to map-ping instances of the source and target domains to a new data spacewhere the distributions of the two domains are similar TCA [46]and TCA-based methods have been widely used in applications oftraditional transfer learning [66] Tzeng et al [67] used maximummean discrepancy (MMD) to measure the sample distribution af-ter deep neural network processing and learned domain invariantrepresentations by introducing an adaptation layer and additionaldomain confusion loss Long et al [68] improved previous work byreplacing MMD with multiple kernel variant MMD (MK-MMD)originally by [69] and proposed a method called deep adaptationnetworks (DAN) Long et al [70] proposed joint maximum meandiscrepancy to promote the transfer learning ability of neural net-works to adapt to the data distribution of different domains TheWassersteinrsquos distance proposed by Arjovsky et al [71] has beenused as a new distance measure to find a better mapping betweendomains

To the best of our knowledge DTL methods have not yet beenused in defect prediction In this paper we propose a novel mapping-based DTL method using a self-attention mechanism for defectprediction

3 APPROACHThe overall framework of our approach deep transfer learning fordefect prediction DTL-DP is shown in Figure 2 It is comprised oftwo stages (1) source code visualization and (2) modeling with DTLIn the first stage we use a visualization method to convert programfiles into images In the second stage we build a DTL model basedon the AlexNet network structure with transfer learning and a self-attention mechanism to construct an end-to-end defect predictionframework We use that framework to predict if a new instance fileis defective or not

In a nutshell our approach takes the raw program files of a train-ing and test sets directly as input and generates images from themwhich are then used to build the evaluation model for defect predic-tion Specifically since the input data of the network model basedon the CNN structure should be in the form of images we builda mapping to convert the files into images Then we use the first5 layers of AlexNet as Feature-Net to generate features from theimages The shallow CNN layers correlate the presence and absenceof defects to the overall code structure gradually deepening thegranularity from functionloop bodies to function names identifiersetc These features are then fed into the Attention Layer where theyare used in assigning weights and highlighting features which aremore helpful in the classification The re-weighted features of thetraining and test sets are used to calculate MK-MMD which is usedto measure the difference between their distributions as the MMDloss After that the re-weighted features of the training set are en-tered into the fully connected layers to calculate the cross entropyas the classification loss The weighted sum of the MMD loss andclassification loss is fed back to train the whole network includingFeature-Net Attention Layer and the fully connected layers Finallybased on the source code visualization method and the DTL model

3

ICSE 2020 May 23-29 2020 Seoul South Korea Chen et al

DataAugmentation

B G R

GRB

R B G

Feature- Net

AttentionLayerBGR

G R B

RBG

Clean Buggy

Clean Buggy

SourceProject

TargetProject

MMD

Clean

Buggy

Clean

Buggy

Defect Visualization Model Building Phase

Prediction Phase

Figure 2 The overall framework of DTL-DP

Figure 3 The process of converting code to color images

we build train and evaluate a defect prediction model We give thedetails in the following

31 Source Code VisualizationConverting code to images retains information and facilitates avail-able deep neural network (DNN) -based methods to improve defectprediction performance Most existing methods ignore associationsbetween extracted features and the classification task Our proposedmethod forms a unified end-to-end framework of feature extractionmodeling and prediction

Figure 3 shows how we convert each program file into 6 differentimages We call this process source code visualization and the im-ages produced code images First each source file is converted into avector of 8-bit unsigned integers corresponding to the ASCII decimalvalues of the characters in the source code eg rsquoarsquo is converted to97 etc We then generate an image from that vector by arrangingits values in rows and columns and interpreting it as a rectangularimage A straight forward way to visualize the 8-bit values as colorintensities would be as levels of gray eg 0=black and 255=whiteHowever the relatively small size of our original training set meansthat if we did that we would end up with a deep model that cannot besufficiently well trained Fortunately an additional benefit to using

images for training is that we can augment the original data set toproduce a larger data set with the same semantics Common dataaugmentation methods for image data sets include flipping (bothvertically and horizontally) rotating cropping translating (movingalong the x or y axis) adding Gaussian noise (distortion of highfrequency features) zooming and scaling Since the sizes of our pro-grams are small and we want to retain the semantic and structuralfeatures in the images the above image data augmentation methodscould result in data loss

Instead we designed a novel color based augmentation methodto generate 6 color images from each source code file Namelyeach pixel in a color image can be represented by three primarycolor components or channels red (R) green (G) and blue (B)The letters R G B and thus the color channels can be orderedin 6 different ways RBG RGB BGR BRG GRB and GBR Byadopting a different order every time as shown in Figs 2 and 3 wegenerate six different images for each program file For examplersquoforrsquo is converted to [102 111 114] and the 3 values are filled intothe R G and B channels in 6 different ways to obtain 6 differentlycolored pixels

The above method expands our data set six-fold and because ofthe nature of the downstream analysis the generated samples arereasonable since the representation is changed only by the order ofthe different channels 2

Whereas we generated 6 images for each instance in the trainingset we randomly selected an RGB permutation for testing We foundthat it was not necessary to generate all six images for testing becauseour experiments on 10 datasets showed that the performance of usingthe different permutations was quite comparable We randomly choseone to increase the speed in practice The data augmentation wasused to improve the efficacy of the model

After the above for each of the 6 orderings of R G B we obtaina vector of pixels of length one third the original code file size(each source code character is assigned to a color channel and thusthree characters in a row represent a pixel) We then turned those

2A CNN model learns the features by convolution Since we use ImageNetrsquos pre-trainedAlexNet model the initial parameters of the convolution kernel are different for eachchannel which means even though byte sequences are the same the final set of featuresobtained by sequentially inputting into the model the different channels is also different

4

Software Visualization and Deep Transfer Learningfor Effective Software Defect Prediction ICSE 2020 May 23-29 2020 Seoul South Korea

Figure 4 The training and testing process of our approach DTL-DP inspired by the original DAN paper [68]

Table 1 Image Width for Various File Sizes

File Size Range Image Width

lt10 kB 3210 kB - 30 kB 6430 kB - 60 kB 12860 kB - 100 kB 256

100 kB - 200 kB 384200 kB - 500 kB 512500 kB - 1000 kB 768

gt1000kB 1024

6 image vectors into image matrices of width and length such thatwidth lowast lenдth = vector_size3 We note that the first convolutionkernel of AlexNet is of size 1111 so if the image is too narrowthe convolution performance will be poor And the image with asuitable width can be convolved to obtain more efficient semanticand structural features in contexts with a proper sequence lengthTable 1 gives some recommended image widths for different filesizes based on previous work [72 73] We adopt those in this work

32 Modeling with Deep Transfer LearningNext we describe our approach DTL-DP The goal is to learn trans-ferable features across projects and build a classifier y = ϕ(x) offile defectiveness with source code supervision where x is the rep-resentation of the sample and y is the predicted label In the defectprediction problem we are given a source project Ps = (xsi y

si )

nsi=1

with ns labeled instances and a target project Pt = (xti yti )

nti=1

with nt unlabeled instances In WPDP the source project is the pre-vious version of the target project while in CPDP the source projectis a related project of the target project The sample distributions pand q in the source and target projects are often similar in WPDPbut different in CPDP

For our deep network architecture we adopt a similar architec-ture as that of deep adaptation networks (DANs) [68] to capturethe semantic and structural information of the source code To theoriginal DAN model we add an attention layer to further enhancethe expressive ability of important features The overall architectureis illustrated in Figure 4 In particular our DTL-DP consists of aninput layer five convolution layers (conv1 minus conv5 from AlexNet)an Attention-Layer and finally four fully-connected hidden layers(f c6 minus f c9) working as a classifier The structure and parameters of

(conv1minusconv5) and (f c6minus f c8) are consistent with a DAN But sincedefect prediction is a binary classification problem a fully-connectedlayer (f c9) is added to obtain a binary result at the end

We adopt the defaults for AlexNet so the input to our DTL-DPmust be a 224224 size image cropped from a three-channel (RGB)image of size 256256 The code image is placed in the approximatecenter of the 256256 image (the determination of the size of eachcode image is described later in Sect 33) We note that the codeimage can be smaller than 224224 pixels due to the variance in thesize of the code files If this happens the image is padded aroundwith blank (zero valued) pixels Padding should not negatively effectthe qualitative performance of feature detection in the images in factfor deeper DL architectures like ours padding has been shown toprovide extra contrast to the embedded image for each of the layersas well as buffering against data loss by each layer [74]

Training deep models requires a significant amount of labeleddata which is not available for a new project so we first pre-trainedan AlexNet model on ImageNet 2012 Unlike a DAN which freezesconv1minusconv3 and fine-tunes conv4minusconv5 we fine tune all convolu-tion layers conv1minusconv5 by taking the parameters of the pre-trainedmodels as initial parameters that we then optimize during the train-ing phase We do that to minimize the differences between our codeimages and the actual object images in ImageNet 2012

In order for DTL-DP to focus on the key features in differentdefect images and thus further improve prediction we employ aself-attention mechanism into our model inspired by the good per-formance of self-attention in GANs [21] As shown in the AttentionLayer in Figure 4 the attention mechanism makes the feature mapgenerated by layer conv5 be the self-attention feature map input tothe next layer fc6 Specifically at first it linearly maps the inputfeatures x (it is a 11 convolution to compress the number of chan-nels ie out_chanels = in_chanels8) and produces f (x)д(x)and h(x) where f (x) = wf x д(x) = wдx h(x) = whx The dif-ference between the three is that the size of h(x) is still the sameas x but the other two are not Thus if the width of x is W theheight H and the number of channelsC the size of x is [CN ] whereN =W lowast H the size of f(x) and g(x) is [C8N ] but the size of h(x)is [CN ] The transposed f (x) and д(x) are matrix-multiplied to ob-tain the autocorrelation in the features ie the relationship of eachpixel to all other pixels where Si j = f (xi )

Tд(x j ) Then softmax isapplied to the autocorrelation features S to get the attention mapcomprised of weights with values between 0 and 1

α ji =exp(Si j )sumNi=1 exp(Si j )

(1)

5

ICSE 2020 May 23-29 2020 Seoul South Korea Chen et al

005 01 02 05 1 2 5MMD Penalty

055

056

057

058

059

060

061

062

063

064

065

F-m

easu

re

WPDP CPDP

(a) Effect of different MMD Penalty λ

BRG BGR GRB GBR RGB RBG040

045

050

055

060

065

070

075

F-m

easu

re

WPDP CPDP

(b) Effect of color channel order

8 4 2 1 12 14 18Width recommended

040

045

050

055

060

065

070

075

F-m

easu

re

WPDP CPDP

(c) Effect of code image widthrecommended ratio

Figure 5 Sensitivity studies to aid in choosing the parameters in the model

After that the output of the AttentionLayer is the self-attentionfeature map o = (o1o2 oj oN ) where

oj =Nsumi=1

α jih(xi ) (2)

Then each fully connected layer learns a nonlinear mappinghli = f l (wlhli + b

l ) where hli is the l th layer hidden representationof feature xi wl and bl are the weights and biases of the l th layerand f l is the activation function using ReLU (f l (o) = max(0o))for f c6 minus f c8 and softmax for f c9 If we let Θ denote the set of allDTL-DP parameters the empirical risk of DTL-DP then is

minΘ

1n

nsumi=1

F (ϕ(oi )yi ) (3)

where F is the cross-entropy loss function n is the number of in-stances and ϕ(oi ) is the conditional probability that the DTL-DPassigns oi to label yi It is used to calculate the final loss in Equation7

To make the distribution of the source and target projects similarthe same multi-layer adaptation and multi-kernel MMD strategy asin a DAN [68] are used in our model The feature mapping functionσ is defined as the combination ofm positive semi-definite kernelsku

lt σ (os )σ (ot ) gt= k(os ot ) (4)

k =msumu=1

βuku (5)

where β ge 0 are the weights of the kernelsThe MK-MMD dk (pq) of the probability distributions p and q

is defined as the reproducing kernel Hilbert space distance betweenthe mean embedding of p and q The square formula of MK-MMDis defined as

d2k (pq) =∥ Ep [σ (xs )] minus Eq [σ (x

t )] ∥2 (6)

We see then that the smaller dk (pq) is the more similar p and qare If dk is 0 the distribution of the target project is the same as thatof the source project So the final loss function for train DPL-DP is

1n

nsumi=1

F (ϕ(oi )yi ) + λ

l2suml=l1

d2k (Dls D

lt ) (7)

where Dls is the l th layer hidden representation for the source and

target d2k (Dls D

lt ) is the MK-MMD between the source and target

evaluated on the l th layer representation λ is a penalty parameterand l1 and l2 are f c layers to calculate the MK-MMD between sourceand target In our implementation of DTL-DP we set λ = 1 l1 = 6and l2 = 9 as per the original DAN work [68]

33 Model Sensitivity to Parameter ChoicesWe have made a number of choices to make our modeling plat-form work effectively Here we justify those choices by presentingsensitivity studies For this analysis we used all the data

Hyperparameter λ The DTL-DP model has a hyper-parameter λfor the MMD penalty that adjusts the final loss We conducted λ pa-rameter sensitivity experiments in both the WPDP and CPDP settingWe fix the other parameters and range λ in 001 005 01 02 05 1 2 5The results are shown in Figure 5(a) While fairly constant across therange small variations exist In CPDP the F-measure first increasesand then decreases along the hyper-parameter λ In WPDP λ affectsthe performance of DTL-DP but in the opposite direction We choseλ = 1 here

Different Color Orders The selection of the RGB permutationsin the target project images may also potentially affect the outcomeWe performed experiments with all six different orderings Whentesting them we set λ to 1 and changed the image type of the tar-get project to each one in RGBRBGBRGBGRGBRGRB Theresults are shown in Figure 5(b) The target images in different colororders have somewhat different performance But overall the valuesare close to DTL-DPrsquos average performance

Different Image Widths To explore the impact of different choicesfor image widths on the results we performed additional experi-ments

For our code images 3 bytes of source code are needed to form 1pixel each byte for one of the R G and B channels The sizes ofthe source files are between 0 and 100 kb corresponding to imageswith pixel count between 0 and 34 133 (10010243) The size ofthe image is (widthheiдht) where width is obtained from Table 1according to the size of the code file (size)

heiдht =size lowast 10243 lowastwidth

(8)

For example the size of image converted from a 20 kb code file is(64 107)

We experimented with image widths of 18 14 12 2 4 and 8times of the recommended widths in Table 1 The larger the mul-tiplier the wider the image For images of different widths we

6

Software Visualization and Deep Transfer Learningfor Effective Software Defect Prediction ICSE 2020 May 23-29 2020 Seoul South Korea

repeated the previous experiment and obtained results for the meanF-measure shown in Figure 5(c) We see that the closer to the recom-mended width from Table 1 the better the average result In additionwe found that a wider image is better than a narrower one And thecloser the image is to a square the better the prediction Thereforewe made the generated images be as close to a square as possible

34 Training and PredictionWhen training the DTL-DP code images are generated from thelabeled instances in the source project and the unlabeled instances inthe target project and then are simultaneously input into the modelThey share the convolution layers conv1minusconv5 andAttentionLayerto extract their respective features and calculate the MK-MMDbetween source and target projects in the fully connected layersf c6 minus f c9 It should be noted that we only calculate cross entropyfor the source project because the target projectrsquos labels are not pro-vided We use a mini-batch stochastic gradient descent along the lossfunction in order to train the parameters of the entire model We trainfor 500 epochs for each pair of source and target projects then pickthe epoch with the best F-measure (described in section 43) fromwhich to read out the parameters of the final model Finally the filesfrom the target project that need to be predicted are converted intocode images and then input into the trained model for classification

Overfitting is always a possibility with such pipelines We moder-ate it here with our choices of (1) augmenting the dataset by using sixcolor channel permutations described in section 31 (2) selecting asimple AlexNet model structure and (3) using the ImageNet 2012pre-trained model to reduce fluctuations

4 EXPERIMENTAL SETUPWe conducted experiments to asses the performance of DTL-DP andto compare it with existing deep learning-based defect predictionapproaches for both within-project WPDP and cross-project CPDPdefect prediction We ran experiments on a modern-day Linux serverwith 3 Titan XP GPUs Unless otherwise stated each experimentwas run 10 times and the average results are reported

41 Dataset DescriptionIn order to directly compare our work with prior research we usedpublicly available data from the PROMISE3 data repository whichhas been widely used in defect prediction work [16ndash18 75] Weselected all open source Java projects from this repository andcollected their version number class names and the buggy labelfor each file Based on the version number and class name weobtained the source code for each file from GitHub4 and fed it toour end-to-end framework In total data for 10 Java projects werecollected Table 2 shows the details of these projects includingproject description versions the total and average number of filesand the defect rate It should be noted that the average numberof files over all projects ranges between 150 and 1046 and thedefect rates of the projects have a minimum value of 134 and amaximum value of 497 The number of files in some projects isnot sufficient to train deep models and the classes are imbalancedthus augmentation is needed as described above

3httpopenscienceusrepodefect4httpsgithubcomapache

42 Baseline Comparison MethodsTo evaluate the performance of our end-to-end framework DTL-DPfor defect prediction we compare it with the following baselinemethods in the WPDP setting

bull Semantic [16 75] the state-of-the-art method which em-ploys deep belief networks DBN on source code to extractsemantic features for defect prediction

bull PROMISE-DP [16] a traditional method which builds analternating decision tree ADTree classifier based on theoriginal 20 features of the PROMISE dataset

bull DP-LSTM [18] a long short-term memory LSTM-baseddeep neural network model which uses ASTs to representsource files and predict whether the file is defective or not

bull DP-CNN [17] a convolutional neural network CNN-basedmodel which is seeded by AST-derived numerical vectorsto automatically learn semantic and structural features ofprograms The CNN-learned features are used to train thefinal classifier in combination with traditional features

For the cross-project settings CPDP Semantic and PROMISE-DPcould not be used directly Instead we used the following

bull DBN-CP [16 75] a variant of Semantic which trains a DBNby using the source project and generates semantic featuresfor both the source and target projects

bull TCA+ [45] the state-of-the-art technique for CPDP

To obtain the training and test data we followed the processesestablished in [16] For WPDP we use two consecutive versions ofeach project listed in Table 2 The older version is used to generatethe training data and the more recent version is used as test data ForCPDP we pick versions randomly from each project for 11 targetprojects And for each target project we select 2 source projects thatare different from the target projects We use the same 22 test pairsas in [16] When implementing the baseline methods we use thesame network architecture and parameter settings as described in thepapers that introduced them

43 Performance measuresTo evaluate the prediction performance we use the F-measure awidely adopted metric in the literature [16ndash18 43 45] The F-measure captures a predictorrsquos accuracy and combines both precisionand recall for a comprehensive evaluation of predictive performance

Specifically a prediction that a file is defective is called a truepositive (TP) if the file is in fact defective and false positive (FP)otherwise Similarly a prediction that a file is not defective is a truenegative (TN) if the file is in fact not defective and false negative(FN) otherwise Then the precision (P) recall (R) and F-measureare defined as

P = TP(TP + FP) (9)

R = TP(TP + FN ) (10)

F = (2 times P times R)(P + R) (11)

5 RESULTSThis section discusses our results of comparing DTL-DP to baselinetools for defect prediction

7

ICSE 2020 May 23-29 2020 Seoul South Korea Chen et al

Table 2 Dataset Description

Project Description Versions Files Avg files Avg size(kb) Defective

ant Java based build tool 151617 1465 488 62 134camel Enterprise integration framework 121416 3140 1046 29 187jEdit Text editor designed for programmers 324041 1935 645 87 192log4j Logging library for Java 1011 300 150 34 497

lucene Text search engine library 202224 607 402 38 358xalan A library for transforming XML files 2425 1984 992 46 296xerces XML parser 1213 1647 549 29 157

ivy Dependency management library 1420 622 311 41 200synapse Data transport adapters 101112 661 220 38 227

poi Java library to access Microsoft format files 152530 1248 416 36 407

Table 3 F-measure of DTL-DP Semantic (Seman) PROMISE-DP (PROM) DP-LSTM (LSTM) and DP-CNN (CNN) forWPDP

Project Version Seman PROM LSTM CNN DTL-DP

ant15-gt16 914 477 452 532 70016-gt17 942 542 379 566 593

camel12-gt14 785 373 323 461 40614-gt16 374 391 400 508 385

jEdit32-gt40 574 556 472 564 49440-gt41 615 546 490 580 687

log4j 10-gt11 701 587 539 632 688

lucene20-gt22 651 502 751 761 78322-gt24 773 605 751 721 776

xalan 24-gt25 595 518 657 601 794xerces 12-gt13 411 238 268 374 820

ivy 14-gt20 350 329 191 347 829

synapse10-gt11 544 476 454 539 54711-gt12 583 533 533 556 659

poi15-gt25 640 558 808 589 68525-gt30 803 754 777 784 426

Average 641 499 515 570 642

51 RQ1 How does DTL-DP compare to feature-based machine learning methods and AST-based deep learning methods in WPDP

We compare DTL-DP to 4 baseline approaches representing twodifferent kinds of defect prediction methods PROMISE-DP is thebaseline representative of traditional feature-based machine learningmethods Semantic DP-LSTM and DP-CNN are the baselines fordeep learning-type methods based on extracting features from ASTGuided by prior work we conducted 16 sets of WPDP experimentseach using two versions of the same project The older version isused to train the prediction model and the newer version is used toevaluate the trained model

Table 3 shows the F-measure values for the within-project WPDPdefect prediction experiments The highest F-measure values of the5 methods are shown in bold Since the methods based on deeplearning include some randomness we run DTL-DP Semantic DP-LSTM and DP-CNN 10 times for each experiment On average

the F-measure of our approach is 0642 and the PROMISE-DPSemantic DP-LSTM and DP-CNN achieve 0499 0641 0515 and0570 respectively The results demonstrate that our approach iscompetitive and may improve on defect prediction compared toPROMISE-DP DP-LSTM and DP-CNN The results of Semanticand our approach are similar

The proposed DTL-DP is effective and it could improvethe performance of WPDP tasks Like other deep learners it issensitive to small file sizes and unbalanced data

511 Case Study WPDP Discrimination of DTL-DP t-SNEis a non-linear dimensional reduction algorithm that is effectivein visualizing similarities and helping identify clusters in complexdata sets [76] To give insight in the performance of DTL-DP wedemonstrate feature transferability by showing t-SNE embeddingsin Figure 6 The blue points are non-defective files and the red aredefective ones5 We observe the following (1) The target instancesare not discriminated very well using either the traditional manualfeatures or the TCA+ improved manual features or the semanticfeatures extracted from ASTs while with our approach the pointsare discriminated much better (2) With the other three approachesthe categories between source and target projects are not well alignedwhile with our approach the categories between the projects aremore consistent These conclusions are derived from the intra- andinter-class distances of the two categories in Figures 6 and 7 Theyare visually apparent

Our method did not perform as well as the comparison methods onsome projects such as ant which is likely caused by large variancein file sizes While the sizes of ant-15 and ant-16 are close there isa marked difference between ant-16 and ant-17 the former beingmuch smaller than the latter From Table 3 the performance of ourmethod on ant15-gtant16 is better than that on ant16-gtant17On the contrary baseline methods other than DP-LSTM performbetter for the task ant16-gtant17 notably Semantic is dominantindicating that the semantic feature-based method is more robust tofile size variability Moreover the ant dataset has two shortcomingsthe first is that the amount of data is small cf Table 2 where theaverage amount is only 488 files in one project The second is that

5Since we use data augmentation we have more samples than the three baselines

8

Software Visualization and Deep Transfer Learningfor Effective Software Defect Prediction ICSE 2020 May 23-29 2020 Seoul South Korea

Figure 6 t-SNE mapping of source and target features (WPDP)

the classes are unbalanced and the proportion of defective files is134 which is the least of all our projects This makes training ofthe deep model more difficult which leads to the poorer performanceof our method on some projects

Table 4 F-measure of DTL-DP DBN-CP (DBN) TCA+ DP-LSTM (LSTM) and DP-CNN (CNN) in CPDP

Source Target DBN TCA+ LSTM CNN DTL-DPant16 camel14 316 292 321 323 395

jEdit41 camel14 693 330 318 651 407camel14 ant16 979 616 448 607 591

poi30 ant16 478 598 386 532 693camel14 jEdit41 615 537 394 547 531log4j11 jEdit41 503 419 389 423 639jEdit41 log4j11 645 574 574 656 783

lucene22 log4j11 618 571 578 632 794lucene22 xalan25 550 530 680 540 689xerces13 xalan25 572 581 676 562 686xalan25 lucene22 594 561 750 621 783log4j11 lucene22 692 524 750 663 769xalan25 xerces13 386 394 340 391 400ivy20 xerces13 426 398 261 421 420

xerces13 ivy20 453 409 264 467 472synapse12 ivy20 824 383 261 371 494

ivy14 synapse11 489 348 451 491 545poi25 synapse11 425 376 435 436 597ivy20 synapse12 433 570 530 456 620poi30 synapse12 514 542 503 532 623

synapse12 poi30 661 651 785 671 827ant16 poi30 619 343 785 627 827

Average 568 479 495 528 618

52 RQ2 How does DTL-DP compare to feature-based Machine Learning and AST-based deeplearning methods in CPDP

Here we compare to TCA+ and DBN-CP instead of PROMISEand Semantic as the baseline approaches as explained in Sect 42

Again guided by prior work we conducted 22 sets of CPDP experi-ments In each we randomly select two project versions from twodifferent projects one as a training set and the other as a test set

Table 4 presents the F-measure results of DTL-CP and the 4 base-line approaches The highest F-measure values are in bold On aver-age the F-measure of our approach in CPDP is 0618 and the DBN-CP TCA+ DP-LSTM and DP-CNN achieve 0568 0479 0495 and0528 Thus DTL-DP outperforms them by 88 290 248 and155 respectively In addition we found that for projects log4j11and poi30 our method does better than the corresponding WPDPbest performing method

Our proposed DTL-DP shows significant improvements onthe state-of-the-art in cross-project defect prediction

521 Case Study CPDP Discrimination of DTL-DP Ourmethod is more obviously dominant in CPDP than in WPDP Apossible reason for that is that deep transfer learning makes the dis-tributions of the training and test samples more similar in featurespace Another reason might be the superior ability of deep modelsto represent features enabling the model to obtain more transferablefeatures from the images To gain more insight we choose the taskpoi30 rarr ant16 and show the t-SNE embeddings in Figure 7 Wemake similar observations as in the RQ1 case study that (1) the tar-get instances are more easily discriminated with our approach and(2) the target instances can be better discriminated with the sourceclassifier This implies that our approach can learn more transferablefeatures for more effective defect prediction

Figure 7 t-SNE mapping of source and target features (CPDP)

53 RQ3 How much does each of the threemechanisms ie data augmentation transferlearning and self-attention mechanismcontribute to DTL-DPrsquos performance

To find out the specific contributions of the three parts to defectprediction we conducted further experiments We built the originalAlexNet model for binary classification and use it as the Base +TL+Attention and +DataAug are three new baselines built by adding tothe base AlexNet one of three mechanisms (transfer learning selfattention and data augmentation) respectively It should be noted

9

ICSE 2020 May 23-29 2020 Seoul South Korea Chen et al

that in addition to +DataAug we use images generated by onesequence of R G and B as training and test sets in each experimentTherefore the experimental results of Base +TL and +Attention inTables 5 and 6 are average results obtained over the 6 different RGBpermutations as described above

Table 5 Contributions of the three mechanisms to WPDP

Project Version Base +TL +Atten +Aug DTL-DP

ant15-gt16 660 676 664 677 70016-gt17 525 552 524 591 593

camel12-gt14 415 381 416 403 40614-gt16 412 396 423 384 385

jEdit32-gt40 484 505 497 485 49440-gt41 629 649 645 679 687

log4j 10-gt11 688 679 692 658 688

lucene20-gt22 667 773 778 761 78322-gt24 749 713 769 758 776

xalan 24-gt25 757 754 758 778 794xerces 12-gt13 809 822 800 788 820

ivy 14-gt20 820 824 827 812 829

synapse10-gt11 525 484 511 530 54711-gt12 618 608 612 616 659

poi15-gt25 663 593 654 614 68525-gt30 410 449 420 432 426

Average 614 616 624 623 642

Table 6 Contributions of the three mechanisms to CPDP

Source Target Base +TL +Atten +Aug DTL-DPant16 camel14 383 379 365 377 395

jEdit41 camel14 368 394 395 371 407camel14 ant16 670 667 653 596 591

poi30 ant16 606 627 634 593 693camel14 jEdit41 663 657 652 647 531log4j11 jEdit41 599 599 592 594 639jEdit41 log4j11 754 800 774 746 783

lucene22 log4j11 759 750 753 783 794lucene22 xalan25 661 664 650 647 689xerces13 xalan25 510 638 626 662 686xalan25 lucene22 730 751 710 769 783log4j11 lucene22 713 745 740 727 769xalan25 xerces13 387 355 393 373 400ivy20 xerces13 413 408 398 378 420

xerces13 ivy20 518 500 504 500 472synapse12 ivy20 521 490 509 476 494

ivy14 synapse11 551 581 553 536 545poi25 synapse11 508 541 516 545 597ivy20 synapse12 604 622 619 611 620poi30 synapse12 600 632 613 612 623

synapse12 poi30 770 804 781 817 827ant16 poi30 777 662 700 812 827

Average 594 603 597 599 618

Table 7 Time cost of the three mechanisms defect visualization(Visualiz) self-attention (Attent) and transfer learning (TL)

ProjectTime (s)

TL Attent Visualizant 4504 131 459

camel 9655 242 785jEdit 5915 266 497log4j 1371 131 176

lucene 3739 186 354xalan 9444 271 823xerces 5117 214 275

ivy 2902 142 305synapse 2042 126 209

poi 3865 157 221Average 4856 187 410

To discern the contribution of each mechanism we compare theperformance of these baseline methods in WPDP and CPDP Fromthe results in Table 5 and Table 6 we observe that the three mecha-nisms each contribute substantially to the accuracy of DTL-DP inboth WPDP and CPDP In terms of the F-measure transfer learn-ing contributes the least improvement and self-attention and dataaugmentation contribute similarly to the final result But in CPDPtransfer learning contributes to the F-measure the most This islikely because of the applicability of transfer learning in the CPDPsetting and is in a way a validation of the approach

We also noted the time cost for the 3 mechanisms in our proposedDTL-DP Table 7 shows the result The most time spent during dataaugmentation is on converting code into images ie source codevisualization For transfer learning most time is spent on the MK-MMD calculation and for self-attention on the calculation of theattention layer in Fig 4 Eg for the project ant Table 3 shows twosets of WPDP experiments ant 15 rarr 16 and ant 16 rarr 17 Onaverage it takes 4504 seconds 1301 seconds and 459 seconds forthe 3 parts respectively for both the training data and the test dataTransfer learning takes the longest time more than the sum of theother two This is because of the large number of matrices neededto calculate MMD with the kernel function The least time cost isincurred by the attention mechanism and its contribution to WPDPis the largest of the three ie it is most cost-effective

The three mechanisms all contribute toward the accuracy ofour proposed end-to-end framework in WPDP and CPDP Self-attention has the greatest contribution to WPDP and transferlearning contributes most to CPDP

6 THREATS TO VALIDITYThreats to internal validity come from experimental errors and thereplication of the baseline methods In order to compare and analyzethe deep learning-based defect prediction techniques we compareour proposed DTL-DP method with Sementic DBN-CP DP-LSTMand DP-CNN In addition our method is also compared with thetransfer learning-based method TCA+ which is the state-of-the-art CPDP technique Since the original implementations were not

10

Software Visualization and Deep Transfer Learningfor Effective Software Defect Prediction ICSE 2020 May 23-29 2020 Seoul South Korea

available we re-implemented our own versions of the baselinesAlthough we strictly follow the procedures described in their workour new implementations may not completely restore all of theiroriginal implementation details And the randomness of the deeplearning-based approach also makes the results of our implementedexperiments different from the original Since we have removed theentries in the PROMISE dataset that cannot retrieve the correspond-ing source files our re-implemented experimental results may notbe consistent with the original baselines

The external threat to the validity of the results lies in the general-izability of the results We have tested our method on 10 open sourceJava projects including 14600 files Defect predictions for instancesof other languages such as C C++ etc need to be validated byadditional experiments in the future

Threats to construct validity depend on the appropriateness ofthe evaluation measurement The F-measure is used as our mainevaluation measure which has been applied in many previous effortsto evaluate defect prediction comprehensively

7 CONCLUSIONS AND FUTURE WORKHere we made two main contributions code visualization for defectprediction and an improved deep transfer learning model Our exper-imental results on 10 open source projects show that deep learningcan be effectively applied directly for defect prediction after apply-ing visualization methods to the code Specifically our approachDTL-DP performs at the top of the range of state-of-the-art WPDPapproaches For CPDP DTL-DP improves on the state-of-the-arttechnique TCA+ built on traditional features by 290 (in the F-measure) It also bests the deep learning-based approaches DBN-CPDP-LSTM and DP-CNN by 88 248 and 155 respectively

DTL-DP still has some limitations For some projects a problemof negative transfer occurs resulting in a worse prediction than adirect prediction Large differences between two projects can causesuch negative transfer Reducing the impact of negative transfer isone of the problems to be solved in the future Additionally theamount of training data we had was small and class imbalance isinherent in software defect prediction Both can be improved withmore data which we plan to obtain in the future

ACKNOWLEDGEMENTSThis work is partially supported by Zhejiang Natural Science Foun-dation(LY19F020025) National Natural Science Foundation ofChina (61502423 61572439 61702534) Zhejiang University OpenFund(2018KFJJ07) Signal Recognition Based on GAN Deep Learn-ing for Enhancement Recognition Project Zhejiang Science andTechnology Plan Project (LGF18F030009 2017C33149) and Zhe-jiang Outstanding Youth Fund (LR19F030001)

REFERENCES[1] Romi Satria Wahono A systematic literature review of software defect predic-

tion research trends datasets methods and frameworks Journal of SoftwareEngineering 1(1)1ndash16 2015

[2] Burak Turhan Tim Menzies Ayse B Bener and Justin Di Stefano On the relativevalue of cross-company and within-company data for defect prediction EmpiricalSoftware Engineering 14(5)540ndash578 2009

[3] Lech Madeyski and Marian Jureczko Which process metrics can significantlyimprove defect prediction models an empirical study Software Quality Journal23(3)393ndash422 2015

[4] Thomas Shippey David Bowes and Tracy Hall Automatically identifying codefeatures for software defect prediction Using ast n-grams Information andSoftware Technology 106142ndash160 2019

[5] Zhiyuan Wan Xin Xia Ahmed E Hassan David Lo Jianwei Yin and XiaohuYang Perceptions expectations and challenges in defect prediction IEEETransactions on Software Engineering 2018

[6] Karim O Elish and Mahmoud O Elish Predicting defect-prone software modulesusing support vector machines Journal of Systems and Software 81(5)649ndash6602008

[7] Puja Ahmad Habibi Victor Amrizal and Rizal Broer Bahaweres Cross-projectdefect prediction for web application using naive bayes (case study Petstoreweb application) In 2018 International Workshop on Big Data and InformationSecurity (IWBIS) pages 13ndash18 IEEE 2018

[8] Michael J Siers and Md Zahidul Islam Software defect prediction using a costsensitive decision forest and voting and a potential solution to the class imbalanceproblem Information Systems 5162ndash71 2015

[9] R Jayanthi and Lilly Florence Software defect prediction techniques using metricsbased on neural network classifier Cluster Computing pages 1ndash12 2018

[10] Yi Liu Yu Fan and Junghui Chen Flame images for oxygen content predictionof combustion systems using dbn Energy amp Fuels 31(8)8776ndash8783 2017

[11] Yi Liu Chao Yang Zengliang Gao and Yuan Yao Ensemble deep kernel learn-ing with application to quality prediction in industrial polymerization processesChemometrics and Intelligent Laboratory Systems 17415ndash21 2018

[12] Qi Xuan Binwei Fang Yi Liu Jinbao Wang Jian Zhang Yayu Zheng and Guan-jun Bao Automatic pearl classification machine based on a multistream convolu-tional neural network IEEE Transactions on Industrial Electronics 65(8)6538ndash6547 2018

[13] Qi Xuan Haoquan Xiao Chenbo Fu and Yi Liu Evolving convolutional neuralnetwork and its application in fine-grained visual categorization IEEE Access631110ndash31116 2018

[14] Qi Xuan Zhuangzhi Chen Yi Liu Huimin Huang Guanjun Bao and Dan ZhangMultiview generative adversarial network and its application in pearl classificationIEEE Transactions on Industrial Electronics 66(10)8244ndash8252 2019

[15] Qi Xuan Fuxian Li Yi Liu and Yun Xiang MV-C3D A spatial correlatedmulti-view 3d convolutional neural networks IEEE Access 792528ndash925382019

[16] Song Wang Taiyue Liu and Lin Tan Automatically learning semantic featuresfor defect prediction In Software Engineering (ICSE) 2016 IEEEACM 38thInternational Conference on pages 297ndash308 IEEE 2016

[17] Jian Li Pinjia He Jieming Zhu and Michael R Lyu Software defect predictionvia convolutional neural network In Software Quality Reliability and Security(QRS) 2017 IEEE International Conference on pages 318ndash328 IEEE 2017

[18] Hoa Khanh Dam Trang Pham Shien Wee Ng Truyen Tran John Grundy AdityaGhose Taeksu Kim and Chul-Joo Kim A deep tree-based model for softwaredefect prediction arXiv preprint arXiv180200921 2018

[19] Alex Krizhevsky Ilya Sutskever and Geoffrey E Hinton Imagenet classificationwith deep convolutional neural networks In Advances in neural informationprocessing systems pages 1097ndash1105 2012

[20] Mingsheng Long Yue Cao Jianmin Wang and Michael I Jordan Learning trans-ferable features with deep adaptation networks arXiv preprint arXiv1502027912015

[21] Han Zhang Ian Goodfellow Dimitris Metaxas and Augustus Odena Self-attention generative adversarial networks arXiv preprint arXiv1805083182018

[22] Tian Jiang Lin Tan and Sunghun Kim Personalized defect prediction In Pro-ceedings of the 28th IEEEACM International Conference on Automated SoftwareEngineering pages 279ndash289 IEEE Press 2013

[23] Andrew Meneely Laurie Williams Will Snipes and Jason Osborne Predictingfailures with developer networks and social network analysis In Proceedings ofthe 16th ACM SIGSOFT International Symposium on Foundations of softwareengineering pages 13ndash23 ACM 2008

[24] Raimund Moser Witold Pedrycz and Giancarlo Succi A comparative analysis ofthe efficiency of change metrics and static code attributes for defect prediction InProceedings of the 30th international conference on Software engineering pages181ndash190 ACM 2008

[25] Foyzur Rahman and Premkumar Devanbu How and why process metrics arebetter In Software Engineering (ICSE) 2013 35th International Conference onpages 432ndash441 IEEE 2013

[26] Maurice Howard Halstead et al Elements of software science volume 7 ElsevierNew York 1977

[27] Thomas J McCabe A complexity measure IEEE Transactions on softwareEngineering (4)308ndash320 1976

[28] Shyam R Chidamber and Chris F Kemerer A metrics suite for object orienteddesign IEEE Transactions on software engineering 20(6)476ndash493 1994

[29] Yue Yu Huaimin Wang Gang Yin and Tao Wang Reviewer recommendation forpull-requests in github What can we learn from code review and bug assignmentInformation and Software Technology 74204ndash218 2016

11

ICSE 2020 May 23-29 2020 Seoul South Korea Chen et al

[30] Nachiappan Nagappan and Thomas Ball Using software dependencies and churnmetrics to predict field failures An empirical case study In null pages 364ndash373IEEE 2007

[31] Ahmed E Hassan Predicting faults using the complexity of code changes InProceedings of the 31st International Conference on Software Engineering pages78ndash88 IEEE Computer Society 2009

[32] Thomas J Ostrand Elaine J Weyuker and Robert M Bell Programmer-basedfault prediction In Proceedings of the 6th International Conference on PredictiveModels in Software Engineering page 19 ACM 2010

[33] Taek Lee Jaechang Nam DongGyun Han Sunghun Kim and Hoh Peter In Microinteraction metrics for defect prediction In Proceedings of the 19th ACM SIG-SOFT symposium and the 13th European conference on Foundations of softwareengineering pages 311ndash321 ACM 2011

[34] Tim Menzies Zach Milton Burak Turhan Bojan Cukic Yue Jiang and AyseBener Defect prediction from static code features current results limitationsnew approaches Automated Software Engineering 17(4)375ndash407 2010

[35] Martin Pinzger Nachiappan Nagappan and Brendan Murphy Can developer-module networks predict failures In Proceedings of the 16th ACM SIGSOFTInternational Symposium on Foundations of software engineering pages 2ndash12ACM 2008

[36] Steffen Herbold Alexander Trautsch and Jens Grabowski A comparative studyto benchmark cross-project defect prediction approaches IEEE Transactions onSoftware Engineering 44(9)811ndash833 2018

[37] Chao Liu Dan Yang Xin Xia Meng Yan and Xiaohong Zhang A two-phasetransfer learning model for cross-project defect prediction Information andSoftware Technology 107125ndash136 2019

[38] Jinyin Chen Keke Hu Yitao Yang Yi Liu and Qi Xuan Col-lective transfer learning for defect prediction Neurocomputing 2019doihttpsdoiorg101016jneucom201812091

[39] Jinyin Chen Yitao Yang Keke Hu Qi Xuan Yi Liu and Chao Yang Multiviewtransfer learning for software defect prediction IEEE Access 78901ndash8916 2019

[40] Annibale Panichella Rocco Oliveto and Andrea De Lucia Cross-project defectprediction models Lrsquounion fait la force In Software Maintenance Reengineeringand Reverse Engineering (CSMR-WCRE) 2014 Software Evolution Week-IEEEConference on pages 164ndash173 IEEE 2014

[41] Burak Turhan Tim Menzies Ayse B Bener and Justin Di Stefano On the relativevalue of cross-company and within-company data for defect prediction volume 14pages 540ndash578 Springer 2009

[42] Fayola Peters Tim Menzies and Andrian Marcus Better cross company defectprediction In Proceedings of the 10th Working Conference on Mining SoftwareRepositories pages 409ndash418 IEEE Press 2013

[43] Xin Xia LO David Sinno Jialin Pan Nachiappan Nagappan and Xinyu WangHydra Massively compositional model for cross-project defect prediction IEEETransactions on software Engineering 42(10)977 2016

[44] Ying Ma Guangchun Luo Xue Zeng and Aiguo Chen Transfer learning forcross-company software defect prediction Information and Software Technology54(3)248ndash256 2012

[45] Jaechang Nam Sinno Jialin Pan and Sunghun Kim Transfer defect learningIn Software Engineering (ICSE) 2013 35th International Conference on pages382ndash391 IEEE 2013

[46] Sinno Jialin Pan Ivor W Tsang James T Kwok and Qiang Yang Domain adap-tation via transfer component analysis IEEE Transactions on Neural Networks22(2)199ndash210 2011

[47] Blaine A Price Ronald M Baecker and Ian S Small A principled taxonomy ofsoftware visualization Journal of Visual Languages amp Computing 4(3)211ndash2661993

[48] Thomas Ball and Stephen G Eick Software visualization in the large Computer29(4)33ndash43 1996

[49] Sarita Bassil and Rudolf K Keller Software visualization tools Survey andanalysis In Proceedings 9th International Workshop on Program ComprehensionIWPC 2001 pages 7ndash17 IEEE 2001

[50] Stephan Diehl Software visualization visualizing the structure behaviour andevolution of software Springer Science amp Business Media 2007

[51] Michele Lanza The evolution matrix Recovering software evolution using soft-ware visualization techniques In Proceedings of the 4th international workshopon principles of software evolution pages 37ndash42 ACM 2001

[52] Denis Gracanin Krešimir Matkovic and Mohamed Eltoweissy Software vi-sualization Innovations in Systems and Software Engineering 1(2)221ndash2302005

[53] Guillaume Langelier Houari Sahraoui and Pierre Poulin Visualization-basedanalysis of quality for large-scale software systems In Proceedings of the 20thIEEEACM international Conference on Automated software engineering pages214ndash223 ACM 2005

[54] Abien Fred Agarap and Francis John Hill Pepito Towards building an intelligentanti-malware system a deep learning approach using support vector machine(svm) for malware classification arXiv preprint arXiv180100318 2017

[55] Karl Weiss Taghi M Khoshgoftaar and DingDing Wang A survey of transferlearning Journal of Big Data 3(1)9 2016

[56] Rahul Krishna and Tim Menzies Bellwethers A baseline method for transferlearning IEEE Transactions on Software Engineering 2018

[57] Chuanqi Tan Fuchun Sun Tao Kong Wenchang Zhang Chao Yang and ChunfangLiu A survey on deep transfer learning In ICANN 2018

[58] Wenyuan Dai Qiang Yang Gui Rong Xue and Yong Yu Boosting for transferlearning In International Conference on Machine Learning pages 193ndash2002007

[59] Chang Wan Rong Pan and Jiefei Li Bi-weighting domain adaptation for cross-language text classification In IJCAI Proceedings-International Joint Conferenceon Artificial Intelligence volume 22 page 1535 2011

[60] Yonghui Xu Sinno Jialin Pan Hui Xiong Qingyao Wu Ronghua Luo HuaqingMin and Hengjie Song A unified framework for metric transfer learning IEEETrans Knowl Data Eng 29(6)1158ndash1171 2017

[61] Chetak Kandaswamy Luiacutes M Silva Luiacutes A Alexandre and Jorge M SantosDeep transfer learning ensemble for classification In Ignacio Rojas GonzaloJoya and Andreu Catala editors Advances in Computational Intelligence pages335ndash348 Cham 2015 Springer International Publishing

[62] Maxime Oquab Leon Bottou Ivan Laptev and Josef Sivic Learning and trans-ferring mid-level image representations using convolutional neural networks InProceedings of the IEEE conference on computer vision and pattern recognitionpages 1717ndash1724 2014

[63] Yann LeCun Leacuteon Bottou Yoshua Bengio and Patrick Haffner Gradient-basedlearning applied to document recognition Proceedings of the IEEE 86(11)2278ndash2324 1998

[64] Hana Ajakan Pascal Germain Hugo Larochelle Franccedilois Laviolette and MarioMarchand Domain-adversarial neural networks arXiv preprint arXiv141244462014

[65] Yaroslav Ganin and Victor Lempitsky Unsupervised domain adaptation by back-propagation arXiv preprint arXiv14097495 2014

[66] Jing Zhang Wanqing Li and Philip Ogunbona Joint geometrical and statisticalalignment for visual domain adaptation arXiv preprint arXiv170505498 2017

[67] Eric Tzeng Judy Hoffman Ning Zhang Kate Saenko and Trevor DarrellDeep domain confusion Maximizing for domain invariance arXiv preprintarXiv14123474 2014

[68] Mingsheng Long Yue Cao Zhangjie Cao Jianmin Wang and Michael I Jor-dan Transferable representation learning with deep adaptation networks IEEEtransactions on pattern analysis and machine intelligence 2018

[69] Arthur Gretton Dino Sejdinovic Heiko Strathmann Sivaraman BalakrishnanMassimiliano Pontil Kenji Fukumizu and Bharath K Sriperumbudur Optimalkernel choice for large-scale two-sample tests In Advances in neural informationprocessing systems pages 1205ndash1213 2012

[70] Mingsheng Long Han Zhu Jianmin Wang and Michael I Jordan Deep transferlearning with joint adaptation networks arXiv preprint arXiv160506636 2016

[71] Martin Arjovsky Soumith Chintala and Leacuteon Bottou Wasserstein gan arXivpreprint arXiv170107875 2017

[72] Kesav Kancherla and Srinivas Mukkamala Image visualization based malwaredetection In 2013 IEEE Symposium on Computational Intelligence in CyberSecurity (CICS) pages 40ndash44 IEEE 2013

[73] Xin Zhou Jianmin Pang and Guanghui Liang Image classification for malwaredetection using extremely randomized trees In 2017 11th IEEE InternationalConference on Anti-counterfeiting Security and Identification (ASID) pages54ndash59 IEEE 2017

[74] John Murphy An overview of convolutional neural network architectures for deeplearning 2016

[75] Song Wang Taiyue Liu Jaechang Nam and Lin Tan Deep semantic feature learn-ing for software defect prediction IEEE Transactions on Software Engineering2018

[76] Laurens van der Maaten and Geoffrey Hinton Visualizing data using t-sneJournal of machine learning research 9(Nov)2579ndash2605 2008

12

• Abstract
• 1 Introduction
• 2 Related Work
• 3 Approach
• • 31 Source Code Visualization
  • 32 Modeling with Deep Transfer Learning
  • 33 Model Sensitivity to Parameter Choices
  • 34 Training and Prediction
  • • 4 Experimental Setup
    • • 41 Dataset Description
      • 42 Baseline Comparison Methods
      • 43 Performance measures
      • • 5 Results
        • • 51 RQ1 How does DTL-DP compare to feature- based machine learning methods and AST- based deep learning methods in WPDP
          • 52 RQ2 How does DTL-DP compare to feature- based Machine Learning and AST-based deep learning methods in CPDP
          • 53 RQ3 How much does each of the three mechanisms ie data augmentation transfer learning and self-attention mechanism contribute to DTL-DPs performance
          • • 6 Threats to Validity
            • 7 Conclusions and Future Work
            • References

Software Visualization and Deep Transfer Learningfor Effective Software Defect Prediction ICSE 2020 May 23-29 2020 Seoul South Korea

nearest neighbor methods to train the prediction models Addition-ally the application of transfer learning is receiving more attentionXia et al [43] proposed an approach named HYDRA to build classi-fiers using genetic algorithm and ensemble learning Ma et al [44]proposed a transfer naive Bayes (TNB) method to assign weightsto training instances and then used them to construct a predictionmodel Nam et al [45] proposed TCA+ which uses TCA [46] andoptimizes the normalization to improve CPDP

Recently some studies have used deep learning to directly get fea-tures from source code for DP Wang et al [16] deployed deep beliefnetworks (DBNs) to learn semantic features from token vectors ex-tracted from ASTs automatically and leveraged the learned semanticfeatures to build machine learning models for DP Li et al [17] usedconvolutional neural networks (CNNs) to generate features fromsource code and combined CNN learned features with traditionalfeatures to further improve upon the prediction Similarly Dam etal [18] proposed a prediction model that takes ASTs representingthe source file as input It used an LSTM architecture to capturelong-term dependencies which often exist between code elements

Our proposed approach differs from the above in that we donot use feature engineering followed by machine learning-basedclassifiers We also do not need ASTs to bridge the gap betweendefect prediction and deep learning Instead our approach is basedon feeding software visualizations into an automatic image basedfeature discovery pipeline

Software visualization (SV) for DP SV has long history in soft-ware engineering research and practice [47 48] It has been usedfor visualizing code structure and features [49] code execution [50]and evolution [51] of large codebases in particular [52] SV hasalso been applied to bug repositories where it has been helpfulin correlating bugs with code structure [53] In fact the way codewas visualized in a recent study on malware code visualization [54]has in part inspired us in this paper However to the best of ourknowledge there is a dearth of applications of SV to software defectprediction in the research literature perhaps because prior to DLapproaches no connection was seen between the two areas

Deep transfer learning (DTL) Transfer learning (TL) is an im-portant tool that can help when there is insufficient training datain machine learning It works by transferring information from asource domain to a target domain The key is to relax the assumptionthat the training and test data must be independent and identicallydistributed [55] This is helpful in areas where it is difficult to getenough training data In the field of DP studies have shown thatCPDP can achieve better performance with transfer learning [56]

Most TL studies are based on traditional machine learning meth-ods Recently deep learning-based transfer learning studies haveemerged called deep transfer learning DTL Based on the tech-niques used in them they can be divided into four categories network-based instance-based mapping-based and adversarial-based [57]

Instance-based DTL is implemented through a specific weightadjustment strategy The instances selected from the source domainare assigned weights that complement the training data in the targetdomain [58ndash61] Network-based DTL refers to the use of a partiallytrained network in the source domain for the deep neural networkof the target domain including its network structure and connecting

parameters [62 63] Adversarial-based DTL refers to introducing ad-versarial technology inspired by generative adversarial nets (GAN)to find a transferable representation that is applicable to the sourceand target domains [64 65] Mapping-based DTL refers to map-ping instances of the source and target domains to a new data spacewhere the distributions of the two domains are similar TCA [46]and TCA-based methods have been widely used in applications oftraditional transfer learning [66] Tzeng et al [67] used maximummean discrepancy (MMD) to measure the sample distribution af-ter deep neural network processing and learned domain invariantrepresentations by introducing an adaptation layer and additionaldomain confusion loss Long et al [68] improved previous work byreplacing MMD with multiple kernel variant MMD (MK-MMD)originally by [69] and proposed a method called deep adaptationnetworks (DAN) Long et al [70] proposed joint maximum meandiscrepancy to promote the transfer learning ability of neural net-works to adapt to the data distribution of different domains TheWassersteinrsquos distance proposed by Arjovsky et al [71] has beenused as a new distance measure to find a better mapping betweendomains

To the best of our knowledge DTL methods have not yet beenused in defect prediction In this paper we propose a novel mapping-based DTL method using a self-attention mechanism for defectprediction

3 APPROACHThe overall framework of our approach deep transfer learning fordefect prediction DTL-DP is shown in Figure 2 It is comprised oftwo stages (1) source code visualization and (2) modeling with DTLIn the first stage we use a visualization method to convert programfiles into images In the second stage we build a DTL model basedon the AlexNet network structure with transfer learning and a self-attention mechanism to construct an end-to-end defect predictionframework We use that framework to predict if a new instance fileis defective or not

In a nutshell our approach takes the raw program files of a train-ing and test sets directly as input and generates images from themwhich are then used to build the evaluation model for defect predic-tion Specifically since the input data of the network model basedon the CNN structure should be in the form of images we builda mapping to convert the files into images Then we use the first5 layers of AlexNet as Feature-Net to generate features from theimages The shallow CNN layers correlate the presence and absenceof defects to the overall code structure gradually deepening thegranularity from functionloop bodies to function names identifiersetc These features are then fed into the Attention Layer where theyare used in assigning weights and highlighting features which aremore helpful in the classification The re-weighted features of thetraining and test sets are used to calculate MK-MMD which is usedto measure the difference between their distributions as the MMDloss After that the re-weighted features of the training set are en-tered into the fully connected layers to calculate the cross entropyas the classification loss The weighted sum of the MMD loss andclassification loss is fed back to train the whole network includingFeature-Net Attention Layer and the fully connected layers Finallybased on the source code visualization method and the DTL model

3

ICSE 2020 May 23-29 2020 Seoul South Korea Chen et al

DataAugmentation

B G R

GRB

R B G

Feature- Net

AttentionLayerBGR

G R B

RBG

Clean Buggy

Clean Buggy

SourceProject

TargetProject

MMD

Clean

Buggy

Clean

Buggy

Defect Visualization Model Building Phase

Prediction Phase

Figure 2 The overall framework of DTL-DP

Figure 3 The process of converting code to color images

we build train and evaluate a defect prediction model We give thedetails in the following

31 Source Code VisualizationConverting code to images retains information and facilitates avail-able deep neural network (DNN) -based methods to improve defectprediction performance Most existing methods ignore associationsbetween extracted features and the classification task Our proposedmethod forms a unified end-to-end framework of feature extractionmodeling and prediction

Figure 3 shows how we convert each program file into 6 differentimages We call this process source code visualization and the im-ages produced code images First each source file is converted into avector of 8-bit unsigned integers corresponding to the ASCII decimalvalues of the characters in the source code eg rsquoarsquo is converted to97 etc We then generate an image from that vector by arrangingits values in rows and columns and interpreting it as a rectangularimage A straight forward way to visualize the 8-bit values as colorintensities would be as levels of gray eg 0=black and 255=whiteHowever the relatively small size of our original training set meansthat if we did that we would end up with a deep model that cannot besufficiently well trained Fortunately an additional benefit to using

images for training is that we can augment the original data set toproduce a larger data set with the same semantics Common dataaugmentation methods for image data sets include flipping (bothvertically and horizontally) rotating cropping translating (movingalong the x or y axis) adding Gaussian noise (distortion of highfrequency features) zooming and scaling Since the sizes of our pro-grams are small and we want to retain the semantic and structuralfeatures in the images the above image data augmentation methodscould result in data loss

Instead we designed a novel color based augmentation methodto generate 6 color images from each source code file Namelyeach pixel in a color image can be represented by three primarycolor components or channels red (R) green (G) and blue (B)The letters R G B and thus the color channels can be orderedin 6 different ways RBG RGB BGR BRG GRB and GBR Byadopting a different order every time as shown in Figs 2 and 3 wegenerate six different images for each program file For examplersquoforrsquo is converted to [102 111 114] and the 3 values are filled intothe R G and B channels in 6 different ways to obtain 6 differentlycolored pixels

The above method expands our data set six-fold and because ofthe nature of the downstream analysis the generated samples arereasonable since the representation is changed only by the order ofthe different channels 2

Whereas we generated 6 images for each instance in the trainingset we randomly selected an RGB permutation for testing We foundthat it was not necessary to generate all six images for testing becauseour experiments on 10 datasets showed that the performance of usingthe different permutations was quite comparable We randomly choseone to increase the speed in practice The data augmentation wasused to improve the efficacy of the model

After the above for each of the 6 orderings of R G B we obtaina vector of pixels of length one third the original code file size(each source code character is assigned to a color channel and thusthree characters in a row represent a pixel) We then turned those

2A CNN model learns the features by convolution Since we use ImageNetrsquos pre-trainedAlexNet model the initial parameters of the convolution kernel are different for eachchannel which means even though byte sequences are the same the final set of featuresobtained by sequentially inputting into the model the different channels is also different

4

Software Visualization and Deep Transfer Learningfor Effective Software Defect Prediction ICSE 2020 May 23-29 2020 Seoul South Korea

Figure 4 The training and testing process of our approach DTL-DP inspired by the original DAN paper [68]

Table 1 Image Width for Various File Sizes

File Size Range Image Width

lt10 kB 3210 kB - 30 kB 6430 kB - 60 kB 12860 kB - 100 kB 256

100 kB - 200 kB 384200 kB - 500 kB 512500 kB - 1000 kB 768

gt1000kB 1024

6 image vectors into image matrices of width and length such thatwidth lowast lenдth = vector_size3 We note that the first convolutionkernel of AlexNet is of size 1111 so if the image is too narrowthe convolution performance will be poor And the image with asuitable width can be convolved to obtain more efficient semanticand structural features in contexts with a proper sequence lengthTable 1 gives some recommended image widths for different filesizes based on previous work [72 73] We adopt those in this work

32 Modeling with Deep Transfer LearningNext we describe our approach DTL-DP The goal is to learn trans-ferable features across projects and build a classifier y = ϕ(x) offile defectiveness with source code supervision where x is the rep-resentation of the sample and y is the predicted label In the defectprediction problem we are given a source project Ps = (xsi y

si )

nsi=1

with ns labeled instances and a target project Pt = (xti yti )

nti=1

with nt unlabeled instances In WPDP the source project is the pre-vious version of the target project while in CPDP the source projectis a related project of the target project The sample distributions pand q in the source and target projects are often similar in WPDPbut different in CPDP

For our deep network architecture we adopt a similar architec-ture as that of deep adaptation networks (DANs) [68] to capturethe semantic and structural information of the source code To theoriginal DAN model we add an attention layer to further enhancethe expressive ability of important features The overall architectureis illustrated in Figure 4 In particular our DTL-DP consists of aninput layer five convolution layers (conv1 minus conv5 from AlexNet)an Attention-Layer and finally four fully-connected hidden layers(f c6 minus f c9) working as a classifier The structure and parameters of

(conv1minusconv5) and (f c6minus f c8) are consistent with a DAN But sincedefect prediction is a binary classification problem a fully-connectedlayer (f c9) is added to obtain a binary result at the end

We adopt the defaults for AlexNet so the input to our DTL-DPmust be a 224224 size image cropped from a three-channel (RGB)image of size 256256 The code image is placed in the approximatecenter of the 256256 image (the determination of the size of eachcode image is described later in Sect 33) We note that the codeimage can be smaller than 224224 pixels due to the variance in thesize of the code files If this happens the image is padded aroundwith blank (zero valued) pixels Padding should not negatively effectthe qualitative performance of feature detection in the images in factfor deeper DL architectures like ours padding has been shown toprovide extra contrast to the embedded image for each of the layersas well as buffering against data loss by each layer [74]

Training deep models requires a significant amount of labeleddata which is not available for a new project so we first pre-trainedan AlexNet model on ImageNet 2012 Unlike a DAN which freezesconv1minusconv3 and fine-tunes conv4minusconv5 we fine tune all convolu-tion layers conv1minusconv5 by taking the parameters of the pre-trainedmodels as initial parameters that we then optimize during the train-ing phase We do that to minimize the differences between our codeimages and the actual object images in ImageNet 2012

In order for DTL-DP to focus on the key features in differentdefect images and thus further improve prediction we employ aself-attention mechanism into our model inspired by the good per-formance of self-attention in GANs [21] As shown in the AttentionLayer in Figure 4 the attention mechanism makes the feature mapgenerated by layer conv5 be the self-attention feature map input tothe next layer fc6 Specifically at first it linearly maps the inputfeatures x (it is a 11 convolution to compress the number of chan-nels ie out_chanels = in_chanels8) and produces f (x)д(x)and h(x) where f (x) = wf x д(x) = wдx h(x) = whx The dif-ference between the three is that the size of h(x) is still the sameas x but the other two are not Thus if the width of x is W theheight H and the number of channelsC the size of x is [CN ] whereN =W lowast H the size of f(x) and g(x) is [C8N ] but the size of h(x)is [CN ] The transposed f (x) and д(x) are matrix-multiplied to ob-tain the autocorrelation in the features ie the relationship of eachpixel to all other pixels where Si j = f (xi )

Tд(x j ) Then softmax isapplied to the autocorrelation features S to get the attention mapcomprised of weights with values between 0 and 1

α ji =exp(Si j )sumNi=1 exp(Si j )

(1)

5

ICSE 2020 May 23-29 2020 Seoul South Korea Chen et al

005 01 02 05 1 2 5MMD Penalty

055

056

057

058

059

060

061

062

063

064

065

F-m

easu

re

WPDP CPDP

(a) Effect of different MMD Penalty λ

BRG BGR GRB GBR RGB RBG040

045

050

055

060

065

070

075

F-m

easu

re

WPDP CPDP

(b) Effect of color channel order

8 4 2 1 12 14 18Width recommended

040

045

050

055

060

065

070

075

F-m

easu

re

WPDP CPDP

(c) Effect of code image widthrecommended ratio

Figure 5 Sensitivity studies to aid in choosing the parameters in the model

After that the output of the AttentionLayer is the self-attentionfeature map o = (o1o2 oj oN ) where

oj =Nsumi=1

α jih(xi ) (2)

Then each fully connected layer learns a nonlinear mappinghli = f l (wlhli + b

l ) where hli is the l th layer hidden representationof feature xi wl and bl are the weights and biases of the l th layerand f l is the activation function using ReLU (f l (o) = max(0o))for f c6 minus f c8 and softmax for f c9 If we let Θ denote the set of allDTL-DP parameters the empirical risk of DTL-DP then is

minΘ

1n

nsumi=1

F (ϕ(oi )yi ) (3)

where F is the cross-entropy loss function n is the number of in-stances and ϕ(oi ) is the conditional probability that the DTL-DPassigns oi to label yi It is used to calculate the final loss in Equation7

To make the distribution of the source and target projects similarthe same multi-layer adaptation and multi-kernel MMD strategy asin a DAN [68] are used in our model The feature mapping functionσ is defined as the combination ofm positive semi-definite kernelsku

lt σ (os )σ (ot ) gt= k(os ot ) (4)

k =msumu=1

βuku (5)

where β ge 0 are the weights of the kernelsThe MK-MMD dk (pq) of the probability distributions p and q

is defined as the reproducing kernel Hilbert space distance betweenthe mean embedding of p and q The square formula of MK-MMDis defined as

d2k (pq) =∥ Ep [σ (xs )] minus Eq [σ (x

t )] ∥2 (6)

We see then that the smaller dk (pq) is the more similar p and qare If dk is 0 the distribution of the target project is the same as thatof the source project So the final loss function for train DPL-DP is

1n

nsumi=1

F (ϕ(oi )yi ) + λ

l2suml=l1

d2k (Dls D

lt ) (7)

where Dls is the l th layer hidden representation for the source and

target d2k (Dls D

lt ) is the MK-MMD between the source and target

evaluated on the l th layer representation λ is a penalty parameterand l1 and l2 are f c layers to calculate the MK-MMD between sourceand target In our implementation of DTL-DP we set λ = 1 l1 = 6and l2 = 9 as per the original DAN work [68]

33 Model Sensitivity to Parameter ChoicesWe have made a number of choices to make our modeling plat-form work effectively Here we justify those choices by presentingsensitivity studies For this analysis we used all the data

Hyperparameter λ The DTL-DP model has a hyper-parameter λfor the MMD penalty that adjusts the final loss We conducted λ pa-rameter sensitivity experiments in both the WPDP and CPDP settingWe fix the other parameters and range λ in 001 005 01 02 05 1 2 5The results are shown in Figure 5(a) While fairly constant across therange small variations exist In CPDP the F-measure first increasesand then decreases along the hyper-parameter λ In WPDP λ affectsthe performance of DTL-DP but in the opposite direction We choseλ = 1 here

Different Color Orders The selection of the RGB permutationsin the target project images may also potentially affect the outcomeWe performed experiments with all six different orderings Whentesting them we set λ to 1 and changed the image type of the tar-get project to each one in RGBRBGBRGBGRGBRGRB Theresults are shown in Figure 5(b) The target images in different colororders have somewhat different performance But overall the valuesare close to DTL-DPrsquos average performance

Different Image Widths To explore the impact of different choicesfor image widths on the results we performed additional experi-ments

For our code images 3 bytes of source code are needed to form 1pixel each byte for one of the R G and B channels The sizes ofthe source files are between 0 and 100 kb corresponding to imageswith pixel count between 0 and 34 133 (10010243) The size ofthe image is (widthheiдht) where width is obtained from Table 1according to the size of the code file (size)

heiдht =size lowast 10243 lowastwidth

(8)

For example the size of image converted from a 20 kb code file is(64 107)

We experimented with image widths of 18 14 12 2 4 and 8times of the recommended widths in Table 1 The larger the mul-tiplier the wider the image For images of different widths we

6

Software Visualization and Deep Transfer Learningfor Effective Software Defect Prediction ICSE 2020 May 23-29 2020 Seoul South Korea

repeated the previous experiment and obtained results for the meanF-measure shown in Figure 5(c) We see that the closer to the recom-mended width from Table 1 the better the average result In additionwe found that a wider image is better than a narrower one And thecloser the image is to a square the better the prediction Thereforewe made the generated images be as close to a square as possible

34 Training and PredictionWhen training the DTL-DP code images are generated from thelabeled instances in the source project and the unlabeled instances inthe target project and then are simultaneously input into the modelThey share the convolution layers conv1minusconv5 andAttentionLayerto extract their respective features and calculate the MK-MMDbetween source and target projects in the fully connected layersf c6 minus f c9 It should be noted that we only calculate cross entropyfor the source project because the target projectrsquos labels are not pro-vided We use a mini-batch stochastic gradient descent along the lossfunction in order to train the parameters of the entire model We trainfor 500 epochs for each pair of source and target projects then pickthe epoch with the best F-measure (described in section 43) fromwhich to read out the parameters of the final model Finally the filesfrom the target project that need to be predicted are converted intocode images and then input into the trained model for classification

Overfitting is always a possibility with such pipelines We moder-ate it here with our choices of (1) augmenting the dataset by using sixcolor channel permutations described in section 31 (2) selecting asimple AlexNet model structure and (3) using the ImageNet 2012pre-trained model to reduce fluctuations

4 EXPERIMENTAL SETUPWe conducted experiments to asses the performance of DTL-DP andto compare it with existing deep learning-based defect predictionapproaches for both within-project WPDP and cross-project CPDPdefect prediction We ran experiments on a modern-day Linux serverwith 3 Titan XP GPUs Unless otherwise stated each experimentwas run 10 times and the average results are reported

41 Dataset DescriptionIn order to directly compare our work with prior research we usedpublicly available data from the PROMISE3 data repository whichhas been widely used in defect prediction work [16ndash18 75] Weselected all open source Java projects from this repository andcollected their version number class names and the buggy labelfor each file Based on the version number and class name weobtained the source code for each file from GitHub4 and fed it toour end-to-end framework In total data for 10 Java projects werecollected Table 2 shows the details of these projects includingproject description versions the total and average number of filesand the defect rate It should be noted that the average numberof files over all projects ranges between 150 and 1046 and thedefect rates of the projects have a minimum value of 134 and amaximum value of 497 The number of files in some projects isnot sufficient to train deep models and the classes are imbalancedthus augmentation is needed as described above

3httpopenscienceusrepodefect4httpsgithubcomapache

42 Baseline Comparison MethodsTo evaluate the performance of our end-to-end framework DTL-DPfor defect prediction we compare it with the following baselinemethods in the WPDP setting

bull Semantic [16 75] the state-of-the-art method which em-ploys deep belief networks DBN on source code to extractsemantic features for defect prediction

bull PROMISE-DP [16] a traditional method which builds analternating decision tree ADTree classifier based on theoriginal 20 features of the PROMISE dataset

bull DP-LSTM [18] a long short-term memory LSTM-baseddeep neural network model which uses ASTs to representsource files and predict whether the file is defective or not

bull DP-CNN [17] a convolutional neural network CNN-basedmodel which is seeded by AST-derived numerical vectorsto automatically learn semantic and structural features ofprograms The CNN-learned features are used to train thefinal classifier in combination with traditional features

For the cross-project settings CPDP Semantic and PROMISE-DPcould not be used directly Instead we used the following

bull DBN-CP [16 75] a variant of Semantic which trains a DBNby using the source project and generates semantic featuresfor both the source and target projects

bull TCA+ [45] the state-of-the-art technique for CPDP

To obtain the training and test data we followed the processesestablished in [16] For WPDP we use two consecutive versions ofeach project listed in Table 2 The older version is used to generatethe training data and the more recent version is used as test data ForCPDP we pick versions randomly from each project for 11 targetprojects And for each target project we select 2 source projects thatare different from the target projects We use the same 22 test pairsas in [16] When implementing the baseline methods we use thesame network architecture and parameter settings as described in thepapers that introduced them

43 Performance measuresTo evaluate the prediction performance we use the F-measure awidely adopted metric in the literature [16ndash18 43 45] The F-measure captures a predictorrsquos accuracy and combines both precisionand recall for a comprehensive evaluation of predictive performance

Specifically a prediction that a file is defective is called a truepositive (TP) if the file is in fact defective and false positive (FP)otherwise Similarly a prediction that a file is not defective is a truenegative (TN) if the file is in fact not defective and false negative(FN) otherwise Then the precision (P) recall (R) and F-measureare defined as

P = TP(TP + FP) (9)

R = TP(TP + FN ) (10)

F = (2 times P times R)(P + R) (11)

5 RESULTSThis section discusses our results of comparing DTL-DP to baselinetools for defect prediction

7

ICSE 2020 May 23-29 2020 Seoul South Korea Chen et al

Table 2 Dataset Description

Project Description Versions Files Avg files Avg size(kb) Defective

ant Java based build tool 151617 1465 488 62 134camel Enterprise integration framework 121416 3140 1046 29 187jEdit Text editor designed for programmers 324041 1935 645 87 192log4j Logging library for Java 1011 300 150 34 497

lucene Text search engine library 202224 607 402 38 358xalan A library for transforming XML files 2425 1984 992 46 296xerces XML parser 1213 1647 549 29 157

ivy Dependency management library 1420 622 311 41 200synapse Data transport adapters 101112 661 220 38 227

poi Java library to access Microsoft format files 152530 1248 416 36 407

Table 3 F-measure of DTL-DP Semantic (Seman) PROMISE-DP (PROM) DP-LSTM (LSTM) and DP-CNN (CNN) forWPDP

Project Version Seman PROM LSTM CNN DTL-DP

ant15-gt16 914 477 452 532 70016-gt17 942 542 379 566 593

camel12-gt14 785 373 323 461 40614-gt16 374 391 400 508 385

jEdit32-gt40 574 556 472 564 49440-gt41 615 546 490 580 687

log4j 10-gt11 701 587 539 632 688

lucene20-gt22 651 502 751 761 78322-gt24 773 605 751 721 776

xalan 24-gt25 595 518 657 601 794xerces 12-gt13 411 238 268 374 820

ivy 14-gt20 350 329 191 347 829

synapse10-gt11 544 476 454 539 54711-gt12 583 533 533 556 659

poi15-gt25 640 558 808 589 68525-gt30 803 754 777 784 426

Average 641 499 515 570 642

51 RQ1 How does DTL-DP compare to feature-based machine learning methods and AST-based deep learning methods in WPDP

We compare DTL-DP to 4 baseline approaches representing twodifferent kinds of defect prediction methods PROMISE-DP is thebaseline representative of traditional feature-based machine learningmethods Semantic DP-LSTM and DP-CNN are the baselines fordeep learning-type methods based on extracting features from ASTGuided by prior work we conducted 16 sets of WPDP experimentseach using two versions of the same project The older version isused to train the prediction model and the newer version is used toevaluate the trained model

Table 3 shows the F-measure values for the within-project WPDPdefect prediction experiments The highest F-measure values of the5 methods are shown in bold Since the methods based on deeplearning include some randomness we run DTL-DP Semantic DP-LSTM and DP-CNN 10 times for each experiment On average

the F-measure of our approach is 0642 and the PROMISE-DPSemantic DP-LSTM and DP-CNN achieve 0499 0641 0515 and0570 respectively The results demonstrate that our approach iscompetitive and may improve on defect prediction compared toPROMISE-DP DP-LSTM and DP-CNN The results of Semanticand our approach are similar

The proposed DTL-DP is effective and it could improvethe performance of WPDP tasks Like other deep learners it issensitive to small file sizes and unbalanced data

511 Case Study WPDP Discrimination of DTL-DP t-SNEis a non-linear dimensional reduction algorithm that is effectivein visualizing similarities and helping identify clusters in complexdata sets [76] To give insight in the performance of DTL-DP wedemonstrate feature transferability by showing t-SNE embeddingsin Figure 6 The blue points are non-defective files and the red aredefective ones5 We observe the following (1) The target instancesare not discriminated very well using either the traditional manualfeatures or the TCA+ improved manual features or the semanticfeatures extracted from ASTs while with our approach the pointsare discriminated much better (2) With the other three approachesthe categories between source and target projects are not well alignedwhile with our approach the categories between the projects aremore consistent These conclusions are derived from the intra- andinter-class distances of the two categories in Figures 6 and 7 Theyare visually apparent

Our method did not perform as well as the comparison methods onsome projects such as ant which is likely caused by large variancein file sizes While the sizes of ant-15 and ant-16 are close there isa marked difference between ant-16 and ant-17 the former beingmuch smaller than the latter From Table 3 the performance of ourmethod on ant15-gtant16 is better than that on ant16-gtant17On the contrary baseline methods other than DP-LSTM performbetter for the task ant16-gtant17 notably Semantic is dominantindicating that the semantic feature-based method is more robust tofile size variability Moreover the ant dataset has two shortcomingsthe first is that the amount of data is small cf Table 2 where theaverage amount is only 488 files in one project The second is that

5Since we use data augmentation we have more samples than the three baselines

8

Software Visualization and Deep Transfer Learningfor Effective Software Defect Prediction ICSE 2020 May 23-29 2020 Seoul South Korea

Figure 6 t-SNE mapping of source and target features (WPDP)

the classes are unbalanced and the proportion of defective files is134 which is the least of all our projects This makes training ofthe deep model more difficult which leads to the poorer performanceof our method on some projects

Table 4 F-measure of DTL-DP DBN-CP (DBN) TCA+ DP-LSTM (LSTM) and DP-CNN (CNN) in CPDP

Source Target DBN TCA+ LSTM CNN DTL-DPant16 camel14 316 292 321 323 395

jEdit41 camel14 693 330 318 651 407camel14 ant16 979 616 448 607 591

poi30 ant16 478 598 386 532 693camel14 jEdit41 615 537 394 547 531log4j11 jEdit41 503 419 389 423 639jEdit41 log4j11 645 574 574 656 783

lucene22 log4j11 618 571 578 632 794lucene22 xalan25 550 530 680 540 689xerces13 xalan25 572 581 676 562 686xalan25 lucene22 594 561 750 621 783log4j11 lucene22 692 524 750 663 769xalan25 xerces13 386 394 340 391 400ivy20 xerces13 426 398 261 421 420

xerces13 ivy20 453 409 264 467 472synapse12 ivy20 824 383 261 371 494

ivy14 synapse11 489 348 451 491 545poi25 synapse11 425 376 435 436 597ivy20 synapse12 433 570 530 456 620poi30 synapse12 514 542 503 532 623

synapse12 poi30 661 651 785 671 827ant16 poi30 619 343 785 627 827

Average 568 479 495 528 618

52 RQ2 How does DTL-DP compare to feature-based Machine Learning and AST-based deeplearning methods in CPDP

Here we compare to TCA+ and DBN-CP instead of PROMISEand Semantic as the baseline approaches as explained in Sect 42

Again guided by prior work we conducted 22 sets of CPDP experi-ments In each we randomly select two project versions from twodifferent projects one as a training set and the other as a test set

Table 4 presents the F-measure results of DTL-CP and the 4 base-line approaches The highest F-measure values are in bold On aver-age the F-measure of our approach in CPDP is 0618 and the DBN-CP TCA+ DP-LSTM and DP-CNN achieve 0568 0479 0495 and0528 Thus DTL-DP outperforms them by 88 290 248 and155 respectively In addition we found that for projects log4j11and poi30 our method does better than the corresponding WPDPbest performing method

Our proposed DTL-DP shows significant improvements onthe state-of-the-art in cross-project defect prediction

521 Case Study CPDP Discrimination of DTL-DP Ourmethod is more obviously dominant in CPDP than in WPDP Apossible reason for that is that deep transfer learning makes the dis-tributions of the training and test samples more similar in featurespace Another reason might be the superior ability of deep modelsto represent features enabling the model to obtain more transferablefeatures from the images To gain more insight we choose the taskpoi30 rarr ant16 and show the t-SNE embeddings in Figure 7 Wemake similar observations as in the RQ1 case study that (1) the tar-get instances are more easily discriminated with our approach and(2) the target instances can be better discriminated with the sourceclassifier This implies that our approach can learn more transferablefeatures for more effective defect prediction

Figure 7 t-SNE mapping of source and target features (CPDP)

53 RQ3 How much does each of the threemechanisms ie data augmentation transferlearning and self-attention mechanismcontribute to DTL-DPrsquos performance

To find out the specific contributions of the three parts to defectprediction we conducted further experiments We built the originalAlexNet model for binary classification and use it as the Base +TL+Attention and +DataAug are three new baselines built by adding tothe base AlexNet one of three mechanisms (transfer learning selfattention and data augmentation) respectively It should be noted

9

ICSE 2020 May 23-29 2020 Seoul South Korea Chen et al

that in addition to +DataAug we use images generated by onesequence of R G and B as training and test sets in each experimentTherefore the experimental results of Base +TL and +Attention inTables 5 and 6 are average results obtained over the 6 different RGBpermutations as described above

Table 5 Contributions of the three mechanisms to WPDP

Project Version Base +TL +Atten +Aug DTL-DP

ant15-gt16 660 676 664 677 70016-gt17 525 552 524 591 593

camel12-gt14 415 381 416 403 40614-gt16 412 396 423 384 385

jEdit32-gt40 484 505 497 485 49440-gt41 629 649 645 679 687

log4j 10-gt11 688 679 692 658 688

lucene20-gt22 667 773 778 761 78322-gt24 749 713 769 758 776

xalan 24-gt25 757 754 758 778 794xerces 12-gt13 809 822 800 788 820

ivy 14-gt20 820 824 827 812 829

synapse10-gt11 525 484 511 530 54711-gt12 618 608 612 616 659

poi15-gt25 663 593 654 614 68525-gt30 410 449 420 432 426

Average 614 616 624 623 642

Table 6 Contributions of the three mechanisms to CPDP

Source Target Base +TL +Atten +Aug DTL-DPant16 camel14 383 379 365 377 395

jEdit41 camel14 368 394 395 371 407camel14 ant16 670 667 653 596 591

poi30 ant16 606 627 634 593 693camel14 jEdit41 663 657 652 647 531log4j11 jEdit41 599 599 592 594 639jEdit41 log4j11 754 800 774 746 783

lucene22 log4j11 759 750 753 783 794lucene22 xalan25 661 664 650 647 689xerces13 xalan25 510 638 626 662 686xalan25 lucene22 730 751 710 769 783log4j11 lucene22 713 745 740 727 769xalan25 xerces13 387 355 393 373 400ivy20 xerces13 413 408 398 378 420

xerces13 ivy20 518 500 504 500 472synapse12 ivy20 521 490 509 476 494

ivy14 synapse11 551 581 553 536 545poi25 synapse11 508 541 516 545 597ivy20 synapse12 604 622 619 611 620poi30 synapse12 600 632 613 612 623

synapse12 poi30 770 804 781 817 827ant16 poi30 777 662 700 812 827

Average 594 603 597 599 618

Table 7 Time cost of the three mechanisms defect visualization(Visualiz) self-attention (Attent) and transfer learning (TL)

ProjectTime (s)

TL Attent Visualizant 4504 131 459

camel 9655 242 785jEdit 5915 266 497log4j 1371 131 176

lucene 3739 186 354xalan 9444 271 823xerces 5117 214 275

ivy 2902 142 305synapse 2042 126 209

poi 3865 157 221Average 4856 187 410

To discern the contribution of each mechanism we compare theperformance of these baseline methods in WPDP and CPDP Fromthe results in Table 5 and Table 6 we observe that the three mecha-nisms each contribute substantially to the accuracy of DTL-DP inboth WPDP and CPDP In terms of the F-measure transfer learn-ing contributes the least improvement and self-attention and dataaugmentation contribute similarly to the final result But in CPDPtransfer learning contributes to the F-measure the most This islikely because of the applicability of transfer learning in the CPDPsetting and is in a way a validation of the approach

We also noted the time cost for the 3 mechanisms in our proposedDTL-DP Table 7 shows the result The most time spent during dataaugmentation is on converting code into images ie source codevisualization For transfer learning most time is spent on the MK-MMD calculation and for self-attention on the calculation of theattention layer in Fig 4 Eg for the project ant Table 3 shows twosets of WPDP experiments ant 15 rarr 16 and ant 16 rarr 17 Onaverage it takes 4504 seconds 1301 seconds and 459 seconds forthe 3 parts respectively for both the training data and the test dataTransfer learning takes the longest time more than the sum of theother two This is because of the large number of matrices neededto calculate MMD with the kernel function The least time cost isincurred by the attention mechanism and its contribution to WPDPis the largest of the three ie it is most cost-effective

The three mechanisms all contribute toward the accuracy ofour proposed end-to-end framework in WPDP and CPDP Self-attention has the greatest contribution to WPDP and transferlearning contributes most to CPDP

6 THREATS TO VALIDITYThreats to internal validity come from experimental errors and thereplication of the baseline methods In order to compare and analyzethe deep learning-based defect prediction techniques we compareour proposed DTL-DP method with Sementic DBN-CP DP-LSTMand DP-CNN In addition our method is also compared with thetransfer learning-based method TCA+ which is the state-of-the-art CPDP technique Since the original implementations were not

10

Software Visualization and Deep Transfer Learningfor Effective Software Defect Prediction ICSE 2020 May 23-29 2020 Seoul South Korea

available we re-implemented our own versions of the baselinesAlthough we strictly follow the procedures described in their workour new implementations may not completely restore all of theiroriginal implementation details And the randomness of the deeplearning-based approach also makes the results of our implementedexperiments different from the original Since we have removed theentries in the PROMISE dataset that cannot retrieve the correspond-ing source files our re-implemented experimental results may notbe consistent with the original baselines

The external threat to the validity of the results lies in the general-izability of the results We have tested our method on 10 open sourceJava projects including 14600 files Defect predictions for instancesof other languages such as C C++ etc need to be validated byadditional experiments in the future

Threats to construct validity depend on the appropriateness ofthe evaluation measurement The F-measure is used as our mainevaluation measure which has been applied in many previous effortsto evaluate defect prediction comprehensively

7 CONCLUSIONS AND FUTURE WORKHere we made two main contributions code visualization for defectprediction and an improved deep transfer learning model Our exper-imental results on 10 open source projects show that deep learningcan be effectively applied directly for defect prediction after apply-ing visualization methods to the code Specifically our approachDTL-DP performs at the top of the range of state-of-the-art WPDPapproaches For CPDP DTL-DP improves on the state-of-the-arttechnique TCA+ built on traditional features by 290 (in the F-measure) It also bests the deep learning-based approaches DBN-CPDP-LSTM and DP-CNN by 88 248 and 155 respectively

DTL-DP still has some limitations For some projects a problemof negative transfer occurs resulting in a worse prediction than adirect prediction Large differences between two projects can causesuch negative transfer Reducing the impact of negative transfer isone of the problems to be solved in the future Additionally theamount of training data we had was small and class imbalance isinherent in software defect prediction Both can be improved withmore data which we plan to obtain in the future

ACKNOWLEDGEMENTSThis work is partially supported by Zhejiang Natural Science Foun-dation(LY19F020025) National Natural Science Foundation ofChina (61502423 61572439 61702534) Zhejiang University OpenFund(2018KFJJ07) Signal Recognition Based on GAN Deep Learn-ing for Enhancement Recognition Project Zhejiang Science andTechnology Plan Project (LGF18F030009 2017C33149) and Zhe-jiang Outstanding Youth Fund (LR19F030001)

REFERENCES[1] Romi Satria Wahono A systematic literature review of software defect predic-

tion research trends datasets methods and frameworks Journal of SoftwareEngineering 1(1)1ndash16 2015

[2] Burak Turhan Tim Menzies Ayse B Bener and Justin Di Stefano On the relativevalue of cross-company and within-company data for defect prediction EmpiricalSoftware Engineering 14(5)540ndash578 2009

[3] Lech Madeyski and Marian Jureczko Which process metrics can significantlyimprove defect prediction models an empirical study Software Quality Journal23(3)393ndash422 2015

[4] Thomas Shippey David Bowes and Tracy Hall Automatically identifying codefeatures for software defect prediction Using ast n-grams Information andSoftware Technology 106142ndash160 2019

[5] Zhiyuan Wan Xin Xia Ahmed E Hassan David Lo Jianwei Yin and XiaohuYang Perceptions expectations and challenges in defect prediction IEEETransactions on Software Engineering 2018

[6] Karim O Elish and Mahmoud O Elish Predicting defect-prone software modulesusing support vector machines Journal of Systems and Software 81(5)649ndash6602008

[7] Puja Ahmad Habibi Victor Amrizal and Rizal Broer Bahaweres Cross-projectdefect prediction for web application using naive bayes (case study Petstoreweb application) In 2018 International Workshop on Big Data and InformationSecurity (IWBIS) pages 13ndash18 IEEE 2018

[8] Michael J Siers and Md Zahidul Islam Software defect prediction using a costsensitive decision forest and voting and a potential solution to the class imbalanceproblem Information Systems 5162ndash71 2015

[9] R Jayanthi and Lilly Florence Software defect prediction techniques using metricsbased on neural network classifier Cluster Computing pages 1ndash12 2018

[10] Yi Liu Yu Fan and Junghui Chen Flame images for oxygen content predictionof combustion systems using dbn Energy amp Fuels 31(8)8776ndash8783 2017

[11] Yi Liu Chao Yang Zengliang Gao and Yuan Yao Ensemble deep kernel learn-ing with application to quality prediction in industrial polymerization processesChemometrics and Intelligent Laboratory Systems 17415ndash21 2018

[12] Qi Xuan Binwei Fang Yi Liu Jinbao Wang Jian Zhang Yayu Zheng and Guan-jun Bao Automatic pearl classification machine based on a multistream convolu-tional neural network IEEE Transactions on Industrial Electronics 65(8)6538ndash6547 2018

[13] Qi Xuan Haoquan Xiao Chenbo Fu and Yi Liu Evolving convolutional neuralnetwork and its application in fine-grained visual categorization IEEE Access631110ndash31116 2018

[14] Qi Xuan Zhuangzhi Chen Yi Liu Huimin Huang Guanjun Bao and Dan ZhangMultiview generative adversarial network and its application in pearl classificationIEEE Transactions on Industrial Electronics 66(10)8244ndash8252 2019

[15] Qi Xuan Fuxian Li Yi Liu and Yun Xiang MV-C3D A spatial correlatedmulti-view 3d convolutional neural networks IEEE Access 792528ndash925382019

[16] Song Wang Taiyue Liu and Lin Tan Automatically learning semantic featuresfor defect prediction In Software Engineering (ICSE) 2016 IEEEACM 38thInternational Conference on pages 297ndash308 IEEE 2016

[17] Jian Li Pinjia He Jieming Zhu and Michael R Lyu Software defect predictionvia convolutional neural network In Software Quality Reliability and Security(QRS) 2017 IEEE International Conference on pages 318ndash328 IEEE 2017

[18] Hoa Khanh Dam Trang Pham Shien Wee Ng Truyen Tran John Grundy AdityaGhose Taeksu Kim and Chul-Joo Kim A deep tree-based model for softwaredefect prediction arXiv preprint arXiv180200921 2018

[19] Alex Krizhevsky Ilya Sutskever and Geoffrey E Hinton Imagenet classificationwith deep convolutional neural networks In Advances in neural informationprocessing systems pages 1097ndash1105 2012

[20] Mingsheng Long Yue Cao Jianmin Wang and Michael I Jordan Learning trans-ferable features with deep adaptation networks arXiv preprint arXiv1502027912015

[21] Han Zhang Ian Goodfellow Dimitris Metaxas and Augustus Odena Self-attention generative adversarial networks arXiv preprint arXiv1805083182018

[22] Tian Jiang Lin Tan and Sunghun Kim Personalized defect prediction In Pro-ceedings of the 28th IEEEACM International Conference on Automated SoftwareEngineering pages 279ndash289 IEEE Press 2013

[23] Andrew Meneely Laurie Williams Will Snipes and Jason Osborne Predictingfailures with developer networks and social network analysis In Proceedings ofthe 16th ACM SIGSOFT International Symposium on Foundations of softwareengineering pages 13ndash23 ACM 2008

[24] Raimund Moser Witold Pedrycz and Giancarlo Succi A comparative analysis ofthe efficiency of change metrics and static code attributes for defect prediction InProceedings of the 30th international conference on Software engineering pages181ndash190 ACM 2008

[25] Foyzur Rahman and Premkumar Devanbu How and why process metrics arebetter In Software Engineering (ICSE) 2013 35th International Conference onpages 432ndash441 IEEE 2013

[26] Maurice Howard Halstead et al Elements of software science volume 7 ElsevierNew York 1977

[27] Thomas J McCabe A complexity measure IEEE Transactions on softwareEngineering (4)308ndash320 1976

[28] Shyam R Chidamber and Chris F Kemerer A metrics suite for object orienteddesign IEEE Transactions on software engineering 20(6)476ndash493 1994

[29] Yue Yu Huaimin Wang Gang Yin and Tao Wang Reviewer recommendation forpull-requests in github What can we learn from code review and bug assignmentInformation and Software Technology 74204ndash218 2016

11

ICSE 2020 May 23-29 2020 Seoul South Korea Chen et al

[30] Nachiappan Nagappan and Thomas Ball Using software dependencies and churnmetrics to predict field failures An empirical case study In null pages 364ndash373IEEE 2007

[31] Ahmed E Hassan Predicting faults using the complexity of code changes InProceedings of the 31st International Conference on Software Engineering pages78ndash88 IEEE Computer Society 2009

[32] Thomas J Ostrand Elaine J Weyuker and Robert M Bell Programmer-basedfault prediction In Proceedings of the 6th International Conference on PredictiveModels in Software Engineering page 19 ACM 2010

[33] Taek Lee Jaechang Nam DongGyun Han Sunghun Kim and Hoh Peter In Microinteraction metrics for defect prediction In Proceedings of the 19th ACM SIG-SOFT symposium and the 13th European conference on Foundations of softwareengineering pages 311ndash321 ACM 2011

[34] Tim Menzies Zach Milton Burak Turhan Bojan Cukic Yue Jiang and AyseBener Defect prediction from static code features current results limitationsnew approaches Automated Software Engineering 17(4)375ndash407 2010

[35] Martin Pinzger Nachiappan Nagappan and Brendan Murphy Can developer-module networks predict failures In Proceedings of the 16th ACM SIGSOFTInternational Symposium on Foundations of software engineering pages 2ndash12ACM 2008

[36] Steffen Herbold Alexander Trautsch and Jens Grabowski A comparative studyto benchmark cross-project defect prediction approaches IEEE Transactions onSoftware Engineering 44(9)811ndash833 2018

[37] Chao Liu Dan Yang Xin Xia Meng Yan and Xiaohong Zhang A two-phasetransfer learning model for cross-project defect prediction Information andSoftware Technology 107125ndash136 2019

[38] Jinyin Chen Keke Hu Yitao Yang Yi Liu and Qi Xuan Col-lective transfer learning for defect prediction Neurocomputing 2019doihttpsdoiorg101016jneucom201812091

[39] Jinyin Chen Yitao Yang Keke Hu Qi Xuan Yi Liu and Chao Yang Multiviewtransfer learning for software defect prediction IEEE Access 78901ndash8916 2019

[40] Annibale Panichella Rocco Oliveto and Andrea De Lucia Cross-project defectprediction models Lrsquounion fait la force In Software Maintenance Reengineeringand Reverse Engineering (CSMR-WCRE) 2014 Software Evolution Week-IEEEConference on pages 164ndash173 IEEE 2014

[41] Burak Turhan Tim Menzies Ayse B Bener and Justin Di Stefano On the relativevalue of cross-company and within-company data for defect prediction volume 14pages 540ndash578 Springer 2009

[42] Fayola Peters Tim Menzies and Andrian Marcus Better cross company defectprediction In Proceedings of the 10th Working Conference on Mining SoftwareRepositories pages 409ndash418 IEEE Press 2013

[43] Xin Xia LO David Sinno Jialin Pan Nachiappan Nagappan and Xinyu WangHydra Massively compositional model for cross-project defect prediction IEEETransactions on software Engineering 42(10)977 2016

[44] Ying Ma Guangchun Luo Xue Zeng and Aiguo Chen Transfer learning forcross-company software defect prediction Information and Software Technology54(3)248ndash256 2012

[45] Jaechang Nam Sinno Jialin Pan and Sunghun Kim Transfer defect learningIn Software Engineering (ICSE) 2013 35th International Conference on pages382ndash391 IEEE 2013

[46] Sinno Jialin Pan Ivor W Tsang James T Kwok and Qiang Yang Domain adap-tation via transfer component analysis IEEE Transactions on Neural Networks22(2)199ndash210 2011

[47] Blaine A Price Ronald M Baecker and Ian S Small A principled taxonomy ofsoftware visualization Journal of Visual Languages amp Computing 4(3)211ndash2661993

[48] Thomas Ball and Stephen G Eick Software visualization in the large Computer29(4)33ndash43 1996

[49] Sarita Bassil and Rudolf K Keller Software visualization tools Survey andanalysis In Proceedings 9th International Workshop on Program ComprehensionIWPC 2001 pages 7ndash17 IEEE 2001

[50] Stephan Diehl Software visualization visualizing the structure behaviour andevolution of software Springer Science amp Business Media 2007

[51] Michele Lanza The evolution matrix Recovering software evolution using soft-ware visualization techniques In Proceedings of the 4th international workshopon principles of software evolution pages 37ndash42 ACM 2001

[52] Denis Gracanin Krešimir Matkovic and Mohamed Eltoweissy Software vi-sualization Innovations in Systems and Software Engineering 1(2)221ndash2302005

[53] Guillaume Langelier Houari Sahraoui and Pierre Poulin Visualization-basedanalysis of quality for large-scale software systems In Proceedings of the 20thIEEEACM international Conference on Automated software engineering pages214ndash223 ACM 2005

[54] Abien Fred Agarap and Francis John Hill Pepito Towards building an intelligentanti-malware system a deep learning approach using support vector machine(svm) for malware classification arXiv preprint arXiv180100318 2017

[55] Karl Weiss Taghi M Khoshgoftaar and DingDing Wang A survey of transferlearning Journal of Big Data 3(1)9 2016

[56] Rahul Krishna and Tim Menzies Bellwethers A baseline method for transferlearning IEEE Transactions on Software Engineering 2018

[57] Chuanqi Tan Fuchun Sun Tao Kong Wenchang Zhang Chao Yang and ChunfangLiu A survey on deep transfer learning In ICANN 2018

[58] Wenyuan Dai Qiang Yang Gui Rong Xue and Yong Yu Boosting for transferlearning In International Conference on Machine Learning pages 193ndash2002007

[59] Chang Wan Rong Pan and Jiefei Li Bi-weighting domain adaptation for cross-language text classification In IJCAI Proceedings-International Joint Conferenceon Artificial Intelligence volume 22 page 1535 2011

[60] Yonghui Xu Sinno Jialin Pan Hui Xiong Qingyao Wu Ronghua Luo HuaqingMin and Hengjie Song A unified framework for metric transfer learning IEEETrans Knowl Data Eng 29(6)1158ndash1171 2017

[61] Chetak Kandaswamy Luiacutes M Silva Luiacutes A Alexandre and Jorge M SantosDeep transfer learning ensemble for classification In Ignacio Rojas GonzaloJoya and Andreu Catala editors Advances in Computational Intelligence pages335ndash348 Cham 2015 Springer International Publishing

[62] Maxime Oquab Leon Bottou Ivan Laptev and Josef Sivic Learning and trans-ferring mid-level image representations using convolutional neural networks InProceedings of the IEEE conference on computer vision and pattern recognitionpages 1717ndash1724 2014

[63] Yann LeCun Leacuteon Bottou Yoshua Bengio and Patrick Haffner Gradient-basedlearning applied to document recognition Proceedings of the IEEE 86(11)2278ndash2324 1998

[64] Hana Ajakan Pascal Germain Hugo Larochelle Franccedilois Laviolette and MarioMarchand Domain-adversarial neural networks arXiv preprint arXiv141244462014

[65] Yaroslav Ganin and Victor Lempitsky Unsupervised domain adaptation by back-propagation arXiv preprint arXiv14097495 2014

[66] Jing Zhang Wanqing Li and Philip Ogunbona Joint geometrical and statisticalalignment for visual domain adaptation arXiv preprint arXiv170505498 2017

[67] Eric Tzeng Judy Hoffman Ning Zhang Kate Saenko and Trevor DarrellDeep domain confusion Maximizing for domain invariance arXiv preprintarXiv14123474 2014

[68] Mingsheng Long Yue Cao Zhangjie Cao Jianmin Wang and Michael I Jor-dan Transferable representation learning with deep adaptation networks IEEEtransactions on pattern analysis and machine intelligence 2018

[69] Arthur Gretton Dino Sejdinovic Heiko Strathmann Sivaraman BalakrishnanMassimiliano Pontil Kenji Fukumizu and Bharath K Sriperumbudur Optimalkernel choice for large-scale two-sample tests In Advances in neural informationprocessing systems pages 1205ndash1213 2012

[70] Mingsheng Long Han Zhu Jianmin Wang and Michael I Jordan Deep transferlearning with joint adaptation networks arXiv preprint arXiv160506636 2016

[71] Martin Arjovsky Soumith Chintala and Leacuteon Bottou Wasserstein gan arXivpreprint arXiv170107875 2017

[72] Kesav Kancherla and Srinivas Mukkamala Image visualization based malwaredetection In 2013 IEEE Symposium on Computational Intelligence in CyberSecurity (CICS) pages 40ndash44 IEEE 2013

[73] Xin Zhou Jianmin Pang and Guanghui Liang Image classification for malwaredetection using extremely randomized trees In 2017 11th IEEE InternationalConference on Anti-counterfeiting Security and Identification (ASID) pages54ndash59 IEEE 2017

[74] John Murphy An overview of convolutional neural network architectures for deeplearning 2016

[75] Song Wang Taiyue Liu Jaechang Nam and Lin Tan Deep semantic feature learn-ing for software defect prediction IEEE Transactions on Software Engineering2018

[76] Laurens van der Maaten and Geoffrey Hinton Visualizing data using t-sneJournal of machine learning research 9(Nov)2579ndash2605 2008

12

• Abstract
• 1 Introduction
• 2 Related Work
• 3 Approach
• • 31 Source Code Visualization
  • 32 Modeling with Deep Transfer Learning
  • 33 Model Sensitivity to Parameter Choices
  • 34 Training and Prediction
  • • 4 Experimental Setup
    • • 41 Dataset Description
      • 42 Baseline Comparison Methods
      • 43 Performance measures
      • • 5 Results
        • • 51 RQ1 How does DTL-DP compare to feature- based machine learning methods and AST- based deep learning methods in WPDP
          • 52 RQ2 How does DTL-DP compare to feature- based Machine Learning and AST-based deep learning methods in CPDP
          • 53 RQ3 How much does each of the three mechanisms ie data augmentation transfer learning and self-attention mechanism contribute to DTL-DPs performance
          • • 6 Threats to Validity
            • 7 Conclusions and Future Work
            • References

ICSE 2020 May 23-29 2020 Seoul South Korea Chen et al

DataAugmentation

B G R

GRB

R B G

Feature- Net

AttentionLayerBGR

G R B

RBG

Clean Buggy

Clean Buggy

SourceProject

TargetProject

MMD

Clean

Buggy

Clean

Buggy

Defect Visualization Model Building Phase

Prediction Phase

Figure 2 The overall framework of DTL-DP

Figure 3 The process of converting code to color images

we build train and evaluate a defect prediction model We give thedetails in the following

31 Source Code VisualizationConverting code to images retains information and facilitates avail-able deep neural network (DNN) -based methods to improve defectprediction performance Most existing methods ignore associationsbetween extracted features and the classification task Our proposedmethod forms a unified end-to-end framework of feature extractionmodeling and prediction

Figure 3 shows how we convert each program file into 6 differentimages We call this process source code visualization and the im-ages produced code images First each source file is converted into avector of 8-bit unsigned integers corresponding to the ASCII decimalvalues of the characters in the source code eg rsquoarsquo is converted to97 etc We then generate an image from that vector by arrangingits values in rows and columns and interpreting it as a rectangularimage A straight forward way to visualize the 8-bit values as colorintensities would be as levels of gray eg 0=black and 255=whiteHowever the relatively small size of our original training set meansthat if we did that we would end up with a deep model that cannot besufficiently well trained Fortunately an additional benefit to using

images for training is that we can augment the original data set toproduce a larger data set with the same semantics Common dataaugmentation methods for image data sets include flipping (bothvertically and horizontally) rotating cropping translating (movingalong the x or y axis) adding Gaussian noise (distortion of highfrequency features) zooming and scaling Since the sizes of our pro-grams are small and we want to retain the semantic and structuralfeatures in the images the above image data augmentation methodscould result in data loss

Instead we designed a novel color based augmentation methodto generate 6 color images from each source code file Namelyeach pixel in a color image can be represented by three primarycolor components or channels red (R) green (G) and blue (B)The letters R G B and thus the color channels can be orderedin 6 different ways RBG RGB BGR BRG GRB and GBR Byadopting a different order every time as shown in Figs 2 and 3 wegenerate six different images for each program file For examplersquoforrsquo is converted to [102 111 114] and the 3 values are filled intothe R G and B channels in 6 different ways to obtain 6 differentlycolored pixels

The above method expands our data set six-fold and because ofthe nature of the downstream analysis the generated samples arereasonable since the representation is changed only by the order ofthe different channels 2

Whereas we generated 6 images for each instance in the trainingset we randomly selected an RGB permutation for testing We foundthat it was not necessary to generate all six images for testing becauseour experiments on 10 datasets showed that the performance of usingthe different permutations was quite comparable We randomly choseone to increase the speed in practice The data augmentation wasused to improve the efficacy of the model

After the above for each of the 6 orderings of R G B we obtaina vector of pixels of length one third the original code file size(each source code character is assigned to a color channel and thusthree characters in a row represent a pixel) We then turned those

2A CNN model learns the features by convolution Since we use ImageNetrsquos pre-trainedAlexNet model the initial parameters of the convolution kernel are different for eachchannel which means even though byte sequences are the same the final set of featuresobtained by sequentially inputting into the model the different channels is also different

4

Software Visualization and Deep Transfer Learningfor Effective Software Defect Prediction ICSE 2020 May 23-29 2020 Seoul South Korea

Figure 4 The training and testing process of our approach DTL-DP inspired by the original DAN paper [68]

Table 1 Image Width for Various File Sizes

File Size Range Image Width

lt10 kB 3210 kB - 30 kB 6430 kB - 60 kB 12860 kB - 100 kB 256

100 kB - 200 kB 384200 kB - 500 kB 512500 kB - 1000 kB 768

gt1000kB 1024

6 image vectors into image matrices of width and length such thatwidth lowast lenдth = vector_size3 We note that the first convolutionkernel of AlexNet is of size 1111 so if the image is too narrowthe convolution performance will be poor And the image with asuitable width can be convolved to obtain more efficient semanticand structural features in contexts with a proper sequence lengthTable 1 gives some recommended image widths for different filesizes based on previous work [72 73] We adopt those in this work

32 Modeling with Deep Transfer LearningNext we describe our approach DTL-DP The goal is to learn trans-ferable features across projects and build a classifier y = ϕ(x) offile defectiveness with source code supervision where x is the rep-resentation of the sample and y is the predicted label In the defectprediction problem we are given a source project Ps = (xsi y

si )

nsi=1

with ns labeled instances and a target project Pt = (xti yti )

nti=1

with nt unlabeled instances In WPDP the source project is the pre-vious version of the target project while in CPDP the source projectis a related project of the target project The sample distributions pand q in the source and target projects are often similar in WPDPbut different in CPDP

For our deep network architecture we adopt a similar architec-ture as that of deep adaptation networks (DANs) [68] to capturethe semantic and structural information of the source code To theoriginal DAN model we add an attention layer to further enhancethe expressive ability of important features The overall architectureis illustrated in Figure 4 In particular our DTL-DP consists of aninput layer five convolution layers (conv1 minus conv5 from AlexNet)an Attention-Layer and finally four fully-connected hidden layers(f c6 minus f c9) working as a classifier The structure and parameters of

(conv1minusconv5) and (f c6minus f c8) are consistent with a DAN But sincedefect prediction is a binary classification problem a fully-connectedlayer (f c9) is added to obtain a binary result at the end

We adopt the defaults for AlexNet so the input to our DTL-DPmust be a 224224 size image cropped from a three-channel (RGB)image of size 256256 The code image is placed in the approximatecenter of the 256256 image (the determination of the size of eachcode image is described later in Sect 33) We note that the codeimage can be smaller than 224224 pixels due to the variance in thesize of the code files If this happens the image is padded aroundwith blank (zero valued) pixels Padding should not negatively effectthe qualitative performance of feature detection in the images in factfor deeper DL architectures like ours padding has been shown toprovide extra contrast to the embedded image for each of the layersas well as buffering against data loss by each layer [74]

Training deep models requires a significant amount of labeleddata which is not available for a new project so we first pre-trainedan AlexNet model on ImageNet 2012 Unlike a DAN which freezesconv1minusconv3 and fine-tunes conv4minusconv5 we fine tune all convolu-tion layers conv1minusconv5 by taking the parameters of the pre-trainedmodels as initial parameters that we then optimize during the train-ing phase We do that to minimize the differences between our codeimages and the actual object images in ImageNet 2012

In order for DTL-DP to focus on the key features in differentdefect images and thus further improve prediction we employ aself-attention mechanism into our model inspired by the good per-formance of self-attention in GANs [21] As shown in the AttentionLayer in Figure 4 the attention mechanism makes the feature mapgenerated by layer conv5 be the self-attention feature map input tothe next layer fc6 Specifically at first it linearly maps the inputfeatures x (it is a 11 convolution to compress the number of chan-nels ie out_chanels = in_chanels8) and produces f (x)д(x)and h(x) where f (x) = wf x д(x) = wдx h(x) = whx The dif-ference between the three is that the size of h(x) is still the sameas x but the other two are not Thus if the width of x is W theheight H and the number of channelsC the size of x is [CN ] whereN =W lowast H the size of f(x) and g(x) is [C8N ] but the size of h(x)is [CN ] The transposed f (x) and д(x) are matrix-multiplied to ob-tain the autocorrelation in the features ie the relationship of eachpixel to all other pixels where Si j = f (xi )

Tд(x j ) Then softmax isapplied to the autocorrelation features S to get the attention mapcomprised of weights with values between 0 and 1

α ji =exp(Si j )sumNi=1 exp(Si j )

(1)

5

ICSE 2020 May 23-29 2020 Seoul South Korea Chen et al

005 01 02 05 1 2 5MMD Penalty

055

056

057

058

059

060

061

062

063

064

065

F-m

easu

re

WPDP CPDP

(a) Effect of different MMD Penalty λ

BRG BGR GRB GBR RGB RBG040

045

050

055

060

065

070

075

F-m

easu

re

WPDP CPDP

(b) Effect of color channel order

8 4 2 1 12 14 18Width recommended

040

045

050

055

060

065

070

075

F-m

easu

re

WPDP CPDP

(c) Effect of code image widthrecommended ratio

Figure 5 Sensitivity studies to aid in choosing the parameters in the model

After that the output of the AttentionLayer is the self-attentionfeature map o = (o1o2 oj oN ) where

oj =Nsumi=1

α jih(xi ) (2)

Then each fully connected layer learns a nonlinear mappinghli = f l (wlhli + b

l ) where hli is the l th layer hidden representationof feature xi wl and bl are the weights and biases of the l th layerand f l is the activation function using ReLU (f l (o) = max(0o))for f c6 minus f c8 and softmax for f c9 If we let Θ denote the set of allDTL-DP parameters the empirical risk of DTL-DP then is

minΘ

1n

nsumi=1

F (ϕ(oi )yi ) (3)

where F is the cross-entropy loss function n is the number of in-stances and ϕ(oi ) is the conditional probability that the DTL-DPassigns oi to label yi It is used to calculate the final loss in Equation7

To make the distribution of the source and target projects similarthe same multi-layer adaptation and multi-kernel MMD strategy asin a DAN [68] are used in our model The feature mapping functionσ is defined as the combination ofm positive semi-definite kernelsku

lt σ (os )σ (ot ) gt= k(os ot ) (4)

k =msumu=1

βuku (5)

where β ge 0 are the weights of the kernelsThe MK-MMD dk (pq) of the probability distributions p and q

is defined as the reproducing kernel Hilbert space distance betweenthe mean embedding of p and q The square formula of MK-MMDis defined as

d2k (pq) =∥ Ep [σ (xs )] minus Eq [σ (x

t )] ∥2 (6)

We see then that the smaller dk (pq) is the more similar p and qare If dk is 0 the distribution of the target project is the same as thatof the source project So the final loss function for train DPL-DP is

1n

nsumi=1

F (ϕ(oi )yi ) + λ

l2suml=l1

d2k (Dls D

lt ) (7)

where Dls is the l th layer hidden representation for the source and

target d2k (Dls D

lt ) is the MK-MMD between the source and target

evaluated on the l th layer representation λ is a penalty parameterand l1 and l2 are f c layers to calculate the MK-MMD between sourceand target In our implementation of DTL-DP we set λ = 1 l1 = 6and l2 = 9 as per the original DAN work [68]

33 Model Sensitivity to Parameter ChoicesWe have made a number of choices to make our modeling plat-form work effectively Here we justify those choices by presentingsensitivity studies For this analysis we used all the data

Hyperparameter λ The DTL-DP model has a hyper-parameter λfor the MMD penalty that adjusts the final loss We conducted λ pa-rameter sensitivity experiments in both the WPDP and CPDP settingWe fix the other parameters and range λ in 001 005 01 02 05 1 2 5The results are shown in Figure 5(a) While fairly constant across therange small variations exist In CPDP the F-measure first increasesand then decreases along the hyper-parameter λ In WPDP λ affectsthe performance of DTL-DP but in the opposite direction We choseλ = 1 here

Different Color Orders The selection of the RGB permutationsin the target project images may also potentially affect the outcomeWe performed experiments with all six different orderings Whentesting them we set λ to 1 and changed the image type of the tar-get project to each one in RGBRBGBRGBGRGBRGRB Theresults are shown in Figure 5(b) The target images in different colororders have somewhat different performance But overall the valuesare close to DTL-DPrsquos average performance

Different Image Widths To explore the impact of different choicesfor image widths on the results we performed additional experi-ments

For our code images 3 bytes of source code are needed to form 1pixel each byte for one of the R G and B channels The sizes ofthe source files are between 0 and 100 kb corresponding to imageswith pixel count between 0 and 34 133 (10010243) The size ofthe image is (widthheiдht) where width is obtained from Table 1according to the size of the code file (size)

heiдht =size lowast 10243 lowastwidth

(8)

For example the size of image converted from a 20 kb code file is(64 107)

We experimented with image widths of 18 14 12 2 4 and 8times of the recommended widths in Table 1 The larger the mul-tiplier the wider the image For images of different widths we

6

Software Visualization and Deep Transfer Learningfor Effective Software Defect Prediction ICSE 2020 May 23-29 2020 Seoul South Korea

repeated the previous experiment and obtained results for the meanF-measure shown in Figure 5(c) We see that the closer to the recom-mended width from Table 1 the better the average result In additionwe found that a wider image is better than a narrower one And thecloser the image is to a square the better the prediction Thereforewe made the generated images be as close to a square as possible

34 Training and PredictionWhen training the DTL-DP code images are generated from thelabeled instances in the source project and the unlabeled instances inthe target project and then are simultaneously input into the modelThey share the convolution layers conv1minusconv5 andAttentionLayerto extract their respective features and calculate the MK-MMDbetween source and target projects in the fully connected layersf c6 minus f c9 It should be noted that we only calculate cross entropyfor the source project because the target projectrsquos labels are not pro-vided We use a mini-batch stochastic gradient descent along the lossfunction in order to train the parameters of the entire model We trainfor 500 epochs for each pair of source and target projects then pickthe epoch with the best F-measure (described in section 43) fromwhich to read out the parameters of the final model Finally the filesfrom the target project that need to be predicted are converted intocode images and then input into the trained model for classification

Overfitting is always a possibility with such pipelines We moder-ate it here with our choices of (1) augmenting the dataset by using sixcolor channel permutations described in section 31 (2) selecting asimple AlexNet model structure and (3) using the ImageNet 2012pre-trained model to reduce fluctuations

4 EXPERIMENTAL SETUPWe conducted experiments to asses the performance of DTL-DP andto compare it with existing deep learning-based defect predictionapproaches for both within-project WPDP and cross-project CPDPdefect prediction We ran experiments on a modern-day Linux serverwith 3 Titan XP GPUs Unless otherwise stated each experimentwas run 10 times and the average results are reported

41 Dataset DescriptionIn order to directly compare our work with prior research we usedpublicly available data from the PROMISE3 data repository whichhas been widely used in defect prediction work [16ndash18 75] Weselected all open source Java projects from this repository andcollected their version number class names and the buggy labelfor each file Based on the version number and class name weobtained the source code for each file from GitHub4 and fed it toour end-to-end framework In total data for 10 Java projects werecollected Table 2 shows the details of these projects includingproject description versions the total and average number of filesand the defect rate It should be noted that the average numberof files over all projects ranges between 150 and 1046 and thedefect rates of the projects have a minimum value of 134 and amaximum value of 497 The number of files in some projects isnot sufficient to train deep models and the classes are imbalancedthus augmentation is needed as described above

3httpopenscienceusrepodefect4httpsgithubcomapache

42 Baseline Comparison MethodsTo evaluate the performance of our end-to-end framework DTL-DPfor defect prediction we compare it with the following baselinemethods in the WPDP setting

bull Semantic [16 75] the state-of-the-art method which em-ploys deep belief networks DBN on source code to extractsemantic features for defect prediction

bull PROMISE-DP [16] a traditional method which builds analternating decision tree ADTree classifier based on theoriginal 20 features of the PROMISE dataset

bull DP-LSTM [18] a long short-term memory LSTM-baseddeep neural network model which uses ASTs to representsource files and predict whether the file is defective or not

bull DP-CNN [17] a convolutional neural network CNN-basedmodel which is seeded by AST-derived numerical vectorsto automatically learn semantic and structural features ofprograms The CNN-learned features are used to train thefinal classifier in combination with traditional features

For the cross-project settings CPDP Semantic and PROMISE-DPcould not be used directly Instead we used the following

bull DBN-CP [16 75] a variant of Semantic which trains a DBNby using the source project and generates semantic featuresfor both the source and target projects

bull TCA+ [45] the state-of-the-art technique for CPDP

To obtain the training and test data we followed the processesestablished in [16] For WPDP we use two consecutive versions ofeach project listed in Table 2 The older version is used to generatethe training data and the more recent version is used as test data ForCPDP we pick versions randomly from each project for 11 targetprojects And for each target project we select 2 source projects thatare different from the target projects We use the same 22 test pairsas in [16] When implementing the baseline methods we use thesame network architecture and parameter settings as described in thepapers that introduced them

43 Performance measuresTo evaluate the prediction performance we use the F-measure awidely adopted metric in the literature [16ndash18 43 45] The F-measure captures a predictorrsquos accuracy and combines both precisionand recall for a comprehensive evaluation of predictive performance

Specifically a prediction that a file is defective is called a truepositive (TP) if the file is in fact defective and false positive (FP)otherwise Similarly a prediction that a file is not defective is a truenegative (TN) if the file is in fact not defective and false negative(FN) otherwise Then the precision (P) recall (R) and F-measureare defined as

P = TP(TP + FP) (9)

R = TP(TP + FN ) (10)

F = (2 times P times R)(P + R) (11)

5 RESULTSThis section discusses our results of comparing DTL-DP to baselinetools for defect prediction

7

ICSE 2020 May 23-29 2020 Seoul South Korea Chen et al

Table 2 Dataset Description

Project Description Versions Files Avg files Avg size(kb) Defective

ant Java based build tool 151617 1465 488 62 134camel Enterprise integration framework 121416 3140 1046 29 187jEdit Text editor designed for programmers 324041 1935 645 87 192log4j Logging library for Java 1011 300 150 34 497

lucene Text search engine library 202224 607 402 38 358xalan A library for transforming XML files 2425 1984 992 46 296xerces XML parser 1213 1647 549 29 157

ivy Dependency management library 1420 622 311 41 200synapse Data transport adapters 101112 661 220 38 227

poi Java library to access Microsoft format files 152530 1248 416 36 407

Table 3 F-measure of DTL-DP Semantic (Seman) PROMISE-DP (PROM) DP-LSTM (LSTM) and DP-CNN (CNN) forWPDP

Project Version Seman PROM LSTM CNN DTL-DP

ant15-gt16 914 477 452 532 70016-gt17 942 542 379 566 593

camel12-gt14 785 373 323 461 40614-gt16 374 391 400 508 385

jEdit32-gt40 574 556 472 564 49440-gt41 615 546 490 580 687

log4j 10-gt11 701 587 539 632 688

lucene20-gt22 651 502 751 761 78322-gt24 773 605 751 721 776

xalan 24-gt25 595 518 657 601 794xerces 12-gt13 411 238 268 374 820

ivy 14-gt20 350 329 191 347 829

synapse10-gt11 544 476 454 539 54711-gt12 583 533 533 556 659

poi15-gt25 640 558 808 589 68525-gt30 803 754 777 784 426

Average 641 499 515 570 642

51 RQ1 How does DTL-DP compare to feature-based machine learning methods and AST-based deep learning methods in WPDP

We compare DTL-DP to 4 baseline approaches representing twodifferent kinds of defect prediction methods PROMISE-DP is thebaseline representative of traditional feature-based machine learningmethods Semantic DP-LSTM and DP-CNN are the baselines fordeep learning-type methods based on extracting features from ASTGuided by prior work we conducted 16 sets of WPDP experimentseach using two versions of the same project The older version isused to train the prediction model and the newer version is used toevaluate the trained model

Table 3 shows the F-measure values for the within-project WPDPdefect prediction experiments The highest F-measure values of the5 methods are shown in bold Since the methods based on deeplearning include some randomness we run DTL-DP Semantic DP-LSTM and DP-CNN 10 times for each experiment On average

the F-measure of our approach is 0642 and the PROMISE-DPSemantic DP-LSTM and DP-CNN achieve 0499 0641 0515 and0570 respectively The results demonstrate that our approach iscompetitive and may improve on defect prediction compared toPROMISE-DP DP-LSTM and DP-CNN The results of Semanticand our approach are similar

The proposed DTL-DP is effective and it could improvethe performance of WPDP tasks Like other deep learners it issensitive to small file sizes and unbalanced data

511 Case Study WPDP Discrimination of DTL-DP t-SNEis a non-linear dimensional reduction algorithm that is effectivein visualizing similarities and helping identify clusters in complexdata sets [76] To give insight in the performance of DTL-DP wedemonstrate feature transferability by showing t-SNE embeddingsin Figure 6 The blue points are non-defective files and the red aredefective ones5 We observe the following (1) The target instancesare not discriminated very well using either the traditional manualfeatures or the TCA+ improved manual features or the semanticfeatures extracted from ASTs while with our approach the pointsare discriminated much better (2) With the other three approachesthe categories between source and target projects are not well alignedwhile with our approach the categories between the projects aremore consistent These conclusions are derived from the intra- andinter-class distances of the two categories in Figures 6 and 7 Theyare visually apparent

Our method did not perform as well as the comparison methods onsome projects such as ant which is likely caused by large variancein file sizes While the sizes of ant-15 and ant-16 are close there isa marked difference between ant-16 and ant-17 the former beingmuch smaller than the latter From Table 3 the performance of ourmethod on ant15-gtant16 is better than that on ant16-gtant17On the contrary baseline methods other than DP-LSTM performbetter for the task ant16-gtant17 notably Semantic is dominantindicating that the semantic feature-based method is more robust tofile size variability Moreover the ant dataset has two shortcomingsthe first is that the amount of data is small cf Table 2 where theaverage amount is only 488 files in one project The second is that

5Since we use data augmentation we have more samples than the three baselines

8

Software Visualization and Deep Transfer Learningfor Effective Software Defect Prediction ICSE 2020 May 23-29 2020 Seoul South Korea

Figure 6 t-SNE mapping of source and target features (WPDP)

the classes are unbalanced and the proportion of defective files is134 which is the least of all our projects This makes training ofthe deep model more difficult which leads to the poorer performanceof our method on some projects

Table 4 F-measure of DTL-DP DBN-CP (DBN) TCA+ DP-LSTM (LSTM) and DP-CNN (CNN) in CPDP

Source Target DBN TCA+ LSTM CNN DTL-DPant16 camel14 316 292 321 323 395

jEdit41 camel14 693 330 318 651 407camel14 ant16 979 616 448 607 591

poi30 ant16 478 598 386 532 693camel14 jEdit41 615 537 394 547 531log4j11 jEdit41 503 419 389 423 639jEdit41 log4j11 645 574 574 656 783

lucene22 log4j11 618 571 578 632 794lucene22 xalan25 550 530 680 540 689xerces13 xalan25 572 581 676 562 686xalan25 lucene22 594 561 750 621 783log4j11 lucene22 692 524 750 663 769xalan25 xerces13 386 394 340 391 400ivy20 xerces13 426 398 261 421 420

xerces13 ivy20 453 409 264 467 472synapse12 ivy20 824 383 261 371 494

ivy14 synapse11 489 348 451 491 545poi25 synapse11 425 376 435 436 597ivy20 synapse12 433 570 530 456 620poi30 synapse12 514 542 503 532 623

synapse12 poi30 661 651 785 671 827ant16 poi30 619 343 785 627 827

Average 568 479 495 528 618

52 RQ2 How does DTL-DP compare to feature-based Machine Learning and AST-based deeplearning methods in CPDP

Here we compare to TCA+ and DBN-CP instead of PROMISEand Semantic as the baseline approaches as explained in Sect 42

Again guided by prior work we conducted 22 sets of CPDP experi-ments In each we randomly select two project versions from twodifferent projects one as a training set and the other as a test set

Table 4 presents the F-measure results of DTL-CP and the 4 base-line approaches The highest F-measure values are in bold On aver-age the F-measure of our approach in CPDP is 0618 and the DBN-CP TCA+ DP-LSTM and DP-CNN achieve 0568 0479 0495 and0528 Thus DTL-DP outperforms them by 88 290 248 and155 respectively In addition we found that for projects log4j11and poi30 our method does better than the corresponding WPDPbest performing method

Our proposed DTL-DP shows significant improvements onthe state-of-the-art in cross-project defect prediction

521 Case Study CPDP Discrimination of DTL-DP Ourmethod is more obviously dominant in CPDP than in WPDP Apossible reason for that is that deep transfer learning makes the dis-tributions of the training and test samples more similar in featurespace Another reason might be the superior ability of deep modelsto represent features enabling the model to obtain more transferablefeatures from the images To gain more insight we choose the taskpoi30 rarr ant16 and show the t-SNE embeddings in Figure 7 Wemake similar observations as in the RQ1 case study that (1) the tar-get instances are more easily discriminated with our approach and(2) the target instances can be better discriminated with the sourceclassifier This implies that our approach can learn more transferablefeatures for more effective defect prediction

Figure 7 t-SNE mapping of source and target features (CPDP)

53 RQ3 How much does each of the threemechanisms ie data augmentation transferlearning and self-attention mechanismcontribute to DTL-DPrsquos performance

To find out the specific contributions of the three parts to defectprediction we conducted further experiments We built the originalAlexNet model for binary classification and use it as the Base +TL+Attention and +DataAug are three new baselines built by adding tothe base AlexNet one of three mechanisms (transfer learning selfattention and data augmentation) respectively It should be noted

9

ICSE 2020 May 23-29 2020 Seoul South Korea Chen et al

that in addition to +DataAug we use images generated by onesequence of R G and B as training and test sets in each experimentTherefore the experimental results of Base +TL and +Attention inTables 5 and 6 are average results obtained over the 6 different RGBpermutations as described above

Table 5 Contributions of the three mechanisms to WPDP

Project Version Base +TL +Atten +Aug DTL-DP

ant15-gt16 660 676 664 677 70016-gt17 525 552 524 591 593

camel12-gt14 415 381 416 403 40614-gt16 412 396 423 384 385

jEdit32-gt40 484 505 497 485 49440-gt41 629 649 645 679 687

log4j 10-gt11 688 679 692 658 688

lucene20-gt22 667 773 778 761 78322-gt24 749 713 769 758 776

xalan 24-gt25 757 754 758 778 794xerces 12-gt13 809 822 800 788 820

ivy 14-gt20 820 824 827 812 829

synapse10-gt11 525 484 511 530 54711-gt12 618 608 612 616 659

poi15-gt25 663 593 654 614 68525-gt30 410 449 420 432 426

Average 614 616 624 623 642

Table 6 Contributions of the three mechanisms to CPDP

Source Target Base +TL +Atten +Aug DTL-DPant16 camel14 383 379 365 377 395

jEdit41 camel14 368 394 395 371 407camel14 ant16 670 667 653 596 591

poi30 ant16 606 627 634 593 693camel14 jEdit41 663 657 652 647 531log4j11 jEdit41 599 599 592 594 639jEdit41 log4j11 754 800 774 746 783

lucene22 log4j11 759 750 753 783 794lucene22 xalan25 661 664 650 647 689xerces13 xalan25 510 638 626 662 686xalan25 lucene22 730 751 710 769 783log4j11 lucene22 713 745 740 727 769xalan25 xerces13 387 355 393 373 400ivy20 xerces13 413 408 398 378 420

xerces13 ivy20 518 500 504 500 472synapse12 ivy20 521 490 509 476 494

ivy14 synapse11 551 581 553 536 545poi25 synapse11 508 541 516 545 597ivy20 synapse12 604 622 619 611 620poi30 synapse12 600 632 613 612 623

synapse12 poi30 770 804 781 817 827ant16 poi30 777 662 700 812 827

Average 594 603 597 599 618

Table 7 Time cost of the three mechanisms defect visualization(Visualiz) self-attention (Attent) and transfer learning (TL)

ProjectTime (s)

TL Attent Visualizant 4504 131 459

camel 9655 242 785jEdit 5915 266 497log4j 1371 131 176

lucene 3739 186 354xalan 9444 271 823xerces 5117 214 275

ivy 2902 142 305synapse 2042 126 209

poi 3865 157 221Average 4856 187 410

To discern the contribution of each mechanism we compare theperformance of these baseline methods in WPDP and CPDP Fromthe results in Table 5 and Table 6 we observe that the three mecha-nisms each contribute substantially to the accuracy of DTL-DP inboth WPDP and CPDP In terms of the F-measure transfer learn-ing contributes the least improvement and self-attention and dataaugmentation contribute similarly to the final result But in CPDPtransfer learning contributes to the F-measure the most This islikely because of the applicability of transfer learning in the CPDPsetting and is in a way a validation of the approach

We also noted the time cost for the 3 mechanisms in our proposedDTL-DP Table 7 shows the result The most time spent during dataaugmentation is on converting code into images ie source codevisualization For transfer learning most time is spent on the MK-MMD calculation and for self-attention on the calculation of theattention layer in Fig 4 Eg for the project ant Table 3 shows twosets of WPDP experiments ant 15 rarr 16 and ant 16 rarr 17 Onaverage it takes 4504 seconds 1301 seconds and 459 seconds forthe 3 parts respectively for both the training data and the test dataTransfer learning takes the longest time more than the sum of theother two This is because of the large number of matrices neededto calculate MMD with the kernel function The least time cost isincurred by the attention mechanism and its contribution to WPDPis the largest of the three ie it is most cost-effective

The three mechanisms all contribute toward the accuracy ofour proposed end-to-end framework in WPDP and CPDP Self-attention has the greatest contribution to WPDP and transferlearning contributes most to CPDP

6 THREATS TO VALIDITYThreats to internal validity come from experimental errors and thereplication of the baseline methods In order to compare and analyzethe deep learning-based defect prediction techniques we compareour proposed DTL-DP method with Sementic DBN-CP DP-LSTMand DP-CNN In addition our method is also compared with thetransfer learning-based method TCA+ which is the state-of-the-art CPDP technique Since the original implementations were not

10

Software Visualization and Deep Transfer Learningfor Effective Software Defect Prediction ICSE 2020 May 23-29 2020 Seoul South Korea

available we re-implemented our own versions of the baselinesAlthough we strictly follow the procedures described in their workour new implementations may not completely restore all of theiroriginal implementation details And the randomness of the deeplearning-based approach also makes the results of our implementedexperiments different from the original Since we have removed theentries in the PROMISE dataset that cannot retrieve the correspond-ing source files our re-implemented experimental results may notbe consistent with the original baselines

The external threat to the validity of the results lies in the general-izability of the results We have tested our method on 10 open sourceJava projects including 14600 files Defect predictions for instancesof other languages such as C C++ etc need to be validated byadditional experiments in the future

Threats to construct validity depend on the appropriateness ofthe evaluation measurement The F-measure is used as our mainevaluation measure which has been applied in many previous effortsto evaluate defect prediction comprehensively

7 CONCLUSIONS AND FUTURE WORKHere we made two main contributions code visualization for defectprediction and an improved deep transfer learning model Our exper-imental results on 10 open source projects show that deep learningcan be effectively applied directly for defect prediction after apply-ing visualization methods to the code Specifically our approachDTL-DP performs at the top of the range of state-of-the-art WPDPapproaches For CPDP DTL-DP improves on the state-of-the-arttechnique TCA+ built on traditional features by 290 (in the F-measure) It also bests the deep learning-based approaches DBN-CPDP-LSTM and DP-CNN by 88 248 and 155 respectively

DTL-DP still has some limitations For some projects a problemof negative transfer occurs resulting in a worse prediction than adirect prediction Large differences between two projects can causesuch negative transfer Reducing the impact of negative transfer isone of the problems to be solved in the future Additionally theamount of training data we had was small and class imbalance isinherent in software defect prediction Both can be improved withmore data which we plan to obtain in the future

ACKNOWLEDGEMENTSThis work is partially supported by Zhejiang Natural Science Foun-dation(LY19F020025) National Natural Science Foundation ofChina (61502423 61572439 61702534) Zhejiang University OpenFund(2018KFJJ07) Signal Recognition Based on GAN Deep Learn-ing for Enhancement Recognition Project Zhejiang Science andTechnology Plan Project (LGF18F030009 2017C33149) and Zhe-jiang Outstanding Youth Fund (LR19F030001)

REFERENCES[1] Romi Satria Wahono A systematic literature review of software defect predic-

tion research trends datasets methods and frameworks Journal of SoftwareEngineering 1(1)1ndash16 2015

[2] Burak Turhan Tim Menzies Ayse B Bener and Justin Di Stefano On the relativevalue of cross-company and within-company data for defect prediction EmpiricalSoftware Engineering 14(5)540ndash578 2009

[3] Lech Madeyski and Marian Jureczko Which process metrics can significantlyimprove defect prediction models an empirical study Software Quality Journal23(3)393ndash422 2015

[4] Thomas Shippey David Bowes and Tracy Hall Automatically identifying codefeatures for software defect prediction Using ast n-grams Information andSoftware Technology 106142ndash160 2019

[5] Zhiyuan Wan Xin Xia Ahmed E Hassan David Lo Jianwei Yin and XiaohuYang Perceptions expectations and challenges in defect prediction IEEETransactions on Software Engineering 2018

[6] Karim O Elish and Mahmoud O Elish Predicting defect-prone software modulesusing support vector machines Journal of Systems and Software 81(5)649ndash6602008

[7] Puja Ahmad Habibi Victor Amrizal and Rizal Broer Bahaweres Cross-projectdefect prediction for web application using naive bayes (case study Petstoreweb application) In 2018 International Workshop on Big Data and InformationSecurity (IWBIS) pages 13ndash18 IEEE 2018

[8] Michael J Siers and Md Zahidul Islam Software defect prediction using a costsensitive decision forest and voting and a potential solution to the class imbalanceproblem Information Systems 5162ndash71 2015

[9] R Jayanthi and Lilly Florence Software defect prediction techniques using metricsbased on neural network classifier Cluster Computing pages 1ndash12 2018

[10] Yi Liu Yu Fan and Junghui Chen Flame images for oxygen content predictionof combustion systems using dbn Energy amp Fuels 31(8)8776ndash8783 2017

[11] Yi Liu Chao Yang Zengliang Gao and Yuan Yao Ensemble deep kernel learn-ing with application to quality prediction in industrial polymerization processesChemometrics and Intelligent Laboratory Systems 17415ndash21 2018

[12] Qi Xuan Binwei Fang Yi Liu Jinbao Wang Jian Zhang Yayu Zheng and Guan-jun Bao Automatic pearl classification machine based on a multistream convolu-tional neural network IEEE Transactions on Industrial Electronics 65(8)6538ndash6547 2018

[13] Qi Xuan Haoquan Xiao Chenbo Fu and Yi Liu Evolving convolutional neuralnetwork and its application in fine-grained visual categorization IEEE Access631110ndash31116 2018

[14] Qi Xuan Zhuangzhi Chen Yi Liu Huimin Huang Guanjun Bao and Dan ZhangMultiview generative adversarial network and its application in pearl classificationIEEE Transactions on Industrial Electronics 66(10)8244ndash8252 2019

[15] Qi Xuan Fuxian Li Yi Liu and Yun Xiang MV-C3D A spatial correlatedmulti-view 3d convolutional neural networks IEEE Access 792528ndash925382019

[16] Song Wang Taiyue Liu and Lin Tan Automatically learning semantic featuresfor defect prediction In Software Engineering (ICSE) 2016 IEEEACM 38thInternational Conference on pages 297ndash308 IEEE 2016

[17] Jian Li Pinjia He Jieming Zhu and Michael R Lyu Software defect predictionvia convolutional neural network In Software Quality Reliability and Security(QRS) 2017 IEEE International Conference on pages 318ndash328 IEEE 2017

[18] Hoa Khanh Dam Trang Pham Shien Wee Ng Truyen Tran John Grundy AdityaGhose Taeksu Kim and Chul-Joo Kim A deep tree-based model for softwaredefect prediction arXiv preprint arXiv180200921 2018

[19] Alex Krizhevsky Ilya Sutskever and Geoffrey E Hinton Imagenet classificationwith deep convolutional neural networks In Advances in neural informationprocessing systems pages 1097ndash1105 2012

[20] Mingsheng Long Yue Cao Jianmin Wang and Michael I Jordan Learning trans-ferable features with deep adaptation networks arXiv preprint arXiv1502027912015

[21] Han Zhang Ian Goodfellow Dimitris Metaxas and Augustus Odena Self-attention generative adversarial networks arXiv preprint arXiv1805083182018

[22] Tian Jiang Lin Tan and Sunghun Kim Personalized defect prediction In Pro-ceedings of the 28th IEEEACM International Conference on Automated SoftwareEngineering pages 279ndash289 IEEE Press 2013

[23] Andrew Meneely Laurie Williams Will Snipes and Jason Osborne Predictingfailures with developer networks and social network analysis In Proceedings ofthe 16th ACM SIGSOFT International Symposium on Foundations of softwareengineering pages 13ndash23 ACM 2008

[24] Raimund Moser Witold Pedrycz and Giancarlo Succi A comparative analysis ofthe efficiency of change metrics and static code attributes for defect prediction InProceedings of the 30th international conference on Software engineering pages181ndash190 ACM 2008

[25] Foyzur Rahman and Premkumar Devanbu How and why process metrics arebetter In Software Engineering (ICSE) 2013 35th International Conference onpages 432ndash441 IEEE 2013

[26] Maurice Howard Halstead et al Elements of software science volume 7 ElsevierNew York 1977

[27] Thomas J McCabe A complexity measure IEEE Transactions on softwareEngineering (4)308ndash320 1976

[28] Shyam R Chidamber and Chris F Kemerer A metrics suite for object orienteddesign IEEE Transactions on software engineering 20(6)476ndash493 1994

[29] Yue Yu Huaimin Wang Gang Yin and Tao Wang Reviewer recommendation forpull-requests in github What can we learn from code review and bug assignmentInformation and Software Technology 74204ndash218 2016

11

ICSE 2020 May 23-29 2020 Seoul South Korea Chen et al

[30] Nachiappan Nagappan and Thomas Ball Using software dependencies and churnmetrics to predict field failures An empirical case study In null pages 364ndash373IEEE 2007

[31] Ahmed E Hassan Predicting faults using the complexity of code changes InProceedings of the 31st International Conference on Software Engineering pages78ndash88 IEEE Computer Society 2009

[32] Thomas J Ostrand Elaine J Weyuker and Robert M Bell Programmer-basedfault prediction In Proceedings of the 6th International Conference on PredictiveModels in Software Engineering page 19 ACM 2010

[33] Taek Lee Jaechang Nam DongGyun Han Sunghun Kim and Hoh Peter In Microinteraction metrics for defect prediction In Proceedings of the 19th ACM SIG-SOFT symposium and the 13th European conference on Foundations of softwareengineering pages 311ndash321 ACM 2011

[34] Tim Menzies Zach Milton Burak Turhan Bojan Cukic Yue Jiang and AyseBener Defect prediction from static code features current results limitationsnew approaches Automated Software Engineering 17(4)375ndash407 2010

[35] Martin Pinzger Nachiappan Nagappan and Brendan Murphy Can developer-module networks predict failures In Proceedings of the 16th ACM SIGSOFTInternational Symposium on Foundations of software engineering pages 2ndash12ACM 2008

[36] Steffen Herbold Alexander Trautsch and Jens Grabowski A comparative studyto benchmark cross-project defect prediction approaches IEEE Transactions onSoftware Engineering 44(9)811ndash833 2018

[37] Chao Liu Dan Yang Xin Xia Meng Yan and Xiaohong Zhang A two-phasetransfer learning model for cross-project defect prediction Information andSoftware Technology 107125ndash136 2019

[38] Jinyin Chen Keke Hu Yitao Yang Yi Liu and Qi Xuan Col-lective transfer learning for defect prediction Neurocomputing 2019doihttpsdoiorg101016jneucom201812091

[39] Jinyin Chen Yitao Yang Keke Hu Qi Xuan Yi Liu and Chao Yang Multiviewtransfer learning for software defect prediction IEEE Access 78901ndash8916 2019

[40] Annibale Panichella Rocco Oliveto and Andrea De Lucia Cross-project defectprediction models Lrsquounion fait la force In Software Maintenance Reengineeringand Reverse Engineering (CSMR-WCRE) 2014 Software Evolution Week-IEEEConference on pages 164ndash173 IEEE 2014

[41] Burak Turhan Tim Menzies Ayse B Bener and Justin Di Stefano On the relativevalue of cross-company and within-company data for defect prediction volume 14pages 540ndash578 Springer 2009

[42] Fayola Peters Tim Menzies and Andrian Marcus Better cross company defectprediction In Proceedings of the 10th Working Conference on Mining SoftwareRepositories pages 409ndash418 IEEE Press 2013

[43] Xin Xia LO David Sinno Jialin Pan Nachiappan Nagappan and Xinyu WangHydra Massively compositional model for cross-project defect prediction IEEETransactions on software Engineering 42(10)977 2016

[44] Ying Ma Guangchun Luo Xue Zeng and Aiguo Chen Transfer learning forcross-company software defect prediction Information and Software Technology54(3)248ndash256 2012

[45] Jaechang Nam Sinno Jialin Pan and Sunghun Kim Transfer defect learningIn Software Engineering (ICSE) 2013 35th International Conference on pages382ndash391 IEEE 2013

[46] Sinno Jialin Pan Ivor W Tsang James T Kwok and Qiang Yang Domain adap-tation via transfer component analysis IEEE Transactions on Neural Networks22(2)199ndash210 2011

[47] Blaine A Price Ronald M Baecker and Ian S Small A principled taxonomy ofsoftware visualization Journal of Visual Languages amp Computing 4(3)211ndash2661993

[48] Thomas Ball and Stephen G Eick Software visualization in the large Computer29(4)33ndash43 1996

[49] Sarita Bassil and Rudolf K Keller Software visualization tools Survey andanalysis In Proceedings 9th International Workshop on Program ComprehensionIWPC 2001 pages 7ndash17 IEEE 2001

[50] Stephan Diehl Software visualization visualizing the structure behaviour andevolution of software Springer Science amp Business Media 2007

[51] Michele Lanza The evolution matrix Recovering software evolution using soft-ware visualization techniques In Proceedings of the 4th international workshopon principles of software evolution pages 37ndash42 ACM 2001

[52] Denis Gracanin Krešimir Matkovic and Mohamed Eltoweissy Software vi-sualization Innovations in Systems and Software Engineering 1(2)221ndash2302005

[53] Guillaume Langelier Houari Sahraoui and Pierre Poulin Visualization-basedanalysis of quality for large-scale software systems In Proceedings of the 20thIEEEACM international Conference on Automated software engineering pages214ndash223 ACM 2005

[54] Abien Fred Agarap and Francis John Hill Pepito Towards building an intelligentanti-malware system a deep learning approach using support vector machine(svm) for malware classification arXiv preprint arXiv180100318 2017

[55] Karl Weiss Taghi M Khoshgoftaar and DingDing Wang A survey of transferlearning Journal of Big Data 3(1)9 2016

[56] Rahul Krishna and Tim Menzies Bellwethers A baseline method for transferlearning IEEE Transactions on Software Engineering 2018

[57] Chuanqi Tan Fuchun Sun Tao Kong Wenchang Zhang Chao Yang and ChunfangLiu A survey on deep transfer learning In ICANN 2018

[58] Wenyuan Dai Qiang Yang Gui Rong Xue and Yong Yu Boosting for transferlearning In International Conference on Machine Learning pages 193ndash2002007

[59] Chang Wan Rong Pan and Jiefei Li Bi-weighting domain adaptation for cross-language text classification In IJCAI Proceedings-International Joint Conferenceon Artificial Intelligence volume 22 page 1535 2011

[60] Yonghui Xu Sinno Jialin Pan Hui Xiong Qingyao Wu Ronghua Luo HuaqingMin and Hengjie Song A unified framework for metric transfer learning IEEETrans Knowl Data Eng 29(6)1158ndash1171 2017

[61] Chetak Kandaswamy Luiacutes M Silva Luiacutes A Alexandre and Jorge M SantosDeep transfer learning ensemble for classification In Ignacio Rojas GonzaloJoya and Andreu Catala editors Advances in Computational Intelligence pages335ndash348 Cham 2015 Springer International Publishing

[62] Maxime Oquab Leon Bottou Ivan Laptev and Josef Sivic Learning and trans-ferring mid-level image representations using convolutional neural networks InProceedings of the IEEE conference on computer vision and pattern recognitionpages 1717ndash1724 2014

[63] Yann LeCun Leacuteon Bottou Yoshua Bengio and Patrick Haffner Gradient-basedlearning applied to document recognition Proceedings of the IEEE 86(11)2278ndash2324 1998

[64] Hana Ajakan Pascal Germain Hugo Larochelle Franccedilois Laviolette and MarioMarchand Domain-adversarial neural networks arXiv preprint arXiv141244462014

[65] Yaroslav Ganin and Victor Lempitsky Unsupervised domain adaptation by back-propagation arXiv preprint arXiv14097495 2014

[66] Jing Zhang Wanqing Li and Philip Ogunbona Joint geometrical and statisticalalignment for visual domain adaptation arXiv preprint arXiv170505498 2017

[67] Eric Tzeng Judy Hoffman Ning Zhang Kate Saenko and Trevor DarrellDeep domain confusion Maximizing for domain invariance arXiv preprintarXiv14123474 2014

[68] Mingsheng Long Yue Cao Zhangjie Cao Jianmin Wang and Michael I Jor-dan Transferable representation learning with deep adaptation networks IEEEtransactions on pattern analysis and machine intelligence 2018

[69] Arthur Gretton Dino Sejdinovic Heiko Strathmann Sivaraman BalakrishnanMassimiliano Pontil Kenji Fukumizu and Bharath K Sriperumbudur Optimalkernel choice for large-scale two-sample tests In Advances in neural informationprocessing systems pages 1205ndash1213 2012

[70] Mingsheng Long Han Zhu Jianmin Wang and Michael I Jordan Deep transferlearning with joint adaptation networks arXiv preprint arXiv160506636 2016

[71] Martin Arjovsky Soumith Chintala and Leacuteon Bottou Wasserstein gan arXivpreprint arXiv170107875 2017

[72] Kesav Kancherla and Srinivas Mukkamala Image visualization based malwaredetection In 2013 IEEE Symposium on Computational Intelligence in CyberSecurity (CICS) pages 40ndash44 IEEE 2013

[73] Xin Zhou Jianmin Pang and Guanghui Liang Image classification for malwaredetection using extremely randomized trees In 2017 11th IEEE InternationalConference on Anti-counterfeiting Security and Identification (ASID) pages54ndash59 IEEE 2017

[74] John Murphy An overview of convolutional neural network architectures for deeplearning 2016

[75] Song Wang Taiyue Liu Jaechang Nam and Lin Tan Deep semantic feature learn-ing for software defect prediction IEEE Transactions on Software Engineering2018

[76] Laurens van der Maaten and Geoffrey Hinton Visualizing data using t-sneJournal of machine learning research 9(Nov)2579ndash2605 2008

12

• Abstract
• 1 Introduction
• 2 Related Work
• 3 Approach
• • 31 Source Code Visualization
  • 32 Modeling with Deep Transfer Learning
  • 33 Model Sensitivity to Parameter Choices
  • 34 Training and Prediction
  • • 4 Experimental Setup
    • • 41 Dataset Description
      • 42 Baseline Comparison Methods
      • 43 Performance measures
      • • 5 Results
        • • 51 RQ1 How does DTL-DP compare to feature- based machine learning methods and AST- based deep learning methods in WPDP
          • 52 RQ2 How does DTL-DP compare to feature- based Machine Learning and AST-based deep learning methods in CPDP
          • 53 RQ3 How much does each of the three mechanisms ie data augmentation transfer learning and self-attention mechanism contribute to DTL-DPs performance
          • • 6 Threats to Validity
            • 7 Conclusions and Future Work
            • References

Software Visualization and Deep Transfer Learningfor Effective Software Defect Prediction ICSE 2020 May 23-29 2020 Seoul South Korea

Figure 4 The training and testing process of our approach DTL-DP inspired by the original DAN paper [68]

Table 1 Image Width for Various File Sizes

File Size Range Image Width

lt10 kB 3210 kB - 30 kB 6430 kB - 60 kB 12860 kB - 100 kB 256

100 kB - 200 kB 384200 kB - 500 kB 512500 kB - 1000 kB 768

gt1000kB 1024

6 image vectors into image matrices of width and length such thatwidth lowast lenдth = vector_size3 We note that the first convolutionkernel of AlexNet is of size 1111 so if the image is too narrowthe convolution performance will be poor And the image with asuitable width can be convolved to obtain more efficient semanticand structural features in contexts with a proper sequence lengthTable 1 gives some recommended image widths for different filesizes based on previous work [72 73] We adopt those in this work

32 Modeling with Deep Transfer LearningNext we describe our approach DTL-DP The goal is to learn trans-ferable features across projects and build a classifier y = ϕ(x) offile defectiveness with source code supervision where x is the rep-resentation of the sample and y is the predicted label In the defectprediction problem we are given a source project Ps = (xsi y

si )

nsi=1

with ns labeled instances and a target project Pt = (xti yti )

nti=1

with nt unlabeled instances In WPDP the source project is the pre-vious version of the target project while in CPDP the source projectis a related project of the target project The sample distributions pand q in the source and target projects are often similar in WPDPbut different in CPDP

For our deep network architecture we adopt a similar architec-ture as that of deep adaptation networks (DANs) [68] to capturethe semantic and structural information of the source code To theoriginal DAN model we add an attention layer to further enhancethe expressive ability of important features The overall architectureis illustrated in Figure 4 In particular our DTL-DP consists of aninput layer five convolution layers (conv1 minus conv5 from AlexNet)an Attention-Layer and finally four fully-connected hidden layers(f c6 minus f c9) working as a classifier The structure and parameters of

(conv1minusconv5) and (f c6minus f c8) are consistent with a DAN But sincedefect prediction is a binary classification problem a fully-connectedlayer (f c9) is added to obtain a binary result at the end

We adopt the defaults for AlexNet so the input to our DTL-DPmust be a 224224 size image cropped from a three-channel (RGB)image of size 256256 The code image is placed in the approximatecenter of the 256256 image (the determination of the size of eachcode image is described later in Sect 33) We note that the codeimage can be smaller than 224224 pixels due to the variance in thesize of the code files If this happens the image is padded aroundwith blank (zero valued) pixels Padding should not negatively effectthe qualitative performance of feature detection in the images in factfor deeper DL architectures like ours padding has been shown toprovide extra contrast to the embedded image for each of the layersas well as buffering against data loss by each layer [74]

Training deep models requires a significant amount of labeleddata which is not available for a new project so we first pre-trainedan AlexNet model on ImageNet 2012 Unlike a DAN which freezesconv1minusconv3 and fine-tunes conv4minusconv5 we fine tune all convolu-tion layers conv1minusconv5 by taking the parameters of the pre-trainedmodels as initial parameters that we then optimize during the train-ing phase We do that to minimize the differences between our codeimages and the actual object images in ImageNet 2012

In order for DTL-DP to focus on the key features in differentdefect images and thus further improve prediction we employ aself-attention mechanism into our model inspired by the good per-formance of self-attention in GANs [21] As shown in the AttentionLayer in Figure 4 the attention mechanism makes the feature mapgenerated by layer conv5 be the self-attention feature map input tothe next layer fc6 Specifically at first it linearly maps the inputfeatures x (it is a 11 convolution to compress the number of chan-nels ie out_chanels = in_chanels8) and produces f (x)д(x)and h(x) where f (x) = wf x д(x) = wдx h(x) = whx The dif-ference between the three is that the size of h(x) is still the sameas x but the other two are not Thus if the width of x is W theheight H and the number of channelsC the size of x is [CN ] whereN =W lowast H the size of f(x) and g(x) is [C8N ] but the size of h(x)is [CN ] The transposed f (x) and д(x) are matrix-multiplied to ob-tain the autocorrelation in the features ie the relationship of eachpixel to all other pixels where Si j = f (xi )

Tд(x j ) Then softmax isapplied to the autocorrelation features S to get the attention mapcomprised of weights with values between 0 and 1

α ji =exp(Si j )sumNi=1 exp(Si j )

(1)

5

ICSE 2020 May 23-29 2020 Seoul South Korea Chen et al

005 01 02 05 1 2 5MMD Penalty

055

056

057

058

059

060

061

062

063

064

065

F-m

easu

re

WPDP CPDP

(a) Effect of different MMD Penalty λ

BRG BGR GRB GBR RGB RBG040

045

050

055

060

065

070

075

F-m

easu

re

WPDP CPDP

(b) Effect of color channel order

8 4 2 1 12 14 18Width recommended

040

045

050

055

060

065

070

075

F-m

easu

re

WPDP CPDP

(c) Effect of code image widthrecommended ratio

Figure 5 Sensitivity studies to aid in choosing the parameters in the model

After that the output of the AttentionLayer is the self-attentionfeature map o = (o1o2 oj oN ) where

oj =Nsumi=1

α jih(xi ) (2)

Then each fully connected layer learns a nonlinear mappinghli = f l (wlhli + b

l ) where hli is the l th layer hidden representationof feature xi wl and bl are the weights and biases of the l th layerand f l is the activation function using ReLU (f l (o) = max(0o))for f c6 minus f c8 and softmax for f c9 If we let Θ denote the set of allDTL-DP parameters the empirical risk of DTL-DP then is

minΘ

1n

nsumi=1

F (ϕ(oi )yi ) (3)

where F is the cross-entropy loss function n is the number of in-stances and ϕ(oi ) is the conditional probability that the DTL-DPassigns oi to label yi It is used to calculate the final loss in Equation7

To make the distribution of the source and target projects similarthe same multi-layer adaptation and multi-kernel MMD strategy asin a DAN [68] are used in our model The feature mapping functionσ is defined as the combination ofm positive semi-definite kernelsku

lt σ (os )σ (ot ) gt= k(os ot ) (4)

k =msumu=1

βuku (5)

where β ge 0 are the weights of the kernelsThe MK-MMD dk (pq) of the probability distributions p and q

is defined as the reproducing kernel Hilbert space distance betweenthe mean embedding of p and q The square formula of MK-MMDis defined as

d2k (pq) =∥ Ep [σ (xs )] minus Eq [σ (x

t )] ∥2 (6)

We see then that the smaller dk (pq) is the more similar p and qare If dk is 0 the distribution of the target project is the same as thatof the source project So the final loss function for train DPL-DP is

1n

nsumi=1

F (ϕ(oi )yi ) + λ

l2suml=l1

d2k (Dls D

lt ) (7)

where Dls is the l th layer hidden representation for the source and

target d2k (Dls D

lt ) is the MK-MMD between the source and target

evaluated on the l th layer representation λ is a penalty parameterand l1 and l2 are f c layers to calculate the MK-MMD between sourceand target In our implementation of DTL-DP we set λ = 1 l1 = 6and l2 = 9 as per the original DAN work [68]

33 Model Sensitivity to Parameter ChoicesWe have made a number of choices to make our modeling plat-form work effectively Here we justify those choices by presentingsensitivity studies For this analysis we used all the data

Hyperparameter λ The DTL-DP model has a hyper-parameter λfor the MMD penalty that adjusts the final loss We conducted λ pa-rameter sensitivity experiments in both the WPDP and CPDP settingWe fix the other parameters and range λ in 001 005 01 02 05 1 2 5The results are shown in Figure 5(a) While fairly constant across therange small variations exist In CPDP the F-measure first increasesand then decreases along the hyper-parameter λ In WPDP λ affectsthe performance of DTL-DP but in the opposite direction We choseλ = 1 here

Different Color Orders The selection of the RGB permutationsin the target project images may also potentially affect the outcomeWe performed experiments with all six different orderings Whentesting them we set λ to 1 and changed the image type of the tar-get project to each one in RGBRBGBRGBGRGBRGRB Theresults are shown in Figure 5(b) The target images in different colororders have somewhat different performance But overall the valuesare close to DTL-DPrsquos average performance

Different Image Widths To explore the impact of different choicesfor image widths on the results we performed additional experi-ments

For our code images 3 bytes of source code are needed to form 1pixel each byte for one of the R G and B channels The sizes ofthe source files are between 0 and 100 kb corresponding to imageswith pixel count between 0 and 34 133 (10010243) The size ofthe image is (widthheiдht) where width is obtained from Table 1according to the size of the code file (size)

heiдht =size lowast 10243 lowastwidth

(8)

For example the size of image converted from a 20 kb code file is(64 107)

We experimented with image widths of 18 14 12 2 4 and 8times of the recommended widths in Table 1 The larger the mul-tiplier the wider the image For images of different widths we

6

Software Visualization and Deep Transfer Learningfor Effective Software Defect Prediction ICSE 2020 May 23-29 2020 Seoul South Korea

repeated the previous experiment and obtained results for the meanF-measure shown in Figure 5(c) We see that the closer to the recom-mended width from Table 1 the better the average result In additionwe found that a wider image is better than a narrower one And thecloser the image is to a square the better the prediction Thereforewe made the generated images be as close to a square as possible

34 Training and PredictionWhen training the DTL-DP code images are generated from thelabeled instances in the source project and the unlabeled instances inthe target project and then are simultaneously input into the modelThey share the convolution layers conv1minusconv5 andAttentionLayerto extract their respective features and calculate the MK-MMDbetween source and target projects in the fully connected layersf c6 minus f c9 It should be noted that we only calculate cross entropyfor the source project because the target projectrsquos labels are not pro-vided We use a mini-batch stochastic gradient descent along the lossfunction in order to train the parameters of the entire model We trainfor 500 epochs for each pair of source and target projects then pickthe epoch with the best F-measure (described in section 43) fromwhich to read out the parameters of the final model Finally the filesfrom the target project that need to be predicted are converted intocode images and then input into the trained model for classification

Overfitting is always a possibility with such pipelines We moder-ate it here with our choices of (1) augmenting the dataset by using sixcolor channel permutations described in section 31 (2) selecting asimple AlexNet model structure and (3) using the ImageNet 2012pre-trained model to reduce fluctuations

4 EXPERIMENTAL SETUPWe conducted experiments to asses the performance of DTL-DP andto compare it with existing deep learning-based defect predictionapproaches for both within-project WPDP and cross-project CPDPdefect prediction We ran experiments on a modern-day Linux serverwith 3 Titan XP GPUs Unless otherwise stated each experimentwas run 10 times and the average results are reported

41 Dataset DescriptionIn order to directly compare our work with prior research we usedpublicly available data from the PROMISE3 data repository whichhas been widely used in defect prediction work [16ndash18 75] Weselected all open source Java projects from this repository andcollected their version number class names and the buggy labelfor each file Based on the version number and class name weobtained the source code for each file from GitHub4 and fed it toour end-to-end framework In total data for 10 Java projects werecollected Table 2 shows the details of these projects includingproject description versions the total and average number of filesand the defect rate It should be noted that the average numberof files over all projects ranges between 150 and 1046 and thedefect rates of the projects have a minimum value of 134 and amaximum value of 497 The number of files in some projects isnot sufficient to train deep models and the classes are imbalancedthus augmentation is needed as described above

3httpopenscienceusrepodefect4httpsgithubcomapache

42 Baseline Comparison MethodsTo evaluate the performance of our end-to-end framework DTL-DPfor defect prediction we compare it with the following baselinemethods in the WPDP setting

bull Semantic [16 75] the state-of-the-art method which em-ploys deep belief networks DBN on source code to extractsemantic features for defect prediction

bull PROMISE-DP [16] a traditional method which builds analternating decision tree ADTree classifier based on theoriginal 20 features of the PROMISE dataset

bull DP-LSTM [18] a long short-term memory LSTM-baseddeep neural network model which uses ASTs to representsource files and predict whether the file is defective or not

bull DP-CNN [17] a convolutional neural network CNN-basedmodel which is seeded by AST-derived numerical vectorsto automatically learn semantic and structural features ofprograms The CNN-learned features are used to train thefinal classifier in combination with traditional features

For the cross-project settings CPDP Semantic and PROMISE-DPcould not be used directly Instead we used the following

bull DBN-CP [16 75] a variant of Semantic which trains a DBNby using the source project and generates semantic featuresfor both the source and target projects

bull TCA+ [45] the state-of-the-art technique for CPDP

To obtain the training and test data we followed the processesestablished in [16] For WPDP we use two consecutive versions ofeach project listed in Table 2 The older version is used to generatethe training data and the more recent version is used as test data ForCPDP we pick versions randomly from each project for 11 targetprojects And for each target project we select 2 source projects thatare different from the target projects We use the same 22 test pairsas in [16] When implementing the baseline methods we use thesame network architecture and parameter settings as described in thepapers that introduced them

43 Performance measuresTo evaluate the prediction performance we use the F-measure awidely adopted metric in the literature [16ndash18 43 45] The F-measure captures a predictorrsquos accuracy and combines both precisionand recall for a comprehensive evaluation of predictive performance

Specifically a prediction that a file is defective is called a truepositive (TP) if the file is in fact defective and false positive (FP)otherwise Similarly a prediction that a file is not defective is a truenegative (TN) if the file is in fact not defective and false negative(FN) otherwise Then the precision (P) recall (R) and F-measureare defined as

P = TP(TP + FP) (9)

R = TP(TP + FN ) (10)

F = (2 times P times R)(P + R) (11)

5 RESULTSThis section discusses our results of comparing DTL-DP to baselinetools for defect prediction

7

ICSE 2020 May 23-29 2020 Seoul South Korea Chen et al

Table 2 Dataset Description

Project Description Versions Files Avg files Avg size(kb) Defective

ant Java based build tool 151617 1465 488 62 134camel Enterprise integration framework 121416 3140 1046 29 187jEdit Text editor designed for programmers 324041 1935 645 87 192log4j Logging library for Java 1011 300 150 34 497

lucene Text search engine library 202224 607 402 38 358xalan A library for transforming XML files 2425 1984 992 46 296xerces XML parser 1213 1647 549 29 157

ivy Dependency management library 1420 622 311 41 200synapse Data transport adapters 101112 661 220 38 227

poi Java library to access Microsoft format files 152530 1248 416 36 407

Table 3 F-measure of DTL-DP Semantic (Seman) PROMISE-DP (PROM) DP-LSTM (LSTM) and DP-CNN (CNN) forWPDP

Project Version Seman PROM LSTM CNN DTL-DP

ant15-gt16 914 477 452 532 70016-gt17 942 542 379 566 593

camel12-gt14 785 373 323 461 40614-gt16 374 391 400 508 385

jEdit32-gt40 574 556 472 564 49440-gt41 615 546 490 580 687

log4j 10-gt11 701 587 539 632 688

lucene20-gt22 651 502 751 761 78322-gt24 773 605 751 721 776

xalan 24-gt25 595 518 657 601 794xerces 12-gt13 411 238 268 374 820

ivy 14-gt20 350 329 191 347 829

synapse10-gt11 544 476 454 539 54711-gt12 583 533 533 556 659

poi15-gt25 640 558 808 589 68525-gt30 803 754 777 784 426

Average 641 499 515 570 642

51 RQ1 How does DTL-DP compare to feature-based machine learning methods and AST-based deep learning methods in WPDP

We compare DTL-DP to 4 baseline approaches representing twodifferent kinds of defect prediction methods PROMISE-DP is thebaseline representative of traditional feature-based machine learningmethods Semantic DP-LSTM and DP-CNN are the baselines fordeep learning-type methods based on extracting features from ASTGuided by prior work we conducted 16 sets of WPDP experimentseach using two versions of the same project The older version isused to train the prediction model and the newer version is used toevaluate the trained model

Table 3 shows the F-measure values for the within-project WPDPdefect prediction experiments The highest F-measure values of the5 methods are shown in bold Since the methods based on deeplearning include some randomness we run DTL-DP Semantic DP-LSTM and DP-CNN 10 times for each experiment On average

the F-measure of our approach is 0642 and the PROMISE-DPSemantic DP-LSTM and DP-CNN achieve 0499 0641 0515 and0570 respectively The results demonstrate that our approach iscompetitive and may improve on defect prediction compared toPROMISE-DP DP-LSTM and DP-CNN The results of Semanticand our approach are similar

The proposed DTL-DP is effective and it could improvethe performance of WPDP tasks Like other deep learners it issensitive to small file sizes and unbalanced data

511 Case Study WPDP Discrimination of DTL-DP t-SNEis a non-linear dimensional reduction algorithm that is effectivein visualizing similarities and helping identify clusters in complexdata sets [76] To give insight in the performance of DTL-DP wedemonstrate feature transferability by showing t-SNE embeddingsin Figure 6 The blue points are non-defective files and the red aredefective ones5 We observe the following (1) The target instancesare not discriminated very well using either the traditional manualfeatures or the TCA+ improved manual features or the semanticfeatures extracted from ASTs while with our approach the pointsare discriminated much better (2) With the other three approachesthe categories between source and target projects are not well alignedwhile with our approach the categories between the projects aremore consistent These conclusions are derived from the intra- andinter-class distances of the two categories in Figures 6 and 7 Theyare visually apparent

Our method did not perform as well as the comparison methods onsome projects such as ant which is likely caused by large variancein file sizes While the sizes of ant-15 and ant-16 are close there isa marked difference between ant-16 and ant-17 the former beingmuch smaller than the latter From Table 3 the performance of ourmethod on ant15-gtant16 is better than that on ant16-gtant17On the contrary baseline methods other than DP-LSTM performbetter for the task ant16-gtant17 notably Semantic is dominantindicating that the semantic feature-based method is more robust tofile size variability Moreover the ant dataset has two shortcomingsthe first is that the amount of data is small cf Table 2 where theaverage amount is only 488 files in one project The second is that

5Since we use data augmentation we have more samples than the three baselines

8

Software Visualization and Deep Transfer Learningfor Effective Software Defect Prediction ICSE 2020 May 23-29 2020 Seoul South Korea

Figure 6 t-SNE mapping of source and target features (WPDP)

the classes are unbalanced and the proportion of defective files is134 which is the least of all our projects This makes training ofthe deep model more difficult which leads to the poorer performanceof our method on some projects

Table 4 F-measure of DTL-DP DBN-CP (DBN) TCA+ DP-LSTM (LSTM) and DP-CNN (CNN) in CPDP

Source Target DBN TCA+ LSTM CNN DTL-DPant16 camel14 316 292 321 323 395

jEdit41 camel14 693 330 318 651 407camel14 ant16 979 616 448 607 591

poi30 ant16 478 598 386 532 693camel14 jEdit41 615 537 394 547 531log4j11 jEdit41 503 419 389 423 639jEdit41 log4j11 645 574 574 656 783

lucene22 log4j11 618 571 578 632 794lucene22 xalan25 550 530 680 540 689xerces13 xalan25 572 581 676 562 686xalan25 lucene22 594 561 750 621 783log4j11 lucene22 692 524 750 663 769xalan25 xerces13 386 394 340 391 400ivy20 xerces13 426 398 261 421 420

xerces13 ivy20 453 409 264 467 472synapse12 ivy20 824 383 261 371 494

ivy14 synapse11 489 348 451 491 545poi25 synapse11 425 376 435 436 597ivy20 synapse12 433 570 530 456 620poi30 synapse12 514 542 503 532 623

synapse12 poi30 661 651 785 671 827ant16 poi30 619 343 785 627 827

Average 568 479 495 528 618

52 RQ2 How does DTL-DP compare to feature-based Machine Learning and AST-based deeplearning methods in CPDP

Here we compare to TCA+ and DBN-CP instead of PROMISEand Semantic as the baseline approaches as explained in Sect 42

Again guided by prior work we conducted 22 sets of CPDP experi-ments In each we randomly select two project versions from twodifferent projects one as a training set and the other as a test set

Table 4 presents the F-measure results of DTL-CP and the 4 base-line approaches The highest F-measure values are in bold On aver-age the F-measure of our approach in CPDP is 0618 and the DBN-CP TCA+ DP-LSTM and DP-CNN achieve 0568 0479 0495 and0528 Thus DTL-DP outperforms them by 88 290 248 and155 respectively In addition we found that for projects log4j11and poi30 our method does better than the corresponding WPDPbest performing method

Our proposed DTL-DP shows significant improvements onthe state-of-the-art in cross-project defect prediction

521 Case Study CPDP Discrimination of DTL-DP Ourmethod is more obviously dominant in CPDP than in WPDP Apossible reason for that is that deep transfer learning makes the dis-tributions of the training and test samples more similar in featurespace Another reason might be the superior ability of deep modelsto represent features enabling the model to obtain more transferablefeatures from the images To gain more insight we choose the taskpoi30 rarr ant16 and show the t-SNE embeddings in Figure 7 Wemake similar observations as in the RQ1 case study that (1) the tar-get instances are more easily discriminated with our approach and(2) the target instances can be better discriminated with the sourceclassifier This implies that our approach can learn more transferablefeatures for more effective defect prediction

Figure 7 t-SNE mapping of source and target features (CPDP)

53 RQ3 How much does each of the threemechanisms ie data augmentation transferlearning and self-attention mechanismcontribute to DTL-DPrsquos performance

To find out the specific contributions of the three parts to defectprediction we conducted further experiments We built the originalAlexNet model for binary classification and use it as the Base +TL+Attention and +DataAug are three new baselines built by adding tothe base AlexNet one of three mechanisms (transfer learning selfattention and data augmentation) respectively It should be noted

9

ICSE 2020 May 23-29 2020 Seoul South Korea Chen et al

that in addition to +DataAug we use images generated by onesequence of R G and B as training and test sets in each experimentTherefore the experimental results of Base +TL and +Attention inTables 5 and 6 are average results obtained over the 6 different RGBpermutations as described above

Table 5 Contributions of the three mechanisms to WPDP

Project Version Base +TL +Atten +Aug DTL-DP

ant15-gt16 660 676 664 677 70016-gt17 525 552 524 591 593

camel12-gt14 415 381 416 403 40614-gt16 412 396 423 384 385

jEdit32-gt40 484 505 497 485 49440-gt41 629 649 645 679 687

log4j 10-gt11 688 679 692 658 688

lucene20-gt22 667 773 778 761 78322-gt24 749 713 769 758 776

xalan 24-gt25 757 754 758 778 794xerces 12-gt13 809 822 800 788 820

ivy 14-gt20 820 824 827 812 829

synapse10-gt11 525 484 511 530 54711-gt12 618 608 612 616 659

poi15-gt25 663 593 654 614 68525-gt30 410 449 420 432 426

Average 614 616 624 623 642

Table 6 Contributions of the three mechanisms to CPDP

Source Target Base +TL +Atten +Aug DTL-DPant16 camel14 383 379 365 377 395

jEdit41 camel14 368 394 395 371 407camel14 ant16 670 667 653 596 591

poi30 ant16 606 627 634 593 693camel14 jEdit41 663 657 652 647 531log4j11 jEdit41 599 599 592 594 639jEdit41 log4j11 754 800 774 746 783

lucene22 log4j11 759 750 753 783 794lucene22 xalan25 661 664 650 647 689xerces13 xalan25 510 638 626 662 686xalan25 lucene22 730 751 710 769 783log4j11 lucene22 713 745 740 727 769xalan25 xerces13 387 355 393 373 400ivy20 xerces13 413 408 398 378 420

xerces13 ivy20 518 500 504 500 472synapse12 ivy20 521 490 509 476 494

ivy14 synapse11 551 581 553 536 545poi25 synapse11 508 541 516 545 597ivy20 synapse12 604 622 619 611 620poi30 synapse12 600 632 613 612 623

synapse12 poi30 770 804 781 817 827ant16 poi30 777 662 700 812 827

Average 594 603 597 599 618

Table 7 Time cost of the three mechanisms defect visualization(Visualiz) self-attention (Attent) and transfer learning (TL)

ProjectTime (s)

TL Attent Visualizant 4504 131 459

camel 9655 242 785jEdit 5915 266 497log4j 1371 131 176

lucene 3739 186 354xalan 9444 271 823xerces 5117 214 275

ivy 2902 142 305synapse 2042 126 209

poi 3865 157 221Average 4856 187 410

To discern the contribution of each mechanism we compare theperformance of these baseline methods in WPDP and CPDP Fromthe results in Table 5 and Table 6 we observe that the three mecha-nisms each contribute substantially to the accuracy of DTL-DP inboth WPDP and CPDP In terms of the F-measure transfer learn-ing contributes the least improvement and self-attention and dataaugmentation contribute similarly to the final result But in CPDPtransfer learning contributes to the F-measure the most This islikely because of the applicability of transfer learning in the CPDPsetting and is in a way a validation of the approach

We also noted the time cost for the 3 mechanisms in our proposedDTL-DP Table 7 shows the result The most time spent during dataaugmentation is on converting code into images ie source codevisualization For transfer learning most time is spent on the MK-MMD calculation and for self-attention on the calculation of theattention layer in Fig 4 Eg for the project ant Table 3 shows twosets of WPDP experiments ant 15 rarr 16 and ant 16 rarr 17 Onaverage it takes 4504 seconds 1301 seconds and 459 seconds forthe 3 parts respectively for both the training data and the test dataTransfer learning takes the longest time more than the sum of theother two This is because of the large number of matrices neededto calculate MMD with the kernel function The least time cost isincurred by the attention mechanism and its contribution to WPDPis the largest of the three ie it is most cost-effective

The three mechanisms all contribute toward the accuracy ofour proposed end-to-end framework in WPDP and CPDP Self-attention has the greatest contribution to WPDP and transferlearning contributes most to CPDP

6 THREATS TO VALIDITYThreats to internal validity come from experimental errors and thereplication of the baseline methods In order to compare and analyzethe deep learning-based defect prediction techniques we compareour proposed DTL-DP method with Sementic DBN-CP DP-LSTMand DP-CNN In addition our method is also compared with thetransfer learning-based method TCA+ which is the state-of-the-art CPDP technique Since the original implementations were not

10

Software Visualization and Deep Transfer Learningfor Effective Software Defect Prediction ICSE 2020 May 23-29 2020 Seoul South Korea

available we re-implemented our own versions of the baselinesAlthough we strictly follow the procedures described in their workour new implementations may not completely restore all of theiroriginal implementation details And the randomness of the deeplearning-based approach also makes the results of our implementedexperiments different from the original Since we have removed theentries in the PROMISE dataset that cannot retrieve the correspond-ing source files our re-implemented experimental results may notbe consistent with the original baselines

The external threat to the validity of the results lies in the general-izability of the results We have tested our method on 10 open sourceJava projects including 14600 files Defect predictions for instancesof other languages such as C C++ etc need to be validated byadditional experiments in the future

Threats to construct validity depend on the appropriateness ofthe evaluation measurement The F-measure is used as our mainevaluation measure which has been applied in many previous effortsto evaluate defect prediction comprehensively

7 CONCLUSIONS AND FUTURE WORKHere we made two main contributions code visualization for defectprediction and an improved deep transfer learning model Our exper-imental results on 10 open source projects show that deep learningcan be effectively applied directly for defect prediction after apply-ing visualization methods to the code Specifically our approachDTL-DP performs at the top of the range of state-of-the-art WPDPapproaches For CPDP DTL-DP improves on the state-of-the-arttechnique TCA+ built on traditional features by 290 (in the F-measure) It also bests the deep learning-based approaches DBN-CPDP-LSTM and DP-CNN by 88 248 and 155 respectively

DTL-DP still has some limitations For some projects a problemof negative transfer occurs resulting in a worse prediction than adirect prediction Large differences between two projects can causesuch negative transfer Reducing the impact of negative transfer isone of the problems to be solved in the future Additionally theamount of training data we had was small and class imbalance isinherent in software defect prediction Both can be improved withmore data which we plan to obtain in the future

ACKNOWLEDGEMENTSThis work is partially supported by Zhejiang Natural Science Foun-dation(LY19F020025) National Natural Science Foundation ofChina (61502423 61572439 61702534) Zhejiang University OpenFund(2018KFJJ07) Signal Recognition Based on GAN Deep Learn-ing for Enhancement Recognition Project Zhejiang Science andTechnology Plan Project (LGF18F030009 2017C33149) and Zhe-jiang Outstanding Youth Fund (LR19F030001)

REFERENCES[1] Romi Satria Wahono A systematic literature review of software defect predic-

tion research trends datasets methods and frameworks Journal of SoftwareEngineering 1(1)1ndash16 2015

[2] Burak Turhan Tim Menzies Ayse B Bener and Justin Di Stefano On the relativevalue of cross-company and within-company data for defect prediction EmpiricalSoftware Engineering 14(5)540ndash578 2009

[3] Lech Madeyski and Marian Jureczko Which process metrics can significantlyimprove defect prediction models an empirical study Software Quality Journal23(3)393ndash422 2015

[4] Thomas Shippey David Bowes and Tracy Hall Automatically identifying codefeatures for software defect prediction Using ast n-grams Information andSoftware Technology 106142ndash160 2019

[5] Zhiyuan Wan Xin Xia Ahmed E Hassan David Lo Jianwei Yin and XiaohuYang Perceptions expectations and challenges in defect prediction IEEETransactions on Software Engineering 2018

[6] Karim O Elish and Mahmoud O Elish Predicting defect-prone software modulesusing support vector machines Journal of Systems and Software 81(5)649ndash6602008

[7] Puja Ahmad Habibi Victor Amrizal and Rizal Broer Bahaweres Cross-projectdefect prediction for web application using naive bayes (case study Petstoreweb application) In 2018 International Workshop on Big Data and InformationSecurity (IWBIS) pages 13ndash18 IEEE 2018

[8] Michael J Siers and Md Zahidul Islam Software defect prediction using a costsensitive decision forest and voting and a potential solution to the class imbalanceproblem Information Systems 5162ndash71 2015

[9] R Jayanthi and Lilly Florence Software defect prediction techniques using metricsbased on neural network classifier Cluster Computing pages 1ndash12 2018

[10] Yi Liu Yu Fan and Junghui Chen Flame images for oxygen content predictionof combustion systems using dbn Energy amp Fuels 31(8)8776ndash8783 2017

[11] Yi Liu Chao Yang Zengliang Gao and Yuan Yao Ensemble deep kernel learn-ing with application to quality prediction in industrial polymerization processesChemometrics and Intelligent Laboratory Systems 17415ndash21 2018

[12] Qi Xuan Binwei Fang Yi Liu Jinbao Wang Jian Zhang Yayu Zheng and Guan-jun Bao Automatic pearl classification machine based on a multistream convolu-tional neural network IEEE Transactions on Industrial Electronics 65(8)6538ndash6547 2018

[13] Qi Xuan Haoquan Xiao Chenbo Fu and Yi Liu Evolving convolutional neuralnetwork and its application in fine-grained visual categorization IEEE Access631110ndash31116 2018

[14] Qi Xuan Zhuangzhi Chen Yi Liu Huimin Huang Guanjun Bao and Dan ZhangMultiview generative adversarial network and its application in pearl classificationIEEE Transactions on Industrial Electronics 66(10)8244ndash8252 2019

[15] Qi Xuan Fuxian Li Yi Liu and Yun Xiang MV-C3D A spatial correlatedmulti-view 3d convolutional neural networks IEEE Access 792528ndash925382019

[16] Song Wang Taiyue Liu and Lin Tan Automatically learning semantic featuresfor defect prediction In Software Engineering (ICSE) 2016 IEEEACM 38thInternational Conference on pages 297ndash308 IEEE 2016

[17] Jian Li Pinjia He Jieming Zhu and Michael R Lyu Software defect predictionvia convolutional neural network In Software Quality Reliability and Security(QRS) 2017 IEEE International Conference on pages 318ndash328 IEEE 2017

[18] Hoa Khanh Dam Trang Pham Shien Wee Ng Truyen Tran John Grundy AdityaGhose Taeksu Kim and Chul-Joo Kim A deep tree-based model for softwaredefect prediction arXiv preprint arXiv180200921 2018

[19] Alex Krizhevsky Ilya Sutskever and Geoffrey E Hinton Imagenet classificationwith deep convolutional neural networks In Advances in neural informationprocessing systems pages 1097ndash1105 2012

[20] Mingsheng Long Yue Cao Jianmin Wang and Michael I Jordan Learning trans-ferable features with deep adaptation networks arXiv preprint arXiv1502027912015

[21] Han Zhang Ian Goodfellow Dimitris Metaxas and Augustus Odena Self-attention generative adversarial networks arXiv preprint arXiv1805083182018

[22] Tian Jiang Lin Tan and Sunghun Kim Personalized defect prediction In Pro-ceedings of the 28th IEEEACM International Conference on Automated SoftwareEngineering pages 279ndash289 IEEE Press 2013

[23] Andrew Meneely Laurie Williams Will Snipes and Jason Osborne Predictingfailures with developer networks and social network analysis In Proceedings ofthe 16th ACM SIGSOFT International Symposium on Foundations of softwareengineering pages 13ndash23 ACM 2008

[24] Raimund Moser Witold Pedrycz and Giancarlo Succi A comparative analysis ofthe efficiency of change metrics and static code attributes for defect prediction InProceedings of the 30th international conference on Software engineering pages181ndash190 ACM 2008

[25] Foyzur Rahman and Premkumar Devanbu How and why process metrics arebetter In Software Engineering (ICSE) 2013 35th International Conference onpages 432ndash441 IEEE 2013

[26] Maurice Howard Halstead et al Elements of software science volume 7 ElsevierNew York 1977

[27] Thomas J McCabe A complexity measure IEEE Transactions on softwareEngineering (4)308ndash320 1976

[28] Shyam R Chidamber and Chris F Kemerer A metrics suite for object orienteddesign IEEE Transactions on software engineering 20(6)476ndash493 1994

[29] Yue Yu Huaimin Wang Gang Yin and Tao Wang Reviewer recommendation forpull-requests in github What can we learn from code review and bug assignmentInformation and Software Technology 74204ndash218 2016

11

ICSE 2020 May 23-29 2020 Seoul South Korea Chen et al

[30] Nachiappan Nagappan and Thomas Ball Using software dependencies and churnmetrics to predict field failures An empirical case study In null pages 364ndash373IEEE 2007

[31] Ahmed E Hassan Predicting faults using the complexity of code changes InProceedings of the 31st International Conference on Software Engineering pages78ndash88 IEEE Computer Society 2009

[32] Thomas J Ostrand Elaine J Weyuker and Robert M Bell Programmer-basedfault prediction In Proceedings of the 6th International Conference on PredictiveModels in Software Engineering page 19 ACM 2010

[33] Taek Lee Jaechang Nam DongGyun Han Sunghun Kim and Hoh Peter In Microinteraction metrics for defect prediction In Proceedings of the 19th ACM SIG-SOFT symposium and the 13th European conference on Foundations of softwareengineering pages 311ndash321 ACM 2011

[34] Tim Menzies Zach Milton Burak Turhan Bojan Cukic Yue Jiang and AyseBener Defect prediction from static code features current results limitationsnew approaches Automated Software Engineering 17(4)375ndash407 2010

[35] Martin Pinzger Nachiappan Nagappan and Brendan Murphy Can developer-module networks predict failures In Proceedings of the 16th ACM SIGSOFTInternational Symposium on Foundations of software engineering pages 2ndash12ACM 2008

[36] Steffen Herbold Alexander Trautsch and Jens Grabowski A comparative studyto benchmark cross-project defect prediction approaches IEEE Transactions onSoftware Engineering 44(9)811ndash833 2018

[37] Chao Liu Dan Yang Xin Xia Meng Yan and Xiaohong Zhang A two-phasetransfer learning model for cross-project defect prediction Information andSoftware Technology 107125ndash136 2019

[38] Jinyin Chen Keke Hu Yitao Yang Yi Liu and Qi Xuan Col-lective transfer learning for defect prediction Neurocomputing 2019doihttpsdoiorg101016jneucom201812091

[39] Jinyin Chen Yitao Yang Keke Hu Qi Xuan Yi Liu and Chao Yang Multiviewtransfer learning for software defect prediction IEEE Access 78901ndash8916 2019

[40] Annibale Panichella Rocco Oliveto and Andrea De Lucia Cross-project defectprediction models Lrsquounion fait la force In Software Maintenance Reengineeringand Reverse Engineering (CSMR-WCRE) 2014 Software Evolution Week-IEEEConference on pages 164ndash173 IEEE 2014

[41] Burak Turhan Tim Menzies Ayse B Bener and Justin Di Stefano On the relativevalue of cross-company and within-company data for defect prediction volume 14pages 540ndash578 Springer 2009

[42] Fayola Peters Tim Menzies and Andrian Marcus Better cross company defectprediction In Proceedings of the 10th Working Conference on Mining SoftwareRepositories pages 409ndash418 IEEE Press 2013

[43] Xin Xia LO David Sinno Jialin Pan Nachiappan Nagappan and Xinyu WangHydra Massively compositional model for cross-project defect prediction IEEETransactions on software Engineering 42(10)977 2016

[44] Ying Ma Guangchun Luo Xue Zeng and Aiguo Chen Transfer learning forcross-company software defect prediction Information and Software Technology54(3)248ndash256 2012

[45] Jaechang Nam Sinno Jialin Pan and Sunghun Kim Transfer defect learningIn Software Engineering (ICSE) 2013 35th International Conference on pages382ndash391 IEEE 2013

[46] Sinno Jialin Pan Ivor W Tsang James T Kwok and Qiang Yang Domain adap-tation via transfer component analysis IEEE Transactions on Neural Networks22(2)199ndash210 2011

[47] Blaine A Price Ronald M Baecker and Ian S Small A principled taxonomy ofsoftware visualization Journal of Visual Languages amp Computing 4(3)211ndash2661993

[48] Thomas Ball and Stephen G Eick Software visualization in the large Computer29(4)33ndash43 1996

[49] Sarita Bassil and Rudolf K Keller Software visualization tools Survey andanalysis In Proceedings 9th International Workshop on Program ComprehensionIWPC 2001 pages 7ndash17 IEEE 2001

[50] Stephan Diehl Software visualization visualizing the structure behaviour andevolution of software Springer Science amp Business Media 2007

[51] Michele Lanza The evolution matrix Recovering software evolution using soft-ware visualization techniques In Proceedings of the 4th international workshopon principles of software evolution pages 37ndash42 ACM 2001

[52] Denis Gracanin Krešimir Matkovic and Mohamed Eltoweissy Software vi-sualization Innovations in Systems and Software Engineering 1(2)221ndash2302005

[53] Guillaume Langelier Houari Sahraoui and Pierre Poulin Visualization-basedanalysis of quality for large-scale software systems In Proceedings of the 20thIEEEACM international Conference on Automated software engineering pages214ndash223 ACM 2005

[54] Abien Fred Agarap and Francis John Hill Pepito Towards building an intelligentanti-malware system a deep learning approach using support vector machine(svm) for malware classification arXiv preprint arXiv180100318 2017

[55] Karl Weiss Taghi M Khoshgoftaar and DingDing Wang A survey of transferlearning Journal of Big Data 3(1)9 2016

[56] Rahul Krishna and Tim Menzies Bellwethers A baseline method for transferlearning IEEE Transactions on Software Engineering 2018

[57] Chuanqi Tan Fuchun Sun Tao Kong Wenchang Zhang Chao Yang and ChunfangLiu A survey on deep transfer learning In ICANN 2018

[58] Wenyuan Dai Qiang Yang Gui Rong Xue and Yong Yu Boosting for transferlearning In International Conference on Machine Learning pages 193ndash2002007

[59] Chang Wan Rong Pan and Jiefei Li Bi-weighting domain adaptation for cross-language text classification In IJCAI Proceedings-International Joint Conferenceon Artificial Intelligence volume 22 page 1535 2011

[60] Yonghui Xu Sinno Jialin Pan Hui Xiong Qingyao Wu Ronghua Luo HuaqingMin and Hengjie Song A unified framework for metric transfer learning IEEETrans Knowl Data Eng 29(6)1158ndash1171 2017

[61] Chetak Kandaswamy Luiacutes M Silva Luiacutes A Alexandre and Jorge M SantosDeep transfer learning ensemble for classification In Ignacio Rojas GonzaloJoya and Andreu Catala editors Advances in Computational Intelligence pages335ndash348 Cham 2015 Springer International Publishing

[62] Maxime Oquab Leon Bottou Ivan Laptev and Josef Sivic Learning and trans-ferring mid-level image representations using convolutional neural networks InProceedings of the IEEE conference on computer vision and pattern recognitionpages 1717ndash1724 2014

[63] Yann LeCun Leacuteon Bottou Yoshua Bengio and Patrick Haffner Gradient-basedlearning applied to document recognition Proceedings of the IEEE 86(11)2278ndash2324 1998

[64] Hana Ajakan Pascal Germain Hugo Larochelle Franccedilois Laviolette and MarioMarchand Domain-adversarial neural networks arXiv preprint arXiv141244462014

[65] Yaroslav Ganin and Victor Lempitsky Unsupervised domain adaptation by back-propagation arXiv preprint arXiv14097495 2014

[66] Jing Zhang Wanqing Li and Philip Ogunbona Joint geometrical and statisticalalignment for visual domain adaptation arXiv preprint arXiv170505498 2017

[67] Eric Tzeng Judy Hoffman Ning Zhang Kate Saenko and Trevor DarrellDeep domain confusion Maximizing for domain invariance arXiv preprintarXiv14123474 2014

[68] Mingsheng Long Yue Cao Zhangjie Cao Jianmin Wang and Michael I Jor-dan Transferable representation learning with deep adaptation networks IEEEtransactions on pattern analysis and machine intelligence 2018

[69] Arthur Gretton Dino Sejdinovic Heiko Strathmann Sivaraman BalakrishnanMassimiliano Pontil Kenji Fukumizu and Bharath K Sriperumbudur Optimalkernel choice for large-scale two-sample tests In Advances in neural informationprocessing systems pages 1205ndash1213 2012

[70] Mingsheng Long Han Zhu Jianmin Wang and Michael I Jordan Deep transferlearning with joint adaptation networks arXiv preprint arXiv160506636 2016

[71] Martin Arjovsky Soumith Chintala and Leacuteon Bottou Wasserstein gan arXivpreprint arXiv170107875 2017

[72] Kesav Kancherla and Srinivas Mukkamala Image visualization based malwaredetection In 2013 IEEE Symposium on Computational Intelligence in CyberSecurity (CICS) pages 40ndash44 IEEE 2013

[73] Xin Zhou Jianmin Pang and Guanghui Liang Image classification for malwaredetection using extremely randomized trees In 2017 11th IEEE InternationalConference on Anti-counterfeiting Security and Identification (ASID) pages54ndash59 IEEE 2017

[74] John Murphy An overview of convolutional neural network architectures for deeplearning 2016

[75] Song Wang Taiyue Liu Jaechang Nam and Lin Tan Deep semantic feature learn-ing for software defect prediction IEEE Transactions on Software Engineering2018

[76] Laurens van der Maaten and Geoffrey Hinton Visualizing data using t-sneJournal of machine learning research 9(Nov)2579ndash2605 2008

12

• Abstract
• 1 Introduction
• 2 Related Work
• 3 Approach
• • 31 Source Code Visualization
  • 32 Modeling with Deep Transfer Learning
  • 33 Model Sensitivity to Parameter Choices
  • 34 Training and Prediction
  • • 4 Experimental Setup
    • • 41 Dataset Description
      • 42 Baseline Comparison Methods
      • 43 Performance measures
      • • 5 Results
        • • 51 RQ1 How does DTL-DP compare to feature- based machine learning methods and AST- based deep learning methods in WPDP
          • 52 RQ2 How does DTL-DP compare to feature- based Machine Learning and AST-based deep learning methods in CPDP
          • 53 RQ3 How much does each of the three mechanisms ie data augmentation transfer learning and self-attention mechanism contribute to DTL-DPs performance
          • • 6 Threats to Validity
            • 7 Conclusions and Future Work
            • References

ICSE 2020 May 23-29 2020 Seoul South Korea Chen et al

005 01 02 05 1 2 5MMD Penalty

055

056

057

058

059

060

061

062

063

064

065

F-m

easu

re

WPDP CPDP

(a) Effect of different MMD Penalty λ

BRG BGR GRB GBR RGB RBG040

045

050

055

060

065

070

075

F-m

easu

re

WPDP CPDP

(b) Effect of color channel order

8 4 2 1 12 14 18Width recommended

040

045

050

055

060

065

070

075

F-m

easu

re

WPDP CPDP

(c) Effect of code image widthrecommended ratio

Figure 5 Sensitivity studies to aid in choosing the parameters in the model

After that the output of the AttentionLayer is the self-attentionfeature map o = (o1o2 oj oN ) where

oj =Nsumi=1

α jih(xi ) (2)

Then each fully connected layer learns a nonlinear mappinghli = f l (wlhli + b

l ) where hli is the l th layer hidden representationof feature xi wl and bl are the weights and biases of the l th layerand f l is the activation function using ReLU (f l (o) = max(0o))for f c6 minus f c8 and softmax for f c9 If we let Θ denote the set of allDTL-DP parameters the empirical risk of DTL-DP then is

minΘ

1n

nsumi=1

F (ϕ(oi )yi ) (3)

where F is the cross-entropy loss function n is the number of in-stances and ϕ(oi ) is the conditional probability that the DTL-DPassigns oi to label yi It is used to calculate the final loss in Equation7

To make the distribution of the source and target projects similarthe same multi-layer adaptation and multi-kernel MMD strategy asin a DAN [68] are used in our model The feature mapping functionσ is defined as the combination ofm positive semi-definite kernelsku

lt σ (os )σ (ot ) gt= k(os ot ) (4)

k =msumu=1

βuku (5)

where β ge 0 are the weights of the kernelsThe MK-MMD dk (pq) of the probability distributions p and q

is defined as the reproducing kernel Hilbert space distance betweenthe mean embedding of p and q The square formula of MK-MMDis defined as

d2k (pq) =∥ Ep [σ (xs )] minus Eq [σ (x

t )] ∥2 (6)

We see then that the smaller dk (pq) is the more similar p and qare If dk is 0 the distribution of the target project is the same as thatof the source project So the final loss function for train DPL-DP is

1n

nsumi=1

F (ϕ(oi )yi ) + λ

l2suml=l1

d2k (Dls D

lt ) (7)

where Dls is the l th layer hidden representation for the source and

target d2k (Dls D

lt ) is the MK-MMD between the source and target

evaluated on the l th layer representation λ is a penalty parameterand l1 and l2 are f c layers to calculate the MK-MMD between sourceand target In our implementation of DTL-DP we set λ = 1 l1 = 6and l2 = 9 as per the original DAN work [68]

33 Model Sensitivity to Parameter ChoicesWe have made a number of choices to make our modeling plat-form work effectively Here we justify those choices by presentingsensitivity studies For this analysis we used all the data

Hyperparameter λ The DTL-DP model has a hyper-parameter λfor the MMD penalty that adjusts the final loss We conducted λ pa-rameter sensitivity experiments in both the WPDP and CPDP settingWe fix the other parameters and range λ in 001 005 01 02 05 1 2 5The results are shown in Figure 5(a) While fairly constant across therange small variations exist In CPDP the F-measure first increasesand then decreases along the hyper-parameter λ In WPDP λ affectsthe performance of DTL-DP but in the opposite direction We choseλ = 1 here

Different Color Orders The selection of the RGB permutationsin the target project images may also potentially affect the outcomeWe performed experiments with all six different orderings Whentesting them we set λ to 1 and changed the image type of the tar-get project to each one in RGBRBGBRGBGRGBRGRB Theresults are shown in Figure 5(b) The target images in different colororders have somewhat different performance But overall the valuesare close to DTL-DPrsquos average performance

Different Image Widths To explore the impact of different choicesfor image widths on the results we performed additional experi-ments

For our code images 3 bytes of source code are needed to form 1pixel each byte for one of the R G and B channels The sizes ofthe source files are between 0 and 100 kb corresponding to imageswith pixel count between 0 and 34 133 (10010243) The size ofthe image is (widthheiдht) where width is obtained from Table 1according to the size of the code file (size)

heiдht =size lowast 10243 lowastwidth

(8)

For example the size of image converted from a 20 kb code file is(64 107)

We experimented with image widths of 18 14 12 2 4 and 8times of the recommended widths in Table 1 The larger the mul-tiplier the wider the image For images of different widths we

6

Software Visualization and Deep Transfer Learningfor Effective Software Defect Prediction ICSE 2020 May 23-29 2020 Seoul South Korea

repeated the previous experiment and obtained results for the meanF-measure shown in Figure 5(c) We see that the closer to the recom-mended width from Table 1 the better the average result In additionwe found that a wider image is better than a narrower one And thecloser the image is to a square the better the prediction Thereforewe made the generated images be as close to a square as possible

34 Training and PredictionWhen training the DTL-DP code images are generated from thelabeled instances in the source project and the unlabeled instances inthe target project and then are simultaneously input into the modelThey share the convolution layers conv1minusconv5 andAttentionLayerto extract their respective features and calculate the MK-MMDbetween source and target projects in the fully connected layersf c6 minus f c9 It should be noted that we only calculate cross entropyfor the source project because the target projectrsquos labels are not pro-vided We use a mini-batch stochastic gradient descent along the lossfunction in order to train the parameters of the entire model We trainfor 500 epochs for each pair of source and target projects then pickthe epoch with the best F-measure (described in section 43) fromwhich to read out the parameters of the final model Finally the filesfrom the target project that need to be predicted are converted intocode images and then input into the trained model for classification

Overfitting is always a possibility with such pipelines We moder-ate it here with our choices of (1) augmenting the dataset by using sixcolor channel permutations described in section 31 (2) selecting asimple AlexNet model structure and (3) using the ImageNet 2012pre-trained model to reduce fluctuations

4 EXPERIMENTAL SETUPWe conducted experiments to asses the performance of DTL-DP andto compare it with existing deep learning-based defect predictionapproaches for both within-project WPDP and cross-project CPDPdefect prediction We ran experiments on a modern-day Linux serverwith 3 Titan XP GPUs Unless otherwise stated each experimentwas run 10 times and the average results are reported

41 Dataset DescriptionIn order to directly compare our work with prior research we usedpublicly available data from the PROMISE3 data repository whichhas been widely used in defect prediction work [16ndash18 75] Weselected all open source Java projects from this repository andcollected their version number class names and the buggy labelfor each file Based on the version number and class name weobtained the source code for each file from GitHub4 and fed it toour end-to-end framework In total data for 10 Java projects werecollected Table 2 shows the details of these projects includingproject description versions the total and average number of filesand the defect rate It should be noted that the average numberof files over all projects ranges between 150 and 1046 and thedefect rates of the projects have a minimum value of 134 and amaximum value of 497 The number of files in some projects isnot sufficient to train deep models and the classes are imbalancedthus augmentation is needed as described above

3httpopenscienceusrepodefect4httpsgithubcomapache

42 Baseline Comparison MethodsTo evaluate the performance of our end-to-end framework DTL-DPfor defect prediction we compare it with the following baselinemethods in the WPDP setting

bull Semantic [16 75] the state-of-the-art method which em-ploys deep belief networks DBN on source code to extractsemantic features for defect prediction

bull PROMISE-DP [16] a traditional method which builds analternating decision tree ADTree classifier based on theoriginal 20 features of the PROMISE dataset

bull DP-LSTM [18] a long short-term memory LSTM-baseddeep neural network model which uses ASTs to representsource files and predict whether the file is defective or not

bull DP-CNN [17] a convolutional neural network CNN-basedmodel which is seeded by AST-derived numerical vectorsto automatically learn semantic and structural features ofprograms The CNN-learned features are used to train thefinal classifier in combination with traditional features

For the cross-project settings CPDP Semantic and PROMISE-DPcould not be used directly Instead we used the following

bull DBN-CP [16 75] a variant of Semantic which trains a DBNby using the source project and generates semantic featuresfor both the source and target projects

bull TCA+ [45] the state-of-the-art technique for CPDP

To obtain the training and test data we followed the processesestablished in [16] For WPDP we use two consecutive versions ofeach project listed in Table 2 The older version is used to generatethe training data and the more recent version is used as test data ForCPDP we pick versions randomly from each project for 11 targetprojects And for each target project we select 2 source projects thatare different from the target projects We use the same 22 test pairsas in [16] When implementing the baseline methods we use thesame network architecture and parameter settings as described in thepapers that introduced them

43 Performance measuresTo evaluate the prediction performance we use the F-measure awidely adopted metric in the literature [16ndash18 43 45] The F-measure captures a predictorrsquos accuracy and combines both precisionand recall for a comprehensive evaluation of predictive performance

Specifically a prediction that a file is defective is called a truepositive (TP) if the file is in fact defective and false positive (FP)otherwise Similarly a prediction that a file is not defective is a truenegative (TN) if the file is in fact not defective and false negative(FN) otherwise Then the precision (P) recall (R) and F-measureare defined as

P = TP(TP + FP) (9)

R = TP(TP + FN ) (10)

F = (2 times P times R)(P + R) (11)

5 RESULTSThis section discusses our results of comparing DTL-DP to baselinetools for defect prediction

7

ICSE 2020 May 23-29 2020 Seoul South Korea Chen et al

Table 2 Dataset Description

Project Description Versions Files Avg files Avg size(kb) Defective

ant Java based build tool 151617 1465 488 62 134camel Enterprise integration framework 121416 3140 1046 29 187jEdit Text editor designed for programmers 324041 1935 645 87 192log4j Logging library for Java 1011 300 150 34 497

lucene Text search engine library 202224 607 402 38 358xalan A library for transforming XML files 2425 1984 992 46 296xerces XML parser 1213 1647 549 29 157

ivy Dependency management library 1420 622 311 41 200synapse Data transport adapters 101112 661 220 38 227

poi Java library to access Microsoft format files 152530 1248 416 36 407

Table 3 F-measure of DTL-DP Semantic (Seman) PROMISE-DP (PROM) DP-LSTM (LSTM) and DP-CNN (CNN) forWPDP

Project Version Seman PROM LSTM CNN DTL-DP

ant15-gt16 914 477 452 532 70016-gt17 942 542 379 566 593

camel12-gt14 785 373 323 461 40614-gt16 374 391 400 508 385

jEdit32-gt40 574 556 472 564 49440-gt41 615 546 490 580 687

log4j 10-gt11 701 587 539 632 688

lucene20-gt22 651 502 751 761 78322-gt24 773 605 751 721 776

xalan 24-gt25 595 518 657 601 794xerces 12-gt13 411 238 268 374 820

ivy 14-gt20 350 329 191 347 829

synapse10-gt11 544 476 454 539 54711-gt12 583 533 533 556 659

poi15-gt25 640 558 808 589 68525-gt30 803 754 777 784 426

Average 641 499 515 570 642

51 RQ1 How does DTL-DP compare to feature-based machine learning methods and AST-based deep learning methods in WPDP

We compare DTL-DP to 4 baseline approaches representing twodifferent kinds of defect prediction methods PROMISE-DP is thebaseline representative of traditional feature-based machine learningmethods Semantic DP-LSTM and DP-CNN are the baselines fordeep learning-type methods based on extracting features from ASTGuided by prior work we conducted 16 sets of WPDP experimentseach using two versions of the same project The older version isused to train the prediction model and the newer version is used toevaluate the trained model

Table 3 shows the F-measure values for the within-project WPDPdefect prediction experiments The highest F-measure values of the5 methods are shown in bold Since the methods based on deeplearning include some randomness we run DTL-DP Semantic DP-LSTM and DP-CNN 10 times for each experiment On average

the F-measure of our approach is 0642 and the PROMISE-DPSemantic DP-LSTM and DP-CNN achieve 0499 0641 0515 and0570 respectively The results demonstrate that our approach iscompetitive and may improve on defect prediction compared toPROMISE-DP DP-LSTM and DP-CNN The results of Semanticand our approach are similar

The proposed DTL-DP is effective and it could improvethe performance of WPDP tasks Like other deep learners it issensitive to small file sizes and unbalanced data

511 Case Study WPDP Discrimination of DTL-DP t-SNEis a non-linear dimensional reduction algorithm that is effectivein visualizing similarities and helping identify clusters in complexdata sets [76] To give insight in the performance of DTL-DP wedemonstrate feature transferability by showing t-SNE embeddingsin Figure 6 The blue points are non-defective files and the red aredefective ones5 We observe the following (1) The target instancesare not discriminated very well using either the traditional manualfeatures or the TCA+ improved manual features or the semanticfeatures extracted from ASTs while with our approach the pointsare discriminated much better (2) With the other three approachesthe categories between source and target projects are not well alignedwhile with our approach the categories between the projects aremore consistent These conclusions are derived from the intra- andinter-class distances of the two categories in Figures 6 and 7 Theyare visually apparent

Our method did not perform as well as the comparison methods onsome projects such as ant which is likely caused by large variancein file sizes While the sizes of ant-15 and ant-16 are close there isa marked difference between ant-16 and ant-17 the former beingmuch smaller than the latter From Table 3 the performance of ourmethod on ant15-gtant16 is better than that on ant16-gtant17On the contrary baseline methods other than DP-LSTM performbetter for the task ant16-gtant17 notably Semantic is dominantindicating that the semantic feature-based method is more robust tofile size variability Moreover the ant dataset has two shortcomingsthe first is that the amount of data is small cf Table 2 where theaverage amount is only 488 files in one project The second is that

5Since we use data augmentation we have more samples than the three baselines

8

Software Visualization and Deep Transfer Learningfor Effective Software Defect Prediction ICSE 2020 May 23-29 2020 Seoul South Korea

Figure 6 t-SNE mapping of source and target features (WPDP)

the classes are unbalanced and the proportion of defective files is134 which is the least of all our projects This makes training ofthe deep model more difficult which leads to the poorer performanceof our method on some projects

Table 4 F-measure of DTL-DP DBN-CP (DBN) TCA+ DP-LSTM (LSTM) and DP-CNN (CNN) in CPDP

Source Target DBN TCA+ LSTM CNN DTL-DPant16 camel14 316 292 321 323 395

jEdit41 camel14 693 330 318 651 407camel14 ant16 979 616 448 607 591

poi30 ant16 478 598 386 532 693camel14 jEdit41 615 537 394 547 531log4j11 jEdit41 503 419 389 423 639jEdit41 log4j11 645 574 574 656 783

lucene22 log4j11 618 571 578 632 794lucene22 xalan25 550 530 680 540 689xerces13 xalan25 572 581 676 562 686xalan25 lucene22 594 561 750 621 783log4j11 lucene22 692 524 750 663 769xalan25 xerces13 386 394 340 391 400ivy20 xerces13 426 398 261 421 420

xerces13 ivy20 453 409 264 467 472synapse12 ivy20 824 383 261 371 494

ivy14 synapse11 489 348 451 491 545poi25 synapse11 425 376 435 436 597ivy20 synapse12 433 570 530 456 620poi30 synapse12 514 542 503 532 623

synapse12 poi30 661 651 785 671 827ant16 poi30 619 343 785 627 827

Average 568 479 495 528 618

52 RQ2 How does DTL-DP compare to feature-based Machine Learning and AST-based deeplearning methods in CPDP

Here we compare to TCA+ and DBN-CP instead of PROMISEand Semantic as the baseline approaches as explained in Sect 42

Again guided by prior work we conducted 22 sets of CPDP experi-ments In each we randomly select two project versions from twodifferent projects one as a training set and the other as a test set

Table 4 presents the F-measure results of DTL-CP and the 4 base-line approaches The highest F-measure values are in bold On aver-age the F-measure of our approach in CPDP is 0618 and the DBN-CP TCA+ DP-LSTM and DP-CNN achieve 0568 0479 0495 and0528 Thus DTL-DP outperforms them by 88 290 248 and155 respectively In addition we found that for projects log4j11and poi30 our method does better than the corresponding WPDPbest performing method

Our proposed DTL-DP shows significant improvements onthe state-of-the-art in cross-project defect prediction

521 Case Study CPDP Discrimination of DTL-DP Ourmethod is more obviously dominant in CPDP than in WPDP Apossible reason for that is that deep transfer learning makes the dis-tributions of the training and test samples more similar in featurespace Another reason might be the superior ability of deep modelsto represent features enabling the model to obtain more transferablefeatures from the images To gain more insight we choose the taskpoi30 rarr ant16 and show the t-SNE embeddings in Figure 7 Wemake similar observations as in the RQ1 case study that (1) the tar-get instances are more easily discriminated with our approach and(2) the target instances can be better discriminated with the sourceclassifier This implies that our approach can learn more transferablefeatures for more effective defect prediction

Figure 7 t-SNE mapping of source and target features (CPDP)

53 RQ3 How much does each of the threemechanisms ie data augmentation transferlearning and self-attention mechanismcontribute to DTL-DPrsquos performance

To find out the specific contributions of the three parts to defectprediction we conducted further experiments We built the originalAlexNet model for binary classification and use it as the Base +TL+Attention and +DataAug are three new baselines built by adding tothe base AlexNet one of three mechanisms (transfer learning selfattention and data augmentation) respectively It should be noted

9

ICSE 2020 May 23-29 2020 Seoul South Korea Chen et al

that in addition to +DataAug we use images generated by onesequence of R G and B as training and test sets in each experimentTherefore the experimental results of Base +TL and +Attention inTables 5 and 6 are average results obtained over the 6 different RGBpermutations as described above

Table 5 Contributions of the three mechanisms to WPDP

Project Version Base +TL +Atten +Aug DTL-DP

ant15-gt16 660 676 664 677 70016-gt17 525 552 524 591 593

camel12-gt14 415 381 416 403 40614-gt16 412 396 423 384 385

jEdit32-gt40 484 505 497 485 49440-gt41 629 649 645 679 687

log4j 10-gt11 688 679 692 658 688

lucene20-gt22 667 773 778 761 78322-gt24 749 713 769 758 776

xalan 24-gt25 757 754 758 778 794xerces 12-gt13 809 822 800 788 820

ivy 14-gt20 820 824 827 812 829

synapse10-gt11 525 484 511 530 54711-gt12 618 608 612 616 659

poi15-gt25 663 593 654 614 68525-gt30 410 449 420 432 426

Average 614 616 624 623 642

Table 6 Contributions of the three mechanisms to CPDP

Source Target Base +TL +Atten +Aug DTL-DPant16 camel14 383 379 365 377 395

jEdit41 camel14 368 394 395 371 407camel14 ant16 670 667 653 596 591

poi30 ant16 606 627 634 593 693camel14 jEdit41 663 657 652 647 531log4j11 jEdit41 599 599 592 594 639jEdit41 log4j11 754 800 774 746 783

lucene22 log4j11 759 750 753 783 794lucene22 xalan25 661 664 650 647 689xerces13 xalan25 510 638 626 662 686xalan25 lucene22 730 751 710 769 783log4j11 lucene22 713 745 740 727 769xalan25 xerces13 387 355 393 373 400ivy20 xerces13 413 408 398 378 420

xerces13 ivy20 518 500 504 500 472synapse12 ivy20 521 490 509 476 494

ivy14 synapse11 551 581 553 536 545poi25 synapse11 508 541 516 545 597ivy20 synapse12 604 622 619 611 620poi30 synapse12 600 632 613 612 623

synapse12 poi30 770 804 781 817 827ant16 poi30 777 662 700 812 827

Average 594 603 597 599 618

Table 7 Time cost of the three mechanisms defect visualization(Visualiz) self-attention (Attent) and transfer learning (TL)

ProjectTime (s)

TL Attent Visualizant 4504 131 459

camel 9655 242 785jEdit 5915 266 497log4j 1371 131 176

lucene 3739 186 354xalan 9444 271 823xerces 5117 214 275

ivy 2902 142 305synapse 2042 126 209

poi 3865 157 221Average 4856 187 410

To discern the contribution of each mechanism we compare theperformance of these baseline methods in WPDP and CPDP Fromthe results in Table 5 and Table 6 we observe that the three mecha-nisms each contribute substantially to the accuracy of DTL-DP inboth WPDP and CPDP In terms of the F-measure transfer learn-ing contributes the least improvement and self-attention and dataaugmentation contribute similarly to the final result But in CPDPtransfer learning contributes to the F-measure the most This islikely because of the applicability of transfer learning in the CPDPsetting and is in a way a validation of the approach

We also noted the time cost for the 3 mechanisms in our proposedDTL-DP Table 7 shows the result The most time spent during dataaugmentation is on converting code into images ie source codevisualization For transfer learning most time is spent on the MK-MMD calculation and for self-attention on the calculation of theattention layer in Fig 4 Eg for the project ant Table 3 shows twosets of WPDP experiments ant 15 rarr 16 and ant 16 rarr 17 Onaverage it takes 4504 seconds 1301 seconds and 459 seconds forthe 3 parts respectively for both the training data and the test dataTransfer learning takes the longest time more than the sum of theother two This is because of the large number of matrices neededto calculate MMD with the kernel function The least time cost isincurred by the attention mechanism and its contribution to WPDPis the largest of the three ie it is most cost-effective

The three mechanisms all contribute toward the accuracy ofour proposed end-to-end framework in WPDP and CPDP Self-attention has the greatest contribution to WPDP and transferlearning contributes most to CPDP

6 THREATS TO VALIDITYThreats to internal validity come from experimental errors and thereplication of the baseline methods In order to compare and analyzethe deep learning-based defect prediction techniques we compareour proposed DTL-DP method with Sementic DBN-CP DP-LSTMand DP-CNN In addition our method is also compared with thetransfer learning-based method TCA+ which is the state-of-the-art CPDP technique Since the original implementations were not

10

Software Visualization and Deep Transfer Learningfor Effective Software Defect Prediction ICSE 2020 May 23-29 2020 Seoul South Korea

available we re-implemented our own versions of the baselinesAlthough we strictly follow the procedures described in their workour new implementations may not completely restore all of theiroriginal implementation details And the randomness of the deeplearning-based approach also makes the results of our implementedexperiments different from the original Since we have removed theentries in the PROMISE dataset that cannot retrieve the correspond-ing source files our re-implemented experimental results may notbe consistent with the original baselines

The external threat to the validity of the results lies in the general-izability of the results We have tested our method on 10 open sourceJava projects including 14600 files Defect predictions for instancesof other languages such as C C++ etc need to be validated byadditional experiments in the future

Threats to construct validity depend on the appropriateness ofthe evaluation measurement The F-measure is used as our mainevaluation measure which has been applied in many previous effortsto evaluate defect prediction comprehensively

7 CONCLUSIONS AND FUTURE WORKHere we made two main contributions code visualization for defectprediction and an improved deep transfer learning model Our exper-imental results on 10 open source projects show that deep learningcan be effectively applied directly for defect prediction after apply-ing visualization methods to the code Specifically our approachDTL-DP performs at the top of the range of state-of-the-art WPDPapproaches For CPDP DTL-DP improves on the state-of-the-arttechnique TCA+ built on traditional features by 290 (in the F-measure) It also bests the deep learning-based approaches DBN-CPDP-LSTM and DP-CNN by 88 248 and 155 respectively

DTL-DP still has some limitations For some projects a problemof negative transfer occurs resulting in a worse prediction than adirect prediction Large differences between two projects can causesuch negative transfer Reducing the impact of negative transfer isone of the problems to be solved in the future Additionally theamount of training data we had was small and class imbalance isinherent in software defect prediction Both can be improved withmore data which we plan to obtain in the future

ACKNOWLEDGEMENTSThis work is partially supported by Zhejiang Natural Science Foun-dation(LY19F020025) National Natural Science Foundation ofChina (61502423 61572439 61702534) Zhejiang University OpenFund(2018KFJJ07) Signal Recognition Based on GAN Deep Learn-ing for Enhancement Recognition Project Zhejiang Science andTechnology Plan Project (LGF18F030009 2017C33149) and Zhe-jiang Outstanding Youth Fund (LR19F030001)

REFERENCES[1] Romi Satria Wahono A systematic literature review of software defect predic-

tion research trends datasets methods and frameworks Journal of SoftwareEngineering 1(1)1ndash16 2015

[2] Burak Turhan Tim Menzies Ayse B Bener and Justin Di Stefano On the relativevalue of cross-company and within-company data for defect prediction EmpiricalSoftware Engineering 14(5)540ndash578 2009

[3] Lech Madeyski and Marian Jureczko Which process metrics can significantlyimprove defect prediction models an empirical study Software Quality Journal23(3)393ndash422 2015

[4] Thomas Shippey David Bowes and Tracy Hall Automatically identifying codefeatures for software defect prediction Using ast n-grams Information andSoftware Technology 106142ndash160 2019

[5] Zhiyuan Wan Xin Xia Ahmed E Hassan David Lo Jianwei Yin and XiaohuYang Perceptions expectations and challenges in defect prediction IEEETransactions on Software Engineering 2018

[6] Karim O Elish and Mahmoud O Elish Predicting defect-prone software modulesusing support vector machines Journal of Systems and Software 81(5)649ndash6602008

[7] Puja Ahmad Habibi Victor Amrizal and Rizal Broer Bahaweres Cross-projectdefect prediction for web application using naive bayes (case study Petstoreweb application) In 2018 International Workshop on Big Data and InformationSecurity (IWBIS) pages 13ndash18 IEEE 2018

[8] Michael J Siers and Md Zahidul Islam Software defect prediction using a costsensitive decision forest and voting and a potential solution to the class imbalanceproblem Information Systems 5162ndash71 2015

[9] R Jayanthi and Lilly Florence Software defect prediction techniques using metricsbased on neural network classifier Cluster Computing pages 1ndash12 2018

[10] Yi Liu Yu Fan and Junghui Chen Flame images for oxygen content predictionof combustion systems using dbn Energy amp Fuels 31(8)8776ndash8783 2017

[11] Yi Liu Chao Yang Zengliang Gao and Yuan Yao Ensemble deep kernel learn-ing with application to quality prediction in industrial polymerization processesChemometrics and Intelligent Laboratory Systems 17415ndash21 2018

[12] Qi Xuan Binwei Fang Yi Liu Jinbao Wang Jian Zhang Yayu Zheng and Guan-jun Bao Automatic pearl classification machine based on a multistream convolu-tional neural network IEEE Transactions on Industrial Electronics 65(8)6538ndash6547 2018

[13] Qi Xuan Haoquan Xiao Chenbo Fu and Yi Liu Evolving convolutional neuralnetwork and its application in fine-grained visual categorization IEEE Access631110ndash31116 2018

[14] Qi Xuan Zhuangzhi Chen Yi Liu Huimin Huang Guanjun Bao and Dan ZhangMultiview generative adversarial network and its application in pearl classificationIEEE Transactions on Industrial Electronics 66(10)8244ndash8252 2019

[15] Qi Xuan Fuxian Li Yi Liu and Yun Xiang MV-C3D A spatial correlatedmulti-view 3d convolutional neural networks IEEE Access 792528ndash925382019

[16] Song Wang Taiyue Liu and Lin Tan Automatically learning semantic featuresfor defect prediction In Software Engineering (ICSE) 2016 IEEEACM 38thInternational Conference on pages 297ndash308 IEEE 2016

[17] Jian Li Pinjia He Jieming Zhu and Michael R Lyu Software defect predictionvia convolutional neural network In Software Quality Reliability and Security(QRS) 2017 IEEE International Conference on pages 318ndash328 IEEE 2017

[18] Hoa Khanh Dam Trang Pham Shien Wee Ng Truyen Tran John Grundy AdityaGhose Taeksu Kim and Chul-Joo Kim A deep tree-based model for softwaredefect prediction arXiv preprint arXiv180200921 2018

[19] Alex Krizhevsky Ilya Sutskever and Geoffrey E Hinton Imagenet classificationwith deep convolutional neural networks In Advances in neural informationprocessing systems pages 1097ndash1105 2012

[20] Mingsheng Long Yue Cao Jianmin Wang and Michael I Jordan Learning trans-ferable features with deep adaptation networks arXiv preprint arXiv1502027912015

[21] Han Zhang Ian Goodfellow Dimitris Metaxas and Augustus Odena Self-attention generative adversarial networks arXiv preprint arXiv1805083182018

[22] Tian Jiang Lin Tan and Sunghun Kim Personalized defect prediction In Pro-ceedings of the 28th IEEEACM International Conference on Automated SoftwareEngineering pages 279ndash289 IEEE Press 2013

[23] Andrew Meneely Laurie Williams Will Snipes and Jason Osborne Predictingfailures with developer networks and social network analysis In Proceedings ofthe 16th ACM SIGSOFT International Symposium on Foundations of softwareengineering pages 13ndash23 ACM 2008

[24] Raimund Moser Witold Pedrycz and Giancarlo Succi A comparative analysis ofthe efficiency of change metrics and static code attributes for defect prediction InProceedings of the 30th international conference on Software engineering pages181ndash190 ACM 2008

[25] Foyzur Rahman and Premkumar Devanbu How and why process metrics arebetter In Software Engineering (ICSE) 2013 35th International Conference onpages 432ndash441 IEEE 2013

[26] Maurice Howard Halstead et al Elements of software science volume 7 ElsevierNew York 1977

[27] Thomas J McCabe A complexity measure IEEE Transactions on softwareEngineering (4)308ndash320 1976

[28] Shyam R Chidamber and Chris F Kemerer A metrics suite for object orienteddesign IEEE Transactions on software engineering 20(6)476ndash493 1994

[29] Yue Yu Huaimin Wang Gang Yin and Tao Wang Reviewer recommendation forpull-requests in github What can we learn from code review and bug assignmentInformation and Software Technology 74204ndash218 2016

11

ICSE 2020 May 23-29 2020 Seoul South Korea Chen et al

[30] Nachiappan Nagappan and Thomas Ball Using software dependencies and churnmetrics to predict field failures An empirical case study In null pages 364ndash373IEEE 2007

[31] Ahmed E Hassan Predicting faults using the complexity of code changes InProceedings of the 31st International Conference on Software Engineering pages78ndash88 IEEE Computer Society 2009

[32] Thomas J Ostrand Elaine J Weyuker and Robert M Bell Programmer-basedfault prediction In Proceedings of the 6th International Conference on PredictiveModels in Software Engineering page 19 ACM 2010

[33] Taek Lee Jaechang Nam DongGyun Han Sunghun Kim and Hoh Peter In Microinteraction metrics for defect prediction In Proceedings of the 19th ACM SIG-SOFT symposium and the 13th European conference on Foundations of softwareengineering pages 311ndash321 ACM 2011

[34] Tim Menzies Zach Milton Burak Turhan Bojan Cukic Yue Jiang and AyseBener Defect prediction from static code features current results limitationsnew approaches Automated Software Engineering 17(4)375ndash407 2010

[35] Martin Pinzger Nachiappan Nagappan and Brendan Murphy Can developer-module networks predict failures In Proceedings of the 16th ACM SIGSOFTInternational Symposium on Foundations of software engineering pages 2ndash12ACM 2008

[36] Steffen Herbold Alexander Trautsch and Jens Grabowski A comparative studyto benchmark cross-project defect prediction approaches IEEE Transactions onSoftware Engineering 44(9)811ndash833 2018

[37] Chao Liu Dan Yang Xin Xia Meng Yan and Xiaohong Zhang A two-phasetransfer learning model for cross-project defect prediction Information andSoftware Technology 107125ndash136 2019

[38] Jinyin Chen Keke Hu Yitao Yang Yi Liu and Qi Xuan Col-lective transfer learning for defect prediction Neurocomputing 2019doihttpsdoiorg101016jneucom201812091

[39] Jinyin Chen Yitao Yang Keke Hu Qi Xuan Yi Liu and Chao Yang Multiviewtransfer learning for software defect prediction IEEE Access 78901ndash8916 2019

[40] Annibale Panichella Rocco Oliveto and Andrea De Lucia Cross-project defectprediction models Lrsquounion fait la force In Software Maintenance Reengineeringand Reverse Engineering (CSMR-WCRE) 2014 Software Evolution Week-IEEEConference on pages 164ndash173 IEEE 2014

[41] Burak Turhan Tim Menzies Ayse B Bener and Justin Di Stefano On the relativevalue of cross-company and within-company data for defect prediction volume 14pages 540ndash578 Springer 2009

[42] Fayola Peters Tim Menzies and Andrian Marcus Better cross company defectprediction In Proceedings of the 10th Working Conference on Mining SoftwareRepositories pages 409ndash418 IEEE Press 2013

[43] Xin Xia LO David Sinno Jialin Pan Nachiappan Nagappan and Xinyu WangHydra Massively compositional model for cross-project defect prediction IEEETransactions on software Engineering 42(10)977 2016

[44] Ying Ma Guangchun Luo Xue Zeng and Aiguo Chen Transfer learning forcross-company software defect prediction Information and Software Technology54(3)248ndash256 2012

[45] Jaechang Nam Sinno Jialin Pan and Sunghun Kim Transfer defect learningIn Software Engineering (ICSE) 2013 35th International Conference on pages382ndash391 IEEE 2013

[46] Sinno Jialin Pan Ivor W Tsang James T Kwok and Qiang Yang Domain adap-tation via transfer component analysis IEEE Transactions on Neural Networks22(2)199ndash210 2011

[47] Blaine A Price Ronald M Baecker and Ian S Small A principled taxonomy ofsoftware visualization Journal of Visual Languages amp Computing 4(3)211ndash2661993

[48] Thomas Ball and Stephen G Eick Software visualization in the large Computer29(4)33ndash43 1996

[49] Sarita Bassil and Rudolf K Keller Software visualization tools Survey andanalysis In Proceedings 9th International Workshop on Program ComprehensionIWPC 2001 pages 7ndash17 IEEE 2001

[50] Stephan Diehl Software visualization visualizing the structure behaviour andevolution of software Springer Science amp Business Media 2007

[51] Michele Lanza The evolution matrix Recovering software evolution using soft-ware visualization techniques In Proceedings of the 4th international workshopon principles of software evolution pages 37ndash42 ACM 2001

[52] Denis Gracanin Krešimir Matkovic and Mohamed Eltoweissy Software vi-sualization Innovations in Systems and Software Engineering 1(2)221ndash2302005

[53] Guillaume Langelier Houari Sahraoui and Pierre Poulin Visualization-basedanalysis of quality for large-scale software systems In Proceedings of the 20thIEEEACM international Conference on Automated software engineering pages214ndash223 ACM 2005

[54] Abien Fred Agarap and Francis John Hill Pepito Towards building an intelligentanti-malware system a deep learning approach using support vector machine(svm) for malware classification arXiv preprint arXiv180100318 2017

[55] Karl Weiss Taghi M Khoshgoftaar and DingDing Wang A survey of transferlearning Journal of Big Data 3(1)9 2016

[56] Rahul Krishna and Tim Menzies Bellwethers A baseline method for transferlearning IEEE Transactions on Software Engineering 2018

[57] Chuanqi Tan Fuchun Sun Tao Kong Wenchang Zhang Chao Yang and ChunfangLiu A survey on deep transfer learning In ICANN 2018

[58] Wenyuan Dai Qiang Yang Gui Rong Xue and Yong Yu Boosting for transferlearning In International Conference on Machine Learning pages 193ndash2002007

[59] Chang Wan Rong Pan and Jiefei Li Bi-weighting domain adaptation for cross-language text classification In IJCAI Proceedings-International Joint Conferenceon Artificial Intelligence volume 22 page 1535 2011

[60] Yonghui Xu Sinno Jialin Pan Hui Xiong Qingyao Wu Ronghua Luo HuaqingMin and Hengjie Song A unified framework for metric transfer learning IEEETrans Knowl Data Eng 29(6)1158ndash1171 2017

[61] Chetak Kandaswamy Luiacutes M Silva Luiacutes A Alexandre and Jorge M SantosDeep transfer learning ensemble for classification In Ignacio Rojas GonzaloJoya and Andreu Catala editors Advances in Computational Intelligence pages335ndash348 Cham 2015 Springer International Publishing

[62] Maxime Oquab Leon Bottou Ivan Laptev and Josef Sivic Learning and trans-ferring mid-level image representations using convolutional neural networks InProceedings of the IEEE conference on computer vision and pattern recognitionpages 1717ndash1724 2014

[63] Yann LeCun Leacuteon Bottou Yoshua Bengio and Patrick Haffner Gradient-basedlearning applied to document recognition Proceedings of the IEEE 86(11)2278ndash2324 1998

[64] Hana Ajakan Pascal Germain Hugo Larochelle Franccedilois Laviolette and MarioMarchand Domain-adversarial neural networks arXiv preprint arXiv141244462014

[65] Yaroslav Ganin and Victor Lempitsky Unsupervised domain adaptation by back-propagation arXiv preprint arXiv14097495 2014

[66] Jing Zhang Wanqing Li and Philip Ogunbona Joint geometrical and statisticalalignment for visual domain adaptation arXiv preprint arXiv170505498 2017

[67] Eric Tzeng Judy Hoffman Ning Zhang Kate Saenko and Trevor DarrellDeep domain confusion Maximizing for domain invariance arXiv preprintarXiv14123474 2014

[68] Mingsheng Long Yue Cao Zhangjie Cao Jianmin Wang and Michael I Jor-dan Transferable representation learning with deep adaptation networks IEEEtransactions on pattern analysis and machine intelligence 2018

[69] Arthur Gretton Dino Sejdinovic Heiko Strathmann Sivaraman BalakrishnanMassimiliano Pontil Kenji Fukumizu and Bharath K Sriperumbudur Optimalkernel choice for large-scale two-sample tests In Advances in neural informationprocessing systems pages 1205ndash1213 2012

[70] Mingsheng Long Han Zhu Jianmin Wang and Michael I Jordan Deep transferlearning with joint adaptation networks arXiv preprint arXiv160506636 2016

[71] Martin Arjovsky Soumith Chintala and Leacuteon Bottou Wasserstein gan arXivpreprint arXiv170107875 2017

[72] Kesav Kancherla and Srinivas Mukkamala Image visualization based malwaredetection In 2013 IEEE Symposium on Computational Intelligence in CyberSecurity (CICS) pages 40ndash44 IEEE 2013

[73] Xin Zhou Jianmin Pang and Guanghui Liang Image classification for malwaredetection using extremely randomized trees In 2017 11th IEEE InternationalConference on Anti-counterfeiting Security and Identification (ASID) pages54ndash59 IEEE 2017

[74] John Murphy An overview of convolutional neural network architectures for deeplearning 2016

[75] Song Wang Taiyue Liu Jaechang Nam and Lin Tan Deep semantic feature learn-ing for software defect prediction IEEE Transactions on Software Engineering2018

[76] Laurens van der Maaten and Geoffrey Hinton Visualizing data using t-sneJournal of machine learning research 9(Nov)2579ndash2605 2008

12

• Abstract
• 1 Introduction
• 2 Related Work
• 3 Approach
• • 31 Source Code Visualization
  • 32 Modeling with Deep Transfer Learning
  • 33 Model Sensitivity to Parameter Choices
  • 34 Training and Prediction
  • • 4 Experimental Setup
    • • 41 Dataset Description
      • 42 Baseline Comparison Methods
      • 43 Performance measures
      • • 5 Results
        • • 51 RQ1 How does DTL-DP compare to feature- based machine learning methods and AST- based deep learning methods in WPDP
          • 52 RQ2 How does DTL-DP compare to feature- based Machine Learning and AST-based deep learning methods in CPDP
          • 53 RQ3 How much does each of the three mechanisms ie data augmentation transfer learning and self-attention mechanism contribute to DTL-DPs performance
          • • 6 Threats to Validity
            • 7 Conclusions and Future Work
            • References

Software Visualization and Deep Transfer Learningfor Effective Software Defect Prediction ICSE 2020 May 23-29 2020 Seoul South Korea

repeated the previous experiment and obtained results for the meanF-measure shown in Figure 5(c) We see that the closer to the recom-mended width from Table 1 the better the average result In additionwe found that a wider image is better than a narrower one And thecloser the image is to a square the better the prediction Thereforewe made the generated images be as close to a square as possible

34 Training and PredictionWhen training the DTL-DP code images are generated from thelabeled instances in the source project and the unlabeled instances inthe target project and then are simultaneously input into the modelThey share the convolution layers conv1minusconv5 andAttentionLayerto extract their respective features and calculate the MK-MMDbetween source and target projects in the fully connected layersf c6 minus f c9 It should be noted that we only calculate cross entropyfor the source project because the target projectrsquos labels are not pro-vided We use a mini-batch stochastic gradient descent along the lossfunction in order to train the parameters of the entire model We trainfor 500 epochs for each pair of source and target projects then pickthe epoch with the best F-measure (described in section 43) fromwhich to read out the parameters of the final model Finally the filesfrom the target project that need to be predicted are converted intocode images and then input into the trained model for classification

Overfitting is always a possibility with such pipelines We moder-ate it here with our choices of (1) augmenting the dataset by using sixcolor channel permutations described in section 31 (2) selecting asimple AlexNet model structure and (3) using the ImageNet 2012pre-trained model to reduce fluctuations

4 EXPERIMENTAL SETUPWe conducted experiments to asses the performance of DTL-DP andto compare it with existing deep learning-based defect predictionapproaches for both within-project WPDP and cross-project CPDPdefect prediction We ran experiments on a modern-day Linux serverwith 3 Titan XP GPUs Unless otherwise stated each experimentwas run 10 times and the average results are reported

41 Dataset DescriptionIn order to directly compare our work with prior research we usedpublicly available data from the PROMISE3 data repository whichhas been widely used in defect prediction work [16ndash18 75] Weselected all open source Java projects from this repository andcollected their version number class names and the buggy labelfor each file Based on the version number and class name weobtained the source code for each file from GitHub4 and fed it toour end-to-end framework In total data for 10 Java projects werecollected Table 2 shows the details of these projects includingproject description versions the total and average number of filesand the defect rate It should be noted that the average numberof files over all projects ranges between 150 and 1046 and thedefect rates of the projects have a minimum value of 134 and amaximum value of 497 The number of files in some projects isnot sufficient to train deep models and the classes are imbalancedthus augmentation is needed as described above

3httpopenscienceusrepodefect4httpsgithubcomapache

42 Baseline Comparison MethodsTo evaluate the performance of our end-to-end framework DTL-DPfor defect prediction we compare it with the following baselinemethods in the WPDP setting

bull Semantic [16 75] the state-of-the-art method which em-ploys deep belief networks DBN on source code to extractsemantic features for defect prediction

bull PROMISE-DP [16] a traditional method which builds analternating decision tree ADTree classifier based on theoriginal 20 features of the PROMISE dataset

bull DP-LSTM [18] a long short-term memory LSTM-baseddeep neural network model which uses ASTs to representsource files and predict whether the file is defective or not

bull DP-CNN [17] a convolutional neural network CNN-basedmodel which is seeded by AST-derived numerical vectorsto automatically learn semantic and structural features ofprograms The CNN-learned features are used to train thefinal classifier in combination with traditional features

For the cross-project settings CPDP Semantic and PROMISE-DPcould not be used directly Instead we used the following

bull DBN-CP [16 75] a variant of Semantic which trains a DBNby using the source project and generates semantic featuresfor both the source and target projects

bull TCA+ [45] the state-of-the-art technique for CPDP

To obtain the training and test data we followed the processesestablished in [16] For WPDP we use two consecutive versions ofeach project listed in Table 2 The older version is used to generatethe training data and the more recent version is used as test data ForCPDP we pick versions randomly from each project for 11 targetprojects And for each target project we select 2 source projects thatare different from the target projects We use the same 22 test pairsas in [16] When implementing the baseline methods we use thesame network architecture and parameter settings as described in thepapers that introduced them

43 Performance measuresTo evaluate the prediction performance we use the F-measure awidely adopted metric in the literature [16ndash18 43 45] The F-measure captures a predictorrsquos accuracy and combines both precisionand recall for a comprehensive evaluation of predictive performance

Specifically a prediction that a file is defective is called a truepositive (TP) if the file is in fact defective and false positive (FP)otherwise Similarly a prediction that a file is not defective is a truenegative (TN) if the file is in fact not defective and false negative(FN) otherwise Then the precision (P) recall (R) and F-measureare defined as

P = TP(TP + FP) (9)

R = TP(TP + FN ) (10)

F = (2 times P times R)(P + R) (11)

5 RESULTSThis section discusses our results of comparing DTL-DP to baselinetools for defect prediction

7

ICSE 2020 May 23-29 2020 Seoul South Korea Chen et al

Table 2 Dataset Description

Project Description Versions Files Avg files Avg size(kb) Defective

ant Java based build tool 151617 1465 488 62 134camel Enterprise integration framework 121416 3140 1046 29 187jEdit Text editor designed for programmers 324041 1935 645 87 192log4j Logging library for Java 1011 300 150 34 497

lucene Text search engine library 202224 607 402 38 358xalan A library for transforming XML files 2425 1984 992 46 296xerces XML parser 1213 1647 549 29 157

ivy Dependency management library 1420 622 311 41 200synapse Data transport adapters 101112 661 220 38 227

poi Java library to access Microsoft format files 152530 1248 416 36 407

Table 3 F-measure of DTL-DP Semantic (Seman) PROMISE-DP (PROM) DP-LSTM (LSTM) and DP-CNN (CNN) forWPDP

Project Version Seman PROM LSTM CNN DTL-DP

ant15-gt16 914 477 452 532 70016-gt17 942 542 379 566 593

camel12-gt14 785 373 323 461 40614-gt16 374 391 400 508 385

jEdit32-gt40 574 556 472 564 49440-gt41 615 546 490 580 687

log4j 10-gt11 701 587 539 632 688

lucene20-gt22 651 502 751 761 78322-gt24 773 605 751 721 776

xalan 24-gt25 595 518 657 601 794xerces 12-gt13 411 238 268 374 820

ivy 14-gt20 350 329 191 347 829

synapse10-gt11 544 476 454 539 54711-gt12 583 533 533 556 659

poi15-gt25 640 558 808 589 68525-gt30 803 754 777 784 426

Average 641 499 515 570 642

51 RQ1 How does DTL-DP compare to feature-based machine learning methods and AST-based deep learning methods in WPDP

We compare DTL-DP to 4 baseline approaches representing twodifferent kinds of defect prediction methods PROMISE-DP is thebaseline representative of traditional feature-based machine learningmethods Semantic DP-LSTM and DP-CNN are the baselines fordeep learning-type methods based on extracting features from ASTGuided by prior work we conducted 16 sets of WPDP experimentseach using two versions of the same project The older version isused to train the prediction model and the newer version is used toevaluate the trained model

Table 3 shows the F-measure values for the within-project WPDPdefect prediction experiments The highest F-measure values of the5 methods are shown in bold Since the methods based on deeplearning include some randomness we run DTL-DP Semantic DP-LSTM and DP-CNN 10 times for each experiment On average

the F-measure of our approach is 0642 and the PROMISE-DPSemantic DP-LSTM and DP-CNN achieve 0499 0641 0515 and0570 respectively The results demonstrate that our approach iscompetitive and may improve on defect prediction compared toPROMISE-DP DP-LSTM and DP-CNN The results of Semanticand our approach are similar

The proposed DTL-DP is effective and it could improvethe performance of WPDP tasks Like other deep learners it issensitive to small file sizes and unbalanced data

511 Case Study WPDP Discrimination of DTL-DP t-SNEis a non-linear dimensional reduction algorithm that is effectivein visualizing similarities and helping identify clusters in complexdata sets [76] To give insight in the performance of DTL-DP wedemonstrate feature transferability by showing t-SNE embeddingsin Figure 6 The blue points are non-defective files and the red aredefective ones5 We observe the following (1) The target instancesare not discriminated very well using either the traditional manualfeatures or the TCA+ improved manual features or the semanticfeatures extracted from ASTs while with our approach the pointsare discriminated much better (2) With the other three approachesthe categories between source and target projects are not well alignedwhile with our approach the categories between the projects aremore consistent These conclusions are derived from the intra- andinter-class distances of the two categories in Figures 6 and 7 Theyare visually apparent

Our method did not perform as well as the comparison methods onsome projects such as ant which is likely caused by large variancein file sizes While the sizes of ant-15 and ant-16 are close there isa marked difference between ant-16 and ant-17 the former beingmuch smaller than the latter From Table 3 the performance of ourmethod on ant15-gtant16 is better than that on ant16-gtant17On the contrary baseline methods other than DP-LSTM performbetter for the task ant16-gtant17 notably Semantic is dominantindicating that the semantic feature-based method is more robust tofile size variability Moreover the ant dataset has two shortcomingsthe first is that the amount of data is small cf Table 2 where theaverage amount is only 488 files in one project The second is that

5Since we use data augmentation we have more samples than the three baselines

8

Software Visualization and Deep Transfer Learningfor Effective Software Defect Prediction ICSE 2020 May 23-29 2020 Seoul South Korea

Figure 6 t-SNE mapping of source and target features (WPDP)

the classes are unbalanced and the proportion of defective files is134 which is the least of all our projects This makes training ofthe deep model more difficult which leads to the poorer performanceof our method on some projects

Table 4 F-measure of DTL-DP DBN-CP (DBN) TCA+ DP-LSTM (LSTM) and DP-CNN (CNN) in CPDP

Source Target DBN TCA+ LSTM CNN DTL-DPant16 camel14 316 292 321 323 395

jEdit41 camel14 693 330 318 651 407camel14 ant16 979 616 448 607 591

poi30 ant16 478 598 386 532 693camel14 jEdit41 615 537 394 547 531log4j11 jEdit41 503 419 389 423 639jEdit41 log4j11 645 574 574 656 783

lucene22 log4j11 618 571 578 632 794lucene22 xalan25 550 530 680 540 689xerces13 xalan25 572 581 676 562 686xalan25 lucene22 594 561 750 621 783log4j11 lucene22 692 524 750 663 769xalan25 xerces13 386 394 340 391 400ivy20 xerces13 426 398 261 421 420

xerces13 ivy20 453 409 264 467 472synapse12 ivy20 824 383 261 371 494

ivy14 synapse11 489 348 451 491 545poi25 synapse11 425 376 435 436 597ivy20 synapse12 433 570 530 456 620poi30 synapse12 514 542 503 532 623

synapse12 poi30 661 651 785 671 827ant16 poi30 619 343 785 627 827

Average 568 479 495 528 618

52 RQ2 How does DTL-DP compare to feature-based Machine Learning and AST-based deeplearning methods in CPDP

Here we compare to TCA+ and DBN-CP instead of PROMISEand Semantic as the baseline approaches as explained in Sect 42

Again guided by prior work we conducted 22 sets of CPDP experi-ments In each we randomly select two project versions from twodifferent projects one as a training set and the other as a test set

Table 4 presents the F-measure results of DTL-CP and the 4 base-line approaches The highest F-measure values are in bold On aver-age the F-measure of our approach in CPDP is 0618 and the DBN-CP TCA+ DP-LSTM and DP-CNN achieve 0568 0479 0495 and0528 Thus DTL-DP outperforms them by 88 290 248 and155 respectively In addition we found that for projects log4j11and poi30 our method does better than the corresponding WPDPbest performing method

Our proposed DTL-DP shows significant improvements onthe state-of-the-art in cross-project defect prediction

521 Case Study CPDP Discrimination of DTL-DP Ourmethod is more obviously dominant in CPDP than in WPDP Apossible reason for that is that deep transfer learning makes the dis-tributions of the training and test samples more similar in featurespace Another reason might be the superior ability of deep modelsto represent features enabling the model to obtain more transferablefeatures from the images To gain more insight we choose the taskpoi30 rarr ant16 and show the t-SNE embeddings in Figure 7 Wemake similar observations as in the RQ1 case study that (1) the tar-get instances are more easily discriminated with our approach and(2) the target instances can be better discriminated with the sourceclassifier This implies that our approach can learn more transferablefeatures for more effective defect prediction

Figure 7 t-SNE mapping of source and target features (CPDP)

53 RQ3 How much does each of the threemechanisms ie data augmentation transferlearning and self-attention mechanismcontribute to DTL-DPrsquos performance

To find out the specific contributions of the three parts to defectprediction we conducted further experiments We built the originalAlexNet model for binary classification and use it as the Base +TL+Attention and +DataAug are three new baselines built by adding tothe base AlexNet one of three mechanisms (transfer learning selfattention and data augmentation) respectively It should be noted

9

ICSE 2020 May 23-29 2020 Seoul South Korea Chen et al

that in addition to +DataAug we use images generated by onesequence of R G and B as training and test sets in each experimentTherefore the experimental results of Base +TL and +Attention inTables 5 and 6 are average results obtained over the 6 different RGBpermutations as described above

Table 5 Contributions of the three mechanisms to WPDP

Project Version Base +TL +Atten +Aug DTL-DP

ant15-gt16 660 676 664 677 70016-gt17 525 552 524 591 593

camel12-gt14 415 381 416 403 40614-gt16 412 396 423 384 385

jEdit32-gt40 484 505 497 485 49440-gt41 629 649 645 679 687

log4j 10-gt11 688 679 692 658 688

lucene20-gt22 667 773 778 761 78322-gt24 749 713 769 758 776

xalan 24-gt25 757 754 758 778 794xerces 12-gt13 809 822 800 788 820

ivy 14-gt20 820 824 827 812 829

synapse10-gt11 525 484 511 530 54711-gt12 618 608 612 616 659

poi15-gt25 663 593 654 614 68525-gt30 410 449 420 432 426

Average 614 616 624 623 642

Table 6 Contributions of the three mechanisms to CPDP

Source Target Base +TL +Atten +Aug DTL-DPant16 camel14 383 379 365 377 395

jEdit41 camel14 368 394 395 371 407camel14 ant16 670 667 653 596 591

poi30 ant16 606 627 634 593 693camel14 jEdit41 663 657 652 647 531log4j11 jEdit41 599 599 592 594 639jEdit41 log4j11 754 800 774 746 783

lucene22 log4j11 759 750 753 783 794lucene22 xalan25 661 664 650 647 689xerces13 xalan25 510 638 626 662 686xalan25 lucene22 730 751 710 769 783log4j11 lucene22 713 745 740 727 769xalan25 xerces13 387 355 393 373 400ivy20 xerces13 413 408 398 378 420

xerces13 ivy20 518 500 504 500 472synapse12 ivy20 521 490 509 476 494

ivy14 synapse11 551 581 553 536 545poi25 synapse11 508 541 516 545 597ivy20 synapse12 604 622 619 611 620poi30 synapse12 600 632 613 612 623

synapse12 poi30 770 804 781 817 827ant16 poi30 777 662 700 812 827

Average 594 603 597 599 618

Table 7 Time cost of the three mechanisms defect visualization(Visualiz) self-attention (Attent) and transfer learning (TL)

ProjectTime (s)

TL Attent Visualizant 4504 131 459

camel 9655 242 785jEdit 5915 266 497log4j 1371 131 176

lucene 3739 186 354xalan 9444 271 823xerces 5117 214 275

ivy 2902 142 305synapse 2042 126 209

poi 3865 157 221Average 4856 187 410

To discern the contribution of each mechanism we compare theperformance of these baseline methods in WPDP and CPDP Fromthe results in Table 5 and Table 6 we observe that the three mecha-nisms each contribute substantially to the accuracy of DTL-DP inboth WPDP and CPDP In terms of the F-measure transfer learn-ing contributes the least improvement and self-attention and dataaugmentation contribute similarly to the final result But in CPDPtransfer learning contributes to the F-measure the most This islikely because of the applicability of transfer learning in the CPDPsetting and is in a way a validation of the approach

We also noted the time cost for the 3 mechanisms in our proposedDTL-DP Table 7 shows the result The most time spent during dataaugmentation is on converting code into images ie source codevisualization For transfer learning most time is spent on the MK-MMD calculation and for self-attention on the calculation of theattention layer in Fig 4 Eg for the project ant Table 3 shows twosets of WPDP experiments ant 15 rarr 16 and ant 16 rarr 17 Onaverage it takes 4504 seconds 1301 seconds and 459 seconds forthe 3 parts respectively for both the training data and the test dataTransfer learning takes the longest time more than the sum of theother two This is because of the large number of matrices neededto calculate MMD with the kernel function The least time cost isincurred by the attention mechanism and its contribution to WPDPis the largest of the three ie it is most cost-effective

The three mechanisms all contribute toward the accuracy ofour proposed end-to-end framework in WPDP and CPDP Self-attention has the greatest contribution to WPDP and transferlearning contributes most to CPDP

6 THREATS TO VALIDITYThreats to internal validity come from experimental errors and thereplication of the baseline methods In order to compare and analyzethe deep learning-based defect prediction techniques we compareour proposed DTL-DP method with Sementic DBN-CP DP-LSTMand DP-CNN In addition our method is also compared with thetransfer learning-based method TCA+ which is the state-of-the-art CPDP technique Since the original implementations were not

10

Software Visualization and Deep Transfer Learningfor Effective Software Defect Prediction ICSE 2020 May 23-29 2020 Seoul South Korea

available we re-implemented our own versions of the baselinesAlthough we strictly follow the procedures described in their workour new implementations may not completely restore all of theiroriginal implementation details And the randomness of the deeplearning-based approach also makes the results of our implementedexperiments different from the original Since we have removed theentries in the PROMISE dataset that cannot retrieve the correspond-ing source files our re-implemented experimental results may notbe consistent with the original baselines

The external threat to the validity of the results lies in the general-izability of the results We have tested our method on 10 open sourceJava projects including 14600 files Defect predictions for instancesof other languages such as C C++ etc need to be validated byadditional experiments in the future

Threats to construct validity depend on the appropriateness ofthe evaluation measurement The F-measure is used as our mainevaluation measure which has been applied in many previous effortsto evaluate defect prediction comprehensively

7 CONCLUSIONS AND FUTURE WORKHere we made two main contributions code visualization for defectprediction and an improved deep transfer learning model Our exper-imental results on 10 open source projects show that deep learningcan be effectively applied directly for defect prediction after apply-ing visualization methods to the code Specifically our approachDTL-DP performs at the top of the range of state-of-the-art WPDPapproaches For CPDP DTL-DP improves on the state-of-the-arttechnique TCA+ built on traditional features by 290 (in the F-measure) It also bests the deep learning-based approaches DBN-CPDP-LSTM and DP-CNN by 88 248 and 155 respectively

DTL-DP still has some limitations For some projects a problemof negative transfer occurs resulting in a worse prediction than adirect prediction Large differences between two projects can causesuch negative transfer Reducing the impact of negative transfer isone of the problems to be solved in the future Additionally theamount of training data we had was small and class imbalance isinherent in software defect prediction Both can be improved withmore data which we plan to obtain in the future

ACKNOWLEDGEMENTSThis work is partially supported by Zhejiang Natural Science Foun-dation(LY19F020025) National Natural Science Foundation ofChina (61502423 61572439 61702534) Zhejiang University OpenFund(2018KFJJ07) Signal Recognition Based on GAN Deep Learn-ing for Enhancement Recognition Project Zhejiang Science andTechnology Plan Project (LGF18F030009 2017C33149) and Zhe-jiang Outstanding Youth Fund (LR19F030001)

REFERENCES[1] Romi Satria Wahono A systematic literature review of software defect predic-

tion research trends datasets methods and frameworks Journal of SoftwareEngineering 1(1)1ndash16 2015

[2] Burak Turhan Tim Menzies Ayse B Bener and Justin Di Stefano On the relativevalue of cross-company and within-company data for defect prediction EmpiricalSoftware Engineering 14(5)540ndash578 2009

[3] Lech Madeyski and Marian Jureczko Which process metrics can significantlyimprove defect prediction models an empirical study Software Quality Journal23(3)393ndash422 2015

[4] Thomas Shippey David Bowes and Tracy Hall Automatically identifying codefeatures for software defect prediction Using ast n-grams Information andSoftware Technology 106142ndash160 2019

[5] Zhiyuan Wan Xin Xia Ahmed E Hassan David Lo Jianwei Yin and XiaohuYang Perceptions expectations and challenges in defect prediction IEEETransactions on Software Engineering 2018

[6] Karim O Elish and Mahmoud O Elish Predicting defect-prone software modulesusing support vector machines Journal of Systems and Software 81(5)649ndash6602008

[7] Puja Ahmad Habibi Victor Amrizal and Rizal Broer Bahaweres Cross-projectdefect prediction for web application using naive bayes (case study Petstoreweb application) In 2018 International Workshop on Big Data and InformationSecurity (IWBIS) pages 13ndash18 IEEE 2018

[8] Michael J Siers and Md Zahidul Islam Software defect prediction using a costsensitive decision forest and voting and a potential solution to the class imbalanceproblem Information Systems 5162ndash71 2015

[9] R Jayanthi and Lilly Florence Software defect prediction techniques using metricsbased on neural network classifier Cluster Computing pages 1ndash12 2018

[10] Yi Liu Yu Fan and Junghui Chen Flame images for oxygen content predictionof combustion systems using dbn Energy amp Fuels 31(8)8776ndash8783 2017

[11] Yi Liu Chao Yang Zengliang Gao and Yuan Yao Ensemble deep kernel learn-ing with application to quality prediction in industrial polymerization processesChemometrics and Intelligent Laboratory Systems 17415ndash21 2018

[12] Qi Xuan Binwei Fang Yi Liu Jinbao Wang Jian Zhang Yayu Zheng and Guan-jun Bao Automatic pearl classification machine based on a multistream convolu-tional neural network IEEE Transactions on Industrial Electronics 65(8)6538ndash6547 2018

[13] Qi Xuan Haoquan Xiao Chenbo Fu and Yi Liu Evolving convolutional neuralnetwork and its application in fine-grained visual categorization IEEE Access631110ndash31116 2018

[14] Qi Xuan Zhuangzhi Chen Yi Liu Huimin Huang Guanjun Bao and Dan ZhangMultiview generative adversarial network and its application in pearl classificationIEEE Transactions on Industrial Electronics 66(10)8244ndash8252 2019

[15] Qi Xuan Fuxian Li Yi Liu and Yun Xiang MV-C3D A spatial correlatedmulti-view 3d convolutional neural networks IEEE Access 792528ndash925382019

[16] Song Wang Taiyue Liu and Lin Tan Automatically learning semantic featuresfor defect prediction In Software Engineering (ICSE) 2016 IEEEACM 38thInternational Conference on pages 297ndash308 IEEE 2016

[17] Jian Li Pinjia He Jieming Zhu and Michael R Lyu Software defect predictionvia convolutional neural network In Software Quality Reliability and Security(QRS) 2017 IEEE International Conference on pages 318ndash328 IEEE 2017

[18] Hoa Khanh Dam Trang Pham Shien Wee Ng Truyen Tran John Grundy AdityaGhose Taeksu Kim and Chul-Joo Kim A deep tree-based model for softwaredefect prediction arXiv preprint arXiv180200921 2018

[19] Alex Krizhevsky Ilya Sutskever and Geoffrey E Hinton Imagenet classificationwith deep convolutional neural networks In Advances in neural informationprocessing systems pages 1097ndash1105 2012

[20] Mingsheng Long Yue Cao Jianmin Wang and Michael I Jordan Learning trans-ferable features with deep adaptation networks arXiv preprint arXiv1502027912015

[21] Han Zhang Ian Goodfellow Dimitris Metaxas and Augustus Odena Self-attention generative adversarial networks arXiv preprint arXiv1805083182018

[22] Tian Jiang Lin Tan and Sunghun Kim Personalized defect prediction In Pro-ceedings of the 28th IEEEACM International Conference on Automated SoftwareEngineering pages 279ndash289 IEEE Press 2013

[23] Andrew Meneely Laurie Williams Will Snipes and Jason Osborne Predictingfailures with developer networks and social network analysis In Proceedings ofthe 16th ACM SIGSOFT International Symposium on Foundations of softwareengineering pages 13ndash23 ACM 2008

[24] Raimund Moser Witold Pedrycz and Giancarlo Succi A comparative analysis ofthe efficiency of change metrics and static code attributes for defect prediction InProceedings of the 30th international conference on Software engineering pages181ndash190 ACM 2008

[25] Foyzur Rahman and Premkumar Devanbu How and why process metrics arebetter In Software Engineering (ICSE) 2013 35th International Conference onpages 432ndash441 IEEE 2013

[26] Maurice Howard Halstead et al Elements of software science volume 7 ElsevierNew York 1977

[27] Thomas J McCabe A complexity measure IEEE Transactions on softwareEngineering (4)308ndash320 1976

[28] Shyam R Chidamber and Chris F Kemerer A metrics suite for object orienteddesign IEEE Transactions on software engineering 20(6)476ndash493 1994

[29] Yue Yu Huaimin Wang Gang Yin and Tao Wang Reviewer recommendation forpull-requests in github What can we learn from code review and bug assignmentInformation and Software Technology 74204ndash218 2016

11

ICSE 2020 May 23-29 2020 Seoul South Korea Chen et al

[30] Nachiappan Nagappan and Thomas Ball Using software dependencies and churnmetrics to predict field failures An empirical case study In null pages 364ndash373IEEE 2007

[31] Ahmed E Hassan Predicting faults using the complexity of code changes InProceedings of the 31st International Conference on Software Engineering pages78ndash88 IEEE Computer Society 2009

[32] Thomas J Ostrand Elaine J Weyuker and Robert M Bell Programmer-basedfault prediction In Proceedings of the 6th International Conference on PredictiveModels in Software Engineering page 19 ACM 2010

[33] Taek Lee Jaechang Nam DongGyun Han Sunghun Kim and Hoh Peter In Microinteraction metrics for defect prediction In Proceedings of the 19th ACM SIG-SOFT symposium and the 13th European conference on Foundations of softwareengineering pages 311ndash321 ACM 2011

[34] Tim Menzies Zach Milton Burak Turhan Bojan Cukic Yue Jiang and AyseBener Defect prediction from static code features current results limitationsnew approaches Automated Software Engineering 17(4)375ndash407 2010

[35] Martin Pinzger Nachiappan Nagappan and Brendan Murphy Can developer-module networks predict failures In Proceedings of the 16th ACM SIGSOFTInternational Symposium on Foundations of software engineering pages 2ndash12ACM 2008

[36] Steffen Herbold Alexander Trautsch and Jens Grabowski A comparative studyto benchmark cross-project defect prediction approaches IEEE Transactions onSoftware Engineering 44(9)811ndash833 2018

[37] Chao Liu Dan Yang Xin Xia Meng Yan and Xiaohong Zhang A two-phasetransfer learning model for cross-project defect prediction Information andSoftware Technology 107125ndash136 2019

[38] Jinyin Chen Keke Hu Yitao Yang Yi Liu and Qi Xuan Col-lective transfer learning for defect prediction Neurocomputing 2019doihttpsdoiorg101016jneucom201812091

[39] Jinyin Chen Yitao Yang Keke Hu Qi Xuan Yi Liu and Chao Yang Multiviewtransfer learning for software defect prediction IEEE Access 78901ndash8916 2019

[40] Annibale Panichella Rocco Oliveto and Andrea De Lucia Cross-project defectprediction models Lrsquounion fait la force In Software Maintenance Reengineeringand Reverse Engineering (CSMR-WCRE) 2014 Software Evolution Week-IEEEConference on pages 164ndash173 IEEE 2014

[41] Burak Turhan Tim Menzies Ayse B Bener and Justin Di Stefano On the relativevalue of cross-company and within-company data for defect prediction volume 14pages 540ndash578 Springer 2009

[42] Fayola Peters Tim Menzies and Andrian Marcus Better cross company defectprediction In Proceedings of the 10th Working Conference on Mining SoftwareRepositories pages 409ndash418 IEEE Press 2013

[43] Xin Xia LO David Sinno Jialin Pan Nachiappan Nagappan and Xinyu WangHydra Massively compositional model for cross-project defect prediction IEEETransactions on software Engineering 42(10)977 2016

[44] Ying Ma Guangchun Luo Xue Zeng and Aiguo Chen Transfer learning forcross-company software defect prediction Information and Software Technology54(3)248ndash256 2012

[45] Jaechang Nam Sinno Jialin Pan and Sunghun Kim Transfer defect learningIn Software Engineering (ICSE) 2013 35th International Conference on pages382ndash391 IEEE 2013

[46] Sinno Jialin Pan Ivor W Tsang James T Kwok and Qiang Yang Domain adap-tation via transfer component analysis IEEE Transactions on Neural Networks22(2)199ndash210 2011

[47] Blaine A Price Ronald M Baecker and Ian S Small A principled taxonomy ofsoftware visualization Journal of Visual Languages amp Computing 4(3)211ndash2661993

[48] Thomas Ball and Stephen G Eick Software visualization in the large Computer29(4)33ndash43 1996

[49] Sarita Bassil and Rudolf K Keller Software visualization tools Survey andanalysis In Proceedings 9th International Workshop on Program ComprehensionIWPC 2001 pages 7ndash17 IEEE 2001

[50] Stephan Diehl Software visualization visualizing the structure behaviour andevolution of software Springer Science amp Business Media 2007

[51] Michele Lanza The evolution matrix Recovering software evolution using soft-ware visualization techniques In Proceedings of the 4th international workshopon principles of software evolution pages 37ndash42 ACM 2001

[52] Denis Gracanin Krešimir Matkovic and Mohamed Eltoweissy Software vi-sualization Innovations in Systems and Software Engineering 1(2)221ndash2302005

[53] Guillaume Langelier Houari Sahraoui and Pierre Poulin Visualization-basedanalysis of quality for large-scale software systems In Proceedings of the 20thIEEEACM international Conference on Automated software engineering pages214ndash223 ACM 2005

[54] Abien Fred Agarap and Francis John Hill Pepito Towards building an intelligentanti-malware system a deep learning approach using support vector machine(svm) for malware classification arXiv preprint arXiv180100318 2017

[55] Karl Weiss Taghi M Khoshgoftaar and DingDing Wang A survey of transferlearning Journal of Big Data 3(1)9 2016

[56] Rahul Krishna and Tim Menzies Bellwethers A baseline method for transferlearning IEEE Transactions on Software Engineering 2018

[57] Chuanqi Tan Fuchun Sun Tao Kong Wenchang Zhang Chao Yang and ChunfangLiu A survey on deep transfer learning In ICANN 2018

[58] Wenyuan Dai Qiang Yang Gui Rong Xue and Yong Yu Boosting for transferlearning In International Conference on Machine Learning pages 193ndash2002007

[59] Chang Wan Rong Pan and Jiefei Li Bi-weighting domain adaptation for cross-language text classification In IJCAI Proceedings-International Joint Conferenceon Artificial Intelligence volume 22 page 1535 2011

[60] Yonghui Xu Sinno Jialin Pan Hui Xiong Qingyao Wu Ronghua Luo HuaqingMin and Hengjie Song A unified framework for metric transfer learning IEEETrans Knowl Data Eng 29(6)1158ndash1171 2017

[61] Chetak Kandaswamy Luiacutes M Silva Luiacutes A Alexandre and Jorge M SantosDeep transfer learning ensemble for classification In Ignacio Rojas GonzaloJoya and Andreu Catala editors Advances in Computational Intelligence pages335ndash348 Cham 2015 Springer International Publishing

[62] Maxime Oquab Leon Bottou Ivan Laptev and Josef Sivic Learning and trans-ferring mid-level image representations using convolutional neural networks InProceedings of the IEEE conference on computer vision and pattern recognitionpages 1717ndash1724 2014

[63] Yann LeCun Leacuteon Bottou Yoshua Bengio and Patrick Haffner Gradient-basedlearning applied to document recognition Proceedings of the IEEE 86(11)2278ndash2324 1998

[64] Hana Ajakan Pascal Germain Hugo Larochelle Franccedilois Laviolette and MarioMarchand Domain-adversarial neural networks arXiv preprint arXiv141244462014

[65] Yaroslav Ganin and Victor Lempitsky Unsupervised domain adaptation by back-propagation arXiv preprint arXiv14097495 2014

[66] Jing Zhang Wanqing Li and Philip Ogunbona Joint geometrical and statisticalalignment for visual domain adaptation arXiv preprint arXiv170505498 2017

[67] Eric Tzeng Judy Hoffman Ning Zhang Kate Saenko and Trevor DarrellDeep domain confusion Maximizing for domain invariance arXiv preprintarXiv14123474 2014

[68] Mingsheng Long Yue Cao Zhangjie Cao Jianmin Wang and Michael I Jor-dan Transferable representation learning with deep adaptation networks IEEEtransactions on pattern analysis and machine intelligence 2018

[69] Arthur Gretton Dino Sejdinovic Heiko Strathmann Sivaraman BalakrishnanMassimiliano Pontil Kenji Fukumizu and Bharath K Sriperumbudur Optimalkernel choice for large-scale two-sample tests In Advances in neural informationprocessing systems pages 1205ndash1213 2012

[70] Mingsheng Long Han Zhu Jianmin Wang and Michael I Jordan Deep transferlearning with joint adaptation networks arXiv preprint arXiv160506636 2016

[71] Martin Arjovsky Soumith Chintala and Leacuteon Bottou Wasserstein gan arXivpreprint arXiv170107875 2017

[72] Kesav Kancherla and Srinivas Mukkamala Image visualization based malwaredetection In 2013 IEEE Symposium on Computational Intelligence in CyberSecurity (CICS) pages 40ndash44 IEEE 2013

[73] Xin Zhou Jianmin Pang and Guanghui Liang Image classification for malwaredetection using extremely randomized trees In 2017 11th IEEE InternationalConference on Anti-counterfeiting Security and Identification (ASID) pages54ndash59 IEEE 2017

[74] John Murphy An overview of convolutional neural network architectures for deeplearning 2016

[75] Song Wang Taiyue Liu Jaechang Nam and Lin Tan Deep semantic feature learn-ing for software defect prediction IEEE Transactions on Software Engineering2018

[76] Laurens van der Maaten and Geoffrey Hinton Visualizing data using t-sneJournal of machine learning research 9(Nov)2579ndash2605 2008

12

• Abstract
• 1 Introduction
• 2 Related Work
• 3 Approach
• • 31 Source Code Visualization
  • 32 Modeling with Deep Transfer Learning
  • 33 Model Sensitivity to Parameter Choices
  • 34 Training and Prediction
  • • 4 Experimental Setup
    • • 41 Dataset Description
      • 42 Baseline Comparison Methods
      • 43 Performance measures
      • • 5 Results
        • • 51 RQ1 How does DTL-DP compare to feature- based machine learning methods and AST- based deep learning methods in WPDP
          • 52 RQ2 How does DTL-DP compare to feature- based Machine Learning and AST-based deep learning methods in CPDP
          • 53 RQ3 How much does each of the three mechanisms ie data augmentation transfer learning and self-attention mechanism contribute to DTL-DPs performance
          • • 6 Threats to Validity
            • 7 Conclusions and Future Work
            • References

ICSE 2020 May 23-29 2020 Seoul South Korea Chen et al

Table 2 Dataset Description

Project Description Versions Files Avg files Avg size(kb) Defective

ant Java based build tool 151617 1465 488 62 134camel Enterprise integration framework 121416 3140 1046 29 187jEdit Text editor designed for programmers 324041 1935 645 87 192log4j Logging library for Java 1011 300 150 34 497

lucene Text search engine library 202224 607 402 38 358xalan A library for transforming XML files 2425 1984 992 46 296xerces XML parser 1213 1647 549 29 157

ivy Dependency management library 1420 622 311 41 200synapse Data transport adapters 101112 661 220 38 227

poi Java library to access Microsoft format files 152530 1248 416 36 407

Table 3 F-measure of DTL-DP Semantic (Seman) PROMISE-DP (PROM) DP-LSTM (LSTM) and DP-CNN (CNN) forWPDP

Project Version Seman PROM LSTM CNN DTL-DP

ant15-gt16 914 477 452 532 70016-gt17 942 542 379 566 593

camel12-gt14 785 373 323 461 40614-gt16 374 391 400 508 385

jEdit32-gt40 574 556 472 564 49440-gt41 615 546 490 580 687

log4j 10-gt11 701 587 539 632 688

lucene20-gt22 651 502 751 761 78322-gt24 773 605 751 721 776

xalan 24-gt25 595 518 657 601 794xerces 12-gt13 411 238 268 374 820

ivy 14-gt20 350 329 191 347 829

synapse10-gt11 544 476 454 539 54711-gt12 583 533 533 556 659

poi15-gt25 640 558 808 589 68525-gt30 803 754 777 784 426

Average 641 499 515 570 642

51 RQ1 How does DTL-DP compare to feature-based machine learning methods and AST-based deep learning methods in WPDP

We compare DTL-DP to 4 baseline approaches representing twodifferent kinds of defect prediction methods PROMISE-DP is thebaseline representative of traditional feature-based machine learningmethods Semantic DP-LSTM and DP-CNN are the baselines fordeep learning-type methods based on extracting features from ASTGuided by prior work we conducted 16 sets of WPDP experimentseach using two versions of the same project The older version isused to train the prediction model and the newer version is used toevaluate the trained model

Table 3 shows the F-measure values for the within-project WPDPdefect prediction experiments The highest F-measure values of the5 methods are shown in bold Since the methods based on deeplearning include some randomness we run DTL-DP Semantic DP-LSTM and DP-CNN 10 times for each experiment On average

the F-measure of our approach is 0642 and the PROMISE-DPSemantic DP-LSTM and DP-CNN achieve 0499 0641 0515 and0570 respectively The results demonstrate that our approach iscompetitive and may improve on defect prediction compared toPROMISE-DP DP-LSTM and DP-CNN The results of Semanticand our approach are similar

The proposed DTL-DP is effective and it could improvethe performance of WPDP tasks Like other deep learners it issensitive to small file sizes and unbalanced data

511 Case Study WPDP Discrimination of DTL-DP t-SNEis a non-linear dimensional reduction algorithm that is effectivein visualizing similarities and helping identify clusters in complexdata sets [76] To give insight in the performance of DTL-DP wedemonstrate feature transferability by showing t-SNE embeddingsin Figure 6 The blue points are non-defective files and the red aredefective ones5 We observe the following (1) The target instancesare not discriminated very well using either the traditional manualfeatures or the TCA+ improved manual features or the semanticfeatures extracted from ASTs while with our approach the pointsare discriminated much better (2) With the other three approachesthe categories between source and target projects are not well alignedwhile with our approach the categories between the projects aremore consistent These conclusions are derived from the intra- andinter-class distances of the two categories in Figures 6 and 7 Theyare visually apparent

Our method did not perform as well as the comparison methods onsome projects such as ant which is likely caused by large variancein file sizes While the sizes of ant-15 and ant-16 are close there isa marked difference between ant-16 and ant-17 the former beingmuch smaller than the latter From Table 3 the performance of ourmethod on ant15-gtant16 is better than that on ant16-gtant17On the contrary baseline methods other than DP-LSTM performbetter for the task ant16-gtant17 notably Semantic is dominantindicating that the semantic feature-based method is more robust tofile size variability Moreover the ant dataset has two shortcomingsthe first is that the amount of data is small cf Table 2 where theaverage amount is only 488 files in one project The second is that

5Since we use data augmentation we have more samples than the three baselines

8

Software Visualization and Deep Transfer Learningfor Effective Software Defect Prediction ICSE 2020 May 23-29 2020 Seoul South Korea

Figure 6 t-SNE mapping of source and target features (WPDP)

the classes are unbalanced and the proportion of defective files is134 which is the least of all our projects This makes training ofthe deep model more difficult which leads to the poorer performanceof our method on some projects

Table 4 F-measure of DTL-DP DBN-CP (DBN) TCA+ DP-LSTM (LSTM) and DP-CNN (CNN) in CPDP

Source Target DBN TCA+ LSTM CNN DTL-DPant16 camel14 316 292 321 323 395

jEdit41 camel14 693 330 318 651 407camel14 ant16 979 616 448 607 591

poi30 ant16 478 598 386 532 693camel14 jEdit41 615 537 394 547 531log4j11 jEdit41 503 419 389 423 639jEdit41 log4j11 645 574 574 656 783

lucene22 log4j11 618 571 578 632 794lucene22 xalan25 550 530 680 540 689xerces13 xalan25 572 581 676 562 686xalan25 lucene22 594 561 750 621 783log4j11 lucene22 692 524 750 663 769xalan25 xerces13 386 394 340 391 400ivy20 xerces13 426 398 261 421 420

xerces13 ivy20 453 409 264 467 472synapse12 ivy20 824 383 261 371 494

ivy14 synapse11 489 348 451 491 545poi25 synapse11 425 376 435 436 597ivy20 synapse12 433 570 530 456 620poi30 synapse12 514 542 503 532 623

synapse12 poi30 661 651 785 671 827ant16 poi30 619 343 785 627 827

Average 568 479 495 528 618

52 RQ2 How does DTL-DP compare to feature-based Machine Learning and AST-based deeplearning methods in CPDP

Here we compare to TCA+ and DBN-CP instead of PROMISEand Semantic as the baseline approaches as explained in Sect 42

Again guided by prior work we conducted 22 sets of CPDP experi-ments In each we randomly select two project versions from twodifferent projects one as a training set and the other as a test set

Table 4 presents the F-measure results of DTL-CP and the 4 base-line approaches The highest F-measure values are in bold On aver-age the F-measure of our approach in CPDP is 0618 and the DBN-CP TCA+ DP-LSTM and DP-CNN achieve 0568 0479 0495 and0528 Thus DTL-DP outperforms them by 88 290 248 and155 respectively In addition we found that for projects log4j11and poi30 our method does better than the corresponding WPDPbest performing method

Our proposed DTL-DP shows significant improvements onthe state-of-the-art in cross-project defect prediction

521 Case Study CPDP Discrimination of DTL-DP Ourmethod is more obviously dominant in CPDP than in WPDP Apossible reason for that is that deep transfer learning makes the dis-tributions of the training and test samples more similar in featurespace Another reason might be the superior ability of deep modelsto represent features enabling the model to obtain more transferablefeatures from the images To gain more insight we choose the taskpoi30 rarr ant16 and show the t-SNE embeddings in Figure 7 Wemake similar observations as in the RQ1 case study that (1) the tar-get instances are more easily discriminated with our approach and(2) the target instances can be better discriminated with the sourceclassifier This implies that our approach can learn more transferablefeatures for more effective defect prediction

Figure 7 t-SNE mapping of source and target features (CPDP)

53 RQ3 How much does each of the threemechanisms ie data augmentation transferlearning and self-attention mechanismcontribute to DTL-DPrsquos performance

To find out the specific contributions of the three parts to defectprediction we conducted further experiments We built the originalAlexNet model for binary classification and use it as the Base +TL+Attention and +DataAug are three new baselines built by adding tothe base AlexNet one of three mechanisms (transfer learning selfattention and data augmentation) respectively It should be noted

9

ICSE 2020 May 23-29 2020 Seoul South Korea Chen et al

that in addition to +DataAug we use images generated by onesequence of R G and B as training and test sets in each experimentTherefore the experimental results of Base +TL and +Attention inTables 5 and 6 are average results obtained over the 6 different RGBpermutations as described above

Table 5 Contributions of the three mechanisms to WPDP

Project Version Base +TL +Atten +Aug DTL-DP

ant15-gt16 660 676 664 677 70016-gt17 525 552 524 591 593

camel12-gt14 415 381 416 403 40614-gt16 412 396 423 384 385

jEdit32-gt40 484 505 497 485 49440-gt41 629 649 645 679 687

log4j 10-gt11 688 679 692 658 688

lucene20-gt22 667 773 778 761 78322-gt24 749 713 769 758 776

xalan 24-gt25 757 754 758 778 794xerces 12-gt13 809 822 800 788 820

ivy 14-gt20 820 824 827 812 829

synapse10-gt11 525 484 511 530 54711-gt12 618 608 612 616 659

poi15-gt25 663 593 654 614 68525-gt30 410 449 420 432 426

Average 614 616 624 623 642

Table 6 Contributions of the three mechanisms to CPDP

Source Target Base +TL +Atten +Aug DTL-DPant16 camel14 383 379 365 377 395

jEdit41 camel14 368 394 395 371 407camel14 ant16 670 667 653 596 591

poi30 ant16 606 627 634 593 693camel14 jEdit41 663 657 652 647 531log4j11 jEdit41 599 599 592 594 639jEdit41 log4j11 754 800 774 746 783

lucene22 log4j11 759 750 753 783 794lucene22 xalan25 661 664 650 647 689xerces13 xalan25 510 638 626 662 686xalan25 lucene22 730 751 710 769 783log4j11 lucene22 713 745 740 727 769xalan25 xerces13 387 355 393 373 400ivy20 xerces13 413 408 398 378 420

xerces13 ivy20 518 500 504 500 472synapse12 ivy20 521 490 509 476 494

ivy14 synapse11 551 581 553 536 545poi25 synapse11 508 541 516 545 597ivy20 synapse12 604 622 619 611 620poi30 synapse12 600 632 613 612 623

synapse12 poi30 770 804 781 817 827ant16 poi30 777 662 700 812 827

Average 594 603 597 599 618

Table 7 Time cost of the three mechanisms defect visualization(Visualiz) self-attention (Attent) and transfer learning (TL)

ProjectTime (s)

TL Attent Visualizant 4504 131 459

camel 9655 242 785jEdit 5915 266 497log4j 1371 131 176

lucene 3739 186 354xalan 9444 271 823xerces 5117 214 275

ivy 2902 142 305synapse 2042 126 209

poi 3865 157 221Average 4856 187 410

To discern the contribution of each mechanism we compare theperformance of these baseline methods in WPDP and CPDP Fromthe results in Table 5 and Table 6 we observe that the three mecha-nisms each contribute substantially to the accuracy of DTL-DP inboth WPDP and CPDP In terms of the F-measure transfer learn-ing contributes the least improvement and self-attention and dataaugmentation contribute similarly to the final result But in CPDPtransfer learning contributes to the F-measure the most This islikely because of the applicability of transfer learning in the CPDPsetting and is in a way a validation of the approach

We also noted the time cost for the 3 mechanisms in our proposedDTL-DP Table 7 shows the result The most time spent during dataaugmentation is on converting code into images ie source codevisualization For transfer learning most time is spent on the MK-MMD calculation and for self-attention on the calculation of theattention layer in Fig 4 Eg for the project ant Table 3 shows twosets of WPDP experiments ant 15 rarr 16 and ant 16 rarr 17 Onaverage it takes 4504 seconds 1301 seconds and 459 seconds forthe 3 parts respectively for both the training data and the test dataTransfer learning takes the longest time more than the sum of theother two This is because of the large number of matrices neededto calculate MMD with the kernel function The least time cost isincurred by the attention mechanism and its contribution to WPDPis the largest of the three ie it is most cost-effective

The three mechanisms all contribute toward the accuracy ofour proposed end-to-end framework in WPDP and CPDP Self-attention has the greatest contribution to WPDP and transferlearning contributes most to CPDP

6 THREATS TO VALIDITYThreats to internal validity come from experimental errors and thereplication of the baseline methods In order to compare and analyzethe deep learning-based defect prediction techniques we compareour proposed DTL-DP method with Sementic DBN-CP DP-LSTMand DP-CNN In addition our method is also compared with thetransfer learning-based method TCA+ which is the state-of-the-art CPDP technique Since the original implementations were not

10

Software Visualization and Deep Transfer Learningfor Effective Software Defect Prediction ICSE 2020 May 23-29 2020 Seoul South Korea

available we re-implemented our own versions of the baselinesAlthough we strictly follow the procedures described in their workour new implementations may not completely restore all of theiroriginal implementation details And the randomness of the deeplearning-based approach also makes the results of our implementedexperiments different from the original Since we have removed theentries in the PROMISE dataset that cannot retrieve the correspond-ing source files our re-implemented experimental results may notbe consistent with the original baselines

The external threat to the validity of the results lies in the general-izability of the results We have tested our method on 10 open sourceJava projects including 14600 files Defect predictions for instancesof other languages such as C C++ etc need to be validated byadditional experiments in the future

Threats to construct validity depend on the appropriateness ofthe evaluation measurement The F-measure is used as our mainevaluation measure which has been applied in many previous effortsto evaluate defect prediction comprehensively

7 CONCLUSIONS AND FUTURE WORKHere we made two main contributions code visualization for defectprediction and an improved deep transfer learning model Our exper-imental results on 10 open source projects show that deep learningcan be effectively applied directly for defect prediction after apply-ing visualization methods to the code Specifically our approachDTL-DP performs at the top of the range of state-of-the-art WPDPapproaches For CPDP DTL-DP improves on the state-of-the-arttechnique TCA+ built on traditional features by 290 (in the F-measure) It also bests the deep learning-based approaches DBN-CPDP-LSTM and DP-CNN by 88 248 and 155 respectively

DTL-DP still has some limitations For some projects a problemof negative transfer occurs resulting in a worse prediction than adirect prediction Large differences between two projects can causesuch negative transfer Reducing the impact of negative transfer isone of the problems to be solved in the future Additionally theamount of training data we had was small and class imbalance isinherent in software defect prediction Both can be improved withmore data which we plan to obtain in the future

ACKNOWLEDGEMENTSThis work is partially supported by Zhejiang Natural Science Foun-dation(LY19F020025) National Natural Science Foundation ofChina (61502423 61572439 61702534) Zhejiang University OpenFund(2018KFJJ07) Signal Recognition Based on GAN Deep Learn-ing for Enhancement Recognition Project Zhejiang Science andTechnology Plan Project (LGF18F030009 2017C33149) and Zhe-jiang Outstanding Youth Fund (LR19F030001)

REFERENCES[1] Romi Satria Wahono A systematic literature review of software defect predic-

tion research trends datasets methods and frameworks Journal of SoftwareEngineering 1(1)1ndash16 2015

[2] Burak Turhan Tim Menzies Ayse B Bener and Justin Di Stefano On the relativevalue of cross-company and within-company data for defect prediction EmpiricalSoftware Engineering 14(5)540ndash578 2009

[3] Lech Madeyski and Marian Jureczko Which process metrics can significantlyimprove defect prediction models an empirical study Software Quality Journal23(3)393ndash422 2015

[4] Thomas Shippey David Bowes and Tracy Hall Automatically identifying codefeatures for software defect prediction Using ast n-grams Information andSoftware Technology 106142ndash160 2019

[5] Zhiyuan Wan Xin Xia Ahmed E Hassan David Lo Jianwei Yin and XiaohuYang Perceptions expectations and challenges in defect prediction IEEETransactions on Software Engineering 2018

[6] Karim O Elish and Mahmoud O Elish Predicting defect-prone software modulesusing support vector machines Journal of Systems and Software 81(5)649ndash6602008

[7] Puja Ahmad Habibi Victor Amrizal and Rizal Broer Bahaweres Cross-projectdefect prediction for web application using naive bayes (case study Petstoreweb application) In 2018 International Workshop on Big Data and InformationSecurity (IWBIS) pages 13ndash18 IEEE 2018

[8] Michael J Siers and Md Zahidul Islam Software defect prediction using a costsensitive decision forest and voting and a potential solution to the class imbalanceproblem Information Systems 5162ndash71 2015

[9] R Jayanthi and Lilly Florence Software defect prediction techniques using metricsbased on neural network classifier Cluster Computing pages 1ndash12 2018

[10] Yi Liu Yu Fan and Junghui Chen Flame images for oxygen content predictionof combustion systems using dbn Energy amp Fuels 31(8)8776ndash8783 2017

[11] Yi Liu Chao Yang Zengliang Gao and Yuan Yao Ensemble deep kernel learn-ing with application to quality prediction in industrial polymerization processesChemometrics and Intelligent Laboratory Systems 17415ndash21 2018

[12] Qi Xuan Binwei Fang Yi Liu Jinbao Wang Jian Zhang Yayu Zheng and Guan-jun Bao Automatic pearl classification machine based on a multistream convolu-tional neural network IEEE Transactions on Industrial Electronics 65(8)6538ndash6547 2018

[13] Qi Xuan Haoquan Xiao Chenbo Fu and Yi Liu Evolving convolutional neuralnetwork and its application in fine-grained visual categorization IEEE Access631110ndash31116 2018

[14] Qi Xuan Zhuangzhi Chen Yi Liu Huimin Huang Guanjun Bao and Dan ZhangMultiview generative adversarial network and its application in pearl classificationIEEE Transactions on Industrial Electronics 66(10)8244ndash8252 2019

[15] Qi Xuan Fuxian Li Yi Liu and Yun Xiang MV-C3D A spatial correlatedmulti-view 3d convolutional neural networks IEEE Access 792528ndash925382019

[16] Song Wang Taiyue Liu and Lin Tan Automatically learning semantic featuresfor defect prediction In Software Engineering (ICSE) 2016 IEEEACM 38thInternational Conference on pages 297ndash308 IEEE 2016

[17] Jian Li Pinjia He Jieming Zhu and Michael R Lyu Software defect predictionvia convolutional neural network In Software Quality Reliability and Security(QRS) 2017 IEEE International Conference on pages 318ndash328 IEEE 2017

[18] Hoa Khanh Dam Trang Pham Shien Wee Ng Truyen Tran John Grundy AdityaGhose Taeksu Kim and Chul-Joo Kim A deep tree-based model for softwaredefect prediction arXiv preprint arXiv180200921 2018

[19] Alex Krizhevsky Ilya Sutskever and Geoffrey E Hinton Imagenet classificationwith deep convolutional neural networks In Advances in neural informationprocessing systems pages 1097ndash1105 2012

[20] Mingsheng Long Yue Cao Jianmin Wang and Michael I Jordan Learning trans-ferable features with deep adaptation networks arXiv preprint arXiv1502027912015

[21] Han Zhang Ian Goodfellow Dimitris Metaxas and Augustus Odena Self-attention generative adversarial networks arXiv preprint arXiv1805083182018

[22] Tian Jiang Lin Tan and Sunghun Kim Personalized defect prediction In Pro-ceedings of the 28th IEEEACM International Conference on Automated SoftwareEngineering pages 279ndash289 IEEE Press 2013

[23] Andrew Meneely Laurie Williams Will Snipes and Jason Osborne Predictingfailures with developer networks and social network analysis In Proceedings ofthe 16th ACM SIGSOFT International Symposium on Foundations of softwareengineering pages 13ndash23 ACM 2008

[24] Raimund Moser Witold Pedrycz and Giancarlo Succi A comparative analysis ofthe efficiency of change metrics and static code attributes for defect prediction InProceedings of the 30th international conference on Software engineering pages181ndash190 ACM 2008

[25] Foyzur Rahman and Premkumar Devanbu How and why process metrics arebetter In Software Engineering (ICSE) 2013 35th International Conference onpages 432ndash441 IEEE 2013

[26] Maurice Howard Halstead et al Elements of software science volume 7 ElsevierNew York 1977

[27] Thomas J McCabe A complexity measure IEEE Transactions on softwareEngineering (4)308ndash320 1976

[28] Shyam R Chidamber and Chris F Kemerer A metrics suite for object orienteddesign IEEE Transactions on software engineering 20(6)476ndash493 1994

[29] Yue Yu Huaimin Wang Gang Yin and Tao Wang Reviewer recommendation forpull-requests in github What can we learn from code review and bug assignmentInformation and Software Technology 74204ndash218 2016

11

ICSE 2020 May 23-29 2020 Seoul South Korea Chen et al

[30] Nachiappan Nagappan and Thomas Ball Using software dependencies and churnmetrics to predict field failures An empirical case study In null pages 364ndash373IEEE 2007

[31] Ahmed E Hassan Predicting faults using the complexity of code changes InProceedings of the 31st International Conference on Software Engineering pages78ndash88 IEEE Computer Society 2009

[32] Thomas J Ostrand Elaine J Weyuker and Robert M Bell Programmer-basedfault prediction In Proceedings of the 6th International Conference on PredictiveModels in Software Engineering page 19 ACM 2010

[33] Taek Lee Jaechang Nam DongGyun Han Sunghun Kim and Hoh Peter In Microinteraction metrics for defect prediction In Proceedings of the 19th ACM SIG-SOFT symposium and the 13th European conference on Foundations of softwareengineering pages 311ndash321 ACM 2011

[34] Tim Menzies Zach Milton Burak Turhan Bojan Cukic Yue Jiang and AyseBener Defect prediction from static code features current results limitationsnew approaches Automated Software Engineering 17(4)375ndash407 2010

[35] Martin Pinzger Nachiappan Nagappan and Brendan Murphy Can developer-module networks predict failures In Proceedings of the 16th ACM SIGSOFTInternational Symposium on Foundations of software engineering pages 2ndash12ACM 2008

[36] Steffen Herbold Alexander Trautsch and Jens Grabowski A comparative studyto benchmark cross-project defect prediction approaches IEEE Transactions onSoftware Engineering 44(9)811ndash833 2018

[37] Chao Liu Dan Yang Xin Xia Meng Yan and Xiaohong Zhang A two-phasetransfer learning model for cross-project defect prediction Information andSoftware Technology 107125ndash136 2019

[38] Jinyin Chen Keke Hu Yitao Yang Yi Liu and Qi Xuan Col-lective transfer learning for defect prediction Neurocomputing 2019doihttpsdoiorg101016jneucom201812091

[39] Jinyin Chen Yitao Yang Keke Hu Qi Xuan Yi Liu and Chao Yang Multiviewtransfer learning for software defect prediction IEEE Access 78901ndash8916 2019

[40] Annibale Panichella Rocco Oliveto and Andrea De Lucia Cross-project defectprediction models Lrsquounion fait la force In Software Maintenance Reengineeringand Reverse Engineering (CSMR-WCRE) 2014 Software Evolution Week-IEEEConference on pages 164ndash173 IEEE 2014

[41] Burak Turhan Tim Menzies Ayse B Bener and Justin Di Stefano On the relativevalue of cross-company and within-company data for defect prediction volume 14pages 540ndash578 Springer 2009

[42] Fayola Peters Tim Menzies and Andrian Marcus Better cross company defectprediction In Proceedings of the 10th Working Conference on Mining SoftwareRepositories pages 409ndash418 IEEE Press 2013

[43] Xin Xia LO David Sinno Jialin Pan Nachiappan Nagappan and Xinyu WangHydra Massively compositional model for cross-project defect prediction IEEETransactions on software Engineering 42(10)977 2016

[44] Ying Ma Guangchun Luo Xue Zeng and Aiguo Chen Transfer learning forcross-company software defect prediction Information and Software Technology54(3)248ndash256 2012

[45] Jaechang Nam Sinno Jialin Pan and Sunghun Kim Transfer defect learningIn Software Engineering (ICSE) 2013 35th International Conference on pages382ndash391 IEEE 2013

[46] Sinno Jialin Pan Ivor W Tsang James T Kwok and Qiang Yang Domain adap-tation via transfer component analysis IEEE Transactions on Neural Networks22(2)199ndash210 2011

[47] Blaine A Price Ronald M Baecker and Ian S Small A principled taxonomy ofsoftware visualization Journal of Visual Languages amp Computing 4(3)211ndash2661993

[48] Thomas Ball and Stephen G Eick Software visualization in the large Computer29(4)33ndash43 1996

[49] Sarita Bassil and Rudolf K Keller Software visualization tools Survey andanalysis In Proceedings 9th International Workshop on Program ComprehensionIWPC 2001 pages 7ndash17 IEEE 2001

[50] Stephan Diehl Software visualization visualizing the structure behaviour andevolution of software Springer Science amp Business Media 2007

[51] Michele Lanza The evolution matrix Recovering software evolution using soft-ware visualization techniques In Proceedings of the 4th international workshopon principles of software evolution pages 37ndash42 ACM 2001

[52] Denis Gracanin Krešimir Matkovic and Mohamed Eltoweissy Software vi-sualization Innovations in Systems and Software Engineering 1(2)221ndash2302005

[53] Guillaume Langelier Houari Sahraoui and Pierre Poulin Visualization-basedanalysis of quality for large-scale software systems In Proceedings of the 20thIEEEACM international Conference on Automated software engineering pages214ndash223 ACM 2005

[54] Abien Fred Agarap and Francis John Hill Pepito Towards building an intelligentanti-malware system a deep learning approach using support vector machine(svm) for malware classification arXiv preprint arXiv180100318 2017

[55] Karl Weiss Taghi M Khoshgoftaar and DingDing Wang A survey of transferlearning Journal of Big Data 3(1)9 2016

[56] Rahul Krishna and Tim Menzies Bellwethers A baseline method for transferlearning IEEE Transactions on Software Engineering 2018

[57] Chuanqi Tan Fuchun Sun Tao Kong Wenchang Zhang Chao Yang and ChunfangLiu A survey on deep transfer learning In ICANN 2018

[58] Wenyuan Dai Qiang Yang Gui Rong Xue and Yong Yu Boosting for transferlearning In International Conference on Machine Learning pages 193ndash2002007

[59] Chang Wan Rong Pan and Jiefei Li Bi-weighting domain adaptation for cross-language text classification In IJCAI Proceedings-International Joint Conferenceon Artificial Intelligence volume 22 page 1535 2011

[60] Yonghui Xu Sinno Jialin Pan Hui Xiong Qingyao Wu Ronghua Luo HuaqingMin and Hengjie Song A unified framework for metric transfer learning IEEETrans Knowl Data Eng 29(6)1158ndash1171 2017

[61] Chetak Kandaswamy Luiacutes M Silva Luiacutes A Alexandre and Jorge M SantosDeep transfer learning ensemble for classification In Ignacio Rojas GonzaloJoya and Andreu Catala editors Advances in Computational Intelligence pages335ndash348 Cham 2015 Springer International Publishing

[62] Maxime Oquab Leon Bottou Ivan Laptev and Josef Sivic Learning and trans-ferring mid-level image representations using convolutional neural networks InProceedings of the IEEE conference on computer vision and pattern recognitionpages 1717ndash1724 2014

[63] Yann LeCun Leacuteon Bottou Yoshua Bengio and Patrick Haffner Gradient-basedlearning applied to document recognition Proceedings of the IEEE 86(11)2278ndash2324 1998

[64] Hana Ajakan Pascal Germain Hugo Larochelle Franccedilois Laviolette and MarioMarchand Domain-adversarial neural networks arXiv preprint arXiv141244462014

[65] Yaroslav Ganin and Victor Lempitsky Unsupervised domain adaptation by back-propagation arXiv preprint arXiv14097495 2014

[66] Jing Zhang Wanqing Li and Philip Ogunbona Joint geometrical and statisticalalignment for visual domain adaptation arXiv preprint arXiv170505498 2017

[67] Eric Tzeng Judy Hoffman Ning Zhang Kate Saenko and Trevor DarrellDeep domain confusion Maximizing for domain invariance arXiv preprintarXiv14123474 2014

[68] Mingsheng Long Yue Cao Zhangjie Cao Jianmin Wang and Michael I Jor-dan Transferable representation learning with deep adaptation networks IEEEtransactions on pattern analysis and machine intelligence 2018

[69] Arthur Gretton Dino Sejdinovic Heiko Strathmann Sivaraman BalakrishnanMassimiliano Pontil Kenji Fukumizu and Bharath K Sriperumbudur Optimalkernel choice for large-scale two-sample tests In Advances in neural informationprocessing systems pages 1205ndash1213 2012

[70] Mingsheng Long Han Zhu Jianmin Wang and Michael I Jordan Deep transferlearning with joint adaptation networks arXiv preprint arXiv160506636 2016

[71] Martin Arjovsky Soumith Chintala and Leacuteon Bottou Wasserstein gan arXivpreprint arXiv170107875 2017

[72] Kesav Kancherla and Srinivas Mukkamala Image visualization based malwaredetection In 2013 IEEE Symposium on Computational Intelligence in CyberSecurity (CICS) pages 40ndash44 IEEE 2013

[73] Xin Zhou Jianmin Pang and Guanghui Liang Image classification for malwaredetection using extremely randomized trees In 2017 11th IEEE InternationalConference on Anti-counterfeiting Security and Identification (ASID) pages54ndash59 IEEE 2017

[74] John Murphy An overview of convolutional neural network architectures for deeplearning 2016

[75] Song Wang Taiyue Liu Jaechang Nam and Lin Tan Deep semantic feature learn-ing for software defect prediction IEEE Transactions on Software Engineering2018

[76] Laurens van der Maaten and Geoffrey Hinton Visualizing data using t-sneJournal of machine learning research 9(Nov)2579ndash2605 2008

12

• Abstract
• 1 Introduction
• 2 Related Work
• 3 Approach
• • 31 Source Code Visualization
  • 32 Modeling with Deep Transfer Learning
  • 33 Model Sensitivity to Parameter Choices
  • 34 Training and Prediction
  • • 4 Experimental Setup
    • • 41 Dataset Description
      • 42 Baseline Comparison Methods
      • 43 Performance measures
      • • 5 Results
        • • 51 RQ1 How does DTL-DP compare to feature- based machine learning methods and AST- based deep learning methods in WPDP
          • 52 RQ2 How does DTL-DP compare to feature- based Machine Learning and AST-based deep learning methods in CPDP
          • 53 RQ3 How much does each of the three mechanisms ie data augmentation transfer learning and self-attention mechanism contribute to DTL-DPs performance
          • • 6 Threats to Validity
            • 7 Conclusions and Future Work
            • References

Software Visualization and Deep Transfer Learningfor Effective Software Defect Prediction ICSE 2020 May 23-29 2020 Seoul South Korea

Figure 6 t-SNE mapping of source and target features (WPDP)

the classes are unbalanced and the proportion of defective files is134 which is the least of all our projects This makes training ofthe deep model more difficult which leads to the poorer performanceof our method on some projects

Table 4 F-measure of DTL-DP DBN-CP (DBN) TCA+ DP-LSTM (LSTM) and DP-CNN (CNN) in CPDP

Source Target DBN TCA+ LSTM CNN DTL-DPant16 camel14 316 292 321 323 395

jEdit41 camel14 693 330 318 651 407camel14 ant16 979 616 448 607 591

poi30 ant16 478 598 386 532 693camel14 jEdit41 615 537 394 547 531log4j11 jEdit41 503 419 389 423 639jEdit41 log4j11 645 574 574 656 783

lucene22 log4j11 618 571 578 632 794lucene22 xalan25 550 530 680 540 689xerces13 xalan25 572 581 676 562 686xalan25 lucene22 594 561 750 621 783log4j11 lucene22 692 524 750 663 769xalan25 xerces13 386 394 340 391 400ivy20 xerces13 426 398 261 421 420

xerces13 ivy20 453 409 264 467 472synapse12 ivy20 824 383 261 371 494

ivy14 synapse11 489 348 451 491 545poi25 synapse11 425 376 435 436 597ivy20 synapse12 433 570 530 456 620poi30 synapse12 514 542 503 532 623

synapse12 poi30 661 651 785 671 827ant16 poi30 619 343 785 627 827

Average 568 479 495 528 618

52 RQ2 How does DTL-DP compare to feature-based Machine Learning and AST-based deeplearning methods in CPDP

Here we compare to TCA+ and DBN-CP instead of PROMISEand Semantic as the baseline approaches as explained in Sect 42

Again guided by prior work we conducted 22 sets of CPDP experi-ments In each we randomly select two project versions from twodifferent projects one as a training set and the other as a test set

Table 4 presents the F-measure results of DTL-CP and the 4 base-line approaches The highest F-measure values are in bold On aver-age the F-measure of our approach in CPDP is 0618 and the DBN-CP TCA+ DP-LSTM and DP-CNN achieve 0568 0479 0495 and0528 Thus DTL-DP outperforms them by 88 290 248 and155 respectively In addition we found that for projects log4j11and poi30 our method does better than the corresponding WPDPbest performing method

Our proposed DTL-DP shows significant improvements onthe state-of-the-art in cross-project defect prediction

521 Case Study CPDP Discrimination of DTL-DP Ourmethod is more obviously dominant in CPDP than in WPDP Apossible reason for that is that deep transfer learning makes the dis-tributions of the training and test samples more similar in featurespace Another reason might be the superior ability of deep modelsto represent features enabling the model to obtain more transferablefeatures from the images To gain more insight we choose the taskpoi30 rarr ant16 and show the t-SNE embeddings in Figure 7 Wemake similar observations as in the RQ1 case study that (1) the tar-get instances are more easily discriminated with our approach and(2) the target instances can be better discriminated with the sourceclassifier This implies that our approach can learn more transferablefeatures for more effective defect prediction

Figure 7 t-SNE mapping of source and target features (CPDP)

53 RQ3 How much does each of the threemechanisms ie data augmentation transferlearning and self-attention mechanismcontribute to DTL-DPrsquos performance

To find out the specific contributions of the three parts to defectprediction we conducted further experiments We built the originalAlexNet model for binary classification and use it as the Base +TL+Attention and +DataAug are three new baselines built by adding tothe base AlexNet one of three mechanisms (transfer learning selfattention and data augmentation) respectively It should be noted

9

ICSE 2020 May 23-29 2020 Seoul South Korea Chen et al

that in addition to +DataAug we use images generated by onesequence of R G and B as training and test sets in each experimentTherefore the experimental results of Base +TL and +Attention inTables 5 and 6 are average results obtained over the 6 different RGBpermutations as described above

Table 5 Contributions of the three mechanisms to WPDP

Project Version Base +TL +Atten +Aug DTL-DP

ant15-gt16 660 676 664 677 70016-gt17 525 552 524 591 593

camel12-gt14 415 381 416 403 40614-gt16 412 396 423 384 385

jEdit32-gt40 484 505 497 485 49440-gt41 629 649 645 679 687

log4j 10-gt11 688 679 692 658 688

lucene20-gt22 667 773 778 761 78322-gt24 749 713 769 758 776

xalan 24-gt25 757 754 758 778 794xerces 12-gt13 809 822 800 788 820

ivy 14-gt20 820 824 827 812 829

synapse10-gt11 525 484 511 530 54711-gt12 618 608 612 616 659

poi15-gt25 663 593 654 614 68525-gt30 410 449 420 432 426

Average 614 616 624 623 642

Table 6 Contributions of the three mechanisms to CPDP

Source Target Base +TL +Atten +Aug DTL-DPant16 camel14 383 379 365 377 395

jEdit41 camel14 368 394 395 371 407camel14 ant16 670 667 653 596 591

poi30 ant16 606 627 634 593 693camel14 jEdit41 663 657 652 647 531log4j11 jEdit41 599 599 592 594 639jEdit41 log4j11 754 800 774 746 783

lucene22 log4j11 759 750 753 783 794lucene22 xalan25 661 664 650 647 689xerces13 xalan25 510 638 626 662 686xalan25 lucene22 730 751 710 769 783log4j11 lucene22 713 745 740 727 769xalan25 xerces13 387 355 393 373 400ivy20 xerces13 413 408 398 378 420

xerces13 ivy20 518 500 504 500 472synapse12 ivy20 521 490 509 476 494

ivy14 synapse11 551 581 553 536 545poi25 synapse11 508 541 516 545 597ivy20 synapse12 604 622 619 611 620poi30 synapse12 600 632 613 612 623

synapse12 poi30 770 804 781 817 827ant16 poi30 777 662 700 812 827

Average 594 603 597 599 618

Table 7 Time cost of the three mechanisms defect visualization(Visualiz) self-attention (Attent) and transfer learning (TL)

ProjectTime (s)

TL Attent Visualizant 4504 131 459

camel 9655 242 785jEdit 5915 266 497log4j 1371 131 176

lucene 3739 186 354xalan 9444 271 823xerces 5117 214 275

ivy 2902 142 305synapse 2042 126 209

poi 3865 157 221Average 4856 187 410

To discern the contribution of each mechanism we compare theperformance of these baseline methods in WPDP and CPDP Fromthe results in Table 5 and Table 6 we observe that the three mecha-nisms each contribute substantially to the accuracy of DTL-DP inboth WPDP and CPDP In terms of the F-measure transfer learn-ing contributes the least improvement and self-attention and dataaugmentation contribute similarly to the final result But in CPDPtransfer learning contributes to the F-measure the most This islikely because of the applicability of transfer learning in the CPDPsetting and is in a way a validation of the approach

We also noted the time cost for the 3 mechanisms in our proposedDTL-DP Table 7 shows the result The most time spent during dataaugmentation is on converting code into images ie source codevisualization For transfer learning most time is spent on the MK-MMD calculation and for self-attention on the calculation of theattention layer in Fig 4 Eg for the project ant Table 3 shows twosets of WPDP experiments ant 15 rarr 16 and ant 16 rarr 17 Onaverage it takes 4504 seconds 1301 seconds and 459 seconds forthe 3 parts respectively for both the training data and the test dataTransfer learning takes the longest time more than the sum of theother two This is because of the large number of matrices neededto calculate MMD with the kernel function The least time cost isincurred by the attention mechanism and its contribution to WPDPis the largest of the three ie it is most cost-effective

The three mechanisms all contribute toward the accuracy ofour proposed end-to-end framework in WPDP and CPDP Self-attention has the greatest contribution to WPDP and transferlearning contributes most to CPDP

6 THREATS TO VALIDITYThreats to internal validity come from experimental errors and thereplication of the baseline methods In order to compare and analyzethe deep learning-based defect prediction techniques we compareour proposed DTL-DP method with Sementic DBN-CP DP-LSTMand DP-CNN In addition our method is also compared with thetransfer learning-based method TCA+ which is the state-of-the-art CPDP technique Since the original implementations were not

10

Software Visualization and Deep Transfer Learningfor Effective Software Defect Prediction ICSE 2020 May 23-29 2020 Seoul South Korea

available we re-implemented our own versions of the baselinesAlthough we strictly follow the procedures described in their workour new implementations may not completely restore all of theiroriginal implementation details And the randomness of the deeplearning-based approach also makes the results of our implementedexperiments different from the original Since we have removed theentries in the PROMISE dataset that cannot retrieve the correspond-ing source files our re-implemented experimental results may notbe consistent with the original baselines

The external threat to the validity of the results lies in the general-izability of the results We have tested our method on 10 open sourceJava projects including 14600 files Defect predictions for instancesof other languages such as C C++ etc need to be validated byadditional experiments in the future

Threats to construct validity depend on the appropriateness ofthe evaluation measurement The F-measure is used as our mainevaluation measure which has been applied in many previous effortsto evaluate defect prediction comprehensively

7 CONCLUSIONS AND FUTURE WORKHere we made two main contributions code visualization for defectprediction and an improved deep transfer learning model Our exper-imental results on 10 open source projects show that deep learningcan be effectively applied directly for defect prediction after apply-ing visualization methods to the code Specifically our approachDTL-DP performs at the top of the range of state-of-the-art WPDPapproaches For CPDP DTL-DP improves on the state-of-the-arttechnique TCA+ built on traditional features by 290 (in the F-measure) It also bests the deep learning-based approaches DBN-CPDP-LSTM and DP-CNN by 88 248 and 155 respectively

DTL-DP still has some limitations For some projects a problemof negative transfer occurs resulting in a worse prediction than adirect prediction Large differences between two projects can causesuch negative transfer Reducing the impact of negative transfer isone of the problems to be solved in the future Additionally theamount of training data we had was small and class imbalance isinherent in software defect prediction Both can be improved withmore data which we plan to obtain in the future

ACKNOWLEDGEMENTSThis work is partially supported by Zhejiang Natural Science Foun-dation(LY19F020025) National Natural Science Foundation ofChina (61502423 61572439 61702534) Zhejiang University OpenFund(2018KFJJ07) Signal Recognition Based on GAN Deep Learn-ing for Enhancement Recognition Project Zhejiang Science andTechnology Plan Project (LGF18F030009 2017C33149) and Zhe-jiang Outstanding Youth Fund (LR19F030001)

REFERENCES[1] Romi Satria Wahono A systematic literature review of software defect predic-

tion research trends datasets methods and frameworks Journal of SoftwareEngineering 1(1)1ndash16 2015

[2] Burak Turhan Tim Menzies Ayse B Bener and Justin Di Stefano On the relativevalue of cross-company and within-company data for defect prediction EmpiricalSoftware Engineering 14(5)540ndash578 2009

[3] Lech Madeyski and Marian Jureczko Which process metrics can significantlyimprove defect prediction models an empirical study Software Quality Journal23(3)393ndash422 2015

[4] Thomas Shippey David Bowes and Tracy Hall Automatically identifying codefeatures for software defect prediction Using ast n-grams Information andSoftware Technology 106142ndash160 2019

[5] Zhiyuan Wan Xin Xia Ahmed E Hassan David Lo Jianwei Yin and XiaohuYang Perceptions expectations and challenges in defect prediction IEEETransactions on Software Engineering 2018

[6] Karim O Elish and Mahmoud O Elish Predicting defect-prone software modulesusing support vector machines Journal of Systems and Software 81(5)649ndash6602008

[7] Puja Ahmad Habibi Victor Amrizal and Rizal Broer Bahaweres Cross-projectdefect prediction for web application using naive bayes (case study Petstoreweb application) In 2018 International Workshop on Big Data and InformationSecurity (IWBIS) pages 13ndash18 IEEE 2018

[8] Michael J Siers and Md Zahidul Islam Software defect prediction using a costsensitive decision forest and voting and a potential solution to the class imbalanceproblem Information Systems 5162ndash71 2015

[9] R Jayanthi and Lilly Florence Software defect prediction techniques using metricsbased on neural network classifier Cluster Computing pages 1ndash12 2018

[10] Yi Liu Yu Fan and Junghui Chen Flame images for oxygen content predictionof combustion systems using dbn Energy amp Fuels 31(8)8776ndash8783 2017

[11] Yi Liu Chao Yang Zengliang Gao and Yuan Yao Ensemble deep kernel learn-ing with application to quality prediction in industrial polymerization processesChemometrics and Intelligent Laboratory Systems 17415ndash21 2018

[12] Qi Xuan Binwei Fang Yi Liu Jinbao Wang Jian Zhang Yayu Zheng and Guan-jun Bao Automatic pearl classification machine based on a multistream convolu-tional neural network IEEE Transactions on Industrial Electronics 65(8)6538ndash6547 2018

[13] Qi Xuan Haoquan Xiao Chenbo Fu and Yi Liu Evolving convolutional neuralnetwork and its application in fine-grained visual categorization IEEE Access631110ndash31116 2018

[14] Qi Xuan Zhuangzhi Chen Yi Liu Huimin Huang Guanjun Bao and Dan ZhangMultiview generative adversarial network and its application in pearl classificationIEEE Transactions on Industrial Electronics 66(10)8244ndash8252 2019

[15] Qi Xuan Fuxian Li Yi Liu and Yun Xiang MV-C3D A spatial correlatedmulti-view 3d convolutional neural networks IEEE Access 792528ndash925382019

[16] Song Wang Taiyue Liu and Lin Tan Automatically learning semantic featuresfor defect prediction In Software Engineering (ICSE) 2016 IEEEACM 38thInternational Conference on pages 297ndash308 IEEE 2016

[17] Jian Li Pinjia He Jieming Zhu and Michael R Lyu Software defect predictionvia convolutional neural network In Software Quality Reliability and Security(QRS) 2017 IEEE International Conference on pages 318ndash328 IEEE 2017

[18] Hoa Khanh Dam Trang Pham Shien Wee Ng Truyen Tran John Grundy AdityaGhose Taeksu Kim and Chul-Joo Kim A deep tree-based model for softwaredefect prediction arXiv preprint arXiv180200921 2018

[19] Alex Krizhevsky Ilya Sutskever and Geoffrey E Hinton Imagenet classificationwith deep convolutional neural networks In Advances in neural informationprocessing systems pages 1097ndash1105 2012

[20] Mingsheng Long Yue Cao Jianmin Wang and Michael I Jordan Learning trans-ferable features with deep adaptation networks arXiv preprint arXiv1502027912015

[21] Han Zhang Ian Goodfellow Dimitris Metaxas and Augustus Odena Self-attention generative adversarial networks arXiv preprint arXiv1805083182018

[22] Tian Jiang Lin Tan and Sunghun Kim Personalized defect prediction In Pro-ceedings of the 28th IEEEACM International Conference on Automated SoftwareEngineering pages 279ndash289 IEEE Press 2013

[23] Andrew Meneely Laurie Williams Will Snipes and Jason Osborne Predictingfailures with developer networks and social network analysis In Proceedings ofthe 16th ACM SIGSOFT International Symposium on Foundations of softwareengineering pages 13ndash23 ACM 2008

[24] Raimund Moser Witold Pedrycz and Giancarlo Succi A comparative analysis ofthe efficiency of change metrics and static code attributes for defect prediction InProceedings of the 30th international conference on Software engineering pages181ndash190 ACM 2008

[25] Foyzur Rahman and Premkumar Devanbu How and why process metrics arebetter In Software Engineering (ICSE) 2013 35th International Conference onpages 432ndash441 IEEE 2013

[26] Maurice Howard Halstead et al Elements of software science volume 7 ElsevierNew York 1977

[27] Thomas J McCabe A complexity measure IEEE Transactions on softwareEngineering (4)308ndash320 1976

[28] Shyam R Chidamber and Chris F Kemerer A metrics suite for object orienteddesign IEEE Transactions on software engineering 20(6)476ndash493 1994

[29] Yue Yu Huaimin Wang Gang Yin and Tao Wang Reviewer recommendation forpull-requests in github What can we learn from code review and bug assignmentInformation and Software Technology 74204ndash218 2016

11

ICSE 2020 May 23-29 2020 Seoul South Korea Chen et al

[30] Nachiappan Nagappan and Thomas Ball Using software dependencies and churnmetrics to predict field failures An empirical case study In null pages 364ndash373IEEE 2007

[31] Ahmed E Hassan Predicting faults using the complexity of code changes InProceedings of the 31st International Conference on Software Engineering pages78ndash88 IEEE Computer Society 2009

[32] Thomas J Ostrand Elaine J Weyuker and Robert M Bell Programmer-basedfault prediction In Proceedings of the 6th International Conference on PredictiveModels in Software Engineering page 19 ACM 2010

[33] Taek Lee Jaechang Nam DongGyun Han Sunghun Kim and Hoh Peter In Microinteraction metrics for defect prediction In Proceedings of the 19th ACM SIG-SOFT symposium and the 13th European conference on Foundations of softwareengineering pages 311ndash321 ACM 2011

[34] Tim Menzies Zach Milton Burak Turhan Bojan Cukic Yue Jiang and AyseBener Defect prediction from static code features current results limitationsnew approaches Automated Software Engineering 17(4)375ndash407 2010

[35] Martin Pinzger Nachiappan Nagappan and Brendan Murphy Can developer-module networks predict failures In Proceedings of the 16th ACM SIGSOFTInternational Symposium on Foundations of software engineering pages 2ndash12ACM 2008

[36] Steffen Herbold Alexander Trautsch and Jens Grabowski A comparative studyto benchmark cross-project defect prediction approaches IEEE Transactions onSoftware Engineering 44(9)811ndash833 2018

[37] Chao Liu Dan Yang Xin Xia Meng Yan and Xiaohong Zhang A two-phasetransfer learning model for cross-project defect prediction Information andSoftware Technology 107125ndash136 2019

[38] Jinyin Chen Keke Hu Yitao Yang Yi Liu and Qi Xuan Col-lective transfer learning for defect prediction Neurocomputing 2019doihttpsdoiorg101016jneucom201812091

[39] Jinyin Chen Yitao Yang Keke Hu Qi Xuan Yi Liu and Chao Yang Multiviewtransfer learning for software defect prediction IEEE Access 78901ndash8916 2019

[40] Annibale Panichella Rocco Oliveto and Andrea De Lucia Cross-project defectprediction models Lrsquounion fait la force In Software Maintenance Reengineeringand Reverse Engineering (CSMR-WCRE) 2014 Software Evolution Week-IEEEConference on pages 164ndash173 IEEE 2014

[41] Burak Turhan Tim Menzies Ayse B Bener and Justin Di Stefano On the relativevalue of cross-company and within-company data for defect prediction volume 14pages 540ndash578 Springer 2009

[42] Fayola Peters Tim Menzies and Andrian Marcus Better cross company defectprediction In Proceedings of the 10th Working Conference on Mining SoftwareRepositories pages 409ndash418 IEEE Press 2013

[43] Xin Xia LO David Sinno Jialin Pan Nachiappan Nagappan and Xinyu WangHydra Massively compositional model for cross-project defect prediction IEEETransactions on software Engineering 42(10)977 2016

[44] Ying Ma Guangchun Luo Xue Zeng and Aiguo Chen Transfer learning forcross-company software defect prediction Information and Software Technology54(3)248ndash256 2012

[45] Jaechang Nam Sinno Jialin Pan and Sunghun Kim Transfer defect learningIn Software Engineering (ICSE) 2013 35th International Conference on pages382ndash391 IEEE 2013

[46] Sinno Jialin Pan Ivor W Tsang James T Kwok and Qiang Yang Domain adap-tation via transfer component analysis IEEE Transactions on Neural Networks22(2)199ndash210 2011

[47] Blaine A Price Ronald M Baecker and Ian S Small A principled taxonomy ofsoftware visualization Journal of Visual Languages amp Computing 4(3)211ndash2661993

[48] Thomas Ball and Stephen G Eick Software visualization in the large Computer29(4)33ndash43 1996

[49] Sarita Bassil and Rudolf K Keller Software visualization tools Survey andanalysis In Proceedings 9th International Workshop on Program ComprehensionIWPC 2001 pages 7ndash17 IEEE 2001

[50] Stephan Diehl Software visualization visualizing the structure behaviour andevolution of software Springer Science amp Business Media 2007

[51] Michele Lanza The evolution matrix Recovering software evolution using soft-ware visualization techniques In Proceedings of the 4th international workshopon principles of software evolution pages 37ndash42 ACM 2001

[52] Denis Gracanin Krešimir Matkovic and Mohamed Eltoweissy Software vi-sualization Innovations in Systems and Software Engineering 1(2)221ndash2302005

[53] Guillaume Langelier Houari Sahraoui and Pierre Poulin Visualization-basedanalysis of quality for large-scale software systems In Proceedings of the 20thIEEEACM international Conference on Automated software engineering pages214ndash223 ACM 2005

[54] Abien Fred Agarap and Francis John Hill Pepito Towards building an intelligentanti-malware system a deep learning approach using support vector machine(svm) for malware classification arXiv preprint arXiv180100318 2017

[55] Karl Weiss Taghi M Khoshgoftaar and DingDing Wang A survey of transferlearning Journal of Big Data 3(1)9 2016

[56] Rahul Krishna and Tim Menzies Bellwethers A baseline method for transferlearning IEEE Transactions on Software Engineering 2018

[57] Chuanqi Tan Fuchun Sun Tao Kong Wenchang Zhang Chao Yang and ChunfangLiu A survey on deep transfer learning In ICANN 2018

[58] Wenyuan Dai Qiang Yang Gui Rong Xue and Yong Yu Boosting for transferlearning In International Conference on Machine Learning pages 193ndash2002007

[59] Chang Wan Rong Pan and Jiefei Li Bi-weighting domain adaptation for cross-language text classification In IJCAI Proceedings-International Joint Conferenceon Artificial Intelligence volume 22 page 1535 2011

[60] Yonghui Xu Sinno Jialin Pan Hui Xiong Qingyao Wu Ronghua Luo HuaqingMin and Hengjie Song A unified framework for metric transfer learning IEEETrans Knowl Data Eng 29(6)1158ndash1171 2017

[61] Chetak Kandaswamy Luiacutes M Silva Luiacutes A Alexandre and Jorge M SantosDeep transfer learning ensemble for classification In Ignacio Rojas GonzaloJoya and Andreu Catala editors Advances in Computational Intelligence pages335ndash348 Cham 2015 Springer International Publishing

[62] Maxime Oquab Leon Bottou Ivan Laptev and Josef Sivic Learning and trans-ferring mid-level image representations using convolutional neural networks InProceedings of the IEEE conference on computer vision and pattern recognitionpages 1717ndash1724 2014

[63] Yann LeCun Leacuteon Bottou Yoshua Bengio and Patrick Haffner Gradient-basedlearning applied to document recognition Proceedings of the IEEE 86(11)2278ndash2324 1998

[64] Hana Ajakan Pascal Germain Hugo Larochelle Franccedilois Laviolette and MarioMarchand Domain-adversarial neural networks arXiv preprint arXiv141244462014

[65] Yaroslav Ganin and Victor Lempitsky Unsupervised domain adaptation by back-propagation arXiv preprint arXiv14097495 2014

[66] Jing Zhang Wanqing Li and Philip Ogunbona Joint geometrical and statisticalalignment for visual domain adaptation arXiv preprint arXiv170505498 2017

[67] Eric Tzeng Judy Hoffman Ning Zhang Kate Saenko and Trevor DarrellDeep domain confusion Maximizing for domain invariance arXiv preprintarXiv14123474 2014

[68] Mingsheng Long Yue Cao Zhangjie Cao Jianmin Wang and Michael I Jor-dan Transferable representation learning with deep adaptation networks IEEEtransactions on pattern analysis and machine intelligence 2018

[69] Arthur Gretton Dino Sejdinovic Heiko Strathmann Sivaraman BalakrishnanMassimiliano Pontil Kenji Fukumizu and Bharath K Sriperumbudur Optimalkernel choice for large-scale two-sample tests In Advances in neural informationprocessing systems pages 1205ndash1213 2012

[70] Mingsheng Long Han Zhu Jianmin Wang and Michael I Jordan Deep transferlearning with joint adaptation networks arXiv preprint arXiv160506636 2016

[71] Martin Arjovsky Soumith Chintala and Leacuteon Bottou Wasserstein gan arXivpreprint arXiv170107875 2017

[72] Kesav Kancherla and Srinivas Mukkamala Image visualization based malwaredetection In 2013 IEEE Symposium on Computational Intelligence in CyberSecurity (CICS) pages 40ndash44 IEEE 2013

[73] Xin Zhou Jianmin Pang and Guanghui Liang Image classification for malwaredetection using extremely randomized trees In 2017 11th IEEE InternationalConference on Anti-counterfeiting Security and Identification (ASID) pages54ndash59 IEEE 2017

[74] John Murphy An overview of convolutional neural network architectures for deeplearning 2016

[75] Song Wang Taiyue Liu Jaechang Nam and Lin Tan Deep semantic feature learn-ing for software defect prediction IEEE Transactions on Software Engineering2018

[76] Laurens van der Maaten and Geoffrey Hinton Visualizing data using t-sneJournal of machine learning research 9(Nov)2579ndash2605 2008

12

• Abstract
• 1 Introduction
• 2 Related Work
• 3 Approach
• • 31 Source Code Visualization
  • 32 Modeling with Deep Transfer Learning
  • 33 Model Sensitivity to Parameter Choices
  • 34 Training and Prediction
  • • 4 Experimental Setup
    • • 41 Dataset Description
      • 42 Baseline Comparison Methods
      • 43 Performance measures
      • • 5 Results
        • • 51 RQ1 How does DTL-DP compare to feature- based machine learning methods and AST- based deep learning methods in WPDP
          • 52 RQ2 How does DTL-DP compare to feature- based Machine Learning and AST-based deep learning methods in CPDP
          • 53 RQ3 How much does each of the three mechanisms ie data augmentation transfer learning and self-attention mechanism contribute to DTL-DPs performance
          • • 6 Threats to Validity
            • 7 Conclusions and Future Work
            • References

ICSE 2020 May 23-29 2020 Seoul South Korea Chen et al

that in addition to +DataAug we use images generated by onesequence of R G and B as training and test sets in each experimentTherefore the experimental results of Base +TL and +Attention inTables 5 and 6 are average results obtained over the 6 different RGBpermutations as described above

Table 5 Contributions of the three mechanisms to WPDP

Project Version Base +TL +Atten +Aug DTL-DP

ant15-gt16 660 676 664 677 70016-gt17 525 552 524 591 593

camel12-gt14 415 381 416 403 40614-gt16 412 396 423 384 385

jEdit32-gt40 484 505 497 485 49440-gt41 629 649 645 679 687

log4j 10-gt11 688 679 692 658 688

lucene20-gt22 667 773 778 761 78322-gt24 749 713 769 758 776

xalan 24-gt25 757 754 758 778 794xerces 12-gt13 809 822 800 788 820

ivy 14-gt20 820 824 827 812 829

synapse10-gt11 525 484 511 530 54711-gt12 618 608 612 616 659

poi15-gt25 663 593 654 614 68525-gt30 410 449 420 432 426

Average 614 616 624 623 642

Table 6 Contributions of the three mechanisms to CPDP

Source Target Base +TL +Atten +Aug DTL-DPant16 camel14 383 379 365 377 395

jEdit41 camel14 368 394 395 371 407camel14 ant16 670 667 653 596 591

poi30 ant16 606 627 634 593 693camel14 jEdit41 663 657 652 647 531log4j11 jEdit41 599 599 592 594 639jEdit41 log4j11 754 800 774 746 783

lucene22 log4j11 759 750 753 783 794lucene22 xalan25 661 664 650 647 689xerces13 xalan25 510 638 626 662 686xalan25 lucene22 730 751 710 769 783log4j11 lucene22 713 745 740 727 769xalan25 xerces13 387 355 393 373 400ivy20 xerces13 413 408 398 378 420

xerces13 ivy20 518 500 504 500 472synapse12 ivy20 521 490 509 476 494

ivy14 synapse11 551 581 553 536 545poi25 synapse11 508 541 516 545 597ivy20 synapse12 604 622 619 611 620poi30 synapse12 600 632 613 612 623

synapse12 poi30 770 804 781 817 827ant16 poi30 777 662 700 812 827

Average 594 603 597 599 618

Table 7 Time cost of the three mechanisms defect visualization(Visualiz) self-attention (Attent) and transfer learning (TL)

ProjectTime (s)

TL Attent Visualizant 4504 131 459

camel 9655 242 785jEdit 5915 266 497log4j 1371 131 176

lucene 3739 186 354xalan 9444 271 823xerces 5117 214 275

ivy 2902 142 305synapse 2042 126 209

poi 3865 157 221Average 4856 187 410

To discern the contribution of each mechanism we compare theperformance of these baseline methods in WPDP and CPDP Fromthe results in Table 5 and Table 6 we observe that the three mecha-nisms each contribute substantially to the accuracy of DTL-DP inboth WPDP and CPDP In terms of the F-measure transfer learn-ing contributes the least improvement and self-attention and dataaugmentation contribute similarly to the final result But in CPDPtransfer learning contributes to the F-measure the most This islikely because of the applicability of transfer learning in the CPDPsetting and is in a way a validation of the approach

We also noted the time cost for the 3 mechanisms in our proposedDTL-DP Table 7 shows the result The most time spent during dataaugmentation is on converting code into images ie source codevisualization For transfer learning most time is spent on the MK-MMD calculation and for self-attention on the calculation of theattention layer in Fig 4 Eg for the project ant Table 3 shows twosets of WPDP experiments ant 15 rarr 16 and ant 16 rarr 17 Onaverage it takes 4504 seconds 1301 seconds and 459 seconds forthe 3 parts respectively for both the training data and the test dataTransfer learning takes the longest time more than the sum of theother two This is because of the large number of matrices neededto calculate MMD with the kernel function The least time cost isincurred by the attention mechanism and its contribution to WPDPis the largest of the three ie it is most cost-effective

The three mechanisms all contribute toward the accuracy ofour proposed end-to-end framework in WPDP and CPDP Self-attention has the greatest contribution to WPDP and transferlearning contributes most to CPDP

6 THREATS TO VALIDITYThreats to internal validity come from experimental errors and thereplication of the baseline methods In order to compare and analyzethe deep learning-based defect prediction techniques we compareour proposed DTL-DP method with Sementic DBN-CP DP-LSTMand DP-CNN In addition our method is also compared with thetransfer learning-based method TCA+ which is the state-of-the-art CPDP technique Since the original implementations were not

10

Software Visualization and Deep Transfer Learningfor Effective Software Defect Prediction ICSE 2020 May 23-29 2020 Seoul South Korea

available we re-implemented our own versions of the baselinesAlthough we strictly follow the procedures described in their workour new implementations may not completely restore all of theiroriginal implementation details And the randomness of the deeplearning-based approach also makes the results of our implementedexperiments different from the original Since we have removed theentries in the PROMISE dataset that cannot retrieve the correspond-ing source files our re-implemented experimental results may notbe consistent with the original baselines

The external threat to the validity of the results lies in the general-izability of the results We have tested our method on 10 open sourceJava projects including 14600 files Defect predictions for instancesof other languages such as C C++ etc need to be validated byadditional experiments in the future

Threats to construct validity depend on the appropriateness ofthe evaluation measurement The F-measure is used as our mainevaluation measure which has been applied in many previous effortsto evaluate defect prediction comprehensively

7 CONCLUSIONS AND FUTURE WORKHere we made two main contributions code visualization for defectprediction and an improved deep transfer learning model Our exper-imental results on 10 open source projects show that deep learningcan be effectively applied directly for defect prediction after apply-ing visualization methods to the code Specifically our approachDTL-DP performs at the top of the range of state-of-the-art WPDPapproaches For CPDP DTL-DP improves on the state-of-the-arttechnique TCA+ built on traditional features by 290 (in the F-measure) It also bests the deep learning-based approaches DBN-CPDP-LSTM and DP-CNN by 88 248 and 155 respectively

DTL-DP still has some limitations For some projects a problemof negative transfer occurs resulting in a worse prediction than adirect prediction Large differences between two projects can causesuch negative transfer Reducing the impact of negative transfer isone of the problems to be solved in the future Additionally theamount of training data we had was small and class imbalance isinherent in software defect prediction Both can be improved withmore data which we plan to obtain in the future

ACKNOWLEDGEMENTSThis work is partially supported by Zhejiang Natural Science Foun-dation(LY19F020025) National Natural Science Foundation ofChina (61502423 61572439 61702534) Zhejiang University OpenFund(2018KFJJ07) Signal Recognition Based on GAN Deep Learn-ing for Enhancement Recognition Project Zhejiang Science andTechnology Plan Project (LGF18F030009 2017C33149) and Zhe-jiang Outstanding Youth Fund (LR19F030001)

REFERENCES[1] Romi Satria Wahono A systematic literature review of software defect predic-

tion research trends datasets methods and frameworks Journal of SoftwareEngineering 1(1)1ndash16 2015

[2] Burak Turhan Tim Menzies Ayse B Bener and Justin Di Stefano On the relativevalue of cross-company and within-company data for defect prediction EmpiricalSoftware Engineering 14(5)540ndash578 2009

[3] Lech Madeyski and Marian Jureczko Which process metrics can significantlyimprove defect prediction models an empirical study Software Quality Journal23(3)393ndash422 2015

[4] Thomas Shippey David Bowes and Tracy Hall Automatically identifying codefeatures for software defect prediction Using ast n-grams Information andSoftware Technology 106142ndash160 2019

[5] Zhiyuan Wan Xin Xia Ahmed E Hassan David Lo Jianwei Yin and XiaohuYang Perceptions expectations and challenges in defect prediction IEEETransactions on Software Engineering 2018

[6] Karim O Elish and Mahmoud O Elish Predicting defect-prone software modulesusing support vector machines Journal of Systems and Software 81(5)649ndash6602008

[7] Puja Ahmad Habibi Victor Amrizal and Rizal Broer Bahaweres Cross-projectdefect prediction for web application using naive bayes (case study Petstoreweb application) In 2018 International Workshop on Big Data and InformationSecurity (IWBIS) pages 13ndash18 IEEE 2018

[8] Michael J Siers and Md Zahidul Islam Software defect prediction using a costsensitive decision forest and voting and a potential solution to the class imbalanceproblem Information Systems 5162ndash71 2015

[9] R Jayanthi and Lilly Florence Software defect prediction techniques using metricsbased on neural network classifier Cluster Computing pages 1ndash12 2018

[10] Yi Liu Yu Fan and Junghui Chen Flame images for oxygen content predictionof combustion systems using dbn Energy amp Fuels 31(8)8776ndash8783 2017

[11] Yi Liu Chao Yang Zengliang Gao and Yuan Yao Ensemble deep kernel learn-ing with application to quality prediction in industrial polymerization processesChemometrics and Intelligent Laboratory Systems 17415ndash21 2018

[12] Qi Xuan Binwei Fang Yi Liu Jinbao Wang Jian Zhang Yayu Zheng and Guan-jun Bao Automatic pearl classification machine based on a multistream convolu-tional neural network IEEE Transactions on Industrial Electronics 65(8)6538ndash6547 2018

[13] Qi Xuan Haoquan Xiao Chenbo Fu and Yi Liu Evolving convolutional neuralnetwork and its application in fine-grained visual categorization IEEE Access631110ndash31116 2018

[14] Qi Xuan Zhuangzhi Chen Yi Liu Huimin Huang Guanjun Bao and Dan ZhangMultiview generative adversarial network and its application in pearl classificationIEEE Transactions on Industrial Electronics 66(10)8244ndash8252 2019

[15] Qi Xuan Fuxian Li Yi Liu and Yun Xiang MV-C3D A spatial correlatedmulti-view 3d convolutional neural networks IEEE Access 792528ndash925382019

[16] Song Wang Taiyue Liu and Lin Tan Automatically learning semantic featuresfor defect prediction In Software Engineering (ICSE) 2016 IEEEACM 38thInternational Conference on pages 297ndash308 IEEE 2016

[17] Jian Li Pinjia He Jieming Zhu and Michael R Lyu Software defect predictionvia convolutional neural network In Software Quality Reliability and Security(QRS) 2017 IEEE International Conference on pages 318ndash328 IEEE 2017

[18] Hoa Khanh Dam Trang Pham Shien Wee Ng Truyen Tran John Grundy AdityaGhose Taeksu Kim and Chul-Joo Kim A deep tree-based model for softwaredefect prediction arXiv preprint arXiv180200921 2018

[19] Alex Krizhevsky Ilya Sutskever and Geoffrey E Hinton Imagenet classificationwith deep convolutional neural networks In Advances in neural informationprocessing systems pages 1097ndash1105 2012

[20] Mingsheng Long Yue Cao Jianmin Wang and Michael I Jordan Learning trans-ferable features with deep adaptation networks arXiv preprint arXiv1502027912015

[21] Han Zhang Ian Goodfellow Dimitris Metaxas and Augustus Odena Self-attention generative adversarial networks arXiv preprint arXiv1805083182018

[22] Tian Jiang Lin Tan and Sunghun Kim Personalized defect prediction In Pro-ceedings of the 28th IEEEACM International Conference on Automated SoftwareEngineering pages 279ndash289 IEEE Press 2013

[23] Andrew Meneely Laurie Williams Will Snipes and Jason Osborne Predictingfailures with developer networks and social network analysis In Proceedings ofthe 16th ACM SIGSOFT International Symposium on Foundations of softwareengineering pages 13ndash23 ACM 2008

[24] Raimund Moser Witold Pedrycz and Giancarlo Succi A comparative analysis ofthe efficiency of change metrics and static code attributes for defect prediction InProceedings of the 30th international conference on Software engineering pages181ndash190 ACM 2008

[25] Foyzur Rahman and Premkumar Devanbu How and why process metrics arebetter In Software Engineering (ICSE) 2013 35th International Conference onpages 432ndash441 IEEE 2013

[26] Maurice Howard Halstead et al Elements of software science volume 7 ElsevierNew York 1977

[27] Thomas J McCabe A complexity measure IEEE Transactions on softwareEngineering (4)308ndash320 1976

[28] Shyam R Chidamber and Chris F Kemerer A metrics suite for object orienteddesign IEEE Transactions on software engineering 20(6)476ndash493 1994

[29] Yue Yu Huaimin Wang Gang Yin and Tao Wang Reviewer recommendation forpull-requests in github What can we learn from code review and bug assignmentInformation and Software Technology 74204ndash218 2016

11

ICSE 2020 May 23-29 2020 Seoul South Korea Chen et al

[30] Nachiappan Nagappan and Thomas Ball Using software dependencies and churnmetrics to predict field failures An empirical case study In null pages 364ndash373IEEE 2007

[31] Ahmed E Hassan Predicting faults using the complexity of code changes InProceedings of the 31st International Conference on Software Engineering pages78ndash88 IEEE Computer Society 2009

[32] Thomas J Ostrand Elaine J Weyuker and Robert M Bell Programmer-basedfault prediction In Proceedings of the 6th International Conference on PredictiveModels in Software Engineering page 19 ACM 2010

[33] Taek Lee Jaechang Nam DongGyun Han Sunghun Kim and Hoh Peter In Microinteraction metrics for defect prediction In Proceedings of the 19th ACM SIG-SOFT symposium and the 13th European conference on Foundations of softwareengineering pages 311ndash321 ACM 2011

[34] Tim Menzies Zach Milton Burak Turhan Bojan Cukic Yue Jiang and AyseBener Defect prediction from static code features current results limitationsnew approaches Automated Software Engineering 17(4)375ndash407 2010

[35] Martin Pinzger Nachiappan Nagappan and Brendan Murphy Can developer-module networks predict failures In Proceedings of the 16th ACM SIGSOFTInternational Symposium on Foundations of software engineering pages 2ndash12ACM 2008

[36] Steffen Herbold Alexander Trautsch and Jens Grabowski A comparative studyto benchmark cross-project defect prediction approaches IEEE Transactions onSoftware Engineering 44(9)811ndash833 2018

[37] Chao Liu Dan Yang Xin Xia Meng Yan and Xiaohong Zhang A two-phasetransfer learning model for cross-project defect prediction Information andSoftware Technology 107125ndash136 2019

[38] Jinyin Chen Keke Hu Yitao Yang Yi Liu and Qi Xuan Col-lective transfer learning for defect prediction Neurocomputing 2019doihttpsdoiorg101016jneucom201812091

[39] Jinyin Chen Yitao Yang Keke Hu Qi Xuan Yi Liu and Chao Yang Multiviewtransfer learning for software defect prediction IEEE Access 78901ndash8916 2019

[40] Annibale Panichella Rocco Oliveto and Andrea De Lucia Cross-project defectprediction models Lrsquounion fait la force In Software Maintenance Reengineeringand Reverse Engineering (CSMR-WCRE) 2014 Software Evolution Week-IEEEConference on pages 164ndash173 IEEE 2014

[41] Burak Turhan Tim Menzies Ayse B Bener and Justin Di Stefano On the relativevalue of cross-company and within-company data for defect prediction volume 14pages 540ndash578 Springer 2009

[42] Fayola Peters Tim Menzies and Andrian Marcus Better cross company defectprediction In Proceedings of the 10th Working Conference on Mining SoftwareRepositories pages 409ndash418 IEEE Press 2013

[43] Xin Xia LO David Sinno Jialin Pan Nachiappan Nagappan and Xinyu WangHydra Massively compositional model for cross-project defect prediction IEEETransactions on software Engineering 42(10)977 2016

[44] Ying Ma Guangchun Luo Xue Zeng and Aiguo Chen Transfer learning forcross-company software defect prediction Information and Software Technology54(3)248ndash256 2012

[45] Jaechang Nam Sinno Jialin Pan and Sunghun Kim Transfer defect learningIn Software Engineering (ICSE) 2013 35th International Conference on pages382ndash391 IEEE 2013

[46] Sinno Jialin Pan Ivor W Tsang James T Kwok and Qiang Yang Domain adap-tation via transfer component analysis IEEE Transactions on Neural Networks22(2)199ndash210 2011

[47] Blaine A Price Ronald M Baecker and Ian S Small A principled taxonomy ofsoftware visualization Journal of Visual Languages amp Computing 4(3)211ndash2661993

[48] Thomas Ball and Stephen G Eick Software visualization in the large Computer29(4)33ndash43 1996

[49] Sarita Bassil and Rudolf K Keller Software visualization tools Survey andanalysis In Proceedings 9th International Workshop on Program ComprehensionIWPC 2001 pages 7ndash17 IEEE 2001

[50] Stephan Diehl Software visualization visualizing the structure behaviour andevolution of software Springer Science amp Business Media 2007

[51] Michele Lanza The evolution matrix Recovering software evolution using soft-ware visualization techniques In Proceedings of the 4th international workshopon principles of software evolution pages 37ndash42 ACM 2001

[52] Denis Gracanin Krešimir Matkovic and Mohamed Eltoweissy Software vi-sualization Innovations in Systems and Software Engineering 1(2)221ndash2302005

[53] Guillaume Langelier Houari Sahraoui and Pierre Poulin Visualization-basedanalysis of quality for large-scale software systems In Proceedings of the 20thIEEEACM international Conference on Automated software engineering pages214ndash223 ACM 2005

[54] Abien Fred Agarap and Francis John Hill Pepito Towards building an intelligentanti-malware system a deep learning approach using support vector machine(svm) for malware classification arXiv preprint arXiv180100318 2017

[55] Karl Weiss Taghi M Khoshgoftaar and DingDing Wang A survey of transferlearning Journal of Big Data 3(1)9 2016

[56] Rahul Krishna and Tim Menzies Bellwethers A baseline method for transferlearning IEEE Transactions on Software Engineering 2018

[57] Chuanqi Tan Fuchun Sun Tao Kong Wenchang Zhang Chao Yang and ChunfangLiu A survey on deep transfer learning In ICANN 2018

[58] Wenyuan Dai Qiang Yang Gui Rong Xue and Yong Yu Boosting for transferlearning In International Conference on Machine Learning pages 193ndash2002007

[59] Chang Wan Rong Pan and Jiefei Li Bi-weighting domain adaptation for cross-language text classification In IJCAI Proceedings-International Joint Conferenceon Artificial Intelligence volume 22 page 1535 2011

[60] Yonghui Xu Sinno Jialin Pan Hui Xiong Qingyao Wu Ronghua Luo HuaqingMin and Hengjie Song A unified framework for metric transfer learning IEEETrans Knowl Data Eng 29(6)1158ndash1171 2017

[61] Chetak Kandaswamy Luiacutes M Silva Luiacutes A Alexandre and Jorge M SantosDeep transfer learning ensemble for classification In Ignacio Rojas GonzaloJoya and Andreu Catala editors Advances in Computational Intelligence pages335ndash348 Cham 2015 Springer International Publishing

[62] Maxime Oquab Leon Bottou Ivan Laptev and Josef Sivic Learning and trans-ferring mid-level image representations using convolutional neural networks InProceedings of the IEEE conference on computer vision and pattern recognitionpages 1717ndash1724 2014

[63] Yann LeCun Leacuteon Bottou Yoshua Bengio and Patrick Haffner Gradient-basedlearning applied to document recognition Proceedings of the IEEE 86(11)2278ndash2324 1998

[64] Hana Ajakan Pascal Germain Hugo Larochelle Franccedilois Laviolette and MarioMarchand Domain-adversarial neural networks arXiv preprint arXiv141244462014

[65] Yaroslav Ganin and Victor Lempitsky Unsupervised domain adaptation by back-propagation arXiv preprint arXiv14097495 2014

[66] Jing Zhang Wanqing Li and Philip Ogunbona Joint geometrical and statisticalalignment for visual domain adaptation arXiv preprint arXiv170505498 2017

[67] Eric Tzeng Judy Hoffman Ning Zhang Kate Saenko and Trevor DarrellDeep domain confusion Maximizing for domain invariance arXiv preprintarXiv14123474 2014

[68] Mingsheng Long Yue Cao Zhangjie Cao Jianmin Wang and Michael I Jor-dan Transferable representation learning with deep adaptation networks IEEEtransactions on pattern analysis and machine intelligence 2018

[69] Arthur Gretton Dino Sejdinovic Heiko Strathmann Sivaraman BalakrishnanMassimiliano Pontil Kenji Fukumizu and Bharath K Sriperumbudur Optimalkernel choice for large-scale two-sample tests In Advances in neural informationprocessing systems pages 1205ndash1213 2012

[70] Mingsheng Long Han Zhu Jianmin Wang and Michael I Jordan Deep transferlearning with joint adaptation networks arXiv preprint arXiv160506636 2016

[71] Martin Arjovsky Soumith Chintala and Leacuteon Bottou Wasserstein gan arXivpreprint arXiv170107875 2017

[72] Kesav Kancherla and Srinivas Mukkamala Image visualization based malwaredetection In 2013 IEEE Symposium on Computational Intelligence in CyberSecurity (CICS) pages 40ndash44 IEEE 2013

[73] Xin Zhou Jianmin Pang and Guanghui Liang Image classification for malwaredetection using extremely randomized trees In 2017 11th IEEE InternationalConference on Anti-counterfeiting Security and Identification (ASID) pages54ndash59 IEEE 2017

[74] John Murphy An overview of convolutional neural network architectures for deeplearning 2016

[75] Song Wang Taiyue Liu Jaechang Nam and Lin Tan Deep semantic feature learn-ing for software defect prediction IEEE Transactions on Software Engineering2018

[76] Laurens van der Maaten and Geoffrey Hinton Visualizing data using t-sneJournal of machine learning research 9(Nov)2579ndash2605 2008

12

• Abstract
• 1 Introduction
• 2 Related Work
• 3 Approach
• • 31 Source Code Visualization
  • 32 Modeling with Deep Transfer Learning
  • 33 Model Sensitivity to Parameter Choices
  • 34 Training and Prediction
  • • 4 Experimental Setup
    • • 41 Dataset Description
      • 42 Baseline Comparison Methods
      • 43 Performance measures
      • • 5 Results
        • • 51 RQ1 How does DTL-DP compare to feature- based machine learning methods and AST- based deep learning methods in WPDP
          • 52 RQ2 How does DTL-DP compare to feature- based Machine Learning and AST-based deep learning methods in CPDP
          • 53 RQ3 How much does each of the three mechanisms ie data augmentation transfer learning and self-attention mechanism contribute to DTL-DPs performance
          • • 6 Threats to Validity
            • 7 Conclusions and Future Work
            • References

Software Visualization and Deep Transfer Learningfor Effective Software Defect Prediction ICSE 2020 May 23-29 2020 Seoul South Korea

available we re-implemented our own versions of the baselinesAlthough we strictly follow the procedures described in their workour new implementations may not completely restore all of theiroriginal implementation details And the randomness of the deeplearning-based approach also makes the results of our implementedexperiments different from the original Since we have removed theentries in the PROMISE dataset that cannot retrieve the correspond-ing source files our re-implemented experimental results may notbe consistent with the original baselines

The external threat to the validity of the results lies in the general-izability of the results We have tested our method on 10 open sourceJava projects including 14600 files Defect predictions for instancesof other languages such as C C++ etc need to be validated byadditional experiments in the future

Threats to construct validity depend on the appropriateness ofthe evaluation measurement The F-measure is used as our mainevaluation measure which has been applied in many previous effortsto evaluate defect prediction comprehensively

7 CONCLUSIONS AND FUTURE WORKHere we made two main contributions code visualization for defectprediction and an improved deep transfer learning model Our exper-imental results on 10 open source projects show that deep learningcan be effectively applied directly for defect prediction after apply-ing visualization methods to the code Specifically our approachDTL-DP performs at the top of the range of state-of-the-art WPDPapproaches For CPDP DTL-DP improves on the state-of-the-arttechnique TCA+ built on traditional features by 290 (in the F-measure) It also bests the deep learning-based approaches DBN-CPDP-LSTM and DP-CNN by 88 248 and 155 respectively

DTL-DP still has some limitations For some projects a problemof negative transfer occurs resulting in a worse prediction than adirect prediction Large differences between two projects can causesuch negative transfer Reducing the impact of negative transfer isone of the problems to be solved in the future Additionally theamount of training data we had was small and class imbalance isinherent in software defect prediction Both can be improved withmore data which we plan to obtain in the future

ACKNOWLEDGEMENTSThis work is partially supported by Zhejiang Natural Science Foun-dation(LY19F020025) National Natural Science Foundation ofChina (61502423 61572439 61702534) Zhejiang University OpenFund(2018KFJJ07) Signal Recognition Based on GAN Deep Learn-ing for Enhancement Recognition Project Zhejiang Science andTechnology Plan Project (LGF18F030009 2017C33149) and Zhe-jiang Outstanding Youth Fund (LR19F030001)

REFERENCES[1] Romi Satria Wahono A systematic literature review of software defect predic-

tion research trends datasets methods and frameworks Journal of SoftwareEngineering 1(1)1ndash16 2015

[2] Burak Turhan Tim Menzies Ayse B Bener and Justin Di Stefano On the relativevalue of cross-company and within-company data for defect prediction EmpiricalSoftware Engineering 14(5)540ndash578 2009

[3] Lech Madeyski and Marian Jureczko Which process metrics can significantlyimprove defect prediction models an empirical study Software Quality Journal23(3)393ndash422 2015

[4] Thomas Shippey David Bowes and Tracy Hall Automatically identifying codefeatures for software defect prediction Using ast n-grams Information andSoftware Technology 106142ndash160 2019

[5] Zhiyuan Wan Xin Xia Ahmed E Hassan David Lo Jianwei Yin and XiaohuYang Perceptions expectations and challenges in defect prediction IEEETransactions on Software Engineering 2018

[6] Karim O Elish and Mahmoud O Elish Predicting defect-prone software modulesusing support vector machines Journal of Systems and Software 81(5)649ndash6602008

[7] Puja Ahmad Habibi Victor Amrizal and Rizal Broer Bahaweres Cross-projectdefect prediction for web application using naive bayes (case study Petstoreweb application) In 2018 International Workshop on Big Data and InformationSecurity (IWBIS) pages 13ndash18 IEEE 2018

[8] Michael J Siers and Md Zahidul Islam Software defect prediction using a costsensitive decision forest and voting and a potential solution to the class imbalanceproblem Information Systems 5162ndash71 2015

[9] R Jayanthi and Lilly Florence Software defect prediction techniques using metricsbased on neural network classifier Cluster Computing pages 1ndash12 2018

[10] Yi Liu Yu Fan and Junghui Chen Flame images for oxygen content predictionof combustion systems using dbn Energy amp Fuels 31(8)8776ndash8783 2017

[11] Yi Liu Chao Yang Zengliang Gao and Yuan Yao Ensemble deep kernel learn-ing with application to quality prediction in industrial polymerization processesChemometrics and Intelligent Laboratory Systems 17415ndash21 2018

[12] Qi Xuan Binwei Fang Yi Liu Jinbao Wang Jian Zhang Yayu Zheng and Guan-jun Bao Automatic pearl classification machine based on a multistream convolu-tional neural network IEEE Transactions on Industrial Electronics 65(8)6538ndash6547 2018

[13] Qi Xuan Haoquan Xiao Chenbo Fu and Yi Liu Evolving convolutional neuralnetwork and its application in fine-grained visual categorization IEEE Access631110ndash31116 2018

[14] Qi Xuan Zhuangzhi Chen Yi Liu Huimin Huang Guanjun Bao and Dan ZhangMultiview generative adversarial network and its application in pearl classificationIEEE Transactions on Industrial Electronics 66(10)8244ndash8252 2019

[15] Qi Xuan Fuxian Li Yi Liu and Yun Xiang MV-C3D A spatial correlatedmulti-view 3d convolutional neural networks IEEE Access 792528ndash925382019

[16] Song Wang Taiyue Liu and Lin Tan Automatically learning semantic featuresfor defect prediction In Software Engineering (ICSE) 2016 IEEEACM 38thInternational Conference on pages 297ndash308 IEEE 2016

[17] Jian Li Pinjia He Jieming Zhu and Michael R Lyu Software defect predictionvia convolutional neural network In Software Quality Reliability and Security(QRS) 2017 IEEE International Conference on pages 318ndash328 IEEE 2017

[18] Hoa Khanh Dam Trang Pham Shien Wee Ng Truyen Tran John Grundy AdityaGhose Taeksu Kim and Chul-Joo Kim A deep tree-based model for softwaredefect prediction arXiv preprint arXiv180200921 2018

[19] Alex Krizhevsky Ilya Sutskever and Geoffrey E Hinton Imagenet classificationwith deep convolutional neural networks In Advances in neural informationprocessing systems pages 1097ndash1105 2012

[20] Mingsheng Long Yue Cao Jianmin Wang and Michael I Jordan Learning trans-ferable features with deep adaptation networks arXiv preprint arXiv1502027912015

[21] Han Zhang Ian Goodfellow Dimitris Metaxas and Augustus Odena Self-attention generative adversarial networks arXiv preprint arXiv1805083182018

[22] Tian Jiang Lin Tan and Sunghun Kim Personalized defect prediction In Pro-ceedings of the 28th IEEEACM International Conference on Automated SoftwareEngineering pages 279ndash289 IEEE Press 2013

[23] Andrew Meneely Laurie Williams Will Snipes and Jason Osborne Predictingfailures with developer networks and social network analysis In Proceedings ofthe 16th ACM SIGSOFT International Symposium on Foundations of softwareengineering pages 13ndash23 ACM 2008

[24] Raimund Moser Witold Pedrycz and Giancarlo Succi A comparative analysis ofthe efficiency of change metrics and static code attributes for defect prediction InProceedings of the 30th international conference on Software engineering pages181ndash190 ACM 2008

[25] Foyzur Rahman and Premkumar Devanbu How and why process metrics arebetter In Software Engineering (ICSE) 2013 35th International Conference onpages 432ndash441 IEEE 2013

[26] Maurice Howard Halstead et al Elements of software science volume 7 ElsevierNew York 1977

[27] Thomas J McCabe A complexity measure IEEE Transactions on softwareEngineering (4)308ndash320 1976

[28] Shyam R Chidamber and Chris F Kemerer A metrics suite for object orienteddesign IEEE Transactions on software engineering 20(6)476ndash493 1994

[29] Yue Yu Huaimin Wang Gang Yin and Tao Wang Reviewer recommendation forpull-requests in github What can we learn from code review and bug assignmentInformation and Software Technology 74204ndash218 2016

11

ICSE 2020 May 23-29 2020 Seoul South Korea Chen et al

[30] Nachiappan Nagappan and Thomas Ball Using software dependencies and churnmetrics to predict field failures An empirical case study In null pages 364ndash373IEEE 2007

[31] Ahmed E Hassan Predicting faults using the complexity of code changes InProceedings of the 31st International Conference on Software Engineering pages78ndash88 IEEE Computer Society 2009

[32] Thomas J Ostrand Elaine J Weyuker and Robert M Bell Programmer-basedfault prediction In Proceedings of the 6th International Conference on PredictiveModels in Software Engineering page 19 ACM 2010

[33] Taek Lee Jaechang Nam DongGyun Han Sunghun Kim and Hoh Peter In Microinteraction metrics for defect prediction In Proceedings of the 19th ACM SIG-SOFT symposium and the 13th European conference on Foundations of softwareengineering pages 311ndash321 ACM 2011

[34] Tim Menzies Zach Milton Burak Turhan Bojan Cukic Yue Jiang and AyseBener Defect prediction from static code features current results limitationsnew approaches Automated Software Engineering 17(4)375ndash407 2010

[35] Martin Pinzger Nachiappan Nagappan and Brendan Murphy Can developer-module networks predict failures In Proceedings of the 16th ACM SIGSOFTInternational Symposium on Foundations of software engineering pages 2ndash12ACM 2008

[36] Steffen Herbold Alexander Trautsch and Jens Grabowski A comparative studyto benchmark cross-project defect prediction approaches IEEE Transactions onSoftware Engineering 44(9)811ndash833 2018

[37] Chao Liu Dan Yang Xin Xia Meng Yan and Xiaohong Zhang A two-phasetransfer learning model for cross-project defect prediction Information andSoftware Technology 107125ndash136 2019

[38] Jinyin Chen Keke Hu Yitao Yang Yi Liu and Qi Xuan Col-lective transfer learning for defect prediction Neurocomputing 2019doihttpsdoiorg101016jneucom201812091

[39] Jinyin Chen Yitao Yang Keke Hu Qi Xuan Yi Liu and Chao Yang Multiviewtransfer learning for software defect prediction IEEE Access 78901ndash8916 2019

[40] Annibale Panichella Rocco Oliveto and Andrea De Lucia Cross-project defectprediction models Lrsquounion fait la force In Software Maintenance Reengineeringand Reverse Engineering (CSMR-WCRE) 2014 Software Evolution Week-IEEEConference on pages 164ndash173 IEEE 2014

[41] Burak Turhan Tim Menzies Ayse B Bener and Justin Di Stefano On the relativevalue of cross-company and within-company data for defect prediction volume 14pages 540ndash578 Springer 2009

[42] Fayola Peters Tim Menzies and Andrian Marcus Better cross company defectprediction In Proceedings of the 10th Working Conference on Mining SoftwareRepositories pages 409ndash418 IEEE Press 2013

[43] Xin Xia LO David Sinno Jialin Pan Nachiappan Nagappan and Xinyu WangHydra Massively compositional model for cross-project defect prediction IEEETransactions on software Engineering 42(10)977 2016

[44] Ying Ma Guangchun Luo Xue Zeng and Aiguo Chen Transfer learning forcross-company software defect prediction Information and Software Technology54(3)248ndash256 2012

[45] Jaechang Nam Sinno Jialin Pan and Sunghun Kim Transfer defect learningIn Software Engineering (ICSE) 2013 35th International Conference on pages382ndash391 IEEE 2013

[46] Sinno Jialin Pan Ivor W Tsang James T Kwok and Qiang Yang Domain adap-tation via transfer component analysis IEEE Transactions on Neural Networks22(2)199ndash210 2011

[47] Blaine A Price Ronald M Baecker and Ian S Small A principled taxonomy ofsoftware visualization Journal of Visual Languages amp Computing 4(3)211ndash2661993

[48] Thomas Ball and Stephen G Eick Software visualization in the large Computer29(4)33ndash43 1996

[49] Sarita Bassil and Rudolf K Keller Software visualization tools Survey andanalysis In Proceedings 9th International Workshop on Program ComprehensionIWPC 2001 pages 7ndash17 IEEE 2001

[50] Stephan Diehl Software visualization visualizing the structure behaviour andevolution of software Springer Science amp Business Media 2007

[51] Michele Lanza The evolution matrix Recovering software evolution using soft-ware visualization techniques In Proceedings of the 4th international workshopon principles of software evolution pages 37ndash42 ACM 2001

[52] Denis Gracanin Krešimir Matkovic and Mohamed Eltoweissy Software vi-sualization Innovations in Systems and Software Engineering 1(2)221ndash2302005

[53] Guillaume Langelier Houari Sahraoui and Pierre Poulin Visualization-basedanalysis of quality for large-scale software systems In Proceedings of the 20thIEEEACM international Conference on Automated software engineering pages214ndash223 ACM 2005

[54] Abien Fred Agarap and Francis John Hill Pepito Towards building an intelligentanti-malware system a deep learning approach using support vector machine(svm) for malware classification arXiv preprint arXiv180100318 2017

[55] Karl Weiss Taghi M Khoshgoftaar and DingDing Wang A survey of transferlearning Journal of Big Data 3(1)9 2016

[56] Rahul Krishna and Tim Menzies Bellwethers A baseline method for transferlearning IEEE Transactions on Software Engineering 2018

[57] Chuanqi Tan Fuchun Sun Tao Kong Wenchang Zhang Chao Yang and ChunfangLiu A survey on deep transfer learning In ICANN 2018

[58] Wenyuan Dai Qiang Yang Gui Rong Xue and Yong Yu Boosting for transferlearning In International Conference on Machine Learning pages 193ndash2002007

[59] Chang Wan Rong Pan and Jiefei Li Bi-weighting domain adaptation for cross-language text classification In IJCAI Proceedings-International Joint Conferenceon Artificial Intelligence volume 22 page 1535 2011

[60] Yonghui Xu Sinno Jialin Pan Hui Xiong Qingyao Wu Ronghua Luo HuaqingMin and Hengjie Song A unified framework for metric transfer learning IEEETrans Knowl Data Eng 29(6)1158ndash1171 2017

[61] Chetak Kandaswamy Luiacutes M Silva Luiacutes A Alexandre and Jorge M SantosDeep transfer learning ensemble for classification In Ignacio Rojas GonzaloJoya and Andreu Catala editors Advances in Computational Intelligence pages335ndash348 Cham 2015 Springer International Publishing

[62] Maxime Oquab Leon Bottou Ivan Laptev and Josef Sivic Learning and trans-ferring mid-level image representations using convolutional neural networks InProceedings of the IEEE conference on computer vision and pattern recognitionpages 1717ndash1724 2014

[63] Yann LeCun Leacuteon Bottou Yoshua Bengio and Patrick Haffner Gradient-basedlearning applied to document recognition Proceedings of the IEEE 86(11)2278ndash2324 1998

[64] Hana Ajakan Pascal Germain Hugo Larochelle Franccedilois Laviolette and MarioMarchand Domain-adversarial neural networks arXiv preprint arXiv141244462014

[65] Yaroslav Ganin and Victor Lempitsky Unsupervised domain adaptation by back-propagation arXiv preprint arXiv14097495 2014

[66] Jing Zhang Wanqing Li and Philip Ogunbona Joint geometrical and statisticalalignment for visual domain adaptation arXiv preprint arXiv170505498 2017

[67] Eric Tzeng Judy Hoffman Ning Zhang Kate Saenko and Trevor DarrellDeep domain confusion Maximizing for domain invariance arXiv preprintarXiv14123474 2014

[68] Mingsheng Long Yue Cao Zhangjie Cao Jianmin Wang and Michael I Jor-dan Transferable representation learning with deep adaptation networks IEEEtransactions on pattern analysis and machine intelligence 2018

[69] Arthur Gretton Dino Sejdinovic Heiko Strathmann Sivaraman BalakrishnanMassimiliano Pontil Kenji Fukumizu and Bharath K Sriperumbudur Optimalkernel choice for large-scale two-sample tests In Advances in neural informationprocessing systems pages 1205ndash1213 2012

[70] Mingsheng Long Han Zhu Jianmin Wang and Michael I Jordan Deep transferlearning with joint adaptation networks arXiv preprint arXiv160506636 2016

[71] Martin Arjovsky Soumith Chintala and Leacuteon Bottou Wasserstein gan arXivpreprint arXiv170107875 2017

[72] Kesav Kancherla and Srinivas Mukkamala Image visualization based malwaredetection In 2013 IEEE Symposium on Computational Intelligence in CyberSecurity (CICS) pages 40ndash44 IEEE 2013

[73] Xin Zhou Jianmin Pang and Guanghui Liang Image classification for malwaredetection using extremely randomized trees In 2017 11th IEEE InternationalConference on Anti-counterfeiting Security and Identification (ASID) pages54ndash59 IEEE 2017

[74] John Murphy An overview of convolutional neural network architectures for deeplearning 2016

[75] Song Wang Taiyue Liu Jaechang Nam and Lin Tan Deep semantic feature learn-ing for software defect prediction IEEE Transactions on Software Engineering2018

[76] Laurens van der Maaten and Geoffrey Hinton Visualizing data using t-sneJournal of machine learning research 9(Nov)2579ndash2605 2008

12

• Abstract
• 1 Introduction
• 2 Related Work
• 3 Approach
• • 31 Source Code Visualization
  • 32 Modeling with Deep Transfer Learning
  • 33 Model Sensitivity to Parameter Choices
  • 34 Training and Prediction
  • • 4 Experimental Setup
    • • 41 Dataset Description
      • 42 Baseline Comparison Methods
      • 43 Performance measures
      • • 5 Results
        • • 51 RQ1 How does DTL-DP compare to feature- based machine learning methods and AST- based deep learning methods in WPDP
          • 52 RQ2 How does DTL-DP compare to feature- based Machine Learning and AST-based deep learning methods in CPDP
          • 53 RQ3 How much does each of the three mechanisms ie data augmentation transfer learning and self-attention mechanism contribute to DTL-DPs performance
          • • 6 Threats to Validity
            • 7 Conclusions and Future Work
            • References

ICSE 2020 May 23-29 2020 Seoul South Korea Chen et al

[30] Nachiappan Nagappan and Thomas Ball Using software dependencies and churnmetrics to predict field failures An empirical case study In null pages 364ndash373IEEE 2007

[31] Ahmed E Hassan Predicting faults using the complexity of code changes InProceedings of the 31st International Conference on Software Engineering pages78ndash88 IEEE Computer Society 2009

[32] Thomas J Ostrand Elaine J Weyuker and Robert M Bell Programmer-basedfault prediction In Proceedings of the 6th International Conference on PredictiveModels in Software Engineering page 19 ACM 2010

[33] Taek Lee Jaechang Nam DongGyun Han Sunghun Kim and Hoh Peter In Microinteraction metrics for defect prediction In Proceedings of the 19th ACM SIG-SOFT symposium and the 13th European conference on Foundations of softwareengineering pages 311ndash321 ACM 2011

[34] Tim Menzies Zach Milton Burak Turhan Bojan Cukic Yue Jiang and AyseBener Defect prediction from static code features current results limitationsnew approaches Automated Software Engineering 17(4)375ndash407 2010

[35] Martin Pinzger Nachiappan Nagappan and Brendan Murphy Can developer-module networks predict failures In Proceedings of the 16th ACM SIGSOFTInternational Symposium on Foundations of software engineering pages 2ndash12ACM 2008

[36] Steffen Herbold Alexander Trautsch and Jens Grabowski A comparative studyto benchmark cross-project defect prediction approaches IEEE Transactions onSoftware Engineering 44(9)811ndash833 2018

[37] Chao Liu Dan Yang Xin Xia Meng Yan and Xiaohong Zhang A two-phasetransfer learning model for cross-project defect prediction Information andSoftware Technology 107125ndash136 2019

[38] Jinyin Chen Keke Hu Yitao Yang Yi Liu and Qi Xuan Col-lective transfer learning for defect prediction Neurocomputing 2019doihttpsdoiorg101016jneucom201812091

[39] Jinyin Chen Yitao Yang Keke Hu Qi Xuan Yi Liu and Chao Yang Multiviewtransfer learning for software defect prediction IEEE Access 78901ndash8916 2019

[40] Annibale Panichella Rocco Oliveto and Andrea De Lucia Cross-project defectprediction models Lrsquounion fait la force In Software Maintenance Reengineeringand Reverse Engineering (CSMR-WCRE) 2014 Software Evolution Week-IEEEConference on pages 164ndash173 IEEE 2014

[41] Burak Turhan Tim Menzies Ayse B Bener and Justin Di Stefano On the relativevalue of cross-company and within-company data for defect prediction volume 14pages 540ndash578 Springer 2009

[42] Fayola Peters Tim Menzies and Andrian Marcus Better cross company defectprediction In Proceedings of the 10th Working Conference on Mining SoftwareRepositories pages 409ndash418 IEEE Press 2013

[43] Xin Xia LO David Sinno Jialin Pan Nachiappan Nagappan and Xinyu WangHydra Massively compositional model for cross-project defect prediction IEEETransactions on software Engineering 42(10)977 2016

[44] Ying Ma Guangchun Luo Xue Zeng and Aiguo Chen Transfer learning forcross-company software defect prediction Information and Software Technology54(3)248ndash256 2012

[45] Jaechang Nam Sinno Jialin Pan and Sunghun Kim Transfer defect learningIn Software Engineering (ICSE) 2013 35th International Conference on pages382ndash391 IEEE 2013

[46] Sinno Jialin Pan Ivor W Tsang James T Kwok and Qiang Yang Domain adap-tation via transfer component analysis IEEE Transactions on Neural Networks22(2)199ndash210 2011

[47] Blaine A Price Ronald M Baecker and Ian S Small A principled taxonomy ofsoftware visualization Journal of Visual Languages amp Computing 4(3)211ndash2661993

[48] Thomas Ball and Stephen G Eick Software visualization in the large Computer29(4)33ndash43 1996

[49] Sarita Bassil and Rudolf K Keller Software visualization tools Survey andanalysis In Proceedings 9th International Workshop on Program ComprehensionIWPC 2001 pages 7ndash17 IEEE 2001

[50] Stephan Diehl Software visualization visualizing the structure behaviour andevolution of software Springer Science amp Business Media 2007

[51] Michele Lanza The evolution matrix Recovering software evolution using soft-ware visualization techniques In Proceedings of the 4th international workshopon principles of software evolution pages 37ndash42 ACM 2001

[52] Denis Gracanin Krešimir Matkovic and Mohamed Eltoweissy Software vi-sualization Innovations in Systems and Software Engineering 1(2)221ndash2302005

[53] Guillaume Langelier Houari Sahraoui and Pierre Poulin Visualization-basedanalysis of quality for large-scale software systems In Proceedings of the 20thIEEEACM international Conference on Automated software engineering pages214ndash223 ACM 2005

[54] Abien Fred Agarap and Francis John Hill Pepito Towards building an intelligentanti-malware system a deep learning approach using support vector machine(svm) for malware classification arXiv preprint arXiv180100318 2017

[55] Karl Weiss Taghi M Khoshgoftaar and DingDing Wang A survey of transferlearning Journal of Big Data 3(1)9 2016

[56] Rahul Krishna and Tim Menzies Bellwethers A baseline method for transferlearning IEEE Transactions on Software Engineering 2018

[57] Chuanqi Tan Fuchun Sun Tao Kong Wenchang Zhang Chao Yang and ChunfangLiu A survey on deep transfer learning In ICANN 2018

[58] Wenyuan Dai Qiang Yang Gui Rong Xue and Yong Yu Boosting for transferlearning In International Conference on Machine Learning pages 193ndash2002007

[59] Chang Wan Rong Pan and Jiefei Li Bi-weighting domain adaptation for cross-language text classification In IJCAI Proceedings-International Joint Conferenceon Artificial Intelligence volume 22 page 1535 2011

[60] Yonghui Xu Sinno Jialin Pan Hui Xiong Qingyao Wu Ronghua Luo HuaqingMin and Hengjie Song A unified framework for metric transfer learning IEEETrans Knowl Data Eng 29(6)1158ndash1171 2017

[61] Chetak Kandaswamy Luiacutes M Silva Luiacutes A Alexandre and Jorge M SantosDeep transfer learning ensemble for classification In Ignacio Rojas GonzaloJoya and Andreu Catala editors Advances in Computational Intelligence pages335ndash348 Cham 2015 Springer International Publishing

[62] Maxime Oquab Leon Bottou Ivan Laptev and Josef Sivic Learning and trans-ferring mid-level image representations using convolutional neural networks InProceedings of the IEEE conference on computer vision and pattern recognitionpages 1717ndash1724 2014

[63] Yann LeCun Leacuteon Bottou Yoshua Bengio and Patrick Haffner Gradient-basedlearning applied to document recognition Proceedings of the IEEE 86(11)2278ndash2324 1998

[64] Hana Ajakan Pascal Germain Hugo Larochelle Franccedilois Laviolette and MarioMarchand Domain-adversarial neural networks arXiv preprint arXiv141244462014

[65] Yaroslav Ganin and Victor Lempitsky Unsupervised domain adaptation by back-propagation arXiv preprint arXiv14097495 2014

[66] Jing Zhang Wanqing Li and Philip Ogunbona Joint geometrical and statisticalalignment for visual domain adaptation arXiv preprint arXiv170505498 2017

[67] Eric Tzeng Judy Hoffman Ning Zhang Kate Saenko and Trevor DarrellDeep domain confusion Maximizing for domain invariance arXiv preprintarXiv14123474 2014

[68] Mingsheng Long Yue Cao Zhangjie Cao Jianmin Wang and Michael I Jor-dan Transferable representation learning with deep adaptation networks IEEEtransactions on pattern analysis and machine intelligence 2018

[69] Arthur Gretton Dino Sejdinovic Heiko Strathmann Sivaraman BalakrishnanMassimiliano Pontil Kenji Fukumizu and Bharath K Sriperumbudur Optimalkernel choice for large-scale two-sample tests In Advances in neural informationprocessing systems pages 1205ndash1213 2012

[70] Mingsheng Long Han Zhu Jianmin Wang and Michael I Jordan Deep transferlearning with joint adaptation networks arXiv preprint arXiv160506636 2016

[71] Martin Arjovsky Soumith Chintala and Leacuteon Bottou Wasserstein gan arXivpreprint arXiv170107875 2017

[72] Kesav Kancherla and Srinivas Mukkamala Image visualization based malwaredetection In 2013 IEEE Symposium on Computational Intelligence in CyberSecurity (CICS) pages 40ndash44 IEEE 2013

[73] Xin Zhou Jianmin Pang and Guanghui Liang Image classification for malwaredetection using extremely randomized trees In 2017 11th IEEE InternationalConference on Anti-counterfeiting Security and Identification (ASID) pages54ndash59 IEEE 2017

[74] John Murphy An overview of convolutional neural network architectures for deeplearning 2016

[75] Song Wang Taiyue Liu Jaechang Nam and Lin Tan Deep semantic feature learn-ing for software defect prediction IEEE Transactions on Software Engineering2018

[76] Laurens van der Maaten and Geoffrey Hinton Visualizing data using t-sneJournal of machine learning research 9(Nov)2579ndash2605 2008

12

• Abstract
• 1 Introduction
• 2 Related Work
• 3 Approach
• • 31 Source Code Visualization
  • 32 Modeling with Deep Transfer Learning
  • 33 Model Sensitivity to Parameter Choices
  • 34 Training and Prediction
  • • 4 Experimental Setup
    • • 41 Dataset Description
      • 42 Baseline Comparison Methods
      • 43 Performance measures
      • • 5 Results
        • • 51 RQ1 How does DTL-DP compare to feature- based machine learning methods and AST- based deep learning methods in WPDP
          • 52 RQ2 How does DTL-DP compare to feature- based Machine Learning and AST-based deep learning methods in CPDP
          • 53 RQ3 How much does each of the three mechanisms ie data augmentation transfer learning and self-attention mechanism contribute to DTL-DPs performance
          • • 6 Threats to Validity
            • 7 Conclusions and Future Work
            • References