arX

iv:1

812.

1052

8v2

[cs

.CR

] 1

Apr

202

01

Adversarial Attack and Defense on Graph Data:A Survey

Lichao Sun, Yingtong Dou, Carl Yang, Ji Wang, Philip S. Yu, Fellow, IEEE, and Bo Li,

✦

Abstract—Deep neural networks (DNNs) have been widely applied to

various applications including image classification, text generation, au-

dio recognition, and graph data analysis. However, recent studies have

shown that DNNs are vulnerable to adversarial attacks. Though there

are several works studying adversarial attack and defense strategies

on domains such as images and natural language processing, it is still

difficult to directly transfer the learned knowledge to graph structure

data due to its representation challenges. Given the importance of

graph analysis, an increasing number of works start to analyze the

robustness of machine learning models on graph data. Nevertheless,

current studies considering adversarial behaviors on graph data usually

focus on specific types of attacks with certain assumptions. In addition,

each work proposes its own mathematical formulation which makes

the comparison among different methods difficult. Therefore, in this

paper, we aim to survey existing adversarial learning strategies on graph

data and first provide a unified formulation for adversarial learning on

graph data which covers most adversarial learning studies on graph.

Moreover, we also compare different attacks and defenses on graph

data and discuss their corresponding contributions and limitations. In

this work, we systemically organize the considered works based on the

features of each topic. This survey not only serves as a reference for the

research community, but also brings a clear image researchers outside

this research domain. Besides, we also create an online resource and

keep updating the relevant papers during the last two years. More

details of the comparisons of various studies based on this survey

are open-sourced at https://github.com/YingtongDou/graph-adversarial-

learning-literature.

Index Terms—adversarial attack, adversarial defense, adversarial

learning, graph data, graph neural networks

1 INTRODUCTION

Recent years have witnessed significant success brought bydeep neural networks (DNNs) in various domains. Suchhighly expressive models outperform other models in fieldsincluding image recognition [43], natural language process-ing [30], graph data applications [42], [53], [86], [87], [102],as well as advanced applications such as healthcare analysis

• Lichao Sun, Yingtong Dou, and Philip S. Yu are with the Uni-versity of Illinois at Chicago, Chicago, IL, 60607 USA. E-mail:{lsun29,ydou5,psyu}@uic.edu

• Ji Wang is with the College of Systems Engineering, National Universityof Defense Technology, Changsha, Hunan, 410073 P. R. China. E-mail:wangji@nudt.edu.cn.

• Carl Yang and Bo Li are with the University of IllinoisUrbana-Champaign at Champaign, IL 61820 USA. E-mail:{jiyang3,lbo}@illinois.edu

[69], brain circuits analysis [61], and functionality of muta-tions in DNA [106].

Given the outstanding performance, deep learning hasbeen applied in some safety and security critical tasks suchas self driving [7], malware detection [78], identification[79] and anomaly detection [34]. However, the lack of inter-pretability and robustness of DNNs makes them vulnerableto adversarial attacks. Szegedy et al. [82] pointed out the sus-ceptibility of DNNs in image classification. The performanceof a well-trained DNN can be significantly degraded by ad-versarial examples, which are carefully crafted inputs withsmall magnitude of perturbations added. Goodfellow etal. [41] analyzed this phenomenon and proposed a gradient-based method (FGSM) to generate adversarial image sam-ples. Different adversarial attack strategies are then pro-posed to demonstrate the vulnerabilities of DNNs in varioussettings [6], [15], [105]. For instance, black-box adversarialattacks are later explored based on transferability [62], [70]and query feedback from the DNN models [5], [13]. Severaldefense and detection methods have also followed up tomitigate such adversarial behaviors [65], [74], while variousadaptive attacks continued to be proposed showing thatdetection/defense is hard in general [3], [14].

Although there are an increasing number of studies onadversarial attack and defense, such adversarial analysismainly focuses on image, natural language, and speechdomains. Related study on graph data is at its infancydespite the importance of graph data in many real-worldapplications. For example, in the credit prediction applica-tion, an adversary can easily disguise himself by adding afriendship connection with others, which may cause severeconsequences [26]. Compared with previous adversarialanalysis in non-graph data, the study on graph data raisesseveral unique challenges: 1) Unlike images consisting ofcontinuous features, the graph structure and nodes’ featuresare discrete. It is difficult to design efficient algorithms thatare able to generate adversarial examples in the discretespaces. 2) Adversarial perturbations are designed to be im-perceptible to humans in the image domain, so one can forcea certain distance function, such as Lp norm distance to besmall between adversarial and benign instances. Howeverin graph data, how to define “imperceptible" or “subtleperturbation" requires further analysis, measurement andstudy.

Given the importance of graph-related applicationsand the successful applications of graph neural networks

2

TABLE 1Attack and Defense works are categorized by GNN or Non-GNN oriented.

Category Type Paper

Attack Model GNN [8], [12], [16], [20], [22], [26], [64], [80], [81], [89], [91], [107], [121], [122][19], [33], [38], [59], [83], [103], [114]

Non-GNN [2], [17], [18], [23], [25], [31], [39], [44], [93], [94], [109], [110], [115], [119]

Defense Model GNN [21], [29], [36], [50], [51], [67], [77], [85], [90], [103], [108], [117], [120], [123][9], [33], [38], [46]–[48], [68], [92]

Non-GNN [2], [27], [39], [44], [49], [73], [115], [118]

(GNNs), both academia and industry are interested in therobustness of GNNs. In recent several months, some re-searchers begin to focus on adversarial attack for a set ofGNN models. In this paper, we contribute the first study onsummarizing different adversarial learning on graph dataand providing taxonomies for them according to variouscriteria. All relevant attack and defense are listed in Tables 2and 3. Despite the more than one hundred papers publishedin the last three years, there are several challenges remainingunsolved until now, which we contribute to summarize andintroduce in this work as below.

Comprehensive Understanding. To our best knowledgethis is the first work to provide a comprehensive under-standing of adversarial attack and defense on graph data.Our first version of this work was released on Arxiv in 2018,which summarized all published and pre-print works at thattime. It has stimulated (and been cited by) various following-up novel research in this line [16], [51], [56], [111], [120] aswell as other attempts of survey [24], [52]. In this work,we substantially improve the coverage over a wide rangeof relevant works, especially those released in the recenttwo years, and we summarize novel elaborated taxonomiesaccording to various criteria. This survey not only includeworks on adversarial attack and defense approaches target-ing on specific GNNs, but also discuss many non-gradientand non-model-based approaches in this area.

Online Updating Resource. We created an open-sourcerepository that includes all relevant works and maintainedthe update on it in the last two years.1 This repositoryprovides all paper links and released code links whichmakes it easier for relevant researchers to use and hasserved as a fundamental benchmark library in this area.Currently, many of the works are only pre-print versions,and please feel free to contact us when the pre-print papersare accepted in any conference or journal. We will updatethe information and keep updating the new works in thisdomain in the future. We hope this open-sourced repositorycan keep shedding light on future research about adversarialanalysis on graph structured data.

Unified Problem Definition. As we know, there havebeen various attack and defense strategies on graph data. Itis a challenge and takes much time to understand the bigpicture of all works in this domain. In order to facilitateeasy understanding over existing research on this line, wepioneer to provide a unified formulation and definitionfor adversarial attacks on graph data in this work. Unlikeattacks, defenses on graph data often go beyond adversarial

1. https://github.com/YingtongDou/graph-adversarial-learning-literature

learning, for which we provide additional categories basedon their unique strategies.

Taxonomy of adversarial analysis on graph data. Thereare already over a hundred papers in this domain. Com-pared with image data and text data, graph data are morecomplex due to various data representations and tasks. Listall papers can help but is not good enough for readers toquickly understand the similarity and difference betweenthe works. To this end, we summaries all existing worksbased on GNN and Non-GNN methods, aiming to helpreaders find the most relevant papers easily. We present ourtaxonomy with more details in Table 1.

Dataset and Metrics. Due to the various goals and dataof attack and defense works, it is hard to compare the resultsbetween each pair of different methods. Currently, no workcan directly answer question “What attack or defense is thebest work in this domain (for this task)?”. The only wayto alleviate this is by building a benchmark like in otherareas [28], [88]. In order to address this problem, we notonly develop taxonomies for all papers based on differentcriteria, but also summarize the corresponding datasets andmetrics that are frequently used. We hope our work can pavethe way for the community to build good benchmark inthis area for future empirical analysis and in-depth technicalunderstanding.

The rest of this survey is organized as follows. Sec-tion 2 provides the necessary background information ofgraph data and common applications. Section 3 providesthe unified problem formulation and discusses the existingadversarial attack studies on graph data. Section 4 providesdiscusses and summaries the existing defense studies ongraph data. Section 5 provides the evaluation and attackmetrics used in different papers. Section 6 provides thedetails of each dataset, and summarizes existing worksbased on the datasets they use. The last section concludesthis survey.

2 GRAPH

In this section, we first give the notations of graph data, andthen introduce the preliminaries about graph types, learningsettings, and application tasks.

2.1 Notations

We use G = {Gi}Ni=1 to represent a set of graphs, where N is

the number of graphs. Each graph Gi is generally denotedby a set of nodes Vi = {v

(i)j } and edges Ei = {e

(i)j }, where

e(i)j = (v

(i)j,1, v

(i)j,2) ∈ Vi × Vi is the edge between the nodes

v(i)j,1 and v

(i)j,2. Optionally, the nodes and the edges can have

other features such as node features, edge weights, and edge

3

direction. According to these features, graph data can beclassified into different types.

2.2 Types of Graph Data

From a temporal perspective, graphs can be grouped into staticgraphs and dynamic graphs.

Dynamic Graph and Static Graph. A graph is dynamic,denoted as G(t), if any of its nodes, edges, node features,or edges features change over time. In the contrast, a staticgraph, denoted as G, consists of a fixed set of nodes andedges that do not change over time.

A typical example of static graph is the molecular struc-ture of drugs [32]. Once a drug is developed, its molecularstructure does not change over time. Social network [72] isa good example of dynamic graphs. As people often add orremove friendship links in their social network, the graphextracted changes over time. In most existing attack works,the researchers study the attacks on dynamic graphs.

Directed Graph and Undirected Graph. A directedgraph, denoted as G(Dr), has direction information asso-ciated with each edge, where any directed edge e

(i)1 =

(v(i)1 , v

(i)2 ) 6= (v

(i)2 , v

(i)1 ) = e

(i)2 . In the contrast, any two

nodes of an undirected graph share the same edge.Twitter, an online social network, is one typical example,

where the directed edge represents the following informa-tion from one user to another. If there is a directed edgeconnecting from user A to user B, it means A follows B,and necessarily the other way around. The graphs extractedfrom such online social networks are directed graphs. Face-book is a classic undirected graph that A is B’s friend meansB is A’s friend too.

Attributed Graph on edge. An attributed graph on edge,denoted as G(Ae), has some features associated with eachedge, which is denoted by x(e

(i)j ) ∈ R

Dedge .The weighted graph where each edge has a weight,

x(e(i)j ) ∈ R, is a special case of attributed graph on edges.

A traffic flow graph [60] is a typical example of weightedgraph where roads are modeled as edges and road condi-tions are represented by weights of edges.

Attributed Graph on node. An attributed graph on node,denoted as G(An), has some features associated with eachnode, which is denoted by x(v

(i)j ) ∈ R

Dnode .The e-commerce network [35] with different users can

be regarded as an example of attributed graph on nodewhere each user is modeled as nodes with some featureslike demographics and clicking history.

Note that, directed graph and heterogeneous informa-tion networks are special cases of attributed graph, which arewidely used to model different applications.

2.3 Learning Settings on Graph Data

This section introduces the different machine learning set-tings used on graph data. Before introducing the learningsettings, we first provide the notations for mathematicalformulation. We associate the target component ci withina graph Gci ∈ G with a corresponding ground truth labelyi ∈ Y = {1, 2, . . . , Y }. Here i ∈ [1,K], K representsthe number of the total target components, and Y is thenumber of classes being predicted. The dataset D(ind) =

{(ci, Gci , yi)}

Ki=1 is represented by the target graph compo-

nent, graph containing ci, and the corresponding groundtruth label of ci. For instance, in a node classification task, cirepresents the node to be classified, and yi denotes its labelwithin Gci . Based on the features of training and testingprocesses, the learning settings can be classified as inductiveand transductive learning.

Inductive Learning. It is the most realistic machine learn-ing setting where the model is trained by labeled examples,and then predicts the labels of examples never seen duringtraining. Under the supervised inductive learning setting,the classifier f (ind) ∈ F (ind) : G → Y is optimized:

L(ind) =1

K

K∑

i=1

L(f(ind)θ (ci, G

ci), yi),

where L(·, ·) is the cross entropy by default, and ci can benode, link or subgraph of its associated graph Gci . Notethat, two or more different instances, c1, c2, . . . , cn can beassociated with the same graph G ∈ G.

Transductive Learning. Different from inductive learn-ing, the testing graphs have been seen during trainingin the transductive learning. In this case, the classifierf (tra) ∈ F (tra) : G → Y is optimized:

L(tra) =1

K

K∑

i=1

L(f(tra)θ (ci, G

ci), yi).

Transductive learning predicts the label of seen instances, butinductive learning predicts the label of unseen instances.

Unified Formulation of Learning on Graph Data. Wegive an uniform formula to represent both supervised in-ductive and transductive learning as below:

L(·) =1

K

K∑

i=1

L(f(·)θ (ci, G

ci), yi), (1)

where f(·)θ = f

(ind)θ is inductive learning and f (·) = f

(tra)θ

is transductive learning.In the unsupervised learning setting, we can use the

unlabelled dataset D(ind) = {(ci, Gj)}Ki=1 and replace the

supervised loss L and function f(ci, Gi) of Eq. 1.In this survey, we mainly focus on the supervised learn-

ing setting, while also introducing a few new works in theunsupervised learning setting.

2.4 Application

In this section, we will introduce the main tasks on graphdata, including node-level, link-level and graph-level appli-cations. Moreover, we also introduce how to use the unifiedformulation of Eq. 1 to define each application task below.

Node-Level Application. The node-level application isthe most popular one in both academia and industry. Aclassic example is labeling the nodes in the Web and socialnetwork graphs, which may contain millions of nodes, suchas Facebook and Twitter.

Most existing papers [8], [9], [26], [89], [103], [107], [120]–[123] focus on node-level applications. All of these papersstudy node classification in the transductive learning settingwhose objective function can be formulated by modifyingEq. 1 where f

(·)θ = f

(tra)θ , ci here is the representation of

4

node target and its associated graph Gci is set as a singlegraph G.

Few existing works have discussed the node-level ap-plications in the inductive leaning setting. However, theseapplications frequently appear in real life. For example,the first party only has several large and public networkinformation, such as Facebook and Twitter. The secondparty has private unlabeled graph data in which the nodescan be predicted by using the information from the firstparty. In this case, the node-level classification task is nolonger transductive learning. It can be easily formulated bymodifying Eq. 1 with f

(·)θ = f

(ind)θ and ci here is still the

representation of node target.Link-Level Application. Link prediction on dynamic

graphs is one of the most common link-level applications.The models try to predict missing links in current networks,as well as new or dissoluted links in future networks. Thecorresponding attacks have been discussed in [20].

Compared with node classification tasks, link predica-tion tasks still use node features, but target at the missing orunlabelled links in the graph. Therefore, we can formulatethe link predication task by slightly modifying Eq. 1 with cibeing the representation of link target, and yi ∈ {0, 1}.

Graph-Level Application. Graph-level tasks are fre-quently seen in the chemistry or medical areas, such as themodeling of drug molecule graphs and brain graphs. In [26],the whole graph is used as the sample instance. Differentfrom this setting, some other graph-level applications usethe subgraphs of a larger graph for particular tasks.

Compared with the existing works on node classificationand link predication, graph classification uses the graph-structure representation as the features to classify the un-labelled graph instances. Therefore, we can formulate thegraph classification task by slightly modifying Eq. 1 bysetting ci as the representation of graph target.

3 ADVERSARIAL ATTACKS ON GRAPH DATA

In this section, we give a general definition and taxonomiesof adversarial attacks on graph data, and then introduce theimperceptibility metrics, attack types, attack tasks and levelsof attack knowledge.

3.1 An Unified Definition and Formulation

Definition 3.1. (General Adversarial Attack on Graph Data)Given a dataset D = (ci, Gi, yi), after slightly modifyingGi (denoted as Gci ), the adversarial samples Gci and Gi

should be similar under the imperceptibility metrics, butthe performance of graph task becomes much worse thanbefore.

Existing papers [8], [16], [20], [25], [26], [33], [44], [59],[80], [89], [103], [107], [121], [122] considering adversarialbehaviors on graph data usually focus on specific types ofattacks with certain assumptions. In addition, each workproposes its own mathematical formulation which makesthe comparison among different methods difficult. In orderto help researchers understand the relations between differ-ent problems, we propose a unified problem formulationthat can cover all current existing works.

Definition 3.2. (Adversarial Attack on Graph Data: A Uni-fied Formulation) f can be any learning task function ongraph data, e.g., link prediction, node-level embedding,node-level classification, graph-level embedding and graph-level classification. Φ(Gi) denotes the space of perturbationon the original graph Gi, and dataset D = {(ci, G

ci , yi)}Ni=1

denote the attacked instances. The attack can be depicted as,

maxGci∈Φ(Gi)

∑

i

L(f(·)θ∗ (ci, G

ci), yi))

s.t. θ∗ = argminθ

∑

j

L(f(·)θ (cj , G

′j), yj)).

(2)

When G′j equals to Gcj , Eq. 4 represents the poisoning

attack, whereas when G′j is the original G without mod-

ification, Eq. 4 denotes the evasion attack. f(·)θ = f

(ind)θ

represents inductive learning and f(·)θ = f

(tra)θ transductive

learning.

Note that, with Gci ∈ Φ(G), (ci, Gci) can represent nodemanipulation, edge manipulation, or both. For any Gci ∈Φ(Gi), Gci is required to be similar or close to the originalgraph Gj , and such similarity measurement can be definedby the general distance function below:

Q(Gci , Gi) < ǫ

s.t. Gci ∈ Φ(Gi)(3)

where Q(·, ·) represents the distance function, and ǫ is a pa-rameter denoting the distance/cost budget for each sample.

Discussion: Graph Distance Function. Graph distancefunctions can be defined in many ways, a lot of which havebeen discussed on graph privacy-preserving related work[54]. Such distance functions include the number of commonneighbours of given nodes, cosine similarity, Jaccard similar-ity and so on. However, few of them are discussed in depthregarding adversarial behaviors (adversarial cost in gametheory). In general, an attacker aims to make “minimal"perturbations on the existing graph and therefore such dis-tance measurement is important to measure the quality ofattacks. How to design and choose proper distance functionto quantify the attack ability under different attack scenariosis also critical towards developing defensive approachesregarding specific threat model. We will discuss potentialperturbation evaluation metrics in detail in Sec 3.2.

In addition to the unique properties of each graphdistance function, it would also be interesting to analyzethe “equivalence" among them. For instance, an attackeraims to attack one node by adding/removing one edgein the graph can encounter similar “adversarial cost" asadding/removing edges. It is not hard to see that by usinga graph distance function or similarity measures, only a fewtargets would be the optimal choices for the attacker (withdifferent distance), so this can also help to optimize the ad-versarial targets. In summary, due to the complexity and di-versity of graph representations and adversarial behaviors,perturbation evaluation or graph similarity measurementwill depend on various factors such as different learningtasks, adversarial strategies, and adversarial cost types.

5

TABLE 2Summary of adversarial attack works on graph data (time ascending).

Ref. Year Venue Task Model Strategy Approach Baseline Metric Dataset

[25] 2017 CCS Graph ClusteringSVD, Node2vec,

Communitydetection algs

Noise injection,Small community

attackAdd/Delete edges - ASR, FPR

NXDOMAIN,Reverse Engineered

DGA Domains

[93] 2018NatureHuman

Behavior

Hide nodesand communities

in a graph

Community detectionalgs Heuristic Rewire edges -

Concealmentmeasures, Graph

statistics

WTC 9/11, Scale-freeFacebook, Twitter,Google+, Random

[121] 2018 KDD Node classification GCN, CLN,DeepWalk Incremental attack Add/Delete edges,

Modify node featuresRandom,

FGSM

Accuracy,Classifcation

margin

Cora-ML,Citeseer,PolBlogs

[26] 2018 ICML Graph classification,Node classification

GNN familymodels

Reinforcementlearning Add/Delete edges Rnd. sampling,

Genetic algs. Accuracy Citeseer,Finance,Pubmed, Cora

[94] 2018 ScientificReports Link prediction Similarity

measures Heuristic Add/Delete edges - AUC, APWTC 9/11, Random,

Scale-Free,Facebook

[22] 2018 arXiv Node classification,Community detection

DeepWalk, GCN,Node2vec, LINE

Check GCNgradients Rewire edges

Random,DICE,

NettackASR, AML Cora, Citeseer,

PolBlogs

[20] 2018 arXiv Link prediction GAE, DeepWalk,Node2vec, Katz

Graphauto-encoder Delete edges Random,

DICE, GA ASR, AML NS, Yeast,Facebook

[91] 2018 arXiv Node classification GCN Greedy, GAN Add fake nodeswith fake features Random, Nettack Accuracy,

F1, ASRCora,

Citeseer

[80] 2018 arXiv Link prediction GAE, DeepWalk,Node2vec, LINE

Projectgradient descent Add/Delete edges

Degree sum,Shortest path,

Random, PageRank

AP,Similarity

score

Cora,Citeseer,Facebook

[8] 2019 ICML Node classification,Link prediction

Node2vec, GCNLP, DeepWalk

Check gradient,Approximate

spectrumAdd/Delete edges Random, Degree,

Eigenvalue

F1 score,Misclassification

rate

Cora, Citeseer,PolBlogs

[122] 2019 ICLR Node classificationGCN, CLNDeepWalk Meta learning Add/Delete edges

DICE, Nettack,First-order attack

Accuracy,Misclassification

rate

Cora, Pubmed,Citeseer,PolBlogs

[119] 2019 AAMAS Link predictionLocal&Global

Similaritymeasures

Submodular Hide edges Random, Greedy Similarityscore

Random,Facebook

[17] 2019 TCSS Community detection Communitydetection algs Genetic algs Rewire edges Random, Degree,

Community detectionNMI,

Modularity

Karate, Dolphin,Football,Polbooks

[89] 2019 CCS Node classification

LinBP, LBP, JW,DeepWalk, LINE,

GCN, RW,Node2vec

Optimization Add/Delete edges Random, Nettack FNR, FPRGoogle+,

Epinions, Twitter,Facebook, Enron

[115] 2019 IJCAIKnowledge graph

fact plausibilityprediction

RESCAL,TransE, TransR

Check targetentity embeddings Add/Delete fact Random MRR,

Hit Rate@K FB15k, WN18

[2] 2019 arXiv Vertex nomination VN·GMM·ASE Random Add/Delete edges - Achievingrank

Bing entitytransition graph

[12] 2019 arXiv Node classification GCN Adversarialgeneration Modify node features Nettack ASR Cora,

Citeseer

[110] 2019 arXiv Node classification,Community detection

HOPE, LPA,EM, DeepWalk

Compare euclideandistance ofDeepWalk

embeddings

Add/Delete edges Random, DICE,Degree, Greedy

NMI,F1 score

Dolphin, Cora,Karate, Game,

Citeseer

[103] 2019 IJCAI Node classification GCN Check gradients Add/Delete edges,Modify node features

Random,Nettack

FGSM, JSMA

Accuracy,Classification

margin

Cora, Citeseer,PolBlogs

[107] 2019 IJCAI Node Classification GCN First-orderoptimization Add/Delete edges DICE, Greedy,

Meta-selfMisclassification

rateCora,

Citeseer

[16] 2019 AAAI Node classification GCN, LINE,SGC, DeepWalk

Approximatespectrum,

Devise new lossAdd/Delete edges Random, Degree,

RL-S2V, Accuracy Cora, Citeseer,Pubmed

[64] 2019 arXiv Node classification GCNReinforcement

learning Rewire edges RL-S2V, Random ASRReddit-Multi,IMDB-Multi

[18] 2019 arXiv Community detection Communitydetection algs

Optimization,Adversarialgeneration

Add/Delete edges

Modularity, Degree,Random, Entropy,

Betweenness,Add/Delete links

NMI, ARI Synthetic, Email,Football, PolBlogs

[44] 2019 CIKM Malware detection,Node classification Metapath2vec Greedy Inject new nodes Anonymous attack %TPR,

TP-FP curve Private dataset

[23] 2019 arXiv Dynamic linkprediction

Deep dynamicnetwork

embedding algsCheck gradients Rewire edges Random, Gradient,

Common neighbor ASR, AML LKML, FB-WOSN,RADOSLAW

[38] 2020 AAAIWorkshop Node classification

GraphIsomorphism

Network

Noisy graphgeneration

Generate new graphwith structural noisy - F1 score Ring of

houses

[31] 2020 AAMAS Node Similarity Similaritymeasures Graph theory Remove edges

Greedy, Random,High jaccard

similarity

# Removededges

Power,web-edu,hamsterster,

euroroad

[81] 2020 WWW Node classification GCN Reinforcementlearning Inject new nodes

Random, FGA,Preferential

attack

Accuracy,Graph

statistics

Cora-ML,Pubmed,Citeseer

[59] 2020 WWW Hide nodein community

Surrogatecommunity

detection model

Graphauto-encoder Add/Delete edges

DICE, Random,Modularitybased attack

Personalizedmetric

DBLP,Finance

[33] 2020 WSDM Node classification GCN, t-PINE Low-rankapproximation Add/Delete edges Nettack

Correctclassification

rate

Cora-ML, Citeseer,PolBlogs

[109] 2020 arXiv Graph property Physical criteria Heuristic Rewire edges - ConcealmentMeasure

Scale-freenetworks

[114] 2020 arXiv Node classification GCN, DeepWalk,Node2vec, GAT Check gradients Add/Delete edges Random, FGA,

Victim-class attack ASR, AML Cora, Citeseer,PolBlogs

[83] 2020 BigData Node classification GCN Check gradients Modify node features Nettack ASR Cora-ML,Citeseer

[19] 2020 arXiv Node classification,Community detection

DeepWalk, GCN,Node2vec

Check gradientmomentum Rewire edges

Nettack,RL-S2V, FGA,GraArgmax

ASR Cora, Citeseer,PolBlogs

[39] 2020 arXiv Manipulatingopinion Graph model Adversarial

optimizationChange initialopinion vector - - -

6

3.2 Adversarial Perturbation

To generate adversarial samples on graph data, we can mod-ify the nodes or edges from the original graph. However,the modified graph G need to be “similar” with the originalgraph G based on certain perturbation evaluation metricsand remain “imperceptible". The following metrics helpunderstand how to define “imperceptible perturbation".

Edge-level Perturbation. In most current papers, theattacker is capable of adding/removing/rewiring edges inthe whole original graph within a given budget. In this case,the number of modified edges is usually used to evaluatethe magnitude of perturbation. In addition to other pertur-bations, edge perturbation is hardly found by the defender,especially in dynamic graphs.

Node-level Perturbation. The attacker is also capableof adding/removing nodes, or manipulating the featuresof target nodes. The evaluation metric in this case can becalculated based on the number of nodes modified or thedistance between the benign and adversarial feature vectors.

Structure Preserving Perturbation. Similar to edge-levelperturbation, an attacker can modify edges in the graphwithin a given budget in terms of graph structure. Com-pared to general edge-level perturbation, this considersmore structural preservation, such as total degree, node dis-tribution, etc. For instance, in [121], the attacker is requiredto preserve the key structural features of a graph such as thedegree distribution. Therefore, the perturbation here can bemeasured by the graph structure drift.

Attribute Preserving Perturbation. In the attributedgraphs, each node or edge has its own features. In addi-tion to manipulating the graph structure, the attacker canchoose to modify the features of nodes or edges to generateadversarial samples on graph data. Various measurementsbased on graph-attribute properties can be analyzed tocharacterize the perturbation magnitude. For instance, in[121], the authors argue adding a feature is imperceptibleif a probabilistic random walker on the co-occurrence graphcan reach it with high probability by starting from existingfeatures.

Note that, most GNN methods learn the feature represen-tation of each node, which means it could be easily attackedby structure-only, feature-only perturbations or both.

Principles of imperceptible perturbation evaluation.Given various graph distance discussion, there is no cleardiscussion in existing research about how to set the ad-versarial cost for attacks on graph data so far. Therefore,we summarize some principles of defining the perturbationevaluation metrics as below for future research.

• For static graph, both the number of modified edgesand the distance between the benign and adversarialfeature vectors should be small.

• For a dynamic graph, we can set the distance oradversarial cost based on the intrinsic changing in-formation over time. For example, by using statis-tic analysis, we can get the upper bound of theinformation manipulated in practice, and use thisinformation to set an imperceptible bound.

• For various learning tasks on graph data, e.g., nodeor graph classification, we need to use a suitablegraph distance function to calculate the similarity

between the benign and its adversarial sample. Forexample, we can use the number of common neigh-bours to evaluate the similarity of two nodes, but thisis not applicable for two individual graphs.

In summary, compared to image and text data, an attackerfirst can modify more features on the information network,and also can explore more angles to define “imperceptible”based on the format of graph data and the application task.

3.3 Attack Stage

The adversarial attacks can happen at two stages: evasion at-tack (model testing) and poisoning attacks (model training).It depends on the attacker’s capacity to insert adversarialperturbations:

Poisoning Attack. Poisoning attack tries to affect theperformance of the model by adding adversarial samplesinto the training dataset. Most existing works are poisoningattacks, and their node classification tasks are performedin the transductive learning setting. In this case, once theattacker changes the data, the model is retrained. Mathe-matically, by setting G′

j = Gcj in Eq. 4, we have a generalformula for adversarial attack on graph data under poison-ing attacks.

Evasion Attack. Evasion attack means that the param-eters of the trained model are assumed to be fixed. Theattacker tries to generate the adversarial samples of thetrained model. Evasion attack only changes the testing data,which does not require to retrain the model. Mathematically,by setting G′

j to original Gj in Eq. 4, we have a generalformula for adversarial attack on graph data under evasionattacks.

3.4 Attack Objective

Though all adversarial attacks are modifying the data, anattacker needs to choose their attack targets or objectives:model or data. In this case, we can summarize them asmodel objective and data objective.

Model Objective. Model objective is attacking a par-ticular model by using any approaches. It could be eitherevasion attack or poisoning attack. Most current adversarialattack is related to model objective attack. The target couldbe either GNN or other learning models. An attacker wantsto make the model become non-functional working in mul-tiple scenarios. Model objective attack can be categorized bywhether using the gradient information of the model or not.

• Gradient-based Attack. In most studies, we can seethat the gradient-based attack is always the simplestand most effective approach. Most gradient-basedattack, no matter white-box or black-box, tries to getor estimate the gradient information to find the mostimportant features to the model. Based on the aboveknowledge, an attacker can choose to modify thelimited information based on the feature importanceto the model and make the model inaccurate whenusing the modified information [8], [26], [121].

• Non-gradient-based Attack. In addition to gradientinformation, an attack could destroy the model with-out any gradient information. As we know, besidesthe gradients, many reinforcement learning based

7

attack methods can attack the model based on long-term rewards [26], [64], [80]. Some works can alsoconstruct the adversarial samples with generativemodels [12], [18], [38]. All the above approaches canattack the model without the gradient informationbut attack the model in practice.

Data Objective. Unlike model objective attacks, dataobjective attacks do not attack a specific model. Such attackshappen when the attacker only has access to the data, butdoes not have enough information about the model. Ingeneral there are two settings when data become the target.

• Model Poisoning. Unsupervised feature analysis ap-proaches can still get useful information from thedata without any knowledge of the training ap-proach. Even with a small perturbation on the data, itcan make general training approaches cease to work.Besides, backdoor attack is another relevant hot topicwhere an attacker only injects the adversarial signalsin the dataset, but does not destroy the model perfor-mance on regular samples.

• Statistic Information. In addition to using the datato train a model, in many studies, researchers usestatistical results or simulation results from the graphdata. In this case, an attacker can break the modelbased on the capturing of the valuable statistical in-formation on graph data. For example, by modifyinga few edges between different communities based onstructural information and analysis, one can makecommunities counting inaccurate under this attack.

3.5 Attack Knowledge

The attacker would receive different information to attackthe system. Based on this, we can characterize the dangerouslevels of existing attacks.

While-box Attack. In this case, an attacker can get allinformation and use it to attack the system, such as theprediction result, gradient information, etc. The attack maynot work if the attacker does not fully break the system first.

Grey-box Attack. An attacker gets limited informationto attack the system. Comparing to white-box attack, it ismore dangerous to the system, since the attacker only needpartial information.

Black-box Attack. Under this setting, an attacker canonly do black-box queries on some of the samples. Thus, theattacker generally can not do poisoning attack on the trainedmodel. However, if black-box attack can work, it would bethe most dangerous attack compared with the other two,because the attacker can attack the model with the mostlimited acknowledge.

Most existing papers only studies white-box attack onthe graph, and there are lots of opportunities to study otherattacks with different levels of knowledge.

3.6 Attack Goal

Generally, an attacker wants to destroy the performance ofthe whole system, but sometimes they prefer to attack a fewimportant target instances in the system. Based on the goalof an attack, we have:

Availability Attack. The adversarial goal of availabilityattack is to reduce the total performance of the system.For example, by giving a modification budget, we wantthe performance of the system decreasing the most as theoptimal attack strategy.

Integrity Attack. The adversarial goal of integrity attackis to reduce the performance of target instances. For exam-ple, in recommendation systems, we want the model to notsuccessfully predict the hidden relation between two targetusers. However, the total performance of the system is thesame or similar to the original system.

Availability attack is easier to detect than integrity attackunder the positioning attack setting. Therefore, meaningfulavailability attack studies are in general under the evasionattack setting.

3.7 Attack Task

Corresponding to various tasks on graph data, we showhow to attack each task and explain the general idea bymodifying the unified formulation.

Node-relevant Task. As mentioned before, most attackpapers focus on node-level tasks, including node classifi-cation [16], [26], [89], [103], [107], [121], [122] and nodeembedding [8], [115]. The main difference is that node em-bedding uses the low dimensional representations of eachnode for an adversarial attack. Mathematically, by setting cias representation of node target in Eq. 4, we have a generalformula for adversarial attack on node-relevant tasks.

Link-relevant Task. Other several existing works [8],[20], [80] study node embedding and use it for link pre-diction. Compared with node classification, link predictionrequires to use different input data, where ci represents linktarget, i.e., the information of a pair of nodes. By settingci as representation of link target and yi ∈ [0, 1] in Eq. 4,we have a general formula for adversarial attack on link-relevant tasks.

Graph-relevant Task. Only one existing paper studiesgraph classification [26]. Compared with node classification,graph classification needs the graph representation insteadof the node representation. By setting ci as representationof graph target in Eq. 4, we have a general formula foradversarial attack on graph-relevant tasks.

3.8 Summary: Attack on Graph

In this subsection, we talk about the contributions andlimitations of existing works. Then we discuss the potentialresearch opportunities in this area.

Contributions. First, we list all released papers and theircharacteristics in Table 2, and then categorize them intoselected main topics in Table 1. Then, we summarize theunique contributions of existing adversarial attacks. Notethat, because 14 of 35 papers we discuss are pre-print ver-sion, we especially list the venue in Table 2. We also firstlyuse Strategy and Approach to differ individual attack method.Strategy refers to the high-level design philosophy of anattack, while Approach represents the concrete approach theattacker takes to perturb the graph data.

Graph Neural Networks. Most adversarial attacks arerelevant to graph neural networks. [26] used reinforcementlearning approach to discover adversarial attack, which is

8

the only approach that supports black-box attack comparedto other works. [121] studied adversarial graph sampleswith traditional machine learning and deep learning. Mean-while, they are the first and only group to discuss the adver-sarial attack on attributed graph. [20], [80] mainly attackedthe link predication task with a deep graph convolutionalembedding model. [8] attacked multiple models by approxi-mating the spectrum and use the gradient information. [89]attacked node classification though optimization approachand systematically discussed adversarial attacks on graphdata. Previous works focused on edge or node modifica-tion, whereas [103] also modified the node features andproposed a hybrid attack on the graph convolutional neuralnetworks (GCN) [53]. In addition to gradient check, [33],[107] attacked GCN by using the first-gradient optimiza-tion and low-rank approximation which makes an attackmore efficient. [16] attacked general learning approaches bydevising new loss and approximating the spectrum. [44]used graph attack knowledge into the malware detectionproblem, which showed various graph-based applicationsto be vulnerable to adversarial attacks. Without gradientcheck and optimization design, [81] used reinforcementlearning to attack GCN. However, it contains an obviousissue that it needs to break the graph structure by injectingnew nodes. [59] tried to hide nodes in the community byattacking the graph auto-encoder model. Instead of using agradient check or other optimization approaches, this worksurrogated the community detection model to achieve theattacking goal.

Others. Though many attack works are relevant to GNN,many recent papers start to focus on other types of adver-sarial attacks on graph data. [25] is one of the first worksto attack the graph data, and it also first proposed theattack approach in the unsupervised learning setting. [93]first attacked community detection though edge rewritingbased on a heuristic approach. [94] attacked link predictionbased on a heuristic approach which is based on the sim-ilarity measures. [119] used a greedy approach to attacklink prediction based local and global similarity measure.In addition to traditional graph applications, [115] first at-tacked knowledge graph and destroyed the basic relationalgraph prediction model. [17] attacked community detectionbased on genetic algorithms. Unlike previous approaches,it chose to use rewiring instead of adding/removing edgeswhile attacking the data. [38] useed a generation approachto create a new isomorphism network to attack node clas-sification. In addition to all previous works, [31] started tostudy attacks through theoretical analysis, and we believemore theoretical works will be seen in this domain. Theycan help us understand the attacks better on graph data.

Limitations. The limitations of most current works aresummarized below. Most existing works do not give veryclear strategies about the setting of the budget and distancewith reasonable explanations in real applications. Differentfrom other adversarial attacks, most graph modificationscan hardly be noticed by humans in real life. To solve thisproblem, we give a more detailed discussion on perturba-tion and evaluation metrics in Section 5. Meanwhile, aboutgraph imperceptible evaluation metrics, most papers [8],[20], [26] use one metric for attack, but these adversarial sam-ples could be detected by other existing imperceptible eval-

uation metrics. In this work, we list all existing evaluationmetrics, and recommend future adversarial samples to beimperceptible with more listed evaluation metrics. Anothermain issue is due to the different problem formulations. Tothis end, we give the unified problem formulation for allexisting works discussed in this survey.

Future Directions. Adversarial attack on graph data isa new and hot area, and many research opportunities aresummarized below: 1) Most graphs are associated withattributes or more complex contents on nodes or edges inthe real life. Currently, very few existing works well studiedadversarial attack on attributed graphs, e.g., heterogeneousinformation networks and the Web. 2) Some advanced ideascan be applied for generating the adversarial samples, e.g.,homomorphism graph. 3) Various learning settings are notsufficiently studied yet, such as graph-level attacks andinductive learning on node-level attacks. 4) Most existingattacks do not consider various imperceptibility metricsinto their attack model. Concise and comprehensive imper-ceptibility metrics are necessary in different tasks. A goodand explainable evaluation metric may easily discover moreexisting adversarial samples created by current methods. 5)Last but not least, the distance or similarity measures ofhigh quality adversarial samples are not well studied in thisarea.

4 ADVERSARIAL DEFENSE ON GRAPH DATA

With graph data, recent intensive studies on adversarialattacks have also triggered the research on adversarial de-fenses. Here we survey existing works in this line andclassify them into the two popular categories of AdversarialTraining and Attack Detection. After them, we use an addi-tional Other Methods subsection to summarize the remainingmethods that do not fit into the two generic categories.

4.1 Adversarial Training

While adversarial training has been widely used by at-tackers to perform effective adversarial intrusion, the samesword can be used by defenders to improve the robustnessof their models against adversarial attacks [41]. In the graphsetting, we formulate the objective of adversarial defense byslightly modifying our unified formulation of adversarialattacks, i.e., Eq. 4, as follows

minθ

maxGci∈Φ(Gi)

∑

i

L(fθ(ci, Gci), yi)). (4)

where meanings of the notations remain the same as definedin Section 3. The idea is to alternatively optimize twocompeting modules during training, where the attacker triesto maximize task-oriented loss by generating adversarialperturbations G on the graph, and the defender tries tominimize the same loss by learning the more robust graphmodel parameters θ under the generated adversarial pertur-bations. In this way, the learned graph model is expected tobe resistant to future adversarial attacks.

Structure Perturbations. The earliest and most primitiveway of perturbing the graph is to randomly drop edges [26].The joint training of such cheap adversarial perturbationsis shown to slightly improve the robustness of standardGNN models towards both graph and node classification

9

TABLE 3Summary of adversarial defense works on graph data (time ascending).

Ref. Year Venue Task Model Corresp. Attack Strategy Baseline Metric Dataset

[108] 2018 OpenReview

Added edgesdetection GNN, GCN N/A Link prediction,

Graph generation LP AUC Cora,Citeseer

[36] 2019 TKDE Node classification GCN N/A Adversarialtraining

DeepWalk, GCN,Planetoid, LP,

GraphVAT,GraphSCAN

AccuracyCora,

NELL,Citeseer

[117] 2019 ICLRWorkshop Node classification GCN, GAT Nettack,

Random

First&Second orderKL divergence

proximity-

Classificationmargin,

Accuracy, AUC

Cora,Citeseer,PolBlogs

[27] 2019 WWW Node classification DeepWalk N/A Adversarialtraining

DeepWalk, LINENode2vec, GraRep,Graph Factorization

Accuracy,AUC

Cora, WikiCiteseer,

CA-GrQc,CA-HepTh

[77] 2019 PRCV Node classification GCN N/AVirtual

adversarialtraining

GCN AccuracyCora,

Citeseer,Pubmed

[21] 2019 arXiv Node embedding GNN Nettack,FGA

Smoothinggradients

Adversarialtraining ADR, ACD

Cora,Citeseer,PolBlogs

[120] 2019 KDD Node classification GCNNettack,RL-S2V,Random

Gaussiandistribution layer,

Variance-basedattention

GCN, GAT AccuracyCora,

Citeseer,Pubmed

[73] 2019 NAACL Link predictionKnowledge

graphembeddings

N/A Adversarialmodification N/A Hits@K, MRR

Nations,WN18,

Kinship,YAGO3-10

[2] 2019 arXiv Vertex nominationGraph

embeddingmodels

N/ANetwork

regularizationwith graph trimming

- Achieving rankBing entitytransition

graph

[90] 2019 arXiv Node classification GCNNettack,RL-S2V,Random

GAN, Graphencoder refining,

Contrastive learning

GCN, GraphSAGE,Refined

GCN&GraphSAGE

Classificationmargin

Cora,Citeseer,PolBlogs

[51] 2019 arXiv Node classification GCN N/A Graph powering GCN, ICA,MeniReg

Accuracy,Robustness merit,

Attack deterioration

Cora,Pubmed,Citeseer

[103] 2019 IJCAI Node classification GCNRandom,Nettack

FGSM, JSMADrop edges GCN

Accuracy,Classification

margin

Cora,Citeseer,PolBlogs

[107] 2019 IJCAI Node classification GCN DICE,Meta-self

Check gradients,Adversarial

trainingGCN

Accuracy,Misclassification

rate

Cora,Citeseer

[50] 2019 ICMLWorkshop Node classification GCN Nettack Adversarial

trainingGCN, SGCN,

FastGCN, SGCASR,

Accuracy

Citeseer, Cora,Pubmed, Cora-ML,

DBLP, PolBlogs

[29] 2019 ICMLWorkshop Node classification GCN N/A Adversarial

training

GCN, GAT, LP,DeepWalk, Planetoid,

Monet, GPNNAccuracy Citeseer, Cora,

Pubmed, NELL

[123] 2019 KDD Node classification GCN, GNN N/A Convexoptimization GNN

Accuracy,Average

worst-case margin

Cora-ML,Pubmed,Citeseer

[67] 2019 KDDWorkshop Node classification GCN,

Node2vec N/A Changetraining set

GCN,Node2vec

Adversary budget,Classification

marginCora, Citeseer

[118] 2019 ICDM Link predictionSimilaritymeasures N/A

BayesianStackelberg gameand optimization

ProtectPotential

Neighbors

Damageprevention ratio

PA,TV Show,PLD, Gov

[9] 2019 NIPS Node classification GCN N/ARobust training,

MDP toget bound

GNNAccuracy,Worst-case

margin

Cora-ML,Pubmed,Citeseer

[46] 2019 arXiv Anomaly detection,Node classification

Anomalydetection algs Nettack

Random sampling,Consensusestimation

GAE, Radar,Degree Cut-ratio AUC

Cora, Pubmed,Citeseer,PolBlogs

[47] 2019 arXiv Node classification GCN NettackAuxiliary graph,

Lossregularization

GCN Accuracy

Cora,Pubmed,Citeseer,PolBlogs

[44] 2019 CIKM Malware detection,Node classification

Heterogeneousgraph,

Metapath2vecN/A Attention

mechanismOther malwaredetection algs

Accuracy, F1,Precision, Recall

Privatedataset

[92] 2019 arXiv Node classification GCN,GraphSAGE N/A Adversarial

training

Drop edges, Discreteadversarial

training

Accuracy,Correct

classification rate

Cora,Citeseer,Reddit

[38] 2020AAAI

Workshop Node classificationGraph

IsomorphismNetwork

N/AAugmented

training - F1 scoreRing ofhouse

[85] 2020 WSDM Node classification GNN MetattackMeta learning,Transfer fromclean graph

GCN, GAT,RGCN, VPN Accuracy

Pubmed,Yelp,

Reddit

[33] 2020 WSDM Node classification GCN, t-PINE Nettack,LowBlow

Low-rankapproximation - Correct

classification rateCora-ML, Citeseer,

PolBlogs

[49] 2020 WWW Community detection Communitydetection algs N/A

Robustcertification

with optimization- Certified

accuracyEmail, DBLP,

Amazon

[68] 2020 arXiv Node classification GCN NettackSelect training set

via node degree andneighborhood label

GCN-SVD, GCN-Jaccard Adversarybudget

Citeseer,Pubmed, Cora,

PolBlogs

[39] 2020 arXiv Manipulatingopinion Graph model N/A Minimax game,

Convex optimization - - -

[48] 2020 arXiv Nodeclassification GCN N/A Tensor-GCN GCN Accuracy

Cora,Pubmed,Citeseer,PolBlogs

10

tasks. One step further, [107] proposed a topology attackgeneration method based on projected gradient descent tooptimize edge perturbation. The topology attack is shownto improve the robustness of the adversarially trained GNNmodels against different gradient-based attacks and greedyattacks without sacrificing node classification accuracy onthe original graph. In the meantime, [27] proposed to learnthe perturbations in an unsupervised fashion by maximiz-ing the influence of random noises in the embedding space,which improved the generalization performance of Deep-Walk [72] on node classification. Towards similarity-basedlink prediction, [118] formalized a Bayesian Stackelberggame to optimize the most robust links to preserve withan adversary deleting the remaining links.

Attribute Perturbations. Besides links, [29], [36], [77]also perturb node features to enable virtual adversarialtraining that enforces the smoothness between originalnodes and adversarial nodes. In particular, [36] designeda dynamic regularizer forcing GNN models to learn to pre-vent the propagation of perturbations on graphs, whereas[77] smooths GCN in its most sensitive directions to im-prove generalization. [29] further conducts virtual adversar-ial training in batch to perceive the connectivity patternsbetween nodes in each sampled subsets. [90] leveragedadversarial contrastive learning to tackle the vulnerabilitiesof GNN models to adversarial attacks due to training datascarcity and applied conditional GAN to utilize graph-levelauxiliary information. Instead of approximating the discretegraph space, [92] proposed to directly perturb the adjacencymatrix and feature matrix by ignoring the discreteness,whereas [50] proposed to focus on the first hidden layer ofGNN models to continuously perturb the adjacency matrixand feature matrix. These frameworks are all shown toimprove GNN models on the node classification task.

Attack-oriented Perturbation Based on existing networkadversarial attack methods of FGA [22] and Nettack [121],[21] designed the adversarial training pipelines with addi-tional smooth defense strategies. The pipeline is shown toimprove GNN models against different adversarial attackson node classification and community detection tasks.

4.2 Attack Detection

Instead of generating adversarial attacks during training,another effective way of defense is to detect and remove (orreduce the effect of) attacks, under the assumption that datahave already been polluted. Due to the complexity of graphdata, the connection structures and auxiliary features can beleveraged based on various ad hoc yet intuitive principlesto essentially differentiate clean data from poison ones andcombat certain types of attacks.

Graph Preprocessing. [108] proposed different ap-proaches to detect potential malicious edges based on graphgeneration models, link prediction and outlier detection.Instead of edges, [46] proposed to filter out sets contami-nated by anomalous nodes based on graph-aware criteriacomputed on randomly drawn subsets of nodes; [117] pro-posed to detect nodes subject to topological perturbations(particularly by Nettack [121]) based on empirical analysison the discrepancy between the proximity distributionsof nodes and their neighbors. These models only rely on

network topology for attack detection. On attributed graphs,based on the observations that attackers prefer adding edgesover removing edges and the edges are often added betweendissimilar nodes, [103] proposed to compute the JaccardSimilarity to remove suspicious edges between suspiciousnodes. All of these models can be used for graph prepro-cessing before training normal graph models like GNNs.

Model Training. Rather than direct detection of sus-picious nodes or edges before training, several works de-signed specific attention mechanisms to dynamically un-cover and down-weigh suspicious data during training.[120] assumed high prediction uncertainty for adversarialnodes and computed the attention weights based on theembedding variance in a Gaussian-based GCN. [85] sug-gested to train an attack-aware GCN based on ground-truthpoisoned links generated by Nettack [121] and transfer theability to assign small attention weights to poisoned linksbased on meta-learning.

Robustness Certification. On the contrary of detectingattacks, [9], [123] designed robustness certificates to measurethe safety of individual nodes under adversarial perturba-tion. In particular, [9] considers structural perturbation and[123] considers attribute perturbation. Training GNN mod-els jointly with these certificates can lead to a rigorous safetyguarantee of more nodes. From a different perspective, [49]derives the robustness certificate of community detectionmethods under structural perturbation.

Complex Graphs Beyond traditional homogeneousgraphs, [73] studied the sensitivity of knowledge graph linkprediction models towards adversarial facts (links) and theidentification of facts. [44] studied the detection of poisoningnodes in heterogeneous graphs to enhance the robustness ofAndroid malware detection systems.

4.3 Other Methods

Now we summarize the remaining graph adversarial de-fense algorithms that are neither based on adversarial train-ing nor aiming at attack detection. We further group theminto three subcategories based on their modifications to thegraph data and graph models.

Data Modifications. We have presented several attackdetection algorithms that can be used for modifying graphdata, i.e., graph preprocessing [46], [108], [117]. There existmethods that modify graph data without directly detect-ing attacks. Based on the insight that Nettack [121] onlyaffects the high-rank singular components of the graph, [33]proposed to reduce the effect of attacks by computing thelow-rank approximation of the graphs before training GNNmodels. [38] proposed an augmented training procedure bygenerating more structurally noisy graphs to train GNNmodels for improved robustness, and showed it to be effec-tive for structural role identification of nodes. [68] analyzedthe topological characteristics of graphs and proposed twotraining data selection techniques to raise the difficulty ofeffective adversarial perturbations towards node classifica-tion. These methods are all based on graph topology alone,and they only modify the graph data instead of the graphmodels.

Model Modifications. On the contrary, there exist meth-ods that only modify the graph models, such as model-structure redesign or loss-function redesign. The simplest

11

way is to redesign the loss function. From several existingworks, the results show some loss functions perform betterperformance against the adversarial examples. For example,[51] designed an alternative operator based on graph pow-ering to replace the classical Laplacian in GNN models withimproved spectral robustness. They demonstrated the com-bination of this operator with vanilla GCN to be effectivein node classification and defense against evasion attacks.Hybrid Modifications. One step further, some methodsmodify both the graph data and graph models. [47] de-signed an edge-dithering approach to restoring unperturbednode neighborhoods with multiple randomly edge-flippedgraphs and proposed an adaptive GCN model that learns tocombine the multiple graphs. The proposed framework isshown to improve the performance and robustness of GCNtowards node classification (in particular, protein functionprediction) on attributed graphs. [67] proposed a heuristicmethod to iteratively select training data based on thedegrees and connection patterns of nodes. They furtherproposed to combine node attributes and structural featuresand use SVM for node classification instead of any GNNmodels.

4.4 Summary: Defense on Graph

From the perspective of defenders, the defense approachescan be designed with or without knowing the specificattacks. Thus, current defense works can be classified intotwo categories: 1) Attack-agnostic defenses are designed to en-hance the robustness of graph models against any possibleattacks instead of a fixed one. 2) Attack-oriented defenses aredesigned according to the characteristics of specific attacks.The attack-agnostic defenses usually have a wider assump-tion space of attacks comparing to attack-oriented attack.Last, we discuss some future opportunities on adversarialdefense in this area.

Attack-agnostic Defense. As we summarized in Sec-tion 4.1, adversarial training is a typical instance of attack-agnostic defense approach [26], [29], [36], [77], [107]. Itusually generates simple perturbations on graphs or modelsto train a defense model. In the test phase, some modelstrained in this way could exhibit good robustness againstthose perturbations. Some methods [107] trained in this wayeven attain good defense performance against other specificattacks like Meta-self proposed in [122]. Note that the de-fense methods are designed and trained without knowingother new attacks.

Besides adversarial training, other works secure thegraph model with heuristic assumptions on the attack strate-gies and outcomes. [85] assumes that there are unpollutedgraphs to aid the detection of attacks. [44], [48], [51], [120]propose new GNN architectures to enhance their robustness.[67], [68] directly curates an optimal training set to mitigatethe vulnerability of trained models.

Attack-oriented Defense. Attack-oriented defenses aredesigned based on the strategy and approach of specificattacks. Namely, the defender has full knowledge of anattack method and the defense method could detect thecorresponding attack or curb its performance. Among cur-rent defense works, [33] first argues the weakness of Net-tack [121] and leverages SVD to defend against Nettack. [50]

analyzes the strategies and approaches of Nettack [121] andRL-S2V [26] and propose an adversarial training method.[103] inspects two gradient-based attack (i.e., FGSM [41] andJSMA [71]) and applies edge-dropping technique duringmodel training to alleviate the influence of such attacks. Sim-ilar to attack-agnostic defenses, some attack-oriented meth-ods exhibit good generability which means it can defendagainst other unknown attacks. For instance, the defensemethod proposed in [103] could defend the Nettack as well.Along with the Corresp. Attack column of Table 3, wecould see that Nettack and RL-S2V have become benchmarkattack methods for defense design and evaluation. Someworks employ the framework of minimax game [39] oroptimization [9], [49], [123] to certify the robustness boundsof graph models under given attacks and defenses. Suchkind of defense works are attack-oriented since they haveassumed specific attacks.

Limitations and future directions. We have been focus-ing on the contributions of different existing works on graphadversarial defense. Now we summarize some commonlimitations we observe in this line of research and hint onfuture directions: 1) Most defense models focus on node-level tasks, especially node classification, while it may beintriguing to shed more light on link- and graph-leveltasks like link prediction and graph classification. There isalso large potential in more real-life tasks like graph-basedsearch, recommendation, advertisement and etc. 2) Whilenetwork data are often associated with complex contentsnowadays (e.g., timestamps, images, texts), existing defensemodels have hardly considered the effect of attacks anddefenses under the settings of dynamic or other content-rich complex networked systems. 3) Most defense modelsare relevant to GNNs or GCN in particular, but there aremany other graph models and analysis methods, possiblymore widely used and less studied (e.g., random walk basedmodels, stochastic block models, and many computationalgraph properties). How are they sensitive and prone tograph adversarial attacks? Can the improvements in GNNmodels transfer and generalize to these traditional methodsand measures? 4) Most existing works do not study theefficiency and scalability of defense models. As we know,real-world networks can be massive and often frequentlyevolve, so how to efficiently learn the models and adaptto changes is very important for defenders. 5) While thereare standard evaluation protocols and optimization goals fordown-stream tasks like node classification and link predic-tion, defense methods are optimized towards heterogeneousgoals like accuracy, robustness, generalizability and so on,and they tend to define their own experimental settingsand metrics, rendering fair and comprehensive evaluationschallenging.

5 METRIC

In this section, we summarize the metrics for evaluatingattack and defense performance on graph data. We firstbriefly introduce the general evaluation metrics along withsome notes on their specific usage in adversarial perfor-mance evaluation. We then give a detailed introductionof particular evaluation metrics designed for attacks anddefenses.

12

5.1 General Metric

5.1.1 Accuracy-based Metric

According to Table 2 and Table 3, many existing workstackle the node classification problem which is usually abinary or multi-class classification problem. The accuracy-based metrics like Accuracy, Recall, Precision, and F1 scoreare all used by existing works to reflect the classificationaccuracy from different angles. Readers can refer to [96] fordetailed explanations of those metrics. Note that the FalseNegative Rate (FNR) and False Positive Rate (FPR) usedby [25], [89] are two metrics derived from the confusionmatrix. FNR is the percentage of false negatives among allactual positive instances, which describes the proportion ofpositive instances missed by the classifier. Similarly, FPRreflects the proportion of negative instances misclassifiedby the classifier. Adjusted Rand Index (ARI) [100] is anaccuracy-based metric without label information. [18] usesit to measure the similarity between two clusters in a graph.

Besides the above metrics, Area-under-the-ROC-curve(AUC) [101] and Average Precision (AP) [95] are widelyused, such as by [46], [80], [94], [108], [120]. AUC is sen-sitive to the probability rank of positive instances, whichis larger when positive instances are ranked higher thannegative instances according to the predicted probabilityof a classifier. AP is a metric balancing the Precision andRecall where AP is higher when Precision is higher asRecall threshold increase from 0 to 1. Those two metricscould better reflect the classification performance as singlescores since they provide an all-around evaluation over thepredicted probabilities of all instances.

5.1.2 Ranking-based Metric

Mean Reciprocal Rank (MRR) [97] and Hits@K are tworanking metrics used by [73], [115] to evaluate the perfor-mance of link prediction on knowledge graphs. Given a listof items retrieved regarding a query and ranked by theirprobabilities, the reciprocal rank of a query response is themultiplicative inverse of the rank of the first correct item: 1for first place, 1⁄2 for second place, 1⁄3 for third place and soon. Hits@K is the number of correct answers among the topK items in the ranking list.

5.1.3 Graph-based Metric

The graph-based metrics indicate the specific properties ofa graph. Normalized Mutual Information (NMI) [99] andModularity [98] are two metrics used by [17], [18], [110]to evaluate the performance of community detection (i.e.,clustering) on graphs. NMI is originated from informationtheory that measures the mutual dependence between twovariables. In a community detection scenario, NMI is usedto measure the amount of shared information (i.e., similar-ity) between two communities. Modularity is designed tomeasure the strength of the division of a graph into clus-ters. Graphs with high Modularity have dense connectionsbetween the nodes within clusters but sparse connectionsbetween nodes in different clusters.

[81] employs a couple of graph property statistics as met-rics to evaluate how much the attacker changed the graph(i.e., the imperceptibility of attacks). The metrics includeGini Coefficient, Characteristic Path Length, Distribution

Entropy, Power Law Exponent, and Triangle Count. Pleaserefer to [10] for more details about those metrics. Some moregraph statistics metrics include Degree Ranking, ClosenessRanking, Betweenness Ranking used by [93] and Cluster-ing Coefficient, Shortest Path-length, Diagonal Distanceused by [109].

5.2 Adversarial Metric

Besides the general metrics above, a number of metricswhich measure the attack and defense performance ongraph data have been proposed or used by existing works.We first present the detailed formulations and descriptionsof widely used metrics, and then briefly summarize someunique metrics used by particular papers. The referenceafter each metric name refers to the first paper that proposesor uses this metric and the references inside the parenthesesrefer to other attack and defense papers using this metric.

5.2.1 Common Metric

• Attack Success Rate (ASR) [25] ( [12], [19], [20],[22], [23], [50], [64], [83], [91], [114]). ASR is the mostfrequently used metric to measure the performanceof a give attack approach:

ASR =# Successful attacks

# All attacks.

• Classification Margin (CM) [121] ( [67], [90], [103],[117]). CM measures the performance of the integrityattack:

CM(t) = pt,ct −maxc 6=ct

pt,c,

where t is the target instance, ct is the ground-truthclass for t, pt,c is the probability of t being c. Theabove equation calculates the maximum differencebetween the probability of ground-truth class andthat of other classes. In other words, it shows theextent of an attack flipping the predicted class of atarget instance. [67] proposed another version of CM:

CM(t) = logpt,ct

maxc 6=ct pt,c.

When the instance is correctly classified, CM will bepositive; otherwise it will be negative.

• Correct/Mis Classification Rate [8] ( [33], [92],[107], [122]). Those two metrics evaluate the at-tack/defense performance based on the classificationresults among all instances.

MCR =# Misclassified instances

# All instances:

CCR = 1− MCR.

• Attacker Budget [67] ( [31], [68]). Attacker budget isa general metric to measure the minimum perturba-tions the attacker needs to fulfill its objective. Thelower value indicates a better attack performanceand a worse defense performance respectively. [31]takes number of removed edges as the attacker bud-get. [67], [68] take the smallest number of perturba-tions for the attacker to successfully cause the targetto be misclassified as the budget.

13

TABLE 4Summary of datasets (ordered by the frequency of usage within each graph type).

Type Task Dataset Source # Nodes # Edges # Features # Classes Paper

CitationNetwork

Node/Link Citeseer [75] 3,327 4,732 3,703 6[121], [26], [22], [91], [80], [8], [122], [12], [110], [103], [107], [16],[108], [36], [117], [77], [21], [120], [90] , [51], [123], [9], [46], [47],[92], [33], [114], [83], [19], [48], [27], [50], [29], [67], [68]

Node/Link Cora [75] 2,708 5,429 1,433 7[26], [22], [91], [80], [8], [122], [12], [110], [103], [107], [16], [108],[36], [117], [77], [21], [120], [90] , [51], [46], [47], [92], [114], [19],[48], [27], [50], [29], [67], [68]

Node Pubmed [75] 19,717 44,338 500 3[26], [122], [16], [81], [77], [120], [51], [123], [9], [46], [47], [85],[48], [50], [29], [68]

Node Cora-ML [66] 2,995 8,416 2,879 7 [121], [123], [9], [81], [33], [83], [50]Node/Community DBLP [84] - - - - [59], [50], [49]

SocialNetwork

Node/Link PolBlogs [1] 1,490 19,025 - 2 [121], [22], [8], [103], [18], [117], [21], [90] , [46], [47], [33], [114],[19], [48], [50], [68]

Node/Link Facebook [57] - - - - [94], [20], [80], [119], [89]Node/Community Google+ [57] 107,614 13,673,453 - - [93], [94], [89]Node Reddit [42] 1,490 19,090 300 2 [92], [85]Community WTC 9/11 [55] 36 64 - - [93], [94],Community Email [57] 1,005 25,571 - - [18], [49]Community Dolphin [63] 62 159 - - [17], [110]Community Karate [113] 34 78 - - [17], [110]Community Football [40] 115 613 - - [17], [18]

KnowledgeGraph

Fact/Link WN18 [11] - - - - [115], [73]Fact FB15k [11] - - - - [115]

Others Node Scale-free [4] - - - - [93], [94], [109]Node NELL [112] 65,755 266,144 5,414 210 [36], [29]

• Average Modified Links (AML) [22] ( [20], [22],[23], [114]). AML is a variance of Adversary budgetintroduced above. It describes the average number ofmodified links the attacker needed to meet the attackobjective:

AML =# Modified links

# All attacks.

• Concealment Measures [93] ( [59], [94], [109]). Theconcealment measures are used to evaluate the per-formance of hiding nodes or communities in agraph [59], [93], [94]. From another perspective, thestructural changes introduced by an attack can beused to quantify the concealment of the attack aswell [109].

• Similarity Score [80] ( [119]). Similarity score is ageneral metric to measure the similarity of giveninstance pairs. It can be used as the goal of integrityattack where the attacker’s goal is either to increaseor decrease the similarity score of a target instancepair. For a node instance in a graph, both of itslocal structure and node embedding can be used tocompute the similarity score.

5.2.2 Unique Metric

• Averaged Worst-case Margin (AWM) [9]. The worst-case margin is the minimum value of the classifica-tion margin defined above. The averaged worse-casemargin means the value is averaged across a worst-case margin of each batch of data.

• Robustness Merit (RM) [51]. RM is the differencebetween the post-attack accuracy of the proposemethod and the post-attack accuracy of the vanillaGCN model. A greater value indicates a better de-fense performance.

• Attack Deterioration (AD) [51]. AD is the ratio ofdecreased amount of accuracy after an attack to theaccuracy without attack.

• Average Defense Rate (ADR) [21]. ADR is a metricevaluating the defense performance according to the

ASR defined above. It compares the ASR after attackswith or without applying the defense approach.

• Average Confidence Different (ACD) [21]. ACD isa metric evaluating the defense performance basedon the average difference between the classificationmargin after and before the attack of a set of nodes.Such a set of nodes includes correctly classified nodesbefore the attack.

• Damage Prevention Ratio (DPR) [118]. Damageprevention measures the amount of damage that canbe prevented by the defense. Let L0 be the defender’saccumulated loss when there is no attack. Let LA

be the defender’s loss under some attack A whenthe defender cannot make any reliable queries. LD

denotes the loss when the defender make reliablequeries according to a certain defense strategy D.DPR can be defined as follows:

DPRDA =

LA − LD

LA − L0.

• Certified Accuracy [49]. It is proposed to evaluatethe certification method for robust community detec-tion models against adversarial attacks. The certifiedaccuracy CK(l) is the fraction of sets of victim nodesthat proposed method can provably detect as in thesame community when an attacker adds or removesat most l edges in the graph.

6 DATASET AND APPLICATION

Table 4 summarizes some common datasets used in ad-versarial attacks and defenses works on graph data. Thefirst four citation graphs have been widely used as nodeclassification benchmarks in previous work [53], [86], [87],[102]. [80] also studies the adversarial link prediction prob-lem on Cora and Citeseer. DBLP includes multiple citationdatasets with more metadata information. Thus it can beused to study the community detection task [49]. Amongthe social network datasets, PolBlogs is another dataset usedespecially in adversarial settings where blogs are nodes and

14

TABLE 5Summary of open-source implementations of algorithms.

Type Paper Algorithm Link

Graph Attack

[93] DICE https://github.com/DSE-MSU/DeepRobust[121] Nettack https://github.com/danielzuegner/nettack[26] RL-S2V, GraArgmax https://github.com/Hanjun-Dai/graph_adversarial_attack

[122] Meta-self, Greedy https://github.com/danielzuegner/gnn-meta-attack[8] ICML-19 https://github.com/abojchevski/node_embedding_attack

[107] PGD, Min-max https://github.com/KaidiXu/GCN_ADV_Train[16] GF-Attack https://github.com/SwiftieH/GFAttack

Graph Defense

[36] GraphAT https://github.com/fulifeng/GraphAT[27] AdvT4NE https://github.com/wonniu/AdvT4NE_WWW2019

[120] RGCN https://github.com/DSE-MSU/DeepRobust[103] GCN-Jaccard https://github.com/DSE-MSU/DeepRobust[107] Adversarial Training https://github.com/KaidiXu/GCN_ADV_Train[123] Robust-GCN https://github.com/danielzuegner/robust-gcn[85] PA-GNN https://github.com/tangxianfeng/PA-GNN[51] r-GCN, VPN https://www.dropbox.com/sh/p36pzx1ock2iamo/AABEr7FtM5nqwC4i9nICLIsta?dl=0[9] Graph-cert https://github.com/abojchevski/graph_cert[33] GCN-SVD https://github.com/DSE-MSU/DeepRobust

Other Baseline

[41] FGSM https://github.com/1Konny/FGSM[71] JSMA https://github.com/tensorflow/cleverhans[6] Gradient Attack (GA) https://github.com/bethgelab/foolbox/blob/master/foolbox/attacks/gradient.py[37] First-order https://github.com/cbfinn/maml

their citations are edges. Reddit and Facebook are two largergraph datasets comparing to citation datasets. Since theremultiple sizes of Facebook datasets, we omit its statistics.WTC 9/11, Email, Dolphin, Karate, and Football five bench-mark datasets for community detection. [73], [115] inves-tigate the adversarial attacks and defenses on knowledgegraphs using two knowledge graph benchmarks WN18and FB15k. Scale-free network is a typical type of graphgenerated by graph generation models. Some works alsoemploy other graph generation models to generate randomgraphs to facilitate their experiments [93], [94], [119].

Future Directions. Besides the datasets listed in Table 4,it is worth noting some other datasets which have lessattention but could shed light on future researches. [44]is the first and only paper investigating the vulnerabilityof Heterogeneous Information Network (HIN) which is agraph model with heterogeneous node and edge types [76].Though HIN has been applied to many security applicationslike malicious user detection [116], spam detection [58], andfinancial fraud detection [45], its robustness against adver-sarial attacks remain largely unexplored. A recent work [39]firstly gives a formulation of adversarial attacks on opinionpropagation on graphs with a spectral form that could beused to study the opinion dynamics of social network. [73],[115] are first two works studying the adversarial attacksand defenses on Knowledge Graph (KG) models. As theresearch of KG becoming popular in recent years, its securityissue needs to be noticed as well. The security of dynamicgraph models [23] is another avenue of research as well.

Besides the above works and datasets. The securityissues of many other graph types and their related appli-cations have not been explored yet. To name a few, thebiology graph, causal graph, and bipartite graph have beentriggering many research works but no work has stud-ied potential attacks and their countermeasures on thosegraphs. From the perspective of applications, as the GNNshaving been successfully applied to recommender system,computer vision and natural language processing [104],

adversarial attacks and defenses on graph data under thosespecific applications is another promising research directionwith de facto impacts.

7 CONCLUSION

In this work, we cover the most released papers aboutadversarial attack and defense on graph data as we know.We firstly provide an unified problem formulation for ad-versarial learning on graph data, and give definitions andtaxonomies to category the papers. Next, we summarymost existing imperceptible perturbations evaluation met-rics, datasets and discuss several principles about imper-ceptibility metric. Then, we analyze the contributions andlimitations of the existing works. Finally, we point out thepotential research opportunities and directions in futurestudies.

APPENDIX

In order to help building a benchmarks like other areas [28],[88], we not only develop taxonomy for all the papers basedon different criteria, but also summary the correspondingdatasets and metrics that are frequently used. Moreover,here we also provide the open-source implementation ofeach paper. We hope our work can help the communityto build a good benchmark in this area and gain in-depthunderstanding.

REFERENCES

[1] Lada A Adamic and Natalie Glance. The political blogosphereand the 2004 us election: divided they blog. In Proceedings of the3rd international workshop on Link discovery, pages 36–43, 2005.

[2] Joshua Agterberg, Youngser Park, Jonathan Larson, ChristopherWhite, Carey E Priebe, and Vince Lyzinski. Vertex nomina-tion, consistent estimation, and adversarial modification. arXivpreprint arXiv:1905.01776, 2019.

[3] Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscatedgradients give a false sense of security: Circumventing defensesto adversarial examples. arXiv preprint arXiv:1802.00420, 2018.

15

[4] Albert-László Barabási and Réka Albert. Emergence of scaling inrandom networks. science, 286(5439):509–512, 1999.

[5] Arjun Nitin Bhagoji, Warren He, Bo Li, and Dawn Song. Ex-ploring the space of black-box attacks on deep neural networks.arXiv:1712.09491v1, 2017.

[6] Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson,Nedim Šrndic, Pavel Laskov, Giorgio Giacinto, and Fabio Roli.Evasion attacks against machine learning at test time. In JointEuropean conference on machine learning and knowledge discovery indatabases, pages 387–402. Springer, 2013.

[7] Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bern-hard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel,Mathew Monfort, Urs Muller, Jiakai Zhang, et al. End to endlearning for self-driving cars. arXiv preprint arXiv:1604.07316,2016.

[8] Aleksandar Bojchevski and Stephan Günnemann. Adversarialattacks on node embeddings via graph poisoning. In InternationalConference on Machine Learning, pages 695–704, 2019.

[9] Aleksandar Bojchevski and Stephan Günnemann. Certifiable ro-bustness to graph perturbations. In Advances in Neural InformationProcessing Systems, pages 8317–8328, 2019.

[10] Aleksandar Bojchevski, Oleksandr Shchur, Daniel Zügner, andStephan Günnemann. Netgan: Generating graphs via randomwalks. arXiv preprint arXiv:1803.00816, 2018.

[11] Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, JasonWeston, and Oksana Yakhnenko. Translating embeddings formodeling multi-relational data. In Advances in neural informationprocessing systems, pages 2787–2795, 2013.

[12] Avishek Joey Bose, Andre Cianflone, and William Hamiltion.Generalizable adversarial attacks using generative models. arXivpreprint arXiv:1905.10864, 2019.

[13] Wieland Brendel, Jonas Rauber, and Matthias Bethge. Decision-based adversarial attacks: Reliable attacks against black-box ma-chine learning models. arXiv preprint arXiv:1712.04248, 2017.

[14] Nicholas Carlini and David Wagner. Adversarial examples arenot easily detected: Bypassing ten detection methods. In Proceed-ings of the 10th ACM Workshop on Artificial Intelligence and Security,pages 3–14. ACM, 2017.

[15] Nicholas Carlini and David Wagner. Towards evaluating therobustness of neural networks. In 2017 IEEE Symposium onSecurity and Privacy (SP), pages 39–57. IEEE, 2017.

[16] Heng Chang, Yu Rong, Tingyang Xu, Wenbing Huang, HongleiZhang, Peng Cui, Wenwu Zhu, and Junzhou Huang. A restrictedblack-box adversarial framework towards attacking graph em-bedding models. AAAI, 2020.

[17] Jinyin Chen, Lihong Chen, Yixian Chen, Minghao Zhao, Shan-qing Yu, Qi Xuan, and Xiaoniu Yang. Ga-based q-attack oncommunity detection. IEEE Transactions on Computational SocialSystems, 6(3):491–503, 2019.

[18] Jinyin Chen, Yixian Chen, Lihong Chen, Minghao Zhao, andQi Xuan. Multiscale evolutionary perturbation attack on com-munity detection. arXiv preprint arXiv:1910.09741, 2019.

[19] Jinyin Chen, Yixian Chen, Haibin Zheng, Shijing Shen, ShanqingYu, Dan Zhang, and Qi Xuan. Mga: Momentum gradient attackon network. arXiv preprint arXiv:2002.11320, 2020.

[20] Jinyin Chen, Ziqiang Shi, Yangyang Wu, Xuanheng Xu, andHaibin Zheng. Link prediction adversarial attack. arXiv preprintarXiv:1810.01110, 2018.

[21] Jinyin Chen, Yangyang Wu, Xiang Lin, and Qi Xuan. Canadversarial network attack be defended? arXiv preprintarXiv:1903.05994, 2019.

[22] Jinyin Chen, Yangyang Wu, Xuanheng Xu, Yixian Chen, HaibinZheng, and Qi Xuan. Fast gradient attack on network embedding.arXiv preprint arXiv:1809.02797, 2018.

[23] Jinyin Chen, Jian Zhang, Zhi Chen, Min Du, Feifei Li, andQi Xuan. Time-aware gradient attack on dynamic network linkprediction. arXiv preprint arXiv:1911.10561, 2019.

[24] Liang Chen, Jintang Li, Jiaying Peng, Tao Xie, Zengxu Cao, KunXu, Xiangnan He, and Zibin Zheng. A survey of adversariallearning on graphs. arXiv preprint arXiv:2003.05730, 2020.

[25] Yizheng Chen, Yacin Nadji, Athanasios Kountouras, FabianMonrose, Roberto Perdisci, Manos Antonakakis, and NikolaosVasiloglou. Practical attacks against graph-based clustering. InProceedings of the 2017 ACM SIGSAC Conference on Computer andCommunications Security, pages 1125–1142, 2017.

[26] Hanjun Dai, Hui Li, Tian Tian, Xin Huang, Lin Wang, Jun Zhu,and Le Song. Adversarial attack on graph structured data. arXivpreprint arXiv:1806.02371, 2018.

[27] Quanyu Dai, Xiao Shen, Liang Zhang, Qiang Li, and Dan Wang.Adversarial training methods for network embedding. In TheWorld Wide Web Conference, pages 329–339, 2019.

[28] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009IEEE conference on computer vision and pattern recognition, pages248–255. Ieee, 2009.

[29] Zhijie Deng, Yinpeng Dong, and Jun Zhu. Batch virtual adver-sarial training for graph convolutional networks. arXiv preprintarXiv:1902.09192, 2019.

[30] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and KristinaToutanova. Bert: Pre-training of deep bidirectional transformersfor language understanding. arXiv preprint arXiv:1810.04805,2018.

[31] Palash Dey and Sourav Medya. Manipulating node similaritymeasures in network. arXiv preprint arXiv:1910.11529, 2019.

[32] David Duvenaud, Dougal Maclaurin, Jorge Aguilera-Iparraguirre, Rafael Gómez-Bombarelli, Timothy Hirzel, AlánAspuru-Guzik, and Ryan P. Adams. Convolutional networkson graphs for learning molecular fingerprints. In Proceedings ofthe 28th International Conference on Neural Information ProcessingSystems, NIPS’15, pages 2224–2232, 2015.

[33] Negin Entezari, Saba A Al-Sayouri, Amirali Darvishzadeh, andEvangelos E Papalexakis. All you need is low (rank) defendingagainst adversarial attacks on graphs. In Proceedings of the 13thInternational Conference on Web Search and Data Mining, pages 169–177, 2020.

[34] Sarah M. Erfani, Sutharshan Rajasegarar, Shanika Karunasek-era, and Christopher Leckie. High-dimensional and large-scaleanomaly detection using a linear one-class svm with deep learn-ing. Pattern Recognition, 58:121 – 134, 2016.

[35] Dhivya Eswaran, Stephan Günnemann, Christos Faloutsos,Disha Makhija, and Mohit Kumar. Zoobp: Belief propagationfor heterogeneous networks. Proceedings of the VLDB Endowment,10(5):625–636, 2017.

[36] Fuli Feng, Xiangnan He, Jie Tang, and Tat-Seng Chua. Graphadversarial training: Dynamically regularizing based on graphstructure. IEEE Transactions on Knowledge and Data Engineering,2019.

[37] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnosticmeta-learning for fast adaptation of deep networks. In Proceed-ings of the 34th International Conference on Machine Learning-Volume70, pages 1126–1135. JMLR. org, 2017.

[38] James Fox and Sivasankaran Rajamanickam. How robust aregraph neural networks to structural noise? arXiv preprintarXiv:1912.10206, 2019.

[39] Jason Gaitonde, Jon Kleinberg, and Eva Tardos. Adversarialperturbations of opinion dynamics in networks. arXiv preprintarXiv:2003.07010, 2020.

[40] Michelle Girvan and Mark EJ Newman. Community structure insocial and biological networks. Proceedings of the national academyof sciences, 99(12):7821–7826, 2002.

[41] Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Ex-plaining and harnessing adversarial examples. arXiv:1412.6572v3,2015.

[42] Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive rep-resentation learning on large graphs. In Advances in neuralinformation processing systems, pages 1024–1034, 2017.

[43] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deepresidual learning for image recognition. In Proceedings of the IEEEconference on computer vision and pattern recognition, pages 770–778,2016.

[44] Shifu Hou, Yujie Fan, Yiming Zhang, Yanfang Ye, Jingwei Lei,Wenqiang Wan, Jiabin Wang, Qi Xiong, and Fudong Shao. αcyber:Enhancing robustness of android malware detection systemagainst adversarial attacks on heterogeneous graph based model.In Proceedings of the 28th ACM International Conference on Informa-tion and Knowledge Management, pages 609–618, 2019.

[45] Binbin Hu, Zhiqiang Zhang, Chuan Shi, Jun Zhou, XiaolongLi, and Yuan Qi. Cash-out user detection based on attributedheterogeneous information network with a hierarchical attentionmechanism. In Proceedings of the AAAI Conference on ArtificialIntelligence, volume 33, pages 946–953, 2019.

16

[46] Vassilis N Ioannidis, Dimitris Berberidis, and Georgios B Gian-nakis. Graphsac: Detecting anomalies in large-scale graphs. arXivpreprint arXiv:1910.09589, 2019.

[47] Vassilis N Ioannidis and Georgios B Giannakis. Edge ditheringfor robust adaptive graph convolutional networks. arXiv preprintarXiv:1910.09590, 2019.

[48] Vassilis N Ioannidis, Antonio G Marques, and Georgios B Gian-nakis. Tensor graph convolutional networks for multi-relationaland robust learning. arXiv preprint arXiv:2003.07729, 2020.

[49] Jinyuan Jia, Binghui Wang, Xiaoyu Cao, and Neil ZhenqiangGong. Certified robustness of community detection againstadversarial structural perturbation via randomized smoothing.arXiv preprint arXiv:2002.03421, 2020.

[50] Hongwei Jin and Xinhua Zhang. Latent adversarial training ofgraph convolution networks. In ICML Workshop on Learning andReasoning with Graph-Structured Representations, 2019.

[51] Ming Jin, Heng Chang, Wenwu Zhu, and Somayeh Sojoudi.Power up! robust graph convolutional network against evasionattacks based on graph powering. arXiv preprint arXiv:1905.10029,2019.

[52] Wei Jin, Yaxin Li, Han Xu, Yiqi Wang, and Jiliang Tang. Adversar-ial attacks and defenses on graphs: A review and empirical study.arXiv preprint arXiv:2003.00653, 2020.

[53] Thomas N Kipf and Max Welling. Semi-supervised classi-fication with graph convolutional networks. arXiv preprintarXiv:1609.02907, 2016.

[54] Danai Koutra, Ankur Parikh, Aaditya Ramdas, and Jing Xiang.Algorithms for graph similarity and subgraph matching. In Proc.Ecol. Inference Conf., 2011.

[55] Valdis E Krebs. Mapping networks of terrorist cells. Connections,24(3):43–52, 2002.

[56] Chetan Kumar, Riazat Ryan, and Ming Shao. Adversary forsocial good: Protecting familial privacy through joint adversarialattacks. In Conference on Artificial Intelligence (AAAI), 2020.

[57] Jure Leskovec, Jon Kleinberg, and Christos Faloutsos. Graph evo-lution: Densification and shrinking diameters. ACM transactionson Knowledge Discovery from Data (TKDD), 1(1):2–es, 2007.

[58] Ao Li, Zhou Qin, Runshi Liu, Yiqun Yang, and Dong Li. Spamreview detection with graph convolutional networks. In Proceed-ings of the 28th ACM International Conference on Information andKnowledge Management, pages 2703–2711, 2019.

[59] Jia Li, Honglei Zhang, Zhichao Han, Yu Rong, Hong Cheng, andJunzhou Huang. Adversarial attack on community detection byhiding individuals. In WWW, 2020.

[60] Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusionconvolutional recurrent neural network: Data-driven traffic fore-casting. arXiv preprint arXiv:1707.01926v3, 2018.

[61] Geert Litjens, Thijs Kooi, Babak Ehteshami Bejnordi, ArnaudArindra Adiyoso Setio, Francesco Ciompi, Mohsen Ghafoorian,Jeroen A.W.M. van der Laak, Bram van Ginneken, and Clara I.Sanchez. A survey on deep learning in medical image analysis.Medical Image Analysis, 42:60 – 88, 2017.

[62] Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delvinginto transferable adversarial examples and black-box attacks.arXiv preprint arXiv:1611.02770, 2016.

[63] David Lusseau, Karsten Schneider, Oliver J Boisseau, Patti Haase,Elisabeth Slooten, and Steve M Dawson. The bottlenose dolphincommunity of doubtful sound features a large proportion of long-lasting associations. Behavioral Ecology and Sociobiology, 54(4):396–405, 2003.

[64] Yao Ma, Suhang Wang, Lingfei Wu, and Jiliang Tang. Attack-ing graph convolutional networks via rewiring. arXiv preprintarXiv:1906.03750, 2019.

[65] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dim-itris Tsipras, and Adrian Vladu. Towards deep learning modelsresistant to adversarial attacks. arXiv preprint arXiv:1706.06083,2017.

[66] Andrew Kachites McCallum, Kamal Nigam, Jason Rennie, andKristie Seymore. Automating the construction of internet portalswith machine learning. Information Retrieval, 3(2):127–163, 2000.

[67] Benjamin A Miller, Mustafa Çamurcu, Alexander J Gomez, KevinChan, and Tina Eliassi-Rad. Improving robustness to attacksagainst vertex classification. In MLG Workshop in KDD, 2019.

[68] Benjamin A Miller, Mustafa Çamurcu, Alexander J Gomez, KevinChan, and Tina Eliassi-Rad. Topological effects on attacks againstvertex classification. arXiv preprint arXiv:2003.05822, 2020.

[69] Riccardo Miotto, Fei Wang, Shuang Wang, Xiaoqian Jiang, andJoel T Dudley. Deep learning for healthcare: review, opportunitiesand challenges. Briefings in bioinformatics, 2017.

[70] Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha,Z Berkay Celik, and Ananthram Swami. Practical black-boxattacks against machine learning. In Proceedings of the 2017 ACMon Asia Conference on Computer and Communications Security, pages506–519. ACM, 2017.

[71] Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrik-son, Z Berkay Celik, and Ananthram Swami. The limitationsof deep learning in adversarial settings. In 2016 IEEE Europeansymposium on security and privacy (EuroS&P), pages 372–387. IEEE,2016.

[72] Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deep-walk: Online learning of social representations. arXiv preprintarXiv:1403.6652v2, 2014.

[73] Pouya Pezeshkpour, Yifan Tian, and Sameer Singh. Investigatingrobustness and interpretability of link prediction via adversarialmodifications. arXiv preprint arXiv:1905.00563, 2019.

[74] Pouya Samangouei, Maya Kabkab, and Rama Chellappa.Defense-gan: Protecting classifiers against adversarial attacksusing generative models. arXiv preprint arXiv:1805.06605, 2018.

[75] Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, BrianGalligher, and Tina Eliassi-Rad. Collective classification in net-work data. AI magazine, 29(3):93–93, 2008.

[76] Chuan Shi, Yitong Li, Jiawei Zhang, Yizhou Sun, and S Yu Philip.A survey of heterogeneous information network analysis. IEEETransactions on Knowledge and Data Engineering, 29(1):17–37, 2016.

[77] Ke Sun, Zhouchen Lin, Hantao Guo, and Zhanxing Zhu. Virtualadversarial training on graph convolutional networks in nodeclassification. In Chinese Conference on Pattern Recognition andComputer Vision (PRCV), pages 431–443. Springer, 2019.

[78] Lichao Sun, Zhiqiang Li, Qiben Yan, Witawas Srisa-an, andYu Pan. Sigpid: significant permission identification for androidmalware detection. In Malicious and Unwanted Software (MAL-WARE), 2016 11th International Conference on, pages 1–8. IEEE,2016.

[79] Lichao Sun, Yuqi Wang, Bokai Cao, S Yu Philip, Witawas Srisa-An, and Alex D Leow. Sequential keystroke behavioral biomet-rics for mobile user identification via multi-view deep learning.In Joint European Conference on Machine Learning and KnowledgeDiscovery in Databases, pages 228–240. Springer, 2017.

[80] M. Sun, J. Tang, H. Li, Bo Li, C. X., Y. Chen, and D. Song. Datapoisoning attack against unsupervised node embedding methods.arXiv preprint arXiv:1810.12881, 2018.

[81] Yiwei Sun, Suhang Wang, Xianfeng Tang, Tsung-Yu Hsieh, andVasant Honavar. Non-target-specific node injection attacks ongraph neural networks: A hierarchical reinforcement learningapproach. In WWW, 2019.

[82] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna,Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguingproperties of neural networks. arXiv:1312.6199v4, 2014.

[83] Tsubasa Takahashi. Indirect adversarial attacks via poisoningneighbors for graph convolutional networks. In 2019 IEEEInternational Conference on Big Data (Big Data), pages 1395–1400.IEEE, 2019.

[84] Jie Tang, Jing Zhang, Limin Yao, Juanzi Li, Li Zhang, andZhong Su. Arnetminer: extraction and mining of academic socialnetworks. In Proceedings of the 14th ACM SIGKDD internationalconference on Knowledge discovery and data mining, 2008.

[85] Xianfeng Tang, Yandong Li, Yiwei Sun, Huaxiu Yao, PrasenjitMitra, and Suhang Wang. Transferring robustness for graphneural network against poisoning attacks. In Proceedings of the13th International Conference on Web Search and Data Mining, pages600–608, 2020.

[86] Petar Velickovic, Guillem Cucurull, Arantxa Casanova, AdrianaRomero, Pietro Lio, and Yoshua Bengio. Graph attention net-works. arXiv preprint arXiv:1710.10903, 2017.

[87] Petar Velickovic, William Fedus, William L Hamilton, Pietro Liò,Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. arXivpreprint arXiv:1809.10341, 2018.

[88] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, OmerLevy, and Samuel R Bowman. Glue: A multi-task benchmarkand analysis platform for natural language understanding. arXivpreprint arXiv:1804.07461, 2018.

[89] Binghui Wang and Neil Zhenqiang Gong. Attacking graph-based classification via manipulating the graph structure. In

17

Proceedings of the 2019 ACM SIGSAC Conference on Computer andCommunications Security, pages 2023–2040, 2019.

[90] Shen Wang, Zhengzhang Chen, Jingchao Ni, Xiao Yu, Zhichun Li,Haifeng Chen, and Philip S Yu. Adversarial defense frameworkfor graph neural network. arXiv preprint arXiv:1905.03679, 2019.

[91] Xiaoyun Wang, Joe Eaton, Cho-Jui Hsieh, and Felix Wu. Attackgraph convolutional networks by adding fake nodes. arXivpreprint arXiv:1810.10751, 2018.

[92] Xiaoyun Wang, Xuanqing Liu, and Cho-Jui Hsieh. Graphdefense:Towards robust graph convolutional networks. arXiv preprintarXiv:1911.04429, 2019.

[93] Marcin Waniek, Tomasz P Michalak, Michael J Wooldridge, andTalal Rahwan. Hiding individuals and communities in a socialnetwork. Nature Human Behaviour, 2(2):139–147, 2018.

[94] Marcin Waniek, Kai Zhou, Yevgeniy Vorobeychik, Esteban Moro,Tomasz P Michalak, and Talal Rahwan. Attack tolerance of linkprediction algorithms: How to hide your relations in a socialnetwork. arXiv preprint arXiv:1809.00152, 2018.

[95] Wikipedia. Average precision. https://bit.ly/2Uz06lL.[96] Wikipedia. Confusion matrix. https://bit.ly/2wHUpcf.[97] Wikipedia. Mean reciprocal rank. https://bit.ly/3aBadMk.[98] Wikipedia. Modularity. https://bit.ly/3dMbsdB).[99] Wikipedia. Mutual information. https://bit.ly/3bBeDCY.[100] Wikipedia. Rand index. https://bit.ly/3azqoK6.[101] Wikipedia. Roc. https://bit.ly/341yHfa.[102] Felix Wu, Tianyi Zhang, Amauri Holanda de Souza Jr, Christo-

pher Fifty, Tao Yu, and Kilian Q Weinberger. Simplifying graphconvolutional networks. arXiv preprint arXiv:1902.07153, 2019.

[103] Huijun Wu, Chen Wang, Yuriy Tyshetskiy, Andrew Docherty, KaiLu, and Liming Zhu. Adversarial examples for graph data: Deepinsights into attack and defense. In International Joint Conferenceon Artificial Intelligence, IJCAI, pages 4816–4823, 2019.

[104] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long,Chengqi Zhang, and Philip S Yu. A comprehensive survey ongraph neural networks. arXiv preprint arXiv:1901.00596, 2019.

[105] Chaowei Xiao, Jun-Yan Zhu, Bo Li, Warren He, Mingyan Liu, andDawn Song. Spatially transformed adversarial examples. arXivpreprint arXiv:1801.02612, 2018.

[106] Hui Y. Xiong, Babak Alipanahi, Leo J. Lee, Hannes Bretschneider,Daniele Merico, Ryan K. C. Yuen, Yimin Hua, Serge Gueroussov,Hamed S. Najafabadi, Timothy R. Hughes, Quaid Morris, YosephBarash, Adrian R. Krainer, Nebojsa Jojic, Stephen W. Scherer, Ben-jamin J. Blencowe, and Brendan J. Frey. The human splicing codereveals new insights into the genetic determinants of disease.Science, 347(6218), 2015.

[107] Kaidi Xu, Hongge Chen, Sijia Liu, Pin-Yu Chen, Tsui-Wei Weng,Mingyi Hong, and Xue Lin. Topology attack and defense forgraph neural networks: an optimization perspective. In Proceed-ings of the 28th International Joint Conference on Artificial Intelligence,pages 3961–3967. AAAI Press, 2019.

[108] Xiaojun Xu, Yue Yu, Bo Li, Le Song, Chengfeng Liu, and CarlGunter. Characterizing malicious edges targeting on graph neu-ral networks. Openreview, 2018.

[109] Qi Xuan, Yalu Shan, Jinhuan Wang, Zhongyuan Ruan, and Guan-rong Chen. Adversarial attacks to scale-free networks: Testingthe robustness of physical criteria. arXiv preprint arXiv:2002.01249,2020.

[110] Qi Xuan, Jun Zheng, Lihong Chen, Shanqing Yu, Jinyin Chen,Dan Zhang, and Qingpeng Zhang Member. Unsupervised eu-clidean distance attack on network embedding. arXiv preprintarXiv:1905.11015, 2019.

[111] Naganand Yadati, Madhav Nimishakavi, Prateek Yadav, VikramNitin, Anand Louis, and Partha Talukdar. Hypergcn: A newmethod for training graph convolutional networks on hyper-graphs. In Advances in Neural Information Processing Systems,pages 1509–1520, 2019.

[112] Zhilin Yang, William W Cohen, and Ruslan Salakhutdinov. Re-visiting semi-supervised learning with graph embeddings. arXivpreprint arXiv:1603.08861, 2016.

[113] Wayne W Zachary. An information flow model for conflictand fission in small groups. Journal of anthropological research,33(4):452–473, 1977.

[114] Xiao Zang, Yi Xie, Jie Chen, and Bo Yuan. Graph universaladversarial attacks: A few bad actors ruin graph learning models.arXiv preprint arXiv:2002.04784, 2020.

[115] Hengtong Zhang, Tianhang Zheng, Jing Gao, Chenglin Miao,Lu Su, Yaliang Li, and Kui Ren. Data poisoning attack against

knowledge graph embedding. arXiv preprint arXiv:1904.12052,2019.

[116] Yiming Zhang, Yujie Fan, Yanfang Ye, Liang Zhao, and Chuan Shi.Key player identification in underground forums over attributedheterogeneous information network embedding framework. InProceedings of the 28th ACM International Conference on Informationand Knowledge Management, pages 549–558, 2019.

[117] Yingxue Zhang, S Khan, and Mark Coates. Comparing anddetecting adversarial attacks for graph deep learning. In Proc.Representation Learning on Graphs and Manifolds Workshop, Int. Conf.Learning Representations, New Orleans, LA, USA, 2019.

[118] Kai Zhou, Tomasz P Michalak, and Yevgeniy Vorobeychik. Ad-versarial robustness of similarity-based link prediction. arXivpreprint arXiv:1909.01432, 2019.

[119] Kai Zhou, Tomasz P Michalak, Marcin Waniek, Talal Rahwan,and Yevgeniy Vorobeychik. Attacking similarity-based link pre-diction in social networks. In Proceedings of the 18th InternationalConference on Autonomous Agents and MultiAgent Systems, pages305–313. International Foundation for Autonomous Agents andMultiagent Systems, 2019.

[120] Dingyuan Zhu, Ziwei Zhang, Peng Cui, and Wenwu Zhu. Robustgraph convolutional networks against adversarial attacks. InProceedings of the 25th ACM SIGKDD International Conference onKnowledge Discovery & Data Mining, pages 1399–1407, 2019.

[121] Daniel Zügner, Amir Akbarnejad, and Stephan Günnemann.Adversarial attacks on neural networks for graph data. InProceedings of the 24th ACM SIGKDD International Conference onKnowledge Discovery & Data Mining, pages 2847–2856, 2018.

[122] Daniel Zügner and Stephan Günnemann. Adversarial attackson graph neural networks via meta learning. arXiv preprintarXiv:1902.08412, 2019.

[123] Daniel Zügner and Stephan Günnemann. Certifiable robust-ness and robust training for graph convolutional networks. InProceedings of the 25th ACM SIGKDD International Conference onKnowledge Discovery & Data Mining, pages 246–256, 2019.