SCSMiner: Mining Social Coding Sites for SoftwareDeveloper Recommendation with Relevance

Propagation

Yao Wana,b, Liang Chenc, Guandong Xud,∗, Zhou Zhaoa, Jie Tange, JianWua,b

aCollege of Computer Science and Technology, Zhejiang University, Hangzhou, ChinabRealdoctor Artificial Intelligence Research Center of Zhejiang University, Hangzhou, China

cSchool of Data and Computer Science, Sun Yat-Sen University, Guangzhou, ChinadAdvanced Analytics Institute, University of Technology, Sydney, Australia

eDepartment of Computer Science and Technology, Tsinghua University, Beijing, China

Abstract

With the advent of social coding sites, software development has entered a new

era of collaborative work. Social coding sites (e.g., GitHub) can integrate social

networking and distributed version control in a unified platform to facilitate

collaborative developments over the world. One unique characteristic of such

sites is that the past development experiences of developers provided on the sites

convey the implicit metrics of developer’s programming capability and expertise,

which can be applied in many areas, such as software developer recruitment for

IT corporations. Motivated by this intuition, we aim to develop a framework to

effectively locate the developers with right coding skills. To achieve this goal, we

devise a generativ e probabilistic expert ranking model upon which a consistency

among projects is incorporated as graph regularization to enhance the expert

ranking and a perspective of relevance propagation illustration is introduced.

For evaluation, StackOverflow is leveraged to complement the ground truth of

expert. Finally, a prototype system, SCSMiner, which provides expert search

service based on a real-world dataset crawled from GitHub is implemented and

demonstrated.

∗Corresponding authorEmail address: guandong.xu@uts.edu.au (Guandong Xu)

Preprint submitted to Journal of LATEX Templates November 24, 2017

Keywords: SCSMiner, social coding sites, expert finding, developer

recommendation, relevance propagation.

1. Introduction

Along with the prevalence of social networks in the world, social coding sites

(SCS) is changing software development toward a more collaborative manner by

the way of integrating social media functionality and distributed version control

tools. For instance, as a large website for social coding, GitHub1 is developing5

fast and growing into one of the best-known websites. Ever since the publication

of Git and its successful application by Ruby community for distributed version

control, the size of GitHub has been increasing since 2008 and this trend has

dramatically accelerated over the past few years. According to our statistical

analysis, until June 2015, the numbers of developers and projects in GitHub10

have reached up to 11,610,094 and 20,598,603, respectively.

A systemic view of social coding sites. Figure 1 describes the schema of

developer and project profiles in SCS. In SCS, information concerning user’s

name, company, location and followers count property is available, and the

name, owner, description property and README file of project are also pro-15

vided. Specifically, for developers, the location feature records where the devel-

oper is; the hireable feature records the current working status of a developer.

For projects, the contributors feature records a project’s contributors and lines

of codes each contributor contributes to; the language feature (e.g., C++, Java,

Python) refers to the main language of a project; the stargazers count is the20

number of developers who have starred the project, and it can represent the

quality of projects in some sense. It’s worth noting that every project in SCS

consists of a brief description, apart from some detailed information in the

README file for most projects. Unlike traditional open source code platforms

1https://www.github.com

2

Figure 1: The schema of developer and project profiles in GitHub, one of the most popular

social coding site. In GitHub, each user has name, company, location and followers count

property, each project has name, owner, description and README property. A heterogeneous

network between users and projects can be constructed based on these information.

such as SourceForge2 and BitBucket3, SCS like GitHub offers not only a code25

hosting service, but also an online tool for collaborative software development.

In SCS, collaborators of projects can make changes to the content of repository

and review the contributions submitted to the repository, while those who are

not collaborators but wish to contribute to a project can fork it. In addition,

users in GitHub can follow others in order to be notified of their actions, and30

users can star interesting repositories if they want to bookmark for later ref-

erence. Based on the information in SCS, we can construct a heterogeneous

collaborative network of developers and projects.

Motivation. The past development experiences of developers in SCS demon-

strate the implicit metrics of developer’s programming capability and expertise35

which can be applied in many areas such as software developer recruitment for

IT corporations. In IT industry, IT professional recruiting has always been a

slog for everyone involved, so finding better and easier ways to achieve successful

recruiting is of great importance. The most common practice in current recruit-

ment is through LinkedIn service by referring to the information published in40

LinkedIn to measure the professional level of targeted candidates. One limita-

2http://www.sourceforge.net/3http://www.bitbucket.org

3

tion of such practice is that a personal profile can only be updated intentionally,

i.e., it’s a static profile rather than a dynamic profile. Another limitation is

that it’s difficult to measure the proficiency of developers only through static

profile, e.g., recruiters can’t judge the degree of proficiency of developer who45

declares that he/she is proficient in Java programming. In contrast, the ex-

pertise information derived from SCS will undoubtedly be more dynamic and

accurate about developers’ portfolio, professional interest and influence through

their participated projects hosted in the site, which makes recruiters easier to

make selections. This intuition mainly motivate us to conduct this study for50

developing a practical system to fulfill such aim.

Challenges. Bearing the above in mind, we propose to develop a coding ex-

pert recommendation system via SCS mining, named SCSMiner4. This system

is able to provide a list of developers with ranking based on the given query and

the rich information embedded in SCSs, which we firmly believe will benefit the55

software developers online recruitment and promote the development of open

source community [1]. Although many SCSs such as GitHub has its own search

engine to rank developers based on some simple metrics, e.g., the number of

their followers, its performance is far from satisfaction. Having a certain num-

ber of followers usually indicates the popularity and professional competence60

of users to some extent, however having less followers does not mean that the

candidate is necessarily uncompetitive. This work aims to address this ranking

problem for a given query. There are mainly two challenges existing in our work.

On the one hand, the textual information in projects is chaotic and limited. In

the README files of projects, there exists many codes and installation instruc-65

tions, making the topic model which performs well in textual information not

satisfactorily in this context. On the other hand, existing expert finding models

heavily rely on the textual information of projects alone, ignoring the implicit

information in the network structure of projects and developers. Combining

the embedded network structure along with the limited textual information70

4http://scsminer.wanyao.me

4

has proved to an option to alleviate the limited textual information problem of

projects [2]. Therefore, the challenge is how to integrate the textual information

with the network structure in a unified framework to enhance the performance

of expert searching and ranking simultaneously. Another difficulty is that unlike

the academic network (e.g., DBLP) that many researches have been conducted75

on, the expertise of retrieved developers is difficult to quantitatively evaluate

for the reason that no ground truth is available.

To implement this system, we design an expertise ranking algorithm. We

first extract the textual information of projects from README file and then use

a vector space model with cosine similarity to calculate the semantic similarity80

between a project and a given query. Furthermore, a graph based regularization

framework is proposed to incorporate consistency hypothesis among the network

structure of projects alongside probabilistic model to measure the expertise score

of each candidate developer. Comprehensive experiments conducted on a real-

world dataset demonstrate the performance of our proposed approach. Finally,85

a prototype system of SCSMiner is developed and demonstrated.

Contributions. The main contributions of this paper can be summarized as

follows.

• To the best of our knowledge, this work is the first attempt to discoverlocal experts on social coding sites, which can benefit many applications.90

• We apply a probabilistic model to evaluate the expertise of SCS devel-opers and extend this model by graph regularization to incorporate the

consistency hypothesis among projects. Furthermore, we also illustrate

the inner connections between those two models from a relevance propa-

gation perspective.95

• Comprehensive experiments based on real-world data crawled from GitHubare conducted to demonstrate the performance of the proposed approach.

Due to the lack of ground truth, we use the experts from StackOverflow

to verify the experts retrieved by our model creatively.

5

• Our prototype SCSMiner providing expert search service is built based100

on the proposed approach. The involved data, codes have been released,

which will facilitate other researchers to repeat our experiments and verify

their own idea5.

Organization. The remainder of this paper is organized as follows. Section

2 highlights some works related to this paper. Section 3 defines our problem105

mathematically and gives an overview of SCSMiner’s system architecture. In

Section 4, a probabilistic model is proposed to measure the expertise of each

candidate related to a given query. Consistency hypothesis are proposed and in-

corporated to improve the performance of expertise ranking in Section 5. Section

6 describes the dataset we use in our experiment and shows the experimental110

results and analysis. Finally, we conclude this paper and propose some future

research directions in Section 7.

2. Related Work

In this section, we will make a thorough investigation of existing works about

the social coding sites and present some related works about it. Generally, the115

related works can be grouped into three categories.

Mining on social coding sites. The important role of social network played

in software development and the transparency and collaboration the social net-

work brings have been confirmed in [3][4]. Social coding sites have been studied

from several different perspectives. The first is the study of social network120

among social coding sites [5][6] [7], where the collaboration between users, the

dissemination of projects and the interaction among them are analyzed system-

atically. In those works, the social coding sites are seen as a kind of social

network. The second is on mining of source code and commit log. For instance,

[8] aims to characterize the known coding errors to improve the automatic de-125

tection of software defects, [9] quantitatively and qualitatively investigates when

5http://wanyao.me/projects/scsminer.html

6

and why developer change software licenses. These works can be used to pro-

mote the development of software engineering. Besides, some interesting related

applications are also emerging. In [10], the authors extract the professional skill

of developers in GitHub and propose a pipeline that automatizes the process130

of recruiting and job recommendation. For software development, [11] investi-

gate the team formation problem for better collaboration. In [1], the authors

analyze the impression formation in an online distributed software development

community with social media functionality. In [12], the authors investigated

the interplay between StackOverflow activities and the development process,135

reflected by code changes committed to the largest social coding repository

GitHub. Different from those works, our paper mainly focus on finding the

experts by their domain knowledge and social network.

README Extraction and Text Analysis. In [13], a new statistical ap-

proach is introduced to automatically partitioning text into coherent segments.140

Sodedant et al. [14] develop an approach to learn text extraction rules auto-

matically for text styles ranging from highly structured to free text. In [15],

a novel measure is proposed for semantic parsing evaluation of interpreters for

instructions in computer program README files. The description information

extracted from README file are used to calculate the semantic similarities in145

our model. Some models can handle it. A non-probabilistic language model [16]

based on TF-IDF is proposed. ln [17], Hofmann et al. propose the probabilistic

latent semantic indexing (pLSI) and apply it to information retrieval. Blei et

al. [18] put forward a generative probabilistic model called Latent Dirichlet

Allocation (LDA). Based on these topic models, some other works have been150

conducted for modeling both author interests and document contents. The

Author-Topic model [19] extends the LDA by integrating the authorship and

can find a topic mixture over documents and authors. Jie et al. [20] proposed

a unified topic model to simultaneously model the topical aspects of different

types of information in the academic network. Conducting experiments using155

those classical methods, we find that the simplest TF-IDF method achieves the

best performance.

7

Expertise finding. Broadly related to our scenario is expert finding. Con-

siderable works have been conducted to associate query topics to people for

expertise retrieval [21]. In [22], the authors propose a general probabilistic160

framework for studying expert finding problem and derived two families of gen-

erative models e.g., candidate generation models and topic generation models

from the framework. In [23], a discriminative learning framework is proposed

for expert finding and two specific probabilistic models i.e., the arithmetic mean

discriminative model and the geometric mean discriminative model are derived165

from this framework. In [24], the authors propose a novel voting model for

predicting and ranking candidate expertise with respect to a query inspired by

techniques from the field of data fusion. In [25] the authors model the multi-step

relevance probability dissemination in topic-specific expertise graphs consisting

of persons and top retrieved documents. In [26], the authors propose a joint170

regularization framework to enhance expertise retrieval by modeling heteroge-

neous networks as regularization constraints on top of document-centric model.

With the succesfully development of deep learning, some deep learning based

method such as RNN is utilized to learn the representation of users’ topics[27].

In [27], Zhao et al. propose a ranking metric network learning framework for175

expert finding in community question answering by exploiting both users’ rela-

tive quality rank to given questions and their social relations. Our work mainly

refers to [26] but not limited to. Although our proposed model is mainly bor-

rowed from [26], we transfer it to a novel scenario that is expert finding in social

coding sites and analyze the topics of developers from the README files of their180

developed projects. Besides, we discuss the connection between our proposed

model and generative probabilistic model from the perspective of random walk.

Furthermore, we design a novel method to indirectly evaluate the performance

of proposed model.

8

3. Preliminaries185

In this section, we introduce the information we can get from a SCS, and

formally define the problem of expertise search and ranking in a heterogeneous

network.

3.1. Problem Definition

As shown in Figure 1, a collaborative heterogeneous network consists of two190

types of object sets, including a developer setD = {d1, d2, · · · , dm} and a projectset R = {r1, r2, · · · , rn}. Such a heterogeneous network can be defined as G =<V,E >, where V = VD

⋃VR and E = ED

⋃ER

⋃ED,R. This network can

be decomposed into three subnetworks, GD is a collaborative network between

developers, GR is a network of projects, while GD,R is a network representing195

the relationship between developers and projects.

In a developer-developer network GD, each node represents a given devel-

oper. A link is assigned to two nodes when the corresponding developers col-

laboratively develop at least one common project. We associate a weight to

each edge taking into account the number of projects where the two developers200

work together. For a project-project network GR, each node represents a given

project, two nodes are connected if the corresponding projects have at least one

common developer. In developer-developer network, we associate a weight to

each edge taking into account the number of common developers the projects

have. For GD,R, a link is assigned, if there exists a “develop” or “developed by”205

relationship between a developer and a project.

Given a heterogeneous network between developers and projects, as well

as the profile information of each project and developer, and the collaborative

relationship between them, the goal is to learn the expertise vector f ∈ RD,where each element corresponds to the expertise of each developer given a query.210

3.2. An Overview of System Architecture

Based on the model proposed in this paper, we implement an expert search

system on social coding sites named SCSMiner. The initial version of this

9

Figure 2: The system architecture of SCSMiner

system is accessible on http://scsminer.wanyao.me. Figure 2 shows the system

architecture of our SCSMiner search system. The system consists of four main215

components.

• Extraction: The crawler(s) crawls all the user and project information

from social coding sites, and the features which will be used in our model

are extracted. Due to the README file of projects information is chaotic

with many codes and installation instruction, some special processes are220

conducted to extract descriptions from README files.

• Storage and Access: This component is the data center of SCSMiner.

The information of users, projects and the relationship between them are

recorded here, and this component provides an interface for invoking.

Specifically, we employ Mongodb for storage and inverted file indexing225

method for index.

• Modeling: It utilizes a probabilistic model to measure the expertise of

each developer for a given query basically, and a regularization framework

is utilized to incorporate one consistency existing in the heterogeneous

network between developers and projects.230

10

• Search Services: Based on the modeling results, SCSMiner provides its

search service to client users.

4. Generative Probabilistic Model for Expertise Ranking

Generative probabilistic models are efficient approaches in expertise retrieval

as their good empirical performance and their potential for incorporating various

extensions in a transparent and theoretically sound fashion. The probabilistic

model proposed in this section follows the basic idea of a document-centric

probabilistic model, which is proposed to estimate the expertise of a candidate

by summing the relevance of its associated documents [28, 26]. In this model for

a given query q, the relevance probability of candidate developer d is determined

by the following equation:

p(d|q) = p(q|d)p(d)p(q)

, (1)

in which, p(d|q) denotes the expertise score of candidate developer d for the givenquery q; p(q|d) is the probability of query q given the candidate developer d; p(d)is the prior probability of being expert for an expert candidate d, and p(q) is

the prior probability of a given query q. As p(q) is a constant in expert ranking,

it can be ignored from above equation. Therefore, relevance probability of each

experts can be estimated by the probability of a query given expert candidate,

weighted by the prior probability that expert candidate d is an expert:

p(d|q) ∝ p(q|d)p(d). (2)

According to the document centric model [28], the expertise score of a developer

related to a given query can be formulated as:

p(q|d) =∑r∈Rd

p(r|d)p(q|r, d), (3)

where Rd indicates the subset of projects associated with the candidate de-

veloper d. In this equation, to simplify the calculations, it is assumed that

candidate developer d is conditionally independent of the query q given project

11

r (i.e., p(q|r, d) ≈ p(q|r)). As a result, by substituting p(q|d) in Eq. (2) andexpanding p(r|d), p(d|q) can be estimated as follows:

p(d|q) =∑r∈Rd

p(q|r)p(r|d)p(d)

=∑r∈Rd

p(q|r)p(d|r)p(r)p(d)

p(d)

=∑r∈Rd

p(q|r)p(d|r)p(r),

(4)

where p(r) denotes the prior relevance probability of project r, p(q|r) is theprobability of a query q given the project r, and p(d|r) is the probability of the235

association between the project and the candidate developer d. The probabil-

ity p(d|r) provides a ranking of candidates associated with a given project r,based on their contribution made to project r. According to Eq.(4), in order to

rank candidate developers, we should estimate three probabilities, namely, prior

probability of retrieval for project r (i.e., p(r)), relevance probability of project240

r (i.e., p(q|r)), and association probability of project r and candidate developerd (i.e., p(d|r)).In Eq.(4), p(q|r) denotes the semantic similarities between project r and a

given query q. In our paper, we simply apply vector space represented by TF-

IDF determine p(q|r), where TF-IDF is a typical vector space model and it isdefined as:

TF -IDF (w, r,R) = f(w, r)× log |D|∣∣∣{r ∈ R : w ∈ r

}∣∣∣, (5)

where f(w, r) is the frequency of word w in project r, |R| is the total numberof documents in the corpus,

∣∣∣{r ∈ R : w ∈ r

}∣∣∣ is the number of documents inwhich the term w appears.245

In our paper, we use cosine similarity to determine the semantic similarity

between a given query and project. The cosine similarity is defined as follows:

sim(q, rj) =rj · q

‖rj‖ · ‖q‖ =∑ni=1 wijwiq√∑n

i=1 w2ij

√∑ni=1 w

2iq

, (6)

where q, rj are the word vector of query q and project rj respectively.

12

For a project r developed by multiple developers, we assume that each devel-

oper has the same level of knowledge about the topics described in the project

and therefore the association probability of project r and candidate developer

d is estimated as follows:

p(d|r) =⎧⎨⎩

cd∑Di ci

d is a contributor of r,

0 otherwise.(7)

where cd is the number of commits submitted by developer d,∑Di ci is the total

number of commits submitted to project r.

In addition, p(r) can be seen as the quality of project. Indeed, p(r) can be

set to be the combination of star number and total lines of code of a project,

which can be represented as followers.

p(r) = ηsr + (1− η)lr, (8)

where sr and lr is the stars count and total lines of code of the project r,

respectively; η is the tuning parameter that controls the weight between quality250

coming from the stars count and lines of code. In our experiments, we scale the

sr and lr to a same range via min-max noamalization, and η is set to be 0.5.

For simplicity, let x be the relevance vector between project ri and query

q with xi = p(q|ri), QR be the diagonal matrix that represents the project

quality, and PRD be the composition matrix between projects and developers.

Then the primary model as shown in Eq.(4) can be rewritten as:

f = PTRDQRx (9)

where f represents the relevance score vector of all candidate developers. The

underlying intuition of this model is to estimate the expertise of candidate

developers based on the relevance and quality of associated projects.255

5. Modeling the Network

In the previous section we propose a probabilistic model to determine the

expertise score of each developer. As shown in Figure 3a, the generative proba-

bilistic model can be considered as an one-step relevance propagation [25] from

13

projects to related candidate developers. In this model, only textual informa-260

tion of projects are used to calculate the relevance between query and projects,

ignoring the network structure among projects. In this section, we propose a

consistency among projects and extend the probabilistic model to incorporate

project consistency via graph regularization. The intuition is that relevance can

not only propagate from projects to developers, it can also propagate among265

projects, which is considered as a consistency in our paper. Figure 3 gives

a graphical illustration of connections between generative probabilistic model

(PM) and that with graph regularization (regPM).

5.1. Project Consistency Regularization

According to our common sense and observation, we propose the following270

consistency hypothesis.

Project Consistency Hypothesis: Usually similar projects tend to be of

similar relevance with respect to a given query. In the project layer network, it

is reasonable to assume that the neighbors of project r are those projects that

are similar to it.275

In the generative probabilistic model, we need to calculate the semantic

similarity between query and project. However, as declared before, the textual

information of projects are limited. Incorporating the project consistency will

connect the neighbor projects, and the similarity of similar projects will consis-

tent with each other, this will alleviate the limited textual information problem.280

In this subsection, we will enforce the consistency hypothesis proposed above

with the probabilistic model by defining the regularization constraints.

Graph regularization is an effective technique in the semi-supervised learning

where the goal is to estimate the labels of unlabeled data using other partially

labeled data and their similarities. For project consistency hypothesis in our

scenario, the goal is to refine relevance score vector x based on the project

consistency regularization that similar projects tend to be of similar relevance

with respect to a given query. In the probabilistic model, the initial relevance

score x0 can be determined by the vector space model. According to [2], the

14

problem can be addressed by minimizing the regularization loss below:

Ω(x) = xT (I− SR)x+ μR∥∥x− x0

∥∥2 (10)

where μR > 0 is the regularization parameter, SR ∈ R|R×R| denotes the pairwise

similarities among projects GR. The first term of the loss function defines the

project consistency, which prefers small difference in relevance scores between285

nearby projects; the second term is the fitting constraint, which measures the

difference between final relevance scores x and the initial relevance scores x0.

The initial relevance score vector x0 can be calculated according to Eq.(9) in

the probabilistic model.

Minimizing Ω(x) will force the neighbor projects to have similar relevance

scores. Setting ∂Ω(x)/∂x = 0, we can see that the solution x∗ is essentially the

solution to the linear equation:

(I− αSR)x∗ = (1− α)x0 (11)

where α = 1/(1+μR). In this equation, we need to calculate the inverse matrix290

(I −αSR)−1. Fortunately, matrix SR is always very sparse, causing calculation

to be costly. One iterative solution to this equation is given in a related work

using a power method [29].

x(t+ 1) = αSRx(t) + (1− α)x0 (12)

where x∗ = x(∞) is the solution. Here L = (I − αSR) is essentially a variant

Laplacian on this graph using SR as the adjacency matrix; and K = (I −295

αSR)−1 = L −1 is the graph diffusion kernel.

Now the interesting question is how to calculate SR in R. For graph data,

several works [30] borrow results form spectral graph theory for obtaining the

similarity measures. Specifically for an undirected graph, SR is simply the

normalized adjacency matrixW:

SR = Π−1/2R WRΠ

−1/2R (13)

whereW is the adjacency matrix of projects in GR,Wij = 1 if node i is linked

to node j, otherwise,Wij = 0, and Π is a diagonal matrix with Πii =∑jWij .

15

(a) PM (b) regPM

Figure 3: Graphical illustration of connections between generative probabilistic model and

that with relevance propagation.

5.2. Connections and Discussions

In the regularization framework, suppose α = 0(μR → ∞); as shown in300

Eq.(10), Ω puts all weight on the second term∥∥x− x0

∥∥2 to ensure that thefinal scores are equal to the initial scores x0. In this case, the regularization

framework boils down to the baseline generative probabilistic model.

The regularization framework can also be interpreted as lazy random walks

with the transition probability matrix P = (1 − ρ)I + ρS−1R WR where ρ is a305

parameter in (0, 1) [31]. This means, with probability ρ, following on a link

coincident with the vertex of the current position and chosen with probability

proportional to the weight of the link, and with the probability 1−ρ, it stays atthe current position. This can also be observed from Figure 3. Comparing with

the generative probabilistic model, the regularized framework introduce more310

random walk rules (e.g., walks among projects).

6. Experiments

6.1. Dataset Collection

As we describe before, our experiments are conducted on the GitHub dataset.

However, the most significantly difficulty is the lack of ground truth of whether315

the retrieved developer is expert or not. One simple approach is to select some

queries and judge the relevance of retrieved results manually. However, this

16

approach can not be implemented in large scale and may be affected by many

human factors. As such, we deliberately utilize StackOverflow6 as an indirect

ground truth to conduct evaluation.320

6.1.1. Dataset1: Intersection of GitHub and StackOverflow

According to [12], expertise derived from StackOverflow can somehow reflect

those from GitHub indirectly. In [12], the authors investigate the interplay

between StackOverflow activities and the development process reflected by code

changes committed to GitHub. They find that the StackOverflow activity rate325

correlates with the code changing activity in GitHub. So we extend their finding

to validate our model via StackOverflow. Here the key is first to identify the

developers both appeared in GitHub and StackOverflow, then to verify the

developers recommended by our GitHub-base system against the results given

by StackOverflow.330

GitHub. GitHub, one of the most popular social coding sites, has gained

much popularity among a large number of software developers around the world.

It has a publicly accessible API. Crawling GitHub website by its API, we get

28,362,019 projects, 15,647,255 users and make out the relationships between

them.335

StackOverflow. StackOverflow is a popular online programming question

and answer community started in 2008. It dumps and releases the dataset every

three months. The data used in our experiment is released in August 2012,

containing about 1,295,622 registered users dating from July 2008 to August

2012.340

Intersection. To intersect data from GitHub and StackOverflow, a conser-

vative approach matching email address is adopted in our experiment. In the

GitHub dataset email address are present, while in the StackOverflow dataset

email addresses are obscured, but their MD5 hashed are available. Therefore,

we merge a GitHub and a StackOverflow user if their MD5 email hashes are345

6http://stackoverflow.com/

17

identical. Furthermore, for computation simplicity, only those users whose rep-

utation in StackOverflow is greater than 5 and followers number in GitHub is

greater than 10 are considered. Finally, we obtain 16,567 users and 458,639

related projects.

Ground Truth. In StackOverflow, each user answers many programming350

problems and the tags and their corresponding count of those problems they

answer are collected 7. In terms of the retrieved results, we consider the retrieved

developer who has also answered some corresponding problems in StackOverflow

as relevant, and the count of answering questions can be taken as the degree

of relevance. Actually, we divide the degree of relevance into five degrees (0-4)355

by four thresholds (e.g., 0, 5, 10, 50). In this evaluation, top 50 queries in

StackOverflow are used as queries (the top 20 queries are shown in Table 1).

6.1.2. Dataset2: Human labeled data

Since the some popular users in GitHub may not appear in StackOverflow,

and to have a clear understanding on the retrieved results, we also evaluate our360

model on some popular users. We extract projects whose star number is greater

than 400 firstly, and then extract users who make contributions to the projects.

Finally, we obtain 11,437 projects and 158,646 users.

Ground Truth. Evaluation on this dataset is also a nontrivial task since

the ground truth is difficult to obtain. Here, we consider this strategy. We365

first random select 8 popular queries (e.g, linux, javascript, app, plugin, an-

gular, framework, game, image) a developer may be interested in. Then a

pooling methods (see Sec. 6.3.2) are used to select an initial set of developers

for judgment. Then we invited 4 computer science graduated students to judge

our retrieved result. The followers count and the projects the developer con-370

tributed to are two criteria the judge referred when determining the relevance

of developers to queries. As it’s difficult to measure the degree of relevance

7https://api.stackexchange.com/2.2/users/1335234/tags?

order=desc&sort=popular&site=stackoverflow

18

(a) Network overview. (b) Collaboration network.

Figure 4: The GitHub network. (a) An overview of network structure in GitHub. (b) Collab-

oration network between developers in GitHub.

by human, the judges just need to determine whether the candidate developer

is expert or not, so the NDCG metric is not calculated on this dataset. The

queries and their corresponding expert used in our experiment are listed in375

http://wanyao.me/projects/scsminer.html.

6.1.3. Dataset characteristics

(a) CDF of # stars of projects (b) CDF of # followers of users

Figure 5: (a) The cumulative distribution function of projects with respect to stars number.

(b) The cumulative distribution function of developers with respect to followers number.

Figure 4 presents an overview of the network structure of GitHub and the

collaboration relationship between developers. Figure 4a shows the number of

participants in each project (the size of each node is related to its indegree).380

19

From this figure, we observe that the number of participants in projects such as

“foundation”, “node”, “three.js” is maximal. To further study the relationship

between the number of participants and projects, we examine each project’s

popularity by its stars, from which we notice that for most projects, the number

of participants indicates its popularity.385

Figure 4b illustrates relationships between developers where each sector rep-

resents a developer. In this figure, we observe that the sectors indicate the

developer’s activity degree of coorperation with others. We note that olliwolli

and vachzar are the most active user in out GitHub dataset.

Table 1: Top 20 tags in StackOverflow

Tag Frequency Tag Frequency

javascript 182,509 ruby 55,463

php 146,926 ios 51,025

java 144,660 css 49,756

c# 143,001 mysql 49,642

python 120,203 .net 48,523

jquery 118,475 objective-c 45,180

c++ 79,503 iphone 43,805

android 76,592 c 39,789

ruby-on-rails 74,207 asp.net 36,026

html 70,047 sql 32,371

Figure 5 shows the cumulative distribution function of star count with re-390

spect to projects as well as the cumulative distribution function of followers

number with respect to developers. Figure 5 (a) shows that in our experimental

dataset, most projects have a little more than 400 stars, only few developers

have many stars. From Figure 5 (b), we note that the distribution of followers

roughly follows a power law. This indicates that only few developers have more395

10 followers; most developers have few or none.

Table 1 lists the Top 20 tags in StackOverflow. The frequency column in this

20

table represents the number of question which are tagged with the corresponding

tag. From this table, we can find that most tags in Q&A sites are programming

languages like java, php and java, which is in accordance with our expectation.400

6.2. Preprocessing

Through statistics of 1,406,409 README files, the README files fall into

three main three categories: markdown (89.14%), txt (7.4%) and html (0.008%).

Since the number of html README files is limited, we consider them as txt

files simply. Regarding the other two types, the process consists of 3 steps:405

(1) separating the README file into paragraphs; (2) selecting the description

paragraphs; and (3) removing unnecessary parts.

In the paragraph separation step, for markdown files, by utilizing regular

expressions, we separate the whole README file by the subtitle separator in

markdown (e.g., ‘##’, ‘==’, ‘–’) into two sequences. One is the separator410

sequence which only contains separators, the other is the words sequence. Then

we check the separator type. If it is “#”, combining the “#” with words to

obtain the paragraph from the two sequences. If it is “-” or “=”, things may

be complex. The subtitle of the paragraph has been split into the previous

paragraph, which is its last line. So we have to choose the last line of the last415

paragraph as the new subtitle, then combine it with the separator (‘–’ or ‘==’)

and the sequence into a new paragraph. For txt files, it’s hard to get a common

separator. The method for finding a subtitle is to judge each line of text by four

conditions: (a) Delete the lines contain a web link. It is probably the download

link won’t be useful for description. (b) The whole line is English characters420

from A-Z or a-z. (c) The line has no special signals (e.g. ‘,’ ‘.’ ‘$’) that seldom

appear in subtitles (d) Shorter than 40 characters. If the line meets conditions

(b), (c) or (d), we regard it as a subtitle. Then we can use it to separate the

text.

When selecting description paragraphs, we only consider the first three para-425

graphs in a README file, because description text is usually fount there. We

check and select the descriptions by using regular expression to search for key

21

words in the subtitle. The key words we use are ‘description’, ‘feature’, the

project name and other words that could represent the meaning of ‘description’

(e.g. introduction). As for removing unnecessary parts, we remove web links430

and common Linux commands (e.g., ‘mkdir ’, ‘apt ’), which won’t contribute

to a useful description. After these three steps, the textual information can

be extracted from the README file (Figure 1 gives a graphical illustration of

these processes). According to our manual analysis based on 100 samples, the

accuracy of the proposed approach reaches as high as 82.0%.435

6.3. Experimental Setup

6.3.1. Evaluation Metrics

For our evaluation, several categories of Web search evaluation metrics are

used to measure the performance of our proposed model from different aspects,

including some relevance based metrics and ranking based metrics. To measure440

the relevance of our search results, we use the precision at rank k (P@k), which is

widely used and is defined as: P@k = # relevant in top k resultsk . P@k measures

the fraction of the top-K retrieved results that are relevant to the given query.

Another metricMAP is the mean value of the average precision computed for all

queries. MAP =∑Q

q=1 AP (q)

|Q| and AP =∑N

n=1(P@n×rel(n))R , where n is the rank,445

N the number retrieved, and rel(n) is a binary function indicating the relevance

of a given rank. We use Mean Reciprocal Rank (MRR) to evaluate the ranking

of our search results. A larger MRR value means a better result. The MRR is

defined as MRR = 1|Q|

∑|Q|i=1

1ranki

, where |Q| is the size of query set. Anotherranking metric we use in our evaluation is normalized discounted cumulative450

gain (NDCG), which measures the performance of a retrieval system based on

the graded relevance of the retrieved entities. The NDCG@k is defined as:

NDCG@k = 1|Q|

∑|Q|q=1 Zkq

∑kj=1

2r(j)−1log(1+j) , where Zkq is a normalization factor

calculated to make it so that a perfect ranking’s NDCG at k for query q is 1;

r(j) is the relevance score assessors give to retrieved entity for query q. Beside455

the measurement of precisions, bpref [32] is a good function that evaluates how

frequently relevant documents are retrieved before non-relevant documents. It

22

is defined as bpref = 1R

∑r

(1− |n ranked higher than r|

R

), where R is the number

of relevant documents, r is the relevant document retrieved and n is member of

the first R non-relevant documents retrieved.460

6.3.2. Comparison Models

In this subsection, comparisons between the models described above and the

state-of-the-art are made to show the effectiveness of our proposed approach.

• Voting Model (Voting). The voting model ranks candidates by the

number of top projects they have contributed to [24]. In [24], the authors465

evaluate 12 voting techniques based on known data fusion techniques.

Here, we only consider the reciprocal rank data fusion technique, which

will highly rank candidates who contribute to many top projects.

• Infinite RandomWalk (IRW). In [25], the authors propose three kinds

of random walks based on the principle of multi-step relevance propagation470

in topic-specific expertise graphs. Here, only the infinite random walk is

considered.

• Probabilistic Model (PM). This is the probabilistic model proposed

in Section 4.

• Probabilistic Model with Graph Regularization (regPM). Based475

on PM model, this model incorporates the consistency hypothesis among

projects with probabilistic model via graph regularization (see Sec. 5).

All the experiments in this paper are implemented with Python 2.7, and run on

a computer with an 2.2 GHz Intel Core i7 CPU and 64 GB 1600 MHz DDR3

RAM, running Debian 7.0.480

6.4. Experimental Results

The experimental results of those comparison methods on two datasets are

shown in Table 2. In this experiment, the regPM model achieves its best per-

formance when we set α = 0.8, #iteration = 10 and Top-K = 50. Some

23

(a) P@K (b) MAP (c) MRR

(d) bpref (e) NDCG

Figure 6: Impact of α on different metrics.

conclusions can be drawn from this table. Firstly, the state-of-the-art voting485

model has a bad performance for the reason that it doesn’t so well integrate the

relevance between query and project as the proposed probabilistic model, nor

does it integrate the quality of the the project and the contribution users make

to the project. The voting model just takes the number of top projects they have

contributed to into consideration. Secondly, when compared with voting model,490

the generative probabilistic model has obtained remarkable results by way of

integrating the relevance between query and project, the quality of project and

the contribution users make to the project. Thirdly, as described before, the

PM model can be interpreted as one-step relevance propagation from projects

to developers. IRW is a multi-step relevance propagation model which considers495

both the relevance propagation from developers to projects, and the relevance

propagation from projects to developers. However, from the experimental re-

sults, we note that IRW model doesn’t perform better than PM model in our

dataset. This may be illustrated by that the propagation between projects and

developers is unidirectional, thus the bidirectional propagation may reduce the500

24

performance instead.

Finally, comparing regPM with PM, we notice that integrating relevance

propagation among similar projects contributes dramatically to performance

improvement in many evaluation metrics, for example, the P@5 of regPM has

reached up to 0.508 on Dataset1, which improves 9.48%, gains outstanding505

performance when compared with the PM model. The above results have veri-

fied our intuition that the network structure among projects is complementary

and the consistency hypothesis characterizes the relevance propagation among

projects well. Another interesting finding from the results is that on Dataset1

the MRR of regPM is smaller than that of PM model. We can illustrate this510

phenomenon from two aspects. On the one hand, it may be caused by the

ground truth since the regPM performs better than PM in MRR on Dataset2.

On the other hand, introducing the consistency among projects may dispersed

the divergence of expert ranking, which may have an impact on the MRR metric.

Impact of α. In our model, α = 1/(μR + 1) where μR is a regularization515

parameter that controls project consistency, i.e., similar projects tend to be of

similar relevance with respect to a given query. To study the impact of α on the

metrics of our approach, we change α from 0 to 1 with a step value of 0.1. We

set λ = 0.8, Top-K = 50, and #iteration = 10 in this experiment. Experiments

are conducted on Dataset1. Figure 6 shows the impact α on different metrics.520

From this figure, we can observe that the value of α has a significant impact

on the performance of searching experts on SCSMiner and the optimal α lies

between 0.4 and 0.8. As α increases, the metrics increase at first, but when α

surpasses a certain threshold, they decrease with further increase in the value

of α. This indicates that incorporating project consistency approximately may525

improve the performance of searching results. While when α approaches 1, the

μR will approach 0, which indicates that the rule that the final relevance score

x∗ should be close to the initial score x0 is ignored. On this condition, the

regPM model gets its worst performance.

25

6.5. Search Example Analysis530

To gain a better insight into the proposed algorithm, we chose “linux” as an

example query for showing detailed results. The top-20 developers are listed in

Table 3. It is obvious that the results of the regPW model make more sense

than those of the PW model. After looking into details, we see that, unlike their

ranking in GitHub, and some other developer ranking in websites according to535

the “followers” number, although the followers number of a developer is small,

it can still get a high rank in our method. This can be illustrated by the ranking

mechanism in the probabilistic model and the consistency among projects we

introduce in this paper. For example, broonie has a high rank, although his

followers number is small. This is because he collaborated much with torvalds540

on the development of the Linux kernel source, with a contribution of 3%, which

is as high as the contribution of torvalds.

6.6. Demonstration

Based on the model proposed in this paper, we implement a search system

based on GitHub named SCSMiner. This system is dedicate to mining of activ-545

ities of developers on social coding sites such as GitHub for online recruitment.

The initial version of this system is accessible on http://scsminer.wanyao.me.

Figure 7 shows a screenshot of SCSMiner search system. This prototype system

just gives a general overview currently, and in our future work, more functions

and visualizations will be integrated.550

We demonstrate SCSMiner search system mainly from two parts, the search

portal part and the returned results display part. As shown in Figure 7(a), in

the homepage of SCSMiner search system, client users can search developers

not only by entering a keyword in the input box but also by clicking the hot

topic word for a quick access. After obtaining the keyword inputted by users,555

SCSMiner need to determine whether it is in the cache. If the keyword is in

cache, corresponding results will be returned immediately, otherwise, the model

proposed in this paper will be executed, which may take about 7 seconds. In

addition, we also show some statistics of GitHub dataset in the foot of this page.

26

(a) Homepage of SCSMiner system. (b) Searching results display page.

Figure 7: Screenshot of SCSMiner search system.

It’s worth mentioning that 7 seconds is really a bit long for online service, it’s560

necessary for us to parallelize the calculation of big matrices in the updated

version.

When the ranked retrieved developers are returned, SCSMiner will jump to

the result display page as shown in Figure 7(b). In this page, information of

developers including name, email and gravatar are displayed. In our future565

work, some other properties of developers such as their ego network will also

be included. The client user can also filter the returned results by some defined

conditions such as followers/following number, location, gender and so on.

7. Conclusion and Future Work

Social coding sites are changing software development. The developers’ pro-570

gramming capability and expertise which conveyed by the past development

experiences can benefit many applications such as software developer online

recruitment for IT corporations. In this paper, we devise a generative proba-

bilistic expertise ranking model and incorporate a consistency among projects

via graph regularization. Meanwhile, a prototype of SCSMiner which provides575

expert search service is implemented. Comprehensive experiments conducted

on the GitHub dataset show that our model performs better than the the state-

of-the-art voting model and random walk model.

27

Besides applied in developer recruitment for IT corporation, expert finding

on social coding sites can also be extended to some other areas such as question580

routing and developer recommendation. a) Question routing. To the best of our

knowledge, most question routing approaches fall into the content-based method

and graph-based method. For these two methods, one of the biggest challenge

is the cold start problem. For programmers, many of them are active in both

SCSs (e.g., GitHub) and Q&A sites (e.g, StackOverflow). Domain knowledge585

retrieved from SCS can be transferred to alleviate the cold start problem in Q&A

sites. b) Developer recommendation. In collaborative development, developers

can easily find a partner with specific expertise to collaborate with through our

system. Moreover, if the SCSs know about each developer’s expertise, it can

recommend appropriate developers to other to encourage them to collaborate.590

In our future work, we will first parallelize the calculation of languages mod-

els and some big matrices to accelerate the performance of SCSMiner. With the

success application of deep learning in natural language processing, maybe in

the future we will apply the deep learning based method such as RNN to learn

the representation of users. Furthermore, we also plan to extend our model for595

question routing and developer recommendation based on the domain skills of

developers via fusing the GitHub with StackOverflow or LinkedIn [10]. In addi-

tion, integrating data sources from multiple social coding and recruitment sites

also attracts us [33][34]. For instance, we can combine the data from GitHub and

that from LinkedIn. After integrating multiple data sources, many interesting600

topics such as entity matching and job recommendation can be attacked.

8. Acknowledgments

This work was down during Yao Wan’s visit to University of Technology Syd-

ney. This research was partially supported by the Natural Science Foundation

of China under grant of No. 61379119 and No. 61672453, Australia Research605

Council Linkage Project (LP140100937). We would like to thank Jie Liang,

Tianhan Xia and Junqing Luan for sharing their crawler source code with us

28

and their demo system githuber.info also gave us some inspiration.

References

[1] J. Marlow, L. Dabbish, J. Herbsleb, Impression formation in online peer610

production: activity traces and personal profiles in github, in: Proceedings

of the 2013 conference on Computer supported cooperative work, ACM,

2013, pp. 117–128.

[2] D. Zhou, O. Bousquet, T. N. Lal, J. Weston, B. Scholkopf, Learning with

local and global consistency, Advances in neural information processing615

systems 16 (16) (2004) 321–328.

[3] A. Begel, J. Bosch, M.-A. Storey, Social networking meets software devel-

opment: Perspectives from github, msdn, stack exchange, and topcoder,

Software, IEEE 30 (1) (2013) 52–66.

[4] L. Dabbish, C. Stuart, J. Tsay, J. Herbsleb, Social coding in github: trans-620

parency and collaboration in an open software repository, in: Proceedings

of the ACM 2012 conference on Computer Supported Cooperative Work,

ACM, 2012, pp. 1277–1286.

[5] J. Jiang, L. Zhang, L. Li, Understanding project dissemination on a social

coding site, in: Reverse Engineering (WCRE), 2013 20th Working Confer-625

ence on, IEEE, 2013, pp. 132–141.

[6] A. Lima, L. Rossi, M. Musolesi, Coding together at scale: Github as a

collaborative social network, arXiv preprint arXiv:1407.2535.

[7] F. Thung, T. F. Bissyande, D. Lo, L. Jiang, Network structure of social

coding in github, in: Software Maintenance and Reengineering (CSMR),630

2013 17th European Conference on, IEEE, 2013, pp. 323–326.

[8] P. Gyimesi, G. Gyimesi, Z. Toth, R. Ferenc, Characterization of source code

defects by data mining conducted on github, in: Computational Science and

Its Applications–ICCSA 2015, Springer, 2015, pp. 47–62.

29

[9] C. Vendome, M. Linares-Vasquez, G. Bavota, M. Di Penta, D. German,635

D. Poshyvanyk, License usage and changes: A large-scale study of java

projects on github, in: Program Comprehension (ICPC), 2015 IEEE 23rd

International Conference on, IEEE, 2015, pp. 218–228.

[10] C. Hauff, G. Gousios, Matching github developer profiles to job advertise-

ments, in: Proceedings of the 12th Working Conference on Mining Software640

Repositories, IEEE Press, 2015, pp. 362–366.

[11] A. Majumder, S. Datta, K. Naidu, Capacitated team formation problem on

social networks, in: Proceedings of the 18th ACM SIGKDD international

conference on Knowledge discovery and data mining, ACM, 2012, pp. 1005–

1013.645

[12] B. Vasilescu, V. Filkov, A. Serebrenik, Stackoverflow and github: asso-

ciations between software development and crowdsourced knowledge, in:

Social Computing (SocialCom), 2013 International Conference on, IEEE,

2013, pp. 188–195.

[13] D. Beeferman, A. Berger, J. Lafferty, Statistical models for text segmenta-650

tion, Machine learning 34 (1-3) (1999) 177–210.

[14] S. Soderland, Learning information extraction rules for semi-structured and

free text, Machine learning 34 (1-3) (1999) 233–272.

[15] J. P. White, Towards readme-eval: Interpreting readme file instructions,

ACL 2014 (2014) 76.655

[16] J. M. Ponte, W. B. Croft, A language modeling approach to information

retrieval, in: Proceedings of the 21st annual international ACM SIGIR

conference on Research and development in information retrieval, ACM,

1998, pp. 275–281.

[17] T. Hofmann, Probabilistic latent semantic indexing, in: Proceedings of660

the 22nd annual international ACM SIGIR conference on Research and

development in information retrieval, ACM, 1999, pp. 50–57.

30

[18] D. M. Blei, A. Y. Ng, M. I. Jordan, Latent dirichlet allocation, the Journal

of machine Learning research 3 (2003) 993–1022.

[19] M. Steyvers, P. Smyth, M. Rosen-Zvi, T. Griffiths, Probabilistic author-665

topic models for information discovery, in: Proceedings of the tenth ACM

SIGKDD international conference on Knowledge discovery and data min-

ing, ACM, 2004, pp. 306–315.

[20] J. Tang, J. Zhang, L. Yao, J. Li, L. Zhang, Z. Su, Arnetminer: extrac-

tion and mining of academic social networks, in: Proceedings of the 14th670

ACM SIGKDD international conference on Knowledge discovery and data

mining, ACM, 2008, pp. 990–998.

[21] K. Balog, Y. Fang, M. de Rijke, P. Serdyukov, L. Si, Expertise retrieval,

Foundations and Trends in Information Retrieval 6 (2–3) (2012) 127–256.

[22] H. Fang, C. Zhai, Probabilistic models for expert finding, Springer, 2007.675

[23] Y. Fang, L. Si, A. P. Mathur, Discriminative models of integrating docu-

ment evidence and document-candidate associations for expert search, in:

Proceedings of the 33rd international ACM SIGIR conference on Research

and development in information retrieval, ACM, 2010, pp. 683–690.

[24] C. Macdonald, I. Ounis, Voting for candidates: adapting data fusion tech-680

niques for an expert search task, in: Proceedings of the 15th ACM interna-

tional conference on Information and knowledge management, ACM, 2006,

pp. 387–396.

[25] P. Serdyukov, H. Rode, D. Hiemstra, Modeling multi-step relevance prop-

agation for expert finding, in: Proceedings of the 17th ACM conference on685

Information and knowledge management, ACM, 2008, pp. 1133–1142.

[26] H. Deng, J. Han, M. R. Lyu, I. King, Modeling and exploiting heteroge-

neous bibliographic networks for expertise ranking, in: Proceedings of the

12th ACM/IEEE-CS joint conference on Digital Libraries, ACM, 2012, pp.

71–80.690

31

[27] Z. Zhao, Q. Yang, D. Cai, X. He, Y. Zhuang, Expert finding for community-

based question answering via ranking metric network learning., in: IJCAI,

2016, pp. 3000–3006.

[28] K. Balog, L. Azzopardi, M. De Rijke, Formal models for expert finding in

enterprise corpora, in: Proceedings of the 29th annual international ACM695

SIGIR conference on Research and development in information retrieval,

ACM, 2006, pp. 43–50.

[29] D. Zhou, J. Huang, B. Scholkopf, Learning from labeled and unlabeled data

on a directed graph, in: Proceedings of the 22nd international conference

on Machine learning, ACM, 2005, pp. 1036–1043.700

[30] F. R. Chung, Spectral graph theory, Vol. 92, American Mathematical Soc.,

1997.

[31] D. Zhou, B. Scholkopf, A regularization framework for learning from graph

data, in: ICML workshop on statistical relational learning and Its connec-

tions to other fields, Vol. 15, 2004, pp. 67–68.705

[32] C. Buckley, E. M. Voorhees, Retrieval evaluation with incomplete informa-

tion, in: Proceedings of the 27th annual international ACM SIGIR confer-

ence on Research and development in information retrieval, ACM, 2004,

pp. 25–32.

[33] Z. Zhao, J. Cheng, F. Wei, M. Zhou, W. Ng, Y. Wu, Socialtransfer: trans-710

ferring social knowledge for cold-start cowdsourcing, in: Proceedings of

the 23rd ACM International Conference on Conference on Information and

Knowledge Management, ACM, 2014, pp. 779–788.

[34] X. Zhu, X. Li, S. Zhang, Block-row sparse multiview multilabel learning

for image classification, IEEE transactions on cybernetics 46 (2) (2016)715

450–461.

32