new method for solving reviewer assignment problem using type-2 fuzzy sets and fuzzy functions

Appl IntellDOI 10.1007/s10489-013-0445-5

New method for solving reviewer assignment problem usingtype-2 fuzzy sets and fuzzy functions

Devendra Kumar Tayal · P.C. Saxena · Ankita Sharma ·Garima Khanna · Shubhangi Gupta

© Springer Science+Business Media New York 2013

Abstract Reviewer Assignment Problem (RAP) is one ofthe cardinal problems in Government Funding agencieswhere the expertise level of the referee reviewing a pro-posal needs to be optimised to guarantee the selection ofgood R&D projects. Although many solutions have beenproposed for RAP in the past, none of them deals with theinherent imprecision associated with the problem. For in-stance, it is not possible to determine the “exact expertiselevel” of a particular reviewer in a particular domain. In thispaper, we propose a novel approach for assigning reviewersto proposals. To calculate the expertise of a reviewer in a par-ticular domain, we create a type-2 fuzzy set by assigning rel-evant weights to the various factors that affect the expertiseof the reviewer in that domain. We also create a fuzzy set of

D.K. TayalIndira Gandhi Institute of Technology, Computer ScienceEngineering Department, Guru Gobind Singh IndraprasthaUniversity, Kashmere Gate, Delhi 110006, India

P.C. SaxenaJawaharlal Nehru University, New Mehrauli Road, NewDelhi 110030, India

A. Sharma (�)Indira Gandhi Institute of Technology, Guru Gobind SinghIndraprastha University, BG-5/41-B, Paschim Vihar, NewDelhi 110063, Indiae-mail: [email protected]

G. KhannaIndira Gandhi Institute of Technology, Guru Gobind SinghIndraprastha University, ED-90, Tagore Garden, NewDelhi 110027, India

S. GuptaIndira Gandhi Institute of Technology, Guru Gobind SinghIndraprastha University, R-6/61 Sector-6, Rajnagar, Ghaziabad,Uttar Pradesh 201002, India

the proposal by selecting three keywords that best representthe proposal. We then use a fuzzy functions based equal-ity operator to compute the equality of the type-2 fuzzy setof experts and the fuzzy set of proposal keywords, whichis then subjected to a set of relevant constraints to optimizethe solution. We consider the four important aspects: work-load balancing of reviewers, avoiding Conflicts of Interest,considering individual preferences by incorporating biddingand mapping multiple keywords of a proposal. As an exten-sion to this approach, we further consider the relative im-portance of each keyword with respect to the submitted pro-posal by using representative percentage weights to createthe FUZZY sets which represent the keywords. Hence, wepropose an integrated solution based on the strong mathe-matical foundation of fuzzy logic, comprised of all the dif-ferent aspects of expertise modeling and reviewer assign-ment. An Expert System has also been developed for thesame.

Keywords Reviewer assignment · Type-2 fuzzy sets ·Fuzzy equality · Fuzzy set theory · Fuzzy functions

1 Introduction

Assignment of submitted research proposals to appropriateexpert reviewers, known as Reviewer Assignment Problem(RAP), is a crucial and challenging task for journal editors,conference program chairs and funding agencies. In general,the Selection of R&D projects comprises of multiple stages.It generally begins with a call for proposals (CFP), whichis distributed to the relevant communities, such as universi-ties and research institutions. Proposals are then submittedto the organization (e.g., funding agencies) that issued theCFP. These proposals are sent to experts for the peer review.

mailto:[email protected]

D.K. Tayal et al.

Experts normally review the proposals according to the in-structions on the rules and criteria of the funding agency.The review results are collected and ranked based on the ag-gregation methods (Sun et al., 2008 [1]).

Traditionally, reviewer assignment was handled by a sin-gle person or a small committee of members and all the taskswere performed manually. The basic approach followed laidfocus on mainly four aspects: matching proposals and re-viewers, assigning adequate number of reviewers to eachproposal, balancing reviewers’ workload and avoiding Con-flicts of Interest. However, the manual assignment methodwas very time-consuming and involved a subjective or bi-ased opinion of the committee. Also it was very challengingto optimise the assignments as all the constraints could notbe considered efficiently. Furthermore, reviewer assignmenthad to be completed under severe timing constraints as avery large number of submissions arrived near an announceddeadline. The manual procedure, thus, was very tedious anddid not always result in the best solution.

To overcome the problems faced by traditional methods,there was a rising need for an automatic mechanism for re-viewer assignment. Wang et al., 2008 [2] have provided arich survey on the various techniques that have been pro-posed in the past to deal with the issue. One of the earliestpapers that addressed an assignment solution is by Dumaisand Nielsen [3] in 1992 who computed the matching degreeof Proposal by using Latent Semantic Indexing (LSI). Theirapproach was relevant whenever an object in one set had tobe assigned a small number of objects in another set. Sincethen, several other approaches were proposed which had fo-cussed on using information retrieval techniques to computethe matching degree between proposals and reviewers (Het-tich and Pazzani, 2006 [4]; Rodriguez and Bollen, 2008 [5]).Several other studies employed intelligent techniques forautomatic key-phrase extraction to mine relevant featuresfrom a paper and reviewer’s profile (expertise) from his orher publications to locate the right reviewer for a particu-lar paper efficiently. Another research (Biswas and Hasan,2007 [6]) used the Vector Space Model to calculate paper-reviewer relevance using three approaches—free-text, ex-tracted keyword and ontology driven topic inference. A dif-ferent approach employed the use of LSI (Latent SemanticIndexing) for extracting the significant components, i.e. ti-tle and abstract, from the paper without requiring the authorto do it manually (Ferilli et al., 2006 [7]). In 2007, a noveltopic model, the Author-Persona-Topic (APT) model wasproposed in a research that allowed each author’s documentsto be divided into one or more clusters, each with its ownseparate topic distribution (Andrew and David, 2007 [8]).It also included a comparison of the language model basedapproaches with this model.

Simultaneously, in the literature, another set of ap-proaches were also proposed which were mainly focussed

on mathematical models and algorithms to tackle RAP.Some of these approaches were also applied to conferencepapers assignment that combined a greedy and an evolution-ary algorithm to assign papers submitted to a conference toreviewers (Guervos and Valdivieso, 2004 [9]). In 2008 Tay-lor [10], described an approach to measure the affinities ormembership between papers and reviewers encoded in anaffinity graph or equivalently a sparse affinity matrix. Someapproaches also considered reviewer preferences by incor-porating bidding into the assignment procedure (Benferhatand Lang, 2001 [11]; Di Mauro et al., 2005 [12]; Goldsmithand Sloan, 2007 [13]; Kolasa and Kr’ol, 2010 [14]; Papage-lis et al., 2005 [15]). These approaches, on one hand, en-sured increased satisfaction of the reviewers, while, on theother hand, they were inconvenient and often gave inaccu-rate feedback.

In the past, many hybrid assignment systems have alsobeen developed which combined decision models andknowledge rules to assist in decision making. Decision mod-els were used to deal with well-structured decision prob-lems, i.e., assignment of external reviewers to proposals,aggregation of reviewer results, and panel evaluation, whileknowledge rules were appropriate for ill-structured decisionsituations like proposal submission, peer review, and finalevaluation (Sun et al., 2007 [16], 2008 [17]; Tian et al.,2002 [18]). This approach was further refined by performingproposal grouping in which knowledge rules were designedto deal with proposal identification and proposal classifica-tion and the genetic algorithm was developed to search forthe expected groupings (Fan et al., 2009 [19]). Extendingthis further, a hybrid GRASP and GA was used to searchfor the desired solutions of assigning proposal groups to re-viewers for the large volume problem (Xu et al., 2010 [20]).Object-oriented method has also been used to design thearchitecture of the decision support systems (Tian et al.,2005 [21]). It included a group based modelling method forR&D project selection, and a corresponding OrganisationalDecision Support System architecture which was based onobject oriented methodology and could support and coordi-nate the work of decision-making groups.

Several heuristic algorithms have also been proposed forautomatically assigning reviewers to papers that providedeffective and good results (Kolasa and Krol, 2011 [22]). Ge-netic Algorithm based techniques offered satisfactory per-formance and usually resulted in optimal or near optimalassignments. Another advantage of this approach was thatit was able to produce a number of feasible project assign-ments by giving the user a number of different choices, thusfacilitating discussion on the merits of various allocations(Harper et al., 2005 [23]). Further, it was observed that thecombination of GA with ACO (Ant Colony Optimization)provided better solutions than when GA and ACO wereused separately (Kolasa and Krol, 2010 [14]). These two

New method for solving reviewer assignment problem using type-2 fuzzy sets and fuzzy functions

algorithms complemented each other and prevented prema-ture convergence. This approach was able to cooperativelyexplore the search space and quickly find good solutionswithin a small region of the search space.

Zhai et al. [25], in 2008 observed that a common short-coming in all existing work was that they did not considerthe multiple aspects of topics or expertise. All these ap-proaches required the matching of the entire document withthe overall expertise of a reviewer. These approaches failedto map the reviewers efficiently when a document containedmultiple subtopics. To combat this, they proposed a newmodel that used three general strategies—redundancy re-moval, reviewer aspect modelling and paper aspect mod-elling, to cover all the subtopics of a paper efficiently (Zhaiet al. [25]). This study was taken a step further to solve theproblem of committee review assignment that could simulta-neously assign papers to a committee of reviewers while alsobalancing their respective review workload (Karimzadehganand Zhai, 2009 [26]).

Another potential research area was the evaluation of ex-perts. A group decision support approach was developedthat was based on Analytic Hierarchy Process (AHP), scor-ing method, and fuzzy linguistic processing (Sun et al.,2008 [27]). Various attributes of the experts were rated onthe basis of formal rules that were a part of the decision sup-port system incorporating various criterions. Finally evalu-ation of the experts was performed using hierarchical struc-tures.

After going through the literature review, we find thatnone of the approaches proposed so far have been able tocapture the imprecision associated with the Reviewer As-signment Problem. The expertise of reviewers in their re-spective domains has often been considered as a crisp setwhich is misleading as it is not likely to determine the exactexpertise level of a particular reviewer in a particular do-main. Moreover, the expertise level of a reviewer also varieswith different domains and their sub-domains. In this pa-per, we focus on these problems that have long been ignoredin the past, by proposing a new model that uses fuzzy setsto represent the imprecision in expertise sets of reviewers.We have also proposed an algorithm to determine the ex-pertise of reviewers in each domain on the basis of a fairlyexhaustive list of factors affecting the expertise. As an ex-tension, we have also considered the relative importance ofeach keyword with respect to the submitted proposal usingfuzzy sets for the purpose. Also, this paper proposes an in-tegrated solution comprised of all the different aspects ofexpertise modelling and reviewer assignment that have beenconsidered in isolation in the previous studies. Four impor-tant aspects are considered here: workload balancing of re-viewers, avoiding Conflicts of Interest (COI), consideringindividual preferences by incorporating bidding and map-ping multiple keywords of a proposal.

The rest of the paper is organised as follows. Section 2contains an introduction to fuzzy logic and fuzzy set the-ory along with the various fuzzy set operators used. Sec-tion 3 is comprised of our proposed approach in the form ofa stepwise Method. Section 4 demonstrates the execution ofthe proposed method with the help of an example. Finally,Sect. 5 concludes the study and discusses the future scopeof the model.

2 Fuzzy set theory and fuzzy functions

The term ‘fuzzy logic’ first emerged in 1965 when it wasused by Lotfi A. Zadeh in the development of the theory offuzzy sets (Zadeh, 1965 [28]). According to Zadeh “A fuzzyset A in X is characterised by a membership function μA(x)

which associates with each point in X a real number in theinterval [0,1], with the value of μA(x) at x representing the“grade of membership” of x in A. A value of μA(x) that iscloser to unity corresponds to a higher membership degreein A. Classical sets (or as they are known as crisp sets) canbe taken as a special case of fuzzy sets, where the mem-bership function μA(x) can take only two values—0 and 1,depending upon whether x belongs to the set or not.

The concept of fuzzy set has been applied widely in solv-ing real world applications, in particular for the problemswhich involve imprecise, vague, uncertain data and the sys-tems based on linguistic information. Some of the recentapplications of fuzzy sets in discrete domains for problemsolving are given here, which is not otherwise exhaustive.

Wang et al., 2003 [29] have developed the fuzzy learningalgorithm to manage linguistic information, which generatesfuzzy linguistic rules from “soft instances”. They had solvedthe Iris classification problem and sport classification prob-lem using their algorithm. Chiang et al., 2005 [30] have ap-plied fuzzy classification trees to highly noisy and uncertaindata of Biochemical Laboratory Examinations. They havedeveloped a Medical Decision Support system to identifythe patients who have obtained “adenomatous polyps” andhave removed the lesions of them. Chen and Chen, 2005 [31]have developed a prioritized information fusion algorithmbased on generalized fuzzy numbers to solve “multicrite-ria fuzzy decision making problems”. Rasmani and Shen,2006 [32] have developed a method to evaluate students’performance by developing fuzzy rules based on the impre-cise problem data. Wang and Chen, 2006 [33], 2008 [34],2009 [35] have extensively applied fuzzy sets for evaluat-ing students’ answer scripts. In 2008, Wang and Chen [34],developed a new method for evaluating students’ answerscripts using vague values. In 2009, Lewis et al. [36] ap-plied fuzzy logic to solve the problem of inferring threatsin Homeland Security where the knowledge of the environ-ment involves multiple types of intelligence and infrastruc-ture data which is by nature imprecise or approximate. Very

D.K. Tayal et al.

recently in 2012, Aksac et al. [37] have developed a real timetraffic simulator utilizing an adaptive fuzzy inference mech-anism by tuning fuzzy parameters. The simulator automat-ically changes the time duration of traffic lights dependingon the vehicles waiting behind the green and red lights atthe crossroad. Therefore the system works on imprecise in-put data which corresponds to the “volume” of the vehicleswaiting at the crossroads. According to Xu et al., 2012 [38],with dramatically growing applications of fuzzy set theory,lots of queries involving fuzzy conditions appear nowadaysand these fuzzy conditions are widely applied for queryingover uncertain data. In their paper, they propose an incre-mental Membership algorithm which efficiently answers F-Ranking queries over fuzzy databases.

However, from the very advent of fuzzy set theory, a lotof criticism was made about the fact that the membershipfunction of a type-1 fuzzy set had no uncertainty associatedwith it. This acts as a contradiction to the definition of fuzzysets. To combat this issue, Lotfi A. Zadeh proposed a moresophisticated type of fuzzy sets known as type-2 fuzzy setsin 1975, which generalised the type-1 fuzzy sets further.

2.1 Type-2 fuzzy sets

Type-2 fuzzy sets were created to incorporate uncertaintyinto the membership function used in the fuzzy set theory.Type-2 fuzzy sets refer to the sets in which the member-ship degrees assigned to the elements of the Universal set bythese generalised fuzzy sets are themselves ordinary fuzzysets. Type-2 fuzzy sets provide us with a way to address thecriticism of type-1 fuzzy sets. Fuzzy set of type-2, as definedby Klir and Yuan, 1995 [39], is a fuzzy set whose member-ship function has the form X → F([0,1]), where F([0,1])denotes the set of ordinary fuzzy sets defined on [0,1].

The use of type-2 fuzzy sets to model uncertain behaviourhas been discussed in several works in the past. Type-1Fuzzy Logic based Systems employ the crisp and preciseT1FSs. For example, consider a T1FS representing the lin-guistic label of “Low” temperature. If the input temperature,say x, is 15 °C, then the membership of this input to the“Low” type-1 set will be a certain and crisp membershipvalue, say 0.4. On the other hand, a T2FS is characterizedby a fuzzy MF, i.e., the membership value (or membershipdegree) for each element of this set is a fuzzy set in [0,1].For example, if the linguistic label of “Low” temperature isrepresented by a T2FS, then the input x of 15 °C, will nolonger have a single value for the MF. Instead, the MF takeson values in a region which is bounded by a lower member-ship function (LMF) and an upper MF (UMF). Hence, 15 °Cwill have primary membership values that lie in an interval.Each point of this interval will also have a weight associatedwith it. Hence, this will create an amplitude distribution inthe third dimension to form what is called a secondary MF

(Lee et al., 2010 [40], Wagner et al., 2010 [41], Jammehet al., 2009 [42]).

Since type-2 fuzzy sets can better model the inherent un-certainty in several applications, they are able to outperformtheir type-1 counterparts. However, general type-2 fuzzylogic based systems have only recently been investigatedin more detail as the high complexity associated with theirdesign and their computational requirements made them ap-pear unsuitable for real-world use (Wagner et al., 2010 [41]).

However, in their paper, Raju and Majumdar [43] havedefined type-2 fuzzy relations as follows:

A fuzzy relation r on a relation scheme R(A1,A2, . . . ,An)

is a fuzzy subset of domain(A1) × domain(A2) × · · · ×domain(An). A fuzzy relation is said to be a type-2 fuzzyrelation if domain(Ai) is a set of fuzzy subsets.

Therefore they considered every domain(Ai ) (which is atype-2 fuzzy set) as a set of fuzzy subsets. In the currentpaper, we will use this notion of the type-2 fuzzy sets torepresent the “Expertise of Reviewers” (Sect. 3).

As an example let us consider a set A = “Adult”.If A is a type-2 fuzzy set, the membership function would

map the whole age to “young”, “manhood” and “senior”.Now we define the membership degree with which ele-

ments x and y belong to the set A:

μA(x) = “young”,

μA(y) = “manhood”,

where “young”, “manhood” etc. are also fuzzy sets.Hence, ‘A’ becomes a type-2 fuzzy set.Type-2 fuzzy sets are used extensively these days is a

variety of real world applications that require processingof imprecise and uncertain information. In his paper, Own2009 [44] has used type-2 fuzzy sets for medical diagnosisreasoning and pattern recognition.

2.2 Fuzzy set operations and fuzzy functions

This section provides a brief introduction of the fuzzy func-tions which are used for the calculation of equality betweentwo type-2 fuzzy sets.

2.2.1 Fuzzy functions

A fuzzy function may be explained by extending the defi-nition of a classical function. In the current section, we de-scribe the notations and fuzzy relationships, in particular thefuzzy inclusion and fuzzy equality for type-2 fuzzy sets de-fined earlier in the literature by Sostak, 1985 [45], Sostak,1988 [50], Sasaki, 1993 [49], Es & Coker, 1995 [24] andDemirci, 1997 [46].

The fuzzy function X(t) may thus be interpreted as beinga set of fuzzy results or fuzzy functional values X(t) ∈ F(x)

belonging to specified t ∈ F(T ).

X(t) = {Xt = X(t)∀t | t ∈ F(T )

}(1)


Here t ∈ T represents the arguments of the function and x ∈X indicates the functional values or results. The set T isreferred to as the argument domain and X denotes the rangeof values of x(t).

The interval [0,1] and the family of all fuzzy subsets ofa crisp set are denoted by I and IX .

As given by Sostak, 1985 [45], I (A,B) is a crisp set, forall A,B ∈ IX and is given by

I (A,B) = {x ∈ X : μA(x) ≤ μB(x)

}(2)

Fuzzy inclusion The fuzzy inclusion, ⊆̃, was given bySostak, 1985 [45]. For fuzzy subsets A and B of set X, wecalculate the value of fuzzy inclusion as follows

(A⊆̃B) =∧

x∈X

(Ac ∨ B

)(x) (3)

Thus the fuzzy inclusion may be considered as a function,⊆̃ : IX × IX → I . In this mapping I x denotes the familyof all fuzzy subsets of a crisp set. The real number A⊆̃B

shows “to what extent” the fuzzy set A is contained in thefuzzy set B .

Fuzzy equality The fuzzy equality, ∼=, was given bySostak, 1985 [45]. For fuzzy subsets A and B of set X,let

(A ∼= B) = (A⊆̃B) ∧ (B⊆̃A) (4)

The fuzzy equality can be considered as a function ∼= :IX × IX → I . In this mapping I x denotes the family of allfuzzy subsets of a crisp set, extending the usual equality ofcrisp sets. Here we are calculating the fuzzy equality by tak-ing the intersection of the separate results obtained by fuzzyinclusion A⊆̃B and fuzzy inclusion B⊆̃A.

Characteristic function As given by Demirci, 1997 [46]any function on crisp sets A and B is a relation on A × B

with the characteristic function represented by χI (A,B) andis calculated as follows

∧

x∈X

{χI (A,B)(x)

} ={

1 if A ⊆ B

0 if A ⊂ B(5)

Demirci, 1997 [46] also defined the following mappings tocalculate the subset relationship and equality between twofuzzy sets, respectively:

The mapping ⊆̃∗,∼=∗ : IX × IX → I is as follows

[⊆̃∗(A,B)] = [⊆̃(A,B)

] ∨(∧

x∈X

{χI (A,B)(x)

})(6)

[∼=∗(A,B)] = [⊆̃∗(A,B)

] ∧ [⊆̃∗(B,A)]

(7)

where,[⊆̃(A,B)

] =∧

x∈X

{(Ac ∨ B

)(x)

} (refer Eq. (3)

)

and Ac is the complement of fuzzy set A. μA is the mem-bership function of a set A and μAc(x) = 1 − μA(x), for allx ∈ X.

The real number [∼=∗(A,B)] = [⊆̃∗(A,B)]∧[⊆̃∗(B,A)];A,B ∈ I x shows the degree of equality to which a fuzzyset A is equal to a fuzzy set B . Thus the mapping ∼=∗ :IX × IX → I is a fuzzy equality on IX .

2.3 Fuzzy equality operator

On the basis of the fuzzy equality operator defined byDemirci, 1997 [46] (Sect. 2.1), Tayal and Saxena, 2007 [47]had defined, an equality operator to find the equality of twotuples in a relational database model. They had used the op-erator to calculate the equality of the fuzzy attributes of rela-tions in a relational database for any two tuples. The equalityoperator defined by them is as follows.

Let t1 and t2 be any two tuples of a fuzzy relation R. LetA be any attribute of R and let t1[A] and t2[A] denote thevalues of the tuples t1 and t2 for the attribute A, then fuzzyequality of t1[A] and t2[A] can be calculated as:

E(t1[A],t2[A])(t1[A], t2[A]) =

⎧⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎨

⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎩

1 if dom(A) is an ordinary set and t1[A] = t2 [A]0 if dom(A) is an ordinary set and t1 [A] = t2 [A]1 if dom(A) is a fuzzy set and t1[A] = t2[A]0 if dom(A) is a fuzzy set and t1[A] = t2[A]μt2[A](t1[A]) if dom(A) is a set of fuzzy subsets and

t1[A] is crisp value and t2[A] is a fuzzy subsetμt1[A](t2[A]) if dom(A) is a set of fuzzy subsets and

t2[A] is crisp value and t1[A] is a fuzzy subset∼=∗(t1[A], t2[A]) if dom(A) is a set of fuzzy subsets and

both t1[A] and t2[A] are fuzzy subsetsEA(t1[A], t2[A]) if t1[A] and t2[A] are crisp values and

dom(A) is a set of fuzzy subsets, whereEA(t1[A], t2[A]) is a domain dependent function

(8)

D.K. Tayal et al.

In our approach, we intend to use this operator in calcu-lating the equality between two fuzzy sets. We will use thisequality operator to calculate the equality between our type-2 fuzzy set (of reviewer’s expertise) and other fuzzy set (ofKeywords of the proposal). According to the equality oper-ator, to find the equality between two different sets, we firsthave to identify the kinds of the two sets and then choosethe case under which they fall, and then we calculate theequality on the basis of the obtained case.

It is easier to determine the equality of crisp sets than thefuzzy sets. Two crisp sets are equal if and only if they havethe same elements.

This can be defined rigorously as:

S = T ⇔ (∀x : x ∈ S ⇔ x ∈ T ), (9)

here S and T are both sets.However, in fuzzy sets, each element belongs to the set

with a certain membership degree whose value ranges from0 to 1. Hence to calculate the fuzzy equality for two type-2fuzzy sets we make use of fuzzy equality operator (Tayal,et al. [48]).

The Appendix demonstrates the computation of equalityusing the mentioned equality operator (Tayal and Saxena,2007 [47, 48]) which is based on fuzzy functions given bySostak, 1985 [45] and Demirci, 1999 [46].

3 Proposed approach

In our approach, we have proposed an algorithm which aimsat creating the expertise sets of the reviewers which rep-resent their proficiency in varied domains. This algorithmtakes into account an exhaustive list of factors based onwhich we construct the type-2 fuzzy sets of expertise ofthe reviewers, as further discussed in Sect. 3.1. Secondly,we have proposed a stepwise method, which makes use ofthese type-2 fuzzy sets for assigning proposals to appropri-ate reviewers. This assignment focuses on the calculation offuzzy equality between the set of keywords representing theproposal and the type-2 fuzzy set of reviewer’s expertise, asexplained in Sect. 3.2.

3.1 Expertise evaluation

Experts always play a very crucial role in R&D project se-lection because their opinions have a great influence on theoutcome of the project selection. It is understood that an ex-pert with a high expertise level in a particular field will makebetter judgment regarding the projects related to that field.Therefore, the correct evaluation of the expertise level of ex-perts plays a very critical role in guaranteeing the selectionof good R&D projects (Sun et al., 2008 [27]).

Here, we propose an algorithm to evaluate the expertiseof the reviewers in different domains. The factors on the

basis of which expert evaluation is done in our approachhave never been considered in such an exhaustive manner inthe past. Using these factors, we have finally created type-2 fuzzy sets to represent the expertise level of the variousReviewers in the different domains with the help of Ex-pert Opinions. There are several well-known methods avail-able for creating fuzzy sets like Expert Opinions, Inference,Rank Ordering, Angular Fuzzy Sets, Neural Networks, Ge-netic Algorithms and Inductive Reasoning (Yen and Lan-gari, 1999 [51]; Sivanandam and Deepa, 2008 [52]). In ourapproach we have used the Expert Opinions to decide thefactors that might affect the expertise of a Reviewer and thencreate appropriate type-2 fuzzy sets for the same.

In order to shortlist the factors on the basis of whichone can potentially evaluate an expert, we conducted a sur-vey in which selected renowned professors and researchersfrom esteemed Universities and Institutes like Indian Insti-tute of Technology, Jawaharlal Nehru University, Guru Gob-ind Singh Indraprastha University, Delhi University, IndiraGandhi Institute of Technology were queried about the pa-rameters to be taken to represent the Expertise of a Reviewerand their relative weights. The questionnaire developed forthe survey consisted of a list of relevant criterions (parame-ters) which represented the technical aptitude and the overallexpertise of a reviewer in any domain. Every participant ofthe survey was required to rate all the factors with a valueranging between 0 and 1. This value represented the relativeimportance of each criterion as per the Expert’s own per-ception and understanding. The final weights for all the cri-terions were calculated by taking the average of the valuesassigned by each Expert. Based upon the Experts’ opinions,the final list of factors for Expert Evaluation was obtainedalong with their respective weights.

The following relevant factors were uncovered throughExperts’ opinion, to represent the expertise of reviewers:

(i) The number of patents (denoted as ‘p’) registered un-der the name of a certain reviewer represents his inno-vation, novelty and research work done in that partic-ular field. This is of utmost relevance as it symbolisesthe highest cadre of research work in Science and Tech-nology.

(ii) Another important aspect is the PhD Thesis of a re-viewer (no. of PhD Thesis denoted as ‘t’), which standsfor acquisition and dissemination of new knowledge.Thesis in a particular domain embodies the years ofhard work and research of a reviewer in that domainand thus, signifies his/her Expertise in that area. Hence,it holds high importance while calculating the expertiseof the reviewer. We have also considered the qualifica-tion of an expert in terms of the number of Bachelorsor Masters Degrees he/she holds which represents howlearned an expert is.


(iii) The number of publications in a particular field (de-noted as ‘n’) is another factor for evaluating the exper-tise. Experts are often judged on the basis of their pub-lished work in academic journals and conferences. Thenumber of publications an expert holds in a particulardomain area represents his foothold and understandingof that domain. In addition to the number of publica-tions, the quality of each publication also needs to betaken into account when the experts are evaluated.

The quality of each publication can be assessed by focussingon the varied factors. The quality of the journal (denotedas ‘J ’) or conference (denoted as ‘C’) depends on whetherit is paid (denoted as ‘P ’) or non-paid (denoted as ‘NP’).As a general notion, the research work published in non-paid Journals or Conferences is considered to be of higherstature as compared to that published in Paid Journals orConferences.

The impact factor (denoted as ‘I ’) is the most signifi-cant of all the factors and is used to compare different jour-nals within a certain field. The higher the impact factor, thehigher the importance of the journal is. Therefore a publica-tion in a journal with higher impact factor implies a betterquality of publication.

Undoubtedly the publications in journals hold higher im-portance than the publications in conferences. Journals areusually peer-reviewed, which means that the papers pub-lished in journals are carefully evaluated for errors and pos-sibly rewritten a few times. However a comparatively lessrigorous peer review is done in conference papers. Thereforea publication in journal implies a higher standard of publi-cation when compared to a publication in a conference.

Refereed materials (denotes ad ‘R’) are publications re-viewed by “expert readers” or referees prior to the publica-tion of the material. After reading and evaluating the mate-rial, the referee informs the publisher if the document shouldbe published or if any changes should be made prior to pub-lication. Refereed materials assure that the information con-veyed is reliable and timely. In a non-refereed publication(denoted as ‘NR’) less rigorous standards of screening arefollowed prior to publication as compared to refereed publi-cations. Therefore getting one’s work published in a refereedpublication denotes a higher degree of weightage.

A citation (denoted as ‘c’) is a specific reference to meri-torious work done in the original publication. The number oftimes a paper is cited represents the significance of that par-ticular publication and the acknowledgment it has receivedfrom its peers. All these aspects of the quality of publica-tion have to be taken into account for each publication ofthe expert.

Number of books (denoted as ‘b’) authored by an expertis an indication of his/her intellect in a particular field. Num-ber of papers previously reviewed as an expert (denoted by‘r’) denotes his/her experience in reviewing similar projects

Table 1 Weights for each criterion

S No. Criteria Variable Weight

1 No. of patents p 0.92

2 PHD Thesis t 0.75

3 No. of publications n 0.85

4 Qualification (Degrees)

Bachelors d 0.26

Masters m 0.53

5 No. of books written b 0.38

6 Quality factors for eachpublication

Journal J 0.75

Conference C 0.32

Impact Factor I 0.95

Paid journal/conference P 0.37

Non-paid journal/conference NP 0.73

Refereed Journal/Conference R 0.62

Non-RefereedJournal/Conference

NR 0.34

No of citations c 0.80

7 Time elapsed since mostrecent publication

Table 2 Displayedseparately

8 No. of papers previouslyreviewed

r 0.73

9 Expert Lectures E 0.28

Table 2 Weights for elapsed time since the most recent publication

Time elapsedsince most recentpublication

0–2 yrs 2–5 yrs 5–10 yrs 10–15 yrs >15 yrs

Weight 0.60 0.55 0.42 0.35 0.25

in such similar domains. An expert with more experiencetends to be a better judge of the research proposal.

The time elapsed since the most recent publication (de-noted as ‘T ’) is a reflection of whether the expert is an ac-tive researcher of the field or not. An active researcher hasa higher expertise for reviewing the proposals as comparedto a passive one. Accordingly, different weights have beenassigned for the calculation of the expertise as given in Ta-ble 2. Expert Lectures conducted by an expert (denoted as‘E’) are an implication of his knowledge of the subject.

All the above factors are to be considered for finding theexpertise of the reviewer in one particular domain. For cre-ating the complete set of expertise for an expert, we have tocalculate the expertise for each domain separately.

Table 1 shows the weights assigned to each factor. Theseweights have been computed using the results of the Ex-perts’ opinion, as discussed in the beginning of Sect. 3.1.

Now we propose an algorithm to develop the Expertiseof each reviewer in every domain.

D.K. Tayal et al.

Table 3 Algorithm for expert evaluation

For i = 1 to i = nr /* ‘nr’ is the number of Reviewers */

For each di ∈ Reviewer_Domain, calculate his Expertise

Step 1:

Set w1 = 0.92 ∗ p /* ‘p’ is the no. of patents of Reviewers in the domain D*/

Step 2:

Set w2 = 0.75 ∗ t /* ‘t’ is the number of PHD Thesis */

Step 3:

Set w3 = 0.85 ∗ n /* ‘n’ is the number of publications */

Step 4:

Set w4 = (0.26 ∗ d + 0.53 ∗ m)/(d + m) /* ‘d’ is the number of Bachelors degree and

‘m’ represents number of Masters degree */

Step 5:

Set w5 = 0.38 ∗ b /* ‘b’ is the number of books published */

Step 6:

For j = 1 to j = n /* ‘n’ denotes the no. of publications */

/* Calculate the Quality of publication for j th publication of the ith Expert */

/* Set S1 = (0.75 ∗ J + 0.32 ∗ C + 0.95 ∗ I + 0.73P + 0.37 ∗ NP + 0.62 ∗ R + 0.34NR + 0.8 ∗ c)/(I + c + 3)/*

Where,

J = 1 and C = 0 if publication corresponds to the journal,

J = 0 and C = 1 if publication corresponds to the conference,

I represents the Impact Factor,

P = 1 and NP = 0 if publication corresponds to the paid journal/conference,

P = 0 and NP = 1 if publication corresponds to the paid journal/conference,

R = 1 and NR = 0 if publication corresponds to the Refereed journal/conference,

R = 0 and NR = 1 if publication corresponds to the Non-Refereed journal/conference,

c represents the number of citations

*/

/* End of for loop for ‘j ’ */

Set w6 = (S1 + S2 + S3 + · · · + Sn)/n

/* ‘n’ is the number of publications, ‘w6’ represent the weight for quality */

Step 7:

Set w7 from the appropriate value in Table 2

/* For this appropriate weight is selected from Table 2; w7 represents the time elapsed since the most recent publication */

Step 8:

Set w8 = 0.73 ∗ r /* ‘r’ is the number of papers previously reviewed */

Step 9:

Set w9 = 0.28 ∗ e /* ‘e’ is the number of Expert Lectures given */

/* End of for loop for all the domains di of the reviewer */

Step 10:

Return (E = (w1 + w2 + · · · + w9)/(p + t + n + b + r + e + 3))

/* where ‘E’ represents the final Expertise value */

/* End of for loop for ‘i’ */

At each step, the algorithm calculates the weights corre-sponding to each factor by multiplying the weight (relativeimportance) of the factor by the value of the factor itself.The algorithm is presented in Table 3.

3.2 Reviewer assignment

In our approach, we have proposed a new method for the Re-viewer Assignment Problem using type-2 fuzzy sets to rep-


resent the expertise levels of the various reviewers in the dif-ferent domains. In addition to this, we also consider the setof keywords relating to each proposal as another fuzzy set.We then compute the fuzzy equality between the two sets byusing the fuzzy equality operator as proposed by Tayal andSaxena, 2007 [47, 48].

In our method, we propose two approaches for calculat-ing the matching degree between the reviewers and the key-word sets:

(1) In the first approach, we consider the set of keywordsrepresenting a proposal as a classical set. We set themembership degree of each keyword in this set as 1.Thus, it becomes a special case of a fuzzy set wherethe membership degree of the element is taken as 1. Wethen calculate the equality between the expertise sets ofreviewers and the keyword set by using the equality op-erator defined in Sect. 2.2.

(2) In the Extended approach, the set of keywords is takenas a fuzzy set. Here the applicant is made to enter threekeywords that best represent the proposal. He is alsomade to enter the relative weights that represent thepercentage (between 0 and 100) by which a particularkeyword can individually represent the proposal. In thiscase, using these weights, the fuzzy set of keywords iscreated. The rest of the procedure, however, remains thesame. Next we compute the equality between the ex-pertise sets of reviewers and the keyword sets using theequality operator described in Sect. 2.2.

Using a fuzzy set to represent the set of Reviewers and theset of Keywords provides a strong mathematical foundationfor representation of imprecision exhibited in the expertiselevel of Reviewers. None of the approaches in the litera-ture deals with the imprecision in the Reviewer AssignmentProblem. To capture this imprecision, we represent the ex-pertise level of the various reviewers in the various domainsas type-2 fuzzy sets. In this paper, we also propose an inte-grated solution consisting of an exhaustive set of constraintsassociated with the assignment. Also, through this model,we have tried to optimize the assignment procedure, takinginto account multiple aspects of the problem such as balanc-ing the workload of reviewers, avoiding conflicts of interest,incorporating bidding for proposals by reviewers and con-sidering multiple keywords. Finally, we formulate the Re-viewer Assignment Problem as follows.

Let D be the set of m domains {d1, d2, d3, . . . , dm} whichrepresent the expertise areas of reviewers and the key-words of the proposals. Here, each domain area can fur-ther be divided into q sub-domains denoted by alphabetsa, b, c, . . . , q as {{d1a, d1b, . . . , d1q}, {d2a, d2b, . . . , d2q}, . . . ,{dma, dmb, . . . , dmq}} with each sub-domain being a rele-vant and concentrated area within the relatively wider corre-sponding domain.

We assume that the set of keywords K ⊆ ρ, where ρ isthe power set of D.

Let R denote the set of n reviewers {R1,R2,R3, . . . ,Rn},where R1, . . . ,Rn are type-2 fuzzy sets representing the ex-pertise levels of the reviewers in the various domains. Letthe expertise level of Reviewer R in ith domain di be repre-sented as αi and the expertise in sub-domain j of domain l

i.e. dlj be represented as βlj .Let us discuss the expertise set for Reviewer 1. Suppose

R1 possesses some expertise in domains d1, d2, d3 and d4.Let the 1st domain, d1 be atomic, the 2nd domain be fur-ther divided into three sub-domains, d2a, d2b, d2c . Similarly,let the 3rd domain d3 be atomic and the 4th domain be fur-ther sub-divided into 4 domains, d4a, d4b, d4c, d4d . Now, theexpertise set for 1st Reviewer, R1 can be represented as:

R1 = {{α1/d1} + {β2a/d2a + β2b/d2b + β2c/d2c} + {α3/d3}+ {β4a/d4a + β4b/d4b + β4c/d4c + β4d/d4d}}

Similarly, we can represent the expertise levels of all review-ers in the form of type-2 fuzzy sets, R1,R2,R3, . . . ,Rn.

Let there be p keywords representing the proposal,k1, k2, . . . , kp . There are two approaches to represent thefuzzy set of keywords representing the proposal:

(1) In the first approach, we represent this as a fuzzy set K ,with each keyword belonging to K with a membershipdegree of 1. Thus we represent the fuzzy set K as:

K = {{1/k1} + {1/k2} + · · · + {1/kp}}

Or in the form of a set of keywords K = {k1, k2, . . . , kp}.(2) In the Extended Approach (discussed in Sect. 3.2), we

ask the applicant of the proposal to provide us the mem-bership values of each Keyword with respect to his pro-posal. The membership of each keyword can be easilycalculated simply by judging to what degree the pro-posal is devoted to the keyword ki . Let these mem-bership degrees be w1,w2, . . . ,wp for the keywordsk1, k2, . . . , kp respectively. Then we can represent theset of keywords as a fuzzy set as:

K = {{w1/k1} + {w2/k2} + · · · + {wp/kp}}

Now we apply the fuzzy equality operator (Tayal and Sax-ena, 2007 [47, 48]) on these sets of Expertise level of n re-viewers, R1,R2,R3, . . . ,Rn and the set of keywords repre-senting the areas in the proposal, K .

Also, all the reviewers are made to bid for the proposalsthat they find interesting. Every reviewer is made to rate eachproposal in the following 3 grades:

1 �⇒ Okay but preferably not2 �⇒ Willing to review3 �⇒ Eager to review

The bidding grades are also considered during the re-viewer assignment procedure in order to ensure increased

D.K. Tayal et al.

Table 4 A new method for assignment of reviewers to proposals

Step 1:For each reviewer Ri , calculate the fuzzy equality Ei of the expertise set of each reviewer with the set of keywords representing the areas ofthe submitted proposal, ∼=∗(Ri,K) where i ∈ {1,2, . . . , n}. Let E1 be the fuzzy equality between R1 and K , E2 between R2 and K, . . . and En

between Rn and K .

Step 2:Out of these values of equalities obtained in E1, . . . ,En select the ones that are greater than zero. After this step, let there be a set of R′shortlisted reviewers each having Ei > 0.

R′ ⊆ R

If there does not appear any Ri with Ri > 0, then the matter is immediately reported to the Directorial Board for further decisions and thecurrent Method is terminated.

Step 3:Any Conflict of Interest between the authors of the paper and the corresponding Reviewer for the paper needs to be prevented in order to avoidany sort of bias that might hamper the decision of the reviewer regarding the paper. For this purpose, a database of all the Reviewers is stored,which contains all the necessary details of their past associations or affiliations to any organisation. The database also stores a list of all thepublications of the reviewer till date along with the details of each publication like the co-authors, the affiliated institution, the name of theJournal/Conference etc. The details of the authors of the paper are then cross-checked with this database to avoid any kind of Conflict of Interestthat might arise. Consider the following constraints to avoid Conflicts of Interest.

(a) The reviewer and the author should not be affiliated to the same University or Organisation currently or at any time in the past.(b) The reviewer and the author should not have been co-authors to a paper at any point of time in the past.(c) There should not have been a student-teacher relationship between the reviewer and the author at any point of time in the past.(d) The reviewer and the author should not have been colleagues at any point of time in the past.

Let R′′ = {R1,R2,R3, . . . ,Rt } be a set of reviewers that satisfy any of the conditions in (a) to (d). Then these reviewers need to be removedfrom shortlisted list of reviewers, i.e. we obtain R′′′ = R′ − R′′ = {R1,R2,R3, . . . ,Ru}.Step 4:Sort the elements in R′′′ obtained in Step 3 in the decreasing order of their matching degree of the fuzzy equality obtained in step 1. Let this setbe R′′′

ordered.For the reviewers with the same matching degree for a proposal, sort them in the decreasing order of the preference grades assigned by themtowards the proposal.

Step 5:Let the maximum number of proposals that can be assigned to a reviewer be “Max”. Then, starting from the first element of the set as obtainedin Step 4, select the reviewer and verify the following for each reviewer, Ri , belonging to the new set of reviewers R′′′ as obtained in Step 3 fori ∈ (1, u)

LRi≥ Max

If the above inequality is satisfied for any Ri then obtain R′′′′ = R′′′ordered − Ri . Repeat Step 5 for the next reviewer. Select three reviewers from

the top of the set R′′′′.

Step 6:These reviewers obtained in Step 5 are the appropriate reviewers to review the required proposal most efficiently. Update the number ofproposals assigned for the selected reviewers. However, if the list exhausts and three reviewers have not been attained, then report the matter tothe Directorial Board immediately. The final set of reviewers obtained is the optimal solution to the problem.

satisfaction of reviewers regarding the proposals they are as-signed.

We propose the following method to compute the match-ing degree between the reviewers’ expertise level and thekeyword list of the proposal and to optimize the assignmentby considering the mentioned constraints.

Also, for each reviewer Ri , we define a quantity LRifor

i = 1 to n (i.e. total number of reviewers), where

0 ≤ LRi≤ 5

Which denotes the number of proposals currently assignedto Ri . LRi

will also be considered while finally assigningthe proposal to Reviewer Ri .

Now we develop a new stepwise method to assign Re-viewers to the proposal submitted. We give the three mostappropriate keywords that can represent the submitted paperas the input and we receive the three most appropriate re-viewers who can best review the paper as the output. Thismethod is given in Table 4.

For the method given in Table 4, the complexity can becomputed as follows.

In Step 1, the fuzzy equality Ei of the expertise set ofeach Reviewer Ri can be calculated with complexity O(n).The Reviewers with Equality greater than zero (denotedas R′) can be shortlisted with again a complexity of O(n).The Conflicts of Interest can be eliminated in Step 3 with


a complexity of O(n). The list of reviewers in Step 4 canbe sorted with a complexity of O(n logn) using any well-known sorting algorithm. Workload balancing in Step 5 canbe performed with a complexity of O(n). Hence, the over-all complexity of the proposed algorithm comes out to beO(n logn). This method has been implemented in a WebApplication developed by us as discussed in Sect. 4. Theresults obtained from the Expert System have been verifiedand are fast enough for the real world application.

Method given in Table 4 has also been summarised in theform of a flow chart in Fig. 1.

Fig. 1 Flowchart for the Proposed Method

3.2.1 Example for proposed method

Example 1—keyword set as a classical set Let us firstshow an example to demonstrate the proposed method ofTable 4 on a small dataset of 10 reviewers and a set of 5domains, d1, d2, d3, d4, d5, where d2 and d5 can be furtherdivided into 3 sub-domains and 2 sub-domains respectivelyviz. d2a, d2b, d2c and d5a, d5b .

Now let the expertise sets of the 10 reviewers, Ri , fori = 1 to 10, be represented as the type-2 fuzzy sets givenbelow:

R1 = {{0.3/d1} + {0.2/d2a + 0.6/d2b + 0.1/d2c}+ {0.5/d3} + {0.7/d4} + {0.2/d5a + 0.6/d5b}

}

R2 = {{0.6/d1} + {0.4/d3} + {0.5/d5a + 0.7/d5b}}

R3 = {{0.2/d3} + {0.8/d4}}

R4 = {{0.8/d1} + {0.6/d3} + {0.6/d5a + 0.7/d5b}}

R5 = {{0.3/d1} + {0.4/d2a + 0.5/d2b + 0.6/d2c}}

R6 = {{0.3/d2a + 0.5/d2b + 0.4/d2c} + {0.6/d4}}

R7 = {{0.8/d3} + {0.7/d5a + 0.9/d5b}}

R8 = {{0.4/d1} + {0.4/d3} + {0.5/d5a + 0.4/d5b}}

R9 = {{0.3/d2a + 0.4/d2b + 0.9/d2c}}

R10 = {{0.8/d3} + {0.7/d5a + 0.9/d5b}}

Let the keywords in the submitted proposal be d3, d5a andd5b , then the fuzzy set of keywords i.e. K can be representedas:

K = {{1/d3} + {1/d5a} + {1/d5b}}

Now we perform the steps of the above mentioned methodto find the optimal assignment of reviewers for reviewingthis proposal.

Step 1 In this step, the fuzzy equality of the first reviewerR1 with K , given as Edom(R1,K) is calculated.

Here we know that the domain of keywords, dom ={d1, d21, d22, d23, d3, d4, d51, d52}. Since,

R1 = {{0.3/d1} + {0.2/d2a + 0.6/d2b + 0.1/d2c}+ {0.5/d3} + {0.7/d4} + {0.2/d5a + 0.6/d5b}

}

Hence the membership function for Reviewer R1 can be rep-resented as:

μR1(x) =

⎧⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎨

⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎩

0.3 if x = d1

0.2 if x = d2a

0.6 if x = d2b

0.1 if x = d2c

0.5 if x = d3

0.7 if x = d4

0.2 if x = d5a

0.6 if x = d5b

D.K. Tayal et al.

Similarly, the membership function for K can be repre-sented as:

μK(x) =⎧⎨

⎩

1 if x = d3

1 if x = d5a

1 if x = d5b

Now, first we compute I (R1,K) = {x ∈ dom : μR1(x) ≤μK(x)} which is calculated as explained in the Appendix.

For

x = d1 : μR1(d1) = 0.3 and μK(d1) = 0

so

μR1(x) � μK(x) for x = d1

Similarly, we calculate for x = d2a , x = d2b , x = d2c, x =d3, x = d4, x = d5a , x = d5b .

Thus we obtain I (R1,K) = {x = d3, x = d5a, x = d5b :μR1(x) ≤ μK(x)} = dom − {d1, d2a, d2b, d2c, d4}.

Now[⊆̃(R1,K)

] =∧

dom

{Rc

1 ∨ K(x)}

=∧

dom

{((1 − R1) ∨ K

)(x)

}

=∧

dom

{(1 − μR1(x)

) ∨ μK(x)}

for

x = d1 : {(1 − μR1(d1)) ∨ μK(d1)

} = {(1 − 0.3) ∨ 0

}

= (0.7 ∨ 0) = 0.7

Similar values can be calculated for x = d2a , x = d2b ,x = d2c, x = d3, x = d4, x = d5a and x = d5b as 0.8, 0.4,0.9, 1, 0.3, 1 and 1 respectively.

Therefore[⊆̃(R1,K)

] = 0.7 ∧ 0.8 ∧ 0.4 ∧ 0.9 ∧ 1 ∧ 0.3 ∧ 1 ∧ 1 = 0.3

Similarly,[⊆̃(K,R1)

] = 0.2

Now by the definition of

{χI (R1,K)(x)

} ={

1 if R1 ⊆ K

0 if R1 ⊂ K

Therefore∧

dom

{χI (R1,K)(x)

} =∧

dom

{χ{x∈dom:μR1 (x)≤μK(x)}(x)

}

which is given by

= {χ{x∈dom:μR1 (d1)≤μK(d1)}(x = d1)

}

∧ {χ{x∈dom:μR1 (d2a)≤μK(d2a)}(x = d2a)

}

∧ {χ{x∈dom:μR1 (d2b)≤μK(d2b)}(x = d2b)

}

∧ {χ{x∈dom:μR1 (d2c)≤μK(d2c)}(x = d2c)

}

∧ {χ{x∈dom:μR1 (d3)≤μK(d3)}(x = d3)

}

∧ {χ{x∈dom:μR1 (d4)≤μK(d4)}(x = d4)

}

∧ {χ{x∈dom:μR1 (d5a)≤μK(d5a)}(x = d5a)

}

∧ {χ{x∈dom:μR1 (d5b)≤μK(d5b)}(x = d5b)

}

= 0 ∧ 0 ∧ 0 ∧ 0 ∧ 1 ∧ 0 ∧ 1 ∧ 1 = 0

Similarly,∧

dom

{χI (K, R1)(x)

} = 0

Hence

[⊆̃∗(R1,K)] = [⊆̃(R1,K)

] ∨(∧

dom

{χI (R1,K)(x)

})

= 0.3 ∨ 0 = 0.3

Similarly

[⊆̃∗(K,R1)] = [⊆̃(K,R1)

] ∨(∧

dom

{χI (K,R1)(x)

})

= 0.2 ∨ 0 = 0.2

Therefore,

Edom(R1,K) = ∼=∗(R1,K) = 0.3 ∧ 0.2 = 0.2

Hence the fuzzy equality between sets R1 and K = 0.2.Now let us calculate the fuzzy equality between sets R3

and K :

R3 = {{0.4/d3} + {0.8/d4}}

Hence the membership function for R1 can be representedas:

μR3(x) ={

0.2 if x = d3

0.8 if x = d4


μK(x) =⎧⎨

⎩

1 if x = d3

1 if x = d5a

1 if x = d5b

In the following calculation of fuzzy equality, we have omit-ted the repetitive calculation steps for the sake of brevity.

Now, first we compute I (R3,K) = {x ∈ dom : μR3(x) ≤μK(x)} as we have calculated in the beginning of Step 1 forI (R1,K).

We obtain the following:

I (R3,K) = {x = d3, x = d5a, x = d5b : μR3(x) ≤ μK(x)

}

= dom − {d1, d2a, d2b, d2c, d4}Now we calculate [⊆̃(R3,K)] as in the beginning of Step 1for [⊆̃(R1,K)].

We obtain the following:[⊆̃(R3,K)

] = 0.2



] = 0

Now we calculate∧

dom{χI (R3,K)(x)}, as in the beginningof Step 1 for

∧dom{χI (R1,K)(x)}.

We find∧

dom

{χI (R3,K)(x)

} = 0

Similarly,∧

dom

{χI (K,R3)(x)

} = 0

Hence

[⊆̃∗(R3,K)] = [⊆̃(R3,K)

] ∨(∧

dom

{χI (R3,K)(x)

})

= 0.2 ∨ 0 = 0.2

Similarly

[⊆̃∗(K,R3)] = [⊆̃(K,R3)

] ∨(∧

dom

{χI (K,R3)(x)

})

= 0 ∨ 0 = 0

Therefore,

Edom(R3,K) = ∼=∗(R3,K) = 0.2 ∧ 0 = 0

Hence the fuzzy equality between sets R3 and K = 0.We calculate the fuzzy equalities of the other expertise

sets with the keyword set in a similar manner. We thus getthe fuzzy equalities as:

Edom(R1,K) = 0.2

Edom(R2,K) = 0.4

Edom(R3,K) = 0

Edom(R4,K) = 0.6

Edom(R5,K) = 0

Edom(R6,K) = 0

Edom(R7,K) = 0.7

Edom(R8,K) = 0.4

Edom(R9,K) = 0

Edom(R10,K) = 0.7

Step 2 The reviewers with a matching degree greater thanzero are shortlisted. The list of shortlisted reviewers is givenby:

R′ = {R1,R2,R4,R7,R8,R10}

Table 5 Reviewers and their respective fuzzy equality & preferencegrades

Reviewer Fuzzy equality Preference grade

R10 0.7 3

R7 0.7 2

R8 0.4 2

R2 0.4 1

Step 3 Here we consider the Conflicts of Interest. Assumethat the reviewer R4 has co-authored a paper with the authorof the submitted proposal in the past. Also, suppose R1 hasthe same affiliation as the author of the submitted proposal.Hence,

R′′ = {R4,R1}R′′′ = R′ − R′′ = {R2,R7,R8,R10}

Step 4 Here we sort the set of reviewers obtained in Step 3in the decreasing order of their matching degree or the fuzzyequality. Also, for the reviewers with the same matching de-gree for a proposal, we sort them in the decreasing orderof the preference grades assigned by them towards the pro-posal. Let the preference grade of R2 be 1, R7 be 2, R8 be 2and of R10 be 3.

This is summarised in Table 5. In Table 5 the first rowrepresents that the reviewer R10 has a fuzzy equality valueof 0.7 and has a preference grade of 3 towards the proposal,which is the highest value and implies that the reviewer iseager to review this proposal. Similar explanation hold truefor the rest of the rows of the table.

Hence, the ordered set R′′′ordered = {R10,R7,R8,R2}.

Step 5 The maximum number of proposals that can be as-signed to a reviewer is 5 i.e. Max = 5. Now, ∀i, i = 1 to 4verify whether

LRi≥ 5

If it is true, then Ri is removed from the set and the sameconstraint is checked for the other reviewers. Here, supposethat we observe that R7 has already been assigned 5 propos-als i.e. LR7 = 5. So R7 is removed from the list. The rest ofthe reviewers satisfy all constraints and hence are the appro-priate reviewers, i.e.:

R′′′′ = R′′′ordered − R7.

Hence,

R′′′′ = {R10,R8,R2}Therefore, R10,R8 and R2 are the most appropriate review-ers who must be assigned to the submitted proposal and whomay be able to provide the best judgement over the proposal.

Observe that R10 with equality value 0.7 and preferencegrade 3 is, thus, the best possible match for the proposal.

D.K. Tayal et al.

3.3 Extended approach

As an extension to our basic approach, we generalise the setof keywords as a fuzzy set. By representing the sets of key-words by a fuzzy set, we capture the relative importance ofeach keyword in the proposal. The author of the proposal cannow specify which keyword is more important in his pro-posal in comparison to the other keywords and with whatmembership degree. Note that the previous approach dis-cussed above considers the set of keywords as a crisp set,thus it assigns equal importance to all the keywords. We be-lieve that the proposals may combine two or more signifi-cant but totally different thrust areas of research by propos-ing a hybridisation of approaches, where the two approachesmay hold with different degrees of relevance in the proposal.Moreover, in this approach, we can appropriately assign theproposal to a reviewer who holds a comparatively higher ex-pertise level in the domain with greater relevance. This en-ables us to capture multiple aspects related to the proposalkeywords efficiently and thus, make suitable reviewer as-signments.

Here the applicant is asked to provide ‘p’ keywords thatbest represent the proposal along with their respective de-gree of belongingness (i.e. degree of membership) in theproposal. The set of keywords is therefore represented as atype-1 fuzzy set. The rest of the procedure remains the same.Next, we compute the equality between the expertise sets ofreviewers and the keyword sets using the equality operatordescribed in Sect. 4.

For demonstration of the Extended Approach, let us as-sume that the applicant provides the ‘p’ keywords k1, k2, . . . ,

kp with their membership degrees w1,w2, . . . ,wp respec-tively. Then we can represent the set of keywords as a fuzzyset as:

K = {{w1/k1} + {w2/k2} + · · · + {wp/kp}}

We now elicit an example to demonstrate the computationof fuzzy equality with these types of keyword sets.

Example 2 Let us again consider the same dataset (asused in Example 1) of 10 reviewers and a set of 5 do-mains, d1, d2, d3, d4, d5, where d2 and d5 can be further di-vided into 3 sub-domains and 2 sub-domains respectively asd2a, d2b, d2c and d5a, d5b .

Further suppose that the proposal contains three key-words d5b, d3 and d5a and their fuzzy sets of keywords isrepresented as:

K = {{0.3/d3} + {0.7/d5a} + {0.8/d5b}}

Now we apply the method proposed in Sect. 3.1 to find theoptimal assignment of reviewers.

Let us first compute the calculation of fuzzy equality forR1 with K given as Edom(R1,K).

Here, the domain of keywords is given by dom ={d1, d2a, d2b, d2c, d3, d4, d5a, d5b}. Again assume that,

R1 = {{0.3/d1} + {0.2/d2a + 0.6/d2b + 0.1/d2c}+ {0.5/d3} + {0.7/d4} + {0.2/d5a + 0.6/d5b}

}

Hence the membership function for R1 can be representedas:

μR1(x) =

⎧⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎨

⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎩

0.3 if x = d1

0.2 if x = d2a

0.6 if x = d2b

0.1 if x = d2c

0.5 if x = d3

0.7 if x = d4

0.2 if x = d5a

0.6 if x = d5b


μK(x) =⎧⎨

⎩

0.3 if x = d3

0.7 if x = d5a

0.8 if x = d5b

In the following computation of fuzzy equality, we omit thedetailed calculation for the sake of brevity. The procedureused in the calculation in Step 1 of Example 1, Sect. 3.2 isagain followed as reiterated below:

We first compute

I (R1,K) = {x ∈ dom : μR1(x) ≤ μK(x)

}

I (R1,K) = {x = d5a, x = d5b : μR1(x) ≤ μK(x)

}

= dom − {d1, d2a, d2b, d2c, d3, d4}Next we calculate[⊆̃(R1,K)

]

= 0.7 ∧ 0.8 ∧ 0.4 ∧ 0.9 ∧ 0.5 ∧ 0.3 ∧ 0.8 ∧ 0.8 = 0.4


] = 0.3

Now we calculate∧

dom

{χI (R1,K)(x)

}

We find∧

dom

{χI (R1,K)(x)

} = 0

Similarly,∧

dom

{χI (K,R1)(x)

} = 0

Hence[⊆̃∗(R1,K)

] = [⊆̃(R1,K)] ∨

(∧

dom

{χI (R1,K)(x)

})

= 0.4 ∨ 0 = 0.4


Similarly,

[⊆̃∗(K,R1)] = [⊆̃(K,R1)

] ∨(∧

dom

{χI (K,R1)(x)

})

= 0.3 ∨ 0 = 0.3

Therefore,

Edom(R1,K) = 0.4 ∧ 0.3 = 0.3

Hence the fuzzy equality between sets R1 and K comes outto be 0.3.

In a similar manner, we calculate the fuzzy equalities ofthe other expertise sets with the keyword set. After compu-tation, we obtain the following fuzzy equalities:

Edom(R1,K) = 0.2

Edom(R2,K) = 0.4

Edom(R3,K) = 0.3

Edom(R4,K) = 0.2

Edom(R5,K) = 0.2

Edom(R6,K) = 0.2

Edom(R7,K) = 0.3

Edom(R8,K) = 0.4

Edom(R9,K) = 0.1

Edom(R10,K) = 0.3

In this way, we compute the equality between the type-2fuzzy expertise sets of reviewers and the fuzzy keyword sets.

The rest of the steps of the proposed method can be ap-plied as demonstrated in Example 1. For the sake of brevity,we are omitting the complete demonstration of our methodfor the fuzzy set of keywords, which is otherwise a simpleprocess.

3.4 Evaluation of the proposed approach

As we have discussed earlier, none of the approaches pro-posed so far have been able to capture the imprecision that isinherent in the Reviewer Assignment Problem. In all the ap-proaches proposed so far, the expertise of reviewers in theirrespective domains has often been considered as a crisp set.This appears to be misleading as it is not very convenientto determine the exact expertise level of any reviewer in aparticular domain. We have also seen that the expertise levelof any reviewer may vary with the different domains andtheir sub-domains under consideration. In our approach, wehave offered a solution to all these problems that have beenignored in the past. Our model makes use of fuzzy sets torepresent the imprecision present in the expertise sets of re-viewers. In addition to this, we have also proposed an al-gorithm to determine the expertise of reviewers in each do-main on the basis of a fairly exhaustive list of factors that

may affect the expertise of the Reviewer. As an extension,we have also considered the relative importance of each key-word with respect to the submitted proposal using fuzzy setsfor the purpose. Through this paper, we have proposed anintegrated solution comprised of all the different aspects ofexpertise modelling and reviewer assignment that have beenconsidered in isolation in the previous studies. Four impor-tant aspects have been considered here: workload balancingof reviewers, avoiding Conflicts of Interest (COI), consid-ering individual preferences by incorporating bidding andmapping multiple keywords of a proposal. Hence, our ap-proach has been capable of capturing the long ignored im-preciseness or uncertainty related with the RAP problem,while also taking into consideration the different aspects ofexpertise modelling.

Our model takes the keywords of a paper as the inputand returns three most appropriate Reviewers as the output.This assignment is based upon a matching degree that is cal-culated using the Algorithm and the method mentioned inTables 3 and 4 respectively. The matching degree finally ob-tained represents the degree to which a Reviewer is appro-priate for reviewing the paper. Because of the mathematicalrepresentation of the set of reviewers and set of keywordsas fuzzy sets and the computation of equality of these twosets using the standard concepts of fuzzy set theory, we areprone to find the correct match for the reviewer every time.

To evaluate the model, we have created an Expert System(discussed in detail in Sect. 4), which consists of a databaseof around 200 well known international experts from vari-ous esteemed organisations across the World. We have cre-ated type-2 fuzzy sets to represent the expertise levels of allthese experts in their various domains using the method dis-cussed in Table 3. Our method then takes the keywords ofthe paper as the input and follows the steps as mentionedin the method given in Table 4. When we run our proposedmethod on this database, it returns the three most appropri-ate Reviewers in decreasing order of their matching degreeswith the paper. These reviewers are best suited to review thepaper since the profiles of these Reviewers, as stored in thedatabase, are consistent with the corresponding domain ar-eas of the submitted proposal. The system has been testedthoroughly using multiple queries with different domain ar-eas for submitted proposals and the results have been foundto be optimal, as the expert list was always consistent withthe domain areas of the submitted manuscript.

4 Expert system for reviewer assignment

An Expert System has been developed based on the ap-proaches given in Sect. 3 using the J2EE Java computingplatform. The Software is a standalone Web based systemand hence, is easily accessible from anywhere via the inter-net. Since the Software is Java based, it is extremely portable

D.K. Tayal et al.

and can be embedded into any Web based Application. TheExpert System completely automates the work of the hu-man expert who is responsible for assigning the appropriatereviewers to each submitted research proposal manually.

The software developed is completely GUI based andtakes three keywords of a submitted proposal as the in-put and determines the three most appropriate reviewers forthe same as the output. The Reviewers are to be shortlistedfrom the pool of reviewers available in a previously storeddatabase through which the software is internally connected.The software employs the method proposed in Table 4 todetermine the most appropriate reviewers for a paper bycomputing the matching degree between the proposal andthe reviewers by performing the steps as explained in theAppendix. The various steps of the method given in Table 4are then applied to resolve any Conflicts of Interest and bal-ance the workload of the reviewers. Finally, the three mostsuitable reviewers are selected for reviewing the paper.

The software provides three separate logins for users,encompassing the different functionalities of an author, re-viewer and administration. Our software, when executedagainst a sample input instance, provided good quality re-sults in terms of the three most appropriate reviewers for theinput proposal. This can be verified by observing the match-ing degree of the reviewers with the proposal keywords,which in such cases, is the maximum amongst all other re-viewers. We also observe that the reviewers selected by thesoftware, not only had the highest matching degree, but alsohad their workload balanced against the set of papers to bereviewed. Any Conflict of Interest was effectively removed.Also, all special cases were handled well by the software us-ing dedicated system messages. No system interrupts wereencountered while executing the software.

We have provided here two snapshots of a running in-stance of the software in the form of Figs. 2 and 3. Figure 2represents the page of the software in which the author isrequired to submit the relevant details regarding his/her pa-per. This includes the details about the Corresponding Au-thor and the three keywords representing the paper. Figure 3then displays the three most appropriate reviewers selectedby the software for reviewing the paper along with their fi-nal matching degree with the proposal calculated using themethod proposed in Table 4.

As can be observed from the snapshots, we had submitteda research paper with the keywords as—Feed-Forward Neu-ral Network, Back Propagation Training Algorithm and Ar-tificial Neural Networks. The Software then automaticallyassigned the appropriate reviewers with the highest match-ing degrees (0.79, 0.78 and 0.7 respectively) to this paperby employing the various steps of the method explained inTable 4. Hence, the software is capable of assigning appro-priate reviewers efficiently according to the keywords of apaper.

5 Conclusion

In this paper, we have proposed a new method for solvingreviewer assignment problem using type-2 fuzzy sets to rep-resent the expertise levels of the various reviewers in the dif-ferent domains. We observed that the expertise of reviewersin their respective domains had long been considered as acrisp set which is actually misleading, as it is not possible todetermine the exact expertise level of a particular reviewerin a particular domain. We have modelled this imprecision inthe problem by assigning for each reviewer a different levelof expertise in different domains and representing it in theform of type-2 fuzzy sets.

In our approach, we have proposed an algorithm whichcalculates the expertise level of each reviewer in differentdomains. This makes use of the various decisive weightsassigned to the various important factors that affect a re-viewer’s expertise. Through this algorithm, we finally ob-tain the type-2 fuzzy sets representing the expertise of thereviewers.

Secondly, we propose a new method to calculate thefuzzy equality between the set representing the expertiselevels of reviewers (as obtained through the first algorithm)and the set representing the keywords of a submitted pro-posal by using the fuzzy equality operator. Further, we haveoptimized the assignment by taking into account multipleaspects of the reviewer assignment problem such as balanc-ing the workload of reviewers, avoiding Conflicts of Interest,considering the preference levels of reviewers for particularproposals by incorporating bidding and mapping multiplekeywords representing the proposal. Thus, we have providedan integrated solution to the problem comprised of all thedifferent aspects of expertise modelling and reviewer assign-ment that had been considered in isolation in the previousstudies. Further, as an extension to our approach, we havealso represented the set of keywords as true fuzzy sets byconsidering the extent to which each keyword individuallyrepresents the submitted proposal using relative weights.Our solution is general in nature and can be applied to anyproblem domain. Because of the mathematical representa-tion of the set of reviewers and set of keywords and compu-tation of equality of these two sets, we are prone to find thecorrect match for the reviewer every time. The list of Re-viewers retrieved after applying our methods provides thename of the Reviewers in decreasing order of their level ofmatching after considering all the relevant constraints pro-posed in the method. Finally, the three most appropriate re-viewers are selected from this list (the ones with the highestmatching degrees) and the proposal is sent to these review-ers for review. Our work can be further extended by takingthe set of keywords to be a type-2 fuzzy set.

An expert system implementing the model proposed inthe paper has also been developed. The software takes the


Fig. 2 Screenshot for entering the details of the paper being submitted

Fig. 3 Result Screenshot: Name of the final reviewers assigned

three keywords of the paper as the input and provides thethree most appropriate reviewers for the same as the output.Based upon the strong mathematical foundation of fuzzy

logic and fuzzy sets proposed in our paper, the software wascapable of selecting the reviewers with the highest matchingdegree every time.

D.K. Tayal et al.

Appendix: Example for calculation of fuzzy equality of2 type-2 fuzzy sets

Let c1, c2 ∈ I domC be fuzzy sets.Let domC = {d1, d2, d3, d4, d5}, let the membership

function of c1 and c2 be given as follows

μc1(x) =

⎧⎪⎪⎪⎪⎨

⎪⎪⎪⎪⎩

1 if x = d1, d1 ∈ domC

0.8 if x = d2, d2 ∈ domC


0.2 if x = d4, d4 ∈ domC

0.7 if x = d5, d5 ∈ domC

μc2(x) =

⎧⎪⎪⎪⎪⎨

⎪⎪⎪⎪⎩

0.5 if x = d1, d1 ∈ domC



0.4 if x = d4, d4 ∈ domC

0.6 if x = d4, d4 ∈ domC

Now we use the equality operator discussed in Sect. 2.2 tocalculate the equality for c1 and c2 i.e. EdomC(c1, c2) as:

I (c1, c2) = {x ∈ domC : μc1(x) ≤ μc2(x)

}

which is calculated as for

x = d1 : μc1(d1) = 1 and μc2(d1) = 0.5

So

μc1(x) � μc2(x) for x = d1

For

x = d2 : μc1(d2) = 0.8 and μc2(d2) = 1

So

μc1(x) ≤ μc2(x) for x = d2

Similarly,



μc1(x) � μc2(x) for x = d5

Thus

I (c1, c2) = {x = d2, x = d3, x = d4 : μc1(x) ≤ μc2(x)

}

= domC − {d1, d5}Now[⊆̃(c1, c2)

]

=∧

domC

{cc

1 ∨ c2(x)}

(where cc1 stands for complement of c1)

=∧

domC

{((1 − c1) ∨ c2

)(x)

}

=∧

domC

{(1 − μc1(x)

) ∨ μc2(x)}

For

x = d1 : {(1 − μc1(d1)) ∨ μc2(d1)

}

= {(1 − 1) ∨ 0.5

} = (0 ∨ 0.5) = 0.5

We can calculate similar values for x = d2, x = d3, x = d4,x = d5.

Therefore[⊆̃(c1, c2)

] = 0.5 ∧ 1 ∧ 1 ∧ 0.8 ∧ 0.6 = 0.5

Similarly,[⊆̃(c2, c1)

] = 0.6

Now by the definition of

{χI (c1,c2)(x)

} ={

1 if c1 ⊆ c2

0 if c1 ⊂ c2

Therefore∧

domC

{χI (c1,c2)(x)

} =∧

domC

{χ{x∈domC:μc1 (x)≤μc2 (x)}(x)

}

which is given by

= {χ{x∈domC:μc1 (d1)≤μc2 (d1)}(x = d1)

}

∧ {χ{x∈domC:μc1 (d2)≤μc2 (d2)}(x = d2)

}

∧ {χ{x∈domC:μc1 (d3)≤μc2 (d3)}(x = d3)

}

∧ {χ{x∈domC:μc1 (d4)≤μc2 (d4)}(x = d4)

}

∧ {χ{x∈domC:μc1 (d5)≤μc2 (d5)}(x = d5)

}

= {χ{d1∈domC:1≥0.5}(x = d1)

}

∧ {χ{d2∈domC:0.8≤1}(x = d2)

}

∧ {χ{d3∈domC:0≤0}(x = d3)

}

∧ {χ{d4∈domC:0.2≤0.5}(x = d4)

}

∧ {χ{d5∈domC:0.7≥0.6}(x = d5)

}

= 0 ∧ 1 ∧ 1 ∧ 1 ∧ 0

= 0

Similarly∧

domC

{χI (c1,c2) (x)

} = 0

Hence[⊆̃∗(c1, c2)

] = [⊆̃(c1, c2)] ∨

( ∧

domC

{χI (c1,c2) (x)

})

= 0.5 ∨ 0 = 0.5

Similarly

[⊆̃∗(c2, c1)] = [⊆̃(c2, c1)

] ∨( ∧

domC

{χI (c2,c1) (x)

})

= 0.6 ∨ 0 = 0.6

So

EdomC(c1, c2) = 0.5 ∧ 0.6 = 0.5

Hence the fuzzy equality between sets c1 and c2 = 0.5.


References

1. Fan ZP, Ma J, Sun YH, Wang J (2008) A hybrid knowledgeand model approach for reviewer assignment. Expert Syst Appl34(2):817–824

2. Wang F, Chen B, Miao ZW (2008) A survey on reviewer assign-ment problem. In: 21st international conference on industrial, en-gineering and other applications of applied intelligent systems,Wroclaw, Poland

3. Dumais ST, Nielsen J (1992) Automating the assignment of sub-mitted manuscripts to reviewers. Research and development in in-formation retrieval, 233–244

4. Hettich S, Pazzani MJ (2006) Mining for proposal reviewers:lessons learned at the national science foundation. In: Proceedingsof the 12th ACM SIGKDD international conference on knowledgediscovery and data mining. ACM, Philadelphia

5. Rodriguez MA, Bollen J (2008) An algorithm to determine peer-reviewers. In: Proceeding of the 17th ACM conference on infor-mation and knowledge management, Napa Valley, CA, USA

6. Biswas HK, Hasan MM (2007) Using publications and domainknowledge to build research profiles: an application in automaticreviewer assignment. In: ICICT 2007, pp 82–86

7. Ferilli S, Di Mauro N, Basile T, Esposito F, Biba M (2006) Auto-matic topics identification for reviewer assignment. Advances inapplied artificial intelligence, pp 721–730

8. Andrew M, David M (2007) Expertise modelling for matching pa-pers with reviewers. In: Proceedings of the 13th ACM SIGKDDinternational conference on knowledge discovery and data mining.ACM, California

9. Merelo-Guervos JJ, Castillo-Valdivieso P (2004) Conference pa-per assignment using a combined greedy/evolutionary algorithm.In: Eighth international conference on parallel problem solvingfrom nature (PPSN VIII), Birmingham, England

10. Taylor CJ (2008) On the optimal assignment of conference pa-pers to reviewers. Tech report MS-CIS-08-30, Science Depart-ment, University of Pennsylvania

11. Benferhat S, Lang J (2001) Conference paper assignment. Int JIntell Syst 16:1183–1192

12. Di Mauro N, Basile TMA, Ferilli S (2005) GRAPE: an expert re-view assignment component for scientific conference managementsystems. In: Innovations in applied artificial intelligence, pp 789–798

13. Goldsmith J, Sloan R (2007) The conference paper assignmentproblem. In: Proc. AAAI workshop on preference handling forartificial intelligence

14. Kolasa T, Król D (2010) ACO-GA approach to paper-reviewer as-signment problem in CMS. Springer, Berlin, pp 360–369

15. Papagelis M, Plexousakis D, Nikolaou PN (2005) CONFIOUS:managing the electronic submission and reviewing process of sci-entific conferences. In: Sixth international conference on web in-formation systems engineering, NY, USA

16. Sun YH, Jian M, Fan ZP, Wang J (2007) A hybrid knowledge andmodel approach for reviewer assignment. In: 40th annual Hawaiiinternational conference on system sciences (HICSS)

17. Sun YH, Ma J, Fan ZP, Wang J (2008) A hybrid knowledgeand model approach for reviewer assignment. Expert Syst Appl34(2):817–824

18. Tian QJ, Ma J, Liu O (2002) A hybrid knowledge and model sys-tem for R&D project selection. Expert Syst Appl 23(3):265–271

19. Fan ZP, Chen Y, Ma J, Zhu Y (2009) Decision support for proposalgrouping: a hybrid approach using knowledge rule and genetic al-gorithm. Expert systems with applications

20. Xu Y, Ma J, Sun Y, Hao G, Xu W, Zhao D (2010) A decisionsupport approach for assigning reviewers to proposals. Expert SystAppl 37:6948–6956

21. Tian QJ, Ma J, Liang JZ, Kwok RCW, Liu O (2005) An organiza-tional decision support system for effective R&D project selection.Decis Support Syst 39(3):403–413

22. Kolasa T, Krol D (2011) A survey of algorithms for paper-reviewerassignment problem. IETE Tech Rev 28:123–134

23. Harper PR, De Senna V, Vieira IT, Shahani AK (2005) A geneticalgorithm for the project assignment problem. Comput Oper Res32(5):1255–1265

24. Es AH, Coker D (1995) On several types of degrees of fuzzy com-pactness in fuzzy topological spaces in Sostak’s sense. J FuzzyMath 3:481–491

25. Zhai C, Karimzadehgan M, Belford G (2008) Multi-aspect exper-tise matching for review assignment. In: CIKM ’08: proceedingof the 17th ACM conference on information and knowledge man-agement. ACM, New York, pp 1113–1122

26. Karimzadehgan M, Zhai C (2009) Constrained multi-aspect exper-tise matching for committee review assignment. In: Proceeding ofthe 18th ACM conference on information and knowledge manage-ment (CIKM ’09). ACM, New York, pp 1697–1700

27. Sun YH, Ma J, Fan ZP, Wang J (2008) A group decision sup-port approach to evaluate experts for R&D project selection. IEEETrans Eng Manag 55(1):158–170

28. Zadeh LA (1965) Fuzzy sets. Inf Control 8(3):338–35329. Wang C-H, Tsai C-J, Hong T-P, Tseng H-S (2003) Fuzzy inductive

learning strategies. In: Springer applied intelligence, vol 18(2), pp179–193, March 2003

30. Chiang I-J, Shieh M-J, Hsu JY-J, Wong J-M (2005) Building amedical decision support system for colon polyp screening by us-ing fuzzy classification trees. In: Springer applied intelligence, vol22(1), pp 61–75, January 2005

31. Chen S-J, Chen S-M (2005) A prioritized information fu-sion method for handling fuzzy decision-making problems. In:Springer applied intelligence, vol 22(3), pp 219–232, May 2005

32. Rasmani KA, Shen Q (2006) Data-driven fuzzy rule generationand its application for student academic performance evaluation.In: Springer applied intelligence, vol 25(3), pp 305–319, Decem-ber 2006

33. Wang HY, Chen SM (2006) New methods for evaluating answer-scripts of the students using fuzzy sets. In: Lecture notes in artifi-cial intelligence, vol 4031, pp 442–451

34. Wang HY, Chen SM (2008) Evaluating students’ answerscripts us-ing fuzzy numbers associated with degrees of confidence. In: IEEEtransactions on fuzzy systems, vol 16(2), pp 403–415

35. Wang HY, Chen SM (2009) Evaluating students’ answerscriptsbased on interval-valued fuzzy grade sheets. Expert Syst Appl,36(6):9839–9846

36. Lewis L, Buford J, Jakobson G (2009) Inferring threats in ur-ban environments with uncertain and approximate data: an agent-based approach. In: Springer applied intelligence, vol 30(3), pp220–232, June 2009

37. Aksaç A, Uzun E, Özyer T (2012) A real time traffic simulatorutilizing an adaptive fuzzy inference mechanism by tuning fuzzyparameters. In: Springer applied intelligence, vol 36(3), pp 698–720, April 2012

38. Xu C, Wang Y, Gu Y, Lin S, Ge Y (2011) Efficient fuzzy rankingqueries in uncertain databases. In: Springer applied intelligence,vol 37(1), pp 47–59, August 2011

39. Klir GJ, Yuan B (1995) Fuzzy sets and fuzzy logic: theory andapplications. Prentice Hall, New York

40. Lee CS, Wang MH, Hagras H (2012) A type-2 fuzzy ontology andits application to personal diabetic-diet recommendation. In: (SCI)IEEE transactions on fuzzy systems, vol 18(2), pp 374–395, Apr2012

41. Wagner C, Hagras H (2010) Towards general type-2 fuzzy logicsystems based on zSlices. In: IEEE transactions on fuzzy systems,vol 18(4), pp 637–660, August 2010

D.K. Tayal et al.

42. Jammeh E, Fleury M, Wagner C, Hagras H, Ghanbari M (2009)Interval type-2 fuzzy logic congestion control for video streamingacross IP networks. IEEE Trans Fuzzy Syst 17(5):1123–1142

43. Raju KVSVN, Majumdar AK (1988) Fuzzy functional depen-dencies and lossless join decompositions of fuzzy relationaldatabases. ACM Trans Database Syst 13(2):129–166

44. Own C-M (2009) Switching between type-2 fuzzy sets and in-tuitionistic fuzzy sets: an application in medical diagnosis. In:Springer applied intelligence, vol 31(3), pp 283–291, December2009

45. Sostak A (1985) On a fuzzy topological structure. In: Supp RendCirc Mat Palermo (Ser II) II, pp 89–103

46. Demirci M (1999) Fuzzy functions and their fundamental proper-ties. Fuzzy Sets Syst 106:239–246

47. Tayal DK, Saxena PC (2007) Fuzzy equi-join operator in type-1& type-2 fuzzy-relational databases. Int J Comput, Multimedia &Intel Technol, Poland 3:1–18

48. Tayal DK, Saxena PC (2007) Fuzzy join dependency in fuzzy Re-lationalDatabases. Int J Intel Technol 2(1):36–48

49. Sasaki M (1993) Fuzzy functions. Fuzzy Sets Syst 55:295–30150. Sostak A (1988) On compactness and connectedness degrees of

fuzzy sets in fuzzy topological spaces. In: General topology andits relations to modem analysis and algebra. Helderman, Berlin,pp 519–532

51. Yen J, Langari R (1999) Fuzzy logic. Prentice Hall, New York52. Sivanandam SN, Deepa SN (2008) Principles of soft computing.

Wiley, India

Devendra Kumar Tayal was bornin 1977 in Delhi, India. He hasacquired the degrees of B.Sc. (H)(Maths), M.Sc. (Maths), M.Tech.(Computer Engineering) & Ph.D.from Jawaharlal Nehru University,Delhi, India in 2006.He is currently the Head of Depart-ment (Department Of Computer En-gineering) at Indira Gandhi DelhiTechnical University for Women(Formerly Indira Gandhi Institute ofTechnology). He also worked as alecturer in University of Delhi forabout three years and as an Assis-

tant Professor in Department of Computer Engg, Jaypee Institute ofInformation Technology University, Noida, UP, India.Dr. Tayal has published several papers in Fuzzy Logic and SoftwareEngg. His areas of interest include Theory of Computation, Algo-rithms, Software Engineering., Database Management Systems, DataMining and Fuzzy Logic.

P.C. Saxena was born in Delhi, In-dia and acquired a degree of M.Sc.(Operational Research) and Ph.D.(Operational Research) from Uni-versity of Delhi in 1970. He isa member of various professionalbodies including IEEE. He is alsoa referee on the panel of variousreputed national and internationaljournals including the InternationalJournal of Fuzzy Sets and Systems.He is currently a professor andEx-Dean in School of Computerand Systems Sciences, JawaharlalNehru University, India and worked

earlier as a lecturer in University of Delhi, India. He has publishedmore than 75 papers in International Journals of high repute.Prof. Saxena is a member of various professional bodies like ComputerSociety of India, Institute of Electrical and Electronic Engineers, Indiaetc.

Ankita Sharma was born in 1990in Delhi, India. She acquired the de-gree of B.Tech. in Computer Sci-ence Engineering from IndiraGandhi Delhi Technical Universityfor Women, Kashmere Gate, Delhi-6 (Formerly Indira Gandhi Instituteof Technology), in the year 2012.She is currently working as a Soft-ware Engineer with Adobe SystemsPvt. Ltd. in Noida, India. Her areasof interests include Fuzzy Logic,Artificial Intelligence, Soft Com-puting, Human Computer Interac-tion and Software Engineering.

Garima Khanna was born in 1989in New Delhi, India. She acquiredthe degree of B.Tech. in ComputerScience Engineering from IndiraGandhi Delhi Technical Universityfor Women, Kashmere Gate, Delhi-6 (Formerly Indira Gandhi Insti-tute of Technology), in the year2012. She is currently working asa Software Engineer with AdobeSystems Pvt. Ltd. in Noida, India.Her areas of interests include FuzzyLogic, Genetic Algorithms, Neu-ral Networks, Soft Computing andDatabase Management Systems.

Shubhangi Gupta was born in 1990in Ghaziabad, India. She acquiredthe degree of B.Tech. in ComputerScience Engineering from IndiraGandhi Delhi Technical Universityfor Women, Kashmere Gate, Delhi-6 (Formerly Indira Gandhi Instituteof Technology), in the year 2012.She is currently working as a Soft-ware Engineer with SAP Labs IndiaPvt. Ltd. in Gurgaon, India. Her ar-eas of interests include Data Min-ing and Warehousing, Fuzzy Logic,Artificial Intelligence, Soft Comput-ing and Database Management Sys-tems.

new method for solving reviewer assignment problem using type-2 fuzzy sets and fuzzy functions

Documents