dedpul: difference-of-estimated-densities-based positive ... · auctions [ivanov and nesterov,...

DEDPUL Difference-of-Estimated-Densities-based Positive-Unlabeled Learning

Dmitry IvanovNational Research University Higher School of Economics

JetBrains Researchdiivanovhseru

AbstractPositive-Unlabeled (PU) learning is an analog tosupervised binary classification for the case whenonly the positive sample is clean while the nega-tive sample is contaminated with latent instancesof positive class and hence can be considered asan unlabeled mixture The objectives are to clas-sify the unlabeled sample and train an unbiased PNclassifier which generally requires to identify themixing proportions of positives and negatives firstRecently unbiased risk estimation framework hasachieved state-of-the-art performance in PU learn-ing This approach however exhibits two ma-jor bottlenecks First the mixing proportions areassumed to be identified ie known in the do-main or estimated with additional methods Sec-ond the approach relies on the classifier being aneural network In this paper we propose DED-PUL a method that solves PU Learning without theaforementioned issues1 The mechanism behindDEDPUL is to apply a computationally cheap post-processing procedure to the predictions of any clas-sifier trained to distinguish positive and unlabeleddata Instead of assuming the proportions to beidentified DEDPUL estimates them alongside withclassifying unlabeled sample Experiments showthat DEDPUL outperforms the current state-of-the-art in both proportion estimation and PU Classifi-cation

1 IntroductionPU Classification naturally emerges in numerous real-worldcases where obtaining labeled data from both classes is com-plicated We informally divide the applications into three cat-egories In the first category PU learning is an analog to bi-nary classification Here the latter could be applied but theformer accounts for the label noise and hence is more accu-rate An example is identification of disease genes Typicallythe already identified disease genes are treated as positiveand the rest unknown genes are treated as negative Instead

1Implementation of DEDPUL is available at httpsgithubcomdimonenkaDEDPUL

[Yang et al 2012] more accurately treat the unknown genesas unlabeled

In the second category PU learning is an analog to one-class classification which learns only from positive datawhile making assumptions about the negative distributionFor instance [Blanchard et al 2010] show that densitylevel set estimation methods [Vert and Vert 2006 Scott andNowak 2006] assume the negative distribution to be uniformOC-SVM [Scholkopf et al 2001] finds decision boundaryagainst the origin and OC-CNN [Oza and Patel 2018] ex-plicitly models the negative distribution as a Gaussian In-stead PU learning algorithms approximate the negative dis-tribution as the difference between the unlabeled and the pos-itive which can be especially crucial when the distributionsoverlap significantly [Scott and Blanchard 2009] Moreoverthe outcome of the one-class approach is usually an anomalymetric whereas PU learning can output unbiased probabili-ties Examples are the tasks of anomaly detection eg de-ceptive reviews [Ren et al 2014] or fraud [Nguyen et al2011]

The third category contains the cases for which neither su-pervised nor one-class classification could be applied An ex-ample is identification of corruption in Russian procurementauctions [Ivanov and Nesterov 2019] An extensive data setof the auctions is available online however it does not containany labels of corruption The proposed solution is to detectonly successful corruptioneers by treating runner-ups as fair(positive) and winners as possibly corrupted (unlabeled) par-ticipants PU learning may then detect suspicious patternsbased on the subtle differences between the winners and therunner-ups a task for which one-class approach is too insen-sitive

One of the early milestones of PU classification is the pa-per of [Elkan and Noto 2008] The paper makes two majorcontributions The first contribution is the notion of Non-Traditional Classifier (NTC) which is any classifier trainedto distinguish positive and unlabeled samples Under the Se-lected Completely At Random (SCAR) assumption whichstates that the labeling probability of any positive instanceis constant the biased predictions of NTC can be trans-formed into the unbiased posterior probabilities of being pos-itive rather than negative The second contribution is to con-sider the unlabeled data as simultaneously positive and nega-tive weighted by the opposite weights By substituting these

arX

iv1

902

0696

5v5

[cs

LG

] 7

Jun

202

0

Figure 1 DEDPUL Data Upper row (ones) represents positive sample Xp lower row (ones and zeros) represents unlabeled sample XuNon-Traditional Classifier NTC is a supervised probabilistic binary classifier trained to distinguish Xp from Xu Predictions Predicted byNTC probabilities Yp and Yu for samples Xp and Xu obtained with cross-validation NTC reduces dimensionality Densities of PredictionsEstimated probability density functions fyp(y) and fyu(y) on Yp and Yu respectively fyu(y) has two distinct peaks one of which coincideswith that of fyp(y) while the other corresponds to the negative component of the mixture Priors Posteriors The posteriors are estimated forfixed priors using the Bayes rule and are clipped The priors are estimated as the average posteriors These two steps iterate until convergenceIf the process converges trivially to 0 an alternative estimate αlowast

n is used instead

weights into the loss function a PN classifier can be learneddirectly from PU data These weights are equal to the pos-terior probabilities of being positive estimated based on thepredictions of NTC The idea of loss function reweightingwould later be adopted by the Risk Estimation framework [duPlessis et al 2014 Du Plessis et al 2015a] and its latestnon-negative modification nnPU [Kiryo et al 2017] whichis currently considered state-of-the-art The crucial insight ofthis framework is that only the mixing proportion ie theprior probability of being positive is required to train an un-biased PN classifier on PU data

While nnPU does not address mixture proportion esti-mation multiple studies explicitly focus on this problem[Sanderson and Scott 2014 Scott 2015a du Plessis et al2015b] The state-of-the-art methods are KM [Ramaswamyet al 2016] which is based on mean embeddings of pos-itive and unlabeled samples into reproducing kernel Hilbertspace and TIcE [Bekker and Davis 2018b] which tries tofind a subset of mostly positive data via decision tree induc-tion Another noteworthy method is AlphaMax [Jain et al2016] which explicitly estimates the total likelihood of thedata as a function of the proportions and seeks for the pointwhere the likelihood starts to change

This paper makes several distinct contributions Firstwe prove that NTC is a posterior-preserving transformation(Section 31) whereas NTC is only known in the litera-ture to preserve the priors [Jain et al 2016] This en-ables practical estimation of the posteriors using probabil-ity density functions of NTC predictions Independentlywe propose two alternative estimates of the mixing propor-tions (Section 32) These estimates are based on the re-lation between the priors and the average posteriors Fi-nally these new ideas are merged in DEDPUL (Difference-of-Estimated-Densities-based Positive-Unlabeled Learning)a method that simultaneously solves both Mixture Propor-tions Estimation and PU Classification outperforms state-of-the-art methods for both problems [Ramaswamy et al 2016Bekker and Davis 2018b Kiryo et al 2017] and is appli-cable to a wide range of mixture proportions and data sets(Section 5)

DEDPUL adheres to the following two-step strategy At

the first step NTC is constructed [Elkan and Noto 2008]Since NTC treats unlabeled sample as negative its predic-tions can be considered as biased posterior probabilities ofbeing positive rather than negative The second step shouldeliminate this bias At the second step the probability den-sity functions of the NTC predictions for both positive andunalbeled samples are explicitly estimated Then both theprior and the posterior probabilities of being positive are es-timated simultaneously by plugging these densities into theBayes rule and performing Expectation-Maximization (EM)In case if EM converges trivially an alternative algorithm isapplied

2 Problem Setup and BackgroundIn this section we introduce the notations and formally de-fine the problems of Mixture Proportions Estimation and PUClassification Additionally we discuss some of the commonassumptions that we use in this work in Section 22

21 PreliminariesLet random variables xp xn xu have probability densityfunctions fxp

(x) fxn(x) fxu

(x) and correspond to positivenegative and unlabeled distributions of x where x isin Rmis a vector of features Let α be the proportion of fxp

(x) infxu

(x)

fxu(x) equiv αfxp

(x) + (1minus α)fxn(x) (1)

Let pp(x) be the posterior probability that x sim fxu(x) is

sampled from fxp(x) rather than fxn

(x) Given the propor-tion α it can be computed using the Bayes rule

pp(x) equivαfxp(x)

αfxp(x) + (1minus α)fxn

(x)= α

fxp(x)

fxu(x)

(2)

22 True Proportions are UnidentifiableThe goal of Mixture Proportions Estimation is to estimatethe proportions of the mixing components in unlabeled datagiven the samples Xp and Xu from the distributions fxp

(x)and fxu

(x) respectively The problem is fundamentally ill-posed even if the distributions fxp

(x) and fxu(x) are known

Algorithm 1 DEDPUL

1 Input Unlabeled sample Xu Positive sample Xp

2 Output Priors αlowast Posteriors plowastp(x) x isin Xu3 Yu Yp = g(Xu) g(Xp) NTC predictions4 fyu(y) fyp(y) = kde(Yu) kde(Yp) kernel density estimation

5 r(y) = fyp (y)fyu (y)

y isin Yu array of density ratios assumed to be sorted

6 r(y) = MONOTONIZE(r(y) Yu) enforces partial monotonicity of the ratio curve defined in Algorithm 27 r(y) = ROLLING MEDIAN(r(y)) smooths the ratio curve defined in Algorithm 28 αlowastc p

lowastpc(y) = EM(r(y)) αlowastn p

lowastpn(y) = MAX SLOPE(r(y)) estimates priors and posteriors defined in Algorithm 2

9 Return αlowastc plowastpc(y) if (αlowastc gt 0) else αlowastn plowastpn(y)

Indeed based on these densities a valid estimate of α is any α(tilde denotes estimate) that fits the following constraint from(1)

forallx fxu(x) ge αfxp(x) (3)In other words the true proportion α is generally unidenti-

fiable [Blanchard et al 2010] as it might be any value in therange α isin [0 αlowast] However the upper bound αlowast of the rangecan be identified directly from (3)

αlowast equiv infxsimfxu (x)

fxu(x)

fxp(x)

(4)

Denote plowastp(x) as the corresponding to αlowast posteriors

plowastp(x) equiv αlowastfxp(x)

fxu(x)

(5)

To cope with unidentifiability a common practice is tomake assumptions regarding the proportions such as mutualirreducibility [Scott et al 2013] or anchor set [Scott 2015bRamaswamy et al 2016] Instead we consider estimationof the identifiable upper-bound αlowast as the objective of Mix-ture Proportions Estimation explicitly acknowledging thatthe true proportion α might be lower While the two ap-proaches are similar the distinction becomes meaningful dur-ing validation in particular on synthetic data In the first ap-proach the choice of data distributions is limited to the caseswhere the assumption holds Otherwise it might be unclearhow to quantify the degree to which the assumption fails andthus its exact effect on misestimation of the proportions Incontrast the upper-bound αlowast can always be computed usingequation (4) and taken as a target estimate We employ thistechnique in our experiments

Note that the probability that a positive instance x is la-beled is assumed to be constant This has been formulatedin the literature as Selected Completely At Random (SCAR)[Elkan and Noto 2008] An emerging alternative is to allowthe labeling probability e(x) to depend on the instance re-ferred to as Selected At Random (SAR) [Bekker and Davis2018a] However it is easy to see that the priors αlowast andthe labeling probability e(x) are not uniquely identifiable un-der SAR without additional assumptions Specifically lowfxu

(x) compared to fxp(x) can be explained by either high

e(x) or low αlowast Moreover setting e(x)1minuse(x) =

fxp (x)

fxu (x) leads

to αlowast = 1 for any densities Finding particular assumptionsthat relax SCAR while maintaining identifiability is an impor-tant problem of independent interest [Hammoudeh and Lowd2020 Shajarisales et al 2020] In this paper however we donot aim towards solving this problem Instead we propose anovel generic algorithm that assumes SCAR but can be aug-mented with less restricting assumptions in the future work

23 Non-Traditional ClassifierDefine Non-Traditional Classifier (NTC) as a function g(x)

g(x) equivfxp(x)

fxp(x) + fxu

(x)(6)

In practice NTC is a probabilistic classifier trained on thesamples Xp and Xu balanced by reweighting the loss func-tion By definition (6) the proportions (4) and the posteriors(5) can be estimated using g(x)

αlowast = infxsimfxu (x)

1minus g(x)g(x)

(7)

plowastp(x) = αlowastg(x)

1minus g(x)(8)

Directly applying (7) and (8) to the output of NTC has beenconsidered by [Elkan and Noto 2008] and is referred to asEN method Its performance is reported in Section 52

Let random variables yp yn yu have probability densityfunctions fyp(y) fyn(y) fyu(y) and correspond to the dis-tributions of y = g(x) where x sim fxp(x) x sim fxn(x) andx sim fxu

(x) respectively

3 Algorithm DevelopmentIn this section we propose DEDPUL The method is summa-rized in Algorithm 1 and is illustrated in Figure 1 while thesecondary functions are presented in Algorithm 2

In the previous section we have discussed the case of ex-plicitly known distributions fxp(x) and fxu(x) Howeveronly the samples Xp and Xu are usually available Can weuse these samples to estimate the densities fxp(x) and fxu(x)and still apply (4) and (5) Formally the answer is positiveFor proportion estimation this idea is explored in [Jain et al2016] as a baseline In practice two crucial issues may arisethe curse of dimensionality and the instability of αlowast estima-tion We attempt to fix these issues in DEDPUL

31 NTC as Dimensionality ReductionThe first issue is that density estimation is difficult in highdimensions [Liu et al 2007] and thus the dimensionalityshould be reduced To this end we propose the NTC transfor-mation (6) that reduces the arbitrary number of dimensions toone Below we prove that it preserves the priors αlowast


fxu(x)

fxp(x)= infysimfyu (y)

fyu(y)

fyp(y)(9)

and the posteriors


fxu(x)

= αlowastfyp(y)

fyu(y)equiv plowastp(y) (10)

Lemma 1 NTC is a posterior- and αlowast-preserving transfor-mation ie (9) and (10) hold2

Proof To prove (9) and (10) we only need to show thatfxp (x)

fxu (x) =fyp (y)

fyu (y) where y = g(x) We will rely on the rule

of variable substitution fyp(y) =sumxrisinr

fxp (xr)

nablag(xr) where r =

x | g(x) = y or equivalently r = x | fxp (x)

fxu (x) =y

1minusy

fyp(y)

fyu(y)=

sumxrisinr

fxp (xr)

nablag(xr)sumxrisinr

fxu (xr)nablag(xr)

=

sumxrisinr

fxu (xr)nablag(xr)

fxp (xr)

fxu (xr)sumxrisinr

fxu (xr)nablag(xr)

=y

1minus y

sumxrisinr

fxu (xr)nablag(xr)sum

xrisinrfxu (xr)nablag(xr)

=y

1minus y=fxp

(xr)

fxu(xr) xr isin r

Remark on Lemma 1 In the proof we only use the prop-erty of NTC that the density ratio

fxp (x)

fxu (x) is constant for allx such that g(x) = y Generally any function g(x) withthis property preserves the priors and the posteriors For in-stance NTC could be biased towards 05 probability estimatedue to regularization In this case (9) and (10) would stillhold whereas (7) and (8) used in EN would yield incorrectestimates since they rely on NTC being unbiased

Note that the choice of NTC is flexible For example NTCand its hyperparameters can be chosen by maximizing ROC-AUC a metric that is invariant to label-dependent contamina-tion [Menon et al 2015 Menon et al 2016]

After the densities of NTC predictions are estimated twosmoothing heuristics (Algorithm 2) are applied to their ratio

32 Alternative αlowast EstimationThe second issue is that (9) may systematically underestimateαlowast as it solely relies on the noisy infimum point This concernis confirmed experimentally in Section 52 To resolve it wepropose two alternative estimates We first prove that theseestimates coincide with αlowast when the densities are known andthen formulate their practical approximations

2While (9) is known in the literature [Jain et al 2016] (10) andthe property that

fxp (x)

fxu (x)=

fyp (y)

fyu (y)have not been proven before

Algorithm 2 Secondary functions for DEDPUL

1 function EM(r(y) tol = 10minus5)2 αlowast = 1 Initialize3 repeat4 αlowastprev = αlowast

5 pp(y) = min(αlowast middot r(y) 1) E-step6 αlowast = 1

|Yu|sumyisinYu

(pp(y)) M-step7 until |αlowastprev minus αlowast| lt tol Convergence8 Return αlowast pp(y)9 function MAX SLOPE(r(y)max D = 005 ε = 10minus3)

10 D = list() Priors subtract average posteriors11 for α in range(start=0 end=1 step=ε) do12 pp(y) = min(α middot r(y) 1)13 idx = αε14 D[idx] = αminus 1

|Yu|sumyisinYu

(pp(y))

15 l = list() Second lags of D16 for i in range(start=1 end= 1

ε minus 1 step=1) do17 l[i] = (D[iminus 1]minusD[i])minus (D[i]minusD[i+ 1])

18 αlowast = argmax (l) s t D lt max D19 pp(y) = min(αlowast middot r(y) 1)20 Return αlowast pp(y)21 function MONOTONIZE(r(y) Yu)22 maxcur = 0 threshold = 1

|Yu|sumyisinYu

y

23 for i in range(start=0 end=length(r(y)) step=1) do24 if Yu[i] ge threshold then25 maxcur = max(r(y)[i]maxcur)26 r(y)[i] = maxcur27 Return r(y)28 function ROLLING MEDIAN(r(y))29 L = length(r(y)) k = L2030 rnew(y) = r(y)31 for i in range(start=0 end=L step=1) do32 kcur = min(k i Lminusi) Number of neighbours33 rnew(y)[i] = median(r(y)[iminus kcur i+ kcur])

34 Return rnew(y)

Lemma 2 Alternative estimates of the priors are equal tothe definition (9) αlowastc = αlowastn = αlowast whereD(α) equiv αminus Eysimfyu (y) [pp(y)]αlowastc equiv max

αisin[01]α | D(α) = 0

αlowastn equiv minαisin[01]

α | D(α+ ε)minus 2D(α) +D(αminus ε) gt 0

Proof We first prove that D(α) behaves in a specific way(Fig 2a) For the valid priors α le αlowast it holds thatD(α) = 0since priors must be equal to expected posteriors p(h) =Exsimf(x)p(h | x) for some hypothesis h and data x sim f(x)

For the invalid priors α gt αlowast the situation changes Bydefinition of αlowast (9) it is now the case that pp(y) = α

fyp (y)

fyu (y) gt

1 for some y To prevent this the posteriors are clipped fromabove like in (11) Under this redefinition the equality of pri-ors to expected posteriors no longer holds Instead clippingdecreases the posteriors in comparison to the priors Thus for

Figure 2 Behaviours of D(α) in theory (a) and practice (b c) In these plots fxp(x) = L(0 1) fxn = L(2 1) α = 05 and αlowast = 0568where L(middot middot) denotes Laplace distribution

α gt αlowast it holds that α gt Eysimfyu (y) [pp(y)] and D(α) gt 0So D(α) equals 0 up until αlowast and is positive after It triv-

ially follows that αlowast is the highest α for whichD(α) = 0 andthe lowest α for which D(α + ε) minus 2 middotD(α) +D(α minus ε) =D(α+ ε) gt 0 for any ε (in practice ε is chosen small)

Figures 2b and 2c reflect howD(α) changes in practice dueto imperfect density estimation in simple words the curveeither lsquorotateslsquo down or up In the more frequent first casenon-trivial αlowastc exists (the curve intersects zero) We identify itusing Expectation-Maximization algorithm but analogs likebinary search could be used as well In the second case (egsometimes when αlowast is extremely low) non-trivial αlowastc doesnot exist and EM converges to α = 0 In this case we in-stead approximate αlowastn using MAX SLOPE function We de-fine these functions in Algorithm 2 and cover them below inmore details In DEDPUL the two estimates are switchedby choosing the maximal In practice we recommend to plotD(α) to visually identify αlowastc (if exists) or αlowastn

Approximation of αlowastc with EM algorithm On Expecta-tion step the proportion α is fixed and the posteriors pp(y)are estimated using clipped (10)

pp(y) = min

(αfyp(y)

fyu(y) 1

)(11)

On Maximization step α is updated as the average value ofthe posteriors over the unlabeled sample Yu

α =1

|Yu|sumyisinYu

(pp(y)) (12)

These two steps iterate until convergence The algorithmshould be initialized with α = 1

Approximation of αlowastn with MAX SLOPE On a grid ofproportion values α isin [0 1] second difference of D(α) iscalculated for each point Then the proportion that corre-sponds to the highest second difference is chosen

4 Experimental ProcedureWe conduct experiments on synthetic and benchmark datasets Although DEDPUL is able to solve both Mixture Pro-portions Estimation and PU Classification simultaneously

some of the algorithms are not For this reason all PU Clas-sification algorithms receive αlowast (synthetic data) or α (bench-mark data) as an input The algorithms are tested on numer-ous data sets that differ in the distributions and the propor-tions of the mixing components Additionally each experi-ment is repeated 10 times and the results are averaged

The statistical significance of the difference between thealgorithms is verified in two steps First the Friedman testwith 002 P-value threshold is applied to check whether allthe algorithms perform equivalently If this null hypothesisis rejected the algorithms are compared pair-wise using thepaired Wilcoxon signed-rank test with 002 P-value thresholdand Holm correction for multiple hypothesis testing The re-sults are summarized via critical difference diagrams (Fig 3)the code for which is taken from [Ismail Fawaz et al 2019]

41 DataIn the synthetic setting we experiment with mixtures of one-dimensional Laplace distributions We fix fxp

(x) as L(0 1)and mix it with different fxn

(x) L(1 1) L(2 1) L(4 1)L(0 2) L(0 4) For each of these cases the proportion α isvaried in 001 005 025 05 075 095 099 The samplesizes Xp and Xu are fixed as 1000 and 10000

In the benchmark setting (Table 1) we experiment witheight data sets from UCI machine learning repository [Bacheand Lichman 2013] MNIST image data set of handwrittendigits [LeCun et al 2010] and CIFAR-10 image data set ofvehicles and animals [Krizhevsky et al 2009] The propor-tion α is varied in 005 025 05 075 095 The methodsrsquoperformance is mostly determined by the size of the labeledsample |Xp| Therefore for a given data set |Xp| is fixedacross the different values of α and for a given α the maxi-mal |Xu| that can satisfy it is chosen Categorical features aretransformed using dummy encoding Numerical features arenormalized Regression and multi-classification target vari-ables are adapted for binary classification

42 Measures for Performance EvaluationThe synthetic setting provides a straightforward way to eval-uate performance Since the underlying distributions fxp

(x)and fxu

(x) are known we calculate the true values of theproportions αlowast and the posteriors plowastp(x) using (4) and (5) re-spectively Then we directly compare these values with al-gorithmrsquos estimates using mean absolute errors In the bench-mark setting the distributions are unknown Here for Mix-

data set totalsize |Xp| dim positive

targetbank 45211 1000 16 yes

concrete 1030 100 8 (358 826)landsat 6435 1000 36 4 5 7

mushroom 8124 1000 22 ppageblock 5473 100 10 2 3 4 5

shuttle 58000 1000 9 2 3 4 5 6 7spambase 4601 400 57 1

wine 6497 500 12 redmnist 70000 1000 784 1 3 5 7 9

cifar10 60000 1000 3072 plane carship track

Table 1 Description of benchmark data sets

ture Proportions Estimation we use a similar measure butsubstitute αlowast with α while for PU Classification we use1minusaccuracy with 05 probability threshold We additionallycompare the algorithms using f1-measure a popular metricfor imbalanced data sets

43 Implementations of MethodsDEDPUL is implemented according to Algorithms 1 and 2As NTC we apply neural networks with 1 hidden layer with16 neurons (1-16-1) to the synthetic data and with 2 hiddenlayers with 512 neurons (dim-512-512-1) and with batch nor-malization [Ioffe and Szegedy 2015] after the hidden layersto the benchmark data To CIFAR-10 a light version of all-convolutional net [Springenberg et al 2015] with 6 convolu-tional and 2 fully-connected layers is applied with 2-d batchnormalization after the convolutional layers These architec-tures are used henceforth in EN and nnPU The networks aretrained on the logistic loss with ADAM [Kingma and Ba2015] optimizer with 00001 weight decay The predictionsof each network are obtained with 5-fold cross-validation

Densities of the predictions are computed using kernel den-sity estimation with Gaussian kernels Instead of y isin [0 1]we estimate densities of appropriately ranged log

(y

1minusy

)and

make according post-transformations Bandwidths are set as01 and 005 for fyp(y) and fyu(y) respectively In prac-tice a well-performing heuristic is to independently choosebandwidths for fyp(y) and fyu(y) that maximize averagelog-likelihood during cross-validation Threshold in MONO-TONIZE is chosen as 1

|Xu|sumxisinXu

(g(x))The implementations of the Kernel Mean based gradient

thresholding algorithm (KM) are retrieved from the originalpaper [Ramaswamy et al 2016]3 We provide experimen-tal results for KM2 a better performing version As advisedin the paper MNIST and CIFAR-10 data are reduced to 50dimensions with Principal Component Analysis [Wold et al1987] Since KM2 cannot handle big data sets it is appliedto subsamples of at most 4000 examples (Xp is prioritized)

The implementation of Tree Induction for label frequencyc Estimation (TIcE) algorithm is retrieved from the original

3httpwebeecsumichedusimcscottcodehtmlkmpe

paper [Bekker and Davis 2018b]4 MNIST and CIFAR-10data are reduced to 200 dimensions with Principal Compo-nent Analysis In addition to the vanilla version we also ap-ply TIcE to the data preprocessed with NTC (NTC+TIcE)The performance of both versions heavily depends on thechoice of the hyperparameters such as the probability lowerbound δ and the number of splits per node s These parame-ters were chosen as δ = 02 and s = 40 for univariate dataand δ = 02 and s = 4 for multivariate data

We explore two versions of the non-negative PU learning(nnPU) algorithm [Kiryo et al 2017] the original versionwith Sigmoid loss function and our modified version withBrier loss function5 The hyperparameters were chosen asβ = 0 and γ = 1 for the benchmark data and as β = 01 andγ = 09 for the synthetic data

Elkan-Noto (EN) is implemented according to the originalpaper [Elkan and Noto 2008] Out of the three proposed es-timators of the labeling probability we report results for e3(equivalent to (7)) since it has shown better performance

5 Experimental results51 Comparison with State-of-the-artResults of comparison between DEPDUL and state-of-the-art are presented in Figures 4-7 and are summarized in Figure3 DEDPUL significantly outperforms state-of-the-art of bothproblems in both setups for all measures with the exceptionof insignificant difference between f1-measures of DEDPULand nnPU-sigmoid in the benchmark setup

On some benchmark data sets (Fig 5 ndash landsat pageblockwine) KM2 performs on par with DEDPUL for all α valuesexcept for the highest α = 095 while on the other data sets(except for bank) KM2 is strictly outmatched The regionsof low proportion of negatives are of special interest in theanomaly detection task and it is a vital property of an algo-rithm to find anomalies that are in fact present Unlike DED-PUL TIcE and EN KM2 fails to meet this requirement Onthe synthetic data (Fig 4) DEDPUL also outperforms KM2especially when the priors are high

Regarding comparison with TIcE DEDPUL outperformsboth its vanilla and NTC versions in both synthetic andbenchmark settings (Fig 4 5) Two insights about TIcEcould be noted First while the NTC preprocessing is re-dundant for one-dimensional synthetic data it can greatlyimprove the performance of TIcE on the high-dimensionalbenchmark data especially on image data Second whileKM2 generally performs a little better than TIcE on the syn-thetic data the situation is reversed on the benchmark data

In the posteriors estimation task DEDPUL matches or out-performs both Sigmoid and Brier versions of nnPU algorithmon all data sets in both setups (Fig 6 7) An insight aboutthe Sigmoid version is that it estimates probabilities relativelypoorly (Fig 6) This is because the Sigmoid loss is mini-mized by the median prediction ie either 0 or 1 label whilethe Brier loss is minimized by the average prediction ie a

4httpsdtaicskuleuvenbesoftwaretice5Sigmoid and Brier loss functions are mean absolute and mean

squared errors between classifierrsquos predictions and true labels

(a) Synthetic data |αlowast minus αlowast| (b) Benchmark data |αminus αlowast|

(c) Synthetic data∣∣plowastp(x)minus plowastp(x)∣∣ (d) Benchmark data 1 - accuracy

(e) Synthetic data 1 - f1-measure (f) Benchmark data 1 - f1-measure

Figure 3 Critical difference diagrams Summarized comparison of the algorithms on synthetic and benchmark data with different perfor-mance measures The horizontal axis represents relative ranking Bold horizontal lines represent statistically insignificant differences In thesynthetic setup accuracy differs insignificantly between the algorithms according to the Friedman test so the diagram is not reported

probability in [0 1] range Conversely there is no clear win-ner in the benchmark setting (Fig 7) On some data sets(bank concrete landsat pageblock spambase) the Brier lossleads to the better training whereas on other data sets (mush-room shuttle wine mnist) the Sigmoid loss prevails

Below we show that the performance of DEDPUL can befurther improved by the appropriate choice of NTC

52 Ablation studyDEDPUL is a combination of multiple parts such as NTCdensity estimation EM algorithm and several heuristics Todisentangle the contribution of these parts to the overall per-formance of the method we perform an ablation study Wemodify different details of DEDPUL and report changes inperformance in Figures 8-11

First we vary NTC Instead of neural networks we trygradient boosting of decision trees We use an open-sourcelibrary CatBoost [Prokhorenkova et al 2018] For the syn-thetic and the benchmark data respectively the hight of thetrees is limited to 4 and 8 and the learning rate is chosen as01 and 001 the amount of trees is defined by early stopping

Second we vary density estimation methods Aside fromdefault kernel density estimation (kde) we try Gaussian Mix-ture Models (GMM) and histograms (hist) The number ofgaussians for GMM is chosen as 20 and 10 for fu(x) andfp(x) respectively and the number of bins for histograms ischosen as 20 for both densities

Third we replace the proposed estimates of priors αlowastc andαlowastn with the default estimate (9) (simple alpha) This is sim-ilar to the pdf-ratio baseline from [Jain et al 2016] We re-place infimum with 005 quantile for improved stability

Fourth we remove MONOTONIZE and ROLLING MEDIANheuristics that smooth the density ratio (no smooth)

Fifth we report performance of EN method [Elkan andNoto 2008] This can be seen as ablation of DEDPUL wheredensity estimation is removed altogether and is replaced bysimpler equations (7) and (8)

CatBoost In the synthetic setting CatBoost is outmatchedby neural networks as NTC on both tasks (Fig 8 10) The sit-

uation changes in the benchmark setting where CatBoost out-performs neural networks on most data sets especially duringlabel assignment (Fig 11) Still neural networks are a betterchoice for image data sets This comparison highlights theflexibility in the choice of NTC for DEDPUL depending onthe data at hand

Density estimation On the task of proportion estimation inthe synthetic data kde performs a little better than analoguesHowever the choice of the density estimation procedure haslittle effect with no clear trend in other setups (Fig 9-11)which indicates robustness of DEDPUL to the choice

Default αlowast On all data sets using the default αlowast estimationmethod (9) massively degrades the performance especiallywhen contamination of the unlabeled sample with positives ishigh (Fig 8 9) This is to be expected since the estimate (9)is a single infimum (or low quantile) point of the noisy den-sity ratio while the alternatives leverage the entire unlabeledsample Thus the proposed estimates αlowastc and αlowastn are vital forDEDPUL stability

No smoothing While not being very impactful on the es-timation of posteriors the smoothing heuristics are vital forthe quality of the proportion estimation (Fig 8 9)

EN In the synthetic setup performance of EN is decent(Fig 4 6) It outperforms both versions of TIcE on mostand nnPU-sigmoid on all synthetic data sets In the bench-mark setup EN is the worst procedure for proportion estima-tion (Fig 5) but it still outperforms nnPU on some data sets(concrete landsat shuttle spambase) during label assign-ment (Fig 7) Nevertheless EN never surpasses DEDPULexcept for L(0 2) The striking difference between the twomethods might be explained with EN requiring more fromNTC as discussed in Section 31

6 ConclusionIn this paper we have proven that NTC is a posterior-preserving transformation proposed an alternative method ofproportion estimation based on D(α) and combined these

findings in DEDPUL We have conducted an extensive ex-perimental study to show superiority of DEDPUL over cur-rent state-of-the-art in Mixture Proportion Estimation and PUClassification DEDPUL is applicable to a wide range of mix-ing proportions is flexible in the choice of the classificationalgorithm is robust to the choice of the density estimationprocedure and overall outperforms analogs

There are several directions to extend our work Despiteour theoretical finding the reasons behind the behaviour ofD(α) in practice remain unclear In particular we are uncer-tain why non-trivial αlowastc does not exist in some cases and whatdistinguishes these cases Next it could be valuable to ex-plore the extensions of DEDPUL to the related problems suchas PU multi-classification or mutual contamination Anotherpromising topic is relaxation of SCAR Finally modifyingDEDPUL with variational inference might help to internalizedensity estimation into the classifierrsquos training procedure

AcknowledgementsI sincerely thank Alexander Nesterov Alexander SirotkinIskander Safiulin Ksenia Balabaeva and Vitalia Eliseeva forregular revisions of the paper Support from the Basic Re-search Program of the National Research University HigherSchool of Economics is gratefully acknowledged

References[Bache and Lichman 2013] K Bache and M Lichman UCI

machine learning repository 2013[Bekker and Davis 2018a] Jessa Bekker and Jesse Davis

Beyond the selected completely at random assumption forlearning from positive and unlabeled data arXiv preprintarXiv180903207 2018

[Bekker and Davis 2018b] Jessa Bekker and Jesse DavisEstimating the class prior in positive and unlabeled datathrough decision tree induction In Proceedings of the 32thAAAI Conference on Artificial Intelligence 2018

[Blanchard et al 2010] Gilles Blanchard Gyemin Lee andClayton Scott Semi-supervised novelty detection Jour-nal of Machine Learning Research 11(Nov)2973ndash30092010

[du Plessis et al 2014] Marthinus C du Plessis Gang Niuand Masashi Sugiyama Analysis of learning from positiveand unlabeled data In Advances in neural informationprocessing systems pages 703ndash711 2014

[Du Plessis et al 2015a] Marthinus Du Plessis Gang Niuand Masashi Sugiyama Convex formulation for learningfrom positive and unlabeled data In International Confer-ence on Machine Learning pages 1386ndash1394 2015

[du Plessis et al 2015b] Marthinus Christoffel du PlessisGang Niu and Masashi Sugiyama Class-prior estimationfor learning from positive and unlabeled data In ACMLpages 221ndash236 2015

[Elkan and Noto 2008] Charles Elkan and Keith NotoLearning classifiers from only positive and unlabeled dataIn Proceedings of the 14th ACM SIGKDD international

conference on Knowledge discovery and data miningpages 213ndash220 ACM 2008

[Hammoudeh and Lowd 2020] Zayd Hammoudeh andDaniel Lowd Learning from positive and unlabeleddata with arbitrary positive shift arXiv preprintarXiv200210261 2020

[Ioffe and Szegedy 2015] Sergey Ioffe and ChristianSzegedy Batch normalization Accelerating deep net-work training by reducing internal covariate shift InInternational Conference on Machine Learning pages448ndash456 2015

[Ismail Fawaz et al 2019] Hassan Ismail Fawaz GermainForestier Jonathan Weber Lhassane Idoumghar andPierre-Alain Muller Deep learning for time series classifi-cation a review Data Mining and Knowledge Discovery33(4)917ndash963 2019

[Ivanov and Nesterov 2019] Dmitry I Ivanov and Alexan-der S Nesterov Identifying bid leakage in procurementauctions Machine learning approach arXiv preprintarXiv190300261 2019

[Jain et al 2016] Shantanu Jain Martha White Michael WTrosset and Predrag Radivojac Nonparametric semi-supervised learning of class proportions arXiv preprintarXiv160101944 2016

[Kingma and Ba 2015] Diederik P Kingma and Jimmy BaAdam A method for stochastic optimization Interna-tional Conference for Learning Representations 2015

[Kiryo et al 2017] Ryuichi Kiryo Gang Niu Marthinus Cdu Plessis and Masashi Sugiyama Positive-unlabeledlearning with non-negative risk estimator In Advancesin Neural Information Processing Systems pages 1675ndash1685 2017

[Krizhevsky et al 2009] Alex Krizhevsky Geoffrey Hintonet al Learning multiple layers of features from tiny im-ages Technical report Citeseer 2009

[LeCun et al 2010] Yann LeCun Corinna Cortes andCJ Burges MNIST handwritten digit databaseATampT Labs [Online] Available httpyann lecuncomexdbmnist 2 2010

[Liu et al 2007] Han Liu John Lafferty and Larry Wasser-man Sparse nonparametric density estimation in high di-mensions using the rodeo In Artificial Intelligence andStatistics pages 283ndash290 2007

[Menon et al 2015] Aditya Menon Brendan Van RooyenCheng Soon Ong and Bob Williamson Learning fromcorrupted binary labels via class-probability estimationIn International Conference on Machine Learning pages125ndash134 2015

[Menon et al 2016] Aditya Krishna Menon BrendanVan Rooyen and Nagarajan Natarajan Learning frombinary labels with instance-dependent corruption arXivpreprint arXiv160500751 2016

[Nguyen et al 2011] Minh Nhut Nguyen Xiaoli-Li Li andSee-Kiong Ng Positive unlabeled leaning for time se-

ries classification In IJCAI volume 11 pages 1421ndash14262011

[Oza and Patel 2018] Poojan Oza and Vishal M Patel One-class convolutional neural network IEEE Signal Process-ing Letters 26(2)277ndash281 2018

[Prokhorenkova et al 2018] Liudmila Prokhorenkova GlebGusev Aleksandr Vorobev Anna Veronika Dorogush andAndrey Gulin Catboost unbiased boosting with categor-ical features In Advances in Neural Information Process-ing Systems pages 6638ndash6648 2018

[Ramaswamy et al 2016] Harish Ramaswamy ClaytonScott and Ambuj Tewari Mixture proportion estimationvia kernel embeddings of distributions In InternationalConference on Machine Learning pages 2052ndash20602016

[Ren et al 2014] Yafeng Ren Donghong Ji and HongbinZhang Positive unlabeled learning for deceptive reviewsdetection In EMNLP pages 488ndash498 2014

[Sanderson and Scott 2014] Tyler Sanderson and ClaytonScott Class proportion estimation with application to mul-ticlass anomaly rejection In Artificial Intelligence andStatistics pages 850ndash858 2014

[Scholkopf et al 2001] Bernhard Scholkopf John C PlattJohn Shawe-Taylor Alex J Smola and Robert CWilliamson Estimating the support of a high-dimensionaldistribution Neural computation 13(7)1443ndash1471 2001

[Scott and Blanchard 2009] Clayton Scott and Gilles Blan-chard Novelty detection Unlabeled data definitely helpIn Artificial Intelligence and Statistics pages 464ndash4712009

[Scott and Nowak 2006] Clayton D Scott and Robert DNowak Learning minimum volume sets Journal of Ma-chine Learning Research 7(Apr)665ndash704 2006

[Scott et al 2013] Clayton Scott Gilles Blanchard andGregory Handy Classification with asymmetric labelnoise Consistency and maximal denoising In ConferenceOn Learning Theory pages 489ndash511 2013

[Scott 2015a] Clayton Scott A rate of convergence for mix-ture proportion estimation with application to learningfrom noisy labels In Artificial Intelligence and Statisticspages 838ndash846 2015

[Scott 2015b] Clayton Scott A rate of convergence for mix-ture proportion estimation with application to learningfrom noisy labels In Artificial Intelligence and Statisticspages 838ndash846 2015

[Shajarisales et al 2020] Naji Shajarisales Peter Spirtesand Kun Zhang Learning from positive and unlabeleddata by identifying the annotation process arXiv preprintarXiv200301067 2020

[Springenberg et al 2015] Jost Tobias SpringenbergAlexey Dosovitskiy Thomas Brox and Martin Ried-miller Striving for simplicity The all convolutional netInternational Conference for Learning RepresentationsWorkshop 2015

[Vert and Vert 2006] Regis Vert and Jean-Philippe VertConsistency and convergence rates of one-class svms andrelated algorithms Journal of Machine Learning Re-search 7(May)817ndash854 2006

[Wold et al 1987] Svante Wold Kim Esbensen and PaulGeladi Principal component analysis Chemometrics andintelligent laboratory systems 2(1-3)37ndash52 1987

[Yang et al 2012] Peng Yang Xiao-Li Li Jian-Ping MeiChee-Keong Kwoh and See-Kiong Ng Positive-unlabeled learning for disease gene identification Bioin-formatics 28(20)2640ndash2647 2012

Figure 4 Comparison of Mixture Proportions Estimation Algorithms Synthetic Data

Figure 5 Comparison of Mixture Proportions Estimation Algorithms Benchmark Data

Figure 6 Comparison of Positive-Unlabeled Classification Algorithms Synthetic Data

Figure 7 Comparison of Positive-Unlabeled Classification Algorithms Benchmark Data

Figure 8 Ablations of DEDPUL Mixture Proportions Estimation Synthetic Data

Figure 9 Ablations of DEDPUL Mixture Proportions Estimation Benchmark Data

Figure 10 Ablations of DEDPUL Positive-Unlabeled Classification Synthetic Data

Figure 11 Ablations of DEDPUL Positive-Unlabeled Classification Benchmark Data

1 Introduction

2 Problem Setup and Background

21 Preliminaries

22 True Proportions are Unidentifiable

23 Non-Traditional Classifier

3 Algorithm Development

31 NTC as Dimensionality Reduction

32 Alternative Estimation

4 Experimental Procedure

41 Data

42 Measures for Performance Evaluation

43 Implementations of Methods

5 Experimental results

51 Comparison with State-of-the-art

52 Ablation study

6 Conclusion

Figure 1 DEDPUL Data Upper row (ones) represents positive sample Xp lower row (ones and zeros) represents unlabeled sample XuNon-Traditional Classifier NTC is a supervised probabilistic binary classifier trained to distinguish Xp from Xu Predictions Predicted byNTC probabilities Yp and Yu for samples Xp and Xu obtained with cross-validation NTC reduces dimensionality Densities of PredictionsEstimated probability density functions fyp(y) and fyu(y) on Yp and Yu respectively fyu(y) has two distinct peaks one of which coincideswith that of fyp(y) while the other corresponds to the negative component of the mixture Priors Posteriors The posteriors are estimated forfixed priors using the Bayes rule and are clipped The priors are estimated as the average posteriors These two steps iterate until convergenceIf the process converges trivially to 0 an alternative estimate αlowast

n is used instead

weights into the loss function a PN classifier can be learneddirectly from PU data These weights are equal to the pos-terior probabilities of being positive estimated based on thepredictions of NTC The idea of loss function reweightingwould later be adopted by the Risk Estimation framework [duPlessis et al 2014 Du Plessis et al 2015a] and its latestnon-negative modification nnPU [Kiryo et al 2017] whichis currently considered state-of-the-art The crucial insight ofthis framework is that only the mixing proportion ie theprior probability of being positive is required to train an un-biased PN classifier on PU data

While nnPU does not address mixture proportion esti-mation multiple studies explicitly focus on this problem[Sanderson and Scott 2014 Scott 2015a du Plessis et al2015b] The state-of-the-art methods are KM [Ramaswamyet al 2016] which is based on mean embeddings of pos-itive and unlabeled samples into reproducing kernel Hilbertspace and TIcE [Bekker and Davis 2018b] which tries tofind a subset of mostly positive data via decision tree induc-tion Another noteworthy method is AlphaMax [Jain et al2016] which explicitly estimates the total likelihood of thedata as a function of the proportions and seeks for the pointwhere the likelihood starts to change

This paper makes several distinct contributions Firstwe prove that NTC is a posterior-preserving transformation(Section 31) whereas NTC is only known in the litera-ture to preserve the priors [Jain et al 2016] This en-ables practical estimation of the posteriors using probabil-ity density functions of NTC predictions Independentlywe propose two alternative estimates of the mixing propor-tions (Section 32) These estimates are based on the re-lation between the priors and the average posteriors Fi-nally these new ideas are merged in DEDPUL (Difference-of-Estimated-Densities-based Positive-Unlabeled Learning)a method that simultaneously solves both Mixture Propor-tions Estimation and PU Classification outperforms state-of-the-art methods for both problems [Ramaswamy et al 2016Bekker and Davis 2018b Kiryo et al 2017] and is appli-cable to a wide range of mixture proportions and data sets(Section 5)

DEDPUL adheres to the following two-step strategy At

the first step NTC is constructed [Elkan and Noto 2008]Since NTC treats unlabeled sample as negative its predic-tions can be considered as biased posterior probabilities ofbeing positive rather than negative The second step shouldeliminate this bias At the second step the probability den-sity functions of the NTC predictions for both positive andunalbeled samples are explicitly estimated Then both theprior and the posterior probabilities of being positive are es-timated simultaneously by plugging these densities into theBayes rule and performing Expectation-Maximization (EM)In case if EM converges trivially an alternative algorithm isapplied

2 Problem Setup and BackgroundIn this section we introduce the notations and formally de-fine the problems of Mixture Proportions Estimation and PUClassification Additionally we discuss some of the commonassumptions that we use in this work in Section 22

21 PreliminariesLet random variables xp xn xu have probability densityfunctions fxp

(x) fxn(x) fxu

(x) and correspond to positivenegative and unlabeled distributions of x where x isin Rmis a vector of features Let α be the proportion of fxp

(x) infxu

(x)

fxu(x) equiv αfxp

(x) + (1minus α)fxn(x) (1)

Let pp(x) be the posterior probability that x sim fxu(x) is

sampled from fxp(x) rather than fxn

(x) Given the propor-tion α it can be computed using the Bayes rule

pp(x) equivαfxp(x)

αfxp(x) + (1minus α)fxn

(x)= α

fxp(x)

fxu(x)

(2)

22 True Proportions are UnidentifiableThe goal of Mixture Proportions Estimation is to estimatethe proportions of the mixing components in unlabeled datagiven the samples Xp and Xu from the distributions fxp

(x)and fxu

(x) respectively The problem is fundamentally ill-posed even if the distributions fxp

(x) and fxu(x) are known

Algorithm 1 DEDPUL













fxu(x)

fxp(x)

(4)



fxu(x)

(5)





fxp (x)

fxu (x) leads



g(x) equivfxp(x)

fxp(x) + fxu

(x)(6)



1minus g(x)g(x)

(7)


1minus g(x)(8)



(x) respectively





fxu(x)


fyu(y)

fyp(y)(9)

and the posteriors


fxu(x)

= αlowastfyp(y)




fxu (x) =fyp (y)



fxp (xr)



fxu (x) =y

1minusy

fyp(y)

fyu(y)=

sumxrisinr

fxp (xr)


fxu (xr)nablag(xr)

=

sumxrisinr

fxu (xr)nablag(xr)

fxp (xr)

fxu (xr)sumxrisinr

fxu (xr)nablag(xr)

=y

1minus y

sumxrisinr



=y

1minus y=fxp

(xr)

fxu(xr) xr isin r


fxp (x)






fxp (x)

fxu (x)=

fyp (y)





|Yu|sumyisinYu



|Yu|sumyisinYu

(pp(y))




|Yu|sumyisinYu

y


34 Return rnew(y)


αisin[01]α | D(α) = 0





fyp (y)

fyu (y) gt







pp(y) = min

(αfyp(y)

fyu(y) 1

)(11)


α =1

|Yu|sumyisinYu

(pp(y)) (12)











(x)and fxu







wine 6497 500 12 redmnist 70000 1000 784 1 3 5 7 9






(y

1minusy

)and


|Xu|sumxisinXu




















































































1 Introduction


21 Preliminaries







41 Data





52 Ablation study

6 Conclusion

Algorithm 1 DEDPUL













fxu(x)

fxp(x)

(4)



fxu(x)

(5)





fxp (x)

fxu (x) leads



g(x) equivfxp(x)

fxp(x) + fxu

(x)(6)



1minus g(x)g(x)

(7)


1minus g(x)(8)



(x) respectively





fxu(x)


fyu(y)

fyp(y)(9)

and the posteriors


fxu(x)

= αlowastfyp(y)




fxu (x) =fyp (y)



fxp (xr)



fxu (x) =y

1minusy

fyp(y)

fyu(y)=

sumxrisinr

fxp (xr)


fxu (xr)nablag(xr)

=

sumxrisinr

fxu (xr)nablag(xr)

fxp (xr)

fxu (xr)sumxrisinr

fxu (xr)nablag(xr)

=y

1minus y

sumxrisinr



=y

1minus y=fxp

(xr)

fxu(xr) xr isin r


fxp (x)






fxp (x)

fxu (x)=

fyp (y)





|Yu|sumyisinYu



|Yu|sumyisinYu

(pp(y))




|Yu|sumyisinYu

y


34 Return rnew(y)


αisin[01]α | D(α) = 0





fyp (y)

fyu (y) gt







pp(y) = min

(αfyp(y)

fyu(y) 1

)(11)


α =1

|Yu|sumyisinYu

(pp(y)) (12)











(x)and fxu







wine 6497 500 12 redmnist 70000 1000 784 1 3 5 7 9






(y

1minusy

)and


|Xu|sumxisinXu




















































































1 Introduction


21 Preliminaries







41 Data





52 Ablation study

6 Conclusion



fxu(x)


fyu(y)

fyp(y)(9)

and the posteriors


fxu(x)

= αlowastfyp(y)




fxu (x) =fyp (y)



fxp (xr)



fxu (x) =y

1minusy

fyp(y)

fyu(y)=

sumxrisinr

fxp (xr)


fxu (xr)nablag(xr)

=

sumxrisinr

fxu (xr)nablag(xr)

fxp (xr)

fxu (xr)sumxrisinr

fxu (xr)nablag(xr)

=y

1minus y

sumxrisinr



=y

1minus y=fxp

(xr)

fxu(xr) xr isin r


fxp (x)






fxp (x)

fxu (x)=

fyp (y)





|Yu|sumyisinYu



|Yu|sumyisinYu

(pp(y))




|Yu|sumyisinYu

y


34 Return rnew(y)


αisin[01]α | D(α) = 0





fyp (y)

fyu (y) gt







pp(y) = min

(αfyp(y)

fyu(y) 1

)(11)


α =1

|Yu|sumyisinYu

(pp(y)) (12)











(x)and fxu







wine 6497 500 12 redmnist 70000 1000 784 1 3 5 7 9






(y

1minusy

)and


|Xu|sumxisinXu




















































































1 Introduction


21 Preliminaries







41 Data





52 Ablation study

6 Conclusion






pp(y) = min

(αfyp(y)

fyu(y) 1

)(11)


α =1

|Yu|sumyisinYu

(pp(y)) (12)











(x)and fxu







wine 6497 500 12 redmnist 70000 1000 784 1 3 5 7 9






(y

1minusy

)and


|Xu|sumxisinXu




















































































1 Introduction


21 Preliminaries







41 Data





52 Ablation study

6 Conclusion






wine 6497 500 12 redmnist 70000 1000 784 1 3 5 7 9






(y

1minusy

)and


|Xu|sumxisinXu




















































































1 Introduction


21 Preliminaries







41 Data





52 Ablation study

6 Conclusion







































































1 Introduction


21 Preliminaries







41 Data





52 Ablation study

6 Conclusion

dedpul: difference-of-estimated-densities-based positive ... · auctions [ivanov and nesterov,...

Documents