nonsparse learning with latent variables

OPERATIONS RESEARCHVol. 69, No. 1, January–February 2021, pp. 346–359

http://pubsonline.informs.org/journal/opre ISSN 0030-364X (print), ISSN 1526-5463 (online)

Methods

Nonsparse Learning with Latent VariablesZemin Zheng,a Jinchi Lv,b Wei Linc

a International Institute of Finance, School of Management, University of Science and Technology of China, Hefei 230026, China; bMarshallSchool of Business, University of Southern California, Los Angeles, California 90089; c School of Mathematical Sciences and Center forStatistical Science, Peking University, Beijing 100871, ChinaContact: [email protected], https://orcid.org/0000-0002-0240-9411 (ZZ); [email protected],

https://orcid.org/0000-0002-5881-9591 (JL); [email protected], https://orcid.org/0000-0002-7598-6199 (WL)

Received: February 23, 2019Revised: July 22, 2019Accepted: January 2, 2020Published Online in Articles in Advance:December 23, 2020

Subject Classification: statistics: data analysis,estimation, regressionArea of Review: High-Dimensional Learning

https://doi.org/10.1287/opre.2020.2005

Copyright: © 2020 INFORMS

Abstract. As a popular tool for producing meaningful and interpretable models, large-scale sparse learning works efficiently in many optimization applications when the un-derlying structures are indeed or close to sparse. However, naively applying the existingregularization methods can result in misleading outcomes because of model mis-specification. In this paper, we consider nonsparse learning under the factors plus sparsitystructure, which yields a joint modeling of sparse individual effects and common latentfactors. A newmethodology of nonsparse learning with latent variables (NSL) is proposedfor joint estimation of the effects of two groups of features, one for individual effects andthe other associated with the latent substructures, when the nonsparse effects are capturedby the leading population principal component score vectors. We derive the convergencerates of both sample principal components and their score vectors that hold for a wide classof distributions. With the properly estimated latent variables, properties including modelselection consistency and oracle inequalities under various prediction and estimation lossesare established. Our new methodology and results are evidenced by simulation and real-data examples.

Funding: This work was supported by the Beijing Natural Science Foundation [Grant Z190001], theNational Key R&D Program of China [Grant 2016YFC0207703], the National Science Foundation[Grant DMS-1953356], the National Natural Science Foundation of China [Grants 72071187,11601501, 11671018, 11671374, 71532001, 71731010, and 71921001], the Natural Science Foundationof Anhui Province [Grant 1708085QA02], Fundamental Research Funds for the Central Universities[Grant WK2040160028], the Adobe Data Science Research Award, the Simons Foundation, and theBeijing Academy of Artificial Intelligence.

Supplemental Material: The e-companion is available at https://doi.org/10.1287/opre.2020.2005.

Keywords: high dimensionality • nonsparse coefficient vectors • factors plus sparsity structure • principal component analysis •spiked covariance • model selection

1. IntroductionAdvances of information technologies have madehigh-dimensional data increasingly frequent not onlyin the domains of machine learning and biology butalso in economics (Belloni et al. 2017, Uematsu andTanaka 2019), marketing (Paulson et al. 2018), andnumerous operations research and engineering op-timization applications (Xu et al. 2016). In the high-dimensional regime, the number of available samplescan be less than the dimensionality of the problem,so the optimization formulations will contain manymore variables and constraints than in fact are neededto obtain a feasible solution. The key assumption thatenables high-dimensional statistical inference is thatthe regression function lies in a low-dimensional mani-fold (Hastie et al. 2009, Fan and Lv 2010, Bühlmann andvan de Geer 2011). Based on this sparsity assumption,a long list of regularization methods has been devel-oped to generate meaningful and interpretable models,

including those of Tibshirani (1996), Fan and Li (2001),Zou andHastie (2005), Candes and Tao (2007), Belloniet al. (2011), Sun and Zhang (2012), and Chen et al.(2016), among many others. Algorithms and theoreticalguarantees were also established for various regulari-zation methods. See, for example, Zhao and Yu (2006),Radchenko and James (2008), Bickel et al. (2009), Tangand Leng (2010), Fan et al. (2012), Candes et al. (2018),and Belloni et al. (2018).Although large-scale sparse learning works effi-

ciently when the underlying structures are indeed orclose to sparse, naively applying the existing regu-larization methods can result in misleading outcomesbecause of model misspecification (White 1982, Lvand Liu 2014, Hsu et al. 2019). In particular, it wasimposed in most high-dimensional inference methodsthat the coefficient vectors are sparse, which has beenquestioned in real applications. For instance, Boyleet al. (2017) suggested the omnigenic model, in which

346

http://pubsonline.informs.org/journal/opre

mailto:[email protected]

https://orcid.org/0000-0002-0240-9411

https://orcid.org/0000-0002-0240-9411


https://orcid.org/0000-0002-5881-9591

https://orcid.org/0000-0002-5881-9591


https://orcid.org/0000-0002-7598-6199

https://orcid.org/0000-0002-7598-6199



the genes associated with complex traits tend to bespread across most of the genome. Similarly, it wasconjectured earlier by Pritchard (2001) that instead ofbeing sparse, the causal variants responsible for a traitcan be distributed.Under such cases,making a correctstatistical inference is an important yet challengingtask. Though it is generally impossible to accuratelyestimate large numbers of nonzero parameters withrelatively low sample size, nonsparse learning maybe achieved by considering a natural extension ofthe sparse scenario—that is, the factors plus sparsitystructure. Specifically, we assume the coefficient vectorof predictors to be sparse after taking out the impactsof certain unobservable factors, which yields a jointmodeling of sparse individual effects and commonlatent factors. A similar ideawas exploited by Fan et al.(2013) using the low-rank-plus-sparse representationfor large covariance estimation, where a sparse errorcovariance structure is imposed after extracting com-mon but unobservable factors.

To characterize the impacts of latent variables,various methods have been proposed under differentmodel settings. For instance, the latent and observedvariables were assumed to be jointly Gaussian byChandrasekaran et al. (2012) for graphical modelselection. To control for confounding in genetic ge-nomics studies, Lin et al. (2015) used genetic variantsas instrumental variables. Pan et al. (2015) charac-terized latent variables by confirmatory factor analysis(CFA) in survival analysis and estimated them usingthe expectation–maximization algorithm. Despite thegrowing literature, relatively few studies deal withlatent variables in high dimensions. In this paper, wefocus on high-dimensional linear regression incor-porating two groups of features besides the responsevariable—that is, predictors with individual effects andcovariates associated with the latent substructures. Thenumbers of both predictors and potential latent variablescan be large, where the latent variables are nonsparselinear combinations of the covariates. To the best of ourknowledge, this is a new contribution to the case ofhigh-dimensional latent variables. Our analysis alsoallows for a special case that the two groups of featuresare identical, meaning that the latent variables areassociated with the original predictors.

We would like to provide a possible methodologyof nonsparse learning when the nonsparse effectsof the covariates can be captured by their leadingpopulation principal component score vectors, whichare unobservable because of the unknown populationcovariance matrix. The main reasons are as follows.Practically, principal components evaluate orthogo-nal directions that reflect maximal variations in thedata, thus often employed as surrogate variables toestimate the unobservable factors in many contem-porary applications such as genome-wide expression

studies (Leek and Storey 2007). In addition, the leadingprincipal components are typically extracted to adjustfor human genetic variations across population sub-structures (Menozzi et al. 1978, Cavalli-Sforza et al.1993) or stratification (Price et al. 2006). From a the-oretical point of view, principal components yield themaximum likelihood estimates of unobservable fac-tors when the factors are uncorrelated with eachother, even subject to certain measurement errors(Mardia et al. 1979). Moreover, the effects of thecovariates are mainly worked through their leadingpopulation principal component score vectors whenthe remaining eigenvalues decay rapidly.The major contributions of this paper are threefold.

First, we propose nonsparse learning with latentvariables based on the aforementioned factors plussparsity structure to simultaneously recover the sig-nificant predictors and latent factors as well as theireffects. Exploring population principal components ascommon latent variables will be helpful in attenuat-ing collinearity and facilitating dimension reduction.Second, to estimate population principal components,we use the sample counterparts and provide theconvergence rates of both sample principal compo-nents and their score vectors that hold for a wide classof distributions. The convergence property of samplescore vectors is critical to the estimation accuracy oflatent variables. This is, however, much less studied inthe literature compared with the principal compo-nents, and our work is among the first attempts inthe high-dimensional case. Third, we characterize themodel identifiability condition and show that theproposed methodology is applicable to general fami-lies with properly estimated latent variables. In par-ticular, under some regularity conditions, NSL via thethreshold regression is proved to enjoymodel selectionconsistency and oracle inequalities under various pre-diction and estimation losses.The rest of this paper is organized as follows.

Section 2 presents the new methodology of nonsparselearning with latent variables. We establish asymptoticproperties of sample principal components and theirscore vectors in high dimensions, as well as theoreticalproperties of the proposed methodology via the thresh-old regression in Section 3. Simulated and real-dataexamples are provided in Section 4. Section 5 dis-cusses extensions and possible future work. All theproofs of the main results and additional technicaldetails are included in the e-companion to this paper.

2. Nonsparse Learning withLatent Variables

2.1. Model SettingDenote byy � (y1, . . . , yn)T the n-dimensional responsevector, X � (x1, . . . , xp) the n × p random design

Zheng, Lv, and Lin: Nonsparse Learning with Latent VariablesOperations Research, 2021, vol. 69, no. 1, pp. 346–359, © 2020 INFORMS 347

matrix with p predictors, andW � (w1, . . . ,wq) the n ×q random matrix with q features. Assume that therows of X are independent with covariance matrix ΣXand that the rows of W are independent with meanzero and covariance matrix ΣW . We consider thefollowing high-dimensional linear regression model:

y � Xβ0 +Wη0 + ε, (1)where β0 � (β0,1, . . . , β0,p)T and η0 � (η0,1, . . . , η0,q)T are,respectively, the regression coefficient vectors ofthe predictors and the additional features, and ε ∼N(0, σ2In) is an n-dimensional error vector indepen-dent of X and W.

The Gaussianity of the random noises is imposedfor simplicity, and our technical arguments still applyas long as the error tail probability bound decaysexponentially. Different from most of the existingliterature, the regression coefficients η0 for covariatesW can be nonsparse, whereas the coefficient vector β0for predictors is assumed to be sparse with many zerocomponents after adjusting for the impacts of addi-tional features. Therefore, Model (1) is a mixture ofsparse and nonsparse effects. Both the dimensionalityp and the number of features q are allowed to grownonpolynomially fast with the sample size n.

As discussed in Section 1, to make the nonsparselearning possible, we impose the assumption that theimpacts of covariates W are captured by their Kleading population principal component score vec-tors fi � Wui for 1 ≤ i ≤ K, where {ui}Ki�1 are the top-Kprincipal components of the covariance matrix ΣW .That is, the coefficient vector η0 lies in the span ofthe top K population principal components, andthus, η0 � U0γ0 for some coefficient vector γ0 andU0 � (u1, . . . ,uK). In fact, when the covariance matrixΣW adopts a spiked structure (to be discussed inSection 3.1), the part of η0 orthogonal to the span ofthe leading population principal components willplay a relatively small role in prediction in view ofWη0 � VDUTη0, where VDUT is the singular valuedecomposition of W.

Denote by F � (f1, . . . , fK) the n × K matrix consist-ing of the K potential latent variables and by γ0 �(γ0,1, . . . , γ0,K)T their true regression coefficient vector.Then Model (1) can be rewritten as

y � Xβ0 + Fγ0 + ε. (2)It is worth pointing out that the latent variables F areunobservable to us because of the unknown vectorsui, which makes our work distinct frommost existingstudies. For the identifiability of population principalcomponents in high dimensions,Kwill be the numberof significant eigenvalues in the spiked covariancestructure of ΣW , and we allow it to diverge with thesample size.

Model (2) is applicable to two different situations.The first is that we aim at recovering the relationshipbetween predictors X and response y, whereas thefeatures W are treated as confounding variables formaking correct inferences on the effects of X, such asthe gene expression studies with sources of hetero-geneity (Leek and Storey 2007). Then the latent var-iables fi are not required to be associated with dif-ferent eigenvalues as long as their joint impacts Fγ0can be estimated. The other situation is that we areinterested in exploring the effects of bothX and F suchthat the latent variables fi should be identifiable. Thisoccurs in applications when the latent variables arealso meaningful. For instance, the principal compo-nents of genes can be biologically interpretable asrepresenting independent regulatory programs orprocesses (referred to as eigengenes) from their ex-pression patterns (Alter et al. 2000, Bair et al. 2006).In this paper, we mainly focus on the second sit-

uation because the latent variables in our motivatingapplication can also be biologically important. Spe-cifically, both nutrient intake and human gut micro-biome composition are believed to be important in theanalysis of body mass index (BMI), and they sharestrong associations (Chen and Li 2013). To alleviatethe strong correlations and facilitate the analysis ofpossibly nonsparse effects, we take nutrient intakeas a predictor and adjust for confounding variablesby incorporating the principal components of gutmicrobiome composition because principal compo-nents of human gut microbiome were found to revealdifferent enterotypes that affect energy extractionfrom the diet (Arumugam et al. 2011). The results ofthis real-data analysis will be presented in Section 4.2.In general applications, which variables should be

chosen as X andwhich should be chosen asW dependon the domain knowledge and research interests.Overall, predictors X stand for features with indi-vidual effects, whereas observable covariates W arecovariates that reflect the confounding substructures.Our analysis also allows for a special case that W is apart of X, meaning that the latent variables are non-sparse linear combinations of the original predictors.The identifiability of Model (2) will be discussed inSection 3.3 after Condition 4.

2.2. Estimation Procedure by NSLWith unobservable latent factors F, it is challenging toconsistently estimate and recover the support of theregression coefficient vector β0 for observable pre-dictors and the coefficients γ0 for latent variables. Wepartially overcome this difficulty by assuming thatthe factors appear in an unknown linear form of thecovariates W. Then F can be estimated by the sampleprincipal component scores of matrix W. Because therows of W have mean zero, the sample covariance

Zheng, Lv, and Lin: Nonsparse Learning with Latent Variables348 Operations Research, 2021, vol. 69, no. 1, pp. 346–359, © 2020 INFORMS

matrix S � n−1WTW is an unbiased estimate of ΣW

with top K principal components {ui}Ki�1. So the esti-mated latent variables are F � (f1, . . . , fK)with fi �Wui

for 1 ≤ i ≤ K. To ensure identifiability, both fi and fiare rescaled to have a common L2-norm n1/2, matchingthat of the constant predictor 1 for the intercept. Forfuture prediction, we can transform the coefficientvector γ back by multiplying the scalars n1/2‖Wui‖−12 .The notation ‖ · ‖q denotes the Lq-norm of a givenvector for q ∈ [0,∞].

To produce a joint estimate for the true coefficientvectors β0 and γ0, we suggest NSL, which minimizes

Q βT,γT( )T{ }� 2n( )−1 y − Xβ − Fγ

22+ pλ βT∗ ,γT( )T{ }

1, (3)

the penalized residual sum of squares with penalty func-tion pλ(·). Here β∗ � (β∗,1, . . . , β∗,p)T is the Hadamard(component-wise) product of two p-dimensional vec-tors (n−1/2‖xk‖2)1≤k≤p and β. It corresponds to the de-sign matrix with each column rescaled to have acommon L2-norm n1/2. The penalty function pλ(t) isdefined on t ∈ [0,∞), indexed by λ ≥ 0, and assumedto be increasing in both λ and t with pλ(0) � 0. Weuse a compact notation for

pλ βT∗ ,γT( )T{ }� pλ |β∗,1|( )

, . . . , pλ |β∗,p|( ), pλ |γ1|( )

, . . . ,{pλ |γK |( )}T.

The proposed methodology in (3) enables the possi-bility of simultaneously estimating β0 and γ0, iden-tifying the significant observable predictors and la-tent factors altogether. However, it is still difficult toobtain accurate estimates because the confoundingfactors F are replaced by the estimate F, and thecorrelations between the observable predictors andlatent variables can aggravate the difficulty. To pre-vent the estimation errors being further magnifiedin prediction, we consider γ in an L∞ ball Bρ � {γ ∈RK : ‖γ‖∞ ≤ ρ}, where any component of γ is assumedto be no larger than ρ in magnitude. We allow ρ todiverge slowly such that it will not deteriorate theoverall prediction accuracy.

2.3. Comparisons with Existing MethodsThe proposed methodology can be regarded as a reali-zation of the aforementioned low-rank plus sparse rep-resentation (Fan et al. 2013) in the high-dimensionallinear regression setting, but there are significantdifferences lying behind them. First, the latent vari-ables in our setup are not necessarily a part of theoriginal predictors but can stem from any sourcesrelated to the underlying features. Second, unlike thetypical assumption in factor analysis that the factors

and the remaining part are uncorrelated, we allowlatent variables to share correlations with the ob-servable predictors. In the extreme case, the latentvariables can be nonsparse linear combinations of thepredictors. Third, latent variables are employed torecover the information beyond the sparse effects ofpredictors, and thus we do not modify or assumesimplified correlations between the original predic-tors even after accounting for the latent substructures.Another method proposed by Kneip and Sarda

(2011) also incorporated principal components asextra predictors in penalized regression, but it appliesto a single group of features. Even if the two groups offeatures X and W are identical, the method differsfrom ours in the following aspects. First of all, basedon the framework of factor analysis, the observedpredictors in Kneip and Sarda (2011) weremixtures ofindividual features and common factors, both ofwhich were unobservable. In view of this, we aim atdifferent scopes of applications. Moreover, Kneip andSarda (2011) suggested sparse regression on the pro-jected model, where individual features were recov-ered as residuals of projecting the observed predictorson the factors. In contrast, we keep the original pre-dictors such that they will not be contaminated whenthe estimated latent variables are irrelevant. Last butnot least, benefiting from factor analysis, the indi-vidual features in Kneip and Sarda (2011) were un-correlated with each other and also shared no cor-relation with the factors. But we do not impose suchassumptions, as explained earlier.The proposed methodology is also closely related

to principal component regression (PCR). PCR sug-gests regressing the response vector on a subset ofprincipal components instead of all explanatory vari-ables, and comprehensive properties have been estab-lished in the literature for its importance in reducingcollinearity and enabling prediction in high dimen-sions. For instance, Cook (2007) explored situationswhere the response can be regressed on the leadingprincipal components of predictors with little loss ofinformation. Probabilistic explanation was providedby Artemiou and Li (2009) to support the phenom-enon that the response is often highly correlated withthe leading principal components. Our new meth-odology takes advantage of the strengths of principalcomponents to extract the most relevant informationfrom additional sources and adjust for confoundingand nonsparse effects, whereas model interpretabil-ity is also retained by exploring the individual effectsof observable predictors.In addition to the aforementioned literature, there

are two recent lines of work addressing nonsparselearning in high dimensions. Essential regression(Bing et al. 2019, 2020) is a new variant of factor re-gressionmodelswhere both the response and covariates


depend linearly on unobserved low-dimensional fac-tors. Our model assumptions are quite different fromthose of Bing et al. (2019, 2020) in that we also allowthe presence of covariates with sparse individualeffects. Another line of work including that of Bradicet al. (2020) and Zhu and Bradic (2018) aims at hy-pothesis testing under nonsparse linear structures.Test statistics were constructed through restructuredregression or marginal correlations, but the originalregression coefficients were not estimated.

3. Theoretical PropertiesWe will first establish the convergence properties ofsample principal components and their score vectorsfor a wide class of distributions under the spikedcovariance structure. With the aid of these convergenceproperties, additional properties including model se-lection consistency and oracle inequalities will beproved for the proposed methodology via the thresholdregression using hard thresholding.

3.1. Spiked Covariance ModelHigh-dimensional principal component analysis (PCA)particularly in the context of spiked covariancemodel, introduced by Johnstone (2001), has beenstudied by Paul (2007), Jung and Marron (2009), Shenet al. (2016), and Wang and Fan (2017), among manyothers. This model assumes that the first few eigen-values of the population covariance matrix deviatefrom one, whereas the rest are equal to one. Althoughsample principal components are generally incon-sistent without strong conditions when the number ofcovariates is comparable to or larger than the samplesize (Johnstone and Lu 2009), with the aid of a spikedcovariance structure, consistency of sample principalcomponents was established in the literature underdifferent high-dimensional settings. For instance, inthe high-dimension, low-sample-size context, Jungand Marron (2009) proved the consistency of sampleprincipal components for spiked eigenvalues. Whenboth the dimensionality and sample size are diverging,phase transition of sample principal components wasstudied by Paul (2007) and Shen et al. (2016) formultivariate Gaussian observations. The asymptoticdistributions of spiked principal components wereestablished byWang and Fan (2017) for sub-Gaussiandistributions with a finite number of distinguishablespiked eigenvalues.

In this section, we adopt the generalized version ofspiked covariance model studied by Jung and Marron(2009) for the covariance structure of covariate matrixW, where the population covariance matrix ΣW isassumed to contain K spiked eigenvalues that can bedivided into m groups. The eigenvalues grow at thesame rate within each group, whereas the orders of

magnitude of the m groups are different from eachother. To be specific, there are positive constants α1 >α2 > · · · > αm > 1 such that the eigenvalues in thelth group grow at the rate of qαl , 1 ≤ l ≤ m, where q isthe dimensionality or number of covariates in W.The constants αl are larger than one because other-wise the sample eigenvectors can be strongly incon-sistent (Jung and Marron 2009). Denote the groupsizes by positive integers k1, . . . , km satisfying

∑ml�1 kl �

K < n. Set km+1 � q − K, which is the number of non-spiked eigenvalues. Then the set of indices for the lthgroup of eigenvalues is

Jl � 1 +∑l−1j�1

kj, . . . , kl +∑l−1j�1

kj

{ }, l � 1, . . . ,m + 1. (4)

Although this eigenstructure looks almost the sameas that in Jung and Marron (2009), the key differencelies in the magnitudes of the sample size n and thenumber of spiked eigenvalues K, both of which areallowed to diverge in our setup instead of being fixed.This makes the original convergence analysis of sam-ple eigenvalues and eigenvectors invalid because thenumber of entries in the dual matrix SD � n−1WWT isno longer finite. We will overcome this difficultyby conducting a delicate analysis on the deviationbound of the entries such that the correspondingmatrices converge in Frobenius norm. Our theoreticalresults are applicable to a wide class of distributions,including sub-Gaussian distributions. For multivar-iate Gaussian or sub-Gaussian observations with afinite number of spiked eigenvalues, the phase tran-sition of PCA consistency was studied by, for in-stance, Shen et al. (2016) and Wang and Fan (2017).Nevertheless, the convergence property of sampleprincipal component score vectors was not providedin the aforementioned references and needs furtherinvestigation.Assume that the eigendecomposition of the pop-

ulation covariance matrix ΣW is given by ΣW � UΛUT,where Λ is a diagonal matrix of eigenvalues λ1 ≥λ2 ≥ · · · ≥ λq ≥ 0, andU � (u1, . . . ,uq) is an orthogonalmatrix consisting of the population principal com-ponents. Analogously, the eigendecomposition of S �UΛUT provides the diagonal matrix Λ of sample ei-genvalues λ1 ≥ λ2 ≥ · · · ≥ λq ≥ 0 and the orthogonalmatrix U � (u1, . . . , uq) consisting of sample principalcomponents. We always assume that the sample prin-cipal components take the correct directions such thatthe angles between sample and population principalcomponents are no more than a right angle.Our main focus is the high-dimensional setting

where the number of covariates q is no less than thesample size n. Denote by

Z � Λ−1/2UTWT (5)


the sphered data matrix. It is clear that the columnsof Z are independent and identically distributed (i.i.d.)with mean zero and covariance matrix Iq. To build ourtheory, we will impose a tail probability bound on theentry of Z and make use of the n-dimensional dualmatrix SD � n−1ZTΛZ, which shares the same nonzeroeigenvalues with S.

3.2. Threshold Regression UsingHard Thresholding

As discussed in Section 1, there is a large spectrum ofregularization methods for sparse learning in highdimensions. It has been demonstrated by Fan andLv (2013) that the popular L1-regularization of leastabsolute shrinkage and selection operator (Lasso) andconcave methods can be asymptotically equivalentin thresholded parameter space for polynomiallygrowing dimensionality, meaning that they share thesame convergence rates in the oracle inequalities. Forexponentially growing dimensionality, concavemethodscan also be asymptotically equivalent and have fasterconvergence rates than Lasso. Therefore, we willshow theoretical properties of the proposed method-ology via a specific concave regularizationmethod, thethreshold regression using hard thresholding (Zhenget al. 2014). It uses either the hard-thresholdingpenalty pH,λ(t) � 1

2 [λ2 − (λ − t)2+] or the L0-penaltypH0,λ(t) � 2−1λ21{t �0} in the penalized least squares (3),both of which enjoy the hard-thresholding property(Zheng et al. 2014, lemma 1) that facilitates sparsemodeling and consistent estimation.

A key concept for characterizing model identifi-ability in Zheng et al. (2014) is the robust sparkrsparkc(X) of agivenn × pdesignmatrixXwith bound c,defined as the smallest possible number τ such thatthere exists a submatrix consisting of τ columns fromn−1/2X with a singular value less than the givenpositive constant c, where X is obtained by rescalingthe columns of X to have a common L2-norm n1/2. Thebound on themagnitude of rsparkc(X)was establishedby Fan and Lv (2013) for Gaussian design matricesand further studied by Lv (2013) for more generalrandom design matrices. Under mild conditions,M �cn/(log p)with some positive constant cwill provide alower bound on rsparkc(X,F) for the augmented de-signmatrix (see Condition 4 in Section 3.3 for details).Following Fan and Lv (2013) and Zheng et al. (2014),we consider the regularized estimator on the union ofcoordinate subspaces SM/2 � {(βT,γT)T ∈ Rp+K : ‖(βT,γT)T‖0 < M/2} to ensure model identifiability andreduce estimation instability. So the joint estimator(βT

, γT)T is defined as the global minimizer of thepenalized least squares (3) constrained on space SM/2.

3.3. Technical ConditionsHere we list a few technical conditions and discusstheir relevance. Let Δ � min1≤l≤m−1(αl − αl+1). Then qΔ

reflects the minimum gap between the magnitudes ofspiked eigenvalues in two successive groups. The firsttwo conditions are imposed for Theorem 1, whereasthe rest are needed in Theorem 2, to be presented inSection 3.4.

Condition 1. There exist positive constants ci and C suchthat uniformly over i ∈ Jl, 1 ≤ l ≤ m,

λi/qαl � ci +O q−Δ( )

with ci ≤ C,

and λj ≤ C for any j ∈ Jm+1.

Condition 2. (a) There exists some positive α < min{Δ,αm − 1} such that uniformly over 1 ≤ i ≤ n and 1 ≤ j ≤ q,the (j, i)th entry zji of the sphered data matrix Z definedin (5) satisfies

P z2ji > K−1qα( )

� o q−1n−1( )

.

(b) For any 1 ≤ l ≤ m, ‖n−1ZlZTl − Ikl‖∞ � op(k−1l ), where

Zl is a submatrix of Z consisting of the rows with indicesin Jl.

Condition 3. Uniformly over j, 1 ≤ j ≤ K, the angle ωjj

between the jth estimated latent vector fj and its population

counterpart fj satisfies cos(ωjj) ≥ 1 − c22 log n/8K2ρ2n with

probability 1 − θ1 that converges to one as n → ∞.

Condition 4. The inequality ‖n−1/2(X, F)δ‖2 ≥ c‖δ‖2 holdsfor any δ satisfying ‖δ‖0 < M with probability 1 − θ2approaching one as n → ∞.

Condition 5. There exists some positive constant Lsuch that

P ∩pj�1 L−1 ≤ ‖xj‖2/

n

√ ≤ L{ }( )

� 1 − θ3,

where θ3 converges to zero as n → ∞.

Condition 6. Denote by s � ‖β0‖0 + ‖γ0‖0 the number ofoverall significant predictors and by b0 � minj∈supp(β0)(|β0,j|) ∧minj∈supp(γ0)(|γ0,j|) the overall minimum signalstrength. It holds that s < M/2 and

b0 >2

√c−11

( )∨ 1

[ ]c−11 c2L

2s + 1( ) log p( )

/n√

for some positive constants c1 defined in Proposition 1 inSection 3.4 and c2 > 2

2

√σ.

Condition 1 requires that the orders of magnitudeof spiked eigenvalues in each group be the same,


whereas their limits can be different depending onthe constants ci. This is weaker than the conditionsusually imposed in the literature, such as in Shenet al. (2016), where the spiked eigenvalues in eachgroup share exactly the same limit. Nevertheless, wewill prove the consistency of spiked sample eigen-values under very mild conditions. To distinguish theeigenvalues in different groups, convergence to thecorresponding limit is assumed to be at a rate ofO(q−Δ). As the number of spiked eigenvalues di-vergeswith q, we impose a constant upper boundConci for simplicity, and our technical argument stillapplies when C diverges slowly with q. Without lossof generality, the upper bound C also controls thenonspiked eigenvalues.

As pointed out earlier, the columns of the sphereddata matrix Z are i.i.d. withmean zero and covariancematrix Ip. Then part (a) of Condition 2 holds as long asthe entries in any column of Z satisfy the tail prob-ability bound.Moreover, it is clear that this tail bounddecays polynomially, so it holds for a wide class ofdistributions including sub-Gaussian distributions.With this tail bound, the larger sample eigenvalueswould dominate the sum of all eigenvalues in thesmaller groups regardless of the randomness. Fur-thermore, by definition, we know that the columns ofZl are i.i.d. with mean zero and covariance matrix Iklsuch that n−1ZlZT

l → Ikl entrywise as n → ∞. Hence,part (b) of Condition 2 is a very mild assumption todeal with the possibly diverging group sizes kl.

Condition 3 imposes a convergence rate of logn/(K2ρ2n) for the estimation accuracy of confoundingfactors, so the estimation errors in F will not deteri-orate the overall estimation and prediction powers.This rate is easy to satisfy in view of the results inTheorem 1 in Section 3.4 because the sample princi-pal component score vectors are shown to convergeto the population counterparts in polynomial or-ders of q, which is typically larger than n in high-dimensional settings.

Condition 4 assumes the robust spark of matrix(X,F) with bound c to be at least M � cn/(log p) withsignificant probability. This is the key for character-izing the model identifiability in our conditionalsparsity structure and also controls the correlationsbetween theobservablepredictorsX and latent factors F.Consider a special case where F consists of nonsparselinear combinations of the original predictors X. ThenModel (2) cannot be identified if we allow for non-sparse regression coefficients. However, if we con-strain the model size by a certain sparsity level, suchas rsparkc(X, F), the model will become identifiablebecause F cannot be represented by sparse linear

combinations of X. Using the same idea, if we imposeconditions such as the minimum eigenvalue for thecovariance matrix of any M1 features in (X,F) beingbounded from below, where M1 � c1n/(log p) withc1 > c denotes the sparsity level, then theorem 2 ofLv (2013) ensures that the robust spark of any sub-matrix consisting of less thanM1 columns of (X,F)willbe no less than M � cn/(log p). This holds for generaldistributions with tail probability decaying expo-nentially fast with the sample size n and the constant cdepending only on c. This justifies the inequality inCondition 4.Although no distributional assumptions are im-

posed on the random design matrix X, Condition 5places a mild constraint that the L2-norm of anycolumn vector ofXdivided by its common scale n1/2 isbounded with significant probability. This can besatisfied by many distributions and is needed due tothe rescaling of β∗ in (3). Condition 6 is similar to thatof Zheng et al. (2014) for deriving the global prop-erties via the threshold regression. The first partputs a sparsity constraint on the true model size s formodel identifiability, as discussed after Condition 4,whereas the secondpart gives a lower boundO{[s(log p)/n]1/2} on the minimum signal strength to distinguishthe significant predictors from the others.

3.4. Main ResultsWe provide two main theorems in this section. Thefirst is concerned with the asymptotic properties ofsample principal components and their score vectors,which serves as a bridge for establishing the globalproperties in the second theorem.A sample principal component is said to be consistent

with its population counterpart if the angle betweenthem converges to zero asymptotically. However,when several population eigenvalues belong to thesame group, the corresponding principal componentsmay not be distinguishable. In this case, subspaceconsistency is essential to characterizing the asymp-totic properties (Jung and Marron 2009). Denote θil �Angle(ui, span{uj : j ∈ Jl}) for i ∈ Jl, 1 ≤ l ≤ m, which isthe angle between the ith sample principal compo-nent and the subspace spanned by population prin-cipal components in the corresponding spiked group.The following theorem presents the convergence ratesof sample principal components in terms of anglesunder the aforementioned generalized spiked covari-ance model. Moreover, for the identifiability of latentfactors, we assume each group size to be one forthe spiked eigenvalues when studying the principalcomponent score vectors. That is, kl � 1 for 1 ≤ l ≤ m,implying that K � m.


Theorem 1 (Convergence Rates). Under Conditions 1and 2, with probability approaching one, the followingstatements hold:

a. Uniformly over i ∈ Jl, 1 ≤ l ≤ m, θil � Angle(ui,span{uj : j ∈ Jl}) is no more than

arccos 1 −∑l−1t�1

∏l−1i�t+1

1 + ki( )[ ]

O ktA t( ){ }[(

−O A l( ){ }]1/2)

, (6)

where A(t) � (∑ml�t+1 klq

αl + km+1)K−1qα−αt , and we de-fine

∑jt�i st � 0 and

∏jt�i st � 1 if j < i for any sequence {st}.

b. If each group of spiked eigenvalues has size one, thenuniformly over 1 ≤ i ≤ K, ωii � Angle(Wui,Wui) is nomore than

arccos 1 −∑i−1t�1

2i−t−1O A t( ){ } −O A i( ){ }[ ]1/2( )

.

Part (a) of Theorem 1 provides the uniform con-vergence rates of sample principal components to thecorresponding subspaces for a general spiked co-variance structure with possibly tiered eigenvaluesunder mild conditions. It holds even if the principalcomponents are not separable, so the results alsoapply to the first kind of applications of Model (2)discussed in Section 2.1. Because the convergencerates of θ2

il to zero and cos2(θil) to one are the same byL’Hôpital’s rule, both of them are

∑l−1t�1[∏l−1

i�t+1(1+ki)]O{ktA(t)} +O{A(l)} in view of (6). Thus, when thegroup sizes kl are relatively small, the convergencerates are determined by A(t), which decays poly-nomially with q and converges to zero fairly fast.This shows the “blessing of dimensionality” underthe spiked covariance structure because the larger qgives faster convergence rates. Furthermore, it is clearthat when the gaps between the magnitudes of dif-ferent spiked groups are large, A(t) decays quicklywith q to accelerate the convergence of sample prin-cipal components.

The uniform convergence rates of sample principalcomponent score vectors are given in part (b) ofTheorem 1when each group contains only one spikedeigenvalue such that the latent factors are separable.In fact, the proof of Theorem 1 shows that the samplescore vectors converge at least as fast as the sampleprincipal components. Then the results in part (b) areessentially the convergence rates in part (a) withkl � 1. Because the number of spiked eigenvalues K ismuch smaller than q, the sample principal compo-nent score vectors will converge to the populationcounterparts polynomially with q. The convergenceproperty of sample score vectors is critical to ourpurpose of nonsparse learning because it offers the

estimation accuracy of latent variables, which is muchless well studied in the literature. To the best of ourknowledge,ourwork isafirst attempt inhighdimensions.The established asymptotic property of sample

principal component score vectors justifies the esti-mation accuracy assumption in Condition 3. To-gether with Condition 4, this leads to the followingproposition.

Proposition 1. Under Conditions 3 and 4, the inequality

‖n−1/2 X, F( )

δ‖2≥ c1‖δ‖2

holds for some positive constant c1 and any δ satisfying‖δ‖0 < M with probability at least 1 − θ1 − θ2.

From the proof of Proposition 1, we see that theconstant c1 is smaller than but can be very close to cwhen n is relatively large. Therefore, Proposition 1shows that the robust spark of the augmented designmatrix (X, F) will be close to that of (X,F) when Fis accurately estimatedby F.Wearenowready topresenttheoretical properties for the proposed methodology.

Theorem 2 (Global properties). Assume that Condi-tions 3–6 hold and

c−11 c22s + 1( ) log p( )

/n√

< λ < L−1b0 1 ∧ c1/2

√( )[ ].

Then, for both the hard-thresholding penalty pH,λ(t) andL0-penalty pH0,λ(t), with probability at least 1−4σ(2/π)1/2

c−12 (logp)−1/2p1−c22

8σ2 −2σ(2/π)1/2c−12 s(logn)−1/2 ·n−c22

8σ2 −θ1−θ2−θ3, the regularized estimator (βT

, γT)T satisfies that

a. Themodel selection consistency supp{(βT, γT)T} �

supp{(βT0 ,γ

T0 )T}, where supp denotes the support of a vector;

b. The prediction error bound n−1/2‖(X, F)(βT, γT)T−

(X, F)(βT0 ,γ

T0 )T‖2 ≤ (c2/2 + 2c2c−11

s

√ ) (logn)/n√;

c. The oracle inequalities ‖β − β0‖q ≤ 2c−21 c2Ls1/q(log n)/n√, ‖γ−γ0‖q ≤ 2c−21 c2s1/q

(logn)/n√for q ∈ [1, 2].

The upper bounds with q � 2 also hold for ‖β − β0‖∞and ‖γ − γ0‖∞.The model selection consistency in Theorem 2(a)

shows that we can recover both the significant ob-servable predictors and the latent variables, so thewhole model would be identified by combining thesetwo parts even if it contains nonsparse coefficients.The prediction loss of the joint estimator is shown tobe within a logarithmic factor (logn)1/2 of that of theoracle estimator when the regularization parameter λis properly chosen, which is similar to the result inZheng et al. (2014). This means that the predictionaccuracy is maintained regardless of the hidden ef-fects as long as the latent factors are properly esti-mated. The extra term (c2/2)

(logn)/n√in the pre-

diction bound reflects the price we pay in estimating


the confounding factors. Furthermore, the oracle in-equalities for both β and γ under Lq-estimation losseswith q∈ [1,2] ∪{∞} are also established inTheorem 2(c).Although the estimation accuracy for the nonsparsecoefficients Uγ0 of W are obtainable, we omit theresults here because their roles in inferring the indi-vidual effects andprediction are equivalent to those ofthe latent variables.

The proposed methodology of NSL under the con-ditional sparsity structure is not restrictive to the po-tential family of population principal components. Itis more broadly applicable to any latent family pro-vided that the estimation accuracy of latent factors inCondition 3 and the correlations between the ob-servable predictors and latent factors characterizedby the robust spark in Condition 4 hold similarly. Thepopulation principal component provides a commonand concrete example to extract the latent variablesfrom additional covariates. A significant advantage ofthis methodology is that even if the estimated latentfactors are irrelevant, they rarely affect the variableselection and effect estimation of the original pre-dictors because the number of potential latent vari-ables is generally a small proportion of that of thepredictors. This also implies that a relatively large Kcan be chosen when we are not sure about how manylatent variables indeed exist. This is a key differencebetween our methodology and those based on factoranalysis, which renders our methodology useful forcombining additional sources.

4. Numerical StudiesIn this section, we investigate the finite sample per-formance of NSL via three regularization methods ofLasso (Tibshirani 1996), smoothly clipped absolutedeviation (SCAD; Fan and Li 2001), and the thresholdregression using hard thresholding (Hard; Zhenget al. 2014). All three methods are implementedthrough the independent component analysis algo-rithm (Fan and Lv 2011) because coordinate optimi-zation enjoys scalability for large-scale problems. Theoracle procedure (Oracle) that knew the truemodel inadvance is also conducted as a benchmark.

Wewill explore two differentmodels, whereModelM1 involves only observable predictors andModelM2

incorporates estimated latent variables as extra pre-dictors. The case of linear regression Model (2) withthe confounding factor as nonsparse combination ofthe existing predictors is considered in the first ex-ample, whereas in the second examplemultiple latentfactors stem from additional observable covariates,and the error vector is relatively heavy tailed with at-distribution.

4.1. Simulation Examples4.1.1. Simulation Example 1. In the first simulationexample, we consider a special case of linear re-gression Model (2) with potential latent factors Fcoming from the existing observable predictors—thatis,W � X. Then Fγ represents the nonsparse effects ofthe predictors X, and it will be interesting to check theimpacts of latent variables when they are dense linearcombinations of the existing predictors. The samplesize n was chosen to be 100 with true regression co-efficient vectors β0 � (vT, . . . ,vT,0)T, γ0 � (0.5, 0)T, andGaussian error vector ε ∼ N(0, σ2In), where v � (0.6,0, 0,−0.6, 0, 0)T is repeated k times, and γ0 is a K-dimensional vector with one nonzero component 0.5,denoting the effect of the significant confoundingfactor. We generated 200 data sets and adopted thesetting of (p, k,K, σ) � (1,000, 3, 10, 0.4) such that thereare 6 nonzero components with magnitude 0.6 inthe true coefficient vector β0 and 10 potential la-tent variables.The key point in the design of this simulation study

is to construct a population covariance matrix Σwithspiked structure. Therefore, for each data set, therows of the n × p design matrix X were sampled asi.i.d. copies from a multivariate normal distributionN(0,Σ)withΣ � 1/2(Σ1 + Σ2), whereΣ1 � (0.5|i−j|)1≤i,j≤pand Σ2 � 0.5Ip + 0.511T. The choice of Σ1 allows forcorrelation between the predictors at the populationlevel, and Σ2 has an eigenstructure such that thespiked eigenvalue is comparable with p. Based on theconstruction ofΣ1 and Σ2, it is easy to check thatΣ hasthe largest eigenvalue 251.75, and the others are allbelow 1.75. For regularization methods, Model M2involved the top K sample principal components asestimated latent variables, whereas Oracle used thetrue confounding factor instead of the estimated one.We applied Lasso, SCAD, and Hard for both M1 andM2 to produce a sequence of sparse models and se-lected the regularization parameter λ by minimizingthe prediction error calculated based on an indepen-dent validation set for fair comparison of all methods.To compare the performance of the aforementioned

methods under two different models, we considerseveral performance measures. The first measure isthe prediction error (PE), defined as E(Y − xTβ)2 inModel M1 and as E(Y − xTβ − fTγ)2 in Model M2,where β or (βT

, γT)T are the estimated coefficients inthe corresponding models, (xT,Y) is an independenttest sample of size 10, 000, and f is the sample prin-cipal component score vector. For Oracle, f is replacedby the true confounding factor f. The second to fourthmeasures are the Lq-estimation losses of β0—that is,‖β − β0‖q with q � 2, 1, and ∞, respectively. The fifthand sixth measures are the false positives (FP), falsely


selected noise predictors, and false negatives (FN),missed true predictors with respect to β0. The seventhmeasure is the model selection consistency (MSC)calculated as the frequency of selecting exactly therelevant variables. We also reported the estimatederror standard deviation σ by all methods in bothmodels. The results are summarized in Table 1. Forthe selection and effect estimation of latent variablesin Model M2, we display in Table 2 the measuressimilar to those defined in Table 1 but with respectto γ0. They are Lq-estimation losses ‖γ − γ0‖q withq � 2, 1, and ∞, FPγ, FNγ, and MSCγ.

In view of Table 1, it is clear that compared withModel M2, the performance measures in variableselection, estimation, and prediction all deterioratedseriously in Model M1, where most of importantpredictors were missed, and both the estimation andprediction errors were quite large. We want to em-phasize that in this first example, the latent variablesare linear combinations of the observable predictorsinitially included in the model, which means that thenonsparse effects would not be captured without thehelp of estimated confounding factors. By contrast,the prediction and estimation errors of all regulari-zation methods were reasonably small in the latentvariable–augmented Model M2. It is worth noticingthat the performance of Hard was comparable to thatof Oracle regardless of the estimation errors of latentfeatures, which is in line with the theoretical resultsin Theorem 2. Furthermore, we can see from Table 2that all methods with the estimated latent variables

correctly identified the true confounding factor andaccurately recovered its effect.

4.1.2. Simulation Example 2. Nowwe consider a moregeneral case where the latent variables stem from agroup of observable covariates instead of the origi-nal predictors. Moreover, we also want to see whethersimilar results hold when more significant confoundingfactors are involved and the errors become relativelyheavy tailed. Thus, there are three main changes inthe setting of this second example. First, the predictorsXand observable covariates W are different, as well astheir covariance structures, which will be specifiedlater. Second, there are two significant latent variablesand the K-dimensional true coefficient vector γ0 �(0.5,−0.5, 0)T. Third, the error vector ε � ση, wherethe components of the n-dimensional randomvector ηare independent and follow the t-distribution withdf � 10 degrees of freedom. The settings of β0 and(n, p,K, σ) are the same as in the first simulation ex-ample in Section 4.1.1, whereas the dimensionality qof covariates W equals 1,000, which is also large.For the covariance structure of X, we set ΣX �

(0.5|i−j|)1≤i,j≤p to allow for correlation at the populationlevel. By contrast, in order to estimate the principalcomponents in high dimensions, the population co-variance matrix of W should have multiple spikedeigenvalues. Thus, we constructed it using the blockdiagonal structure such that

ΣW � Σ11 0

0 Σ22

( ),

where Σ11 � 3/4(Σ1 + Σ2)1≤i,j≤200 and Σ22 � 1/2 (Σ1 +Σ2)1≤i,j≤800 with the definitions of Σ1 and Σ2 similar tothose in Section 4.1.1 except for different dimensions.Under such construction, the two largest eigenvaluesof ΣW are 201.75 and 77.61, respectively, whereas theothers are less than 2.63. Based on the aforementionedcovariance structures, for each data set, the rows of X

Table 1. Means and Standard Errors (in Parentheses) ofDifferent Performance Measures by All Methods over 200Simulations in Section 4.1.1

Model Measure Lasso SCAD Hard Oracle

M1 PE 65.27 (1.35) 65.29 (1.40) 68.80 (6.45) —L2-loss 1.61 (0.24) 1.61 (0.25) 2.25 (1.07) —L1-loss 4.69 (1.84) 4.70 (1.88) 5.03 (2.31) —L∞-loss 0.65 (0.13) 0.65 (0.15) 1.48 (1.10) —FP 4.45 (7.15) 4.45 (7.16) 0.51 (0.90) —FN 5.93 (0.26) 5.93 (0.26) 5.98 (0.16) —MSC 0 (0) 0 (0) 0 (0) —σ 7.88 (0.57) 7.88 (0.57) 7.78 (0.60) —

M2 PE 0.39 (0.16) 0.19 (0.01) 0.19 (0.01) 0.17 (0.01)L2-loss 0.43 (0.13) 0.13 (0.03) 0.10 (0.03) 0.10 (0.03)L1-loss 1.52 (0.40) 0.44 (0.07) 0.21 (0.13) 0.21 (0.06)L∞-loss 0.23 (0.07) 0.07 (0.02) 0.07 (0.02) 0.07 (0.02)FP 28.79 (6.52) 15.99 (5.63) 0.02 (0.28) 0 (0)FN 0.02 (0.16) 0 (0) 0 (0) 0 (0)MSC 0 (0) 0 (0) 1.00 (0.07) 1 (0)σ 0.47 (0.06) 0.38 (0.03) 0.41 (0.03) 0.40 (0.03)

Note. M1, model with only observable predictors;M2, model includesestimated latent variables; SCAD, smoothly clipped absolute deviation;PE, prediction error; FP, false positives; FN, false negatives; MSC,model selection consistency.

Table 2. Means and Standard Errors (in Parentheses) ofDifferent Performance Measures for Regression Coefficientsof Confounding Factors by All Methods over 200Simulations in Section 4.1.1

Measure Lasso SCAD Hard Oracle

L2-loss 0.02 (0.00) 0.01 (0.00) 0.01 (0.00) 0.00 (0.00)L1-loss 0.02 (0.01) 0.01 (0.00) 0.01 (0.00) 0.00 (0.00)L∞-loss 0.02 (0.00) 0.01 (0.00) 0.01 (0.00) 0.00 (0.00)FPγ 0.29 (0.55) 0.21 (0.43) 0 (0) 0 (0)FNγ 0 (0) 0 (0) 0 (0) 0 (0)MSCγ 0.73 (0.45) 0.79 (0.41) 1 (0) 1 (0)

Notes. The notation 0.00 denotes a number less than 0.005. SCAD,smoothly clipped absolute deviation.


and W were sampled as i.i.d. copies from the corre-sponding multivariate normal distribution.

We included the top K sample principal compo-nents in Model M2 as potential latent factors andcompared the performance of Lasso, SCAD, Hard,and Oracle by the same performance measures asdefined in Section 4.1.1. The results are summarizedin Tables 3 and 4. From Table 3, it is clear that themethods that relied only on the observable predictorsstill suffered a lot under this more difficult setting,where all true predictors were missed, prediction er-rors were large, and the error standard deviation (SD)was poorly estimated. In contrast, the new NSLmethodology via Lasso, SCAD, and Hard was able totackle the issues associated with variable selection,coefficient estimation, prediction, and error SD esti-mation. With the latent variable–augmented ModelM2, Hard almost recovered the exact underlyingmodel. Similar to the first example, in view of Table 4,all methods correctly identified the significant con-founding factors and estimated their effects accu-rately. However, compared with Tables 1 and 2, mostof the performance measures deteriorated in this secondexample. This is mainly due to the relatively heavy-tailedrandom errors, as well as the difficulty in estimatingmultiple high-dimensional principal components.

4.2. Application to Nutrient Intake and Human GutMicrobiome Data

Nutrient intake strongly affects human health anddiseases such as obesity, whereas gut microbiomecomposition is an important factor in energy ex-traction from the diet. We illustrate the usefulness of

our proposed methodology by applying it to the dataset reported by Wu et al. (2011) and previouslystudied by Chen and Li (2013) and Lin et al. (2014),where a cross-sectional study of 98 healthy volunteerswas carried out to investigate the habitual diet effecton the human gut microbiome. The nutrient intakeconsisted of 214 micronutrients collected from thevolunteers by a food frequency questionnaire. Thevalues were normalized by the residual method toadjust for caloric intake and then standardized tohave mean zero and SD one. Similar to Chen andLi (2013), we used one representative for a set ofhighly correlated micronutrients whose correlationcoefficients are larger than 0.9, resulting in 119 rep-resentative micronutrients in total. Furthermore, stoolsamples were collected, and DNA samples were an-alyzed by Roche 454 pyrosequencing of 16S rDNAgene segments from the V1–V2 region. After taxo-nomic assignment of the denoised pyrosequences,the operational taxonomic units were combined into87 genera that appeared in at least one sample.We areinterested in identifying the important micronutrientsand potential latent factors from the gut microbiomegenera that are associated with the BMI.Because of the high correlations between the micro-

nutrients, we applied NSL via the elastic net (Zou andHastie 2005) to this data set by treating BMI, nutrientintake, and gut microbiome composition (after thecentered log-ratio transformation; Aitchison 1983)as the response, predictors, and covariates of con-founding factors, respectively. The data set was split100 times into a training set of 60 samples and avalidation set of the remaining samples. For each

Table 3. Means and Standard Errors (in Parentheses) of Different Performance Measuresby All Methods over 200 Simulations in Section 4.1.2

Model Measure Lasso SCAD Hard Oracle

M1 PE 72.33 (1.53) 72.33 (1.53) 76.04 (6.62) —L2-loss 1.58 (0.24) 1.58 (0.24) 2.25 (1.09) —L1-loss 4.49 (1.86) 4.49 (1.86) 5.00 (2.10) —L∞-loss 0.64 (0.13) 0.64 (0.13) 1.50 (1.15) —FP 3.59 (6.52) 3.59 (6.52) 0.49 (0.72) —FN 5.95 (0.23) 5.95 (0.23) 6.00 (0.07) —MSC 0 (0) 0 (0) 0 (0) —Error SD 8.33 (0.59) 8.33 (0.59) 8.20 (0.63) —

M2 PE 1.74 (1.08) 1.10 (1.05) 1.04 (0.99) 0.22 (0.01)L2-loss 0.70 (0.22) 0.25 (0.22) 0.16 (0.18) 0.11 (0.03)L1-loss 2.18 (0.50) 0.87 (0.50) 0.39 (0.53) 0.23 (0.07)L∞-loss 0.37 (0.12) 0.13 (0.10) 0.10 (0.09) 0.08 (0.03)FP 20.63 (13.60) 23.29 (12.52) 0.70 (3.34) 0 (0)FN 0.09 (0.38) 0.15 (0.94) 0.09 (0.63) 0 (0)MSC 0 (0) 0.09 (0.29) 0.92 (0.27) 1 (0)Error SD 0.74 (0.20) 0.48 (0.21) 0.50 (0.12) 0.45 (0.04)

Notes. M1, model with only observable predictors; M2, model includes estimated latent variables.The population error standard deviation σ

df /(df − 2)√

equals to 0.45. SCAD, smoothly clippedabsolute deviation; PE, prediction error; FP, false positives; FN, false negatives; MSC, modelselection consistency; SD, standard deviation.


splitting of the data set, we explored two differentmodels,M1 andM2, as defined in Section 4.1, with thetop 20 sample principal components of gut micro-biome composition included in Model M2 to estimatethe potential latent factors. All predictors were rescaledto have a common L2-norm of n1/2, and the tuningparameter was chosen by minimizing the predictionerror calculated on the validation set. We summarizein Table 5 the selection probabilities and coefficientsof the significant micronutrients and latent variableswhose selection probabilities were greater than 0.9 inM1 or greater than 0.85 in M2. The means (withstandard errors in parentheses) of the prediction errorsaveraged over 100 random splittings were 167.9 (7.2)in ModelM1 and 110.3 (4.0) in ModelM2, whereas themedian model size also decreased from 93 to 69 afterapplying the NSL methodology. This shows that theprediction performance was improved after using theinformation on gut microbiome genera.

In view of the model selection results in Table 5,many significant micronutrients in Model M1 becameinsignificant after adjusting for the latent substructures,which implies that either they affect BMI through thegut microbiome genera or their combinative effects arecaptured by the latent variables. This was also evi-denced by the reduction in model size mentioned

earlier. Moreover, the effects of some micronutrientschanged signs in Model M2, and the subsequent as-sociations with BMI are consistent with scientificdiscoveries (Gul et al. 2017). For instance, aspartameis a sugar substitute widely used in beverages such asdiet soda, and it was negatively associated with BMIinModelM1 but tended to share a positive associationafter accounting for the gut microbiome genera. Apotential reason is that the people who drink dietsoda can have a relatively healthy habitual diet andgut microbiome composition that, in turn, lower theBMI, but the diet soda itself does not reduce fat.Similar phenomena happened with both acrylamideand vitamin E as well.We also applied the model-free knockoffs (Candes

et al. 2018) with a target false discovery rate (FDR)level of 0.2 to Model M2, and the most significantfactors identified were the latent variables of 7th and9th principal components, whichmay be explained asBMI-associated enterotypes while adjusting for nu-trient intake (Arumugam et al. 2011, Wu et al. 2011).Themajor gutmicrobiome genera in the compositionsof these two latent variables are displayed in Table 6.At the phylum level, the latent factors mainly con-sist of Bacteroidetes and Firmicutes, whose relativeproportion has been shown to affect human obesity(Ley et al. 2006). In view of the associations with BMI,both the 7th and 9th principal components confirmthe claim that the Firmicutes-enriched microbiomeholds a greater metabolic potential for energy gainfrom the diet that results in weight gain (Turnbaughet al. 2006). Furthermore, one of the major micro-biome genera in the latent factor of the 9th principalcomponent, Acidaminococcus, was also found to bepositively associated with BMI by Lin et al. (2014),who show that human obesity can be affected at thegenus level.

5. DiscussionIn this paper, we have introduced a new methodol-ogy, NSL, for prediction and variable selection in the

Table 5. Selection Probabilities and Rescaled Coefficients (in Parentheses) of the MostFrequently Selected Predictors by Each Model Across 100 Random Splittings in Section 4.2

Predictor Model M1 Model M2 Predictor Model M1 Model M2

Sodium 0.98 (1.35) 0.67 (0.55) PC(7th) — 0.99 (1.76)Eicosenoic acid 0.98 (–2.47) 0.80 (–1.24) — — 0.96 (–1.21)Vitamin B12 0.96 (0.43) 0.62 (0.30) Apigenin 0.95 (–1.67) 0.93 (–1.88)Gallocatechin 0.96 (–4.81) 0.84 (–1.70) PC(9th) — 0.88 (–0.87)Riboflavin pills 0.94 (1.71) 0.55 (0.61) PC(10th) — 0.86 (0.78)Acrylamide 0.94 (–0.34) 0.62 (0.32) Iron 0.93 (1.22) 0.86 (0.75)Naringenin 0.94 (1.11) 0.58 (0.32) Aspartame 0.93 (–0.46) 0.79 (0.59)Pelargonidin 0.94 (–1.15) 0.75 (–1.03) Vitamin C 0.93 (–0.71) 0.76 (–0.39)Lauric acid 0.93 (1.88) 0.71 (0.50) Vitamin E 0.92 (0.45) 0.65 (–0.29)

Note. M1, model with only micronutrients as predictors; M2, model includes latent variables from gutmicrobiome composition; PC, principal component.

Table 4. Means and Standard Errors (in Parentheses) ofDifferent Performance Measures for Regression Coefficientsof Confounding Factors by All Methods over 200Simulations in Section 4.1.2

Measure Lasso SCAD Hard Oracle

L2-loss 0.08 (0.03) 0.07 (0.04) 0.07 (0.04) 0.01 (0.00)L1-loss 0.10 (0.04) 0.09 (0.05) 0.08 (0.05) 0.01 (0.00)L∞-loss 0.08 (0.03) 0.06 (0.03) 0.06 (0.03) 0.01 (0.00)FPγ 0.21 (0.45) 0.34 (0.60) 0.01 (0.10) 0 (0)FNγ 0 (0) 0 (0) 0 (0) 0 (0)MSCγ 0.81 (0.39) 0.72 (0.45) 0.99 (0.10) 1 (0)

Notes. The notation 0.00 denotes a number less than 0.005. SCAD,smoothly clipped absolute deviation,


presence of nonsparse coefficient vectors through thefactors plus sparsity structure, where latent variablesare exploited to capture the nonsparse combinationsof either the original predictors or additional cova-riates. The suggested methodology is ideal for theapplications involving two sets of features that arestrongly correlated, as in our BMI study. Both theo-retical guarantees and empirical performance of thepotential latent family incorporating population princi-pal components have been demonstrated. And ourmethodology is also applicable to more general fam-ilies with properly estimated latent variables andidentifiable models.

It would be interesting to further investigate sev-eral problems, such as hypothesis testing and FDRcontrol in nonsparse learning by the idea of NSL.Based on the established model identifiability con-dition that characterizes the correlations betweenobservable and latent predictors, hypothesis testingcan proceed using the debiasing idea in JavanmardandMontanari (2014), van deGeer et al. (2014), Zhangand Zhang (2014). FDR could be controlled by ap-plying the knockoffs inference procedures (Barberand Candes 2015, Candes et al. 2018, Fan et al.2020) on the latent variable–augmented model. Themain difficulty lies in analyzing how the estimationerrors of unobservable factors affect the correspondingprocedures. Another possible direction is to exploremore general ways of modeling the latent variables todeal with the nonsparse coefficient vectors. Theseproblems are beyond the scope of this paper and willbe interesting topics for future research.

AcknowledgmentsThe authors sincerely thank the editors and referees for theirvaluable comments that helped improve the article substantially.

ReferencesAitchison J (1983) Principal component analysis of compositional

data. Biometrika 70(1):57–65.Alter O, Brown P, Botstein D (2000) Singular value decomposition

for genome-wide expression data processing and modeling.Proc. Natl. Acad. Sci. USA 97(18):10101–10106.

Artemiou A, Li B (2009) On principal components and regression: Astatistical explanation of a natural phenomenon. Statist. Sinica19(4):1557–1565.

ArumugamM, Raes J, Pelletier E, Le Paslier D, Yamada T,MendeDR,Fernandes GR, et al. (2011) Enterotypes of the human gutmicrobiome. Nature 473(7346):174–180.

Bair E, Hastie T, Paul D, Tibshirani R (2006) Prediction by supervisedprincipal components. J. Amer. Statist. Assoc. 101(437):119–137.

Barber R, Candes EJ (2015) Controlling the false discovery rate viaknockoffs. Ann. Statist. 43(5):2055–2085.

Belloni A, Chernozhukov V,Wang L (2011) Square-root lasso: Pivotalrecovery of sparse signals via conic programming. Biometrika98(4):791–806.

Belloni A, Chernozhukov V, Chetverikov D, Wei Y (2018) Uni-formly valid post-regularization confidence regions for manyfunctional parameters in Z-estimation framework. Ann. Statist.46(6B):3643–3675.

Belloni A, Chernozhukov V, Fernandez-Val I, Hansen C (2017)Program evaluation and causal inference with high-dimensionaldata. Econometrica 85(1):233–298.

Bickel PJ, Ritov Y, Tsybakov AB (2009) Simultaneous analysis ofLasso and Dantzig selector. Ann. Statist. 37(4):1705–1732.

Bing X, Bunea F, Ning Y, Wegkamp M (2020) Sparse latent factormodels with pure variables for overlapping clustering. Ann.Statist. 48(4):2055–2081.

Bing X, Bunea F, Wegkamp M, Strimas-Mackey S (2019) Essentialregression. arXiv:1905.12696.

Boyle E, Li Y, Pritchard J (2017) An expanded view of complex traits:From polygenic to omnigenic. Cell 169(7):1177–1186.

Bradic J, Fan J, Zhu Y (2020) Testability of high-dimensional linearmodels with non-sparse structures. Ann. Statist. Forthcoming.

Bühlmann P, van de Geer S (2011) Statistics for High-Dimensional Data:Methods, Theory and Applications (Springer, Berlin).

Candes EJ, Fan Y, Janson L, Lv J (2018) Panning for gold: “Model-X”knockoffs for high-dimensional controlled variable selection.J. Royal Statist. Soc. B 80(3):551–577.

Candes EJ, Tao T (2007) The Dantzig selector: Statistical estimationwhen p is much larger than n (with discussion). Ann. Statist.35(6):2313–2404.

Cavalli-Sforza LL, Menozzi P, Piazza A (1993) Demic expansions andhuman evolution. Science 259(5095):639–646.

Chandrasekaran V, Parrilo PA, Willsky AS (2012) Latent variablegraphical model selection via convex optimization (with dis-cussion). Ann. Statist. 40(4):1935–1967.

Chen J, Li H (2013) Variable selection for sparseDirichlet-multinomialregression with an application to microbiome data analysis.Ann. Appl. Stat. 7(1):418–442.

ChenM, Ren Z, ZhaoH, ZhouHH (2016) Asymptotically normal andefficient estimation of covariate-adjusted Gaussian graphicalmodel. J. Amer. Statist. Assoc. 111513:394–406.

Cook RD (2007) Fisher lecture: Dimension reduction in regression.Statist. Sci. 22(1):1–40.

Fan J, Li R (2001) Variable selection via nonconcave penalizedlikelihood and its oracle properties. J. Amer. Statist. Assoc.96(456):1348–1360.

Fan J, Lv J (2010) A selective overview of variable selection in highdimensional feature space. Statist. Sinica 20(1):101–148.

Fan J, Lv J (2011) Nonconcave penalized likelihood with NP-di-mensionality. IEEE Trans. Inform. Theory 57(8):5467–5484.

Fan J, Guo S, Hao N (2012) Variance estimation using refitted cross-validation in ultrahigh dimensional regression. J. Royal Statist.Soc. B 74(1):37–65.

Fan J, Liao Y, Micheva M (2013) Large covariance estimation bythresholding principal orthogonal complements (with discus-sion). J. Royal Statist. Soc. B 75(4):603–680.

Table 6. Major Gut Microbiome Genera in theCompositions of the Two Significant Latent VariablesIdentified by the Model-Free Knockoffs in Section 4.2

Latent variable Phylum Genus Weight

PC(7th) Firmicutes Dialister −0.40Firmicutes Eubacterium 0.39Bacteroidetes Barnesiella −0.28

PC(9th) Firmicutes Acidaminococcus −0.51Firmicutes Megasphaera −0.36Firmicutes Ruminococcus −0.30

Note. PC, principal component.


Fan Y, Lv J (2013) Asymptotic equivalence of regularization methodsin thresholded parameter space. J. Amer. Statist. Assoc. 108(503):247–264.

Fan Y, Demirkaya E, Li G, Lv J (2020) RANK: Large-scale inference withgraphical nonlinear knockoffs. J. Amer. Statist. Assoc. 115(529):362–379.

Gul S, Hamilton AR, Munoz AR, Phupitakphol T, Liu W, Hyoju SK,Economopoulos KP, et al. (2017) Inhibition of the gut enzymeintestinal alkaline phosphatase may explain how aspartamepromotes glucose intolerance and obesity in mice. Appl. Physiol.Nutrition Metabolism 42(1):77–83.

Hastie T, Tibshirani R, Friedman J (2009) The Elements of StatisticalLearning: DataMining, Inference, and Prediction, 2nd ed. (Springer,New York).

Hsu HL, Ing CK, Tong H (2019) On model selection from a finitefamily of possibly misspecified time series models. Ann. Statist.47(2):1061–1087.

Javanmard A, Montanari A (2014) Confidence intervals and hy-pothesis testing for high-dimensional regression. J. MachineLearn. Res. 15(82):2869–2909.

Johnstone I (2001) On the distribution of the largest eigenvalue inprincipal components analysis. Ann. Statist. 29(2):295–327.

Johnstone I, Lu A (2009) On consistency and sparsity for principalcomponents analysis in high dimensions. J. Amer. Statist. Assoc.104(486):682–693.

Jung S, Marron J (2009) PCA consistency in high dimension, lowsample size context. Ann. Statist. 37(6):4104–4130.

Kneip A, Sarda P (2011) Factor models and variable selection in high-dimensional regression analysis. Ann. Statist. 39(5):2410–2447.

Leek J, Storey J (2007) Capturing heterogeneity in gene expressionstudies by surrogate variable analysis. PLoS Genetics 3(9):1724–1735.

Ley RE, Turnbaugh PJ, Klein S, Gordon JI (2006) Human gutmicrobesassociated with obesity. Nature 444(7122):1022–1023.

Lin W, Feng R, Li H (2015) Regularization methods for high-dimensional instrumental variables regression with an appli-cation to genetical genomics. J. Amer. Statist. Assoc. 110(509):270–288.

LinW, Shi P, Feng R, Li H (2014) Variable selection in regression withcompositional covariates. Biometrika 101(4):785–797.

Lv J (2013) Impacts of high dimensionality in finite samples. Ann.Statist. 41(4):2236–2262.

Lv J, Liu JS (2014) Model selection principles in misspecified models.J. Royal Statist. Soc. B 76(1):141–167.

Mardia K, Kent J, Bibby J (1979) Multivariate Analysis (AcademicPress, New York).

Menozzi P, Piazza A, Cavalli-Sforza LL (1978) Syntheticmaps of humangene frequencies in Europeans. Science 201(4358):786–792.

Pan D, He H, Song X, Sun L (2015) Regression analysis of additivehazards model with latent variables. J. Amer. Statist. Assoc.110(511):1148–1159.

Paul D (2007) Asymptotics of sample eigenstructure for a large di-mensional spiked covariance model. Statist. Sinica 17(4):1617–1642.

Paulson C, Luo L, James G (2018) Efficient large-scale internet mediaselection optimization for online display advertising. J. MarketingRes. 55(4):489–506.

Price A, Patterson N, Plenge R, Weinblatt M, Shadick N, Reich D(2006) Principal components analysis corrects for stratificationin genome-wide association studies. Nat. Genetics 38(8):904–909.

Pritchard J (2001) Are rare variants responsible for susceptibility tocomplex diseases? Amer. J. Human Genetics 69(1):124–137.

Radchenko P, James G (2008) Variable inclusion and shrinkage al-gorithms. J. Amer. Statist. Assoc. 103(483):1304–1315.

Shen D, Shen H, Marron J (2016) A general framework for con-sistency of principal component analysis. J. Machine Learn. Res.17(150):5218–5251.

Sun T, Zhang C-H (2012) Scaled sparse linear regression. Biometrika99(4):879–898.

Tang CY, Leng C (2010) Penalized high-dimensional empiricallikelihood. Biometrika 97(4):905–920.

Tibshirani R (1996) Regression shrinkage and selection via the Lasso.J. Royal Statist. Soc. B 58(1):267–288.

Turnbaugh PJ, Ley RE, Mahowald MA, Magrini V, Mardis ER,Gordon JI (2006) An obesity-associated gut microbiome with in-creased capacity for energy harvest. Nature 444(7122):1027–1031.

Uematsu Y, Tanaka S (2019) High-dimensional macroeconomicforecasting and variable selection via penalized regression.Econom. J. 22(1):34–56.

van de Geer S, Bühlmann P, Ritov Y, Dezeure R (2014) On as-ymptotically optimal confidence regions and tests for high-dimensional models. Ann. Statist. 42(3):1166–1202.

Wang W, Fan J (2017) Asymptotics of empirical eigen-structurefor high dimensional spiked covariance. Ann. Statist. 45(3):1342–1374.

White H (1982) Maximum likelihood estimation of misspecifiedmodels. Econometrica 50(1):1–25.

Wu GD, Chen J, Hoffmann C, Bittinger K, Chen Y-Y, Keilbaugh SA,Bewtra M, et al. (2011) Linking long-term dietary patterns withgut microbial enterotypes. Science 334(6052):105–108.

Xu H, Caramanis C, Mannor S (2016) Statistical optimization in highdimensions. Oper. Res. 64(4):958–979.

Zhang S, Zhang C-H (2014) Confidence intervals for low dimensionalparameters in high dimensional linear models. J. Royal Statist.Soc. B 76(1):217–242.

Zhao P, Yu B (2006) On model selection consistency of Lasso.J. Machine Learn. Res. 7(November):2541–2563.

Zheng Z, Fan Y, Lv J (2014) High-dimensional thresholded regressionand shrinkage effect. J. Royal Statist. Soc. B 76(3):627–649.

Zhu Y, Bradic J (2018) Linear hypothesis testing in dense high-dimensional linear models. J. Amer. Statist. Assoc. 113(524):1583–1600.

Zou H, Hastie T (2005) Regularization and variable selection via theelastic net. J. Royal Statist. Soc. B 67(2):301–320.

Zemin Zheng is a professor in the Department of Sta-tistics and Finance of the School of Management at theUniversity of Science and Technology of China (USTC)and a professor in the School of Data Science and Inter-national Institute of Finance at the USTC. His researchinterests include statistics, machine learning, data science,and business applications.

Jinchi Lv is the Kenneth King Stonier Chair in BusinessAdministration, and professor in the Data Sciences andOperations Department of the Marshall School of Business atthe University of Southern California (USC), a professor in theDepartment of Mathematics at USC, and an associate fellowof USC Dornsife Institute for New Economic Thinking. Heis the recipient of an Adobe Data Science Research Award(2017), the Royal Statistical Society Guy Medal in Bronze(2015), and the National Science Foundation Faculty EarlyCareer Development Award (2010).

Wei Lin is an associate professor in the Department ofProbability and Statistics of the School of MathematicalSciences and Center for Statistical Science at Peking Uni-versity. His research interests lie in the broad areas of high-dimensional statistics and statistical machine learning, withparticular emphasis on compositional data analysis andstatistical learning from high-dimensional complex data.


nonsparse learning with latent variables

Documents