Optimal uniform convergence rates and asymptotic normality for series estimators under weak dependence and weak conditions

Abstract

We show that spline and wavelet series regression estimators for weakly dependent regressors attain the optimal uniform (i.e. sup-norm) convergence rate ${(n/logn)}^{-p/(2p+d)}$ of Stone (1982), where $d$ is the number of regressors and $p$ is the smoothness of the regression function. The optimal rate is achieved even for heavy-tailed martingale difference errors with finite $(2+(d/p))$th absolute moment for $d/p<2$. We also establish the asymptotic normality of $t$ statistics for possibly nonlinear, irregular functionals of the conditional mean function under weak conditions. The results are proved by deriving a new exponential inequality for sums of weakly dependent random matrices, which is of independent interest.

Introduction

We study the nonparametric regression model

$$\begin{array}{c}{Y}_i=h_0\left(X_i\right)+{ϵ}_i\hfill \\ E\left[{ϵ}_i\right|X_i]=0\hfill \end{array}$$

where $Y_i\in \mathbb{R}$ is a scalar response variable, $X_i\in \mathcal{X}\subseteq {\mathbb{R}}^d$ is a $d$-dimensional regressor (predictor variable), and the conditional mean function $h_0\left(x\right)=E\left[Y_i\right|X_i=x]$ belongs to a Hölder space of smoothness $p>0$. We are interested in series least squares (LS) estimation¹ of $h_0$ under sup-norm loss and inference on possibly nonlinear functionals of $h_0$ allowing for weakly dependent regressors and heavy-tailed errors ${ϵ}_i$.²

For i.i.d. data,  Stone (1982) shows that ${(n/logn)}^{-p/(2p+d)}$ is the minimax lower bound in sup-norm risk for estimation of $h_0$ over a Hölder ball of smoothness $p>0$. For strictly stationary beta-mixing regressors, we show that spline and wavelet series LS estimators $\widehat{h}$ of $h_0$ attain the optimal uniform rate of  Stone (1982) under a mild unconditional moment condition $E\left[{\left|{ϵ}_i\right|}^{2+(d/p)}\right]<\infty$ imposed on the martingale difference errors.

More generally, we assume the error process ${\left\{{ϵ}_i\right\}}_{i=-\infty }^{\infty }$ is a martingale difference sequence but impose no explicit weak dependence condition on the regressor process ${\left\{X_i\right\}}_{i=-\infty }^{\infty }$. Rather, weak dependence of the regressor process is formulated in terms of convergence of a certain random matrix. We verify this condition for absolutely regular (beta-mixing) sequences by deriving a new exponential inequality for sums of weakly dependent random matrices. This new inequality then leads to a sharp upper bound on the sup-norm variance term of series LS estimators with an arbitrary basis. When combined with a general upper bound on the sup-norm bias term of series LS estimators, the sharp sup-norm variance bound immediately leads to a general upper bound on the sup-norm convergence rate of series LS estimators with an arbitrary basis and weakly dependent data.

In our sup-norm bias and variance decomposition of series LS estimators, the bound on the sup-norm bias term depends on the sup norm of the empirical $L^2$ projection onto the linear sieve space. The sup norm of the empirical $L^2$ projection varies with the choice of the (linear sieve) basis. For spline regression with i.i.d. data,  Huang (2003b) shows that the sup norm of the empirical $L^2$ projection onto splines is bounded with probability approaching one (wpa1). Using our new exponential inequality for sums of weakly dependent random matrices, his bound is easily extended to spline regression with weakly dependent regressors. In addition, we show in Theorem 5.2 that, for either i.i.d. or weakly dependent regressors, the sup norm of the empirical $L^2$ projection onto compactly supported wavelet bases is also bounded wpa1 (this property is called sup-norm stability of empirical $L^2$ projection). These tight bounds lead to sharp sup-norm bias control for spline and wavelet series LS estimators. They allow us to show that spline and wavelet series LS estimators achieve the optimal sup-norm convergence rate with weakly dependent data and heavy-tailed errors (e.g.,  $E\left[{ϵ}_i^4\right]=\infty$ is allowed).

Sup-norm (uniform) convergence rates of series LS estimators have been studied previously by  Newey (1997) and  de Jong (2002) for i.i.d. data and  Lee and Robinson (2013) for spatially dependent data. But the uniform convergence rates obtained in these papers are slower than the optimal rate of  Stone (1982).³ In an important paper on series LS regression with i.i.d. data,  Belloni et al. (2014) establish the attainability of the optimal sup-norm rates of series LS estimators using spline, local polynomial partition, wavelet and other series possessing the property of sup-norm stability of the $L^2$ projection (or bounded Lebesgue constant) under the conditional moment condition ${sup}_{x}E\left[{\left|{ϵ}_i\right|}^{2+\delta }\right|X_i=x]<\infty$ for some $\delta >d/p$. Our Theorem 5.1 on the sup-norm stability of $L^2$ projection of the wavelet basis is used by  Belloni et al. (2014) to show that the wavelet series LS estimator achieves the optimal sup-norm rate under their conditional moment requirement. Our paper contributes to the literature by showing that spline and wavelet series LS estimators attain the optimal sup-norm rate with strictly stationary beta-mixing regressors under the weaker unconditional moment requirement $E\left[{\left|{ϵ}_i\right|}^{2+(d/p)}\right]<\infty$.

As another application of our new exponential inequality, under very weak conditions we obtain sharp $L^2$ convergence rates for series LS estimators with weakly dependent regressors. For example, under the minimal bounded conditional second moment restriction (${sup}_{x}E\left[{\left|{ϵ}_i\right|}^2\right|X_i=x]<\infty$), our $L^2$-norm rates for trigonometric polynomial, spline or wavelet series LS estimators attain  Stone (1982)’s optimal $L^2$-norm rate of $n^{-p/(2p+d)}$ with strictly stationary, exponentially beta-mixing (respectively algebraically beta-mixing at rate $\gamma$) regressors with $p>0$ (resp. $p>d/\left(2\gamma \right)$), while the power series LS estimator attains the same optimal rate with exponentially (resp. algebraically) beta-mixing regressors for $p>d/2$ (resp. $p>d(2+\gamma )/\left(2\gamma \right)$). It is interesting to note that for a smooth conditional mean function, we obtain the optimal $L^2$ convergence rates for these commonly used series LS estimators with weakly dependent regressors without requiring the existence of higher-than-second unconditional moments of the error terms. Previously,  Newey (1997) derived the optimal $L^2$ convergence rates of series LS estimator under i.i.d. data and the restriction of $K^2/n=o\left(1\right)$ for spline and trigonometric series (and $K^3/n=o\left(1\right)$ for power series), where $K$ is the series number of terms. The restriction on $K$ is relaxed to $K(logK)/n=o\left(1\right)$ in  Huang (2003a) for splines and in  Belloni et al. (2014) for wavelets, trigonometric and other series under i.i.d. data. We show that the optimal $L^2$ convergence rates are still attainable for splines, wavelets, trigonometric and other series under exponentially beta-mixing and $K{(logK)}^2/n=o\left(1\right)$.

We also show that feasible asymptotic inference can be performed on a possibly nonlinear functional $f\left(h_0\right)$ using the plug-in series LS estimator $f\left(\widehat{h}\right)$. We establish the asymptotic normality of $f\left(\widehat{h}\right)$ and of the corresponding Student t statistic for weakly dependent data under mild low-level conditions. When specializing to general irregular (i.e., slower than $\sqrt{n}$-estimable) but sup-norm bounded linear functionals of spline or wavelet series LS estimators with i.i.d. data, we obtain the asymptotic normality of $f\left(\widehat{h}\right)$

$$\frac{\sqrt{n}(f\left(\widehat{h}\right)-f\left(h_0\right))}{V_K^{1/2}}{\to }_{d}N(0,1)$$

under remarkably mild conditions of (1) uniform integrability (${sup}_{x\in \mathcal{X}}E\left[{ϵ}_i^2\{\left|{ϵ}_i\right|>\ell \left(n\right)\}\right|X_i=x]\to 0$ for any $\ell \left(n\right)\to \infty$ as $n\to \infty$), and (2) $K^{-p/d}\sqrt{n/V_K}=o\left(1\right)$, $(KlogK)/n=o\left(1\right)$, where $V_K$ is the sieve variance that grows with $K$ for irregular functionals. These conditions coincide with the weakest known conditions in  Huang (2003b) for the pointwise asymptotic normality of spline LS estimators, except we also allow for other irregular linear functionals of spline or wavelet LS estimators.⁴ When specializing to general irregular but sup-norm bounded nonlinear functionals of spline or wavelet series LS estimators with i.i.d. data, we obtain asymptotic normality of $f\left(\widehat{h}\right)$ (and of its t statistic) under conditions (1) and (3) $K^{-p/d}\sqrt{n/V_K}=o\left(1\right)$, $K^{(2+\delta )/\delta }(logn)/n\lesssim 1$ (and $K^{(2+\delta )/\delta }(logn)/n=o\left(1\right)$ for the t statistic) for $\delta \in (0,2)$ such that $E\left[{\left|{ϵ}_i\right|}^{2+\delta }\right]<\infty$. These conditions are much weaker than the well-known conditions in  Newey (1997) for the asymptotic normality of a nonlinear functional and its t statistic of spline LS estimator, namely $K^{-p/d}\sqrt{n}=o\left(1\right)$, $K^4/n=o\left(1\right)$ and ${sup}_{x}E\left[{\left|{ϵ}_i\right|}^4\right|X_i=x]<\infty$. Moreover, under a slightly more restrictive growth condition on $K$ but without the need to increase $\delta$, we show that our mild sufficient conditions for the i.i.d. case extend naturally to the weakly dependent case. Our conditions for the weakly dependent case relax the higher-order-moment requirement in  Chen et al. (2014) for sieve $t$ inference on nonlinear functionals of linear sieve time series LS regressions.

Our paper improves upon the existing results on the asymptotic normality of $t$ statistics of possibly nonlinear functionals of a series LS estimator under dependent data by allowing for heavy-tailed errors ${ϵ}_i$ and relaxing the growth rates of the series term $K$ but maintaining the bounded conditional error variance assumption. For i.i.d. data,  Hansen (2014) derives pointwise asymptotic normality for linear functionals of a series LS estimator allowing for unbounded regressors, unbounded conditional error variance, but requiring $E\left[{\left|{ϵ}_i\right|}^{4+\eta }\right]<\infty$ for some $\eta >0$. In addition to pointwise limiting distribution results,  Belloni et al. (2014) also provide uniform limit theory and uniform confidence intervals for linear functionals of a series LS estimator with i.i.d. data.

Our paper,  Belloni et al. (2014) and  Hansen (2014) all employ tools from recent random matrix theory to derive various new results for series LS estimation.  Belloni et al. (2014) are the first to apply the non-commutative Khinchin random matrix inequality for i.i.d. data. Instead, we apply an exponential inequality for sums of independent random matrices due to  Tropp (2012). Our results for series LS with weakly dependent data rely crucially on our extension of Tropp’s matrix exponential inequality from i.i.d. data to weakly dependent data. See  Hansen (2014) for other applications of Tropp’s inequality to series LS estimators with i.i.d. data.

Since economic and financial time series data often have infinite forth moments, the new improved rates and inference results in our paper should be very useful to the literatures on nonparametric estimation and testing of nonlinear time series models (see, e.g.,  Robinson, 1989, Li et al., 2003, Fan and Yao, 2003, Chen, 2013). Moreover, our new exponential inequality for sums of weakly dependent random matrices should be useful in series LS estimation of spatially dependent models and in other contexts as well.⁵

The rest of the paper is organized as follows. Section  2 first derives general upper bounds on the sup-norm convergence rates of series LS estimators with an arbitrary basis. It then shows that spline and wavelet series LS estimators attain the optimal sup-norm rates, allowing for weakly dependent data and heavy tailed error terms. It also presents general sharp $L^2$-norm convergence rates of series LS estimators with an arbitrary basis under very mild conditions. Section  3 provides the asymptotic normality of sieve $t$ statistics for possibly nonlinear functionals of $h_0$. Section  4 provides new exponential inequalities for sums of weakly dependent random matrices, and a reinterpretation of equivalence of the theoretical and empirical $L^2$ norms as a criterion regarding convergence of a certain random matrix. Section  5 shows the sup-norm stability of the empirical $L^2$ projections onto compactly supported wavelet bases, which provides a tight upper bound on the sup-norm bias term for the wavelet series LS estimator. The results in Sections  4 Useful results on random matrices, 5 Sup-norm stability of are of independent interest. Section  6 contains a brief review of spline and wavelet sieve bases. Proofs and ancillary results are presented in Section  7.

Notation: Let ${\lambda }_{\min }(\cdot )$ and ${\lambda }_{\max }(\cdot )$ denote the smallest and largest eigenvalues, respectively, of a matrix. The exponent ${}^{-}$ denotes the Moore–Penrose generalized inverse. $\Vert \cdot \Vert$ denotes the Euclidean norm when applied to vectors and the matrix spectral norm (i.e., largest singular value) when applied to matrices, and ${\Vert \cdot \Vert }_{{\ell }^p}$ denotes the ${\ell }^p$ norm when applied to vectors and its induced operator norm when applied to matrices (thus $\Vert \cdot \Vert ={\Vert \cdot \Vert }_{{\ell }^2}$). If $\{a_n:n\ge 1\}$ and $\{b_n:n\ge 1\}$ are two sequences of non-negative numbers, $a_n\lesssim b_n$ means there exists a finite positive $C$ such that $a_n\le C{b}_n$ for all $n$ sufficiently large, and $a_n\asymp b_n$ means $a_n\lesssim b_n$ and $b_n\lesssim a_n$. $\#\mathcal{S}$ denotes the cardinality of a set $\mathcal{S}$ of finitely many elements. Given a strictly stationary process $\left\{X_i\right\}$ and $1\le p<\infty$, we let $L^p\left(X\right)$ denote the function space consisting of all (equivalence classes) of measurable functions $f$ for which the $L^p\left(X\right)$ norm ${\Vert f\Vert }_{L^p\left(X\right)}\equiv E{\left[{\left|f\left(X_i\right)\right|}^p\right]}^{1/p}$ is finite, and we let $L^{\infty }\left(X\right)$ denote the space of bounded functions under the sup norm ${\Vert \cdot \Vert }_{\infty }$, i.e., if $f:\mathcal{X}\to \mathbb{R}$ then ${\Vert f\Vert }_{\infty }\equiv {sup}_{x\in \mathcal{X}}\left|f\left(x\right)\right|$.