Consider a linear regression problem with elastic net regularization. THe optimization objective is: \[\arg\min_{\beta}||Y-X\beta||_2^2+\lambda\Big(\alpha||\beta||_1+(1-\alpha)||\beta||_2^2\Big)\]
In many practical applications, it may be desirable to impose additional structure on the model coefficients. For instance, one might want to enforce monotonicity, apply budget constraints, or–our focus here–limit the contribution of each covariates to the final prediction. This is especially relevant in settings where:
We define the contribution of covariates \(i\) as the absolute value of the product \(\beta_iZ_i\), where \(Z_i\) is a data matrix used to evaluate feature contributions, and seek to penalize contributions that exceed a specific upper bound \(L_i\). This leads to either a hard-constrained formulation: \[\arg\min_{\beta}||Y-X\beta||_2^2+\lambda\Big(\alpha||\beta||_1+(1-\alpha)||\beta||_2^2\Big)\quad\text{s.t.}\quad \beta_iZ_i\leq L\quad\forall i\] or a soft-constrained formulation, where the violation of contribution caps is penalized: \[\arg\min_{\beta}||Y-X\beta||_2^2+\lambda\Big(\alpha||\beta||_1+(1-\alpha)||\beta||_2^2\Big)+\mu\sum_{i=1}^k\Big|\Big|\max\big(0,|\beta_iZ_i|-L\big)\Big|\Big|_p^p\] where \(k\) is the number of covariates and \(L\) is the contribution cap vector. When \(p=2\), the cap penalty becomes a sum of squared violations, which is differentiable almost everywhere and easier to optimize compared to an \(\mathcal{l}_1\)-based formulation. While the overall penalty remains non-smooth due to the \(\max(0,\cdot)\), the use of squared penalty reduces sensitivity near the boundary and facilitates convergence with quasi-Newton methods.
This formulation resembles generalized LASSO-type problems (Tibshirani and Taylor 2011) and constrained penalization frameworks (James, Paulson, and Rusmevichientong 2020), but with an important difference: the penalty is applied to the feature contributions, not the coefficients themselves. Because the capped contributions involve data-dependent terms (\(\beta_iX_i\)) and a non-linear transformation, there is no simple matrix \(D\) such that this penalty reduces to a standard form \(|\|D\beta\||_1\).
The inclusion of the contribution cap penalty renders off-the-shelf
solvers like glmnet unsuitable. Instead, we turn to
general-purpose optimizers:
To align with glmnet conventions, we rewrite the loss in
a normalized form as \[\min_{\beta_0,\beta}\frac{1}{2N}\sum_{i=1}^N\Big(y_i-(\beta_0+x_i^T\beta)\Big)^2+\lambda\bigg[\frac{(1-\alpha)}{2}||\beta||_2^2+\alpha||\beta||_1\bigg]+\mu\sum_{i=1}^k\max\big(0,|\beta_iZ_i|-L\big)^2\]
To see how capnet works in practice. We first call the data simulator, which generates the X and y matrices, a true beta vector, and a sample contribution cap limit. The default setting produces a input matrix of size 100 \(\times\) 20.
example <- simulate_capnet_data()
X <- example$X
y <- example$y
n <- nrow(X)
p <- ncol(X)
L <- example$L
beta_init <- rep(0, p + 1)We then call capnet.cv to find the best \(\alpha\) and \(\lambda\) values through cross-validation.
The default method uses 5-fold cross-validation but more complicated
split methods can be specified.
capnet.cv1 <- cv_capnet(X, y, 0, L, standardize = FALSE)
alpha1 <- capnet.cv1$best_alpha
lambda1 <- capnet.cv1$best_lambda
capnet.mod1 <- capnet(X, y, alpha = alpha1, lambda = lambda1,
mu = 0, L = L, standardize = FALSE)Best alpha value: 1.000 | Best lambda value 0.0179
capnet.cv2 <- cv_capnet(X, y, 1, L, standardize = FALSE)
alpha2 <- capnet.cv2$best_alpha
lambda2 <- capnet.cv2$best_lambda
capnet.mod2 <- capnet(X, y, alpha = alpha2, lambda = lambda2,
mu = 0.5, L = L, standardize = FALSE)Best alpha value: 1.000 | Best lambda value 0.0067
## [1] "System runtime for unconstrained 5-fold cross-validation"
## user system elapsed
## 14.304 0.508 14.826
## [1] "System runtime for constrained 5-fold cross-validation"
## user system elapsed
## 15.303 0.513 15.829
## [1] "System runtime for constrained model fit"
## user system elapsed
## 0.023 0.000 0.023
Similar to glmnet, capnet also has a
standardize parameter that is default to TRUE.
This controls standardization of the X and y
matrices. It is often recommended to scale X and
y terms for elastic net regressions and that also applies
to capnet as well. The function has been optimized to
handle variable scaling such that the penalty term, which is calculated
at the original variable scale, is scaled to be in the same unit as the
rest of the loss and gradient terms to avoid convergence failure due to
unit mismatch. Therefore, the user should supply newx and
L in the same scale as X and y
regardless of the value of the standardize parameter.
The multiplier parameter in capnet controls
the scaling applied to feature contributions. It is particularly useful
when the contribution cap needs to be evaluated across multiple
observations with non-uniform scaling factors for each observation. For
example, if the user has a constant contribution limit L
but wishes to evaluate contributions for several prediction points, each
requiring different scaling, multiplier can be specified as
a \(n\times1\) vector of scalars
corresponding to each observation. This avoids the need to manually
transform L into a matrix aligned with each observation.
Currently, the implementation does not support L as a
matrix input, so multiplier provides a flexible alternative
for handling observation-specific scaling requirements efficiently.
The capnet function also allows users to enforce box
constraints on model parameters using the upper.limits and
lower.limits arguments. These specify the maximum and
minimum allowable values for each coefficient, respectively. This is
useful when prior knowledge or practical considerations dictate that
parameter estimates must remain within specific bounds. The box
constraints is achieved via gradient masking, and tol sets
a tolerance threshold for enforcing such box constraints. If a parameter
is within tol of lower or upper bounds and the current
gradient would push it toward the limit, the gradient is manually set to
zero for that parameter. The users can thus increase the
tol variable if default value fails to enforce the box
constraint and decrease the tol if it restrict parameter
updates prematurally.
Parameter initialization: Because box constraints
are achieved via gradient masking, the initial parameter values
par needs to be within the specified limits. If initial
values fall outside these bounds, optimization may fail to converge or
produce invalid results. By default, par initializes to
zeros, so users should adjust the initialization if zeros fall outside
the desired parameter range.
Because capnet relies on the lbfgs package
for optimization, convergence failure can be caused by multiple factors
and may be hard for users to detect and debug. The
check.finite parameter can help the user determine whether
convergence failure occurred due to non-finite values when calculating
loss and gradients. By setting check.finite=TRUE, the
function will catch and throw non-finite value errors. It is also
important to note that the model does not stop or return error even when
convergence failed, so users are advised to manually check model
convergence after fitting. Because lbfgs does not handle
and store messages for all convergence failure codes, the message output
may not provide useful information on why convergence failed. Generally,
when users encounter a \(-998\) error
code, it is usually that the line search algorithm failed because the
gradient values are too unstable.
Social Science