# Adding a "cost term" to a linear regression, so solution values are minimized

I'm using Python's optimize.lsq_linear method to run a linear regression with the bounds set between 0% and 100% power usage.

x = optimize.lsq_linear(A, b, bounds=[0,100], method='trf')


The A matrix is sparse and there are many situations where some of the X values have very, very little effect on the results. Sometimes the regression sets these "far away" X values to 0, sometimes they are set very high (close to 100%, presumably because they help some tiny amount), but usually they are set to exactly 10. I have no idea why 10.

In linear algebra class back in college (almost 30 years ago!) I vaguely remember being told about adding a "cost function" to a regression to tell the regression to minimize the values of X where they "didn't help much". Lower X is less power consumption, which is a good thing here. I think this was sometimes done as an argument in the solver but it could also be done by adding a row to matrix A and vector b to assign the "cost of using more x".

The only words I can remember that might describe this are "cost" and "weight." However, all my searches about cost yield something about the cost function of the solution, such as the float that's returned by optimize.lsq_linear. All of my searches about weight yield something about weighting the different data sets (rows in the A matrix) differently because some of the rows might be more reliable than others. Neither of these is what I'm after.

• What is the name of the technique I'm after?
• Can someone provide a reference or a brief refresher on how to add some "cost" to my x values (so x values are mimimized when the effect of increasing them is negligible)?
• Why does optimize.lsq_linear set the "far away" x values to 10?

UPDATE: The comments, and reference, below point to Ridge regression, Lasso regression and Elastic-Net regression. All are linear regression methods with a penalty added on the size of the X coefficients: Ridge minimizes the sum of the squares of X, Lasso minimizes the values of X and Elastic-Net is a combination of both Ridge and Lasso. After reading the reference, it looks like Lasso may be most interesting here as it handles sparse matrices and prefers solutions with more coefficients set to zero.

I'm also wondering about weighting the penalty on the coefficients. Some of my X values use more power than others, so it would be nice to include this information in the regression, so if the benefit of using an efficient X and an inefficient X are the same, then the regression can preference the efficient X. I vaguely recall a method of doing this by adding a row to A and b, so Ax=b without penalties might be:

| A11 A12 | * | x1 | = | b1 b2 |
| A21 A22 |   | x2 |


| A11 A12 | * | x1 | = | b1 b2 0 |
| A21 A22 |   | x2 |
|  1   1  |


Changing the coefficients of the penalties could add more detail, such as if x1 was 10 times less efficient than x2:

| A11 A12 | * | x1 | = | b1 b2 0 |
| A21 A22 |   | x2 |
|  10  1  |


More questions:

• Is my method of adding a row correct?
• Is there a name for this method?
• Can something like this weighted penalties method be incorporated into a more standard method like Lasso?
• How can Lasso use bounds? All of my X coefficients need to be between 0 and 100.
• Traditionally the modeler performs variable selection (e.g. using experience, domain knowledge, theory, or existing literature). An example of an automatic approach is Lasso Regression. Jul 31 at 21:46
• There are several linear regressions with regularization (that's the term for what you describe by "add some "cost" to my x values"). The simplest one only doing what you want is called Ridge regression. As mentioned in the other comment Lasso Regression is also one of them that also does so called variable selection. You'll probably want to use scikit-learn for this sort of task and read their user guide on linear models to learn about the details which this margin is too narrow to contain: scikit-learn.org/stable/modules/linear_model.html
– user2640045
Jul 31 at 23:31
• Thanks @user2640045. That's very helpful. Lasso Regression looks particularly relevant here. I've added an update, above. Do you know of a way to add weightings to the coefficients in something like Lasso? Aug 4 at 14:07
• Here's a relevant reference I'm looking into: stackoverflow.com/questions/10154922/… Aug 4 at 14:13
• @Casey I don't think you can use ridge or lasso from scikit-learn with bounds. But I also don't get why you would want to. Crank up the alpha(s) and the coefficients won't get large. Apparently for lasso the alpha parameter is a number i.e. can't be chosen individually but for ridge regression I remember that you can give an array penalizing each parameter to your liking :). If you get fancy with parameters btw. I suggest you use their cross validation methods to find them like RidgeCV or wrap things in GridSearchCV/RandomizedSearchCV. Enjoy :)
– user2640045
Aug 4 at 15:42