Adding a "cost term" to a linear regression, so solution values are minimized

Question

I'm using Python's optimize.lsq_linear method to run a linear regression with the bounds set between 0% and 100% power usage.

x = optimize.lsq_linear(A, b, bounds=[0,100], method='trf')

The A matrix is sparse and there are many situations where some of the X values have very, very little effect on the results. Sometimes the regression sets these "far away" X values to 0, sometimes they are set very high (close to 100%, presumably because they help some tiny amount), but usually they are set to exactly 10. I have no idea why 10.

In linear algebra class back in college (almost 30 years ago!) I vaguely remember being told about adding a "cost function" to a regression to tell the regression to minimize the values of X where they "didn't help much". Lower X is less power consumption, which is a good thing here. I think this was sometimes done as an argument in the solver but it could also be done by adding a row to matrix A and vector b to assign the "cost of using more x".

The only words I can remember that might describe this are "cost" and "weight." However, all my searches about cost yield something about the cost function of the solution, such as the float that's returned by optimize.lsq_linear. All of my searches about weight yield something about weighting the different data sets (rows in the A matrix) differently because some of the rows might be more reliable than others. Neither of these is what I'm after.

• What is the name of the technique I'm after?
• Can someone provide a reference or a brief refresher on how to add some "cost" to my x values (so x values are mimimized when the effect of increasing them is negligible)?
• Why does optimize.lsq_linear set the "far away" x values to 10?

UPDATE: The comments, and reference, below point to Ridge regression, Lasso regression and Elastic-Net regression. All are linear regression methods with a penalty added on the size of the X coefficients: Ridge minimizes the sum of the squares of X, Lasso minimizes the values of X and Elastic-Net is a combination of both Ridge and Lasso. After reading the reference, it looks like Lasso may be most interesting here as it handles sparse matrices and prefers solutions with more coefficients set to zero.

I'm also wondering about weighting the penalty on the coefficients. Some of my X values use more power than others, so it would be nice to include this information in the regression, so if the benefit of using an efficient X and an inefficient X are the same, then the regression can preference the efficient X. I vaguely recall a method of doing this by adding a row to A and b, so Ax=b without penalties might be:

| A11 A12 | * | x1 | = | b1 b2 |
| A21 A22 |   | x2 |   

Adding even penalties for each x would add something like this:

| A11 A12 | * | x1 | = | b1 b2 0 |
| A21 A22 |   | x2 | 
|  1   1  |

Changing the coefficients of the penalties could add more detail, such as if x1 was 10 times less efficient than x2:

| A11 A12 | * | x1 | = | b1 b2 0 |
| A21 A22 |   | x2 | 
|  10  1  |

More questions:

• Is my method of adding a row correct?
• Is there a name for this method?
• Can something like this weighted penalties method be incorporated into a more standard method like Lasso?
• How can Lasso use bounds? All of my X coefficients need to be between 0 and 100.

Answer 1

In the first answer to question "Multiple Linear Regression with specific constraint on each coefficients on Python" David Dale lays out how this could work and links to his github project with the code.

I haven't tried this yet, but once I give it a try and confirm that it works, then I'll mark this as the answer to this question.