Questions tagged [least-squares]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
5
votes
1answer
223 views

Non-negative least squares with very small numbers

(I have asked this question on StackOverflow previously but it has been pointed to me that CSSE or MSE could be more appropriate) I have to solve a constrained optimization problem of the following ...
4
votes
0answers
124 views

Do computational scientists think about the statistical aspects of using least squares?

Correct me if I'm wrong, but when least squares is used in the computational science community, it's typically not in the context of regression. It could be used to solve for gradients in discretized ...
0
votes
0answers
39 views

Blown-up iterates in Gauss-Newton method

I am working on a non-linear least squares problem with standard form, in which I need to calibrate a parameter vector $\Theta$ to a set of inputs $\mathbf{x}$ and outputs $\mathbf{y}$: $$\begin{align}...
1
vote
0answers
43 views

Spline interpolation for vector-valued data in 3D space

I have output from a 3D linear elasticity finite element simulation which uses linear tetrahedral elements, such that the displacement is continuous over the nodes but the gradient is not ($C_0$ ...
0
votes
3answers
119 views

How can we solve the normal equations with limited memory?

I was asked this open ended question in an interview once: How would you find a solution to the normal equations with limited memory? Unlike Solving sparse least squares system with limited memory, ...
0
votes
1answer
287 views

Levenberg-Marquardt Algorithm for Black-box optimizations

I would like to create an optimization solution for black-box software calculations. Currently, I am using the Levenberg-Marquardt algorithm to update a vector of parameters, $\beta$, with residuals, $...
1
vote
0answers
60 views

How applying the gradient descent method for solving a least square problem can remove the blur from an image?

I got an assignment where it asked to implement (in MATLAB) the gradient descent algorithm in order to resolve an ill posed least square problem: $$ \min_u \Vert Gu - f \Vert $$ where $u$ is the ...
-1
votes
1answer
38 views

Trouble Estimating Motor Parameters with Least Squares in MATLAB

Basically, I'm trying to use Least-Squares to estimate the parameters of a DC motor. My system can be modeled by the following matrix equation: $$\begin{bmatrix}V_{input}(t)\\0\end{bmatrix}=\begin{...
1
vote
0answers
32 views

What's the best way to implement a least-squares estimation of a motor system in MATLAB?

Basically, I'm trying to use Least-Squares to estimate the parameters of a DC motor. My system can be modeled by the following matrix equation: $$\begin{bmatrix}V_{input}(t)\\0\end{bmatrix}=\begin{...
1
vote
1answer
116 views

Complexity of solving an image differential linear system

Define an "image differential linear system" as a linear system $A\mathbf{x}=\mathbf{b}$ wherein $\mathbf{x}$ contains the ($\mathbb{R}$) pixels of an image and each row of $A$ constrains ...
0
votes
0answers
37 views

Implementation method selection for sparse constrained linear least squares or quadratic programming

I need to slove one optimization problem of quadratic programming. The number of optimization variables is about 16,000. The constraints include equality constraints and inequality constraints. I have ...
2
votes
2answers
136 views

L1 least squares minimization with a sparse matrix

I have the following problem: $$\min_{x\in \mathbb{R}^n}\|Ax-b\|_1$$ where the matrix $A$ is large and sparse. I am looking for methods/code that can minimize this efficiently. References are very ...
1
vote
1answer
83 views

Least Square fit of a system of equations

I've a matrix equation $\bar z^{(r)}(k + 1) = \bar{DF} ~ \bar z^{(r)}(k)$. Now the $\bar z$'s and $\bar{DF}$ are $d \times d$ matrices. Now $\bar{DF} = \begin{pmatrix} 0 & 1 & 0 & 0 & \...
1
vote
0answers
43 views

Numerically estimate the Jacobian from a scalar time-series

I'm trying to numerically estimate the Jacobian from a time-series. Following the paper, https://journals.aps.org/pra/abstract/10.1103/PhysRevA.43.2787. Consider that I have a scalar time series $x = (...
2
votes
0answers
58 views

Non-negative Least Squares to perform Inverse Laplace with weights

I'm trying to perform the inverse Laplace transform of a (noisy) dataset $y_i$ using Tikhonov regularization: $$\min \sum_{i=1}^{N} \left(\int_0^\infty e^{-s_i t} f(t) \, dt - y_i \right)^2 - \lambda^...
3
votes
0answers
53 views

Least-squares fit of explicit parabolic sheet to data points

For a given set of data points $$\{(x_i, y_i, z_i)\}$$ there exists some $$f_{ABC}(x,y)=Ax^2+Bxy+Cy^2$$ that minimizes $$\sum_i(f_{ABC}(x_i,y_i)-z_i)^2$$ $A$, $B$, and $C$ can be found quickly ...
1
vote
0answers
45 views

Triangle on top of diagonal least squares

I need to solve many least squares problems with the following matrices: $$ \pmatrix{ R \\ D_i } $$ where $R$ is upper triangular and $D_i$ is diagonal. $R$ is the same for all the problems, while $...
5
votes
0answers
146 views

Solve ill-posed linear system without transposing matrices?

I am attempting to use an iterative solver to solve $p$ in $$ Jp = -r $$ where $J$ is an $m\times m$ matrix ($m$ is in the order of $10^5$ and never explicitly stored). $J$ is a dense matrix ...
0
votes
2answers
212 views

Update for QR factorization least squares

I found after some research that the most numerically stable way to solve the least squares problem is through QR factorization. For $n$ number of observations and $p$ number of parameters it takes ...
1
vote
2answers
62 views

Augmented Dickey Fuller (ADF) test statistics GPU formulation

I have followed different sources of information and achieved the following formulation for the ADF $t$ test statistics. I implemented it to run several hundred thousands of ...
3
votes
2answers
1k views

Cholesky decomposition vs LDL decomposition

In different books and on Wikipedia, you can see mentions of Cholesky decomposition and only sometimes of LDL decomposition. As far as I understand, LDL decomposition can be applied to a broader ...
1
vote
1answer
50 views

How to properly compute weights for Weighted Least Squares (WLS)?

I want to apply the weighted least squares method in order to identify parameters of a dynamic process. The process is described by a second order differential equation of the form: $$ \ddot{y}+a_1\...
2
votes
0answers
48 views

Fitting a plane with the Prewitt gradient operator

Prewitt gradient operator Show that the Prewitt gradient operator can be obtained by fitting the least-squares plane through the 3 × 3 neighborhood of the intensity function. Hint: Fit a plane to ...
3
votes
1answer
52 views

Nonlinear least squares resolution matrix

For a linear least squares problem, it is possible to define a resolution matrix, relating the estimated model parameters to the true model parameters. If we are solving a regularized problem, $$ \...
5
votes
0answers
50 views

Nonlinear least squares optimized Jacobian calculation

I have a nonlinear least squares problem, in which I am trying to minimize residuals which can be divided into four classes: $$ \min_x ||\epsilon(x)||^2 + ||\xi(x)||^2 + ||\delta(x)||^2 + ||s(x)||^2 $$...
1
vote
1answer
91 views

Spline regularization

I am fitting some B-splines to data, but the data has a "gap" region where the spline is less constrained by the data. I want to devise a regularization scheme to help prevent the spline from ...
7
votes
0answers
125 views

“Geometry of ill-conditioning” for least-squares problems

It is an idea that dates back to Demmel, 1987 that the condition number of a problem is often related to the distance to the closest ill-posed problems. In Section 3 of the above paper, the author ...
1
vote
1answer
352 views

Minimize cost with Levenberg-Marquart method

I want to minimize a cost function of the form, $$ \min_{q,t}\left(q^T\left(\mathcal A + \mathcal B\right)q + t^T\mathcal C t+\delta t+\varepsilon Q(q)^TW(q)t+\lambda\left(1-q^Tq\right)^2\right) $$ ...
1
vote
2answers
484 views

Fastest algorithm for pseudoinverse of skinny matrices

For a performance-sensitive problem, I need to compute the pseudoinverse of a skinny matrix (#rows = 1000–10000, #cols= 10–20). I already employ the traditional SVD econ method. For some problem ...
1
vote
1answer
201 views

What is a good library in Python for correlated fits in both the $x$ and $y$ data?

I have $x$ and $y$ data, both of which have their own covariance matrices. scipy.optimize.curve_fit will accept a covariance matrix for the $y$ data, called ...
1
vote
1answer
340 views

Preconditionning for solving a non-linear system of equations with least squares

I am trying to solve a large system of non-linear equations (about a few hundred equations and variable but with less variable than equations). Given that the system is really sparse and large I am ...
2
votes
0answers
223 views

Regularized least squares with QR factorization

Consider the regularized least squares problem $$ \min_x || b - A x ||^2 + \lambda^2 ||x||^2 $$ which is equivalent to $$ \min_x \left|\left| \pmatrix{b \\ 0} - \pmatrix{A \\ \lambda I} x \right|\...
0
votes
1answer
281 views

How to use LAPACK function (DGELSY) in Fortran

I am trying to use Least Squares Minimization to solve a the matrix problem: b = A*x for x. The system is overdetermined, and A is a dense matrix. In the LAPACK library, I believe the routine DGELSY ...
1
vote
1answer
221 views

Pivoted Cholesky vs Modified Cholesky

I am solving nonlinear least squares problems with the normal equations approach, so on each iteration, I need to solve: $$ J^T J \delta = -J^T f $$ for the step $\delta$, where $J$ is a large (...
2
votes
1answer
265 views

Formulation of the least-squares parameter estimation problem

I have a system of 10 ordinary differential equations of the form, $$\frac{dy_1}{dt} = f1(V1,k1,y1,y2)\\ \vdots \\ \frac{dy_{10}}{dt} = f_{10}(V_{10},k_{10},y_{9},y_{10}) $$ I want to estimate the ...
0
votes
0answers
106 views

Parameter estimation using fmincon

This is a follow up to my previous question posted here. I am solving an optimization problem using fmincon in MATLAB. There are no equality constraints in my model....
-1
votes
1answer
33 views

Minimize squared error of linear function

Let $M$ be a $m \times n$ matrix, $x$ a $n$-vector, $y$ a $m$-vector, and $\|\cdot\|_2$ represent the $L_2$ norm (i.e., Euclidean norm). Given $M,y$, the goal is to find $x$ that minimizes the ...
1
vote
1answer
70 views

Correct weighting in least squares fitting

I am trying to fit some data points $d_i$ to a non-linear model function $m_i$, which depends on a number of fit parameters $f_k$ (I want to determine these) and also on some known, constant values $...
11
votes
3answers
1k views

Least squares approximation question

I am taking a course on scientific computation, and we just went over least squares approximation. My question is specifically about approximating using polynomials. I understand that if you have n+1 ...
4
votes
0answers
269 views

Nonlinear least squares and regularization

Consider the nonlinear least-squares minimization of a vector of $n$ residuals $\mathbf{f}$ in $p$ parameters $\mathbf{x}$: $$ \min_{\mathbf{x}} || \mathbf{f}(\mathbf{x}) ||^2 $$ This can be done with ...
3
votes
1answer
111 views

Nonlinear least squares when some parameters are linear

Consider the least squares problem, $$ \min_{\mathbf{a},\mathbf{b}} || \mathbf{f}(\mathbf{a},\mathbf{b})||^2 $$ where $\mathbf{a},\mathbf{b}$ represent the unknown parameters to be found. In my ...
1
vote
0answers
55 views

B-splines least squares with equality constraints

Can someone recommend the best way to solve a least squares fitting problem with B-splines, with additional equality constraints? I want to solve: $$ \min_x || b - A x ||^2, \textrm{subject to: } C x =...
3
votes
0answers
40 views

Reweighted least squares factorization

This is a continuation of the question asked here. I want to solve numerous least squares systems of the form $$ D_i A x \approx D_i b $$ where $D_i$ are $m \times m$ diagonal matrices with positive ...
4
votes
1answer
90 views

Method to Efficiently Solve “Centered” Least Squares without centering “A”

Suppose I want to solve $$\text{arg min}_x \frac{1}{2}\|\tilde{A}x - b\|_2^2 + \frac{1}{2}\|x - c\|_2^2$$ where $A$ is a wide sparse matrix and $\tilde{A} = A C_n = A (I - \mathbf{1}\, \mathbf{1}^T/...
12
votes
2answers
6k views

Solving a least squares problem with linear constraints in Python

I need to solve \begin{alignat}{1} & \min_{x}\|Ax - b\|^2_{2}, \\ \mathrm{s.t.} & \quad\sum_{i}x_{i} = 1, \\ & \quad x_{i} \geq 0, \quad \forall{i}. \end{alignat} I think it is a ...
2
votes
0answers
286 views

Convergence of a very large non-linear least squares optimization

(note: I also posted this question on stackoverflow before finding this community here, which seems a better place for it) I'm trying to solve the following problem: I have a lot (~80000) surface ...
7
votes
1answer
280 views

Least Squares with Dense-Block Diagonal Structure

I need to solve a least squares problem that takes the following form: $$p = \arg \min_{x}\Vert J V x - y \Vert_2, $$ where $J \in \mathbb{R}^{N \times N}$ is a general dense matrix, and $V \in \...
4
votes
1answer
405 views

How to solve the following Frobenius norm-minimization problem?

Background We know how to solve the following minimization problem $$ \min_{X} \lVert AX - B \rVert_F^2 $$ But what about the extended version? $$ \min_{X} \lVert A \begin{bmatrix} X & X^2 \...
1
vote
0answers
47 views

How to model a non-linear least-squares problem for triangles

I have a non-linear least-squares problem to solve and with my current modeling the solver is either very slow or does not converge to a correct solution. For the problem I need to minimize an energy ...
3
votes
1answer
204 views

How to solve the inverse problem of least-squares?

Focusing on following least squares problem: $$\min\limits_{V} \lVert Z - WV \rVert _{_F}^2$$ $$Z∈{R}^{m\times n},\quad W∈{R}^{m\times k},\quad V∈{R}^{k\times n},\quad k\lt m\lt n $$ This problem ...