Questions tagged [least-squares]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1
vote
0answers
35 views

Triangle on top of diagonal least squares

I need to solve many least squares problems with the following matrices: $$ \pmatrix{ R \\ D_i } $$ where $R$ is upper triangular and $D_i$ is diagonal. $R$ is the same for all the problems, while $...
5
votes
0answers
142 views

Solve ill-posed linear system without transposing matrices?

I am attempting to use an iterative solver to solve $p$ in $$ Jp = -r $$ where $J$ is an $m\times m$ matrix ($m$ is in the order of $10^5$ and never explicitly stored). $J$ is a dense matrix ...
0
votes
2answers
72 views

Update for QR factorization least squares

I found after some research that the most numerically stable way to solve the least squares problem is through QR factorization. For $n$ number of observations and $p$ number of parameters it takes ...
1
vote
2answers
48 views

Augmented Dickey Fuller (ADF) test statistics GPU formulation

I have followed different sources of information and achieved the following formulation for the ADF $t$ test statistics. I implemented it to run several hundred thousands of ...
2
votes
2answers
156 views

Cholesky decomposition vs LDL decomposition

In different books and on Wikipedia, you can see mentions of Cholesky decomposition and only sometimes of LDL decomposition. As far as I understand, LDL decomposition can be applied to a broader ...
1
vote
1answer
41 views

How to properly compute weights for Weighted Least Squares (WLS)?

I want to apply the weighted least squares method in order to identify parameters of a dynamic process. The process is described by a second order differential equation of the form: $$ \ddot{y}+a_1\...
2
votes
0answers
41 views

Fitting a plane with the Prewitt gradient operator

Prewitt gradient operator Show that the Prewitt gradient operator can be obtained by fitting the least-squares plane through the 3 × 3 neighborhood of the intensity function. Hint: Fit a plane to ...
3
votes
1answer
38 views

Nonlinear least squares resolution matrix

For a linear least squares problem, it is possible to define a resolution matrix, relating the estimated model parameters to the true model parameters. If we are solving a regularized problem, $$ \...
4
votes
0answers
42 views

Nonlinear least squares optimized Jacobian calculation

I have a nonlinear least squares problem, in which I am trying to minimize residuals which can be divided into four classes: $$ \min_x ||\epsilon(x)||^2 + ||\xi(x)||^2 + ||\delta(x)||^2 + ||s(x)||^2 $$...
0
votes
1answer
50 views

Spline regularization

I am fitting some B-splines to data, but the data has a "gap" region where the spline is less constrained by the data. I want to devise a regularization scheme to help prevent the spline from ...
6
votes
0answers
108 views

“Geometry of ill-conditioning” for least-squares problems

It is an idea that dates back to Demmel, 1987 that the condition number of a problem is often related to the distance to the closest ill-posed problems. In Section 3 of the above paper, the author ...
1
vote
1answer
146 views

Minimize cost with Levenberg-Marquart method

I want to minimize a cost function of the form, $$ \min_{q,t}\left(q^T\left(\mathcal A + \mathcal B\right)q + t^T\mathcal C t+\delta t+\varepsilon Q(q)^TW(q)t+\lambda\left(1-q^Tq\right)^2\right) $$ ...
1
vote
2answers
215 views

Fastest algorithm for pseudoinverse of skinny matrices

For a performance-sensitive problem, I need to compute the pseudoinverse of a skinny matrix (#rows = 1000–10000, #cols= 10–20). I already employ the traditional SVD econ method. For some problem ...
0
votes
0answers
48 views

Least square approximation of a polynomial with a constraint on the derivative in Python

I'm trying to fit a polynomial of the third degree through a number of points. This could be a very simple problem when not constraining the derivative. I found some promising solutions using CVXPY to ...
1
vote
1answer
97 views

What is a good library in Python for correlated fits in both the $x$ and $y$ data?

I have $x$ and $y$ data, both of which have their own covariance matrices. scipy.optimize.curve_fit will accept a covariance matrix for the $y$ data, called ...
1
vote
1answer
135 views

Preconditionning for solving a non-linear system of equations with least squares

I am trying to solve a large system of non-linear equations (about a few hundred equations and variable but with less variable than equations). Given that the system is really sparse and large I am ...
2
votes
0answers
122 views

Regularized least squares with QR factorization

Consider the regularized least squares problem $$ \min_x || b - A x ||^2 + \lambda^2 ||x||^2 $$ which is equivalent to $$ \min_x \left|\left| \pmatrix{b \\ 0} - \pmatrix{A \\ \lambda I} x \right|\...
0
votes
1answer
158 views

How to use LAPACK function (DGELSY) in Fortran

I am trying to use Least Squares Minimization to solve a the matrix problem: b = A*x for x. The system is overdetermined, and A is a dense matrix. In the LAPACK library, I believe the routine DGELSY ...
1
vote
1answer
118 views

Pivoted Cholesky vs Modified Cholesky

I am solving nonlinear least squares problems with the normal equations approach, so on each iteration, I need to solve: $$ J^T J \delta = -J^T f $$ for the step $\delta$, where $J$ is a large (...
2
votes
1answer
144 views

Formulation of the least-squares parameter estimation problem

I have a system of 10 ordinary differential equations of the form, $$\frac{dy_1}{dt} = f1(V1,k1,y1,y2)\\ \vdots \\ \frac{dy_{10}}{dt} = f_{10}(V_{10},k_{10},y_{9},y_{10}) $$ I want to estimate the ...
0
votes
0answers
49 views

Parameter estimation using fmincon

This is a follow up to my previous question posted here. I am solving an optimization problem using fmincon in MATLAB. There are no equality constraints in my model....
0
votes
1answer
24 views

Minimize squared error of linear function

Let $M$ be a $m \times n$ matrix, $x$ a $n$-vector, $y$ a $m$-vector, and $\|\cdot\|_2$ represent the $L_2$ norm (i.e., Euclidean norm). Given $M,y$, the goal is to find $x$ that minimizes the ...
1
vote
1answer
64 views

Correct weighting in least squares fitting

I am trying to fit some data points $d_i$ to a non-linear model function $m_i$, which depends on a number of fit parameters $f_k$ (I want to determine these) and also on some known, constant values $...
11
votes
3answers
1k views

Least squares approximation question

I am taking a course on scientific computation, and we just went over least squares approximation. My question is specifically about approximating using polynomials. I understand that if you have n+1 ...
3
votes
0answers
167 views

Nonlinear least squares and regularization

Consider the nonlinear least-squares minimization of a vector of $n$ residuals $\mathbf{f}$ in $p$ parameters $\mathbf{x}$: $$ \min_{\mathbf{x}} || \mathbf{f}(\mathbf{x}) ||^2 $$ This can be done with ...
3
votes
1answer
88 views

Nonlinear least squares when some parameters are linear

Consider the least squares problem, $$ \min_{\mathbf{a},\mathbf{b}} || \mathbf{f}(\mathbf{a},\mathbf{b})||^2 $$ where $\mathbf{a},\mathbf{b}$ represent the unknown parameters to be found. In my ...
1
vote
0answers
42 views

B-splines least squares with equality constraints

Can someone recommend the best way to solve a least squares fitting problem with B-splines, with additional equality constraints? I want to solve: $$ \min_x || b - A x ||^2, \textrm{subject to: } C x =...
3
votes
0answers
38 views

Reweighted least squares factorization

This is a continuation of the question asked here. I want to solve numerous least squares systems of the form $$ D_i A x \approx D_i b $$ where $D_i$ are $m \times m$ diagonal matrices with positive ...
4
votes
1answer
76 views

Method to Efficiently Solve “Centered” Least Squares without centering “A”

Suppose I want to solve $$\text{arg min}_x \frac{1}{2}\|\tilde{A}x - b\|_2^2 + \frac{1}{2}\|x - c\|_2^2$$ where $A$ is a wide sparse matrix and $\tilde{A} = A C_n = A (I - \mathbf{1}\, \mathbf{1}^T/...
12
votes
2answers
5k views

Solving a least squares problem with linear constraints in Python

I need to solve \begin{alignat}{1} & \min_{x}\|Ax - b\|^2_{2}, \\ \mathrm{s.t.} & \quad\sum_{i}x_{i} = 1, \\ & \quad x_{i} \geq 0, \quad \forall{i}. \end{alignat} I think it is a ...
2
votes
0answers
278 views

Convergence of a very large non-linear least squares optimization

(note: I also posted this question on stackoverflow before finding this community here, which seems a better place for it) I'm trying to solve the following problem: I have a lot (~80000) surface ...
7
votes
1answer
215 views

Least Squares with Dense-Block Diagonal Structure

I need to solve a least squares problem that takes the following form: $$p = \arg \min_{x}\Vert J V x - y \Vert_2, $$ where $J \in \mathbb{R}^{N \times N}$ is a general dense matrix, and $V \in \...
4
votes
1answer
207 views

How to solve the following Frobenius norm-minimization problem?

Background We know how to solve the following minimization problem $$ \min_{X} \lVert AX - B \rVert_F^2 $$ But what about the extended version? $$ \min_{X} \lVert A \begin{bmatrix} X & X^2 \...
1
vote
0answers
34 views

How to model a non-linear least-squares problem for triangles

I have a non-linear least-squares problem to solve and with my current modeling the solver is either very slow or does not converge to a correct solution. For the problem I need to minimize an energy ...
3
votes
1answer
196 views

How to solve the inverse problem of least-squares?

Focusing on following least squares problem: $$\min\limits_{V} \lVert Z - WV \rVert _{_F}^2$$ $$Z∈{R}^{m\times n},\quad W∈{R}^{m\times k},\quad V∈{R}^{k\times n},\quad k\lt m\lt n $$ This problem ...
5
votes
1answer
163 views

Pseudoinverse of perturbed matrix

How does the pseudo inverse of a full column rank matrix change if I rescale a single row? In more detail the problem is the following: We have a fixed matrix $V$ with linear independent columns and ...
0
votes
1answer
110 views

Using least squares for computing gradients

I am developing a code where I am using the least squares method to compute gradients. Generally, we use least squares to obtain some model based on a set of data (${q_1 \cdots q_N}$) at locations ($...
4
votes
2answers
196 views

treating “almost linear” nonlinear least-squares problems

As a model for a nonlinear least-squares problem with a large linear part problem, consider $$ \Delta u = 0 \quad\text{in } \Omega,\\ n\cdot\nabla u = 0 \quad \text{on } \Gamma,\\ (u(x_i) - u(y_i))^2 =...
12
votes
1answer
1k views

Purely rotational least squares match

Could anyone recommend a method for the following least-squares problem: find $R \in \mathbb{R}^{3 \times 3}$ that minimizes: $\sum\limits_{i=0}^N (Rx_i - b_i)^2 \rightarrow \min$, where $R$ is a ...
4
votes
1answer
118 views

Nonlinear least-squares solvers vs. generic minimization

A nonlinear least-squares problem with $F:\mathbb{R}^m\to\mathbb{R}^n$, $$ F(x) \to \min_x \quad (\text{in the least-squares sense}) $$ really means minimizing $$ \frac{1}{2} \|F(x)\|^2 \to \min_x. $$ ...
1
vote
0answers
34 views

fitting exponential versus exponential w/ power

I have two models which I would like to investigate for my data. One form is: \begin{equation} \label{one} f(r) = A e^{-B r} \end{equation} and the second is: \begin{equation} \label{two} g(r)...
1
vote
1answer
146 views

Matlab interface for minpack2

Is there a Matlab interface for minpack2? minpack2 http://ftp.mcs.anl.gov/pub/MINPACK-2 consists of Fortran programs , but i would like to call the test problems from matlab. What can i do ?
3
votes
1answer
134 views

Obtaining a feasible solution for underdetermined system of linear equations satisfying inequality constraints

I would like to obtain a feasible solution for an under-determined system of linear equations, $$Ax=b$$ where, $A \in \mathbb{R}^{7\times9}, \, x \in \mathbb{R}^{9\times1}\text{and } b\in\mathbb{R}^...
1
vote
0answers
67 views

Stable Method of orthogonal projection onto a subspace with the help of Moore-Penrose inverse,

Projection of a vector $v$ onto the column space of a matrix $A$ is given by $AA^\dagger v$. From the definition of Moore-Penrose Inverse we know that $AA^\dagger v = (A^T)^\dagger A^T v $. Below is ...
3
votes
0answers
81 views

Backward stable algorithm to get orthogonal projection onto the column space of a matrix

I have to find the orthogonal projection of a vector $b$ onto the matrix $A$ of size $m \times n$. In my application, I don't have the luxury of calculating the QR factorization. All I have are ...
0
votes
1answer
193 views

Linear Least-Squares Point-to-Plane ICP degenerative case

I'm trying to implement Linear Least-Squares Optimization for Point-to-Plane ICP Surface Registration Paper describes linear approximation for point to plane distance for rigid ICP. This approach is ...
2
votes
1answer
141 views

Solving sparse least squares system with limited memory

This was a question on a past final that we can't figure out. Take the least squares system $$\min_x ||Ax-b||_2\, ,$$ where $A\in\mathbb{R}^{mxn}$, $m<n$, and A is full rank. A has $\mathcal{O}(n)...
0
votes
2answers
56 views

Finding parameters numerically

I suspect that a function $f(x,y)$ is of the form $f(x,y)=a(bx+c)^{dy+e}$. I have access to several values of $f(x,y)$. How do I proceed numerically to find the parameters $\{a,b,c,d,e\}$? By ...
1
vote
1answer
71 views

Solving for $C$ in $Q = YCZ$ using least squares in Matlab

I am trying to solve for the matrix $C$ in $Q = YCZ$ in matlab. I have preliminary results but they don't seem realistic. Here, $Q$ is $n \times m-1$, $Y$ is $n \times p$, $C$ is $p \times m$ and $Z$ ...
7
votes
2answers
7k views

Least Squares and Fourier Series

I have a little bit of problem figuring out the relation between Fourier series and Least Squares. As far as I understand, LS is a way of minimizing the quadratic error between a measured value $y_i$ ...