Questions tagged [conjugate-gradient]
A popular krylov subspace method for solving linear systems of equations, particularly those that exhibit symmetric positive definiteness.
80
questions
3
votes
1answer
64 views
Why does smoothed aggregation multigrid method used as preconditioner in conjugate gradient slows down the solution time?
I'm solving a system of linear equations obtained from the FEM discretization of a simple linear elasticity problem on a cube with zero displacements at one plane and a load on the opposite one. The ...
3
votes
1answer
86 views
What happens when I use a conjugate gradient solver with a symmetric positive semi-definite matrix?
I have a symmetric positive semi-definite matrix, i.e., a laplacian and wonder what may happen when I use a CG solver, that is an algorithm for positive definite matrices.
What happens when the ...
3
votes
0answers
58 views
Conjugate Gradient for nonlinear equation system
Is it possible to apply adaptions of the conjugate gradient algorithm i.e. Fletcher-Reeves, Polak-Ribere or others to systems of nonlinear equations? How should the equation system be adjusted so one ...
0
votes
0answers
59 views
Arnoldi Decomposition Algorithm
I try to get into GMRES via Arnoldi-Decomposition. For my understanding, I Implemented the Arnoldi-Decomposition in python.
...
1
vote
1answer
72 views
In which cases does the nonlinear conjugate gradient method take more than $n$ steps?
I have programmed a couple of Matlab implementations of nonlinear Conjugate Gradient methods (Fletcher Reeves and Polak Ribeire). However, I am concerned with how many steps it's taking to optimise ...
4
votes
0answers
132 views
Explanation of subspace strategy regarding CG described in Golub's book
I was wondering about the last paragraph in Matrix Computations (4th edition) by Golub, Chapter 11 (11.3.3), specifically his explanation of subspace strategy for Conjugate Gradient.
Note that in ...
2
votes
1answer
162 views
What's wrong with the **PCG and MINRES** in matlab?
Last week, I have learned the details of the robust iterative methods of PCG, MINRES, GMRES, which will converges to the exact solution $x^*$ of nonsingular system within $N$ steps for $A\in \mathbb{R}...
3
votes
0answers
44 views
Subspaces for Iterative methods
In the original paper of Conjugate Gradients, the authors mention that if we pick the canonical basis $\{e_1,e_2,\ldots,e_n\}$, to obtain A-orthonormal vectors, we end up with the Gaussian elimination ...
1
vote
2answers
53 views
What's the difference between the 2 ways of definitions of function handle? which is robust and better?
Recently, I have been studying Krylov subspace iterative methods. I find the matlab robust command pcg and the new concept of the function handle to return a matrix-vector product. Then I use help pcg ...
1
vote
1answer
38 views
Accelerating Conjugate Gradients fitting for small localized kernel (like cubic B-spline)
Question:
Is there some pre-conditioner for Conjugate-Gradient (CG) cheap enough, that it is worth using even if my operator is very local (i.e. already has a low number of non-zero elements), as it ...
1
vote
1answer
36 views
Why the iteration steps become twice if the step size reduces half for CG methods?
For CG method for SPD matrices, (Ax = b arising from Poisson equation with homogeneous boundary condition) we know that the convergence theorem:
After m steps of iteration, the error $e^{(m)}=x-x_m$ ...
3
votes
1answer
83 views
Conjugate Gradient for singular 2D poisson finite element with Neumann Boundary Conditions
Heavily edited question after I realised partly what the problem was
I have programmed a simple 2D square finite element solution to the Poisson equation
$-\Delta u = f$
The source function ...
6
votes
0answers
66 views
Why not use the preconditioned residual as termination criterion for preconditioned CG?
I have a Poisson equation with wildly varying material parameters (1 .. 1000), wildly varying element sizes (5 nm .. 100 um) and some quite anisotropic (tetrahedral) elements (100 nm x 100 um). I use (...
2
votes
1answer
77 views
Blowup of error in Conjugate Gradient method with periodic Dirichlet Poisson matrix
My problem is that the L2-Norm of the residual for the periodic Poisson matrix $P$ is initially decreasing but starts to blow up after a certain number of iterations. The blowup happens earlier the ...
2
votes
1answer
81 views
Nonlinear conjugate gradient with orthogonality constraint
I have to solve a set of nonlinear optimization problems in the subspace defined as the orthogonal space to a given vector.
More precisely,
$$
\arg\min f(\vec x) \qquad \text{with} \qquad \vec x \...
2
votes
0answers
116 views
Richardson's Iteration, Gradient Method and Spectral Radius
Richardson's iteration introduce a scalar $\alpha$ to the update formula:
$$ \textbf{x}^{(k+1)} = \textbf{x}^{(k)} + \alpha \textbf{r}^{(k)} $$
And compute $\alpha$ by minimizing the spectral radius:...
1
vote
1answer
104 views
Conjugate gradient - ill-conditioning and numerical tolerance
I would like to solve system $Ax=b$, where $A$ is SPD, but very ill-conditioned ($\text{cond}(A)>10^{11}$). I am interested in using UNpreconditioned version of the conjugate gradient method.
Is ...
1
vote
1answer
110 views
Relation between conjugate gradient method and finite elements method
What is difference beetwen this two method? Are these methods far from each other or are these methods complement each other? Could you take an example?
4
votes
1answer
2k views
Which C++ linear algebra library is probably the fastest on solving huge sparse [square matrix] linear system?
I am developing a 2D CFD solver for fluid-particle interaction. To solve Navier-Stokes equations on a grid of size $10000\times 10000$ cells (or >1 million cells), a large linear system $Ax=b$ with $A$...
8
votes
1answer
222 views
How to find a good preconditioner to the system $(A^T A + \lambda I) x = A^T b$?
The system in the title has a damper factor $\lambda > 0$ and the matrix $A$ is sparse and rectangular, with a structure I can exploit to solve matrix vector products very fast. My current solver, ...
4
votes
0answers
34 views
Stochastic conjugate directions to improve convergence in narrow valleys
My question concerns a specific statement in this paper:
N. N. Schraudolph and T. Graepel, "Conjugate Directions for Stochastic Gradient Descent," in Int. Conf. Artificial Neural Networks, Berlin, ...
3
votes
1answer
159 views
How can a CG solver solve a non positive definite sparse matrix
I am using the CUSP CG solver and I ran it on couple of sparse matrices from the University of Florida sparse matrix collection. The solver was able to solve non positive definite sparse matrices. My ...
6
votes
1answer
311 views
What is required of the objective function in order to use Gauss Newton method?
From what I understand, the Gauss-Newton method is used to find a search direction, then the step size, etc., can be determined by some other method.
In addition to that, are the following ...
3
votes
1answer
377 views
Linear constraints for L-BFGS-B
I know L-BFGS-B only supports simple box constrains of the form: $l_i \leq x_i \leq u_i$, where $l_i$ and $u_i$ are constants. For my specific optimization problem, I need to specify some simple ...
0
votes
1answer
65 views
Adaptive gradient descent
I want to minimize some multivariable function $\Delta(\alpha, \beta)$. I know that this function has a zero point, $\Delta(5, 5) = 0$.
Starting from some $(\alpha, \beta)$ close to $(5,5)$ (e.g. (4....
0
votes
1answer
144 views
How to verify solution to pre-conditioned linear systems solver?
I am solving Ax=b. A has a very large condition number (> O(10^10))
I am using the conjugate gradients method with point jacobi pre-conditioning. I obtained a solution 'x' that "looks" reasonable. ...
9
votes
1answer
4k views
What are the differences between the different gradient-based numerical optimization methods?
I am interested in the specific differences of the following methods:
The conjugate gradient method (CGM) is an algorithm for the numerical solution of particular systems of linear equations.
The ...
1
vote
1answer
160 views
Computing preconditioner for a non-linear conjugate gradient implementation
Consider the following steps for the $i$-th non-linear conjugate gradient iteration, in the context of 3D electromagnetic inversion, and as discussed in (Newman and Boggs, 2004):
(1) set $i = 1$, ...
1
vote
0answers
159 views
Search direction for CG method
I am studying optimization methods and I was able to understand and derive the search direction
$$ p_k = r_{k-1} + \beta p_{k-1} $$
for Conjugate Gradient Method, with
$$ \beta = -\frac{p_{k-1}^...
4
votes
2answers
1k views
Conjugate gradient method to minimize a function
I am having some serious difficulties trying to understand how to use (apply) CG to minimize a function. In all the textbooks and notes, the step size $\alpha$ is give by the following expression
$$ \...
2
votes
0answers
101 views
Optimization based integration for MPM
I'm considering implementing (just for simplicity) the unconstrained implicit optimization based integration for Material Point Method as described in Chenfanfu Jiang's thesis on MPM (the minimization ...
3
votes
0answers
2k views
Understanding MATLAB's `fmincg` optimization function
I'm researching numerical optimization. Recently I've come across a variant of a conjugate gradient method named fmincg.
The function is written in MATLAB and is ...
1
vote
1answer
2k views
Solving the Poisson equation with Neumann Boundary Conditions - Finite Difference, BiCGSTAB
I am trying to solve the Poisson equation in a rectangular domain using a finite difference scheme with a rectangular mesh.
I have happily generated the matrix system of equations Ax = b which is ...
3
votes
1answer
147 views
CG question: is symmetry always necessary?
Consider the 1D Poisson equation
$$\nabla^2 u = f.$$
Using finite difference method on cell corner data and a uniform grid with ghost points, I think we can write the system of equations with Neumann ...
1
vote
2answers
857 views
Boundary conditions in conjugate gradient method for poisson's equation
I want to use the conjugate gradient method to solve poisson's equation in an electrostatic setup:
\begin{align}
\rho=-\nabla^2\phi
\end{align}
I am however a little confused when it comes to the ...
9
votes
1answer
1k views
Adaptive gradient descent step size when you can't do a line search
I have an objective function $E$ dependent on a value $\phi(x, t = 1.0)$, where $\phi(x, t)$ is the solution to a PDE. I am optimizing $E$ by gradient descent on the initial condition of the PDE: $\...
3
votes
2answers
357 views
2D Laplace problem with mixed boundary conditions using Conjugate Gradients
I am being asked for one of my classes to solve 2D Laplace equations with mixed boundary conditions using the Conjugate Gradient method. The equations and
conditions are given as:
$$
\frac{\partial^...
1
vote
1answer
103 views
The linear system in Quasi Newton method
I have implemented a Quasi Newton method for my problem, where I use the Hessian matrix approximation based approach. Hence, there is a linear system solve in every iteration. I solve the linear ...
5
votes
1answer
451 views
Solving a set of linear equations with block structure and weak coupling
I have a standard set of linear equations $Ax=b$ where the Hessian matrix $A$ has the special block structure as shown:
$A=
\begin{pmatrix}
T & U\\
U^T & V
\end{pmatrix}$,
$x=
\begin{...
1
vote
1answer
71 views
Conjugate Gradient, initial direction set to initial residual
In the (iterative) Conjugate Gradient (CG) algorithm: https://en.wikipedia.org/wiki/Conjugate_gradient_method
The initial search direction $p_{0}$ is set to the initial residual $r_{0}$. But I can't ...
4
votes
0answers
215 views
Nonlinear conjugate gradient restart threshold 1/10
Nocedal and Wright
on Conjugate Gradient Methods, p. 123, describe a
restart strategy ... whenever two consecutive gradients are far from orthogonal
$\qquad {{| \nabla f_k^T \ \nabla f_{k-1} |}
\...
1
vote
1answer
116 views
Can this equation be solved with the conjugate gradient method?
Let $A$ be positive-definite and $C$ diagonal positive-definite, consider the problem of solving the following equation for $\bf x$
$$A{\bf x}+C\begin{bmatrix}
e^{x_1} \\
\vdots \\
e^{x_n}
\end{...
9
votes
2answers
2k views
What is the worst case complexity of Conjugate Gradient?
Let $A\in \mathbb{R}^{n\times n}$, symmetric and positive definite. Suppose it takes $m$ units of work to multiply a vector by $A$. It is well known that performing the CG algorithm on $A$ with ...
8
votes
2answers
310 views
Why does conjugate gradient work with this nonsymmetric preconditioner?
In this previous thread the following multiplicative way to combine symmetric preconditioners $P_1$ and $P_2$ for the symmetric system $Ax=b$ was suggested:
\begin{align}
P_\text{combo}^{-1} :=& ...
3
votes
0answers
109 views
Conjugate gradient: the 1-norm of the residual
I am trying to solve $Ax=b$ using the conjugate gradient method. However, it is important to me to obtain a bound not only on the usual residual $||b-Ax_k||_2$ but also on the quantity $||b-Ax_k||_1$. ...
2
votes
1answer
134 views
Solve FEM matrix from coupled system
I'm developing an FEM solver for a coupled system. I have diffusion and potential equations which result in positive definite matrices for each equation, but the coupling makes the overall system ...
3
votes
0answers
90 views
How to implement conjugate gradient method to minimize this nonlinear action?
Given a 2D stochastic differential equation:
\begin{align}
\dot{x}_{i}=f_{i}(\textbf{x})+g_{ij}\xi_{j}(t),
\end{align}
where $i=2$, $g_{ij}g_{jk}=2\epsilon\delta_{ik}$ and $f(\textbf{x})=-\nabla\phi(\...
3
votes
1answer
218 views
Trust-region Newton: implementation issue with Conjugate Gradient calculations
UPDATE: The problem turned out to be the step (refer penultimate paragraph below) where I was factoring out a small value from the vectors of the numerator and denominator and then computed dot ...
2
votes
1answer
469 views
Hessian-free and Truncated Newton methods
In this paper on Deep Learning for Machine Learning, the approach is referred to as Hessian-free method. That is because the Hessian is never computed explicitly. Instead, the product of the Hessian ...
6
votes
1answer
358 views
Caveats of Hessian free method
Hessian free iterative optimization techniques like Newton-CG, do not explicitly compute the Hessian but instead approximate the product of the Hessian with a vector through finite difference. The ...
