Comparing Jacobi and Gauss-Seidel methods for nonlinear iterations

Question

It is well known that for certain linear systems Jacobi and Gauss-Seidel iterative methods have the same convergence behavior, e.g. Stein-Rosenberg Theorem. I am wondering if similar results exist for nonlinear iterations, where at step $k$ the Jacobi iterations on, say $\mathbb{R}^n$, are defined as

$$x_i^{k+1}=F_i(x_1^{k+1},\cdots,x_{i-1}^{k+1},x_i^k,\cdots,x_n^k),$$

and the Gauss-Seidel iterations are defined as

$$x_i^{k+1}=F_i(x_1^k,\cdots,x_n^k)$$

for $i=1,\cdots,n$ and we have a set of nonlinear functions $F_i(\cdot): \mathbb{R}^n\to\mathbb{R}$.

Answer 1

This paper ( http://www.dcs.bbk.ac.uk/~gmagoulas/APNUM.PDF ) seems to be relevant to your question.

Edit (more than a year later): The paper above is "From linear to nonlinear iterative methods" by Vrahatis et al (2003). Their main application is neural network training, but they also present some results for a few classical optimization problems.

The part that is relevant to this question is Theorems 1 and 2. Theorem 1 states that if $f$ is twice differentiable in an open neighbourhood around the minimizer $x^*$ and the Hessian (of $f$) $H(x^*)$ is positive definite with Property A, then the nonlinear Jacobi iterations for $f$ converges to $x^*$ given that the initial guess $x_0$ is sufficiently good. Theorem 2 says that the non-linear SOR scheme will converge for any sufficiently close initial guess $x_0$ if $f$ is twice differentiable in an open neighborhood $S_0$ around the minimizer $x^*$.

As a further summary, both methods are locally convergent given that the function is nice, and you have a good initial guess.

There are further results presented in the paper, in addition to some suggestions for developing globally convergent algorithms. But, note that this particular paper is published in 2003, so the state-of-the-art is more advanced now, and I didn't do a thorough review.