Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [optimization]

This tag is intended for questions on methods for the (constrained or unconstrained) minimization or maximization of functions.

21
votes
8answers
2k views

Software package for constrained optimization?

I am looking to solve a constrained optimization problem where I know the bounds on some of the variables (specifically a boxed constraint). $$ \arg \min_u f(u,x) $$ subject to $$ c(u,x) = 0 $$ $$ ...
12
votes
2answers
2k views

Newton-based methods in optimization vs. solving systems of nonlinear equations

I asked for clarification about a recent question about minpack, and got the following comment: Any system of equations is equivalent to an optimization problem, which is why Newton-based methods ...
10
votes
1answer
5k views

Are there any heuristics for optimizing the successive over-relaxation (SOR) method?

As I understand it, successive over relaxation works by choosing a parameter $0\leq\omega\leq2$ and using a linear combination of a (quasi) Gauss-Seidel iteration and the value at the previous ...
10
votes
2answers
360 views

Eigenvectors of a small norm adjustment

I have a dataset that is slowly changing, and I need to keep track of eigenvectors/eigenvalues of its covariance matrix. I've been using scipy.linalg.eigh, but it'...
15
votes
4answers
1k views

Testing numerical optimization methods: Rosenbrock vs. real test functions

There seem to be two main kinds of test function for no-derivative optimizers: one-liners like the Rosenbrock function ff., with start points sets of real data points, with an interpolator Is it ...
11
votes
5answers
1k views

Global maximization of expensive objective function

I am interested in globally maximizing a function of many ($\approx 30$) real parameters (a result of a complex simulation). However, the function in question is relatively expensive to evaluate, ...
8
votes
5answers
919 views

How can I automate the process of optimizing the design of a physical object?

I'm trying to optimize a flow distributor in a tank such that the velocity and temperature distribution across any cross-section is relatively uniform. There are many parameters I can adjust to the ...
4
votes
2answers
6k views

Tikhonov regularization in the non-negative least square - NNLS (python:scipy)

I am working on a project that I need to add a regularization into the NNLS algorithm. Is there a way to add the Tikhonov regularization into the NNLS implementation of scipy [1]? [2] talks about it, ...
5
votes
2answers
240 views

Approximating and visualizing basins of attraction

I am working on estimating the position and orientation (pose) of a model (rigid object) from its silhouette in an image. For this, I have constructed an error measure between the model in its pose ...
3
votes
1answer
202 views

Finding zeroes of an infinitely differentiable function of ~100 to ~1000 variables

I have a function that is not only infinitely differentiable, but it is also very cheap to calculate any of those derivatives. It looks like: $f(\boldsymbol{C}, \boldsymbol{x})=\sum_{i} C_{i} \prod_{...
1
vote
0answers
144 views

Oscillating convergence in my Resilient BackPropagation (RPROP) implementation

I have implemented in matlab a neural network that uses rprop's algorithm to update its weights. Strangely the error on the training set does not converge to a local minimum, but oscillates. Here is ...
1
vote
1answer
215 views

How to formulate variance minimization as a mixed integer quadratic program

I have a mixed integer quadratic problem and my objective function is as follows $$\arg \min \operatorname{Var}(f(x),g(x)) + \operatorname{Var}(c(x),d(x)) + \cdots$$ where $f$, $g$, $c$ $d$ are ...
25
votes
3answers
18k views

BFGS vs. Conjugate Gradient Method

What considerations should I be making when choosing between BFGS and conjugate gradient for optimization? The function I am trying to fit with these variables are exponential functions; however, the ...
13
votes
5answers
4k views

Parallel optimization algorithms for a problem with very expensive objective function

I am optimizing a function of 10-20 variables. The bad news is that each function evaluation is expensive, approx 30 min of serial computation. The good news is that I have a cluster with a few dozen ...
19
votes
3answers
651 views

Is it well known that some optimization problems are equivalent to time-stepping?

Given a desired state $y_0$ and a regularization parameter $\beta \in \mathbb R$, consider the problem of finding a state $y$ and a control $u$ to minimize a functional \begin{equation} \frac{1}{2} \...
11
votes
1answer
6k views

Understanding the Wolfe Conditions for an Inexact line search

According to Nocedal & Wright's Book Numerical Optimization (2006), the Wolfe's conditions for an inexact line search are, for a descent direction $p$, Sufficient Decrease: $f(x+\alpha p)\le f(x)+...
18
votes
3answers
6k views

Why should non-convexity be a problem in optimization?

I was very surprised when I started to read something about non-convex optimization in general and I saw statements like this: Many practical problems of importance are non-convex, and most non-...
14
votes
4answers
2k views

Selecting most scattered points from a set of points

Is there any (efficient) algorithm to select subset of $M$ points from a set of $N$ points ($M < N$) such that they "cover" most area (over all possible subsets of size $M$)? I assume the points ...
4
votes
2answers
1k views

Line search for Newton method

If we want to solve nonlinear minimization problem $$\min_{x} f(x),$$ making least-squares assumption and using Gauss-Newton method so that at k$th$ iteration we have: $$J_k^T J_k p_k = - J_k^T ...
17
votes
5answers
2k views

Finding a global minimum of a smooth, bounded, non-convex 2D function that is costly to evaluate

I have a bounded non-convex 2-D function which I'd like to find the minimum of. The function is quite smooth. Evaluating it is costly. An acceptable error is about 3% of the function's domain in each ...
15
votes
1answer
1k views

Intuitive motivation for BFGS update

I am teaching a numerical analysis survey class and am seeking motivation for the BFGS method for students with limited background/intuition in optimization! While I don't have time to prove ...
12
votes
2answers
10k views

Confusion about Armijo rule

I have this confusion about Armijo rule used in line search. I was reading back tracking line search but didn't get what this Armijo rule is all about. Can anyone elaborate what Armijo rule is? The ...
10
votes
2answers
647 views

Trace An Isoline of an Expensive 2D Function

I have a problem similar in formulation to this post, with a few notable differences: What simple methods are there for adaptively sampling a 2D function? Like in that post: I have a $f(x,y)$ and ...
11
votes
2answers
3k views

Strategies for Newton's Method when the Jacobian at the solution is singular

I'm trying to solve the following system of equations for the variables $P,x_1$ and $x_2$ (all else are constants): $$\frac{A(1-P)}{2}-k_1x_1=0 \\ \frac{AP}{2}-k_2x_2=0 \\ \frac{(1-P)(r_1+x_1)^4}{L_1}...
10
votes
4answers
3k views

Nonlinear least squares with box constraints

What are recommended ways of doing nonlinear least squares, min $\sum err_i(p)^2$, with box constraints $lo_j <= p_j <= hi_j$ ? It seems to me (fools rush in) that one could make the box ...
9
votes
1answer
1k views

Sensitivity of BFGS to initial Hessian approximations

I'm trying to implement the Broyden-Fletcher-Goldfarb-Shanno method to find the minimum of a function. I need two initial guesses $x_{-1}$ & $x_0$ and an initial Hessian Matrix approximation $B_0$...
7
votes
1answer
449 views

Linear system solution with inequality constraints - methods?

First of all, I hope I am posting this in the correct place. If not, I'm sorry and could you please direct me to where I should post this? Problem: You are given a set of vectors, $\{\mathbf{a}^i\}_{...
6
votes
1answer
742 views

The speed and memory requirement of minpack

I am considering minpack software package to solve my optimization problem ( this is the kind of question that I am facing), but I don't quite know what is the memory requirement and the speed of this ...
2
votes
1answer
621 views

Doubt regarding stopping criterion for Newton method

I am solving an unconstrained convex optimization problem, which can easily have a million variables. I am trying to get a working system with a toy problem of around 200 variables. I am noticing that ...
13
votes
3answers
5k views

Fortran: Best way to time sections of your code?

Sometimes while optimizing code it is required to time certain portions of the code, I have been using the following for years but was wondering if there is a simpler/better way to do it? ...
11
votes
3answers
3k views

Optimize an unknown function which can be evaluated only?

Given an unknown function $f:\mathbb R^d \to \mathbb R$, we can evaluate its value at any point in its domain, but we don't have its expression. In other words, $f$ is like a black box to us. What is ...
9
votes
2answers
1k views

Safe application of iterative methods on diagonally dominant matrices

Suppose the following linear system is given $$Lx=c,\tag1$$ where $L$ is the weighted Laplacian known to be positive $semi-$definite with a one dimensional null space spanned by $1_n=(1,\dots,1)\in\...
7
votes
3answers
494 views

Finding the first N roots of transcendental equation

I need to find the first $n$ roots of the transcendental equation \begin{equation} F(k) = J_m'(kr)Y_m'(k)-J'_m(k)Y'_m(kr) \end{equation} for integer values of $m$ and any $r \in [0,1)$ where $J'$ ...
6
votes
2answers
447 views

Recommendations for a usable, fast GPL-compatible derivative-free numerical optimization library that can be interfaced to C++

I am dealing with optimization of functions for which I do not have derivatives available, and the optimization is not constrained. I am searching for a high quality GNU Public License-compatible ...
4
votes
1answer
219 views

constrained minimization in N dimensions

I am looking to create an algorithm to minimize an N dimensional problem. I am unsure how to write it in its generic form, so I will show it in 1, 2 and 3 dimensions Minimize $ \sum_{i} x_i\left [ f\...
4
votes
1answer
122 views

zero terminal value of the adjoint based optimal control

I have been pondering about this issue for some time... Say, I want to minimize a costfunctional $$ \tilde J(u) = J(v(u),u) = \frac 12 \int_0^T (v-v_0)^2 + \alpha u^2 dt $$ subject to $$ \dot v = v^2 ...
3
votes
0answers
134 views

Find constrained vectors maximizing angles between them - methods?

This is related to a question I had asked earlier, with the distinction that earlier I did not have a non-linear objective functional to minimize. The problem is reproduced below with added ...
2
votes
2answers
922 views

LP feasibility checking

I have a linear programming problem. I want to know if this LP is feasible. What is the best known algorithm for checking feasibility of an LP or a linear system of equations?
10
votes
2answers
2k views

Calculating Lagrange coefficients for SVM in Python

I'm trying to write a full SVM implementation in Python and I have a few issues computing the Lagrange coefficients. First let me rephrase what I understand from the algorithm to make sure I'm on the ...
8
votes
2answers
6k views

Simultaneous maximization of two functions without available derivatives

I have two variables k and t as functions of two other variables p1 and ...
5
votes
2answers
763 views

minimization problem: sum of Rayleigh quotients

I would like to find $x$ which minimizes the following equation: $\frac{x^HAx}{x^HBx} + \frac{x^HCx}{x^HDx}$ where A, B, C, D are positive-definite. $x$ is not a very large vector (<1000 elements ...
4
votes
2answers
123 views

Maximization variant of semidefinite programming (SDP)

Consider the following program: $$\max_{\pmb a} \sum_i z_i\\ u.c. \pmb a \pmb P_i\pmb a^\top\geq z_i$$ where $\pmb a \in\mathbb{R}^p$ and the $\pmb P_i$ are all symmetric positive semidefinite ...
4
votes
2answers
1k views

Subgradients of non-convex functions

In these notes (section 2.3), it is stated that: A point $x^*$ is a minimizer of a function $f$ (not necessarily convex) if and only if $f$ is subdifferentiable at $x^*$ and $0 \in\partial f(x^*).$ ...
4
votes
2answers
165 views

Optimization of known function with respect to two unknown function arguments

I have a data set, composed of points $(x_i, y_i)$ for $i=1,N$. I also have a known function $F$, which maps these points $x_i$ to $y_i$ as such $F(x_i, a(x_i),b(x_i)) = y_i$, where $a(x_i)$ and $b(...
3
votes
2answers
161 views

polynomial time solvability

Consider the following optimization problem: $Min \qquad C^TX$ S.t.: $\qquad AX=0;$ $x_ix_j=x_kx_t$$\quad $for some $i\neq j\neq k\neq t$ $X=(x_1,x_2,...x_n)$ and $\quad x_j\geq 0\;\; j=1,2,......
3
votes
2answers
369 views

Low-rank updates in BFGS

I have read this and other threads on this site on BFGS, but I still don't have a clear understanding of what it's meant by low-rank updates. For example, I read the following in this book: The ...
3
votes
2answers
169 views

Find $\min x^TAy$ subject to $1^Tx=1^Ty=1,x\ge 0,y\ge 0$

In the following problem, $A$ is a given $\mathbb{R}^{m\times n}$ matrix: \begin{align} \mbox{minimize}\quad & x^TAy \\ \mbox{subject to}\quad & 1^Tx=1^Ty=1, \\ & x\ge 0,y\ge 0. \end{...
2
votes
3answers
843 views

How to avoid the round-off errors in the larger calculations?

Now I need to sum up more than one thousands of terms and then make the four-dimmensional integral in my Fortran program. I found that there are some numerical errors. Can you give me some suggestions ...
1
vote
3answers
155 views

Optimization of a blackbox function

Let's say that we have an objective function $f(\mathbf x,\mathbf y)$ which has the parameters $\mathbf x=[x_1\ldots x_n]$ and $\mathbf y=[y_1\ldots y_n]$. Here, $\mathbf y$ is a blackbox variable ...
1
vote
1answer
295 views

Feasibility checking

Consider the following optimization problem: $Min\;\;\; CX$ $AX\geq b$ $x_ix_j= x_s x_t\;\;\; i\neq j \neq s\neq t$ $x_j\geq 0;$ Where $A$ is the adjacency matrix and $C$ is a constant vector. ...