Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [optimization]

This tag is intended for questions on methods for the (constrained or unconstrained) minimization or maximization of functions.

72
votes
17answers
78k views

Is there a high quality nonlinear programming solver for Python?

I have several challenging non-convex global optimization problems to solve. Currently I use MATLAB's Optimization Toolbox (specifically, fmincon() with algorithm=<...
25
votes
3answers
18k views

BFGS vs. Conjugate Gradient Method

What considerations should I be making when choosing between BFGS and conjugate gradient for optimization? The function I am trying to fit with these variables are exponential functions; however, the ...
21
votes
8answers
2k views

Software package for constrained optimization?

I am looking to solve a constrained optimization problem where I know the bounds on some of the variables (specifically a boxed constraint). $$ \arg \min_u f(u,x) $$ subject to $$ c(u,x) = 0 $$ $$ ...
19
votes
3answers
652 views

Is it well known that some optimization problems are equivalent to time-stepping?

Given a desired state $y_0$ and a regularization parameter $\beta \in \mathbb R$, consider the problem of finding a state $y$ and a control $u$ to minimize a functional \begin{equation} \frac{1}{2} \...
18
votes
3answers
6k views

Why should non-convexity be a problem in optimization?

I was very surprised when I started to read something about non-convex optimization in general and I saw statements like this: Many practical problems of importance are non-convex, and most non-...
17
votes
5answers
2k views

Finding a global minimum of a smooth, bounded, non-convex 2D function that is costly to evaluate

I have a bounded non-convex 2-D function which I'd like to find the minimum of. The function is quite smooth. Evaluating it is costly. An acceptable error is about 3% of the function's domain in each ...
16
votes
3answers
4k views

Solving unconstrained nonlinear optimization problems on GPU

I am trying to solve some unconstrained nonlinear optimization problems on GPU (CUDA). The objective function is a smooth nonlinear function, and its gradient is relatively cheap to compute ...
15
votes
4answers
1k views

Testing numerical optimization methods: Rosenbrock vs. real test functions

There seem to be two main kinds of test function for no-derivative optimizers: one-liners like the Rosenbrock function ff., with start points sets of real data points, with an interpolator Is it ...
15
votes
3answers
442 views

Scientific Programming Contests

I regularly compete in so called "Programming Contests", where you solve difficult algorithmic problems with your own code and problem solving skills during a limited time-frame. For referential ...
15
votes
1answer
1k views

Intuitive motivation for BFGS update

I am teaching a numerical analysis survey class and am seeking motivation for the BFGS method for students with limited background/intuition in optimization! While I don't have time to prove ...
14
votes
4answers
5k views

Linear programming feasibility problem with strict positivity constraints

There is a system of linear constraints ${\bf Ax} \leq {\bf b}$ . I wish to find a strictly positive vector ${\bf x} > 0$ that satisfies these constraints. That means, $x_i > 0$ is required for ...
14
votes
6answers
17k views

Constraints involving $\max$ in a linear program?

Suppose $$\begin{align*} \min A &\mathrm{vec}(U) \\ &\text{subject to } U_{i,j} \leq \max\{U_{i,k}, U_{k,j}\}, \quad i,j,k = 1, \ldots, n \end{align*}$$ where $U$ is a symmetric $n\times ...
14
votes
4answers
2k views

Selecting most scattered points from a set of points

Is there any (efficient) algorithm to select subset of $M$ points from a set of $N$ points ($M < N$) such that they "cover" most area (over all possible subsets of size $M$)? I assume the points ...
14
votes
1answer
3k views

The Remez Algorithm

The Remez algorithm is a well-known iterative routine to approximate a function by a polynomial in the minimax norm. But, as Nick Trefethen [1] says about it: Most of these [implementations] go ...
13
votes
5answers
11k views

Minimizing the Sum of Absolute Deviation ($ {L}_{1} $ Distance)

I have a data set $x_{1}, x_{2}, \ldots, x_{k}$ and want to find the parameter $m$ such that it minimizes the sum $$\sum_{i=1}^{k}\big|m-x_i\big|.$$ that is $$\min_{m}\sum_{i=1}^{k}\big|m-x_i\big|.$$...
13
votes
5answers
4k views

Parallel optimization algorithms for a problem with very expensive objective function

I am optimizing a function of 10-20 variables. The bad news is that each function evaluation is expensive, approx 30 min of serial computation. The good news is that I have a cluster with a few dozen ...
13
votes
3answers
5k views

Fortran: Best way to time sections of your code?

Sometimes while optimizing code it is required to time certain portions of the code, I have been using the following for years but was wondering if there is a simpler/better way to do it? ...
12
votes
3answers
498 views

Confusion about compressed sensing problem

I read some references including this. I am kind of confused what optimization problem compressed sensing builds and tries to solve. Is it $$\begin{array}{ll} \text{minimize} & \|x\|_1\\ \text{...
12
votes
2answers
2k views

Newton-based methods in optimization vs. solving systems of nonlinear equations

I asked for clarification about a recent question about minpack, and got the following comment: Any system of equations is equivalent to an optimization problem, which is why Newton-based methods ...
12
votes
1answer
3k views

Pressure as a Lagrange Multiplier

In the incompressible Navier-Stokes equations, \begin{align*} \rho\left(\mathbf{u}_t + (\mathbf{u} \cdot \nabla)\mathbf{u}\right) &= - \nabla p + \mu\Delta\mathbf{u} + \mathbf{f}\\ \nabla\cdot\...
12
votes
2answers
10k views

Confusion about Armijo rule

I have this confusion about Armijo rule used in line search. I was reading back tracking line search but didn't get what this Armijo rule is all about. Can anyone elaborate what Armijo rule is? The ...
12
votes
2answers
1k views

Absolute Value in Linear Constraints

I have the following optimization problem where I have absolute value in my constraints: Let $\mathbf{x} \in \mathbb{R}^n$ and $\mathbf{f}_0, \mathbf{f}_1, \ldots, \mathbf{f}_m$ be column vectors of ...
12
votes
2answers
1k views

Decomposition methods for solving large optimization problems

I was wondering if anybody had any suggestions for texts or survey articles on decomposition methods (e.g. primal, dual, Dantzig–Wolfe decompositions) for solving large mathematical programming ...
11
votes
4answers
742 views

Calculating determinant while solving $Ax=b$ using CG

I am solving $Ax=b$ for a huge sparse positive definite matrix $A$ using the conjugate gradient (CG) method. It is possible to compute the determinant of $A$ using the information produced during the ...
11
votes
3answers
3k views

Optimize an unknown function which can be evaluated only?

Given an unknown function $f:\mathbb R^d \to \mathbb R$, we can evaluate its value at any point in its domain, but we don't have its expression. In other words, $f$ is like a black box to us. What is ...
11
votes
2answers
4k views

Solving a least squares problem with linear constraints in Python

I need to solve \begin{alignat}{1} & \min_{x}\|Ax - b\|^2_{2}, \\ \mathrm{s.t.} & \quad\sum_{i}x_{i} = 1, \\ & \quad x_{i} \geq 0, \quad \forall{i}. \end{alignat} I think it is a ...
11
votes
5answers
1k views

Global maximization of expensive objective function

I am interested in globally maximizing a function of many ($\approx 30$) real parameters (a result of a complex simulation). However, the function in question is relatively expensive to evaluate, ...
11
votes
1answer
6k views

Understanding the Wolfe Conditions for an Inexact line search

According to Nocedal & Wright's Book Numerical Optimization (2006), the Wolfe's conditions for an inexact line search are, for a descent direction $p$, Sufficient Decrease: $f(x+\alpha p)\le f(x)+...
11
votes
2answers
3k views

Strategies for Newton's Method when the Jacobian at the solution is singular

I'm trying to solve the following system of equations for the variables $P,x_1$ and $x_2$ (all else are constants): $$\frac{A(1-P)}{2}-k_1x_1=0 \\ \frac{AP}{2}-k_2x_2=0 \\ \frac{(1-P)(r_1+x_1)^4}{L_1}...
11
votes
1answer
639 views

Projecting out the null-space of $A$ from $b$ in $Ax=b$

Given the system $$Ax=b,$$ where $A\in\mathbb{R}^{n\times n}$, I read that, in case Jacobi iteration is used as a solver, the method will not converge if $b$ has a non-zero component in the null-space ...
11
votes
1answer
564 views

Computing standard errors for linear regression problems without calculating inverse

Is there a speedier way to calculate standard errors for linear regression problems, than by inverting $X'X$? Here I assume we have regression: $$y=X\beta+\varepsilon,$$ where $X$ is $n\times k$ ...
11
votes
1answer
326 views

scale invariance for line-search and trust region algorithms

In Nocedal & Wright's book on Numerical Optimization, there is a statement in section 2.2 (page 27), "Generally speaking, it is easier to preserve scale invariance for line search algorithms than ...
10
votes
3answers
2k views

Is providing approximate gradients to a gradient based optimizer useless?

Is it pointless to use gradient based optimization algorithms if you can only provide a numerical gradient? If not, why provide a numerical gradient in the first place if it is trivial to perform ...
10
votes
4answers
2k views

Linear programming with matrix constraints

I have an optimization problem that looks like the following $$ \begin{array}{rl} \min_{J,B} & \sum_{ij} |J_{ij}|\\ \textrm{s.t.} & MJ + BY =X \end{array} $$ Here, my variables are matrices $...
10
votes
2answers
5k views

Gradient descent and conjugate gradient descent

For a project, I have to implement these two methods and compare how they perform on different functions. It looks like the conjugate gradient method is meant to solve systems of linear equations of ...
10
votes
3answers
739 views

Maximizing unknown noisy function

I'm interested in maximizing a function $f(\mathbf \theta)$, where $\theta \in \mathbb R^p$. The problem is that I don't know the analytic form of the function, or of its derivatives. The only thing ...
10
votes
4answers
3k views

Nonlinear least squares with box constraints

What are recommended ways of doing nonlinear least squares, min $\sum err_i(p)^2$, with box constraints $lo_j <= p_j <= hi_j$ ? It seems to me (fools rush in) that one could make the box ...
10
votes
1answer
921 views

Adaptive gradient descent step size when you can't do a line search

I have an objective function $E$ dependent on a value $\phi(x, t = 1.0)$, where $\phi(x, t)$ is the solution to a PDE. I am optimizing $E$ by gradient descent on the initial condition of the PDE: $\...
10
votes
2answers
1k views

Understanding the cost of adjoint method for pde-constrained optimization

I'm trying to understand how the adjoint-based optimization method works for a PDE constrained optimization. Particularly, I'm trying to understand why the adjoint method is more efficient for ...
10
votes
2answers
4k views

Meaning of (meta)heuristic methods

For optimization, from Wikipedia: In computer science, metaheuristic designates a computational method that optimizes a problem by iteratively trying to improve a candidate solution with regard ...
10
votes
1answer
3k views

Efficient solution of mixed integer linear programs

Many important problems can be expressed as a mixed integer linear program. Unfortunately computing the optimal solution to this class of problems is NP-Complete. Luckily there are approximation ...
10
votes
2answers
360 views

Eigenvectors of a small norm adjustment

I have a dataset that is slowly changing, and I need to keep track of eigenvectors/eigenvalues of its covariance matrix. I've been using scipy.linalg.eigh, but it'...
10
votes
1answer
5k views

Are there any heuristics for optimizing the successive over-relaxation (SOR) method?

As I understand it, successive over relaxation works by choosing a parameter $0\leq\omega\leq2$ and using a linear combination of a (quasi) Gauss-Seidel iteration and the value at the previous ...
10
votes
1answer
2k views

Why are interior point methods difficult to warm start?

I often encounter the general adage that interior point methods are difficult to warm start. Is there an intuitive explanation behind this advice? Are there situations in which one can expect benefits ...
10
votes
4answers
6k views

Maximizing a convex function (minimizing a concave function) with a linear constraint

The problem is $$\max f(\mathbf{x}) \text{ subject to } \mathbf{Ax} = \mathbf{b}$$ where $f(\mathbf{x}) = \sum_{i=1}^N\sqrt{1+\frac{x_i^4}{(\sum_{i=1}^{N}x_i^2)^2}}$, $\mathbf{x} = [x_1,x_2,...,x_N]...
10
votes
2answers
2k views

How is geometric programming different from convex programming?

How is (generalized) geometric programming different from general convex programming? A geometric program can be transformed into a convex program, and is typically solved by an interior point method....
10
votes
2answers
2k views

Calculating Lagrange coefficients for SVM in Python

I'm trying to write a full SVM implementation in Python and I have a few issues computing the Lagrange coefficients. First let me rephrase what I understand from the algorithm to make sure I'm on the ...
10
votes
2answers
647 views

Trace An Isoline of an Expensive 2D Function

I have a problem similar in formulation to this post, with a few notable differences: What simple methods are there for adaptively sampling a 2D function? Like in that post: I have a $f(x,y)$ and ...
9
votes
2answers
772 views

Meaning of search methods and optimization methods

I was wondering what differences and relations are between "search methods" and "optimization methods"? Especially when solving an optimization problem? I emphasize the context of solving ...
9
votes
2answers
706 views

Initially Bracketing Minimum for Line Search

Leafing through a few textbooks, I've noticed that the problem of initially bracketing a minimum during a line search tends be an afterthought (at least in my undergraduate texts). Are there well-...