Questions tagged [optimization]

This tag is intended for questions on methods for the (constrained or unconstrained) minimization or maximization of functions.

Filter by
Sorted by
Tagged with
22
votes
8answers
3k views

Software package for constrained optimization?

I am looking to solve a constrained optimization problem where I know the bounds on some of the variables (specifically a boxed constraint). $$ \arg \min_u f(u,x) $$ subject to $$ c(u,x) = 0 $$ $$ ...
12
votes
2answers
2k views

Newton-based methods in optimization vs. solving systems of nonlinear equations

I asked for clarification about a recent question about minpack, and got the following comment: Any system of equations is equivalent to an optimization problem, which is why Newton-based methods ...
10
votes
1answer
6k views

Are there any heuristics for optimizing the successive over-relaxation (SOR) method?

As I understand it, successive over relaxation works by choosing a parameter $0\leq\omega\leq2$ and using a linear combination of a (quasi) Gauss-Seidel iteration and the value at the previous ...
4
votes
2answers
8k views

Tikhonov regularization in the non-negative least square - NNLS (python:scipy)

I am working on a project that I need to add a regularization into the NNLS algorithm. Is there a way to add the Tikhonov regularization into the NNLS implementation of scipy [1]? [2] talks about it, ...
10
votes
2answers
475 views

Eigenvectors of a small norm adjustment

I have a dataset that is slowly changing, and I need to keep track of eigenvectors/eigenvalues of its covariance matrix. I've been using scipy.linalg.eigh, but it'...
15
votes
4answers
2k views

Testing numerical optimization methods: Rosenbrock vs. real test functions

There seem to be two main kinds of test function for no-derivative optimizers: one-liners like the Rosenbrock function ff., with start points sets of real data points, with an interpolator Is it ...
12
votes
5answers
2k views

Global maximization of expensive objective function

I am interested in globally maximizing a function of many ($\approx 30$) real parameters (a result of a complex simulation). However, the function in question is relatively expensive to evaluate, ...
9
votes
5answers
1k views

How can I automate the process of optimizing the design of a physical object?

I'm trying to optimize a flow distributor in a tank such that the velocity and temperature distribution across any cross-section is relatively uniform. There are many parameters I can adjust to the ...
10
votes
1answer
2k views

Sensitivity of BFGS to initial Hessian approximations

I'm trying to implement the Broyden-Fletcher-Goldfarb-Shanno method to find the minimum of a function. I need two initial guesses $x_{-1}$ & $x_0$ and an initial Hessian Matrix approximation $B_0$...
5
votes
2answers
315 views

Approximating and visualizing basins of attraction

I am working on estimating the position and orientation (pose) of a model (rigid object) from its silhouette in an image. For this, I have constructed an error measure between the model in its pose ...
3
votes
1answer
213 views

Finding zeroes of an infinitely differentiable function of ~100 to ~1000 variables

I have a function that is not only infinitely differentiable, but it is also very cheap to calculate any of those derivatives. It looks like: $f(\boldsymbol{C}, \boldsymbol{x})=\sum_{i} C_{i} \prod_{...
1
vote
0answers
188 views

Oscillating convergence in my Resilient BackPropagation (RPROP) implementation

I have implemented in matlab a neural network that uses rprop's algorithm to update its weights. Strangely the error on the training set does not converge to a local minimum, but oscillates. Here is ...
1
vote
1answer
343 views

How to formulate variance minimization as a mixed integer quadratic program

I have a mixed integer quadratic problem and my objective function is as follows $$\arg \min \operatorname{Var}(f(x),g(x)) + \operatorname{Var}(c(x),d(x)) + \cdots$$ where $f$, $g$, $c$ $d$ are ...
27
votes
3answers
21k views

BFGS vs. Conjugate Gradient Method

What considerations should I be making when choosing between BFGS and conjugate gradient for optimization? The function I am trying to fit with these variables are exponential functions; however, the ...
16
votes
5answers
5k views

Parallel optimization algorithms for a problem with very expensive objective function

I am optimizing a function of 10-20 variables. The bad news is that each function evaluation is expensive, approx 30 min of serial computation. The good news is that I have a cluster with a few dozen ...
19
votes
3answers
794 views

Is it well known that some optimization problems are equivalent to time-stepping?

Given a desired state $y_0$ and a regularization parameter $\beta \in \mathbb R$, consider the problem of finding a state $y$ and a control $u$ to minimize a functional \begin{equation} \frac{1}{2} \...
16
votes
4answers
4k views

Selecting most scattered points from a set of points

Is there any (efficient) algorithm to select subset of $M$ points from a set of $N$ points ($M < N$) such that they "cover" most area (over all possible subsets of size $M$)? I assume the points ...
13
votes
1answer
7k views

Understanding the Wolfe Conditions for an Inexact line search

According to Nocedal & Wright's Book Numerical Optimization (2006), the Wolfe's conditions for an inexact line search are, for a descent direction $p$, Sufficient Decrease: $f(x+\alpha p)\le f(x)+...
20
votes
3answers
8k views

Why should non-convexity be a problem in optimization?

I was very surprised when I started to read something about non-convex optimization in general and I saw statements like this: Many practical problems of importance are non-convex, and most non-...
17
votes
5answers
2k views

Finding a global minimum of a smooth, bounded, non-convex 2D function that is costly to evaluate

I have a bounded non-convex 2-D function which I'd like to find the minimum of. The function is quite smooth. Evaluating it is costly. An acceptable error is about 3% of the function's domain in each ...
15
votes
1answer
2k views

Intuitive motivation for BFGS update

I am teaching a numerical analysis survey class and am seeking motivation for the BFGS method for students with limited background/intuition in optimization! While I don't have time to prove ...
13
votes
2answers
13k views

Confusion about Armijo rule

I have this confusion about Armijo rule used in line search. I was reading back tracking line search but didn't get what this Armijo rule is all about. Can anyone elaborate what Armijo rule is? The ...
11
votes
2answers
2k views

Understanding the cost of adjoint method for pde-constrained optimization

I'm trying to understand how the adjoint-based optimization method works for a PDE constrained optimization. Particularly, I'm trying to understand why the adjoint method is more efficient for ...
4
votes
2answers
1k views

Line search for Newton method

If we want to solve nonlinear minimization problem $$\min_{x} f(x),$$ making least-squares assumption and using Gauss-Newton method so that at k$th$ iteration we have: $$J_k^T J_k p_k = - J_k^T ...
10
votes
2answers
741 views

Trace An Isoline of an Expensive 2D Function

I have a problem similar in formulation to this post, with a few notable differences: What simple methods are there for adaptively sampling a 2D function? Like in that post: I have a $f(x,y)$ and ...
4
votes
1answer
123 views

Which optimization method can be used to do the following?

I've the following system of equations for studying information flow in the below graph, $$ \frac{d \phi}{dt} = -M^TDM\phi + \text{noise effects} \hspace{1cm} (1)$$ Here, M is the incidence ...
12
votes
2answers
4k views

Strategies for Newton's Method when the Jacobian at the solution is singular

I'm trying to solve the following system of equations for the variables $P,x_1$ and $x_2$ (all else are constants): $$\frac{A(1-P)}{2}-k_1x_1=0 \\ \frac{AP}{2}-k_2x_2=0 \\ \frac{(1-P)(r_1+x_1)^4}{L_1}...
12
votes
3answers
5k views

Optimize an unknown function which can be evaluated only?

Given an unknown function $f:\mathbb R^d \to \mathbb R$, we can evaluate its value at any point in its domain, but we don't have its expression. In other words, $f$ is like a black box to us. What is ...
12
votes
2answers
5k views

Solving a least squares problem with linear constraints in Python

I need to solve \begin{alignat}{1} & \min_{x}\|Ax - b\|^2_{2}, \\ \mathrm{s.t.} & \quad\sum_{i}x_{i} = 1, \\ & \quad x_{i} \geq 0, \quad \forall{i}. \end{alignat} I think it is a ...
10
votes
2answers
3k views

Calculating Lagrange coefficients for SVM in Python

I'm trying to write a full SVM implementation in Python and I have a few issues computing the Lagrange coefficients. First let me rephrase what I understand from the algorithm to make sure I'm on the ...
10
votes
4answers
4k views

Nonlinear least squares with box constraints

What are recommended ways of doing nonlinear least squares, min $\sum err_i(p)^2$, with box constraints $lo_j <= p_j <= hi_j$ ? It seems to me (fools rush in) that one could make the box ...
8
votes
1answer
645 views

Linear system solution with inequality constraints - methods?

First of all, I hope I am posting this in the correct place. If not, I'm sorry and could you please direct me to where I should post this? Problem: You are given a set of vectors, $\{\mathbf{a}^i\}_{...
6
votes
1answer
859 views

The speed and memory requirement of minpack

I am considering minpack software package to solve my optimization problem ( this is the kind of question that I am facing), but I don't quite know what is the memory requirement and the speed of this ...
6
votes
1answer
167 views

zero terminal value of the adjoint based optimal control

I have been pondering about this issue for some time... Say, I want to minimize a costfunctional $$ \tilde J(u) = J(v(u),u) = \frac 12 \int_0^T (v-v_0)^2 + \alpha u^2 dt $$ subject to $$ \dot v = v^2 ...
5
votes
0answers
523 views

Best way to add a positivity constraint to Newton's Method

So given an objective function $f({\bf x})$, I would like to include a positivity constraint when I perform the fixed point iteration: $${\bf x}^{(t+1)}={\bf x}^{(t)} - \text{H}_f^{-1}\nabla f({\bf x}^...
2
votes
1answer
894 views

Doubt regarding stopping criterion for Newton method

I am solving an unconstrained convex optimization problem, which can easily have a million variables. I am trying to get a working system with a toy problem of around 200 variables. I am noticing that ...
15
votes
3answers
7k views

Fortran: Best way to time sections of your code?

Sometimes while optimizing code it is required to time certain portions of the code, I have been using the following for years but was wondering if there is a simpler/better way to do it? ...
9
votes
2answers
1k views

Safe application of iterative methods on diagonally dominant matrices

Suppose the following linear system is given $$Lx=c,\tag1$$ where $L$ is the weighted Laplacian known to be positive $semi-$definite with a one dimensional null space spanned by $1_n=(1,\dots,1)\in\...
8
votes
2answers
8k views

Simultaneous maximization of two functions without available derivatives

I have two variables k and t as functions of two other variables p1 and ...
7
votes
3answers
1k views

Nearest positive semidefinite matrix to a symmetric matrix in the spectral norm

So I have a symmetric matrix $A$ and I would like to solve the optimization problem, $$\hspace{2.5mm}\text{Minimize}\;\; \|A-S\|_2$$ $$\hspace{-5mm}\text{Subject to}\;\; S\geq0.$$ $A$ is given and $S$ ...
7
votes
3answers
1k views

Finding the first N roots of transcendental equation

I need to find the first $n$ roots of the transcendental equation \begin{equation} F(k) = J_m'(kr)Y_m'(k)-J'_m(k)Y'_m(kr) \end{equation} for integer values of $m$ and any $r \in [0,1)$ where $J'$ ...
6
votes
2answers
563 views

Recommendations for a usable, fast GPL-compatible derivative-free numerical optimization library that can be interfaced to C++

I am dealing with optimization of functions for which I do not have derivatives available, and the optimization is not constrained. I am searching for a high quality GNU Public License-compatible ...
6
votes
2answers
2k views

What is the most appropriate derivative free optimization algorithm

We can use random optimization/ derivative free/ direct search to find the minimum of some black box function $f$. If I have some 2D black box function, $f(x,y)$ - which I know to be convex - what ...
5
votes
0answers
387 views

Optimization on the manifold of stochastic matrices

So I have an optimization problem of the form $$\text{maximize}\hspace{3mm}f(A):{\bf R}^{K\times K}\rightarrow{\bf R}$$ $$\text{subject to}\hspace{19mm}A^T{\bf 1}=\bf{1}$$ $$\hspace{33mm}A\geq 0$$ ...
5
votes
2answers
2k views

LP feasibility checking

I have a linear programming problem. I want to know if this LP is feasible. What is the best known algorithm for checking feasibility of an LP or a linear system of equations?
4
votes
1answer
236 views

constrained minimization in N dimensions

I am looking to create an algorithm to minimize an N dimensional problem. I am unsure how to write it in its generic form, so I will show it in 1, 2 and 3 dimensions Minimize $ \sum_{i} x_i\left [ f\...
4
votes
1answer
3k views

Using scipy.optimize to implement a neural network with back propagation

My problem is something similar to this. I'm trying to implement a (Neural Network) Cost function, Back propogation algorithm in Python. The Neural Network has 3 layers. Hence 2 parameters to optimize ...
3
votes
2answers
759 views

Low-rank updates in BFGS

I have read this and other threads on this site on BFGS, but I still don't have a clear understanding of what it's meant by low-rank updates. For example, I read the following in this book: The ...
3
votes
1answer
77 views

Evaluate 3D Shape Descriptor

I'm trying to create my own 3d shape descriptor, the idea is that how I may evaluate how much my descriptor is well and good? What I checked is that they evaluate descriptors through shape matching, ...
3
votes
0answers
176 views

Find constrained vectors maximizing angles between them - methods?

This is related to a question I had asked earlier, with the distinction that earlier I did not have a non-linear objective functional to minimize. The problem is reproduced below with added ...