20
It's a long-running joke that CFD stands for "colorful fluid dynamics". Nevertheless, it is used -- and useful -- in a wide range of applications.
I believe your discontent stems from not sufficiently distinguishing between two interconnected but different steps: creating a mathematical model of a physical process and solving it numerically. Let me comment ...
11
Reducing the kernel width $\sigma_m$ will usually reduce the condition number.
However, kernel matrices can become singular, or close to singular, for any basis function or point distribution, provided the basis functions overlap. The reason for this is actually quite simple:
The kernel matrix $K$ is singular when its determinant $\det(K)$ is zero.
...
11
I am not an expert in machine learning, but I can outline the considerations that are relevant.
The numerical calculations in machine learning are generally linear algebra -- either solving linear systems or linear least squares. For both types of problems, there are well-known backward-stable methods, so I will assume you are using a backward-stable ...
10
I have recently been working on exactly this topic. You may want to take a look at our paper: http://arxiv.org/abs/1209.2364.
Why are you interested in the runtime prediction of linear algebra routines? Do you intend to use the model for a certain purpose?
8
I suggest using SLEPc to compute a partial SVD. See Chapter 4 of the User's Manual and the SVD man pages for details.
7
You shouldn't get the impression from the cited question that there is no theory, such that brute force testing of "all methods" (whatever that means; it's easy to produce infinite sequences of methods that are hard to exclude a priori) is the only viable approach. The problem is that if you don't know enough about your problem to formulate a coherent ...
7
I think you are mixing a couple different ideas that are causing confusion. Yes, there are a wide variety of ways to discretize a given problem. Choosing an appropriate way may look like "voodoo" when you are learning these things in class, but when researchers choose them, they do so drawing on the combined experience of the field, as published in ...
7
So the way I went about formulating the problem was to essentially write the following equations:
The state that will be estimated, which is defined as a column vector, is the following:
$$w = [vec(A)^{T},\bar{N}^{T}]^{T}$$
where $\bar{N}$ is the unknown average $N$ vector and $vec(A)$ is the vectorization operator on the unknown matrix A.
Based on the ...
7
I've implemented this recently, basically it counts how many times each specific colour borders another colour to make up a frequency table. To generate an image, a random colour and position are selected and the rest of the image is built up from there. Results aren't very coherent, but they match the colour palette of the original.
In order to make sure ...
6
As a future hint, use the double question mark ?? to pull the source of the function:
Looking at the scipy sources this error message comes up if the internal parameter alpha_k is zero or None.
This value in turn is tied to the internal Wolfe Line search algorithm. In particular it is called when the search doesn't find a better value along the search ...
6
There is lots of preexisting work. Most linear algebra library developers publish performance results in terms of floating-point performance which can be converted into run times.
Googling for "DGEMM performance" for example, yields the following: http://math-atlas.sourceforge.net/timing/3_5_10/index.html.
Generally, you can expect the answers to be non-...
6
If I understand the link of hardmath correctly, a "support vector machine" in its most simple form is just a glorious name for a linear function $f(x)=b^Tx+b_0$ dividing the space into two half spaces according to the sign of the values. And separating two given sets (with disjoint convex hulls for existence of a separating hyperplane).
The original ...
5
I wrote a Python package called PyPGE.
PyPGE is a Symbolic Regression implementation based on Prioritized Grammar Enumeration (1), not Evolutionary or Genetic Programming. It produces a deterministic Symbolic Regression algorithm.
(1) Worm, Tony, and Kenneth Chiu. "Prioritized grammar enumeration: symbolic regression by dynamic programming." Proceedings of ...
5
I suppose, you right and your network is not that big to 100%-utilize the GPU. The bottle-neck here seems to be not the GPU itself, but the transfer rate between RAM and VRAM and here the difference between 750 Ti and 780 Ti is not that significant. You can try to improve the training speed by hiding the latency of memory transfer - you have to assure you ...
4
I suggest the irlba package - it produces virtually the same results as svd, yet you can define a smaller number of singular values to solve for. An example, using sparse matrices to solve the Netflix prize, can be found here: http://bigcomputing.blogspot.de/2011/05/bryan-lewiss-vignette-on-irlba-for-svd.html
4
There are many different types of approximations (or "surrogate models") you could try. Some that come to mind are Kriging, MARS, and Radial Basis Functions. These types of surrogate models (as opposed to polynomial regression) can accommodate a wide range of functional relationships, but you might need to experiment a bit to find which works best for your ...
4
I have used cvxopt to implement an SVM before, however in matlab not python. It will definitely serve your purpose, whether its efficient enough will depend on what you are using it for. The most efficient SVMs do not use a QP solver package, they take advantage of some optimizations unique to SVM. Many use an SMO style algorithm to solve it.
LibSVM is ...
4
Here's the
code
and the
output
of a solution that uses the development version of
algopy for
automatic differentiation. As someone else said in the comments, you could also try other python packages. Theano and statsmodels come to mind.
4
First, just to clarify things, you're talking about solving optimization problems of the form
$\min \| x \|_{1}$
subject to
$\| Ax - b \|_{2} \leq \delta$
and related forms, right?
There are many different applications in which problems are formulated as the minimization of the 1-norm of a vector subject to linear or least squares constraints.
An ...
4
This topic has been discussed at some length on Cross Validated (aka stats.stackexchange) and Reddit:
Why is Newton's method not widely used in machine learning? (see in particular Nick Alger's answer)
Why use gradient descent with neural networks?
L-BFGS and neural nets
Why second order SGD convergence methods are unpopular for deep learning?
How does the ...
3
Sounds too simple so maybe I've misunderstood you, but they have binary variables, hence obviously a nonconvex problem (a binary variable is either 0 or 1, which is a nonconvex set)
3
If I interpret the problem right, the task is to assign to each $j$ in the column index set $J$ of the distance matrix an index $x_j$ from the row index set $I$
such that the $x_j$ are all different and the $\sum_j d(x_j,j)$ is minimal.
This can be posed as a combinatorial constraint satisfaction problem to a number of constraint solvers. See, e.g.,
http://...
3
The neural net in the second article doesn't use linear activation functions. It uses thresholded on-off functions (e.g. f(x) = 1 if x > threshold, else 0), hence can model XOR. (A linear activation function would just be f(x) = x).
3
PCA is called this way since it picks the principal components. If you happen to have several components with the same or almost the same eigenvalue and you pick one but not the other, then you can't claim that you picked the principal components. You picked a subset of the principal components. In other words, if your second eigenvalue is doubled, then it's ...
3
A couple of suggestions:
Choose $\sigma \sim$ the average distance | random $x$ - nearest $x_i$.
(A cheap approximation for $N$ points uniformly distributed in the unit cube in $\mathbb{R}^d, d\ 2 .. 5$, is
0.5 / $N^{1/d}$.)
We want $\phi( |x - x_i| )$ to be large for $x_i$ near $x$,
small for background noise; plot that for a few random $x$.
Shift $K$ ...
3
I think I can address the second part of your Question.
The phrase "sum-normalized to zero" is a fancy way of saying "subtract the mean (average)", i.e. subtract the constant needed to give a zero sum over the resulting function (filter) values.
The phrase "square-normalized to 1" applies to the result of the first phrase and means dividing by a (...
3
After a cursory google search on the subject, it appears that "symbolic regression" is a problem that lends itself greatly to stochastic optimization algorithms like genetic programming (GP). It is conceivable that you should look for an open source genetic programming library with modules specifically for symbolic regression, such as DEAP (Distributed ...
3
I once started writing anopen source version of Eureqa in Java. The project has limited capabilities but it implements the fitness function described in [1] and couple optimizations mentioned by the authors in other publications (e.g., searching for solutions in Pareto front).
Link: https://github.com/pkoperek/hubert
[1] Schmidt, Michael, and Hod Lipson. "...
Only top voted, non community-wiki answers of a minimum length are eligible
