20
It's a long-running joke that CFD stands for "colorful fluid dynamics". Nevertheless, it is used -- and useful -- in a wide range of applications.
I believe your discontent stems from not sufficiently distinguishing between two interconnected but different steps: creating a mathematical model of a physical process and solving it numerically. Let me ...
12
Reducing the kernel width $\sigma_m$ will usually reduce the condition number.
However, kernel matrices can become singular, or close to singular, for any basis function or point distribution, provided the basis functions overlap. The reason for this is actually quite simple:
The kernel matrix $K$ is singular when its determinant $\det(K)$ is zero.
...
11
I am not an expert in machine learning, but I can outline the considerations that are relevant.
The numerical calculations in machine learning are generally linear algebra -- either solving linear systems or linear least squares. For both types of problems, there are well-known backward-stable methods, so I will assume you are using a backward-stable ...
9
I've implemented this recently, basically it counts how many times each specific colour borders another colour to make up a frequency table. To generate an image, a random colour and position are selected and the rest of the image is built up from there. Results aren't very coherent, but they match the colour palette of the original.
In order to make sure ...
7
I wrote a Python package called PyPGE.
PyPGE is a Symbolic Regression implementation based on Prioritized Grammar Enumeration (1), not Evolutionary or Genetic Programming. It produces a deterministic Symbolic Regression algorithm.
(1) Worm, Tony, and Kenneth Chiu. "Prioritized grammar enumeration: symbolic regression by dynamic programming." Proceedings of ...
7
I think you are mixing a couple different ideas that are causing confusion. Yes, there are a wide variety of ways to discretize a given problem. Choosing an appropriate way may look like "voodoo" when you are learning these things in class, but when researchers choose them, they do so drawing on the combined experience of the field, as published in ...
7
So the way I went about formulating the problem was to essentially write the following equations:
The state that will be estimated, which is defined as a column vector, is the following:
$$w = [vec(A)^{T},\bar{N}^{T}]^{T}$$
where $\bar{N}$ is the unknown average $N$ vector and $vec(A)$ is the vectorization operator on the unknown matrix A.
Based on the ...
7
First of all, interpolation and approximation are slightly different from each other.
Given a sufficiently smooth function $f$ (sufficiently smooth just means that I am covering my bases, there are many theorems in approximation theory considering different function classes, from discontinuous -measure zero- functions to infinitely smooth or analytic or ...
6
If I understand the link of hardmath correctly, a "support vector machine" in its most simple form is just a glorious name for a linear function $f(x)=b^Tx+b_0$ dividing the space into two half spaces according to the sign of the values. And separating two given sets (with disjoint convex hulls for existence of a separating hyperplane).
The original ...
5
I suppose, you right and your network is not that big to 100%-utilize the GPU. The bottle-neck here seems to be not the GPU itself, but the transfer rate between RAM and VRAM and here the difference between 750 Ti and 780 Ti is not that significant. You can try to improve the training speed by hiding the latency of memory transfer - you have to assure you ...
5
The general idea that you have of learning an easy to compute model from results of your detailed simulation model and then optimizing the easy to compute model is long-established. The easy to compute model is typically called a surrogate model or a response surface model. Once the surrogate is available, you can use conventional optimization techniques ...
5
So bottom line is I don't see any comprehensive work on the use of AI in M&S as a whole, let's say having models that can learn how to produce new improved models using the existing models.
There's definitely some work out there on this. This is the field of scientific machine learning. Currently there's three major paths that I'd break it into:
Neural ...
5
So, you want to invert your matrix $A=\Phi^T\Phi$. For $A$ to be invertible it must not have zero eigenvalues. We can show that $A$ is positive semi-definite as follows. Positive semi-definite means that the eigenvalues of $A$ are $\geq 0$. This is equivalent to showing $y^TAy \geq 0, \forall y \neq 0 $.
$$
y^TAy = y^T\Phi^T\Phi{y}=(\Phi{y})^T(\Phi{y}) \geq ...
4
First, just to clarify things, you're talking about solving optimization problems of the form
$\min \| x \|_{1}$
subject to
$\| Ax - b \|_{2} \leq \delta$
and related forms, right?
There are many different applications in which problems are formulated as the minimization of the 1-norm of a vector subject to linear or least squares constraints.
An ...
4
A couple of suggestions:
Choose $\sigma \sim$ the average distance | random $x$ - nearest $x_i$.
(A cheap approximation for $N$ points uniformly distributed in the unit cube in $\mathbb{R}^d, d\ 2 .. 5$, is
0.5 / $N^{1/d}$.)
We want $\phi( |x - x_i| )$ to be large for $x_i$ near $x$,
small for background noise; plot that for a few random $x$.
Shift $K$ ...
4
I once started writing anopen source version of Eureqa in Java. The project has limited capabilities but it implements the fitness function described in [1] and couple optimizations mentioned by the authors in other publications (e.g., searching for solutions in Pareto front).
Link: https://github.com/pkoperek/hubert
[1] Schmidt, Michael, and Hod Lipson. "...
4
After a cursory google search on the subject, it appears that "symbolic regression" is a problem that lends itself greatly to stochastic optimization algorithms like genetic programming (GP). It is conceivable that you should look for an open source genetic programming library with modules specifically for symbolic regression, such as DEAP (Distributed ...
4
This topic has been discussed at some length on Cross Validated (aka stats.stackexchange) and Reddit:
Why is Newton's method not widely used in machine learning? (see in particular Nick Alger's answer)
Why use gradient descent with neural networks?
L-BFGS and neural nets
Why second order SGD convergence methods are unpopular for deep learning?
How does the ...
4
Think of the simplest case when $\Phi$ is a scalar value.
Not well defined:
$$ \boldsymbol \theta^\text{ML} = (0^T 0)^{-1}0^T ~ y = \frac{1}{0} 0~y= \frac{0}{0} $$
Well defined:
$$ \boldsymbol \theta^\text{ML} = (0^T 0 + \kappa)^{-1}0^T~y =\frac{1}{\kappa} 0 ~y= 0 $$
4
Look up something on Tikhonov regularization, also known as ridge regression in machine learning. This is a standard technique (but I agree that the explanation in that notebook is somewhat poor).
Technically speaking, it does not affect the numerical stability of that algorithm, but it modifies the problem to a more well-conditioned one, from $\min \|\Phi \...
3
The problem is not convex, as one can verify using the simplest possible choice: $N=2, d=1$ and with data $x_1=x_2=1, f_1=1, f_2=0$. In that case, your objective function is
$$
F(C) = 2\left(1-\exp(-C/2)\right)^2.
$$
Plotting this function shows that it is not convex.
3
I found the gramEvol R package flexible and easy to use.
They have a small tutorial in which they rederive Kepler's third law from data.
Note that it relies on Genetic Programmic for its optimisation and thus might return different results if you run it twice.
3
I think I can address the second part of your Question.
The phrase "sum-normalized to zero" is a fancy way of saying "subtract the mean (average)", i.e. subtract the constant needed to give a zero sum over the resulting function (filter) values.
The phrase "square-normalized to 1" applies to the result of the first phrase and means dividing by a (...
3
I learn computation science through Practical Numerical Methods with Python.
https://github.com/numerical-mooc/numerical-mooc/wiki
It covers finite differencing and many other numerical algorithms. For me the most interesting part is that it shows how various factors impact the stability, accuracy, and performance through small working examples.
3
This metric is pretty much as misleading (or useful, depending on your perspective) for GPUs as it is for CPUs.
Currently, a lot of applications/algorithm's implementations are limited more by memory throughput rather than FLOPs. Memory throughput (in GB/s) is also always listed for GPU specification and those two numbers together give a much better view on ...
3
I have limited experience in machine learning; however, simplifying, you can think of it as a "trained black box". I would say, you have to know a lot about your problem, the behaviour of your functions in order to successfully and reliably apply any form of machine learning. You would have to decide on how to get the training data, size of the training set, ...
3
This seems to be the "Cocktail Party Problem". Andrew Ng's machine learning course on Coursera gives a solution based on SVD for this problem. See the first week's course notes. Ng refers to Sam Roweis, Yair Weiss & Eero Simoncelli but I can't seem to find the reference on Google Scholar.
2
I assume you didn't specify the fprime parameter. If you don't provide this param fmin_cg has to figure out its own solution what usually is much slower than which a provided optimal solution. Your code might look like this:
theta = fmin_cg(compute_cost_reg, fprime=compute_gradient_reg,
x0=theta, args=(X, y, lambd), maxiter=50)
2
They are simply referring to the fact that the kernel matrix itself is a central quantity in the algorithm. The problem is that it is $O(n^2)$ in size, where $n$ is the number of points being examined. So the storage and computational requirements surrounding the kernel matrix rapidly become impractical as $n$ gets large.
Only top voted, non community-wiki answers of a minimum length are eligible
