Skip to main content
20 votes
Accepted

Sensitivity of $y$ w.r.t. to $x$ in $y=f(x)$ where f is a routine

The act of using automatic differentiation for derivative calculations of general programs has become known as differentiable programming. There are many grades of differentiable programming across ...
Chris Rackauckas's user avatar
17 votes
Accepted

Why aren't Krylov subspace methods popular in the Machine Learning community compared to Gradient Descent?

On a basic level, I don't buy the argument that you have to "solve a linear system for many machine learning algorithms". Much more, you usually have to optimize a non-linear equation which ...
davidhigh's user avatar
  • 3,177
13 votes

Faster Logistic Function

Yes! There are nice approximations of the logistic. Plot of Approximating Functions As shown below, several functions approximate the logistic (shown as blue dots). This graph is available ...
Richard's user avatar
  • 3,971
11 votes
Accepted

Markov (Chain) image generators?

I've implemented this recently, basically it counts how many times each specific colour borders another colour to make up a frequency table. To generate an image, a random colour and position are ...
Jonno_FTW's user avatar
  • 226
9 votes
Accepted

When training a neural network, why choose Adam over L-BGFS for the optimizer?

This topic has been discussed at some length on Cross Validated (aka stats.stackexchange) and Reddit: Why is Newton's method not widely used in machine learning? (see in particular Nick Alger's ...
GoHokies's user avatar
  • 2,216
7 votes
Accepted

What are the benefits of using machine learning for interpolation over traditional interpolation methods?

First of all, interpolation and approximation are slightly different from each other. Given a sufficiently smooth function $f$ (sufficiently smooth just means that I am covering my bases, there are ...
Abdullah Ali Sivas's user avatar
7 votes

Why aren't Krylov subspace methods popular in the Machine Learning community compared to Gradient Descent?

They aren't popular because they don't work. Nicol N. Schraudolph spent a few years on Krylov-like methods for Machine Learning. I first learned of his work at 2004 Machine Learning Summer School in ...
Yaroslav Bulatov's user avatar
6 votes
Accepted

Artificial Intelligence, Modeling and Simulation

So bottom line is I don't see any comprehensive work on the use of AI in M&S as a whole, let's say having models that can learn how to produce new improved models using the existing models. There'...
Chris Rackauckas's user avatar
6 votes

Faster Logistic Function

If only low-accuracy approximations are needed, it is highly advisable to perform all computation in single precision, for example IEEE-754 binary32 format, usually ...
njuffa's user avatar
  • 1,895
5 votes

Why am I not seeing faster neural network training after upgrading to a vastly better GPU?

I suppose, you right and your network is not that big to 100%-utilize the GPU. The bottle-neck here seems to be not the GPU itself, but the transfer rate between RAM and VRAM and here the difference ...
Vlad's user avatar
  • 114
5 votes
Accepted

Machine Learning for Optimization

The general idea that you have of learning an easy to compute model from results of your detailed simulation model and then optimizing the easy to compute model is long-established. The easy to ...
Brian Borchers's user avatar
5 votes

Advantage of diagonal "jitter" for numerical stability?

Look up something on Tikhonov regularization, also known as ridge regression in machine learning. This is a standard technique (but I agree that the explanation in that notebook is somewhat poor). ...
Federico Poloni's user avatar
5 votes
Accepted

Advantage of diagonal "jitter" for numerical stability?

So, you want to invert your matrix $A=\Phi^T\Phi$. For $A$ to be invertible it must not have zero eigenvalues. We can show that $A$ is positive semi-definite as follows. Positive semi-definite means ...
NNN's user avatar
  • 760
5 votes

Is it possible to express an arbitrary tensor contraction in terms of BLAS routines?

You can use reshape and pointwise multiply to reduce the operation to a matmul in terms of two temporary tensors $c_1,c_2$. Consider following transformations: pointwise mul to turn $a_i * b_i$ into $...
Yaroslav Bulatov's user avatar
4 votes

Advantage of diagonal "jitter" for numerical stability?

Think of the simplest case when $\Phi$ is a scalar value. Not well defined: $$ \boldsymbol \theta^\text{ML} = (0^T 0)^{-1}0^T ~ y = \frac{1}{0} 0~y= \frac{0}{0} $$ Well defined: $$ \boldsymbol \theta^...
ConvexHull's user avatar
  • 1,388
3 votes
Accepted

Interpolation vs. Neural network

I have limited experience in machine learning; however, simplifying, you can think of it as a "trained black box". I would say, you have to know a lot about your problem, the behaviour of your ...
Anton Menshov's user avatar
  • 8,692
3 votes

Learning computational science through guided discovery

I learn computation science through Practical Numerical Methods with Python. https://github.com/numerical-mooc/numerical-mooc/wiki It covers finite differencing and many other numerical algorithms. ...
TheBusyTypist's user avatar
3 votes
Accepted

Why is FLOP(Floating Point Operations Per Second) mentioned as a specification on every GPU?

This metric is pretty much as misleading (or useful, depending on your perspective) for GPUs as it is for CPUs. Currently, a lot of applications/algorithm's implementations are limited more by memory ...
Anton Menshov's user avatar
  • 8,692
3 votes

Seeking a free symbolic regression software

I found the gramEvol R package flexible and easy to use. They have a small tutorial in which they rederive Kepler's third law from data. Note that it relies on Genetic Programmic for its optimisation ...
Nestor Demeure's user avatar
3 votes

How to filter customer voice from customer - agent conversation recordings?

This seems to be the "Cocktail Party Problem". Andrew Ng's machine learning course on Coursera gives a solution based on SVD for this problem. See the first week's course notes. Ng refers to ...
NNN's user avatar
  • 760
3 votes

Variational loss of hp-Variational Physics Informed Neural Networks for 2D-Poisson Equation in Tensorflow

Here are the few observations based on your code. 1, Add a jacobian transformation to the 2D integral as you are doing it for the forcing term as shown in below example ( taken from the 2D hp-PINNS ...
ThivinAnandh's user avatar
3 votes
Accepted

Does the loss function in a deep neural network act as a norm?

Connection between Mean squared error and $L_2$ norm: First the $L_2$ norm of a vector $\boldsymbol{x}\in\mathbb{R}^n$ is defined as \begin{align*} \lVert \boldsymbol{x}\rVert_2=\left(\sum\limits_{i=1}...
Pepe's user avatar
  • 459
3 votes
Accepted

What problems does softmax() solve and when should I think of using it - in simple terms

Suppose you are training a neural network to predict the probability that a given picture is a picture of a cat, dog or tiger. This is an example of a problem called multi-class classification. The ...
NNN's user avatar
  • 760
2 votes

Are there tasks in machine learning which require double precision floating points?

Perhaps it's better to say that it depends on the nature (details) of the algorithm and linear algebra implementation. More critically, while deep neural networks (DNN) have become widely popular, ...
rwong's user avatar
  • 121
2 votes

Feedforward net lagged prediction

The reason it looks like it's just shifted is an illusion. One step prediction plots can be very deceiving for the unseasoned. Basically what's happening is that your time series is not very ...
Memming's user avatar
  • 870
2 votes

Algorithms for searching in high-dimensional binary data spaces

You may be able to formulate your problem as a binary integer program, which even though in general they are NP hard, some instances can be solved very quickly through branch and cut algorithms (if ...
Septimus G's user avatar
2 votes

How are the outcomes that generated from different predictive models combined to get more accurate predictions?

Bagging, Boosting, and Bayesian Model Averaging/Combination are all widely used techniques for doing this. These are discussed in many textbooks on machine learning.
Brian Borchers's user avatar
2 votes
Accepted

Why can't we just use machine learning to select which model to use for a given model?

You can. This is called an ensemble model. For example, a linear regression between the solutions of different predictive models is a way to take a weighted average of different models. Normally, the ...
Chris Rackauckas's user avatar
2 votes

Using low rank property for maximal/minimal value search (or sorting)

A couple of possible approaches: N. J. Higham and S. D. Relton, "Estimating the largest elements of a matrix", SIAM J. Sci. Comput., vol. 38, no. 5, pp. C584–C601. Alternative link for a preprint. S. ...
Federico Poloni's user avatar
2 votes
Accepted

Using low rank property for maximal/minimal value search (or sorting)

This problem is rather non-trivial. The low-rank representation of $X_{n\times n}=A_{n\times r}B^T_{r\times n}$ with the rank $r$ does not lead to an easy way of finding the minimum and maximum ...
Anton Menshov's user avatar
  • 8,692

Only top scored, non community-wiki answers of a minimum length are eligible