1
$\begingroup$

Is it possible that an ODE (with an IC) solution by physics informed neural networks (PINNs) turns out to be a mixture of several branch solutions of the same bulk ODE but with different ICs, even when the loss function is very low? E.g. u'(t)=1 is solved by PINNs as the union of u=t for some t interval and u=t+1 for the remaining t interval.

$\endgroup$
12
  • 2
    $\begingroup$ Does anyone have any idea what PINNs actually do? Is what you describe possible? It would say anything is possible with PINNs. $\endgroup$ Apr 16 at 16:30
  • 3
    $\begingroup$ @WolfgangBangerth no one has a serious answer for that, yet. It is sad to see computational math is getting rid of math. $\endgroup$
    – Yimin
    Apr 16 at 17:26
  • 2
    $\begingroup$ @WolfgangBangerth, maybe a handful of people put effort to make it rigorous. The results are lacking. Basically, the neural network is a proposed solution to the ODE/PDE, if the neural network "space" contains the solution then the optimization process will find it. If the "space" is rich enough (basically universal approximation theorem) then the solution can be general enough to depend on the parameters of the problem. I think there are some researchers who derived worst case bounds, and the neural network has to be massive to have any sort of error bound guarantees. $\endgroup$ Apr 17 at 1:09
  • 4
    $\begingroup$ @AbdullahAliSivas I am also an outsider. I agree that the PINN community is application-driven and might be pleased with two-digit accuracy. I am just wondering if someone can provide a deeper understanding, e.g., before solving a problem with PINN, can we at least show a guaranteed error analysis like other traditional methods? $\endgroup$
    – Yimin
    Apr 17 at 2:10
  • 2
    $\begingroup$ @NNN just went through the paper, it made an unreasonable assumption about having the global minimizer found for any sample size. This might be possible if the landscape is trivial, e.g. quadratic-like, under NTK regime, this is true, but that is an uninteresting case. Otherwise, the landscape is guaranteed to be non-convex. I would say the paper made some effort but not what I ask for. $\endgroup$
    – Yimin
    Apr 17 at 3:43

1 Answer 1

-2
$\begingroup$

Yes, I believe it is possible.

For the ODE you showed, one would minimize a loss of the following form

$$ \mathcal{L} = w_{init}\mathcal{L}_{init} + w_{pde}\mathcal{L}_{pde} $$

where $\mathcal{L}_{init}$ is the initial condition loss, and $\mathcal{L}_{pde}$ is the pde loss. As long as these losses are balanced, one would expect the solution obtained from the PINN to be close to the true solution of the ODE.

However, if $w_{pde}\mathcal{L}_{pde} \gg w_{init}\mathcal{L}_{init}$ because the weights are imbalanced, or if the pde loss is very high, then the optimizer may effectively ignore the initial condition and preferentially minimize $\mathcal{L}_{pde}$ and you might get the non-uniqueness you are asking about.

$\endgroup$
6
  • $\begingroup$ Thanks for your answer. But what my question assumes is that the loss function is sufficiently low so that both in your terminology Ls are small, then ICs are assumed satisfied. $\endgroup$
    – feynman
    Apr 19 at 7:45
  • 1
    $\begingroup$ Sorry I misunderstood. In "Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations" by Raissi et. al. JCP, the authors present several examples in which the PINN solution is effectively discontinuous in space Fig 2, Fig A.6. Perhaps due to some pathological reason, your PINN predicts solutions which are discontinuous in time. For t < T_crit it's solutions satisfy the IC and for t > T_crit it's solutions do not satisfy the IC. The loss will be low, but IC information does not get to T > T_crit. $\endgroup$
    – NNN
    Apr 19 at 10:01
  • $\begingroup$ Thanks for pointing that JCP paper out. That the loss will be low but IC information does not get to T > T_crit, I wonder if this is your viewpoint or is that from the JCP paper? $\endgroup$
    – feynman
    Apr 19 at 10:33
  • 1
    $\begingroup$ This is my viewpoint, not the JCP paper's viewpoint. From my experience with physics based losses, they seem to be highly finicky and sensitive to parameters in the pde and possibly neural network and training hyperparameters, and getting stuck in a local minimum might be one of the symptoms of bad pde parameters/hyperparameters. Data based networks seem easier to train. See also section 5.1 in "An Expert’s Guide To Training Physics-Informed Neural Networks" by Wang, Perdikaris et. al. Also see "Characterizing possible failure modes in physics-informed neural networks" by Krishnapriyan et.al. $\endgroup$
    – NNN
    Apr 19 at 10:49
  • 1
    $\begingroup$ I'm not aware existing work in support of my comment. But the section 5.1 I referred to in my previous comment is titled "Respecting Temporal Causality" and it may be relevant (I have't read it, it's on my reading list) $\endgroup$
    – NNN
    Apr 21 at 3:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.