Benchmark Neural Networks on High-Dimensional Functions

Question

For a personal project, I am interested in benchmarking certain neural network architectures in the context of high-dimensional function approximation. Specifically, I am interested in continuous, smooth, Hölder, and Sobolev functions defined on $[0,1]^d$ in $\mathbb{R}^d$.

• Does anyone know if a standard list of high-dimensional functions is commonly used in the literature to benchmark such models? For example, in the optimization literature, there is a standard list of functions, such as the ones found here, to benchmark various optimization algorithms.

• If no such list is available, how should one construct a representative list of functions? This choice will introduce an inductive bias in the problem, so I'd like to ensure the list is as representative as possible.

Answer 1

A quick answer. In 2013, Derek Bingham and his student Sonja Surjanovic created a list of multi-d test functions for uncertainty quantification methods by reviewing existing literature. https://www.sfu.ca/~ssurjano/uq.html

A longer answer from my experience working with methods for high-dimensional approximation. There's a large conceptual gap between mathematical function classes ($C^k$, Holder, Sobolev, etc.) and challenges in numerical approximation. The mathematical categories don't map well to degrees of numerical difficulty. For example, a "Gaussian" bump like $f(x) = \exp(-(x-c)^T(x-c)/\sigma^2)$ with $x,c\in\mathbb{R}^n$ is as nice as it gets in terms of mathematical continuity. But I can always choose $\sigma$ small enough to "hide" the bump centered at $c$ from any finite-data approximation scheme---especially in finite precision arithmetic. Like: (1) choose $\sigma$ to be smaller than machine epsilon and (2) don't let your approximation scheme know $c$. You can play the same conceptual game with highly oscillatory functions. Sure, there are plenty of published papers for approximating "spikey" or "highly oscillatory" functions, but the methods require some knowledge of the spikes or oscillations. In asymptotic analysis, you might make quantitative approximation statements in terms of $\sigma$ or oscillation frequency. But for testing neural net approximations with finite data, I think constructing a list of test functions (like the optimizers have) is a fool's errand. I say that having sat through 10 years of uncertainty quantification conferences where the biggest names tried to construct universal test cases for approximation methods. Nothing's ever stuck---and not for lack of either smarts or effort.

A better idea, in my opinion, is to find a specific type of function from an interesting application and parameterize the function in such a way that certain parameter regimes are numerically easy (fast convergence with increasing data) and others are numerically difficult (slow-or-no convergence). For example, high-d integration methods often use examples from quantitative finance. Then at least you're comparing apples to apples. The idea that one approximation method works better than another given only the continuity class of the functions is overly simplistic because it ignores the wide conceptual gap between asymptotic mathematical approximation and practical numerical approximation.

Answer 2

(My answer didn't fit in the little box, so here's another big box answer.)

I think your notion of "application" is too abstract. An application is a science or engineering field where a model with your desired characteristics might arise. The idea is to find a science/engineering community with a paradigm that values mathematical modeling. Applications include: environmental resource management, airfoil design, drug design, solar cell modeling, etc. A PDE is a type of model. What makes a PDE an application includes choices for model terms, boundary/initial data, variables and scalings, etc. A specific application will determine the available choices. If you try to make "PDEs" an application, then you still need to make all those choices to pose a particular computational model, except you must do so without the aid of science/engineering. Invariably, you end up working with simplistic test problems that the scientists and engineers will call "toys" or (oddly) "Mickey Mouse." This oversimplification-for-the-sake-of-a-demo routinely happens in computational science literature (and it's so annoying). Many PDEs have faux-application names (heat equation, shallow water equations, advection-diffusion equation), which sound like applications, but they're really just mnemonic labels. To understand how $u_t=-ku_{xx}$ might arise in an application, you should find an engineering text on (or colleague interested in) bulk heat transfer. Such a resource should have example problems that set all the terms in the model based on scientific reasoning---instead of being motivated as a benchmark for a different computational modeling tool.

If you're trying to find a range of applications as benchmarks, then you're back to the problem of finding test functions. The problem with a collection of benchmarks is: how does a potential user of your tool know that their problem is like any of your benchmarks? (The optimization test suites have the same problem.) If someone can't see their workbench in your benchmark, then they're unlikely to find value in your tool (e.g. neural net) comparisons. The answer (IMO) is to go look at their workbench.

Re: Question 2. Your categories for application difficulty (continuity, smoothness, integrability) are usually not things that applications engineers think about. Those are things that mathematicians think about. A prominent example: mathematicians have not concluded that the three-dimensional Navier-Stokes equations are well-posed. And yet, engineers routinely approximate solutions to 3D-NS in particular applications. The challenges for a computational fluids engineer usually involve balancing grid resolution, numerical stability, and time-to-solution. Again, there's a large (arguably infinite) gap between mathematical categories and practical computations.

Switching gears entirely. The CFD example suggests a range of abstract benchmark problems that are distinct from mathematical categories. How well does your neural net model balance data resolution, numerical stability, and time-to-solution? There are many functions that could challenge this balance:

• spikey functions
• highly oscillatory functions
• corner singularities
• discontinuous functions
• discontinuous derivative functions

And surely others; just imagine functions where limited resolution will harm your accuracy. Draw pictures. But I stress: if you go that route of trying to list numerically challenging functions, then you're back to the problem of who cares? Specifically, can someone see their own problem within your benchmarks?

Answer 3

The medicine for naivete is curiosite. Let's see if I can spark your curiosity. Note: I'm more concerned with 'context' and 'value' and less concerned with 'approximation' and its mathematics.

Let's critique the article you referenced: Deep Network Approximation for Smooth Functions, which was published in SIAM's Journal on Mathematical Analysis in 2021 (doi.org/10.1137/20M134695X). By publishing in a journal devoted to mathematical analysis, the authors have avoided any serious critique of their discussion about applications. Nevertheless, you are considering giving this article to undergraduates to ask them to test the article's claims with numerics. I suggest you not do this. The article's claims are analytical---built on models of mathematical analysis that are detached from applications. I do not know how anyone---much less an undergrad---could test these claims empirically. Despite certain characterizations by the authors, the claims were not meant to be tested empirically, e.g. with test functions, finite data, and specific implementations. The article's results were constructed to compare with other articles with similarly constructed results---e.g. comparing rates of convergence---constrained by the models, tools, and paradigms for analysis. The content of the paper is the proofs. There are no serious numerical demonstrations. The main results are bounds based on "there exists a $\phi$". They do not offer constructions of $\phi$ that someone could use to build a numerical approximation. The authors admit this in a section called "Application scope of our theory in machine learning", where they try to sell the value of their analytical results for numerical approximation. I do not find their pitch compelling; evidently, it was sufficiently compelling for reviewers and editors of an analysis journal.

I think their results have no value for practical approximation. To support my claim, consider their discussion of applications in the first paragraph. The authors make strong claims about the "impact" of neural nets, but they have no citations. Although they describe the impact on "many fields," they cannot cite one specific technical article. Although they claim that modern advances have made neural nets "very successful in important real applications," they cannot cite one specific technical article detailing the successes. Although they say "great advantages ... have been demonstrated," they cannot cite one specific demonstration. Although they claim, "A large number of experiments have shown ...," they cannot cite one experiment. They use these no-citation claims to argue for the value of studying approximation error in neural nets. At the end of section 1, there is one citation to Hinton's ImageNet paper, which has about 130,000 citations. They do not discuss how their results relate to any specific image classification applications. They simply say that the ImageNet paper motivates their focus on ReLU activation functions.

This exercise is called narrative construction. It is not science. For the journal audience---who share a paradigm for the value of the mathematical analysis---this narrative is (evidently) sufficient (for publication). For real applications, this narrative is sadly insufficient. No one who is curious about neural nets for a specific approximation application will find this paper useful. If your goal is to demonstrate the value of neural nets for practical approximation (e.g. by choosing benchmark problems), then this paper---and likely every paper it cites---will not help you.

If you try to poach a benchmark function from an application, then you are removing the function from its context. Outside proper context, value changes. The value of a test problem is directly related to the value of the application. That value is best assessed from within the application paradigm---i.e. engaging with application scientists who understand how such a function was derived and why that function is important inside the proper application context. "High-dimensional PDE in optimal control" is not an application; it's a type of model that might arise in applications where "control" is defined and valued---like aerospace design. You could collect toy problems from various mathematical subfields. Then the value of your benchmarks rests on the value of the toy problems. Surely, there's enough toy-problem value to publish in a journal or venue that values toy problems. But, in my opinion, this does not imply value for the real applications where the toy problems might have originated. To answer your question of how to make a benchmarking idea practical and relevant, you ought to first answer: practical and relevant to whom?