Seminar Mathematical Optimization (4 ECTS)

Neural Network based One-Shot Inversion and Optimization

We study the use of novel techniques arising in machine learning for inverse problems. Our approach replaces the complex forward model by a neural network, which is trained simultaneously in a one-shot sense when estimating the unknown parameters from data, i.e. the neural network is trained only for the unknown parameter. By establishing a link to the Bayesian approach to inverse problems, an algorithmic framework is developed which ensures the feasibility of the parameter estimate w.r.\ to the forward model. We propose an efficient, derivative-free optimization method based on variants of the ensemble Kalman inversion. Numerical experiments show that the ensemble Kalman filter for neural network based one-shot inversion is a promising direction combining optimization and machine learning techniques for inverse problems.

Stochastic gradient descent in continuous time: discrete and continuous data Abstract: Optimisation problems with discrete and continuous data appear in statistical estimation, machine learning, functional data science, robust optimal control, and variational inference. The `full' target function in such an optimisation problems is given by the integral over a family of parameterised target functions with respect to a discrete or continuous probability measure. Such problems can often be solved by stochastic optimisation methods: performing optimisation steps with respect to the parameterised target function with randomly switched parameter values. In this talk, we discuss a continuous-time variant of the stochastic gradient descent algorithm. This so-called stochastic gradient process couples a gradient flow minimising a parameterised target function and a continuous-time `index' process which determines the parameter. We first briefly introduce the stochastic gradient processes for finite, discrete data which uses pure jump index processes. Then, we move on to continuous data. Here, we allow for very general index processes: reflected diffusions, pure jump processes, as well as other Lévy processes on compact spaces. Thus, we study multiple sampling patterns for the continuous data space. We show that the stochastic gradient process can approximate the gradient flow minimising the full target function at any accuracy. Moreover, we give convexity assumptions under which the stochastic gradient process with constant learning rate is geometrically ergodic. In the same setting, we also obtain ergodicity and convergence to the minimiser of the full target function when the learning rate decreases over time sufficiently slowly. We illustrate the applicability of the stochastic gradient process in a simple polynomial regression problem with noisy functional data, as well as in physics-informed neural networks approximating the solution to certain partial differential equations.

Vesa Kaarnioja (LUT University)

Modeling random domains using periodic random variables

Abstract: Computational measurement models may involve several uncertain simulation parameters: not only can the material properties of a heterogeneous medium be unknown, but the shape of the structure itself can be uncertain as well. In this talk, we discuss a parameterization for an uncertain domain using a random perturbation field in which a countable number of independent random variables enter the random field as periodic functions. The random field can be constructed to have a prescribed mean and covariance function. As an application, we design simple quasi-Monte Carlo cubature rules in order to study how uncertainty in the domain shape impacts the stochastic response of an elliptic PDE.

This is joint work with Harri Hakula (Aalto University), Helmut Harbrecht (University of Basel), Frances Y. Kuo (UNSW Sydney), and Ian H. Sloan (UNSW Sydney).