Date/Time : 01 Nov 2024 15:00
Venue : S17 #04-06 (SR1)
Speaker : Nathan Ross, University of Melbourne

Neural networks are random parametric functions, where the parameters are chosen according to stochastic gradient descent, based on the sample of training data. It has long been known that at initialisation, a neural network with “large” width and “fixed” depth can be well-approximated by a limiting Gaussian random field. But what quantities are well-approximated, and how wide does a neural network need to be for the approximation to be good? There has been a flurry of recent work trying to answer these and related questions, by deriving rates of convergence between the distribution of the neural network and the limiting Gaussian random field in various probability metrics, including some at the process level.

The aim of these lectures is to explain a general approach that can be applied to derive such rates at the process level, which uses Stein’s method, functional smoothing inequalities, and classical Gaussian process concepts such as Karhunen-Loeve expansion and Cameron-Martin/Reproducing Kernel Hilbert Space. The target audience is PhD students planning to do research in probability or statistics. In particular, the definitions and relevant properties of the concepts above will be covered in detail. Open questions and future directions will also be discussed.

The approach is based on joint work with (A.D. Barbour, K. Balasubramanian, L. Goldstein, A. Salim, and G. Zheng).

Add to calendar :

Date/Time : 30 Oct 2024 15:00
Venue : S17 #04-04 (SR3)
Speaker : Nathan Ross, University of Melbourne

Neural networks are random parametric functions, where the parameters are chosen according to stochastic gradient descent, based on the sample of training data. It has long been known that at initialisation, a neural network with “large” width and “fixed” depth can be well-approximated by a limiting Gaussian random field. But what quantities are well-approximated, and how wide does a neural network need to be for the approximation to be good? There has been a flurry of recent work trying to answer these and related questions, by deriving rates of convergence between the distribution of the neural network and the limiting Gaussian random field in various probability metrics, including some at the process level.

The aim of these lectures is to explain a general approach that can be applied to derive such rates at the process level, which uses Stein’s method, functional smoothing inequalities, and classical Gaussian process concepts such as Karhunen-Loeve expansion and Cameron-Martin/Reproducing Kernel Hilbert Space. The target audience is PhD students planning to do research in probability or statistics. In particular, the definitions and relevant properties of the concepts above will be covered in detail. Open questions and future directions will also be discussed.

The approach is based on joint work with (A.D. Barbour, K. Balasubramanian, L. Goldstein, A. Salim, and G. Zheng).

Add to calendar :

Date/Time : 29 Oct 2024 14:00
Venue : S17 #04-06 (SR1)
Speaker : Nathan Ross, University of Melbourne

Neural networks are random parametric functions, where the parameters are chosen according to stochastic gradient descent, based on the sample of training data. It has long been known that at initialisation, a neural network with “large” width and “fixed” depth can be well-approximated by a limiting Gaussian random field. But what quantities are well-approximated, and how wide does a neural network need to be for the approximation to be good? There has been a flurry of recent work trying to answer these and related questions, by deriving rates of convergence between the distribution of the neural network and the limiting Gaussian random field in various probability metrics, including some at the process level.

The aim of these lectures is to explain a general approach that can be applied to derive such rates at the process level, which uses Stein’s method, functional smoothing inequalities, and classical Gaussian process concepts such as Karhunen-Loeve expansion and Cameron-Martin/Reproducing Kernel Hilbert Space. The target audience is PhD students planning to do research in probability or statistics. In particular, the definitions and relevant properties of the concepts above will be covered in detail. Open questions and future directions will also be discussed.

The approach is based on joint work with (A.D. Barbour, K. Balasubramanian, L. Goldstein, A. Salim, and G. Zheng).

Add to calendar :

Date/Time : 28 Oct 2024 16:00
Venue : S17 #04-06 (SR1)
Speaker : Nathan Ross, University of Melbourne

Neural networks are random parametric functions, where the parameters are chosen according to stochastic gradient descent, based on the sample of training data. It has long been known that at initialisation, a neural network with “large” width and “fixed” depth can be well-approximated by a limiting Gaussian random field. But what quantities are well-approximated, and how wide does a neural network need to be for the approximation to be good? There has been a flurry of recent work trying to answer these and related questions, by deriving rates of convergence between the distribution of the neural network and the limiting Gaussian random field in various probability metrics, including some at the process level.

The aim of these lectures is to explain a general approach that can be applied to derive such rates at the process level, which uses Stein’s method, functional smoothing inequalities, and classical Gaussian process concepts such as Karhunen-Loeve expansion and Cameron-Martin/Reproducing Kernel Hilbert Space. The target audience is PhD students planning to do research in probability or statistics. In particular, the definitions and relevant properties of the concepts above will be covered in detail. Open questions and future directions will also be discussed.

The approach is based on joint work with (A.D. Barbour, K. Balasubramanian, L. Goldstein, A. Salim, and G. Zheng).

Add to calendar :

Date/Time : 09 Oct 2024 15:00
Venue : S17 #04-05 (SR2)
Speaker : Pierre Alquier, ESSEC

The PAC-Bayesian theory provides tools to understand the accuracy of Bayes-inspired algorithms that learn probability distributions on parameters. This theory was initially developed by McAllester about 20 years ago, and applied successfully to various machine learning algorithms in various problems. Recently, it led to tight generalization bounds for deep neural networks, a task that could not be achieved by standard “worst-case” generalization bounds such as Vapnik-Chervonenkis bounds. In this talk, I will provide a brief introduction to PAC-Bayes bounds, and explain the core ideas of the theory. I will also provide an overview of the recent research directions. In particular, I will highlight the application of PAC-Bayes bounds to derive minimax-optimal rates of convergence in classification and in regression, and the connection to mutual-information bounds.

Add to calendar :