Date/Time : 10 Jan 2024 15:00
Venue : S17 #04-05 (SR2)
Speaker : Yu-Xiang Wang, University of California, Santa Barbara

Why do deep neural networks work better in practice than classical methods, e.g., kernels / splines? The talk covers a recent line of research that inspects DNNs from the classical non-parametric regression (or “curve fitting”) which reveals that the reason why DNNs work better might be due to its adaptivity when we tune its standard hyperparameters, which implicitly discovers hidden sparsity and low-dimensional structures. I will go over theory and examples to illustrate this point. For example, the effect of “Weight Decay” is in fact to select a small number of learned basis functions and the deeper the network is, the sparser it gets no matter how overparameterized the network is. Also, weight-decay helps a ConvResNeXT architecture to discover avoid the curse-of-dimensionality when the high-dimensional data is embedded in an unknown low-dimensional manifold. Last but not least, I will explain how the choice of “learning rate” determines the smoothness of functions that SGD can stably converge to, thus informing us on generalization.

References:
[1] Deep Learning meets Nonparametric Regression: Are Weight-Decayed DNNs Locally Adaptive? https://arxiv.org/abs/2204.09664

[2] Nonparametric Classification on Low Dimensional Manifolds using Overparameterized Convolutional Residual Networks. https://arxiv.org/abs/2307.01649

Add to calendar :

Date/Time : 04 Jan 2024 15:00
Venue : S17 #04-06 (SR1)
Speaker : Gal Chechik, Bar-Ilan University

Machine learning now heavily relies on “foundation models”, representations trained with massive data in a task-agnostic way, and then used for downstream tasks. But how can we make them most useful for our own tasks? I will discuss a series of studies addressing this problem in the context of vision-and-language models. These include textual-inversion techniques – where we teach new personalized visual concepts to the model that can be combined with known concepts, and ways to improve their editability and speed. I will also discuss generation of rare classes, personalized approaches to discriminative problems, and future directions in model personalization.

Add to calendar :

Date/Time : 11 Dec 2023 15:00
Venue : S17 #04-06 (SR1)
Speaker : Heng Yang, Harvard University

Many fundamental estimation and decision-making problems in robotics are formulated as non-convex mathematical optimization problems. Decades of research have produced numerous algorithms to efficiently solve these problems, yet offering few performance guarantees. In this talk, I aim to advocate using convex semidefinite relaxations to design a new paradigm of algorithms that work out of the box and offer strong performance certificates.

I first briefly introduce the celebrated Lasserre’s hierarchy of moment and sums-of-squares (SOS) relaxations, which relaxes polynomial optimization and decision problems as a hierarchy of convex semidefinite programs (SDP). I then present three applications of this hierarchy in robotics. In the first application, we apply the hierarchy to solve outlier-robust estimation problems in robot perception and show that SDP relaxations provide certificates of optimality. In the second application, we combine this hierarchy with conformal prediction from statistics and set-membership estimation from control theory and demonstrate such relaxations provide certificates of uncertainty. In the third application, we revisit trajectory optimization problems arising in nonlinear optimal control and show that, by exploiting correlative sparsity, SDP relaxations produce near-optimal control trajectories. I end the talk by discussing computational challenges of this framework in our applications and opportunities to tackle them.

Add to calendar :

Date/Time : 12 Oct 2023 14:00
Venue : S16 #03-04
Speaker : Wang Peng, University of Michigan

Over the past decade, deep learning has proven to be a highly effective method for extracting meaningful features from high-dimensional data. This work attempts to unveil the mystery of feature learning in deep networks. Specifically, for a multi-class classification problem, we explore how the features of training data evolve across the intermediate layers of a trained neural network. We investigate this problem based on simple deep linear networks trained on linearly separable data, and we analyze how the output features in each layer concentrate around the means of their respective classes. Remarkably, when the deep linear network is trained using gradient descent from a small orthogonal initialization, we theoretically prove that the features exhibit a linear decay in the measure of within-class feature variability as we move from shallow to deep layers. Moreover, our extensive experiments not only validate our theoretical findings numerically but also reveal a similar pattern in deep nonlinear networks which well aligns with recent empirical studies.

Add to calendar :

Date/Time : 10 Oct 2023 10:30
Venue : S17 #04-06 (SR1)
Speaker : Charles Cheung, NVIDIA AI Technology Center

In this talk, I will talk about a few directions and use cases of AI for Science. PINNs and neural operator are used for solving many engineering problems that involves differential equations. We will walk through the basic concept of PINNs and neural operator and introduce NVIDIA modulus, an SDK for training PINNs and neural operator. We will also share a recent project about transfer learning.

Add to calendar :

Date/Time : 22 Aug 2023 14:00
Venue : S17 #04-06 (SR1)
Speaker : Shahar Mendelson, Australian National University

Consider an isotropic measure \mu on R^d (i.e., centred and whose covariance is the identity) and let X_1,…,X_m be independent, selected according to \mu. If \Gamma=m^{-1/2} \sum_{i=1}^m <X_i,->e_i is the random operator whose rows are X_i/\sqrt{m}, how does the random set \Gamma S^{d-1} typically look like? For example, if the extremal singular values of \Gamma are close to 1 then \Gamma S^{d-1} is “well approximated” by a d-dimensional section of S^{m-1} and vice-versa. But is it possible to give a more accurate description of that set?

I will show that under minimal assumptions on \mu, with high probability and uniformly in t \in S^{d-1}, each vector \Gamma t inherits the structure of the one-dimensional marginal <X,t> in a strong sense. If time permits I will also present a few generalisations of this fact – for an arbitrary indexing set.

Add to calendar :

Date/Time : 13 Apr 2023 10:00
Venue : via Zoom
Speaker : Daniel Sanz-Alonso, University of Chicago

Join Zoom Meeting
https://nus-sg.zoom.us/j/81261210349?pwd=M2VlVjUvR1o4dGVodDY4MnBtRzMzUT09

Meeting ID: 812 6121 0349
Passcode: 557052

Semi-supervised learning refers to the problem of recovering an input-output map using many unlabeled examples and a few labeled ones. In this talk I will survey several mathematical questions arising from the Bayesian formulation of graph-based semi-supervised learning. These questions include the modeling of prior distributions for functions on graphs, the derivation of continuum limits for the posterior, the design of scalable posterior sampling algorithms, and the contraction of the posterior in the large data limit.

Add to calendar :

Date/Time : 13 Apr 2023 16:00
Venue : S16 #03-06
Speaker : Karthyek Murthy, Singapore University of Technology and Design

Several problems in data-driven decision-making and risk management suffer from data imbalance, a term referring to settings where a small fraction of data has an outsized impact on estimating one or more decision-making criteria. Due to the paucity of relevant samples, such problems are usually approached with the “estimate, then optimize” workflow involving a model estimation from data in the first step before plugging in the trained model to solve various downstream optimization tasks. As biases due to model selection, misspecification, and overfitting to in-sample data are especially difficult to avoid in the first-step estimated model in settings affected by data imbalance, we construct novel locally robust optimization formulations in which the first-step estimation has no effect, locally, on the optimal solutions obtained.

We show that this local insensitivity translates to improved out-of-sample performance freed from the first-order impact of model errors introduced in the first-step estimation. A key ingredient in achieving this local robustness is a novel debiasing procedure that adds a non-parametric bias correction term to the objective. The debiased formulation retains convexity, and the imputation of the correction term relies only on a non-restrictive large deviations behavior conducive for transferring knowledge from representative data-rich regions to the data-scarce tail regions suffering from imbalance. The bias correction gets determined by the extent of model error in the estimation step and the specifics of the stochastic program in the optimization step, thereby serving as a scalable “smart-correction” step bridging the disparate goals in estimation and optimization.

Add to calendar :

Date/Time : 30 Mar 2023 10:00
Venue : via Zoom
Speaker : Xun Huan, University of Michigan

Join Zoom Meeting
https://nus-sg.zoom.us/j/9775606531?pwd=cVZ6eTRYS3ZzVUlFNXRuSlhMVjJSUT09

Meeting ID: 977 560 6531
Passcode: 123321

Experiments are indispensable for learning and developing models in engineering and science. A careful design of these often-expensive data-acquisition opportunities can be immensely beneficial. Simulation-based optimal experimental design, while leveraging a predictive model, offers a framework to systematically quantify and maximize the value of experiments. We focus on designing a finite sequence of experiments, seeking optimal design policies that can (a) adapt to newly collected data during the sequence (i.e. feedback) and (b) anticipate future changes (i.e. lookahead). We cast this sequential decision-making problem in a Bayesian setting with information-based utilities, and solve it numerically via policy gradient methods from reinforcement learning. In particular, we directly parameterize the policies and value functions by neural networks—thus adopting an actor-critic approach—and improve them using gradient estimates produced from simulated design and observation sequences. We demonstrate the overall method on a problem of optimal sensor movement for contaminant source inversion.

Add to calendar :