Kate Lin
Research Areas
Authored Publications
Sort By
RADAR: Benchmarking Language Models on Imperfect Tabular Data
Ken Gu
Kumar Ayush
Hong Yu
Zhihan Zhang
Yuzhe Yang
Shwetak Patel
Max Xu
Mark Malhotra
Orson Xu
Evelyn Zhang
Tim Althoff
2025
Preview abstract
Language models (LMs) are increasingly being deployed to perform autonomous data analyses, yet their~\textit{\robustnessTerm}-- the ability to recognize, reason over, and appropriately handle data artifacts such as missing values, outliers, and logical inconsistencies—remains under-explored. These artifacts are common in real-world tabular data and, if mishandled, can significantly compromise the validity of analytical conclusions. To address this gap, we present RADAR, a benchmark for systematically evaluating data awareness on tabular data. RADAR introduces programmatic perturbations for each unique query table pair, enabling targeted evaluation of model behavior. RADAR~ comprises 2500 queries for data analysis across 55 datasets spanning 20 domains and 5 data awareness dimensions. In addition to evaluating artifact handling, RADAR systematically varies table size to study how reasoning performance scales with input length. In our evaluation, we identify fundamental gaps in their ability to perform reliable, data-aware analyses. Designed to be flexible and extensible, RADAR supports diverse perturbation types and controllable table sizes, offering a valuable resource for advancing tabular reasoning.
View details
Preview abstract
The Web today has millions of datasets, and the number of datasets continues to grow at a rapid pace. These datasets are not standalone entities; rather, they are intricately connected through complex relationships. Semantic relationships between datasets provide critical insights for research and decision-making processes. In this paper, we study dataset relationships from the perspective of users who discover, use, and share datasets on the Web: what relationships are important for different tasks? What contextual information might users want to know? We first present a comprehensive taxonomy of relationships between datasets on the Web and map these relationships to user tasks performed during dataset discovery. We develop a series of methods to identify these relationships and compare their performance on a large corpus of datasets generated from Web pages with schema.org markup. We demonstrate that machine-learning based methods that use dataset metadata achieve multi-class classification accuracy of 90%. Finally, we highlight gaps in available semantic markup for datasets and discuss how incorporating comprehensive semantics can facilitate the identification of dataset relationships. By providing a comprehensive overview of dataset relationships at scale, this paper sets a benchmark for future research.
View details
Automatic Structured Variational Inference
Luca Ambrogioni
Max Hinne
Dave Moore
Marcel van Gerven
AISTATS (2021)
Preview abstract
Probabilistic programming is concerned with the symbolic specification of probabilistic models for which inference can be performed automatically. Gradient-based automatic differentiation stochastic variational inference offers an attractive option as the default method for (differentiable) probabilistic programming. However, the performance of any (parametric) variational approach depends on the choice of an appropriate variational family. Here, we introduce automated structured variational inference (ASVI), a fully automated method for constructing structured variational families, inspired by the closed-form update in conjugate Bayesian models. These pseudo-conjugate families incorporate the forward pass of the input probabilistic program and can therefore capture complex statistical dependencies. Pseudo-conjugate families have the same space and time complexity of the input probabilistic program and are therefore tractable for a very large family of models including both continuous and discrete variables. We provide a fully automatic implementation in TensorFlow Probability. We validate our automatic variational method on a wide range of both low- and high-dimensional inference problems including deep learning components.
View details
