publications

Calibration for causal inference

2025

1. Stabilized Inverse Probability Weighting via Isotonic Calibration

   Lars van der Laan, Ziming Lin, Marco Carone, and 1 more author

   In Proceedings of the 3rd Conference on Causal Learning and Reasoning (CLeaR), 2025

   To appear

   Abs PDF Code

   Inverse weighting with an estimated propensity score is widely used by estimation methods in causal inference to adjust for confounding bias. However, directly inverting propensity score estimates can lead to instability, bias, and excessive variability due to large inverse weights, especially when treatment overlap is limited. In this work, we propose a post-hoc calibration algorithm for inverse propensity weights that generates well-calibrated, stabilized weights from user-supplied, cross-fitted propensity score estimates. Our approach employs a variant of isotonic regression with a loss function specifically tailored to the inverse propensity weights. Through theoretical analysis and empirical studies, we demonstrate that isotonic calibration improves the performance of doubly robust estimators of the average treatment effect.

2024

1. Automatic doubly robust inference for linear functionals via calibrated debiased machine learning

   Lars van der Laan, Alex Luedtke, and Marco Carone

   arXiv preprint arXiv:2411.02771, 2024

   Abs PDF Code Poster

   In causal inference, many estimands of interest can be expressed as a linear functional of the outcome regression function; this includes, for example, average causal effects of static, dynamic and stochastic interventions. For learning such estimands, in this work, we propose novel debiased machine learning estimators that are doubly robust asymptotically linear, thus providing not only doubly robust consistency but also facilitating doubly robust inference (e.g., confidence intervals and hypothesis tests). To do so, we first establish a key link between calibration, a machine learning technique typically used in prediction and classification tasks, and the conditions needed to achieve doubly robust asymptotic linearity. We then introduce calibrated debiased machine learning (C-DML), a unified framework for doubly robust inference, and propose a specific C-DML estimator that integrates cross-fitting, isotonic calibration, and debiased machine learning estimation. A C-DML estimator maintains asymptotic linearity when either the outcome regression or the Riesz representer of the linear functional is estimated sufficiently well, allowing the other to be estimated at arbitrarily slow rates or even inconsistently. We propose a simple bootstrap-assisted approach for constructing doubly robust confidence intervals. Our theoretical and empirical results support the use of C-DML to mitigate bias arising from the inconsistent or slow estimation of nuisance functions.

2023

1. Causal isotonic calibration for heterogeneous treatment effects

   Lars van der Laan, Ernesto Ulloa-Pérez, Marco Carone, and 1 more author

   In Proceedings of the 40th International Conference on Machine Learning (ICML), 2023

   Abs Blog PDF Twitter Code Poster

   We propose causal isotonic calibration, a novel nonparametric method for calibrating predictors of heterogeneous treatment effects. Furthermore, we introduce cross-calibration, a data-efficient variant of calibration that eliminates the need for hold-out calibration sets. Cross-calibration leverages cross-fitted predictors and generates a single calibrated predictor using all available data. Under weak conditions that do not assume monotonicity, we establish that both causal isotonic calibration and cross-calibration achieve fast doubly-robust calibration rates, as long as either the propensity score or outcome regression is estimated accurately in a suitable sense. The proposed causal isotonic calibrator can be wrapped around any black-box learning algorithm, providing robust and distribution-free calibration guarantees while preserving predictive performance.

Debiased and efficient estimation

2025

1. Nonparametric Instrumental Variable Inference with Many Weak Instruments

   Lars van der Laan, Nathan Kallus, and Aurélien Bibaut

   2025

   Abs PDF

   We study inference on linear functionals in the nonparametric instrumental variable (NPIV) problem with a discretely-valued instrument under a many-weak-instruments asymptotic regime, where the number of instrument values grows with the sample size. A key motivating example is estimating long-term causal effects in a new experiment with only short-term outcomes, using past experiments to instrument for the effect of short- on long-term outcomes. Here, the assignment to a past experiment serves as the instrument: we have many past experiments but only a limited number of units in each. Since the structural function is nonparametric but constrained by only finitely many moment restrictions, point identification typically fails. To address this, we consider linear functionals of the minimum-norm solution to the moment restrictions, which is always well-defined. As the number of instrument levels grows, these functionals define an approximating sequence to a target functional, replacing point identification with a weaker asymptotic notion suited to discrete instruments. Extending the Jackknife Instrumental Variable Estimator (JIVE) beyond the classical parametric setting, we propose npJIVE, a nonparametric estimator for solutions to linear inverse problems with many weak instruments. We construct automatic debiased machine learning estimators for linear functionals of both the structural function and its minimum-norm projection, and establish their efficiency in the many-weak-instruments regime.

2. Automatic Debiased Machine Learning for Smooth Functionals of Nonparametric M-Estimands

   Lars van der Laan, Aurelien Bibaut, Nathan Kallus, and 1 more author

   2025

   Abs PDF

   We propose a unified framework for automatic debiased machine learning (autoDML) to perform inference on smooth functionals of infinite-dimensional M-estimands, defined as population risk minimizers over Hilbert spaces. By automating debiased estimation and inference procedures in causal inference and semiparametric statistics, our framework enables practitioners to construct valid estimators for complex parameters without requiring specialized expertise. The framework supports Neyman-orthogonal loss functions with unknown nuisance parameters requiring data-driven estimation, as well as vector-valued M-estimands involving simultaneous loss minimization across multiple Hilbert space models. We formalize the class of parameters efficiently estimable by autoDML as a novel class of nonparametric projection parameters, defined via orthogonal minimum loss objectives. We introduce three autoDML estimators based on one-step estimation, targeted minimum loss-based estimation, and the method of sieves. For data-driven model selection, we derive a novel decomposition of model approximation error for smooth functionals of M-estimands and propose adaptive debiased machine learning estimators that are superefficient and adaptive to the functional form of the M-estimand. Finally, we illustrate the flexibility of our framework by constructing autoDML estimators for the long-term survival under a beta-geometric model.

3. Automatic Double Reinforcement Learning in Semiparametric Markov Decision Processes with Applications to Long-Term Causal Inference

   Lars van der Laan, David Hubbard, Allen Tran, and 2 more authors

   2025

   Abs PDF Poster

   Double reinforcement learning (DRL) enables statistically efficient inference on the value of a policy in a nonparametric Markov Decision Process (MDP) given trajectories generated by another policy. However, this approach necessarily requires stringent overlap between the state distributions of the two policies, which is often violated in practice. To relax this requirement and extend DRL, we study efficient inference on linear functionals of the Q-function (of which policy value is a special case) in infinite-horizon, time-invariant MDPs under semiparametric restrictions on the Q-function. These restrictions can alleviate the overlap requirement and lower the efficiency bound, yielding more precise estimates. As an important example, we study the evaluation of long-term value under domain adaptation, given a few short trajectories from the new domain and restrictions on the difference between the domains. This can be applied to long-term causal inference by combining short and scarce experimental data with long and abundant observational data. Our method combines flexible estimates of the Q-function and the Riesz representer of the functional of interest (e.g., the stationary state density ratio for policy value) and is automatic in that it does not require knowledge of the form of the latter - only the functional of interest. To address potential model misspecification bias, we extend the adaptive debiased machine learning (ADML) framework of van der Laan et al. (2023) to construct nonparametrically valid and superefficient estimators that adapt to the functional form of the Q-function. As a special case, we propose a novel adaptive debiased plug-in estimator that uses isotonic-calibrated fitted Q-iteration - a new calibration algorithm for MDPs - to circumvent the computational challenges of estimating debiasing nuisances from min-max objectives.

2024

1. Combining T-learning and DR-learning: a framework for oracle-efficient estimation of causal contrasts

   Lars van der Laan, Marco Carone, and Alex Luedtke

   arXiv preprint arXiv:2402.01972, 2024

   Abs PDF Code Poster

   We introduce efficient plug-in (EP) learning, a novel framework for the estimation of heterogeneous causal contrasts, such as the conditional average treatment effect and conditional relative risk. The EP-learning framework enjoys the same oracle-efficiency as Neyman-orthogonal learning strategies, such as DR-learning and R-learning, while addressing some of their primary drawbacks, including that (i) their practical applicability can be hindered by loss function non-convexity; and (ii) they may suffer from poor performance and instability due to inverse probability weighting and pseudo-outcomes that violate bounds. To avoid these drawbacks, EP-learner constructs an efficient plug-in estimator of the population risk function for the causal contrast, thereby inheriting the stability and robustness properties of plug-in estimation strategies like T-learning. Under reasonable conditions, EP-learners based on empirical risk minimization are oracle-efficient, exhibiting asymptotic equivalence to the minimizer of an oracle-efficient one-step debiased estimator of the population risk function. In simulation experiments, we illustrate that EP-learners of the conditional average treatment effect and conditional relative risk outperform state-of-the-art competitors, including T-learner, R-learner, and DR-learner. Open-source implementations of the proposed methods are available in our R package hte3.

2023

1. Adaptive debiased machine learning using data-driven model selection techniques

   Lars van der Laan, Marco Carone, Alex Luedtke, and 1 more author

   arXiv preprint arXiv:2307.12544, 2023

   Abs PDF Twitter Code

   Debiased machine learning estimators for nonparametric inference of smooth functionals of the data-generating distribution can suffer from excessive variability and instability. For this reason, practitioners may resort to simpler models based on parametric or semiparametric assumptions. However, such simplifying assumptions may fail to hold, and estimates may then be biased due to model misspecification. To address this problem, we propose Adaptive Debiased Machine Learning (ADML), a nonparametric framework that combines data-driven model selection and debiased machine learning techniques to construct asymptotically linear, adaptive, and superefficient estimators for pathwise differentiable functionals. By learning model structure directly from data, ADML avoids the bias introduced by model misspecification and remains free from the restrictions of parametric and semiparametric models. While they may exhibit irregular behavior for the target parameter in a nonparametric statistical model, we demonstrate that ADML estimators provides regular and locally uniformly valid inference for a projection-based oracle parameter. Importantly, this oracle parameter agrees with the original target parameter for distributions within an unknown but correctly specified oracle statistical submodel that is learned from the data. This finding implies that there is no penalty, in a local asymptotic sense, for conducting data-driven model selection compared to having prior knowledge of the oracle submodel and oracle parameter. To demonstrate the practical applicability of our theory, we provide a broad class of ADML estimators for estimating the average treatment effect in adaptive partially linear regression models.

2021

1. Higher order targeted maximum likelihood estimation

   Mark van der Laan, Zeyi Wang, and Lars van der Laan

   arXiv preprint arXiv:2101.06290, 2021

   PDF

Causal inference methodology

2025

1. Targeted maximum likelihood based estimation for longitudinal mediation analysis

   Zeyi Wang, Lars van der Laan, Maya Petersen, and 3 more authors

   Journal of Causal Inference, 2025

   PDF

2024

1. Adaptive-TMLE for the Average Treatment Effect based on Randomized Controlled Trial Augmented with Real-World Data

   Mark van der Laan, Sky Qiu, and Lars van der Laan

   arXiv preprint arXiv:2405.07186, 2024

   Abs PDF

   We consider the problem of estimating the average treatment effect (ATE) when both randomized control trial (RCT) data and real-world data (RWD) are available. We decompose the ATE estimand as the difference between a pooled-ATE estimand that integrates RCT and RWD and a bias estimand that captures the conditional effect of RCT enrollment on the outcome. We introduce an adaptive targeted minimum loss-based estimation (A-TMLE) framework to estimate them. We prove that the A-TMLE estimator is root-n-consistent and asymptotically normal. Moreover, in finite sample, it achieves the super-efficiency one would obtain had one known the oracle model for the conditional effect of the RCT enrollment on the outcome. Consequently, the smaller the working model of the bias induced by the RWD is, the greater our estimator’s efficiency, while our estimator will always be at least as efficient as an efficient estimator that uses the RCT data only. A-TMLE outperforms existing methods in simulations by having smaller mean-squared-error and 95% confidence intervals. A-TMLE could help utilize RWD to improve the efficiency of randomized trial results without biasing the estimates of intervention effects. This approach could allow for smaller, faster trials, decreasing the time until patients can receive effective treatments.

2023

1. Semiparametric logistic regression for inference on relative vaccine efficacy in case-only studies with informative missingness

   Lars van der Laan, and Peter B Gilbert

   arXiv preprint arXiv:2303.11462, 2023

   Abs PDF

   We develop semiparametric methods for estimating subgroup-specific relative vaccine efficacy against multiple viral strains in a partially vaccinated population. Focusing on observational case-only studies, we address informative missingness in strain type due to vaccination status, pre-vaccination character- istics, and post-infection factors such as viral load. We establish general conditions for the nonpara- metric identification of relative conditional vaccine efficacy between strains using covariate-adjusted conditional odds ratio parameters. Assuming a log-linear parametric form for strain-specific conditional vaccine efficacy, we propose targeted maximum likelihood estimators based on partially linear logis- tic regression, leveraging machine learning for flexible confounding adjustment. Finally, we apply our methods to estimate relative strain-specific conditional vaccine efficacy in the ENSEMBLE COVID-19 vaccine trial.

2022

1. Nonparametric estimation of the causal effect of a stochastic threshold-based intervention

   Lars van der Laan, Wenbo Zhang, and Peter B Gilbert

   Biometrics, 2022

   Abs PDF

   Identifying a biomarker or treatment-dose threshold that marks a specified level of risk is an important problem, especially in clinical trials. In view of this goal, we consider a covariate-adjusted threshold-based interventional estimand, which happens to equal the binary treatment–specific mean estimand from the causal inference literature obtained by dichotomizing the continuous biomarker or treatment as above or below a threshold. The unadjusted version of this estimand was considered in Donovan et al.. Expanding upon Stitelman et al., we show that this estimand, under conditions, identifies the expected outcome of a stochastic intervention that sets the treatment dose of all participants above the threshold. We propose a novel nonparametric efficient estimator for the covariate-adjusted threshold-response function for the case of informative outcome missingness, which utilizes machine learning and targeted minimum-loss estimation (TMLE). We prove the estimator is efficient and characterize its asymptotic distribution and robustness properties. Construction of simultaneous 95% confidence bands for the threshold-specific estimand across a set of thresholds is discussed. In the Supporting Information, we discuss how to adjust our estimator when the biomarker is missing at random, as occurs in clinical trials with biased sampling designs, using inverse probability weighting. Efficiency and bias reduction of the proposed estimator are assessed in simulations. The methods are employed to estimate neutralizing antibody thresholds for virologically confirmed dengue risk in the CYD14 and CYD15 dengue vaccine trials.

Conformal prediction and predictive uncertainty

2025

1. Generalized Venn and Venn-Abers Calibration with Applications in Conformal Prediction

   Lars van der Laan, and Ahmed Alaa

   In Proceedings of the 42nd International Conference on Machine Learning (ICML), 2025

   To appear

   Abs PDF

   Ensuring model calibration is critical for reliable predictions, yet popular distribution-free methods, such as histogram binning and isotonic regression, provide only asymptotic guarantees. We introduce a unified framework for Venn and Venn-Abers calibration, generalizing Vovk’s binary classification approach to arbitrary prediction tasks and loss functions. Venn calibration leverages binning calibrators to construct prediction sets that contain at least one marginally perfectly calibrated point prediction in finite samples, capturing epistemic uncertainty in the calibration process. The width of these sets shrinks asymptotically to zero, converging to a conditionally calibrated point prediction. Furthermore, we propose Venn multicalibration, a novel methodology for finite-sample calibration across subpopulations. For quantile loss, group-conditional and multicalibrated conformal prediction arise as special cases of Venn multicalibration, and Venn calibration produces novel conformal prediction intervals that achieve quantile-conditional coverage. As a separate contribution, we extend distribution-free conditional calibration guarantees of histogram binning and isotonic calibration to general losses.

2024

1. Self-Calibrating Conformal Prediction

   Lars van der Laan, and Ahmed M. Alaa

   The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

   Abs PDF Code Poster

   In machine learning, model calibration and predictive inference are essential for producing reliable predictions and quantifying uncertainty to support decision- making. Recognizing the complementary roles of point and interval predictions, we introduce Self-Calibrating Conformal Prediction, a method that combines Venn- Abers calibration and conformal prediction to deliver calibrated point predictions alongside prediction intervals with finite-sample validity conditional on these pre- dictions. To achieve this, we extend the original Venn-Abers procedure from binary classification to regression. Our theoretical framework supports analyzing confor- mal prediction methods that involve calibrating model predictions and subsequently constructing conditionally valid prediction intervals on the same data, where the conditioning set or conformity scores may depend on the calibrated predictions. Real-data experiments show that our method improves interval efficiency through model calibration and offers a practical alternative to feature-conditional validity.

2023

1. Estimating Uncertainty in Multimodal Foundation Models using Public Internet Data

   Shiladitya Dutta, Hongbo Wei, Lars van der Laan, and 1 more author

   arXiv preprint arXiv:2310.09926, 2023

   Abs PDF

   Foundation models are trained on vast amounts of data at scale using self-supervised learning, enabling adaptation to a wide range of downstream tasks. At test time, these models exhibit zero-shot capabilities through which they can classify previously unseen (user-specified) categories. In this paper, we address the problem of quantifying uncertainty in these zero-shot predictions. We propose a heuristic approach for uncertainty estimation in zero-shot settings using conformal prediction with web data. Given a set of classes at test time, we conduct zero-shot classification with CLIP-style models using a prompt template, e.g., "an image of a ", and use the same template as a search query to source calibration data from the open web. Given a web-based calibration set, we apply conformal prediction with a novel conformity score that accounts for potential errors in retrieved web data. We evaluate the utility of our proposed method in Biomedical foundation models; our preliminary results show that web-based conformal prediction sets achieve the target coverage with satisfactory efficiency on a variety of biomedical datasets.

Software and computational tools

2022

1. hal9001: The scalable highly adaptive lasso

   Jeremy R Coyle, Nima S Hejazi, Rachael V Phillips, and 2 more authors

   2022

   R package version 0.4.2

   PDF