FoCAT: Foundation Model for Estimating the Conditional Average Treatment Effect

Abstract

The paper presents a novel foundation model, FoCAT (Foundation Causal Adaptive Transformer), developed for estimating the conditional treatment effect. The model addresses several key challenges inherent in causal inference tasks, including a limited sample size in the treatment group, the impossibility of simultaneously observing patient outcomes before and after intervention, and difficulties in testing models on real data. FoCAT employs a hypernetwork architecture. Unlike existing approaches that predict separate outcome functions for control and treatment groups, FoCAT directly estimates the conditional treatment effect. The model allows for control of the context informativeness through specialized classification tokens. Numerical experiments on synthetic and real-world datasets demonstrate superiority of FoCAT in estimation of the treatment effect. The code implementing FoCAT is publicly available.