A generalized single-index linear threshold model for identifying treatment-sensitive subsets based on multiple covariates and longitudinal measurements

Identification of a subset of patients who may be sensitive to a specific treatment is an important step towards personalized medicine. We consider the case where the effect of a treatment is assessed by longitudinal measurements, which may be continuous or categorical, such as quality of life scores assessed over the duration of a clinical trial. We assume that multiple baseline covariates, such as age and expression levels of genes, are available, and propose a generalized single-index linear threshold model to identify the treatment-sensitive subset and assess the treatment-by-subset interaction after combining these covariates. Because the model involves an indicator function with unknown parameters, conventional procedures are difficult to apply for inferences of the parameters in the model. We define smoothed generalized estimating equations and propose an inference procedure based on these equations with an efficient spectral algorithm to find their solutions. The proposed procedure is evaluated through simulation studies and an application to the analysis of data from a randomized clinical trial in advanced pancreatic cancer.