Sieve maximum likelihood estimation for generalized linear mixed models with an unknown link function

We study the generalized linear mixed models with an unknown link function for correlated outcome data. We propose sieve maximum likelihood estimation procedures by using B‑splines. Specifically, we estimate the unknown link function in a sieve space spanned by the B‑spline basis of the linear predictor that includes both the fixed and random terms. We establish the consistency and asymptotic normality of the proposed sieve maximum likelihood estimators. Extensive simulation studies, along with an application to an epileptic study, are provided to evaluate the finite-sample performance of the proposed method.