Distributed optimization for penalized regression in massive compositional data

Abstract

Compositional data have been widely used in various fields to analyze parts of a whole, providing insights into proportional relationships. With the increasing availability of extraordinarily large compositional datasets, addressing the challenges of distributed statistical methodologies and computations has become essential in the era of big data. This paper focuses on the optimization methodology and practical application of the distributed sparse penalized linear log-contrast model for massive compositional data, specifically in the context of medical insurance reimbursement ratio prediction. We propose two distributed optimization techniques tailored for centralized and decentralized topologies to effectively tackle the constrained convex optimization problems that arise in this application. Our algorithms are rooted in the frameworks of the alternating direction method of multipliers and the coordinate descent method of multipliers, making them available for distributed data scenarios. Notably, in the decentralized topology, we introduce a distributed coordinate-wise descent algorithm that employs a group alternating direction method of multipliers to achieve efficient distributed regularized estimation. We rigorously present convergence analysis for our decentralized algorithm, ensuring its reliability for practical applications. Through numerical experiments on both simulated datasets and a real-world medical insurance dataset, we evaluate the performance of our proposed algorithms.