Causal Relationships amongst Sensors in the Trinity Supercomputer: work in progress

Abstract

HPC systems are inherently complex, both to work with and to maintain. Trying to anticipate a sudden event, such as component failure or how the system will react to a newly installed module, is too large and convoluted of a problem for a single person or group of people to solve manually. In this paper, we attempt to explore the causal relationships present amongst sensors and monitoring data found in these kinds of machines. The intent of this study is to both better understand how different components and modules of the machines interact with each other, as well as get a better understanding of how a change in one part of the machine effects another part. To achieve this, we apply both a Bayesian network and logistic regression, in conjunction with a causal graph generator (TETRAD), on sensor data generated from the Trinity supercomputer in Los Alamos, NM. In particular, the data that was examined in this study focused on data from 4 slot-level sensors and 3 row-level sensors. It was found that, while these sensors do contain causal structure by themselves, they do not seem to makeup the entire causal structure, only a portion of it. The presence of latent variables, as well as possibly more interconnections (i.e. causal relationships) between each of the sensors, are likely having an effect on the predictive accuracy of the Bayesian network and logistic regression experiments conducted in this study. Therefore, it is recommended, for future work, that more experiments are conducted involving more sensors and possibly other relevant data.