Automated Analysis of Load Test Results of Systems with Equilibrium or Transient Behavior: Invited Talk

Abstract

Performance test data should be analyzed to determine if performance requirements are being met, to see if they reveal opportunities for performance improvement, and to see if they show signs of lurking performance issues or malfunctions. Automated analyses of the measurements can be useful when the number of resource usage measures and performance measures is large, when the number of nodes under test is large, or when the number of test cases is large. We shall examine this for cases in which the system under test is subjected to a constant load, as might be the case for an online transaction processing system, and for the case where the load is inherently bursty, as would be the case for an alarm or monitoring system that is receiving streams of notifications from many sensors at once. We expect a well behaved system under constant load to reach steady state shortly after the test load begins, and to ramp down to its previous state once the load is withdrawn. This corresponds to a system reaching equilibrium in stochastic terms. We also expect a well behaved system to return to steady state after a surge of traffic abates. Failure to achieve equilibrium under constant load is a sign of a problem that should be investigated. Automated analysis of voluminous test data facilitates the identification of intervals of steady operation and cases when steady operation has not occurred during performance tests. An embedded control or monitoring system, such as a building security system, might be subject to a sustained burst of message traffic in an emergency situation. Fire alarm systems might have to respond to at least one of these messages within seconds of the onset of the burst by triggering bells and sirens, closing doors, and automatically alerting emergency services. They might consist of only a few hosts. An automated tool could use statistical methods to identify phases of execution during an emergency by noting when each thread's processor consumption changes. This facilitates the identification of areas for performance improvement, especially if the system is implemented with so many processes or threads that visual identification of heavy consumers of processing power is difficult. We shall elaborate on these issues in this talk, and also discuss methods for performing automated analyses of load tests of systems whose loads are expected to be steady and of systems whose loads are expected to be intense for a transient period.