All-Spark: Using Simulation Tests Directly in Production Environments to Detect System Bottlenecks in Large-Scale Systems

With the rapid growth in e-commerce, large-scale promotional activities have become a popular concept. However, when the existing system cannot be adjusted efficiently to adapt to the tremendous traffic in the promotion period, which is hundreds of times more than the volume on normal days, it be-comes a bottleneck that restricts the continuous growth of the online business. Traditional capacity prediction methods have been proven to be incapable of making accurate predictions for such special scenarios, because of a variety of unpredictable system bottlenecks. Simulation testing in a completely new test environment for such a large scale has a number of defects and limitations, such as the high cost of setting up the environment and the difficulty of testing the entire environment. Moreover, bottlenecks found in the test server may be different from those in the production server. We investigated online simulations in the production environment and built a complete simulation test system called All-Sparks. This solution solved a long-standing problem of simulation testing with large traffic in the production environment without causing any data pollution. The simulation test revealed hundreds of bottlenecks under a high workload pressure every year to eliminate the hidden problems caused by new applications. The final capacity evaluation result was deviated by less than 5% from the actual capacity, and the error rate was small (<2%); both of these are significant improvements over the traditional prediction results. This solution also provided a framework with good expansibility to multiple scenarios other than stress testing.