Task Scheduling Optimization Based on Firefly Algorithm in Storm

Abstract

As an open source distributed real-time computing framework, Storm has been widely used in social network, e-commerce, stock analysis and other fields. The default scheduler of Storm try to distribute all executors of topology among all worker nodes via an even strategy using a round-robin algorithm, which may result in performance bottleneck due to high topology processing latency and low throughput. Aiming at optimizating it, we design a task scheduling optimization based on firefly algorithm to reallocate tasks to more suitable nodes according to a task scheduling scheme. We use the location of firefly to represent a feasible scheduling scheme, and the fluorescence brightness represents the node’s ability to process tasks, while the process of finding the best task scheduling scheme is simulated as the process of firefly approaching the brightest position. The Experimental results show that compared to the default scheduling algorithm, the scheduling algorithm we proposed has better task scheduling efficiency, less average processing time and higher throughput, which can optimize the performance of the cluster.