IDL-BiGRU: Integrated deep learning assisted smart scheduling of big data over cloud environment

The rapid expansion of Internet of Things (IoT) applications generates a continuous and massive flow of data, creating significant challenges in both data processing and storage management. Cloud computing offers scalable infrastructure to handle such data intensive workloads, but optimal task scheduling remains critical to ensure performance and resource efficiency. Traditional scheduling algorithms often fall short due to limited adaptability and consideration of only a few system parameters. In this paper, a novel integrated deep learning-assisted scheduling framework is utilized for scheduling big data over a cloud environment. The proposed framework integrated deep reinforcement learning with the bidirectional gated recurrent unit (IDL-BiGRU) model to intelligently schedule tasks based on real-time system states. The IDL-BiGRU model leverages the advantage of deep Q-learning for decision making and BiGRU's ability to capture bidirectional temporal dependencies in task and resource usage patterns. In this work, RAM, CPU, bandwidth utilization of the network, and disk storage are considered for scheduling purposes. The suggested method is to shorten the makespan and increase resource utilization. The Java tool is utilized for conducting the experimental verifications. Analysis and comparison of the suggested deep learning framework's performance with current methods are done. For 1000 tasks, the proposed method attains 0.90 degrees of imbalance, 291.17 ms downtime, 1050 ms throughput, and 721.58 makespan. The performance analysis demonstrates that the suggested strategy outperforms previous methods.