Task allocation algorithm for distributed large data stream group computing in the era of digital intelligence

Abstract

In the era of digital intelligence, this paper studies the task allocation algorithm of distributed large data stream group computing, and reasonably allocates the task of group computing to meet the needs of massive computing and analysis of distributed large data stream. According to the idea of swarm intelligence perception and crowdsourcing platform, the task allocation model of distributed large data stream group computing is constructed to realize the task allocation of group computing. A distributed large data stream group computing task model and a user model are constructed, user attributes are initialized by using the accuracy of the answers submitted by users, the possibility that users can participate in the group computing task is predicted by a logistic regression algorithm, so that user candidate sequences participating in the computing task can be obtained, and the accuracy of the user’s real topics and corresponding topics can be grasped by capturing the candidate users’ real topics and evaluating the accuracy algorithm. Select the users who meet the subject area, update the candidate user sequence, and filter the users again on the basis of fully considering the factors such as information gain, user integrity and cost, so as to get the final user sequence and complete the task allocation of group computing. Experiments show that this method can solve the problem of distributed large data flow group computing task allocation, achieve high accuracy, reduce the cost, and effectively improve the information gain.