Research on posts analysis based on data process automation

Abstract

The structural contradiction between talent supply and demand is the key problem to be solved in higher vocational colleges. Recruitment website provides massive recruitment data. The analysis of recruitment data has important practical significance for promoting the reform and innovation of talent training mode. Based on big data technology, the distributed real-time incremental collection of posts information is realized by Redis and Scrapy technology. The crawled posts information is stored in HBase database. The posts data is analyzed by spark platform, and the analysis result is stored in MySQL database. The charts are displayed by Flask framework and Echarts library. The system is closely linked to the pain spot of the current higher vocational talent training, and it is closely combined the skills required by the post with the courses offered by the school. It is helpful to improve the quality of talent training and cultivate more high-quality skilled talents.