OptSpec: Optimization of Specifications Data for Engine Anatomy

Search engines are progressing with design and technology to meet the business models' continually changing and evolving needs. From crawling the links for indexing them to presenting the results based on the user query, search engine components have been researched in depth and breadth. This research investigation is focused on managing and processing the name-value pair data. 21K e-commerce data items were processed into JSON format in key-value pairs to explore the combinatorial explosion and its impact of the smaller subsets, items segregated by the domains. An item might not have all of its specifications contributing to the results in the query management. We propose a method to generate subsets of categories that can potentially contribute to the processing and results. One of the ways to manage huge data is using threads and parallel processing. We use the approach and compare the results with single threaded model. The processed and raw data storage methods have also been compared for different item sets. The model also presents the approaches of selecting the right set of specifications depending on the user query. Several methods are compared based on the limiting set threshold. The results appear to be an effective way of managing associative data. The known methods and parameters are positively better than the neural and deep learning networks that keep the methodology hidden and in the black box from the users. When the thresholds and sets are known, the algorithms can make better decisions.