Application of the Variational Principle to Create a Measurable Assessment of the Relevance of Objects Included in Training Databases

Abstract

We consider the problem of obtaining a measurable assessment of the quality of empirical training data selected by experts. This problem can be solved in those cases where the data can be displayed in the form of histograms. This class includes any diagrams of frequency of occurrence of linguistic objects in samples, for example, lemmas in a text. It also includes discretized temporal signals from different branches of science, technology, and medicine. The proposed method, as well as other known methods, is based on the use of weight functions. With its help, the weight of each histogram is defined as the sum over all its columns of the products of column height by the value of weight function for the corresponding column. However, in contrast to the well-known approaches, the weight function in the proposed approach is not found empirically, but on the basis of the following variation principle. The weight function is considered optimal if the weight of the lightest histogram found with its help is greater than or equal to the weight of the lightest histogram determined by any other weight function. The application of the developed approach to the task of thematic classification of ad texts on electronic trading floors showed that for the selected topics approximately 90% of the lemmas (words) encountered in the training corpus had the weight equal to zero, and almost all words with nonzero weight were semantically related to the topic.