Leveraging Feature Headers to Learn Sparse and Semantically Pertinent Linear Models

Abstract

Readily available data and software tools have turned "analytics" into a game anyone can play. But genuine, serious modeling demands prudence: domain experts routinely use their knowledge to assess the relevance of various input features and to be judicious with model selection criteria. While engaged in analysis, they marshal knowledge to consider the meaning of the data involved. Seeking to automate and reproduce such aspects, the present paper proposes a framework that makes use of semantics latent within given feature headers to help produce sparse and semantically pertinent linear models, rather than exploiting mere correlations or (potentially spurious) patterns. This framework enables a model builder to employ both features’ data and certain semantic information derived from their headers to search for an optimal feature subset in order to improve generalization of a linear model being built. To do so, a characteristic called "semantic inconsistency" is formulated in order to quantify the degree of conflict between weights learned from data and the amount of relationship between a set of input features and the output being predicted in the semantic space. Using this quantity, semantic information can be incorporated into a regularization procedure in a manner that is quite general and may be computed from any form of background knowledge. The results obtained from validating the framework with four datasets indicate that taking the semantics of features into account can improve model generalization: the approach is shown to perform better than classic linear regression and regularization techniques that consider only complexity of learned models.