The Tale of the Tail: Inference for Customer Purchase Behavior in the Long Tail

A large product assortment is typically characterized by many products that are rarely purchased: the long tail. Combined, these products make a sizable contribution to the total purchase volume. A retailer that better understands the purchase behavior in its long tail can increase the value of these products. Yet, analyzing tail purchases at the customer level is challenging: The available purchases per product in the tail are limited, while the number of customers and products are large. We develop new methodology that overcomes these challenges and sheds light on customer-specific purchase behavior for the long tail. The idea underlying our approach is a dimension reduction that uses latent product groups to summarize tail purchase behavior. We rely on variational inference to apply our method to a large-scale purchase history dataset with almost 50,000 products and over 3 million shopping trips. We are able to identify the customers that are likely to purchase in the tail of the assortment, how this varies across product categories, and how tail purchases relate to purchases of other products in the assortment. These insights can be used to improve recommendations of tail products, facilitate navigation through the assortment, and inform assortment management decisions.