Fertilitopes

Abstract We introduce tools from discrete convexity theory and polyhedral geometry into the theory of West’s stack-sorting map s . Associated to each permutation $$\pi $$ π is a particular set $$\mathcal V(\pi )$$ V ( π ) of integer compositions that appears in a formula for the fertility of $$\pi $$ π , which is defined to be $$|s^{-1}(\pi )|$$ | s - 1 ( π ) | . These compositions also feature prominently in more general formulas involving families of colored binary plane trees called troupes and in a formula that converts from free to classical cumulants in noncommutative probability theory. We show that $$\mathcal V(\pi )$$ V ( π ) is a transversal discrete polymatroid when it is nonempty. We define the fertilitope of $$\pi $$ π to be the convex hull of $$\mathcal V(\pi )$$ V ( π ) , and we prove a surprisingly simple characterization of fertilitopes as nestohedra arising from full binary plane trees. Using known facts about nestohedra, we provide a procedure for describing the structure of the fertilitope of $$\pi $$ π directly from $$\pi $$ π using Bousquet-Mélou’s notion of the canonical tree of $$\pi $$ π . As a byproduct, we obtain a new combinatorial cumulant conversion formula in terms of generalizations of canonical trees that we call quasicanonical trees . We also apply our results on fertilitopes to study combinatorial properties of the stack-sorting map. In particular, we show that the set of fertility numbers has density 1, and we determine all infertility numbers of size at most 126. Finally, we reformulate the conjecture that $$\sum _{\sigma \in s^{-1}(\pi )}x^{\textrm{des}(\sigma )+1}$$ ∑ σ ∈ s - 1 ( π ) x des ( σ ) + 1 is always real-rooted in terms of nestohedra, and we propose natural ways in which this new version of the conjecture could be extended.