Towards automated generation of fast and accurate algorithms for recursive matrix multiplication

Abstract

We propose a strategy for the generation of fast and accurate versions of non-commutative recursive matrix multiplication algorithms. To generate these algorithms, we consider matrix and tensor norm bounds governing the stability and accuracy of numerical matrix multiplication. We start by a unification on known max-norm bounds on matrix multiplication stability and then extend them to further norms and more generally to recursive bilinear algorithms and the alternative basis matrix multiplication algorithms. Then our strategy has three phases. First, we reduce those bounds by minimizing a growth factor along the orbits of the associated matrix multiplication tensor decomposition. Second, we develop heuristics that minimize the number of operations required to realize a bilinear formula, while further improving its accuracy. Third, we perform an alternative basis sparsification that improves on the time complexity constant and mostly preserves the overall accuracy. Let $〈m\times k\times n:r〉$ denote a non-commutative algorithm multiplying an $m\times k$ matrix with an $k\times n$ matrix, using r coefficient products. Our strategy allows us to propose a $〈2\times 2\times 2:7〉$ algorithm that reaches simultaneously a better accuracy in practice, compared to previously known such fast $〈2\times 2\times 2:7〉$ Strassen-like algorithms, and a time complexity bound with the best currently known leading term (obtained via alternative basis sparsification). We also present detailed results of our technique on Smirnov's $〈3\times 3\times 6:40〉$ family of algorithms.