How orthogonal are we? A note on fast and accurate inner product computation in the floating-point arithmetic

Abstract

The multiply-and-accumulate (MAC) belongs to the most fundamental operations in digital signal processing. It constitute the key operation for digital filters as well as object classification, to name a few. Recently we faced implementation issues related to tensor decomposition into orthogonal factors, as well as tensor projections onto the orthogonal tensor bases. Also these rely heavily on the MAC operations. However, a serious problem of the numerical accuracy vs. operation speed of these operations can be observed when implemented with the floating point arithmetic realized in a hardware or a software platform. If not carefully approached, this can lead to significant numerical errors which are frequently overlooked by inexperienced engineers critically relying on standard libraries. In this paper we remind, dust off and discuss some computational aspects of the floating point implementation of the MAC operations in big data signal processing tasks on various platforms, and in different operation modes (serial, parallel, software, hardware). Such algorithms as simple summation, the Kahan algorithm, as well as some hybrid solutions are analyzed in respect to their accuracy, simplicity in implementation, and resource consumption, as well as speed of execution.