Can the Dirac deltas in dipole fields be ignored in classical interactions?

When studying (or teaching) classical electromagnetism, one is bound to deal with the electric field of an ideal electric dipole, as well as its magnetic counterpart. A careful analysis then reveals that each of those fields must include, for consistency, a term proportional to a Dirac delta function localized at the position of the dipole. However, one is usually told not to worry about those terms since, as classical interactions always involve sources which are spatially separated, the Dirac-delta terms are only relevant for quantum mechanics, where they are directly related to important phenomena. In this work, we pose and solve a purely classical problem in electrostatics in which the Dirac-delta terms in the dipole fields are indispensable. It involves the computation of the interaction energy between a conductor with a spherical cavity and an (ideal) electric dipole located at the center of that cavity. We also solve its magnetic counterpart, replacing the conductor with a superconductor and the electric dipole with a magnetic one.