As an emerging platform gaining significant attention from the biomedical community, multiplexed fluorescent imaging from imaging flow cytometry enables simultaneous detection of numerous biological targets within a single cell. Due to the spectral overlap, signals from one fluorophore can bleed into other detection channels, leading to spillover artifacts, which cause erroneous results and false discoveries. Existing color compensation algorithms use special samples to calibrate the fluorophores individually, a time-consuming and laborious process that is cumbersome and hard to scale. While recent developments in calibration-free algorithms produce promising results in multi-color microscope images, these algorithms, when applied to single-cell images with all the fluorophores within a small and constrained area, tend to cause overcorrection by treating real signals as crosstalk and triggering stability problems during the iterative computation process. Here we demonstrate a simple and intuitive algorithm that greatly reduces overcorrection and is computationally efficient. While designed for imaging flow cytometers, our calibration-free crosstalk removal algorithm can be readily applied to microscopy as well. We have validated its effectiveness on various datasets, including simulated cell images, 2D and 3D imaging flow cytometry images, and microscopic images. Our algorithm offers an effective solution for multi-parameter single-cell images where channels are often both spectrally and spatially overlapped within the limited area of a single cell.