BER modeling for interference canceling FIR Wiener equalizer

The performance of a narrowband interference canceling FIR Wiener equalizer is analyzed. While mean squared error (MSE) relates to bit error rate (BER), their connection is not necessarily a direct one when the detector output noise is not Gaussian. We show that BER can be increasing for increasing signal power (or decreasing noise power) even though MSE is decreasing. A Gaussian BER model may not be accurate then. For digital modulation schemes using FIR Wiener equalizers in a narrowband interference dominated environment, a Gaussian sum model is derived for the Wiener filter output. The analytical evaluation of the probability of bit error based on the Gaussian sum model produces a BER prediction that is shown to provide a close match with observed/estimated BER, in particular for lower order equalizers.