On the measurement uncertainty caused by finite resolution and its relation to the rectangular distribution

Abstract

The paper points out that the current practice of evaluation of measurement uncertainty for a finite resolution of measuring instruments uses the rectangular distribution to describe such an effect. The approximate formula for a calculation of the expanded uncertainty of the normal distribution convolved with the rectangular distribution as a contributing distribution is presented and compared with the direct effect of a finite resolution simulated on the normally distributed data. The interpretation of the numeric rounding caused by a finite resolution as a variable with the rectangular distribution is discussed and the solution for estimations with a quantized distribution that is observed is suggested. This approach allows to avoid underestimation in the evaluation of measurement uncertainties, especially for routinely calibrated instruments, where the finite resolution is often the dominant contribution to the uncertainty budget.