The effects of large round-off errors on the performance of control charts for the mean when the quality characteristic is normally distributed with a known variance

Abstract

This research discusses the effects of large round-off errors on the performance of control charts for means when a process is normally distributed with a known variance and a fixed sample size. Quality control in practice uses control charts for means as a process monitoring tool, even when the data is significantly rounded. The objective of this research is to demonstrate how ignoring the round-off errors and using a standard Shewhart chart affects the quality control of a measured process. The first part of the research includes theoretical calculations for estimating the values of alpha and beta, relating to the unrounded data and the large-rounded data. For the rounded data, normality can no longer be assumed because the data is discrete; therefore, the multinomial distribution is used. The results show that given an in-control process, alpha indicates that false alarms are more frequent, whereas given an out-of-control process, the influence on beta is minor and inconsistent. For some rounding levels, there is a decline in the control chart performances, and in others, there is an improvement. In the second part, a simulation study is used to evaluate the performances of the control chart based on a single sample, checking whether the conclusion (reject or fail to reject) for a sample is consistent for rounded and unrounded data. The results of the simulation match the theoretical calculations.