Run-time validation of knowledge-based systems

As knowledge bases become more complex it is increasingly unlikely that they will have been validated against all possible data and therefore an increasing risk of making errors. Run-time validation is checking whether the output of a knowledge base for some data is likely to be correct at the time the data is processed. We have investigated various techniques for runtime validation. The most successful technique has been to constantly re-build a separate knowledge base using a different learning technique with cases labeled by the knowledge base being validated, as training data. Any new cases are processed by both knowledge bases and if the knowledge bases disagree the case is referred for manual checking as a possible outlier. If an outlier is detected the knowledge base is edited to give the correct answer and as cases are processed they are added to the training data for the machine learning knowledge base.