On benchmarks for learning algorithms

Abstract

Comparisons of learning algorithms are often dominated by the time taken to approach optimal weights at infinity, in typical benchmark problems with binary output targets. It is suggested that this slow final convergence be replaced by a scaling step shown to arbitrarily reduce error, for a clearer comparison of the searching power. Stopping a benchmark test by the good point criterion, rather than by a small sum-of-squared-errors, concentrates the test on this more difficult challenge, and thus reveals more about the promise of the algorithm for practical engineering use.<>