ADMM penalty parameter selection with krylov subspace recycling technique for sparse coding

Sparse representations are widely used in a broad variety of fields. A number of different methods have been proposed to solve the sparse coding problem, of which the alternating direction method of multipliers (ADMM) is one of the most popular. One of the disadvantages of this method, however, is the need to select an algorithm parameter, the penalty parameter, that has a significant effect on the rate of convergence of the algorithm. Although a number of heuristic methods have been proposed, as yet there is no general theory providing a good choice of this parameter for all problems. One obvious approach would be to try a number of different parameters at each iteration, proceeding further with the one that delivers the best reduction in functional value, but this would involve a substantial increase in computational cost. We show that, when solving the sparse coding problem for a dictionary corresponding to an operator with a fast transform, requiring iterative methods to solve the main linear system arising in the ADMM solution, it is possible to explore a large range of parameters at marginal additional cost, thus greatly improving the robustness of the method to the choice of penalty parameter.