0.1 Introduction

Consider a function F (w) that we seek to optimize, min w F (w), which is the sum of constituent functions, F (w) = ∑n i=1 fi(w). We will be assuming n is large, w ∈ Rd, and d fits in memory on a single machine. Now, we can calculate the gradient of F (w) as a simple sum of the gradients of the constituent fi(w) functions, ∇F (w) = ∑n i=1∇fi(w), which we can then compute in O(nd). For example, if we have a least squares objective, i.e. fi(w) = (x > i w−yi), then∇fi(w) = 2(wxi−yi)xi, which is just a re-weighting of the original vector fi(w).