Heckerthoughts

Abstract

In 1987, Eric Horvitz, Greg Cooper, and I visited I.J. Good at his university. We wanted to see him was not because he worked with Alan Turing to help win WWII by decoding encrypted messages from the Germans, although that certainly intrigued us. Rather, we wanted to see him because we had just finished reading his book "Good Thinking," which summarized his life's work in Probability and its Applications. We were graduate students at Stanford working in AI, and amazed that his thinking was so similar to ours, having worked decades before us and coming from such a seemingly different perspective not involving AI. This story is a fitting introduction this manuscript. Now having years to look back on my work, to boil it down to its essence, and to better appreciate its significance (if any) in the evolution of AI and ML, I realized it was time to put my work in perspective, providing a roadmap to any who would like to explore it. After I had this realization, it occurred to me that this is what I.J. Good did in his book. This manuscript is for those who want to understand basic concepts central to ML and AI and to learn about early applications of these concepts. Ironically, after I finished writing this manuscript, I realized that a lot of the concepts that I included are missing in modern courses on ML. I hope this work will help to make up for these omissions. The presentation gets somewhat technical in parts, but I've tried to keep the math to the bare minimum. In addition to the technical presentations, I include stories about how the ideas came to be and the effects they have had. When I was a student in physics, I was given dry texts to read. In class, however, several of my physics professors would tell stories around the work. Those stories fascinated me and really made the theory stick. So here, I do my best to present both the ideas and the stories behind them.