Probability theory is the formal and rigorous study of randomness. It forms the basis for all of statistics, in a similar way that analysis forms the basis for calculus. In fact, probability can be considered a special case of analysis and measure theory. The subject is quite rich and deep, and we recommend anyone looking to get into statistics of machine learning take the time to really understand the fundamentals.