Statistics 101 — Law of Large Numbers

In probability theory, the law of large numbers (LLN) is a theorem that describes the result of performing the same experiment a large number of times. According to the law, the average of the results obtained from a large number of trials should be close to the expected value and will tend to become closer to the expected value as more trials are performed.

weak law of large numbers

The weak law of large numbers is a result of probability theory also known as Bernoulli’s theorem.

strong law of large numbers

The strong law of large numbers states that the sample average converges almost surely to the expected value.

What this means is that the probability that, as the number of trials n goes to infinity, the average of the observations converges to the expected value, is equal to one.

The proof is more complex than that of the weak law.[17] This law justifies the intuitive interpretation of the expected value (for Lebesgue integration only) of a random variable when sampled repeatedly as the “long-term average”.

Almost sure convergence is also called a strong convergence of random variables. This version is called the strong law because random variables that converge strongly (almost surely) are guaranteed to converge weakly (in probability). However, the weak law is known to hold in certain conditions where the strong law does not hold and then the convergence is only weak (in probability). See #Differences between the weak law and the strong law.

Limitation of the Law of Large Numbers
The average of the results obtained from a large number of trials may fail to converge in some cases. For instance, the average of the results from Cauchy distribution or some Pareto distribution (α<1) will not converge as x becomes larger since they have heavy tails. Cauchy distribution and Pareto distribution represent two different cases: the Cauchy distribution does not have an expectation[4], while the expectation of Pareto distribution (α<1) is infinite[5]. Another example is where the random numbers equal the tangent of an angle uniformly distributed between −90° and +90°. The median is zero, but the expected value does not exist, and indeed the average of n such variables have the same distribution as one such variable. It does not converge in probability towards zero (or any other value) as n goes to infinity.