## FANDOM

186 Pages

If the number of trials becomes larger and larger, the empirical average value goes to the ensemble average of the event. The empirical average value is given by

$\bar{X}_n = \frac{X_1 + X_2 + ... + X_n}{n}$

where $X_n$ is an instant output of the event.

This law can be applied to the information theory. The entropy of an event is represented as

$H_n = -\log_2 \frac{P_1 + P_2 + ... + P_n}{n}$

where $P_n$ is a probability of event $n$. Now, we will show that the empirical entropy goes to the ensemble entropy if the number of samples becomes larger and larger, where the ensemble average is defined as

$\bar{H} = E[ -p(x) \log_2 p(x)]$

where $p(x)$ is a probability of the given event. If $n$ becomes higher,

## Non-technical Memo

What is the definition of law of large number? Although it seems to be very simple to prove since it is very intuitive, the real formulation to prove it is not a simple. How this phenomena happens?

Why it is difficult to prove some idea which is intuitively simple but mathematically difficult? We all knows that 1+1 = 2 but it is really hard to prove it. THe proving has benn innitiated from Euclid's math. He initially suggested 6 axioms before he prooves all other theorems by the use of 6 original axiums.