If the number of trials becomes larger and larger, the empirical average value goes to the ensemble average of the event. The empirical average value is given by

- $ \bar{X}_n = \frac{X_1 + X_2 + ... + X_n}{n} $

where $ X_n $ is an instant output of the event.

This law can be applied to the information theory. The entropy of an event is represented as

- $ H_n = -\log_2 \frac{P_1 + P_2 + ... + P_n}{n} $

where $ P_n $ is a probability of event $ n $. Now, we will show that the empirical entropy goes to the ensemble entropy if the number of samples becomes larger and larger, where the ensemble average is defined as

- $ \bar{H} = E[ -p(x) \log_2 p(x)] $

where $ p(x) $ is a probability of the given event. If $ n $ becomes higher,

## Non-technical Memo

What is the definition of law of large number? Although it seems to be very simple to prove since it is very intuitive, the real formulation to prove it is not a simple. How this phenomena happens?

Why it is difficult to prove some idea which is intuitively simple but mathematically difficult? We all knows that 1+1 = 2 but it is really hard to prove it. THe proving has benn innitiated from Euclid's math. He initially suggested 6 axioms before he prooves all other theorems by the use of 6 original axiums.

### Associated Information

- First principle in Wikipedia

## Comments

### Emotional math and English

Although many students want to be good at math and English currently, they think that the expert in that areas can be made by passive studying. Even if someone can be good at understanding and solving math formulations, it is hard to be a high-level expert in mathematics. The high-level experts will write math formulations to support his ideas freely and naturally.

How can be an expert in mathematics? It is highly truth that no practice can be make one to be an expert. Long-time practicing is really important to be an expert and at the same, the trial and error approach is also highly necessary to be an expert in one area. One's ability will grow by continuous trials and errors. That is surely similar to being mathematical experts. If he think he want to write perfect texts and formulations too early, he will loose a chance to be a real expert. Ability growth is usually came from his error experience. It is okay to practice his writing in mathematics continuously regardless how many errors he makes during his practicing.