back to arrowtheory.com
On the concept of entropy of a finite probabilistic scheme
D. K. Faddeev, 1956.
English translation by Arina Zinovyeva.
Section 1. A. Ya. Khinchin's article  established that the entropy of a finite probabilistic scheme can be characterised by the following system of axioms.
1. is a continuous function of in the domain .
2. is a symmetric function of .
where (this axiom is formulated in the article  in other terms).
Entropy is defined by these axioms uniquely up to a positive multiplier. The aim of this note is a simplification of this system of axioms. Namely, we offer the following axioms:
1'. is continuous, with and positive at least at one point.
2'. is a symmetric function of .
3'. With Here .
The difference between these two systems of axioms is, first of all, axiom 5 (extremality) is replaced with the requirement of positivity of the entropy at one point, and second of all, axioms 3 and 4 are replaced with axiom 3' that is a very natural one if we consider entropy as a measure of the "indefiniteness" of a scheme. Truly, the "indefiniteness" of a scheme is different from the "indefiniteness" of the scheme occurring from dividing events into two sub-events , with conditional probabilities . This "indefiniteness" is overcome only when the event is realised, the probability of which equals .
Section 2. First of all, we establish that axioms 2, 3, 4 are equivalent to 2', 3'.
Lemma 1. From 2, 3, 4, we conclude
Proof. We calculate with two methods. On one hand, by the consequence of 3, we have On the other hand, and by the consequence of 4, we get Therefore,
Lemma 2. From 2, 3, 4 we conclude 3'.
Proof. equals, by the consequence of 2 and 3, , which equals, by the consequence of 4,
Lemma 3. From 2', 3', we conclude that
Proof. We calculate with two methods On one hand, by the consequence of 3', we have On the other hand, Therefore,
Lemma 4. From 2', 3' we conclude 3.
Lemma 5. From 3', we conclude that: Here ,
Proof. With the statement of lemma is the same as axiom 3', with the lemma is easily proven by induction on .
Lemma 6. From 2', 3', we conclude that: Here
Proof. We need to apply lemma 5, times to each group of arguments, which is possible by the consequence of symmetry. Setting we get that from 2', 3' we can conclude 4.
So, we have established the equivalence of axiom groups 2, 3, 4 and 2', 3'.
Section 3. Assume, following the article , with and Applying lemma 6 to the case we get with With or equation (1) is trivial. Further, applying lemma 5 to , we get from that we conclude:
Lemma 7. With
Proof. By the consequence of the continuity of with
Furthermore, where and But is an arithmetic average of the first members of the sequence which tends to zero. Therefore, and Furthermore, The lemma is proven.
By the consequence of equation (1), will be defined for all natural , as soon as we set the values of the function on prime numbers. Precisely, if where are prime, then Assuming Then
Lemma 8. There is a greatest number among .
Proof. We assume the opposite and show that this assumption will contradict the assumption about the continuity of at It's true, if there is no greatest number in the sequence then we can build an infinite sequence of prime numbers so that is the smallest prime number such that It is clear from the method of building this sequence that when a prime number is smaller than , then
Let and be a canonical decomposition of the number . Consider It is clear that because and by the consequence of the parity of among there is a number 2 with a nonzero index. Then with and
Therefore, which is impossible.
Using exactly the same method, we establish that there is a smallest among
Lemma 9. where c is a constant.
Proof. It's enough to establish that all are equal.
Assume that such a prime number is found, that We set a prime number for which takes the greatest value. Then and
Let be a natural number and a canonical decomposition of . We consider Under the transition to the limit we establish: which is impossible. Therefore, takes place for all
Using exactly the same method, it's established that for all . The lemma is proven.
Proof. We first consider the case Let with r and s whole. We apply lemma 6 to by combining arguments into two groups with and elements. We get: Where
By the consequence of the continuity of , the formula that we get can be also applied for irrational values of by the consequence of positivity of at least at one point.
The transition to the general case is realised by induction, based on axiom 3'.
 A. Ya. Khinchin, On the concept of entropy in the theory of probabilities, Uspekhi Mat. Nauk, VIII, issue 3 (55), (1953).
Copyright (c) 2018 - 2019.