Mutual information evolution and Conditional Entropy Asymmetry: work in progress

 

The mutual information between two processes is defined as:

 

(1)

 

 

By the above definition implies the symmetry of the Mutual information (MI):

 

(2)

 

 

Following (1), MI can be defined by :

 

(3)

 

Where:

 

Another version of MI is the following:

 

(4)

 

By combining (2) and (4), one gets:

 

(5)

 

As a consequence, MI’s symmetry property captured by (2) does not put into evidence possible differences between conditional entropies.

This property represents a significant limit in the adoption of MI in evolving in time processes of self-organisation of complex objects.

Knowing  about X₁ it is not equivalent to knowing  about X₂. Mutual information does not measure this asymmetry, which evidently exists. This seems to be a limit of Mutual Information, which possibly affects our capability of understanding some basic mechanism of information redistribution in a physical object by physical modularisation. It is interesting to notice how the appearance of asymmetry in mutual information between modules reminds some form of symmetry-breaking, a mechanism particularly important in the understanding of complex systems dynamics.

 

Therefore a different form of MI is needed, which should be capable to capture possible asymmetry in conditional entropies. To do that one can take advantage of some property of involved quantities. More precisely, al the quantities involved in MI definition are positive or, at most null. This being the case, one can make use of the following simple equation:

 

(6)

 

 

 

 

Taking advantage of MI symmetry, one can write it as:

 

(7)

 

 

 

And, making use of (6):

 

(8)

 

Which, while being equivalent to equation (3), captures, by the second term on the right hand side, the asymmetry in conditional entropies.

This is a constrain due to the meaning of symbols H(X) and, more precisely to their properties of never being negative quantities. Equation (8) puts into evidence how, all other terms being given and coherently valued, the asymmetry term contributes positively to MI.

 

Assume that the second term on the right hand side is null. Equation (8) reduces to:

 

(9)

 

 

Because of equation (5), we obtain, for MI=0, the same conditions that one would have obtained by the standard definition of MI. Equation (8) is consistent and

 

can be checked by assuming, for example, that . Then:

 

 

 

which is evidently correct.

The practical consequences of equation (8) are, as suggested above, particularly evident in evolving situations. Suppose for a moment that evolution in time of MI takes place while the entropy of the two modules are kept constant. Equation (8) reduces to:

 

 

(10)

 

An increase of mutual information implies a growth of the asymmetry term higher than that of the maximum. This implies that, in order to increase MI, the lower conditional entropy term needs to be reduced until  the asymmetric term reaches values such to make possible a growth of the last term on the right hand side of equation (9). We can then conclude that the growth of MI between the two modules will cause the domination of one module over the other.