Algorithmic Information Dynamics Unit 6 Hector Zenil and Narsis Kiani

1. Limitations of Shannon
Entropy
and Computable
Measures
Dr. Hector Zenil

4. To illustrate the limitations of entropy
we created a very simple graph:
1. Let 1 → 2 be a starting graph G connecting a
node with label 1 to a node with label 2. If a node
with label n has degree n, we call it a core node;
otherwise, we call it a supportive node.
2. Iteratively add a node n + 1 to G such that the
number of core nodes in G is maximised.

6. Algorithm:
The degree sequence d of the labeled nodes
d = 1,2, 3, 4, . . . , n
is the Champernowne constant in base 10, a transcendental real
whose decimal expansion is Borel normal.

7. Algorithm:
The sequence of number of edges is a function of core and supportive nodes, defined by
[1/r] + [2/r] +···+ [n/r],
where r = (1 + √5)/2 is the golden ratio and [· ] the floor
function (sequence A183136 in the OEIS), whose values are
1, 2, 4, 7, 10, 14, 18, 23, 29, 35, 42, 50, 58, 67, 76, 86, 97, 108,
120, 132, 145,....

8. Algorithm:
From iteration 1 to
iteration 3:

11. Algorithm:

12. Maximum Entropy (MaxEnt)
Model Principle
The principle of maximum entropy states that the
probability distribution which best represents the
current state of knowledge is the one with largest
entropy…
E.T. Jaynes

14. Computing the
Uncomputable:
The Coding Theorem
Method
Dr. Hector Zenil

15. K(s) = min{ |p| | U(p) = s}
Computability, Algorithmic
Complexity & Causality
(Un)Computability mediates in the challenge of causality
by way of Algorithmic Information Theory:

16. Computability, Algorithmic
Complexity & Causality
(Un)Computability mediates in the challenge of causality
by way of Algorithmic Information Theory:
Generating
mechanism
Observation
K(s) = min{ |p| | U(p) = s}

17. Generating
mechanism
Observation
K(s) = min{ |p| | U(p) = s}
Most likely
mechanism
according
to Ockham’s
Computability, Algorithmic
Complexity & Causality
(Un)Computability mediates in the challenge of causality
by way of Algorithmic Information Theory:

18. Constructor
Generating
mechanism
Observation
K(s) = min{ |p| | U(p) = s}
Most likely
mechanism
according
to Ockham’s
Computability, Algorithmic
Complexity & Causality
(Un)Computability mediates in the challenge of causality
by way of Algorithmic Information Theory:

19. Compatible
with the
observation
Constructor
Generating
mechanism
Observation
K(s) = min{ |p| | U(p) = s}
Most likely
mechanism
according
to Ockham’s
Computability, Algorithmic
Complexity & Causality
(Un)Computability mediates in the challenge of causality
by way of Algorithmic Information Theory:

20. Compatible
with the
observation
Constructor
Generating
mechanism
Observation
K(s) = min{ |p| | U(p) = s}
Most likely
mechanism
according
to Ockham’s
Computability, Algorithmic
Complexity & Causality
(Un)Computability mediates in the challenge of causality
by way of Algorithmic Information Theory:

21. Algorithmic Data Analytics, Small Data Matters and Correlation
versus Causation. In Computability of the World? Philosophy and
Science in the Age of Big Data), Springer Verlag, 2017
suggestive
methods
Coding Theorem Method
(CTM)

22. Algorithmic Data Analytics, Small Data Matters and Correlation
versus Causation. In Computability of the World? Philosophy and
Science in the Age of Big Data), Springer Verlag, 2017
Coding Theorem Method
(CTM)
suggestive
methods

23. The Coding Theorem Method
(CTM)

24. The Coding Theorem Method
(CTM)

25. The Coding Theorem Method
(CTM)

29. https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0096223

35. How it Looks Like?
https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0096223

36. How it Looks Like?

37. So what about the constant in
the Invariance Theorem?
https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0096223

38. Shannon Entropy and
Algorithmic Complexity
Joining Forces:
The Block Decomposition
Method (BDM)
Dr. Hector Zenil

39. Divide and Conquer
sample
sample
sample
sample
sample
If any part of the whole system (samples) is of high m(x) and low K(x), then that part can
be generated by mechanistic/algorithmic means and thus is causal. The lower BDM the
more causal.
Rube
Goldberg
machine

40. Block Decomposition Method
(BDM)
sample
sample
sample
sample
If any part of the whole system (samples) is of high m(x) and low K(x), then that part can
be generated by mechanistic/algorithmic means and thus is causal. The lower BDM the
more causal.
Rube
Goldberg
machine

41. Formally:
Block Decomposition Method
(BDM)
H Zenil, F Soler-Toscano, N.A. Kiani, S. Hernández-Orozco, A. Rueda-Toicen
A Decomposition Method for Global Evaluation of Shannon Entropy and Local Estimations of Algorithmic
Complexity, Entropy 20(8), 605, 2018.

42. Block Decomposition Method
(BDM)
H Zenil, F Soler-Toscano, N.A. Kiani, S. Hernández-Orozco, A. Rueda-Toicen
A Decomposition Method for Global Evaluation of Shannon Entropy and Local Estimations of Algorithmic
Complexity, Entropy 20(8), 605, 2018.

43. Block Decomposition Method
(BDM)
H Zenil, F Soler-Toscano, N.A. Kiani, S. Hernández-Orozco, A. Rueda-Toicen
A Decomposition Method for Global Evaluation of Shannon Entropy and Local Estimations of Algorithmic
Complexity, Entropy 20(8), 605, 2018.

44. Block Decomposition Method
(BDM)
H Zenil, F Soler-Toscano, N.A. Kiani, S. Hernández-Orozco, A. Rueda-Toicen
A Decomposition Method for Global Evaluation of Shannon Entropy and Local Estimations of Algorithmic
Complexity, Entropy 20(8), 605, 2018.

45. Block Decomposition Method
(BDM)
For example:
If 2 strings are 10111 and 10111, a tighter upper bound of their
algorithmic complexity K is not:
|K(U(p)=10111)| + |K(U(p)=10111)| but
|K(U(p)=10111 2 times)|
H Zenil, F Soler-Toscano, N.A. Kiani, S. Hernández-Orozco, A. Rueda-Toicen
A Decomposition Method for Global Evaluation of Shannon Entropy and Local Estimations of Algorithmic
Complexity, Entropy 20(8), 605, 2018.

46. Hybrid Measure:
Shannon Entropy +
Local Algorithmic Complexity
Because clearly:
|K(U(p)=10111)| + |K(U(p)=10111)| >|K(U(p)=10111 2 times)|
H Zenil, F Soler-Toscano, N.A. Kiani, S. Hernández-Orozco, A. Rueda-Toicen
A Decomposition Method for Global Evaluation of Shannon Entropy and Local Estimations of Algorithmic
Complexity, Entropy 20(8), 605, 2018.

47. Hybrid Measure:
Shannon Entropy +
Local Algorithmic Complexity
H Zenil, F Soler-Toscano, N.A. Kiani, S. Hernández-Orozco, A. Rueda-Toicen
A Decomposition Method for Global Evaluation of Shannon Entropy and Local Estimations of Algorithmic
Complexity, Entropy 20(8), 605, 2018.

48. e.g. These 2 strings are of high Entropy (and high Entropy rate)
but low K:
010101110100, 011011010010
Gaining Causation Power
H Zenil, F Soler-Toscano, N.A. Kiani, S. Hernández-Orozco, A. Rueda-Toicen
A Decomposition Method for Global Evaluation of Shannon Entropy and Local Estimations of Algorithmic
Complexity, Entropy 20(8), 605, 2018.

49. Examples
H Zenil, F Soler-Toscano, N.A. Kiani, S. Hernández-Orozco, A. Rueda-Toicen
A Decomposition Method for Global Evaluation of Shannon Entropy and Local Estimations of Algorithmic
Complexity, Entropy 20(8), 605, 2018.

50. CTM and BDM conform with the
theoretical expectation
H Zenil, F Soler-Toscano, N.A. Kiani, S. Hernández-Orozco, A. Rueda-Toicen
A Decomposition Method for Global Evaluation of Shannon Entropy and Local Estimations of Algorithmic
Complexity, Entropy 20(8), 605, 2018.

51. Example of a generating
mechanism/model
https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0096223

52. Algorithmic Sequence Model
Sequential aggregation of small models produce a model able to explain larger
data. TMs: 2205 <> 1351 <> 1447 <> 599063 can produce:
S = 00000101111111111111111111110
+ +
+

53. The Online Algorithmic
Complexity Calculator
complexitycalculator.com

54. Version history and future
of the OACC
We keep expanding the calculator to wider
horizons of methodological and numerical capabilities:
• Version 1: Estimations of K for short binary strings
• Version 2: Expanded CTM and BDM capabilities for non-binary
strings and arrays
• Version 2.5: Estimations of Bennett's logical depth based on
CTM
• Version 3 (current version): Algorithmic Information dynamics
of strings, arrays and networks
• Version 4: Algorithmic complexity of models, algorithmic feature
selection, algorithmic dimensionality reduction and model
generator. (BETA version ready!)
complexitycalculator.com

55. We have seen how important AIT and
the role of Computability theory is in
the challenge of causality in science in
the 4th revolution of causal discover
towards model-driven approaches
https://www.youtube.com/watch?v=BEaXyDS_1Cw&t=30s

56. Graph and Tensor
Algorithmic
Complexity
Dr. Hector Zenil

61. A
B
C
Random versus simple 2D objects

63. Boundary Conditions
H Zenil, F Soler-Toscano, N.A. Kiani, S. Hernández-Orozco, A. Rueda-Toicen
A Decomposition Method for Global Evaluation of Shannon Entropy and Local Estimations of Algorithmic
Complexity, Entropy 20(8), 605, 2018.

70. Scatterplot of 2-dimensional CTM versus
1-dimensional CTM
https://peerj.com/articles/cs-23/

71. https://peerj.com/articles/cs-23/
ECAs sorted by
BDM

72. https://peerj.com/articles/cs-23/
BDM on CAs
compared to
compression:

77. Adding a causal dimension

78. Adding a causal dimension

79. Adding a causal dimension

80. Adding a causal dimension

81. Adding a causal dimension

82. Adding a causal dimension

83. Adding a causal dimension

84. Adding a causal dimension

85. Adding a causal dimension

86. Algorithmic
Information Dynamics
Dr. Hector Zenil

87. The 2D CA Game of Life (GoL)
Density diagram showing the
persistence of structures:
The Game of Life

88. The Game of Life Space-time
evolution
H. Zenil et al., Algorithmic-information Properties of Dynamic Persistent
Patterns and Colliding Particles in the Game of Life.
In A. Adamatzky (ed), From Parallel to Emergent Computing (book),
Taylor & Francis / CRC Press, 2018

89. The Game of Life Space-time
evolution
H. Zenil et al., Algorithmic-information Properties of Dynamic Persistent
Patterns and Colliding Particles in the Game of Life.
In A. Adamatzky (ed), From Parallel to Emergent Computing (book),
Taylor & Francis / CRC Press, 2018

90. GoL’s glider
(a ‘moving’ particle)
A piece of information moving through the grid from one side to another

91. Pattern algorithmic dynamics
GoL’s glider algorithmic dynamics
The amount of information is preserved,
neither lost or increased

92. Characterization of evolving
patterns
H. Zenil et al., Algorithmic-information Properties of Dynamic Persistent
Patterns and Colliding Particles in the Game of Life.
In A. Adamatzky (ed), From Parallel to Emergent Computing (book),
Taylor & Francis / CRC Press, 2018

93. Characterization of local dynamic patterns

94. Particle collision characterisation
Free particle 2-particle front collision
2-particle side collision 3-particle collision 4-particle side collision

95. Dynamics of a 4-particle
collision
New particles
Fixed point
H. Zenil et al., Algorithmic-information Properties of Dynamic Persistent Patterns and Colliding Particles in the Game
of Life. In A. Adamatzky (ed), From Parallel to Emergent Computin, Taylor & Francis / CRC, 2018

96. Algorithmic dynamics of a stable
‘near miss’
Zenil et al., Algorithmic-information Properties of Dynamic Persistent
Patterns and Colliding Particles in the Game of Life.
In A. Adamatzky (ed), From Parallel to Emergent Computing (book),
Taylor & Francis / CRC Press, 2018

97. Sensitivity of dynamics to initial conditions
Periodic fixed point
Zenil et al., Algorithmic-information Properties of Dynamic Persistent
Patterns and Colliding Particles in the Game of Life.
In A. Adamatzky (ed), From Parallel to Emergent Computing (book),
Taylor & Francis / CRC Press, 2018

98. Particle annihilation
H. Zenil, N.A. Kiani and J. Tegnér, Algorithmic-
information Properties of Dynamic Persistent
Patterns and Colliding Particles in the Game of
Life, In A. Adamatzky (ed), From Parallel to
Emergent Computing (book), Taylor & Francis /
CRC Press, 2018

99. Open-ended collision

100. All cases can be reduced to 4 density
plots (+ annihilation)
Algorithmicdynamics
ofglidercollisions
0 50 100 150 200
50
100
500
1000
BDM all cases

101. Moving
Objects Towards
& Away From
Randomness
Dr. Hector Zenil

102. A Causal Calculus Based on
Algorithmic Information Dynamics
Let S be a non-random binary file containing a string in binary:
S= 10101010101010101010101010101010101010101010101010101010101010
Clearly, S is algorithmically compressible, with pS = “Print(01) 31 times” a short
computer program generating S. Let S’ be equal to S except for a bitwise operation
(bitwise NOT, or complement) in, say, position 23:
S’ = 10101010101010101010100010101010101010101010101010101010101010
We have that the relationship of their algorithmic complexity denoted by C is:
C(S) < C(S’)
for any single-bit mutation of S, where C(S) = |pS| and C(S’) = |pS’| are the lengths
of the shortest computer programs generating S and S’.

103. The algorithmic information dynamics of a string (step 0) pushed towards and away
from randomness by digit removal using BDM.
The initial string consists of a simple segment of ten 1s followed by a random-looking
segment of 10 digits. The resulting strings mostly extract each of the respective
segments.
Moving a string

104. Neutral Deletion
Neutral digit deletion maximises the preservation of the elements contributing to
the algorithmic information content of the original string thus the most important
(computable) features (of which statistical regularities are a subset). Here applied
to a string (step 0) after
removal of 10 digits.

105. Random data is of
different nature
Now consider S to be a binary file of the same size but consisting of, say,
random data:
01101100100011001101010001110110001100011010011100010000110
So pS = Print(S) is the shortest possible generating program, then, in a
random object:
All perturbations are neutral!
H Zenil, N.A. Kiani, F. Marabita, Y. Deng, S. Elias, A. Schmidt, G. Ball, J. Tegnér,
An Algorithmic Information Calculus for Causal Discovery and Reprogramming
Systems, ​bioaRXiv DOI: https://doi.org/10.1101/185637

106. Identifying random objects by
the effect of perturbations
01101100100011001101010001110110001100011010011100010000110
H Zenil, N.A. Kiani, F. Marabita, Y. Deng, S.
Elias, A. Schmidt, G. Ball, J. Tegnér, An
Algorithmic Information Calculus for Causal
Discovery and Reprogramming
Systems, ​bioaRXiv DOI:
https://doi.org/10.1101/185637

107. The Principle of Algorithmic
Information Dynamics
C(rule 30)
C(rule 30 after 1 step) = C(rule 30 after 2 steps) + log(2) = …
C(rule 30 after 100 steps) + log(100) … = C(rule 30 after 1000 steps) + log(1000)
In a deterministic isolated dynamical system algorithmic complexity never
changes if it did, then it is neither deterministic or isolated and we can
identify those elements

108. Networks as programs
H Zenil, N.A. Kiani, F. Marabita, Y. Deng, S. Elias, A. Schmidt, G. Ball, J. Tegnér,
An Algorithmic Information Calculus for Causal Discovery and Reprogramming Systems, ​bioaRXiv DOI:
https://doi.org/10.1101/185637

109. Information Ranking and
Information Spectra
H Zenil, N.A. Kiani, F. Marabita, Y. Deng, S. Elias, A. Schmidt, G. Ball, J. Tegnér,
An Algorithmic Information Calculus for Causal Discovery and Reprogramming Systems, ​bioaRXiv DOI:
https://doi.org/10.1101/185637

110. Information Signatures
σ(G)
H Zenil, N.A. Kiani, F. Marabita, Y.
Deng, S. Elias, A. Schmidt, G. Ball,
J. Tegnér,
An Algorithmic Information Calculus
for Causal Discovery and
Reprogramming Systems, ​bioaRXiv
DOI: https://doi.org/10.1101/185637

111. H Zenil, N.A. Kiani, F. Marabita, Y. Deng, S. Elias, A. Schmidt, G. Ball, J. Tegnér,
An Algorithmic Information Calculus for Causal Discovery and Reprogramming Systems, ​bioaRXiv DOI:
https://doi.org/10.1101/185637

112. Moving Networks
H Zenil, N.A. Kiani, F. Marabita, Y. Deng, S. Elias, A. Schmidt, G. Ball, J. Tegnér,
An Algorithmic Information Calculus for Causal Discovery and Reprogramming Systems, ​bioaRXiv DOI:
https://doi.org/10.1101/185637

113. Reconstructing
Dynamical
Systems
Dr. Hector Zenil

114. AID Application to
Dynamical Systems
Original
candidate
model set M
with individual
models mi in M
After negative
perturbation:
candidate
models M’
grow in length (in
average):
Σi=1
|M’||m’i|>Σi=1
|M||mi|
Positive
perturbation:
candidate
models
decrease in length
Neutral
perturbation
|M| is the
cardinality of M
and |mi| the size
of model m in M.

115. Reconstructing space-time
dynamics of ECAs
original
reconstructed
H Zenil, N.A. Kiani, F. Marabita, Y. Deng, S. Elias, A. Schmidt, G. Ball, J. Tegnér,
An Algorithmic Information Calculus for Causal Discovery and Reprogramming Systems, ​bioaRXiv DOI:
https://doi.org/10.1101/185637

116. Reconstructing time in space-time
dynamics
We are generalizing these tools to continuous dynamical systems such as strange attractors
original
reconstructed
H Zenil, N.A. Kiani, F. Marabita, Y. Deng, S. Elias, A. Schmidt, G. Ball, J. Tegnér,
An Algorithmic Information Calculus for Causal Discovery and Reprogramming Systems, ​bioaRXiv DOI:
https://doi.org/10.1101/185637

117. Reconstructing space-phase dynamics of
ECA (more observations)
H Zenil, N.A. Kiani, F. Marabita, Y. Deng, S. Elias, A. Schmidt, G. Ball, J. Tegnér,
An Algorithmic Information Calculus for Causal Discovery and Reprogramming Systems, ​bioaRXiv DOI:
https://doi.org/10.1101/185637

118. Sensitivity to perturbations at
difference time steps
H Zenil, N.A. Kiani, F. Marabita, Y. Deng, S. Elias, A. Schmidt, G. Ball, J. Tegnér,
An Algorithmic Information Calculus for Causal Discovery and Reprogramming Systems, ​bioaRXiv DOI:
https://doi.org/10.1101/185637

119. System’s
Reprogrammability
Dr. Hector Zenil

120. The Observer Challenge

121. Evidence of pervasive
Reprogrammability
Reprogrammability of ECA and
CA
J. Riedel, H. Zenil
Cross-boundary Behavioural Reprogrammability Reveals Evidence of Pervasive Universality
International Journal of Unconventional Computing, vol 13:14-15 pp. 309-357

122. J. Riedel, H. Zenil
Cross-boundary Behavioural Reprogrammability Reveals Evidence of Pervasive Universality
International Journal of Unconventional Computing, vol 13:14-15 pp. 309-357
Reprogrammability Networks
Evidence of pervasive
Turing universality

123. J. Riedel and H. Zenil
Rule Primality, Minimal Generating Sets and Turing-Universality in the Causal Decomposition of Elementary Cellular
Automata
Journal of Cellular Automata, vol. 13, pp. 479–497, 2018
New Universal
Cellular Automata

124. Networks
Reprogrammability

125. An Algorithmic Information Calculus for Causal Discovery and Reprogramming Systems
H Zenil, N.A. Kiani, F. Marabita, Y. Deng, S. Elias, A. Schmidt, G. Ball, J. Tegnér
doi: https://doi.org/10.1101/185637
Algorithmic space Dynamical phase space
Algorithmic/Dynamic
Landscape Relationship

126. Information
signature of
complete graph
Information
signature of E-R
random graph
K(E-R)
~ |E(G)|K(k)
~ log|V(G)|
Reprogrammability Asymmetry

127. Reprogrammability Curve

128. Reprogrammability Space
Pr(G) x PA(G)