Scientific context
An important early contribution to technological improvement comes from the so called Second Industrial Revolution that was different in its character than the First Industrial Revolution. Starting with the invention of the steam engine, the First Industrial Revolution dates back to the 18th century, whereas Second Industrial Revolution is marked by further improvements in technology and production processes and is usually dated between 1870 and 1914 [77]. Main differences to the first revolution where that the technological improvements also drove much of the scientific research in maths and physics. In the mid 19th century, machines needed to be based on theoretical concepts to further improve their efficiency, but also to invent new devices. For example, the invention of the combustion engine as the successor of the steam engine did not happen accidentally, but was rather a result of a better understanding in thermodynamics. Some of those early developments in maths and physics that also influenced the work of Turing, Shannon, McCulloch and Pitts are presented here. Due to the chronological order, the concept of entropy in statistical mechanics and the development of quantum mechanics is outlined. Followed by this is the concept of computation theory and the works of Gödel and Neumann that played an important role.
Finally, it can be said that cybernetics is considered as a transdiscipline, namely a field that connects the researches coming from diverse branches. Surely, the most involved sciences in cybernetics are Physics and Biology which gave birth to Physiology. Those disciplines and their sub-branches can be considered as the ”steering wheel” of the study of the nerve net model conducted by McCulloch and Pitts which will be described in here.
​
​
​
1. Statistical mechanics and quantum physics
​
From the mid of the 19th century until the begin of the 20th century some important paradigm changes took place in the fields of physics. The first to be mentioned is statistical mechanics. It was developed from the need to describe a macroscopic state where the large number of degrees of freedom of the microscopic system does not allow a direct computation [47]. In this context the concept of uncertainty emerges as a powerful tool to fill the gap between the underlying physical principle and the imperfect knowledge about it. Ludwig Boltzmann was one of the first applying this
method to the broad field of gas dynamics. His underlying thought to make use of this statistical description was rooted in the belief that the macroscopic world consists of discrete particles, called “atoms”. The breakthrough was the description
of ”heat” as the “mean kinetic energy of particles” where Boltzmann derived the ideas from Maxwell and formulated the concept of entropy [23]. Furthermore, in the work Weitere Studien über das Wärmegleichgewicht unter Gasmolekülen (engl.: Further studies on the thermal equilibrium of gas molecules) he showed that this entity can only increase or remain at least constant (the famous H-theorem , where in the original work, E was used instead of H).
In this equation, f (x; t) is a particle distribution that is described by the Boltzmann Equation and is a function of kinetic energy x and time t . Gibbs took this equation to bring it in the more evident form
where pi describes the probability of a micro state i to occur and kB is the Boltzmann Constant . Figure 1 summarizes the different historical developments of the entropy equation.
Fig. 1 Sketch of the historical use and the evolution of the term ’entropy’
It is believed that Shannon developed his formula independently and called it entropy owing to an advice of John von Neumann (1). Quantum physics is another major breakthrough that changed the perception of the world fundamentally at the turn of the 20th century, as it made a transition from continuous to discrete physics. As the reader will notice, the duality between the continuous and the discrete (or the analogue and the digital) is central to the information age. The transition from statistical mechanics to quantum physics is fluent and it is also driven by Boltzmann to a big extend [23]. In his atomistic view, Boltzmann plays an important role in the development of the entropy equation. He first took the original
entropy formulation from Clausius to bring it into a probabilistic law, but also influenced strongly Max Planck in the evolution of quantum physics. As a direct result of Boltzmann’s work, Gibbs equation was later taken from John von Neumann in order to define the quantum entropy, where hi can be seen as the probability of quantum states (Figure 1). It has in many ways similar properties to Shannon’s entropy and in contrast to that, the properties of Boltzmann’s entropy equation exhibit most differences to both (see table in [12][p.303]). Today it is commonly accepted that Shannon’s entropy equation represents a more general case than Boltzmann’s entropy equation, while the latter one can be derived from the first one [59].
​
While at the end of the 19th century many branches of physics and engineering were rooted in classical and thus continuous mechanics, quantum mechanics assumed nature to be discrete in its character. This was the great breakthrough when Max Planck turned to the problem of black-body radiation in 1897 and formulated in the paper On the Theory of the Energy Distribution Law in the Normal Spectrum in the end of 1900 that radiative energy dissipation can only occur under discrete portions called ”quanta”. Almost all of his former studies were dedicated to the second law of thermodynamics and therefore entropy. His research and reasoning are thus clearly influenced by Boltzmann and his preceding work. Boltzmann was the first to perform vehemently the transition from a discrete description in favour of
a continuous one [66]. A similar approach is present in Shannon’s theory, where information is described in a discrete manner but also the all-or-none principle as it will be explained here.
​
​
​
2. Probability and computability - the mathematical background
​
A mentionable impact on the mathematics beforehand the work of the three authors described in this survey came from some Russian mathematicians at the end of the 19th century. Among them is a very modern and politically involved professor - Andrei Andreevich Markov. As a student of Chebyshev, Markov was introduced early to stochastic processes [45]. He continued working in the field and introduced a new sequence of dependent variables, nowadays called Markov chains . They must have had also an influence on the work of Shannon who used them in his trailblazing paper (see here), giving favour to the idea of describing information statistically rather than deterministically.
When tracing the major events in computation back in history, Gödel’s incompleteness theorems [46] play a pivoting role. Some of the very basic ideas can be traced back to Leibniz [29]. It was an astonishing outcome in the mathematical community in the early 1930's which set clear boundaries to the completeness and therefore on the inviolability of mathematics itself. Gödel showed that a mathematical set of axioms is either consistent (it has no contradictory properties) and inherently unprovable, or the system can be proved but is then inconsistent. In this work, the idea of general recursive functions was already created which Gödel presented widely in his lectures starting from 1934. Kleene [63] then took this idea and formulated a theory of general recursive functions which gives a basement for the development for functions that internally make use of themselves. Closely related to the incompleteness theorems of Gödel is the completeness theorem that (among others) led Hilbert to the formulation of the so called Entscheidung's problem (engl.: decision problem) which is very important for the idea of computation [27]. The decision problem asks whether an algorithm is able to answer a syntactic question wrong or correctly due to its formal logic. In other words, this problem can be stated like the following: “Can the proof of axioms be reduced to a mechanical sequence of simple (arithmetic) operations?”. Influenced by Gödel, Church and Turing gave two different approaches to the problem. While Church introduced the so called λ-calculus, Turing used the concept of the Turing Machine representing a fundamental model of computation (see [27]). Both mathematicians are said to be driven by the Gödel numbering [109] and came to the similar result that there are problems that can not be solved by any algorithm. However, the work on the problem led to the more curtailed idea of a computer, where a distinction of machine (hardware), program (software) and data was made [29].
After all, the basic mathematical ideas of recursive functions resulted successively in the development of the first computer. Although the mathematical questions were of theoretical nature, they shaped the concept of computation and quickly led to practical applications as they were used in the second world war [84].
​
​
3. Investigating the brain
​
Neuroscience, and in particular, electro-physiology is another fundamental root of the information age, that can be traced back to the early efforts in the 19th century to study information propagation in nerves. Electro-physiology is a pioneer discipline that investigates on electrical properties of biological cells and tissues. Furthermore, it focuses on electrical activity of neurons, and particularly action potential activity.
​
Early contributors include researchers such as Samuel Thomas von Sommerring, Karl Asmund Rudolphi, Johann Evangelista Purkinje, Ernst Heinrich Weber, Johannes Peter Muller, Friedrich Gustav Jackob Henle and Gabriel Gustav
Valentin [41].
Table 1 illustrates historical milestones in the development of the neuron doctrine.
Table 1 Historical milestones in the development of the neuron doctrine [41]
The transmission of a nerve signal through a neuron occurs as a result of electrical changes across the membrane of the neuron itself. Furthermore, it is known that the propagation of information along a neuron occurs upon excitation by the action current. Those concepts have been discussed, analysed and proved by A. L. Hodgkin in the paper ”Evidence for electrical transmission in nerve” [68], published on March 1937. From a biological point of view, a neuron is mainly characterized by an internal nucleon wrapped by a membrane to which several branches are attached. Those branches are called dendrites and they permit the link with another neuron and the communication through receptors and neurotransmitters. The information travels from one neuron to another along the axon. The membrane of an unstimulated
neuron is considered to be polarized, namely maintains a potential typically around -65 mV across the cellular membrane [87]. Here chemical particles (more precisely ions) come into play: in the intracellular part there is high concentration
of Anions (A-) and Potassium (K+) while outside the neuron the main ions are Sodium (Na+) and Chloride (Cl-) . This state can be also associated to homeostasis whereas the particles, without any incoming signals, preserve their status.
Whenever a signal is coming, an ion channel called ligand-gated opens to let (Na+) flow in or (K+) flow out. Walther Nernst, a German chemist, applied this concept in experiments to discover nervous excitability, and concluded that the local excitatory process through a semi-permeable membrane depends upon the ionic concentration. If a proper concentration of ions was reached, excitation would certainly occur. This was the basis for discovering the threshold value [104]. A few decades later, A. L.Hodgkin and A. F. Huxley described the conductance-based model, which explains
the ionic mechanisms underlying the initiation and propagation of action potentials [53]. The computation of the membrane potential can be done using the formula provided by D. E. Goldman, A. L. Hodgkin and B. Katz, the so called Goldman-Hodgkin-Katz equation which uses the concentration units of all the ions present in the membrane [88].
​
Fig. 2 Events that characterize the transmission of a nerve impulse [85]
The nerve impulse is also called an all-or-none reaction since there are no gradations between threshold potential and fully activated potential. The neuron is either at rest with a polarized membrane, or it is conducting a nerve impulse at reverse polarization. That concept was established by an American physiologist Henry Pickering Bowditch in 1871. He described the response of heart muscles and nerves to stimulations [22]. In 1914, E. D. Adrian, an English electro-physiologist, wrote a paper that gives confirmation of the all-or-none character of the nervous impulse through experiments. Adrian explains the relation between the stimulus and the propagated disturbance. First, he deducts from previous experiments that the size of the propagated disturbance in a nerve fiber depends on the local conditions only
and not on the past disturbances down the nerve. Then, his experiments showed that once the stimulus has reached a certain critical intensity, the nerve will be in a refractory state and the disturbance will be set in motion. Any further increase in intensity will not affect this disturbance. Therefore, the disturbance is independent of the strength of the stimulus [5]. This phenomenon is called the all-or-none principle. Stimuli of insufficient intensity fail to initiate a response whereas those higher than a threshold induce a high response followed by an undershoot.
​
This step response is naturally found in biological systems but today, it is also found in control theory and electronics.
1. [...] Von Neumann told me, “You should call it entropy, for two reasons. In the first place your uncertainty function has been used in statistical mechanics under that name, so it already has a name. In the second place, and more important, nobody knows what entropy really is, so in a debate you will always have the advantage.” [114]