McCulloch&Pitts :
From biological to digital intelligence
In 1943, Warren McCulloch and Walter Pitts published A logical calculus of the ideas immanent in nervous activity [74], This article would later become the foundation of artificial intelligence, influencing as well computer science in general. The
article emerged from previous studies of the brain, as underlined here, and the increasing interest for mathematical models.
1. Summary of brain studies
​
In the beginning of the 20th century, as described on the page History, a lot of research had already been done in order to better understand the brain. Although no full understanding had been achieved, many principles and phenomena had already been discovered.
First, the brain was known to be composed of cells called neurons. These neurons are interconnected by chemical synapses. They transmit signals, specifically electrical impulses, ultimately controlling the organism. The all-or-none principle, namely studied by Adrian [5], showed a discrete aspect of neurons. Indeed, each neuron has some threshold, and it will trigger only if the incoming stimulus intensity exceeds this threshold. In 1937, Hodgkin [68] showed the existence of the summation principle. The idea is that sometimes, a neuron cannot be triggered by a single stimulus and needs the addition of several stimulus coming from different neurons. Finally, it was also shown that a neuron’s threshold could be temporarily or permanently modified by previous stimuli.
Even though biological knowledge was increasing rapidly, little was known about the more advanced activities of the brain, i.e. intelligence and learning. Research had already been done by Dusser de Barenne and Warren McCulloch concerning the localization of functions in the cerebral cortex ([30], [31]), but nothing that could have explained how intelligence could appear. Most studies were focused on pure biology, describing only phenomena in terms of continuous chemical reactions and physical, organic elements.
​
​
2. The rise of mathematical biophysics
​
In parallel with all these classic biology studies, a new discipline appeared, from the influence of researcher Nicolas Rashevsky. In the 1930s, he developed the fields of mathematical biology and mathematical biophysics, whose aim was to study biological behaviors theoretically, with the help of mathematics only. Rashevsky was actually criticized because he never did any experimental work, which was inconceivable for some people at the time [4].
More precisely, the goal of mathematical biophysics was to work on mathematical descriptions of highly idealized biological systems. Rashevsky wanted to adapt the philosophy of theoretical physics to the domain of biology. The idea was that the developed models often have practical applications despite their unrealistic idealizations, because the ideal model still has common properties with the real one [95]. By simplifying the problem, the solution can also be found step by step, approaching gradually the reality.
Rashevsky was interested in applying his discipline to the brain. In 1936, he published Mathematical biophysics and psychology [95]. In this paper, he tries to develop a mathematical model of neural networks, using a highly simplified model of a neuron. Starting with a series of assumptions, he then develops his theory by analyzing models that are in-creasingly complex. His paper has many similarities with the one published by McCulloch and Pitts seven years later.
The initial assumptions of Rashevsky were the following :
-
The brain is composed of neurons in contact with each other. They can be of two types, excitatory or inhibitory, and they produce a corresponding substance upon stimulation, influencing the stimulation of other connected neurons.
-
Neurons follow the all-or-nothing law : basically, the neuron will be excited only if the ratio e=i of the concentration of the incoming excitatory substance over the one of the inhibitory substance is greater than 1 (which somehow introduces the summation principle). Concentrations e and i are deduced from incoming signals by differential equations.
-
The threshold of a nerve is permanently lowered by previous excitation (this will allow learning).
​
Fig. 20 Rashevsky’s neuron model, as described in his paper Mathematical biophysics and psychology [95] in 1936.
Figure 20 shows an abstract representation of the model. From there, Rashevsky developed models based on more and more complex interconnections, analyzing the output in different cases and showing how complex behaviors could emerge from these networks. The most interesting part comes at the end where he manages to explain the emergence of conditional reflex (permanent modification of the behavior of the network after some stimuli). He further shows examples of ”retraction” after an incorrect action, which can be seen as the start of a basic learning process. He finally imagines how rational thinking can occur, i.e. the elimination of several possibilities at the same time, without trying them out. He even provides an expression of the upper limit of complexity of a problem which can be solved by pure reasoning, and demonstrates how a more complex problem requires longer ”thinking”.
Many achievements which, for some reasons that will be discussed later, did not cause a breakthrough as the paper of McCulloch and Pitts. It did not prevent Rashevsky to continue his research and to promote mathematical biophysics. This paper was indeed only one paper among many others published by Rashevsky and his students at that time. In 1939, he created the Bulletin of Mathematical Biophysics, a journal entirely dedicated to that field. It is only one year later, in 1940, that the young Walter Pitts joined his research group.
​
​
​
3. A word aboutWalter Pitts and Warren McCulloch
​
In 1940, Walter Pitts was only 17 and was considered a genius by his peers. At the age of 12, he spent three days in a library reading Principia Mathematica , a huge volume about mathematical logic, written by B. Russell and A. Whitehead [123]. After his reading, he sent a letter to Bertrand Russell, pointing out serious problems in the first volume. Russell replied with a very appreciative letter and invited Pitts to come in the UK to study. This propelled Pitts into the academic world where he met many important researchers, including Rashevsky in 1940. In his team, he worked on a mathematical analysis of excitatory and inhibitory activity in a simple neuron circuit [89]. In 1942, Pitts met Warren McCulloch and they became close friends.
Born in 1896, McCulloch had a more common career. In 1927, he obtained a master’s degree in philosophy and psychology and became a neurophysiologist. He performed a lot of research in neuropsychiatry and was driven by the will to explain the biology behind human thoughts.
At the time of his meeting with Pitts, McCulloch had already tried to describe the all-or-none activity of neurons using logical calculus, without success. He introduced his idea to Pitts who was a better mathematician than him. They were also
interested by Alan Turing’s recent theory of computation. They wondered if the nervous system could be conceived as a Turing machine [4]. These discussions led to a collaboration resulting in the famous ”A logical calculus of the ideas immanent in nervous activity ” [74], published in Rashevsky’s journal in 1943.
​
​
​
4. A logical calculus of the ideas immanent in nervous activity
​
As stated before, McCulloch and Pitts observed an equivalence between neurons which can be ”on” or ”off” (sending a signal or not) and propositions in propositional logic which can be ”true” or ”false”. It was this observation that made them
think that relations among neurons could be described mathematically as relations between propositions, and that the brain could be seen as a kind of universal computing device [4].
In the spirit of mathematical biophysics, McCulloch and Pitts were not concerned about having a realistic model, corresponding closely to real neural networks. Instead, they proposed a simplified version of neural networks based on logic, in order to demonstrate their computational power. This showed that computational theory could probably be used to study limitations of the real brain.
The model of McCulloch and Pitts was the following : each neuron follows the all-or-none principle and can thus be fired (1) or not (0). It receives binary inputs from other neurons, which can be excitatory or inhibitory (the output of a neuron
itself has no such distinction, it is only at the connection that the excitatory or inhibitory aspect is decided). It also has an integer threshold θ. The output will be 1 if and only if at least θ excitatory inputs are 1 and all inhibitory inputs are 0. Figure 21 shows an abstract representation of their neuron model.
​
​
Fig. 21 Neuron model of McCulloch and Pitts ,as defined in their paper in 1943.
Neurons are then connected together to form a network. A distinction is made between the input neurons (or peripheral afferent) which have no inputs and the inner neurons . Furthermore, time is discrete and neurons are updated simultaneously at each time step. Any signal takes one time step to go from the input to the output. Neural network outputs can then be formalized using temporal propositional expressions (TPE), i.e. restricted logical expressions combining outputs of other neurons at different time points.
The main part of the paper focuses on nets without circles (cycles). With TPEs, McCulloch and Pitts demonstrated what kind of input-output function their nerve nets can compute. They also showed how to build a nerve net that behaves in a
specified way, i.e. how to build a nerve net corresponding to a given TPE. Despite their idealizations, they showed that the behavior of more realistic models could be represented with simplified nerve nets. Thus, by adapting the network structure, it is possible to obtain relative inhibition (instead of full veto), variations of threshold when the neuron is fired, or permanently alterable synapses (which are necessary for learning).
The second part of the paper develops nets with circles but it is more obscure and less interesting.
Fig. 23 Examples of neural networks and their corresponding TPE [74]
It might be interesting to compare this model with the one of Rashevsky described here to see what made the model of McCulloch and Pitts successful. Both follow the all-or-none principle, a neuron having two output values chosen based on some threshold and the input values. Inputs can be excitatory or inhibitory in both cases. However, the first big difference is the switch from continuous to discrete (even binary) mathematics. Differential equations were replaced by much simpler logical propositions and the real values for inputs and outputs changed to binary values. This change from ”analog” to ”digital” was what made the model suitable for analysis with computational theory, which was very much related to intelligence as discussed namely by Turing. This also allowed other impacts on computer science that will be explained later.
Another difference, although less important, is that the excitatory or inhibitory nature of an output signal was fixed in Rashevsky’s model while it depends only on where the connection is made in McCulloch and Pitts’ model. This shows how they separated even further from reality than Rashevsky. This is important as their paper was a turning point from biological brain studies to more fictional nets studies, or in a way, from animal to artificial intelligence.
​
​
​
5. From brains to digital computers
​
McCulloch and Pitts’ theory had a huge impact, in several ways. First, their fictional nets inspired the creation of artificial neural networks, one of the first machine learning techniques, initiating at the same time the field of artificial intelligence. The evolution of their networks up to current deep learning techniques is a complex story that will be described here.
Their paper also influenced the work of Von Neumann when he designed computer architectures. The Von Neumann architecture marked the start of program based computers, and was thus were very important in the evolution of computer
science.
Finally, the article also interested Stephen Cole Kleene, an American mathematician who was the founder of recursion theory. This theory helps to solve computable function problems. He was one of Alonzo Church’s students along with Alan Turing. In 1951, he studied McCulloch and Pitts’ paper on nerve nets and produced a RAND report that had a significant influence on the study of automata [65]. In his paper, he makes the link between McCulloch and Pitts’ nerve nets and finite automata. Kleene defined different types of events, built from possible input combinations of a nerve net. Inputs correspond to the value of input neurons N1,...,Nk at different time steps. Figure 23 shows an example of such events. Kleene demonstrated how regular events in particular can be represented by nerve nets. Even more, he showed that the event consists of the firing or the non-firing of a single neuron at a time step ulterior to the ending time step of the event. In the second part of the paper, Kleene generalized the representation of events and showed that events can be represented by finite automata. These automata are still nowadays a very important concept in computation theory.
Fig. 23 Examples of event tables and their corresponding logical expressions, as used by Kleene in his paper [65]. Here, the values of two input neurons N1 and N2 at three different time steps are considered when building the events.
From these examples, it is clear that after McCulloch and Pitts, most related studies abandoned the analog world and switched to digital systems, influencing in multiple ways computer science. The later impacts will be discussed here.