Learning by association: the Hebb's rule.

Learning by associations

Children learn quickly by making associations. Then they remember through those associations. A child, even a small child who may not yet been able to walk, can recognise a chair whether it is large or small or regardless of its colour. Similarly, people will be able to identify a chair even from a bad drawing. And they will recognise (remember) people even when they are wearing a hat covering their head or a scarf covering their mouth.

How do humans (and animals) recognise and remember objects even when only partially visible? Why do we suddenly remember past events or people when we find ourselves in the locations where these events occurred? (This is also called involuntary memory, term coined by Marcel Proust, author of À la recherche du temps perdu).

Everybody can recognise a chair from a simple quick sketch

Learning by making connections

Neural nets, artificial and biological, are made of neurons.

Very schematically speaking, a neuron comprises several inputs (dendrites) a soma (the actual cell body) and an axon (through which it sends out the output). In artificial neural networks neurons get their inputs from other neurons, process the incoming information and then output the result to other neurons for further information processing.

Neurons are connected to thousands of other neurons, but there is a difference - not all those connections are equal.

In artificial neural networks, neurons are connected by weights that are numerical parameters measuring the “strength” of the connection. When a neuron sends the information on to other neurons, it is said to “fire”.

Neurons fire and transmit information to new neurons via connections determined by these weights. The weights constitute the sole variables of the network, driving its learning process. They dictate the strength of the input signals, and in a system where neuron activity is governed by a function operating on a fixed threshold, high positive weights for connections tend to synchronize firing among neurons, even with weaker inputs.

When we say neurons fire together, it means that if one neuron fires and the weight of its connection to another neuron is high (indicating a strong connection), then the other neuron will also fire.

Donald Hebb was a Canadian psychologist, who lived during the 20th century, and who, in 1949, in his The organization of behavior, proposed a fundamental principle known as the Hebb’s rule, suggesting that when neurons fire together their connection strengthens; conversely when they do not fire together, their connection weakens.

More precisely, he writes: “When an axon of cell A is near enough to excite a cell B and repeatedly or persistently takes part in firing it, some growth process or metabolic change takes place in one or both cells such that A's efficiency, as one of the cells firing B, is increased”. This is sometimes succinctly summarised with ‘what fires together, wires together’.

Involuntary Memory

Biological neural networks, such as our brains, exhibit the phenomenon of creating stronger connections in response to intense emotional experiences. For instance, when we find ourselves in a novel environment accompanied by someone to whom we have a deep emotional attachment, the neural activity triggered by the surroundings may be linked to the neuronal activity triggered by that individual. Upon revisiting the same location, even years later, the neurons associated with that place are reactivated. Due to the robust connection created in the past, the neurons associated with the person also tend to be activated, even if the individual is not physically present at the time. Being in the same place where we were with someone may make us remember that person. This interconnected neural activation underscores the enduring influence of strong emotional bonds on memory and cognition.

Proust did not have a psychological background, he was a writer. However, in his novel, when the protagonist eats a tea-soaked madeleine, a long-forgotten childhood memory of eating tea-soaked madeleines with his aunt is restored to him. Neurons that had fired together and created strong connections were reawakened when only some of them were reactivated. From this memory, he then proceeds to recall the childhood home he was in and even the town itself. In fact, chaining is not rare when we have involuntary memories. A set of neurons may trigger another set of neurons which, in turn, will trigger another set of neurons and so on. Each set of neurons may represent an image, a place, a person, or any other memory.

When we see a familiar face, our brain processes various facial features such as the eyes, nose, and mouth, leading to the firing of neurons associated with each of these features. This simultaneous activation creates strong connections between these neurons. Consequently, upon encountering the same face again, even if certain parts are obscured by a hat or a scarf, the neurons linked to the visible features will fire and subsequently trigger those associated with the concealed portions of the face. Despite only perceiving a portion of the face, internally, within our brain, all the neurons connected to the different facial features will still activate together, effectively reconstructing the complete facial image. This process underscores that our brain does not merely perceive reality but actively reconstructs it based on neural connections and associations.

In the context of artificial neural networks, this characteristic offers a significant advantage: robustness. Human vision demonstrates an ability to recognise objects even when the view is partially obscured. We can identify individuals even when they might be wearing hats or scarves that cover parts of their face. We are not sensitive to small image noise and exhibit some tolerance.

Similarly, when employing artificial neural networks for image recognition, this robustness is mirrored. Even if we slightly alter an image, such as by modifying a person's mouth by a few pixels, the signals from the remaining pixels are typically strong enough for recognition to occur. Consequently, this system displays a resistance to noise and possesses the capability to make auto-corrections.

Memory and automatisms

Involuntary memory can be viewed as a consequence of Hebb's rule, but this rule also facilitates the creation of automatisms by synchronising the firing of all the correct neurons required for various tasks, such as playing tennis or driving a car. As a result, we don't need to consciously think through each individual movement; instead, everything becomes natural and flows seamlessly without the need for deliberate cognitive effort.

Neural networks demonstrate remarkable robustness to noise and the ability to learn through association. This biological advantage is tremendously helpful—for instance, when traversing an area where predators may be hidden, it is evolutionarily advantageous to remember this fact each time we enter that location. In addition, this capability may conserve resources: while it may not be necessary to recognise some persons we have encountered just once before in a completely different context, it would be beneficial to remember them in the same context if they had previously been helpful or, conversely, particularly hostile.

In conclusion, the Hebb's rule stands as a foundational principle in understanding the dynamics of neural networks, both biological and artificial. Through the concept of "what fires together, wires together," the Hebb's rule encapsulates the essence of associative learning and memory formation. This principle not only elucidates how neurons forge connections based on correlated activity but also sheds light on the development of automatisms and the robustness of memory recall.