At eNeuroLearn, we will occasionally blog on aspects of Neuroscience, AI, and matters in between. These blogs are aimed at practitioners of Machine Learning, Deep Learning, and AI.
This post briefly introduces some unique aspects of mammalian sensory systems that are relevant to AI. These include: 1) Hierarchy; 2) Feedback; 3) Generic architecture; and 4) Sensory integration. Of these, hierarchy and the generic architecture are used in deep learning (DL) architectures. We point out how feedback and sensory integration can create powerful DL systems.
1. Hierarchy: The mammalian sensory systems are deep and hierarchical. For example, Fig. 1 shows a schematic of the first several stages (levels) of the mammalian visual system. It has been referred to as “The Oil-Refinery Diagram”. Van Essen and his collaborators created this hierarchy after careful consideration of the complexity of “receptive fields” of typical neurons at each level. For example, the receptive fields of Retinal Ganglion Cells (RGC) and the Lateral Geniculate Nucleus (LGN) neurons are center-surround, with a core circular area that is excitatory and a circular surround area that is inhibitory. As we go up the hierarchy to V1, the neurons are characterized as simple and complex cells. These have elongated receptive fields with excitatory and inhibitory areas. People have even identified neurons in area MT that respond selectively to the infamous face of actress Jennifer Aniston.
Figure 2 shows the location of some of these centers on a brain outline along with the delays (earliest and average) in response to a visual stimulus. We can see that a visual stimulus produces a response in the LGN within 30 ms and a response in V1 within another 10 ms. A response from the fingertip can be in 180 ms. Most of these delays are at the synapses. For example, if there is a connection between a LGN neuron and a V1 neuron, the transmission through the synapse introduces a delay of about 10 ms. The delays also allow integration of information coming from other centers.
Figure 3 shows a comparison between the first few stages of the mammalian visual system and the typical receptive fields observed in a deep CNN (Convolutional Neural Network), the workhorse of today’s Deep Learning systems. The hierarchy of a CNN allows it to process visual data very efficiently.
2. Feedback: This is an aspect of the sensory systems very critical to brain function, but is universally neglected in Deep Learning architectures. Feedback allows information flow in the “backward direction”, i.e., from higher levels in a hierarchical system to lower levels.
Figure 2
For example, nearly all of the 187 links in the oil-refinery diagram are bidirectional, allowing information flow from higher to lower levels. In many cases, the number of neurons or the number of connections involved in the backward flow of information is ten-fold or more, compared to those involved in the forward flow. Then, we can genuinely ask, “which is the primary direction of information flow in mammalian sensory systems, top-down or bottom-up”? In subsequent blogs, we will discuss the role of feedback in perception.
Any system with feedback is dynamical, i.e., the mammalian sensory system is a dynamical system with all its benefits and drawbacks. The main benefit of a dynamical system is that it can process time-evolving input information without difficulty. Our sensory systems are continuously processing sensory information and making sense. In other words, the brain uses time as an information-bearing dimension and treats it with respect, i.e., real-time adjustments are common and implicitly expected.
Figure 3
3. Generic architecture: All primary sensory cortical areas (visual, auditory, somato-sensory, etc.) can be viewed as universal building blocks, capable of processing information in any sensory modality. These cortical areas, at least during early stages of development, are completely generic. For example, if the visual information during development is fed into the auditory cortex, the auditory cortex develops into a well-functioning visual cortex.
Figure 4
4. Sensory integration: Our senses do not operate in isolation, i.e., for visual perception, we integrate visual, auditory, tactile, and olfactory information. Sensory systems that operates in isolation are brittle. For example, see Fig. 4; even though they look like roses, none of them are. This becomes abundantly clear if we smell them; each flower has a distinct odor, different from that of a rose.
Actually, we can fool the brain and create interesting illusions like the McGurk effect. Here is how Wikipedia describes it: “The McGurk effect is a perceptual phenomenon that demonstrates an interaction between hearing and vision in speech perception. It occurs when the auditory component of one sound is paired with the visual component of another sound, leading to the perception of a third sound.”
Most vision systems in current generation autonomous vehicles use lidar and radar. But for robust perception of visual scenes, the vehicle should also be listening.
Finally, we illustrate some weaknesses of current vision systems, using an example from our own work on recognizing hand-written digits in the MNIST database. All humans immediately recognize the image in Fig. 5 below as the digit seven. But there is not a single Deep Learning system, ours included, that correctly recognizes this image as a seven.
Figure 5
In closing, we ask: “How we can make the current deep learning systems smarter?”. Is Deep Learning a “damsel in distress,” and is Neuroscience the “knight in shining armor?” Interested? Stay tuned for further blogs.
References:
[1] Figure 1 is taken from Felleman, D. J., & Van, D. E. (1991). Distributed hierarchical processing in the primate cerebral cortex. Cerebral cortex (New York, NY: 1991), 1(1), 1-47.
[2] The receptive field of a neuron is the pattern presented to the animal which elicits maximal response from it. If we view the neurons in the visual pathway as feature extractors or pattern analyzers, the complexity of the receptive field is an indicator of its positon in the hierarchy. This is evidence from Physiology. There is also evidence from Anatomy; Neuroscientists painstakingly trace which neuron is connected to which neuron and which area is connected to which area.
[3] Figure 2 is taken from Thorpe, S. J., & Fabre-Thorpe, M. (2001). Seeking categories in the brain. Science, 291(5502), 260-263.
[4] Figure 3 is taken from Yamins, D. L., & DiCarlo, J. J. (2016). Using goal-driven deep learning models to understand sensory cortex. Nature neuroscience, 19(3), 356.
[5] “Visual Projections Routed to the Auditory Pathway in Ferrets: Receptive Fields of Visual Neurons in Primary Auditory Cortex” Roe, Pallas, Kwon, Sur, J Neurosci, 12: 3651-3664, 1992
[6] en.wikipedia.org/wiki/McGurk_effect
[7] Figure 5 is from Zhao, Z., Kleinhans, A., Sandhu, G., Patel, I., & Unnikrishnan, K. P. (2019). Capsule Networks with Max-Min Normalization. arXiv preprint arXiv:1903.09662
[a]Not considered a good reference
[b]Can you add a sentence about the dynamic integration of the sensory systems with feedback and feed-forward intelligence creating clever or more intelligent and effective systems.
0 Comments