What is the brain’s visual system?

Of the seven main brain networks, the visual system’s name best describes its function as “the observer.”

The visual system controls sight and visual processing, making the system simple to understand even though it handles complicated tasks.

It is a highly complex series of areas that work toward the common goal of turning light into something recognizable. Though at first it appears to be a single region of the brain, the visual system is made up of multiple, neighboring functional areas that work collectively for cohesive visual processing.

Together, they:

    • Engage in visual processing
    • Analyzing motion
    • Recognize patterns, textures, and faces
    • Identify the location of objects, buildings, or people in space
    • Determining the function and permanence of objects
    • Aid in problem solving
    • Reinforce visually learned behaviors

Today, modern neuroscience continues to explore how the visual system makes sense of the world around us.

Yet, there are many capabilities still to be studied, as our understanding of this complex system continually expands.


The visual system’s complex roles in the brain

The visual system can quickly identify visual changes, consistencies, or inconsistencies that aid in problem solving and evaluating the world around us.

To break this down, take a look at the following pattern-based question. Which image comes next in the sequence?


Figure 1. Recognizing visual patterns (image: Brilliant)

This problem could be solved mathematically by assigning each vertex of the pentagon a numerical value and determining the relationship between vertex 1 and 4, 2 and 5, 3 and 1, etc.

Without mathematics, the problem can be visually solved by recognizing that the arrow moves one vertex clockwise in each sequential image. This indicates that the next image would be image B.

The visual system is responsible for making sense of what we see, but it can also be tricked based on what it thinks it perceives — giving us optical illusions. 

This optical illusion plays with the visual system’s ability to process light, dark, and linear elements. 


Figure 2. Line optical illusions (image: Brilliant)

At first glance, the grey lines in this optical illusion appear to be slanted, but when you look more closely, you can see that the grey lines are parallel and the squares are equal size. (If your eyes won’t stop playing tricks on you, cover all but the top row with a piece of paper and compare the size of the squares.)

With the squares shifted off-center to the left or right, they give a warped perception that the grey lines are changing in perspective or growing in size. This is due to our visual system’s ability to process depth perception1 using size and perspective cues.2 In these examples, and in most of our daily visual processing, the visual system analyzes images and movement to make sense of what we see. This can be applied to learning movement behaviors, understanding maps, reading written text,3 and the recognition of facial expressions.4 Each of these functions is based on patterns that can be identified in a single moment or over the course of an individual's learned experience.

The visual system’s pattern recognition capabilities have also become a key basis for creating computer-based neural networks. Developers model machine learning algorithms after human visual processes to design computer-based image recognition or object identification (more on this later).

To understand how the visual system can be leveraged for computer technology, it’s important to first look at how the system transfers information from the eyes throughout the occipital lobe.

Discovery of the visual system and its cortical organization

The visual system is one of the older brain networks, having been studied since before the turn of the 20th century. German physiologist, Hermann Munk, identified the occipital cortex as central to vision capabilities in 1881 through experiments on monkeys and dogs,5 and he first determined a connection between the eyes and the rear of the brain.

The Russo-Japanese War and World War I later led to increased awareness of the visual cortex as military doctors studied visual impairments — including blindness — in soldiers who had suffered rear-brain injuries.6 Japanese physician, Tatsuji Inouye, was the first to develop retinotopic maps of the visual cortex,7 which were reinforced by subsequent wartime studies.8,9

Using this foundation, neuroscience has continued to map the relationship between the eyes, the visual pathway, and the visual cortex. The visual pathway extends from the retina through the optic nerves10 and the lateral geniculate nuclei (LGN) to the visual cortex. Within the visual cortex, information is then processed through a series of closely connected — but functionally selective11 — nodes in the brain.

Each of these nodes are functionally and structurally connected (see Connectomics) as distributed networks that make up the dorsal and ventral pathways.

Figure 3. Parallel visual pathways in humans11

Hierarchy of functional areas in the dorsal and ventral visual pathways

The functional nodes of the visual system have a hierarchical organization for how they distribute and process information through the dorsal and ventral visual pathways.

Upon receiving visual information from the retina, imagery is initially processed in the primary node of the visual cortex, V1. It is then passed through the connected nodes V2 and V3, and through the dorsal or ventral visual pathways where task segregation occurs in highly specialized areas12 of the visual cortex.

The dorsal visual pathway is responsible for the perception of objects,13 global motion,14 and processing visual stimuli15 for a coordinated response. At the same time, the ventral visual pathway is thought to be involved in the recognition of faces, places,16 bodies,17 shapes, letters,18 and words.19

The following chart outlines how visual information can be initially processed in the primary nodes of the visual cortex, before being deployed through the dorsal and ventral pathways to highly specialized areas of the visual system.12


Figure 4. Hierarchy of visual processing in ventral and dorsal streams20

As an example, when you look at a person’s face, both the dorsal and ventral visual pathways are thought to be involved in the initial stages of processing.21 The specialized recognition of facial features through spatial relationships then occurs later in the dorsal visual pathway with strong functional connectivity to the fusiform face area (FFA).22

The hierarchy of the visual system shows us that visual neural networks are closely connected, highly specialized, and synchronized.

Yet, just because the visual cortex appears relatively self-contained anatomically — unlike the default mode network (DMN), for example, which can be found in multiple separate areas in each hemisphere — it still interacts frequently with other networks.

Network integration and impact on visual and mental health

The visual system works closely with many of the brain’s networks:

These are just some of the ways the visual system primarily partners with other networks to carry out complex functions. However, there are also many occasions where this coordination breaks down causing visual impairments or mental health issues.

Many individuals find it difficult to stay focused while reading, and this may correlate with a decoupling of the visual system and the default mode network.29 It has been suggested that stronger connectivity with the DMN is linked to better text processing,29 otherwise it is easy for our minds to wander when we are reading.

Likewise, increased asynchrony between the visual system and the sensorimotor network has been shown to correlate with mild to severe impairment in children with autism spectrum disorder (ASD).30 This impairment likely impacts motor and social skill acquisition. Evaluating brain biomarkers based on the overconnectivity or underconnectivity31 of networks may present an opportunity for identifying and diagnosing ASD.32

Decreased connectivity in the visual system can also be present in individuals with schizophrenia. Diminished connectivity has been shown to correlate with the severity of psychotic symptoms,33 and it may lead to impaired visual object perception.33

Studying changes in functional connectivity of the visual system is also relevant to vision disorders. In patients with primary open angle glaucoma (POAG) which causes blindness, there are not only structural abnormalities34 within the eyes. Atrophy also occurs in the visual cortex32 as well distant gray matter areas of the brain — including decreased functional connectivity in working memory and the dorsal attention network (DAN).34 This degradation suggests that POAG could be considered a neurodegenerative condition.

While studies of the visual system are providing ongoing insight into mental illnesses and neurological conditions, understanding the system’s functional connectivity has also led to advances in computer technology.

From the human brain to the neural network

Moving beyond our understanding of the brain, the visual system’s functional connectivity has become a model for computer-based neural networks. A neural network is a series of algorithms that can be trained to look for visual patterns in large datasets.35 Using computer algorithms, we can recreate cognitive processes36 like visual perception because neural networks mimic the hierarchical organization that occurs in the visual pathways.

Furthermore, the dorsal and ventral pathways present a useful model for close-proximity, small-world networks.37 When developing computer-based information processing systems, the anatomically localized connections between the primary visual nodes and the highly specialized areas of the visual system are a key example for short-range connections in neural networks.

Deep learning, neural networks can then be leveraged as predictive models for image recognition38 by comparing objects, faces, places, and other key tasks. Models can also be further trained to recognize patterns and movement.39

Computationally modeling neural networks poses two unique opportunities:

  1. It gives the ability to use machine learning algorithms to process information in a similar method as a human viewer
  2. It offers better possibilities for understanding the human brain.

If we can correctly model the brain’s intricate functions, we can use computer technology to dive deeper into how the brain operates when it is healthy versus when it experiences functional problems. As neuroscience continues to study the abnormalities resulting in physical or mental health disorders, we are also paving the way for discovering possible brain-focused treatments.

The visual system has led to incredible technological advances, but other networks have also posed new opportunities for the field of neuroscience.

Expand for full list of references
  1. Poggio GF, Gonzalez F, Krause F. Stereoscopic mechanisms in monkey visual cortex: binocular correlation and disparity selectivity. J Neurosci. 1988;8(12):4531-4550. doi:10.1523/JNEUROSCI.08-12-04531.1988
  2. Kalloniatis M, Luu C. The perception of depth. In: Kolb H, Fernandez E, Nelson R, eds. Webvision: The Organization of the Retina and Visual System [Internet]. Salt Lake City, Utah: University of Utah Health Sciences Center; 1995. https://www.ncbi.nlm.nih.gov/books/NBK11512/. Published May 1, 2005. Updated June 6, 2007. Accessed January 15, 2019.
  3. Sajedinia Z, Pizlo Z, Hélie S. Investigating the role of the visual system in solving the traveling salesperson problem. Proceedings of the 41st Annual Meeting of the Cognitive Science Society. Retrieved from https://par.nsf.gov/biblio/10177542. Accessed January 14, 2021.
  4. Bayet, Laurie. The development of facial expressions perception. Psychology. Université Grenoble Alpes; 2015. English. NNT : 2015GREAS049ff. Tel-01475668v2f.
  5. Munk H. Uber die funktionen der grosshirnrinde. A. Hirschwald, Berlin. 1881.
  6. Lanska DJ. Elaboration of the Visual Pathways from the Study of War-Related Cranial Injuries: The Period from the Russo-Japanese War to World War I. In: Tatu L, Bogousslavsky J, eds. War Neurology. Front Neurol Neurosci. Basel, Switzerland: Karger; 2016;38:31-42. doi.org/10.1159/000442567
  7. Glickstein M. The discovery of the visual cortex. Sci Am. 1988;259(3):118-127. http://www.jstor.org/stable/24989235. Accessed January 4, 2021.
  8. Holmes G. DISTURBANCES OF VISION BY CEREBRAL LESIONS. Br J Ophthalmol. 1918;2(7):353-384. doi.org/10.1136/bjo.2.7.353
  9. Benson NC, Butt OH, Datta R, Radoeva PD, Brainard DH, Aguirre GK. The retinotopic organization of striate cortex is well predicted by surface topology. Curr Biol. 2012;22(21):2081-2085. doi.org/10.1016/j.cub.2012.09.014
  10. Crumble L. Visual pathway. Salvador F, ed. Kenhub website.https://www.kenhub.com/en/library/anatomy/the-visual-pathway. Updated November 13, 2020. Accessed January 14, 2021.
  11. Yamasaki T, Tobimatsu S. Parallel visual pathways in humans. In: Driving ability in the Alzheimer disease spectrum: Neural basis, assessment, and potential use of optic flow event-related potentials. Front Neurol. 2018;9(750). doi:10.3389/fneur.2018.00750
  12. Kanwisher N. Functional imaging of the human brain: A window into the architecture of the mind. Lecture presented at CNS 2020 Virtual; May 5, 2020.
  13. Freud E, Plaut DC, Behrmann M. ‘What’ is happening in the dorsal visual pathway. Trends Cogn Sci. 2016;20(10):773-784. doi.org/10.1016/j.tics.2016.08.003
  14. Pellicano E, Gibson LY Investigating the functional integrity of the dorsal visual pathway in autism and dyslexia. Neuropsychologia. 2008;46(10):2593-2596. doi.org/10.1016/j.neuropsychologia.2008.04.008
  15. Siegel M, Donner TH, Oostenveld R, Fries P, Engel AK. Neuronal synchronization along the dorsal visual pathway reflects the focus of spatial attention. Neuron. 2008;60(4):709-719. doi.org/10.1016/j.neuron.2008.09.010
  16. Kanwisher N. Functional specificity in the human brain: A window into the functional architecture of the mind. Proc Natl Acad Sci U S A. 2010;107(25):11163-11170. doi:10.1073/pnas.1005062107
  17. Schwarzlose RF, Baker CI, Kanwisher N. Separate face and body selectivity on the fusiform gyrus. J Neurosci. 2005;25(47):11055-11059. doi:10.1523/JNEUROSCI.2621-05.2005
  18. Kanwisher N, Dilks D. The Functional Organization of the Ventral Visual Pathway in Humans. In: Chalupa L, Werner J, eds. The New Visual Neurosciences. Cambridge, Mass: MIT Press; 2013. https://citeseerx.ist.psu.edu/viewdoc/download?doi= Accessed January 27, 2019.
  19. Ishai A, Ungerleider LG, Martin A, Schouten JL, Haxby JV. Distributed representation of objects in the human ventral visual pathway. Proc Natl Acad Sci U S A. 1999;96(16):9379-9384. doi.org/10.1073/pnas.96.16.9379
  20. Perry C, Fallah, M. Hierarchy of visual processing in ventral and dorsal streams. In: Feature integration and object representations along the dorsal stream visual hierarchy. Front Comput Neurosci. 2014;8(84). doi:10.3389/fncom.2014.00084
  21. Rosazza C, Cai Q, Minati L, Paulignan Y, Nazir TA. Early involvement of dorsal and ventral pathways in visual word recognition: An ERP study. Brain Res. 2009;1272:32-44. doi:10.1016/j.brainres.2009.03.033
  22. Zachariou V, Nikas CV, Safiullah ZN, Gotts SJ, Ungerleider LG. Spatial mechanisms within the dorsal visual pathway contribute to the configural processing of faces. Cereb Cortex. 2017;27(8):4124-4138. doi:10.1093/cercor/bhw224
  23. Vogel AC, Miezin FM, Petersen SE, Schlaggar BL. The putative visual word form area is functionally connected to the dorsal attention network. Cereb Cortex. 2012;22(3):537-549. doi.org/10.1093/cercor/bhr100
  24. Vogel A, Petersen S, Schlaggar B. The VWFA: it's not just for words anymore. Front Hum Neurosci. 2014;8:88. doi:10.3389/fnhum.2014.00088
  25. Sale A, Putignano E, Cancedda L, et al. Enriched environment and acceleration of visual system development. Neuropharmacology. 2004;47(5):649-660. doi.org/10.1016/j.neuropharm.2004.07.008
  26. Iacoboni M. Neurobiology of imitation. Curr Opin Neurobiol. 2009;19(6):661-665. doi.org/10.1016/j.conb.2009.09.008
  27. Steimke R, Nomi JS, Calhoun VD et al. Salience network dynamics underlying successful resistance of temptation. Soc Cogn Affect Neurosci. 2017;12(12):1928-1939. doi.org/10.1093/scan/nsx123
  28. Bilevicius E, Kolesar TA, Smith SD, Trapnell PD, Kornelsen J. Trait emotional empathy and resting state functional connectivity in default mode, salience, and central executive networks. Brain Sci. 2018; 8(7):128. https://doi.org/10.3390/brainsci8070128
  29. Zhang M, Savill N, Margulies DS. et al. Distinct individual differences in default mode network connectivity relate to off-task thought and text memory during reading. Sci Rep. 2019;9:16220. doi.org/10.1038/s41598-019-52674-9
  30. Nebel MB, Eloyan A, Nettles CA. Intrinsic visual-motor synchrony correlates with social deficits in autism. Biol Psychiatry. 2016;79(8):633-641. doi:10.1016/j.biopsych.2015.08.029
  31. Keown CL, Shih P, Nair A, Peterson N, Mulvey ME, Müller R-A. Local functional overconnectivity in posterior brain regions is associated with symptom severity in autism spectrum disorders. Cell Rep. 2013;5(3):567-572. doi.org/10.1016/j.celrep.2013.10.003
  32. Chen CP, Keown CL, Jahedi A, et al. Diagnostic classification of intrinsic functional connectivity highlights somatosensory, default mode, and visual regions in autism, Neuroimage Clin. 2015;8:238-245. doi.org/10.1016/j.nicl.2015.04.002
  33. van de Ven V, Jagiela AR, Oertel-Knöchel V, Linden DEJ. Reduced intrinsic visual cortical connectivity is associated with impaired perceptual closure in schizophrenia. Neuroimage Clin. 2017;15:45-52. doi.org/10.1016/j.nicl.2017.04.012
  34. Frezzotti P, Giorgio A, Motolese I. Structural and functional brain changes beyond visual system in patients with advanced glaucoma. PLoS One. 2014. doi.org/10.1371/journal.pone.0105931
  35. Hardesty L. Explained: Neural networks. MIT News website. https://news.mit.edu/2017/explained-neural-networks-deep-learning-0414. Published April 14, 2017. Accessed January 21, 2021
  36. Yang YL, Deng HX, Xing GY, Xia XL, Li HF. Brain functional network connectivity based on a visual task: visual information processing-related brain regions are significantly activated in the task state. Neural Regen Res. 2015;10(2):298-307. doi:10.4103/1673-5374.152386
  37. Bassett DS, Bullmore E. Small-World Brain Networks. Neuroscientist. 2006;12(6):512-523. doi:10.1177/1073858406293182
  38. Eickenberg M, Gramfort A, Varoquaux G, Thirion B. Seeing it all: Convolutional network layers map the function of the human visual system. Neuroimage. 2017;152:184-194. doi.org/10.1016/j.neuroimage.2016.10.001
  39. George D, Hawkins J. A hierarchical Bayesian model of invariant pattern recognition in the visual cortex. Proc Int Jt Conf Neural Netw. 2005;3:1812-1817. doi:10.1109/IJCNN.2005.1556155