Dopamine and temporal difference learning: A fruitful relationship between neuroscience and AI

Learning and motivation are driven by internal and external rewards. Many of our day-to-day behaviours are guided by predicting, or anticipating, whether a given action will result in a positive (that is, rewarding) outcome. The study of how organisms learn from experience to correctly anticipate rewards has been a productive research field for well over a century, since Ivan Pavlov’s seminal psychological work. In his most famous experiment, dogs were trained to expect food some time after a buzzer sounded. These dogs began salivating as soon as they heard the sound, before the food had arrived, indicating they’d learned to predict the reward. In the original experiment, Pavlov estimated the dogs’ anticipation by measuring the volume of saliva they produced. But in recent decades, scientists have begun to decipher the inner workings of how the brain learns these expectations. Meanwhile, in close contact with this study of reward learning in animals, computer scientists have developed algorithms for reinforcement learning in artificial systems. These algorithms enable AI systems to learn complex strategies without external instruction, guided instead by reward predictions.

The contribution of our new work, published in Nature (PDF), is finding that a recent development in computer science – which yields significant improvements in performance on reinforcement learning problems – may provide a deep, parsimonious explanation for several previously unexplained features of reward learning in the brain, and opens up new avenues of research into the brain’s dopamine system, with potential implications for learning and motivation disorders.

Autonomy: Differentiation, Agency, and Existence

“Would you like a little choice in your life?” ” Do you need to be able to decide for yourself?” “Do you love it when you get to follow your own emergence?” One of the needs that is most missing in our world, from childhood, through work, and on through old age, is the need for self-determination. What changes can we make so that this energy is more alive for us in our lives? What unconscious contracts can we release to make sure that we have a sense of freedom? And what is happening in the brain with the longing for choice, and what happens when that need is met? In this 90-minute session, we’ll learn a little about the brain, a little about unconscious contracts, and a lot about ourselves.

Human emotion and memory: interactions of the amygdala and hippocampal complex

The amygdala and hippocampal complex, two medial temporal lobe structures, are linked to two independent memory systems, each with unique characteristic functions. In emotional situations, these two systems interact in subtle but important ways. Specifically, the amygdala can modulate both the encoding and the storage of hippocampal-dependent memories. The hippocampal complex, by forming episodic representations of the emotional significance and interpretation of events, can influence the amygdala response when emotional stimuli are encountered. Although these are independent memory systems, they act in concert when emotion meets memory.

Amygdala-hippocampus dynamic interaction in relation to memory

Typically the term “memory” refers to the ability to consciously remember past experiences or previously learned information. This kind of memory is considered to be dependent upon the hippocampal system. However, our emotional state seems to considerably affect the way in which we retain information and the accuracy with which the retention occurs. The amygdala is the most notably involved brain structure in emotional responses and the formation of emotional memories. In this review we describe a system, composed of the amygdala and the hippocampus, that acts synergistically to form long-term memories of significantly emotional events. These brain structures are activated following an emotional event and cross-talk with each other in the process of consolidation. This dual activation of the amygdala and the hippocampus and the dynamics between them may be what gives emotionally based memories their uniqueness.


Four experiments are reported which examined memory capacity and retrieval speed for pictures and for words. Single-trial learning tasks were employed throughout, with memory performance assessed by forced-choice recognition, recall measures or choice reaction-time tasks. The main experimental findings were: (I) memory capacity, as a function of the amount of material presented, follows a general power law with a characteristic exponent for each task; (2) pictorial material obeys this power law and shows an overall superiority to verbal material. The capacity of recognition memory for pictures is almost limitless, when measured under appropriate conditions; (3) when the recognition task is made harder by using more alternatives, memory capacity stays constant and the superiority of pictures is maintained; (4) picture memory also exceeds verbal memory in terms of verbal recall; comparable recognition/recall ratios are obtained for pictures, words and nonsense syllables; (5) verbal memory shows a higher retrieval speed than picture memory, as inferred from reaction-time measures. Both types of material obey a power law, when reaction-time is measured for various sizes of learning set, and both show very rapid rates of memory search.

From a consideration of the experimental results and other data it is concluded that the superiority of the pictorial mode in recognition and free recall learning tasks is well established and cannot be attributed to methodological artifact.

Understanding Why You Are Who You Are with Dr. Stephen Porges and Luke Iorio

Who are you and what makes you whole? Do you have triggers such as sounds or scents that remind you of a familiar feeling? Why do you look at the world the way you do? So many of your questions can be answered by scientist and Professor of Psychiatry, Dr. Stephen Porges, whose current work is unlocking the ways we can see another side of ourselves–our truest form.

Stephen is a distinguished University scientist at Indiana University where he’s the Founding Director of the Traumatic Stress Research Consortium. He’s a Professor of Psychiatry at the University of North Carolina and Professor Emeritus at the University of Illinois at Chicago as well as the University of Maryland. Stephen has served as president of the Society for Psychophysiological Research and the Federation of Associations in Behavioral & Brain Sciences. He’s a former recipient of the National Institute of Mental Health Research Scientist Development Award.

Stephen has published more than 300 peer-reviewed articles and in 1994 he first proposed and pioneered the Polyvagal Theory, a theory that links the evolution of the mammalian autonomic nervous system to social behavior and emphasizes the importance of the physiological state in the expression of behavioral problems and psychiatric disorders.

Listen in and learn more about who you are–more now than ever!

Kay Takeaways:

Butterfly effect. Did you know we go through three evolutionary stages directly linked to those of vertebrates? To be clear, mammals are vertebrate and humans are mammals. The correlation between the two is eye-opening on how it affects your physiological state, let alone your health. Your THREE transformative stages are . . . [13:41].

Tranquility zone. Did you know that you have triggers that give you a sense of security? The sounds of certain music or someone’s voice are a couple of them. All your senses slowly drop, as do your defenses. There are simple ways to develop this sense of safety in ourselves and in others, start HERE . . . [25:34].

Connection boost. Did you know there’s no such thing as winning an argument? Once a person becomes physical–such as crossing their arms or retracting in some way–the conversation is over and therefore, the discussion is over. Winning isn’t an option. To strengthen any relationship–work, home, social–give these TWO evaluation methods a try . . . [39:44].

Tune in and turn the volume up for a dose of inspiration and life lessons. You’re never more than One Idea Away from a whole, new reality.


Luke Iorio is President of The Institute for Professional Excellence in Coaching (iPEC) and has graduated thousands of coaches, leaders, athletes, and professionals across 44 countries, all of whom share his vision and desire for expanding our human potential and creating lasting, conscious change. He has been quoted in The Huffington Post, Fox Business, and Next Avenue, and is currently taking to the airwaves on the One Idea Away Podcast to entertain life’s pivotal questions with the help of celebrated thought leaders, mentors, and everyday unsung heroes.

Cartographers of the Brain: Mapping the Connectome

Scientists are attempting to map the wiring of the nearly 100 billion neurons in the human brain. Are we close to uncovering the mysteries of the mind or are we only at the beginning of a new frontier?

PARTICIPANTS: Deanna Barch, Jeff Lichtman, Nim Tottenham, David Van Essen
MODERATOR: John Hockenberry
Original program date: JUNE 4, 2017


Imagine navigating the globe with a map that only sketched out the continents. That’s pretty much how neuroscientists have been operating for decades. But one of the most ambitious programs in all of neuroscience, the Human Connectome Project, has just yielded a “network map” that is shedding light on the intricate connectivity in the brain. Join leading neuroscientists and psychologists as they explore how the connectome promises to revolutionize treatments for psychiatric and neurological disorders, answer profound questions regarding the electrochemical roots of memory and behavior, and clarify the link between our upbringing and brain development.

Consilience: The Unity of Knowledge

One of our greatest living scientists–and the winner of two Pulitzer Prizes for On Human Nature and The Ants–gives us a work of visionary importance that may be the crowning achievement of his career. In Consilience (a word that originally meant “jumping together”), Edward O. Wilson renews the Enlightenment’s search for a unified theory of knowledge in disciplines that range from physics to biology, the social sciences and the humanities.

Using the natural sciences as his model, Wilson forges dramatic links between fields. He explores the chemistry of the mind and the genetic bases of culture. He postulates the biological principles underlying works of art from cave-drawings to Lolita. Presenting the latest findings in prose of wonderful clarity and oratorical eloquence, and synthesizing it into a dazzling whole, Consilience is science in the path-clearing traditions of Newton, Einstein, and Richard Feynman.

Prof. Robert Sapolsky – The Neuroscience Behind Behavior

Robert Sapolsky is an American neuroendocrinologist and author. He is currently a professor of biology, and professor of neurology and neurological sciences and, by courtesy, neurosurgery, at Stanford University.

Recorded: May 2017

Medical Neuroscience

Medical Neuroscience explores the functional organization and neurophysiology of the human central nervous system, while providing a neurobiological framework for understanding human behavior. In this course, you will discover the organization of the neural systems in the brain and spinal cord that mediate sensation, motivate bodily action, and integrate sensorimotor signals with memory, emotion and related faculties of cognition. The overall goal of this course is to provide the foundation for understanding the impairments of sensation, action and cognition that accompany injury, disease or dysfunction in the central nervous system. The course will build upon knowledge acquired through prior studies of cell and molecular biology, general physiology and human anatomy, as we focus primarily on the central nervous system.

This online course is designed to include all of the core concepts in neurophysiology and clinical neuroanatomy that would be presented in most first-year neuroscience courses in schools of medicine. However, there are some topics (e.g., biological psychiatry) and several learning experiences (e.g., hands-on brain dissection) that we provide in the corresponding course offered in the Duke University School of Medicine on campus that we are not attempting to reproduce in Medical Neuroscience online. Nevertheless, our aim is to faithfully present in scope and rigor a medical school caliber course experience.

This course comprises six units of content organized into 12 weeks, with an additional week for a comprehensive final exam:
– Unit 1 Neuroanatomy (weeks 1-2). This unit covers the surface anatomy of the human brain, its internal structure, and the overall organization of sensory and motor systems in the brainstem and spinal cord.
– Unit 2 Neural signaling (weeks 3-4). This unit addresses the fundamental mechanisms of neuronal excitability, signal generation and propagation, synaptic transmission, post synaptic mechanisms of signal integration, and neural plasticity.
– Unit 3 Sensory systems (weeks 5-7). Here, you will learn the overall organization and function of the sensory systems that contribute to our sense of self relative to the world around us: somatic sensory systems, proprioception, vision, audition, and balance senses.
– Unit 4 Motor systems (weeks 8-9). In this unit, we will examine the organization and function of the brain and spinal mechanisms that govern bodily movement.
– Unit 5 Brain Development (week 10). Next, we turn our attention to the neurobiological mechanisms for building the nervous system in embryonic development and in early postnatal life; we will also consider how the brain changes across the lifespan.
– Unit 6 Cognition (weeks 11-12). The course concludes with a survey of the association systems of the cerebral hemispheres, with an emphasis on cortical networks that integrate perception, memory and emotion in organizing behavior and planning for the future; we will also consider brain systems for maintaining homeostasis and regulating brain state.