It’s receiving global media coverage, so you may already have heard about this, but the news from researchers at University of California, Berkeley is that neuroscientists may one day be able to hear the imagined speech of a patient unable to speak due to stroke or paralysis.
Frequency spectrograms of the actual spoken words and the sounds as reconstructed by two separate models based solely on recorded temporal lobe activity in a volunteer subject. The words – Waldo, structure, doubt and property – are more or less recognizable, even though the model had never encountered these specific words before.
These scientists have succeeded in decoding electrical activity in the brain’s temporal lobe – the seat of the auditory system – as a person listens to normal conversation. Based on this correlation between sound and brain activity, they then were able to predict the words the person had heard solely from the temporal lobe activity.
“This research is based on sounds a person actually hears, but to use it for reconstructing imagined conversations, these principles would have to apply to someone’s internal verbalizations,” cautioned first author Brian N. Pasley, a post-doctoral researcher in the center. “There is some evidence that hearing the sound and imagining the sound activate similar areas of the brain. If you can understand the relationship well enough between the brain recordings and sound, you could either synthesize the actual sound a person is thinking, or just write out the words with a type of interface device.”
“This is huge for patients who have damage to their speech mechanisms because of a stroke or Lou Gehrig’s disease and can’t speak,” said co-author Robert Knight, a UC Berkeley professor of psychology and neuroscience. “If you could eventually reconstruct imagined conversations from brain activity, thousands of people could benefit.”
In addition to the potential for expanding the communication ability of the severely disabled, he noted, the research also “is telling us a lot about how the brain in normal people represents and processes speech sounds.”
People undergoing brain surgery participated in the study, to determine the location of intractable seizures so that the area can be removed in a second surgery. Neurosurgeons typically cut a hole in the skull and safely place electrodes on the brain surface or cortex – in this case, up to 256 electrodes covering the temporal lobe – to record activity over a period of a week to pinpoint the seizures. For this study, 15 neurosurgical patients volunteered to participate.
Pasley visited each person in the hospital to record the brain activity detected by the electrodes as they heard 5-10 minutes of conversation. Pasley used this data to reconstruct and play back the sounds the patients heard. He was able to do this because there is evidence that the brain breaks down sound into its component acoustic frequencies – for example, between a low of about 1 Hertz (cycles per second) to a high of about 8,000 Hertz –that are important for speech sounds.
Pasley tested two different computational models to match spoken sounds to the pattern of activity in the electrodes. The patients then heard a single word, and Pasley used the models to predict the word based on electrode recordings.
“We are looking at which cortical sites are increasing activity at particular acoustic frequencies, and from that, we map back to the sound,” Pasley said. He compared the technique to a pianist who knows the sounds of the keys so well that she can look at the keys another pianist is playing in a sound-proof room and “hear” the music, much as Ludwig van Beethoven was able to “hear” his compositions despite being deaf.
“This research is a major step towards understanding what features of speech are represented in the human brain” Knight said. “Brian’s analysis can reproduce the sound the patient heard, and you can actually recognize the word, although not at a perfect level.”
Knight predicts that this success can be extended to imagined, internal verbalizations, because scientific studies have shown that when people are asked to imagine speaking a word, similar brain regions are activated as when the person actually utters the word.
“With neuroprosthetics, people have shown that it’s possible to control movement with brain activity,” Knight said. “But that work, while not easy, is relatively simple compared to reconstructing language. This experiment takes that earlier work to a whole new level.”
Source: University of California, Berkeley