First brain map of speech units could aid mind-reading

Video: How speech sounds activate the brain

"He moistened his lips uneasily." It sounds like a cheap romance novel, but this line is actually lifted from quite a different type of prose: a neuroscience study.

Along with other sentences, including "Have you got enough blankets?" and "And what eyes they were", it was used to build the first map of how the brain processes the building blocks of speech – distinct units of sound known as phonemes.

The map reveals that the brain devotes distinct areas to processing different types of phonemes. It might one day help efforts to read off what someone is hearing from a brain scan.

"If you could see the brain of someone who is listening to speech, there is a rapid activation of different areas, each responding specifically to a particular feature the speaker is producing," says Nima Mesgarani, an electrical engineer at Columbia University in New York City.

Snakes on a brain

To build the map, Mesgarani's team turned to a group of volunteers who already had electrodes implanted in their brains as part of an unrelated treatment for epilepsy. The invasive electrodes sit directly on the surface of the brain, providing a unique and detailed view of neural activity.

The researchers got the volunteers to listen to hundreds of snippets of speech taken from a database designed to provide an efficient way to cycle through a wide variety of phonemes, while monitoring the signals from the electrodes. As well as those already mentioned, sentences ran the gamut from "It had gone like clockwork" to "Junior, what on Earth's the matter with you?" to "Nobody likes snakes".

This showed that there are distinct areas in a brain region called the superior temporal gyrus that are dedicated to different types of sounds – the STG is already known to be involved in filtering incoming sound.

One cluster of neurons responded only to consonants, while another responded only to vowels. These two areas then appeared to divide into even smaller groups. For example, some of the consonant neurons responded only to fricatives – sounds that force air through a narrow channel, like the s sound at the beginning of "scientist". Others responded only to plosives – sounds that block airflow, like the b in "brain".

Other scientists have used neural activity in attempts to recreate what people are looking at or capture people's thoughts in the form of their inner voice.

Mystery of meaning

Mesgarani thinks that the phoneme map may make it easier to figure out what someone is hearing from nothing but brain signals. He has tried to do this before by analysing neural responses to sound in ferrets and in people. The map should make relating the brain activity to specific sounds much easier.

Sophie Scott at the Institute of Cognitive Neuroscience at University College London cautions that the way we hear language is more complicated than just lining up different phonemes. Sounds change in subtle ways when they come together, so the brain must analyse the sequence as a whole.

"I think it's a shame that they haven't taken a more realistic approach to what we know actually happens when people listen to speech," Scott said. "Information isn't located at the level of the segments, it's actually extending over the sequence."

Mesgarani agrees that how the brain turns phonemes into meaning is still unknown. "How the brain takes these basic units, and puts them together to form sub-word and words, and ultimately meaning, remains a mystery," he says. He hopes the phoneme map will one day help answer this question, too.

Journal reference: Science, DOI: 10.1126/science.1245994

ORIGINAL: New Scientist
by Aviva Rutkin
19:00 30 January 2014