Machine learning turns thoughts into words
Understandable speech has been reconstructed from brain cell activity using machine learning, raising the possibility of giving a voice to the voiceless.
When we speak, or listen to someone speaking, a part of the brain called the auditory cortex becomes active as it processes the speech. When someone loses the ability to speak, owing to diseases or trauma, while the parts of the nervous system required to make speech movements may not work, the auditory cortex can remain unharmed. For instance, a person with motor neurone disease is likely to have a perfectly healthy auditory cortex yet be unable to speak because of paralysed muscles. But by studying the patterns of brain cell activity that arise in the auditory cortex when a patient hears speech, scientists have recreated speech artificially at a higher clarity than ever before, in an important step towards restoring speech to those who have lost it.
The research, published in the journal Scientific Reports, comes from Nima Mesgerani’s group at the Neural Acoustic Processing Lab, Columbia University. According to Mesgerani, "what we're hoping to do is to directly read speech from the brain of a person so that they do not have to actually say it… As the brain activity is produced we can directly detect and decode it."
This kind of speech reconstruction has been studied before, but Mesgerani and his team brought together two state of the art technologies - machine learning, and modern speech creation software (of the likes used in your virtual assistants, Siri and Alexa) - to reproduce much clearer speech than previous attempts.
To achieve this, the researchers worked with five patients who were undergoing brain surgery to treat epilepsy, all of whom had normal hearing, and placed electrodes directly onto the surface of the auditory cortex in each of their brains. While the patients then listened to 30 minutes of short stories, the team recorded the neural activity as their brains processed the speech. The data were fed through a machine learning algorithm called a neural network, a complex set of instructions that mimics the dense network of connections found in the brain, to teach the algorithm how the brain processes speech.
This allowed the researchers to play the patient a reading like this: