Science News

Movies in the mind read by brain scanner

Thu, 22nd Sep 2011

Chris Smith

Using a brain scanner, scientists have successfully decoded and reconstructed the visual images experienced by volunteers viewing a sequence of Hollywood movies.

Writing in Current Biology, University of California, Berkeley researcher Jack Gallant and his colleagues placed three volunteers in an MRI scanner and asked them to watch a series of film trailer clips. As the scanner recorded their brains' responses to the movie information, a computer programme matched up how changes in the moving images were correlated with changes in small portions of the brain called voxels.

"This enabled us to create a 'dictionary' - a computational model - of how the brain was responding to shapes moving through the visual world," says Gallant.

HollywoodNext the volunteers were asked to watch a separate set of movies that they hadn't seen before. This time, based upon what it had learned from the initial training sessions, the computer was asked to predict what the subjects had been watching and then, using 100 clips selected from over 18 million seconds of video footage from YouTube, build a reconstruction of what it thought they had seen. Though blurred, the results are breathtaking.

"What we've effectively solved in this paper - with a computational model - is a way to link dynamic changes in the world, using natural movies, with changes in brain activity," says Gallant.

This work overcomes a significant constraint of functional MRI, which deduces brain activity by picking up changes in blood flow and oxygen consumption. But this technique is relatively slow and limited to changes taking place on a second by second basis. The team have surmounted this by using the scanner to look at how the brain itself responds to the much faster changes happening in the movies. It's these changes that can then be used to reconstruct the visual experience. 

Commenting on the implications of the results, Gallant stresses that this is not a technique that could be used to eavesdrop illicitly on peoples' thoughts "because, at the very least, you need an MRI scanner!" But the technique they have developed could, he points out, be applied more generally to assist scientists studying almost any other thought process in the nervous system. It could also, in the future, be used to decode dreams and then ask if what we think we dreamed was what we really dreamed after all...

References

Subscribe Free

Related Content

Comments

Make a comment

This seems to validate a pet theory of mine, that there is a one-to-one correspondence between our experiences and our emotions. The article seems to say that the emotions recorded during the subject's experience were reproduced in subsequent experiences, letting the computer infer what the experience was from the emotions detected by the MRI. grizelda, Thu, 22nd Sep 2011

Getting inside someone's head to interpret what they are dreaming about would in a way be an invasion of people's privacy. Dreams are a private matter. As long as technologies like this are used for something good it's OK. But there's a danger of such techniques being abused. vernahll2, Fri, 23rd Sep 2011



It wasn't so much emotions as direct neuronal responses to the applied stimulus. The computer is correlating which groups of neurones repond reproducibly to certain aspect of the moving visual world.

Then the equation is reversed and the question "if I know what pattern of neuronal activity was displayed, can I work backwards to recreate what the subject must have seen?"

This is what they've done, with breathtaking effect...

Jack Gallant will be on this week's show to talk about the work.

Chris chris, Fri, 23rd Sep 2011

This is awesome! I wish however people would read the article thoroughly before jumping to conclusions. Bayani, Fri, 23rd Sep 2011

See the whole discussion | Make a comment

Not working please enable javascript
Genetics Society
EPSRC
Powered by UKfast
STFC
ipDTL