Scientists have reconstructed short movies from the brain activity of mice that watched videos for a project that aspires to lift the veil on how animals perceive the world.
The brief movie clips are grainy and pixellated, but provide a glimpse of how mice processed footage that featured people taking part in various sports from gymnastics to horse riding and wrestling.
The work is in its infancy, but as technology advances, scientists hope to eavesdrop on a richer suite of animal perceptions and ultimately gain fresh insights into their experiences and how brains more broadly respond to their surroundings.
“The nice thing with humans is you can just ask someone, what did you dream about? What did you see? What are you hallucinating?” said Dr Joel Bauer at the Sainsbury Wellcome Centre at University College London. “But we don’t have that access with animals in the same way.”
Movie reconstruction from mouse visual cortex activity
Central to the work was an artificial intelligence program that won a recent scientific competition to predict how electrical activity in the visual cortex of the mouse brain changes depending on what the animals are seeing. The visual cortex receives raw input from the retina and turns it into a coherent view of the world.
To reconstruct what mice were watching, the scientists first used an infrared laser to record how neurons were firing in the visual cortex as the rodents watched 10-second-long movie clips. They then fed blank video data into the AI program and steadily altered the imagery until the AI predicted the same patterns of brain activity as those seen in the mice. Details are published in the journal eLife.
Mice have poor eyesight compared with humans, so the reconstructed videos may never be as clear as the originals. But at a rough guess, Bauer suspects scientists could make the footage about seven times sharper than it is at present.
It is not the only area where there is room for improvement. The reconstructed videos are essentially a pinhole view of the screen the mice see, but future work could reconstruct the animal’s entire field of view, drawing on brain activity sparked by information from both eyes individually.
Researchers used AI to reconstruct movie clips from mouse brain activity
While Bauer is enthusiastic about reading animal brains, he is more cautious about parallel work in humans. Several research groups are devising ways to reconstruct images and other perceptions from human brain scans. Ultimately, this could lead to techniques that infringe on people’s privacy, he said.
“The risk in humans would be if you can reconstruct not what they see, but what they imagine,” he said. “We don’t necessarily want to share everything that’s happening in our heads,” he added. “The privacy of our neural data is important and will become more and more important.”
As for animals, he believes the approach could give scientists radical insights into how they experience the world, providing answers to questions such as what they see in dreams, are they fooled by the same optical illusions as humans, and even whether they hallucinate on magic mushrooms.
In the far future, he said, it may be possible to reconstruct a rich sense of an animal’s experience along with any accompanying emotions, leading to “a very deep kind of empathy” between humans and other species. So could humans finally understand what it is like to be a bat? “That would be cool,” Bauer said.
