Faces everywhere: pareidolia in machine learning
A new machine learning model and dataset reveal insights about how and why we hallucinate faces in inanimate objects—and give computers a more human way of seeing the world.
FacesInThings / MIT CSAIL
FacesInThings / MIT CSAIL
Look at an electrical outlet, and you can almost see a little shocked face: two slits for eyes, and a ground-socket mouth. This is an instance of pareidolia, in which we see faces in everyday objects. It’s a phenomenon that’s been examined in countless studies: magnetoencephalography shows that pareidolic images activate the fusiform face area, a part of the brain responsible for facial recognition. Electroencephalogram measurements show that brain waves are generated earlier in the prefrontal cortex when the brain detects a face compared to when it doesn’t. From birth, pareidolia is hardwired into our visual perception.
Yet its core mechanism remains largely unexplained. How, and why, did we evolve to see faces everywhere? A new study from the MIT Computer Science & Artificial Intelligence Laboratory (CSAIL) presented at the European Conference on Computer Vision (ECCV) in October tries to answer that question by training a machine learning model to see faces in things too. “What this paper really tries to do is explore the intersection between pareidolia and computer vision,” says Mark Hamilton, MIT PhD student in electrical engineering and computer science and the lead researcher on the work.
To test how well computers could see these illusory faces, Hamilton and his team built the world’s first large-scale pareidolia dataset. Filtering from a set of 5.85 billion images, the researchers—plus Hamilton’s mother—put together 5,000 pictures of pareidolic faces.
“The hope is that this kind of data set can be used to test psychological things in silico,” or using computer models, says Hamilton. Researchers think these models offer a way to test hypotheses about human cognition without the ethical and logistical constraints of traditional experiments, “like how people create fancy rat models for testing new drugs.”
In one experiment, the researchers quizzed a state-of-the-art face detector on the pareidolia dataset. It had a hard time until they gave it some extra training data with animal faces. And that might reveal something about why our brains evolved to exhibit pareidolia. “Previously people might have thought pareidolia was because of emotion detection, but this result seems to say that maybe it’s so that we don't get eaten by a tiger,” says Hamilton. In environments where survival depended on recognizing hidden dangers, the ability to perceive face-like patterns—even in ambiguous situations—might have provided a crucial edge.
This highlights one advantage of using a computer model to explore questions in cognition: researchers can manipulate variables in ways that would be impossible with real humans. “You could never test a human being who's never seen an animal,” says Hamilton. But with a machine learning system, all you have to do is remove the animals from its training dataset.
In another experiment, the researchers intentionally used very simple machine learning systems, such as Gaussian models that represent data with a series of simple probability distributions, to represent pareidolic faces, showing that they tend to match the low-frequency features of faces—the overall structure or contours—while ignoring high-frequency details like textures. Modern neural networks are often black boxes too complicated to analyze, but it’s easy to visualize a simple model and understand what’s going on under the hood. “You can write down a closed form equation, graph it out, and ask questions about how it behaves,” says Hamilton.
But how well do the machine learning models match up with reality? Greg Borenstein, a graduate of the Playful Systems Group at the MIT Media Lab who was not involved in the study, cautions against drawing direct parallels between machines and humans. “The thing that a human being sees as a face is very different,” he said in an interview with Science, from what face-detection algorithms might look for; a convolutional neural network detects very specific patterns of pixels rather than the concept of two eyes and a mouth. Algorithms don’t exhibit pareidolia because of evolutionary pressures, but because of the way they’re trained, he says, and every algorithm will express pareidolia differently.
There are, however, indications that the computers match human perception in some aspects. Hamilton’s team showed pictures of random noise at different levels of detail to human subjects, and asked them to count how many faces they saw. At low detail, people see few faces, because they look too blurry. At high detail, people see few faces, because high-resolution faces rarely occur randomly. The “pareidolic peak,” where people see the most faces, is somewhere in the middle. When given the same experiment, the computers exhibited the same kind of peak.
And what about uses beyond psychological experiments? The most direct application is for fine-tuning facial detectors, either to reduce false positives, or to make them more generalized (to see cartoon faces, for example).
Another possible application is to give computers an understanding of friendliness that could be used in generative design. For example, if a computer is helping to design, say, a dental scanner used on children, it needs to look approachable so that children aren’t terrified of having it shoved into their mouths. “You don't want to make this accidentally menacing,” says Hamilton. One coauthor, from Toyota Research, was interested in whether cars could be made to look more or less intimidating based on their perceived pareidolic emotion.
By creating a connection between computer vision and pareidolia, this research could both help advance cognitive science and help teach machines to see the world like us—soon, your computer might be able to detect that shocked electrical-outlet face too.