The Biology of Reading People

Anthropologists and Exolinguists (also called Xenolinguists) argue about how to communicate with extraterrestrials, but we are still being surprised by the subtleties of communicating with humans.  For example, a new study in Psychological Science (in press), analyzes how Dutch and Japanese people assess others’ emotions.  Just watch their faces?  Not quite.  Akihiro Tanaka et al. find that Dutch people pay attention to the facial expression more than Japanese people do.  “I think Japanese people tend to hide their negative emotions by smiling, but it’s more difficult to hide negative emotions in the voice,” Tanaka says.  So Japanese people may be used to listening for emotional cues.

Tanaka and colleagues made video clips of actors saying “Is that so?”— an ostensibly neutral phrase.  They used two different versions — angry and happy — in both Japanese and Dutch.  Then tricky video edits constructed an actor saying the phrase angrily but with a happy face and happily with an angry face.  Volunteers peered at these videos both in their native language and in the other language.  Then the psychologists asked each subject whether the person was angry or happy.  Statistically, Japanese people paid attention to the voice more than Dutch people did, regardless of being instructed to ignore the voice and judge the emotion by the faces.

This kind of research is not entirely new.  As Sherlock Holmes put it: “I see no more than you, but I have trained myself to notice what I see.”  The best card players have long mastered the ability to read faces, voices, and body language for “tells.” Edgar Allen Poe described this in “The Murders in the Rue Morgue]:

He notes every variation of face as the play progresses, gathering a fund of thought from the differences in the expression of certainty, of surprise, of triumph or of chagrin.  From the manner of gathering up a trick he judges whether the person taking it can make another in the suit.  He recognizes what is played through feint, by the air with which it is thrown upon the table.  A casual or inadvertent word; the accidental dropping or turning of a card, with the accompanying anxiety or carelessness in regard to its concealment; the counting of the tricks, with the order of their arrangement; embarrassment, hesitation, eagerness or trepidation — all afford, to his apparently intuitive perception, indications of the true state of affairs.  The first two or three rounds having been played, he is in full possession of the contents of each hand, and thenceforward puts down his cards with as absolute a precision of purpose as if the rest of the party had turned outward the faces of their own.

Charles Darwin performed early studies on how people recognize emotion in faces and Peter Snyder, a neuroscientist at Brown University, later proved his studies. (See “The evolution of emotion: Charles Darwin’s little-known psychology experiment.” Darwin was in communication with French neurologist and physician Guillaume-Benjamin-Amand Duchenne de Boulogne (September 17, 1806 – September 15, 1875).  Duchenne held that human faces expressed at least 60 discrete emotions, each depending on its own dedicated group of facial muscles.  Darwin published his alternative theory in the 1872 book The Expression of the Emotions in Man and Animals, in which he emphasized homologies between how all humans, and many other animals, exhibit emotion based on common descent and evolutionary history that extended across cultures and species.  Most psychologists share a consensus that certain emotions are human universals: anger, disgust, fear, happiness sadness, and surprise, to pick an alphabetized subset.

Paul Ekman and W. V. Friesen have identified that same list of six basic emotions that are expressed by facial expressions.  The eyes themselves can send messages.  Meeting someone’s glance with your eyes is usually a sign of involvement, while the opposite, looking away, indicates a desire to avoid contact.  This is why panhandlers or salesmen try to catch your eye.  Ekman argues: “Previous researchers had probably confused these culture specific modifications of emotional behaviors with the universals of expression.  For example, Samurai women were reported to smile rather than to cry when hearing that their loved ones had died in battle… Although such observations were taken as evidence of cultural variability in the meaning of smiles, these smiles may not have been signs of grief, but rather could have been culturally required masks implementing the display rule to show joy and hide distress in this public situation.”

In recent years, with the use of video and sophisticated computer software, answers have begun to emerge to three basic questions about the face and emotion: Is there any relationship at all? Are facial expressions culturally bound or universal? And, are any universals in expression biologically based?

The notion that experts can recognize when a lie is being told is fascinating.  Lie to Me, the American television series in which Dr. Cal Lightman (played by Tim Roth) and his colleagues accept assignments from third parties, usually in law enforcement, to uncover the truth through interpreting microexpressions, is based on this idea.  The protagonists’ use of FACS, the Facial Action Coding System, and body language is not just a TV invention.  It’s real — and the brainchild of the same Dr. Ekman.  And it doesn’t just apply to humans.  A variant of FACS has been developed to analyze facial expressions in chimpanzees, our closest evolutionary relatives.

“Today, we use almost the same [1872] technique, and even stimuli, to evaluate emotional recognition in a variety of psychiatric diseases, like autism and schizophrenia,” Snyder wrote, “Darwin’s method and approach are not locked in time.”

This may be the neuroevolutionary reason why we want robots to show human facial expressions of emotions and, at lower resolutions, why emoticons were invented for email.

Do you wish that you could be better at reading the physical emotional expressions of your fellow humans (or your own)?  Technology to the rescue.  Christine Lætitia Lisetti and Fatma Nasoz propose wearable computers in a multimodal system that aims at recognizing its users’ emotions and at responding to them accordingly depending upon the current context or application.  The wearables would input physiological signals from the user’s autonomic nervous system (galvanic skin response, heart rate, temperature) and map them to certain emotions (sadness, anger, fear, surprise, frustration, and amusement).

If you make first contact with extraterrestrials in your back yard, I made some suggestions of my own in “Me Human, You Alien: How to Talk to an Extraterrestrial”, which involve a handful of coins or a loop of string.  They might not have faces or voices, after all.


Akihiro Tanaka et al. I Feel Your Voice: Cultural Differences in the Multisensory Perception of Emotion. Psychological Science, (in press)

Christine Lætitia Lisetti and Fatma Nasoz, “Using noninvasive wearable computers to recognize human emotions from physiological signals”, EURASIP Journal on Applied Signal Processing, Volume 2004, January 2004.

Parr LA, Waller BM, Vick SJ, Bard KA (2007). “Classifying chimpanzee facial expressions using muscle action”. Emotion (Washington, D.C.) 7 (1): 172–81. doi:10.1037/1528-3542.7.1.172. PMID 17352572

See also:

Mind Reading (Neural Decoding) Goes Mainstream

Geek’s Guide to Getting Girls

Leave a Reply