How are facial movements related to emotion expressions?

The issue of how the muscular actions are related to concepts such as emotion is one of validation, to show that specific combinations of muscular actions reflect a particular emotion. A vast accumulated scientific literature over the last century and a half concerns the relation of facial expression to emotion. Here are some important points relevant to computer measurement of expression.

In regard to validity, the relationship between facial expressions and emotion is based only on muscular movements. Thus, if you develop a computer method for measuring emotion expression, you have two choices: 1) Clearly show that the measurement is based on specific muscular actions, or 2) Conduct arduous and difficult validation experiments to show that your new measurements are related to each emotion of interest. Consider, for example, a newly developed method that measures the movement of an elastic grid placed artificially on a facial image. Even though measurements of the grid's deformation can be accurately tracked and vary reliably with changes of expression, it would not be useful in measuring emotion, or any other psychological concept, unless one of the two paths enumerated above were followed. The same is true for tracking facial landmarks, or any other technique.

A second point is that for the few emotion categories that behavioral scientists have identified, several prototype emotion expressions may exist for each one. But these are only prototypes, and these prototypes are relatively infrequently observed in natural situations, especially at the extreme intensities portrayed in textbooks. A computer that was able to distinguish only among these different prototypes would not be useful in a practical situation.

Related to this point is that we do not know what all the emotion expressions are. Thus, it is not practical to train a computer to recognize all the emotion expressions because we do not have the complete training set, nor will it soon be available. There is a database that contains over 6000 different expressions and how they are related to emotion, and these are not nearly a complete set. As illustrated in a later answer about smiling and happiness, the important differences for interpreting the emotion in a facial expression can be quite subtle changes in muscular activation.

There is also the issue of simulated versus genuine emotional expressions. Humans are quite capable of mimicing an emotional expression. There are probably measurable differences between most fake and genuine expressions. Remember that most of the prototype expressions you see as illustrations in books are not really expressions of emotion, but rather posed, simulated productions used to illustrate such expressions. So, if you decide to build a computer that distinguishes these expressions in textbooks or photo sets of requested expressions, do not advertise it as an emotion expression recognizer, because these expressions are not really expressing genuine emotion. Your method might work with both deliberate and genuine emotion expressions, but you have another step to show that it can actually distinguish genuine emotion expressions. A more serious problem might develop if a measurement system were trained using motion records that had only simulated emotion expressions because it is likely that many, if not most, of the differences between genuine and simulated emotion expressions are in the motion parameters.

Finally, there are a number of variations on emotion expressions that contribute to a large number of members in the set and make the task for a measurement system difficult. Intensities of each action in a combination of actions can be different, and this may determine the appropriate emotion classification. A partial expression is when only some of the actions in a prototype expression appear. The partial expression often involves only one area of the face, e.g., only the forehead.  Actions in a particular expression typically do not onset and offset at the same time, and may be dispersed in time. Blends of expressions occur when different expressions appear at one time, such as anger and fear, or happy and surprise. A background expression is a continuous display of one expression that appears as a background for other expressions. A smile is often a background expression. Psychologically ill people often have a background expression. Background expressions need to be partialled out of the measurement when an emotion is predicted.