Dynamic Analysis of Automatic Facial Expressions Recognition ‗in the Wild‘ Using Generalized Additive Mixed Models and Significant Zero Crossing of the Derivatives

The analysis of facial expressions is currently a favoured method of inferring experienced emotion, and consequently significant efforts are currently being made to develop improved facial expression recognition techniques. Among these new techniques, those which allow the automatic recognition of facial expression appear to be most promising. This paper presents a new method of facial expression analysis with a focus on the continuous evolution of emotions using Generalized Additive Mixed Models (GAMM) and Significant Zero Crossing of the Derivatives (SiZer). The time-series analysis of the emotions experienced by participants watching a series of three different online videos suggests that analysis of facial expressions at the overall level may lead to misinterpretation of the emotional experience whereas non-linear analysis allows the significant expressive sequences to be identified.

Content

Author and article information

Contributors

Damien Dupré

Nicole Andelic

Gawain Morrison

Gary McKeown

Conference

Publication date: July 2018

Publication date (Print): July 2018

Pages: 1-8

Affiliations

[0001]Queen‘s University Belfast Belfast, United Kindgom

[0002]Sensum Belfast, United Kindgom

Article

DOI: 10.14236/ewic/HCI2018.1

SO-VID: 369e7701-a141-4c9d-8808-df03ec46e2eb

License:

This work is licensed under a Creative Commons Attribution 4.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/

Conference name: Proceedings of the 32nd International BCS Human Computer Interaction Conference

Conference acronym: HCI

Conference number: 32

Conference location: Belfast, UK

Conference date: 4 - 6 July 2018

Conference sponsor: Electronic Workshops in Computing (eWiC)

Conference theme: Human Computer Interaction Conference

History

Product

1477-9358 BCS Learning & Development

Self URI (article page): https://www.scienceopen.com/hosted-document?doi=10.14236/ewic/HCI2018.1

Self URI (journal page): https://ewic.bcs.org/

REFERENCES

P. ChaudhuriJ.S. Marron 1999 SiZer for exploration of structures in curves. Journal of the American Statistical Association 94 447 807 823
N. Chovil 1997 Facing others: A social communicative perspective on facial displays J. A. RussellJ. M. Fernández-Dols (eds.) The psychology of facial expression Cambridge university press Cambridge, UK 321 333
C. Darwin 1872 The expression of the emotions in man and animals John Murray London, UK
D. DupréA. BoothA. BolsterG. Morrison 2017 Dynamic Analysis of Automatic Emotion Recognition Using Generalized Additive Mixed Models In: Symposium on Computational Modelling of Emotion:Theory and Applications Bath, UK 158 163
P. Ekman 1992 An argument for basic emotions. Cognition & Emotion 6 3-4 169 200
P. EkmanW. FriesenJ. Hager 2002 New Version of the Facial Action Coding System A Human Face Publication Salt Lake City, USA
P. EkmanW.V. Friesen 1978 Facial action coding system: A technique for the measurement of facial movement Consulting Psychologists Press Palo Alto, USA
W.V. FriesenP. Ekman 1983 EMFACS-7: Emotional facial action coding system
D.A. HavasA.M. GlenbergK.A. GutowskiM.J. Lucarelli 2010 Cosmetic use of botulinum toxin-A affects processing of emotional language. Psychological Science 21 7 895 900
X. LinD. Zhang 1999 Inference in generalized additive mixed models by using smoothing splines. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 61 2 381 400
G.J. McKeownI. Sneddon 2014 Modelling continuous self-report measures of perceived emotion using generalized additive mixed models. Psychological methods 19 1 155 174
C.N. MoridisA.A. Economides 2012 Affective learning: Empathetic agents with emotional facial and tone of voice expressions. IEEE Transactions on Affective Computing 3 3 260 272
M.A. NicolaouH. GunesM. Pantic 2011 Continuous prediction of spontaneous affect from multiple cues and modalities in valence-arousal space. IEEE Transactions on Affective Computing 2 2 92 105
E. Parzen 1962 On estimation of a probability density function and mode. The annals of mathematical statistics 33 3 1065 1076
R.W. PicardE. VyzasJ. Healey 2001 Toward machine emotional intelligence: Analysis of affective physiological state. IEEE Transactions on Pattern Analysis and Machine Intelligence 23 10 1175 1191
M. Rosenblatt 1956 Remarks on some nonparametric estimates of a density function. The Annals of Mathematical Statistics 832 837
J.A. Russell 2016 A Sceptical Look at Faces as Emotion Signals C. AbellJ. Smith (eds.) The Expression of Emotion: Philosophical, Psychological and Legal Perspectives Cambridge, UK Cambridge University Press 157 172
G. SandbachS. ZafeiriouM. PanticL. Yin 2012 Static and dynamic 3D facial expression recognition: A comprehensive survey. Image and Vision Computing 30 10 683 697
D. SchacterC. WangG. NejatB. Benhabib 2013 A two-dimensional facial-affect estimation system for human–robot interaction using facial expression parameters. Advanced Robotics 27 4 259 273
K.R. Scherer 2005 What are emotions? And how can they be measured? Social Science Information 44 4 695 729
B. SchullerM. ValstarF. EybenG. McKeown 2011 Avec 2011–the first international audio/visual emotion challenge In: International Conference on Affective Computing and Intelligent Interaction 415 424
M.L. SmithG.W. CottrellF. GosselinP.G. Schyns 2005 Transmitting and decoding facial expressions. Psychological science 16 3 184 189
M.F. ValstarT. AlmaevJ.M. GirardG. McKeown 2015 Fera 2015-second facial expression recognition and analysis challenge In: International Conference and Workshops on Automatic Face and Gesture Recognition 2015 Ljubljana, Slovenia 1 8
S.N. Wood 2011 Fast stable restricted maximum likelihood and marginal likelihood estimation of semiparametric generalized linear models. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 73 1 3 36
S.N. Wood 2006 Low‐Rank Scale‐Invariant Tensor Product Smooths for Generalized Additive Mixed Models. Biometrics 62 4 1025 1036
S.N. WoodN. PyaB. Säfken 2016 Smoothing parameter and model selection for general smooth models. Journal of the American Statistical Association 111 516 1548 1563
X. XiongF. De la Torre 2013 Supervised descent method and its applications to face alignment In: IEEE Conference on Conference on Computer Vision and Pattern Recognition 2013 Portland, USA 532 539
Z. ZengM. PanticG.I. RoismanT.S. Huang 2009 A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence 31 1 39 58
A. ZuurE. IenoN. WalkerA. Saveliev 2009 Mixed effects models and extensions in ecology Spring Science and Business Media New York, USA

Comments

Comment on this article

[1] P. ChaudhuriJ.S. Marron 1999 SiZer for exploration of structures in curves. Journal of the American Statistical Association 94 447 807 823

[2] N. Chovil 1997 Facing others: A social communicative perspective on facial displays J. A. RussellJ. M. Fernández-Dols (eds.) The psychology of facial expression Cambridge university press Cambridge, UK 321 333

[3] C. Darwin 1872 The expression of the emotions in man and animals John Murray London, UK

[4] D. DupréA. BoothA. BolsterG. Morrison 2017 Dynamic Analysis of Automatic Emotion Recognition Using Generalized Additive Mixed Models In: Symposium on Computational Modelling of Emotion:Theory and Applications Bath, UK 158 163

[5] P. Ekman 1992 An argument for basic emotions. Cognition & Emotion 6 3-4 169 200

[6] P. EkmanW. FriesenJ. Hager 2002 New Version of the Facial Action Coding System A Human Face Publication Salt Lake City, USA

[7] P. EkmanW.V. Friesen 1978 Facial action coding system: A technique for the measurement of facial movement Consulting Psychologists Press Palo Alto, USA

[8] W.V. FriesenP. Ekman 1983 EMFACS-7: Emotional facial action coding system

[9] D.A. HavasA.M. GlenbergK.A. GutowskiM.J. Lucarelli 2010 Cosmetic use of botulinum toxin-A affects processing of emotional language. Psychological Science 21 7 895 900

[10] X. LinD. Zhang 1999 Inference in generalized additive mixed models by using smoothing splines. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 61 2 381 400

[11] G.J. McKeownI. Sneddon 2014 Modelling continuous self-report measures of perceived emotion using generalized additive mixed models. Psychological methods 19 1 155 174

[12] C.N. MoridisA.A. Economides 2012 Affective learning: Empathetic agents with emotional facial and tone of voice expressions. IEEE Transactions on Affective Computing 3 3 260 272

[13] M.A. NicolaouH. GunesM. Pantic 2011 Continuous prediction of spontaneous affect from multiple cues and modalities in valence-arousal space. IEEE Transactions on Affective Computing 2 2 92 105

[14] E. Parzen 1962 On estimation of a probability density function and mode. The annals of mathematical statistics 33 3 1065 1076

[15] R.W. PicardE. VyzasJ. Healey 2001 Toward machine emotional intelligence: Analysis of affective physiological state. IEEE Transactions on Pattern Analysis and Machine Intelligence 23 10 1175 1191

[16] M. Rosenblatt 1956 Remarks on some nonparametric estimates of a density function. The Annals of Mathematical Statistics 832 837

[17] J.A. Russell 2016 A Sceptical Look at Faces as Emotion Signals C. AbellJ. Smith (eds.) The Expression of Emotion: Philosophical, Psychological and Legal Perspectives Cambridge, UK Cambridge University Press 157 172

[18] G. SandbachS. ZafeiriouM. PanticL. Yin 2012 Static and dynamic 3D facial expression recognition: A comprehensive survey. Image and Vision Computing 30 10 683 697

[19] D. SchacterC. WangG. NejatB. Benhabib 2013 A two-dimensional facial-affect estimation system for human–robot interaction using facial expression parameters. Advanced Robotics 27 4 259 273

[20] K.R. Scherer 2005 What are emotions? And how can they be measured? Social Science Information 44 4 695 729

[21] B. SchullerM. ValstarF. EybenG. McKeown 2011 Avec 2011–the first international audio/visual emotion challenge In: International Conference on Affective Computing and Intelligent Interaction 415 424

[22] M.L. SmithG.W. CottrellF. GosselinP.G. Schyns 2005 Transmitting and decoding facial expressions. Psychological science 16 3 184 189

[23] M.F. ValstarT. AlmaevJ.M. GirardG. McKeown 2015 Fera 2015-second facial expression recognition and analysis challenge In: International Conference and Workshops on Automatic Face and Gesture Recognition 2015 Ljubljana, Slovenia 1 8

[24] S.N. Wood 2011 Fast stable restricted maximum likelihood and marginal likelihood estimation of semiparametric generalized linear models. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 73 1 3 36

[25] S.N. Wood 2006 Low‐Rank Scale‐Invariant Tensor Product Smooths for Generalized Additive Mixed Models. Biometrics 62 4 1025 1036

[26] S.N. WoodN. PyaB. Säfken 2016 Smoothing parameter and model selection for general smooth models. Journal of the American Statistical Association 111 516 1548 1563

[27] X. XiongF. De la Torre 2013 Supervised descent method and its applications to face alignment In: IEEE Conference on Conference on Computer Vision and Pattern Recognition 2013 Portland, USA 532 539

[28] Z. ZengM. PanticG.I. RoismanT.S. Huang 2009 A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence 31 1 39 58

[29] A. ZuurE. IenoN. WalkerA. Saveliev 2009 Mixed effects models and extensions in ecology Spring Science and Business Media New York, USA

Celebrating 65 years of The Computer Journal - free-to-read perspectives - bcs.org/tcj65