Engagement with Artificial Intelligence through Natural Interaction Models

As Artificial Intelligence (AI) systems become more ubiquitous, what user experience design paradigms will be used by humans to impart their needs and intents to an AI system, in order to engage in a more social interaction? In our work, we look mainly at expression and creativity based systems, where the AI both attempts to model or understand/assist in processes of human expression and creativity. We therefore have designed and implemented a prototype system with more natural interaction modes for engagement with AI as well as other human computer interaction (HCI) where a more open natural communication stream is beneficial. Our proposed conversational agent system makes use of the affective signals from the gestural behaviour of the user and the semantic information from the speech input in order to generate a personalised, human-like conversation that is expressed in the visual and conversational output of the 3D virtual avatar system. We describe our system and two application spaces we are using it in – a care advisor / assistant for the elderly and an interactive creative assistant for uses to produce art forms.

Content

Author and article information

Contributors

Sara (Salevati) Feldman

Ozge Nilay Yalcin

Steve DiPaola

Conference

Publication date: July 2017

Publication date (Print): July 2017

Pages: 296-303

Affiliations

[0001]Simon Fraser University

Vancouver, Canada

Article

DOI: 10.14236/ewic/EVA2017.60

SO-VID: 0e70cbb5-3bc0-41f4-a651-8252f8bba62c

License:

This work is licensed under a Creative Commons Attribution 4.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/

Conference name: Electronic Visualisation and the Arts (EVA 2017)

Conference acronym: EVA

Conference number:

Conference location: London, UK

Conference date: 11 – 13 July 2017

Conference sponsor: Electronic Workshops in Computing (eWiC)

Conference theme: Electronic Visualisation and the Arts

History

Product

1477-9358 BCS Learning & Development

Self URI (article page): https://www.scienceopen.com/hosted-document?doi=10.14236/ewic/EVA2017.60

Self URI (journal page): https://ewic.bcs.org/

7. REFERENCES

C. AlmD. RothR. Sproat 2005 Emotions from Text: Machine Learning for Text-based Emotion Prediction. In Proceedings of the Human Language Technology Conference and the 2005 Conference on Empirical Methods in Natural Language Processing Vancouver, Canada 6-8 October 579 586
API.AI https://docs.api.ai/v25 10 Februray 2017
J. Bellegarda 2010 Emotion analysis using latent affective folding and embedding Proceedings of the NAACL-HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text Los Angeles, California
A. Ben-DavidO. Bat-El 2016 Developing phonology A. Bar-OnD. Ravid Handbook of Communication Disorders: Theoretical, Empirical, and Applied Linguistic Perspectives. Mouton De Gruyter Berlin
U. BernardetS. DiPaola 2015 A Framework for Exogenous and Endogenous Reflexive Behavior in Virtual Characters International Conference on Intelligent Virtual Agents 132 136 Springer 2015
U. BernardetM. CholletS. DiPaolaS. Scherer 2016 An Architecture for Biologically Grounded Real-time Reflexive Behavior Intelligent Virtual Agents. Lecture Notes in Computer Science Springer
U. BernardetM. SaberiS. DiPaola 2016 Simulink Toolbox for Real-time Virtual Character Control Intelligent Virtual Agents. Lecture Notes in Computer Science Springer
S. BirdE. KleinE. Loper 2009 Natural language processing with Python: analyzing text with the natural language toolkit. O’Reilly Media Inc
G. Bradski 2000 OpenCV computer vision library Dr. Dobb’s Journal of Software Tools.
C. BussoZ. DengS. YildirimM. BulutC. M. LeeA. KazemzadehS. LeeU. NeumannS. Narayanan 2004 Analysis of emotion recognition using facial expressions, speech and multimodal information Proceedings of Sixth International Conference on Multimodal Interfaces 205 211
R. A. CalvoS. D’Mello 2010 Affect detection: An interdisciplinary review of models, methods, and their applications IEEE Transactions on affective computing 1 1 18 37
C. ClavelG. AddaF. CailliauM. Garnier-RizetA. CavetG. ChapuisP. Suignard 2013 Spontaneous speech and opinion detection: mining call-centre transcripts Language Resources and Evaluation 47 4 1089 1125
S. DiPaola 2014 Computer Modelling Fine Art Painting using a Cognitive Correlative Heuristics Approach, Proc: Biologically Inspired Cognitive Architectures. MIT, MA USA
P. EkmanH. Oster 1979 Facial expressions of emotion Annual Review Psychology 30 527 554
P. Ekman 2004 Emotional and conversational nonverbal signals Language, knowledge, and representation 39 50 Springer Netherlands
J. M. GirardJ. F. CohnM. H. MahoorS. M. MavadatiZ. HammalD. P. Rosenwald 2014 Nonverbal social withdrawal in depression: Evidence from manual and automatic analyses Image and Vision Computing 32 10 641 647
C. J. HuttoE. E. Gilbert 2014 VADER: A Parsimonious Rule-based Model for Sentiment Analysis of Social Media Text. Eighth International Conference on Weblogs and Social Media (ICWSM-14). Ann Arbor, MI June
J. IrastorzaM. I. Torres 2016 Identification of emotional shifts on customer services. Submitted to Inter-speech 2016
A. JaimesN. Sebe 2007 Multimodal human-computer interaction: A survey Computer Vision and Image Understanding 108 1 116 134
P. N. JuslinK. R. Scherer 2005 Vocal expression of affect J. HarriganR. RosenthalK. Scherer (eds). The New Handbook of Methods in Nonverbal Behavior Research. Oxford University Press Oxford, UK
T. Justo 2014 Extracting relevant knowledge for the detection of sarcasm and nastiness in the social web Knowledge-Based Systems 69 124 133
T. KanadeJ. F. CohnY. Tian 2000 Comprehensive database for facial expression analysis. Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition (FG’00) Grenoble, France 46 53
S. G. KaradoğanJ. Larsen 2012 Combining semantic and acoustic features for valence and arousal recognition in speech 3rd International Workshop on Cognitive Information Processing (CIP) 1 6 IEEE
C. M. LeeS. S. Narayanan 2005 Toward detecting emotions in spoken dialogs IEEE Transactions on Speech and Audio Processing 13 2 293 303
C. LeeS. NarayananR. Pieraccini 2002 Combining acoustic and language information for emotion recognition Proceedings of ICSLP 2002 873 876
K. Lehrer 1974 Knowledge. Oxford University Press New York
P. LuceyJ. F. CohnT. KanadeJ. SaragihZ. AmbadarI. Matthews 2010 The Extended Cohn-Kanade Dataset (CK+): A complete expression dataset for action unit and emotion-specified expression. Proceedings of the Third International Workshop on CVPR for Human Communicative Behavior Analysis (CVPR4HB 2010). San Francisco, USA 94 101
S. M. MohammadC. DunneB. Dorr 2009 Generating high-coverage semantic orientation lexicons from overtly marked words and a thesaurus Proceedings of Empirical Methods in Natural Language Processing (EMNLP-2009) 599 608 Singapore
K. A. Morrissey 2002 Pathways Among Objects and Museum Visitors S. G. Paris Perspectives on Object-Centered Learning in Museums Mahwah Lawrence Erlbaum Associates
I. R. MurrayJ. L. Arnott 1993 Toward a simulation of emotion in synthetic speech: A review of the literature on human vocal emotion The Journal of the Acoustical Society of America 93 2 1097 1108
R. W. Picard 1997 Affective Computing 252 Cambridge MIT Press
J. A. RussellJ. BachorowskiJ. Fernandez-Dols 2003 Facial and vocal expressions of emotion Annual Review of Psychology 54 329 349
J. A. Russell 1980 A circumplex model of affect Journal of Personality and Social Psychology 39 6 1161 1178
M. SaberiU. BernardetS. DiPaola 2014 An Architecture for Personality-Based, Nonverbal Behavior in Affective Virtual Humanoid Character Procedia Computer Science. Bio Inspired Cognitive Architectures 41 204 211
B. SchullerG. RigollM. Lang 2004 Speech emotion recognition combining acoustic features and linguistic information in a hybrid support vector machine-belief network architecture Proceedings of ICASSP 2004 577 580
M. ThiebauxS. MarsellaA. N. MarshallM. Kallmann 2008 Smartbody: Behavior realization for embodied conversational agents Proceedings of the 7th international joint conference on Autonomous agents and multiagent systems, Volume 1 151 158 International Foundation for Autonomous Agents and Multiagent Systems
P. TurneyM. Littman 2003 Measuring praise and criticism: Inference of semantic orientation from association ACM Transactions on Information Systems (TOIS) 21 4 315 346
D. VerveridisC. Kotropoulos 2006 Emotional speech recognition: Resources, features, and methods Speech Communication 48 9 1162 1181
Z. ZengM. PanticG. I. RoismanT. S. Huang 2009 A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE Ttransactions on Pattern Analysis and Machine Intelligence 31 1 39 58

Comments

Comment on this article

[1] C. AlmD. RothR. Sproat 2005 Emotions from Text: Machine Learning for Text-based Emotion Prediction. In Proceedings of the Human Language Technology Conference and the 2005 Conference on Empirical Methods in Natural Language Processing Vancouver, Canada 6-8 October 579 586

[2] API.AI https://docs.api.ai/v25 10 Februray 2017

[3] J. Bellegarda 2010 Emotion analysis using latent affective folding and embedding Proceedings of the NAACL-HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text Los Angeles, California

[4] A. Ben-DavidO. Bat-El 2016 Developing phonology A. Bar-OnD. Ravid Handbook of Communication Disorders: Theoretical, Empirical, and Applied Linguistic Perspectives. Mouton De Gruyter Berlin

[5] U. BernardetS. DiPaola 2015 A Framework for Exogenous and Endogenous Reflexive Behavior in Virtual Characters International Conference on Intelligent Virtual Agents 132 136 Springer 2015

[6] U. BernardetM. CholletS. DiPaolaS. Scherer 2016 An Architecture for Biologically Grounded Real-time Reflexive Behavior Intelligent Virtual Agents. Lecture Notes in Computer Science Springer

[7] U. BernardetM. SaberiS. DiPaola 2016 Simulink Toolbox for Real-time Virtual Character Control Intelligent Virtual Agents. Lecture Notes in Computer Science Springer

[8] S. BirdE. KleinE. Loper 2009 Natural language processing with Python: analyzing text with the natural language toolkit. O’Reilly Media Inc

[9] G. Bradski 2000 OpenCV computer vision library Dr. Dobb’s Journal of Software Tools.

[10] C. BussoZ. DengS. YildirimM. BulutC. M. LeeA. KazemzadehS. LeeU. NeumannS. Narayanan 2004 Analysis of emotion recognition using facial expressions, speech and multimodal information Proceedings of Sixth International Conference on Multimodal Interfaces 205 211

[11] R. A. CalvoS. D’Mello 2010 Affect detection: An interdisciplinary review of models, methods, and their applications IEEE Transactions on affective computing 1 1 18 37

[12] C. ClavelG. AddaF. CailliauM. Garnier-RizetA. CavetG. ChapuisP. Suignard 2013 Spontaneous speech and opinion detection: mining call-centre transcripts Language Resources and Evaluation 47 4 1089 1125

[13] S. DiPaola 2014 Computer Modelling Fine Art Painting using a Cognitive Correlative Heuristics Approach, Proc: Biologically Inspired Cognitive Architectures. MIT, MA USA

[14] P. EkmanH. Oster 1979 Facial expressions of emotion Annual Review Psychology 30 527 554

[15] P. Ekman 2004 Emotional and conversational nonverbal signals Language, knowledge, and representation 39 50 Springer Netherlands

[16] J. M. GirardJ. F. CohnM. H. MahoorS. M. MavadatiZ. HammalD. P. Rosenwald 2014 Nonverbal social withdrawal in depression: Evidence from manual and automatic analyses Image and Vision Computing 32 10 641 647

[17] C. J. HuttoE. E. Gilbert 2014 VADER: A Parsimonious Rule-based Model for Sentiment Analysis of Social Media Text. Eighth International Conference on Weblogs and Social Media (ICWSM-14). Ann Arbor, MI June

[18] J. IrastorzaM. I. Torres 2016 Identification of emotional shifts on customer services. Submitted to Inter-speech 2016

[19] A. JaimesN. Sebe 2007 Multimodal human-computer interaction: A survey Computer Vision and Image Understanding 108 1 116 134

[20] P. N. JuslinK. R. Scherer 2005 Vocal expression of affect J. HarriganR. RosenthalK. Scherer (eds). The New Handbook of Methods in Nonverbal Behavior Research. Oxford University Press Oxford, UK

[21] T. Justo 2014 Extracting relevant knowledge for the detection of sarcasm and nastiness in the social web Knowledge-Based Systems 69 124 133

[22] T. KanadeJ. F. CohnY. Tian 2000 Comprehensive database for facial expression analysis. Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition (FG’00) Grenoble, France 46 53

[23] S. G. KaradoğanJ. Larsen 2012 Combining semantic and acoustic features for valence and arousal recognition in speech 3rd International Workshop on Cognitive Information Processing (CIP) 1 6 IEEE

[24] C. M. LeeS. S. Narayanan 2005 Toward detecting emotions in spoken dialogs IEEE Transactions on Speech and Audio Processing 13 2 293 303

[25] C. LeeS. NarayananR. Pieraccini 2002 Combining acoustic and language information for emotion recognition Proceedings of ICSLP 2002 873 876

[26] K. Lehrer 1974 Knowledge. Oxford University Press New York

[27] P. LuceyJ. F. CohnT. KanadeJ. SaragihZ. AmbadarI. Matthews 2010 The Extended Cohn-Kanade Dataset (CK+): A complete expression dataset for action unit and emotion-specified expression. Proceedings of the Third International Workshop on CVPR for Human Communicative Behavior Analysis (CVPR4HB 2010). San Francisco, USA 94 101

[28] S. M. MohammadC. DunneB. Dorr 2009 Generating high-coverage semantic orientation lexicons from overtly marked words and a thesaurus Proceedings of Empirical Methods in Natural Language Processing (EMNLP-2009) 599 608 Singapore

[29] K. A. Morrissey 2002 Pathways Among Objects and Museum Visitors S. G. Paris Perspectives on Object-Centered Learning in Museums Mahwah Lawrence Erlbaum Associates

[30] I. R. MurrayJ. L. Arnott 1993 Toward a simulation of emotion in synthetic speech: A review of the literature on human vocal emotion The Journal of the Acoustical Society of America 93 2 1097 1108

[31] R. W. Picard 1997 Affective Computing 252 Cambridge MIT Press

[32] J. A. RussellJ. BachorowskiJ. Fernandez-Dols 2003 Facial and vocal expressions of emotion Annual Review of Psychology 54 329 349

[33] J. A. Russell 1980 A circumplex model of affect Journal of Personality and Social Psychology 39 6 1161 1178

[34] M. SaberiU. BernardetS. DiPaola 2014 An Architecture for Personality-Based, Nonverbal Behavior in Affective Virtual Humanoid Character Procedia Computer Science. Bio Inspired Cognitive Architectures 41 204 211

[35] B. SchullerG. RigollM. Lang 2004 Speech emotion recognition combining acoustic features and linguistic information in a hybrid support vector machine-belief network architecture Proceedings of ICASSP 2004 577 580

[36] M. ThiebauxS. MarsellaA. N. MarshallM. Kallmann 2008 Smartbody: Behavior realization for embodied conversational agents Proceedings of the 7th international joint conference on Autonomous agents and multiagent systems, Volume 1 151 158 International Foundation for Autonomous Agents and Multiagent Systems

[37] P. TurneyM. Littman 2003 Measuring praise and criticism: Inference of semantic orientation from association ACM Transactions on Information Systems (TOIS) 21 4 315 346

[38] D. VerveridisC. Kotropoulos 2006 Emotional speech recognition: Resources, features, and methods Speech Communication 48 9 1162 1181

[39] Z. ZengM. PanticG. I. RoismanT. S. Huang 2009 A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE Ttransactions on Pattern Analysis and Machine Intelligence 31 1 39 58

Studying business & IT? Drive your professional career forwards with BCS books - for a 20% discount click here: shop.bcs.org