Deictic Communication across Distances: Visualising Remote Pointing Gestures on Mobile Devices

Deictic expressions, such as ‘what is that’ while pointing at an object, play an important role in face-to-face communication, for example when describing locations and orientation or when identifying objects. If two parties are not collocated, e.g. when communicating via mobile phones, such deictic expressions cannot easily be exchanged between the remote parties. In this paper, we propose three ways to visualise deictic pointing gestures to a remote communication partner: 1) fingerprint overlay, 2) natural hand overlay and 3) map-with-viewshed (see Fig. 1). We evaluated these visualisations in a lab-based user study, where participants had to identify various realistic targets on a mobile phone. Overall, participants preferred and were most successful with fingerprint overlay. We also identified properties of the target objects that affected how well a pointing gesture could be transmitted. Our results can inform the design of future interfaces to transmit pointing gestures across distances.

Content

Author and article information

Contributors

Samuel Navas Medrano

Max Pfeiffer

Christian Kray

Conference

Publication date: July 2018

Publication date (Print): July 2018

Pages: 1-13

Affiliations

[0001]University of Münster Münster, Germany

Article

DOI: 10.14236/ewic/HCI2018.11

SO-VID: 45dfd425-3dd6-48e5-af6d-64f5e585ca15

License:

This work is licensed under a Creative Commons Attribution 4.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/

Conference name: Proceedings of the 32nd International BCS Human Computer Interaction Conference

Conference acronym: HCI

Conference number: 32

Conference location: Belfast, UK

Conference date: 4 - 6 July 2018

Conference sponsor: Electronic Workshops in Computing (eWiC)

Conference theme: Human Computer Interaction Conference

History

Product

1477-9358 BCS Learning & Development

Self URI (article page): https://www.scienceopen.com/hosted-document?doi=10.14236/ewic/HCI2018.11

Self URI (journal page): https://ewic.bcs.org/

REFERENCES

D. AkkilP. Isokoski 2016 Accuracy of interpreting pointing gestures in egocentric view In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing 262 273 ACM
M. W. AlibaliD. C. HeathH. J. Myers 2001 Effects of visibility between speaker and listener on gesture production: Some gestures are meant to be seen Journal of Memory and Language 44 2 169 188
G. L. Allen 2003 Gestures accompanying verbal route directions: Do they point to a new avenue for examining spatial representations? Spatial cognition and computation 3 4 259 268
J. AmoresX. BenavidesP. Maes 2015 Showme: A remote collaboration system that supports immersive gestural communication In Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems 1343 1348 ACM
I. AvellinoC. FleuryM. Beaudouin-Lafon 2015 Accuracy of deictic gestures to support telepresence on wall-sized displays In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems 2393 2396 ACM
A. D. BalakrishnanS. R. FussellS. Kiesler 2008 Do visualizations improve synchronous remote collaboration? In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems 1227 1236 ACM
E. BatesB. O’ConnellC. Shore 1987 Language and communication in infancy
Z. Bauman 2000 Liquid modernity 2000 Polity, Cambridge
F. BioccaA. TangC. OwenF. Xiao 2006 Attention funnel: omnidirectional 3d cursor for mobile augmented reality platforms In Proceedings of the SIGCHI conference on Human Factors in computing systems 1115 1122 ACM
R. A. Bolt 1980 Put-that-there: Voice and gesture at the graphics interface In Proceedings of the 7th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH ’80 New York, NY, USA 262 270 ACM
F. Cairncross 2001 The death of distance: How the communications revolution is changing our lives Harvard Business Press
G. Calbris 1990 The semiotics of French gestures 1900 Indiana Univ Pr
M. CarpenterK. NagellM. TomaselloG. ButterworthC. Moore 1998 Social cognition, joint attention, and communicative competence from 9 to 15 months of age Monographs of the society for research in child development, i–174
S. D’AngeloD. Gergle 2016 Gazed and confused: Understanding and designing shared gaze for remote collaboration In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, CHI ’16 New York, NY, USA 2492 2496 ACM
H. Diessel 1999 Demonstratives: Form, function and grammaticalization 42 John Benjamins Publishing
H. Diessel 2006 Demonstratives, joint attention, and the emergence of grammar Cognitive linguistics 17 4 463 489
Economic Commission for Europe, Transport Division 2007 Convention on road signs and signals of 1968; European agreement supplementing the convention and protocol on road markings, additional to the European agreement ix 276
D. Efron 1972 Gesture, race and culture JSTOR
C. Egido 1988 Video conferencing as a technology to support group work: a review of its failures In Proceedings of the 1988 ACM conference on Computersupported cooperative work 13 24 ACM
C. Egido 1990 Teleconferencing as a technology to support cooperative work: Its possibilities and limitations
P. Eisert 2003 Immersive 3d video conferencing: challenges, concepts, and implementation In VCIP 69 79
P. EkmanW. V. Friesen 1969 The repertoire of nonverbal behavior: Categories, origins, usage, and coding semiotica 1 1 49 98
T. FechnerD. WilhelmC. Kray 2015 Ethermap: real-time collaborative map editing In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems 3583 3592 ACM
H. Feldman 1986 A grammar of awtuw.(= pacific linguistics series b 94) Canberra: Australian National University
F. FrancoG. Butterworth 1996 Pointing and social awareness: Declaring and requesting in the second year Journal of child language 23 2 307 336
S. R. FussellL. D. SetlockJ. YangJ. OuE. MauerA. D. Kramer 2004 Gestures over video streams to support remote collaboration on physical tasks HumanComputer Interaction 19 3 273 309
S. GauglitzC. LeeM. TurkT. Höllerer 2012 Integrating the physical environment into mobile remote collaboration In Proceedings of the 14th international conference on Human-computer interaction with mobile devices and services 241 250 ACM
S. GauglitzB. NuernbergerM. TurkT. Höllerer 2014 World-stabilized annotations and virtual scene navigation for remote collaboration In Proceedings of the 27th annual ACM symposium on User interface software and technology 449 459 ACM
A. Giddens 1984 The constitution of society: Outline of the theory of structuration Univ of California Press
J. A. GrahamM. Argyle 1975 A cross-cultural study of the communication of extra-verbal meaning by gestures International Journal of Psychology 10 1 57 67
F. HaqueM. NancelD. Vogel 2015 Myopoint: Pointing and clicking using forearm mounted electromyography and inertial motion sensors In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems 3653 3656 ACM
D. Harvey 1989 The condition of postmodernity: An enquiry into the origins of social change Malden, MA: Blackwell
M. HegartyA. E. RichardsonD. R. MontelloK. LovelaceI. Subbiah 2002 Development of a self-report measure of environmental spatial ability Intelligence 30 5 425 447
G. W. Hewes 1981 Pointing and language Advances in Psychology 7 263 269
G. W. Hewes 1996 A history of the study of language origins and the gestural primacy hypothesis Handbook of human symbolic evolution 571 595
K. HirataY. HaradaT. TakadaN. YamashitaS. AoyagiY. ShiraiK. KajiJ. YamatoK. Nakazawa 2008 Video communication system supporting spatial cues of mobile users Proc. CollabTech 122 127
E. A. IsaacsJ. C. Tang 1994 What video can and cannot do for collaboration: a case study Multimedia systems 2 2 63 73
O. B. KaulM. PfeifferM. Rohs 2016 Follow the force: Steering the index finger towards targets using ems In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems EA ’16, New York, NY, USA 2526 2532 ACM
A. KendonL. Versante 2003 Pointing by hand in neapolitan Pointing: where language, culture, and cognition meet 109 137
S. Kern 1983 The culture of time and space Cambridge, MA: Harvard UP
D. KirkT. RoddenD. S. Fraser 2007 Turn it this way: grounding collaborative action with remote gestures In Proceedings of the SIGCHI conference on Human Factors in Computing Systems 1039 1048 ACM
S. Kita 2003 Pointing: Where language, culture, and cognition meet Psychology Press
S. KitaA. Özyürek 2003 What does cross-linguistic variation in semantic coordination of speech and gesture reveal?: Evidence for an interface representation of spatial thinking and speaking Journal of Memory and language 48 1 16 32
Y. LiuY. GuoC. Liang 2008 A survey on peer-topeer video streaming systems Peer-to-peer Networking and Applications 1 1 18 28
S. MayerK. WolfS. SchneegassN. Henze 2015 Modeling distant pointing for compensating systematic displacements In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems 4165 4168 ACM
D. McNeill 1992 Hand and mind: What gestures reveal about thought University of Chicago press
D. McNeill 2008 Gesture and thought University of Chicago press
S. MitraT. Acharya 2007 Gesture recognition: A survey IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 37 3 311 324
O. MorikawaT. Maesako 1998 Hypermirror: toward pleasant-to-use video mediated communication system In Proceedings of the 1998 ACM conference on Computer supported cooperative work 149 158 ACM
D. Morris 1978 Manwatching: A fold guide to human behaviour
S. Navas MedranoM. PfeifferC. Kray 2017 Enabling remote deictic communication with mobile devices: An elicitation study In Proceedings of the 19th International Conference on HumanComputer Interaction with Mobile Devices and Services, MobileHCI ’17, New York, NY, USA 19:1 19:13 ACM
D. J. PovinelliD. R. Davis 1994 Differences between chimpanzees (pan troglodytes) and humans (homo sapiens) in the resting state of the index finger: Implications for pointing Journal of Comparative Psychology 108 2134
F. H. RauscherR. M. KraussY. Chen 1996 Gesture, speech, and lexical access: The role of lexical movements in speech production Psychological Science 7 4 226 231
S. S. RautarayA. Agrawal 2015 Vision based hand gesture recognition for human computer interaction: a survey Artificial Intelligence Review 43 1 1 54
L. Rolfe 1996 Theoretical stages in the prehistory of grammar Handbook of human symbolic evolution 776 792
M. SeyfeddinipurS. KitaC. CaveI. GuaitellaS. Santi 2001 Gesture and dysfluency in speech Oralite et gesturalite: Interactions et comportemetns multimodaux dans la communication 266 270
J. Sherzer 1973 Verbal and nonverbal deixis: The pointed lip gesture among the san blas cuna Language in Society 2 1 117 131
J. Sherzer 1993 Pointed lips, thumbs up, and cheek puffs: Some emblematic gestures in social interactional and ethnographic context In Texas linguistic forum 33 196 211 University of Texas, Department of Linguistics
K. TsukadaM. Yasumura 2004 Activebelt: Belttype wearable tactile display for directional navigation In UbiComp 3205 384 399 Springer
D. Wilkins 2003 Why pointing with the index finger is not a universal (in sociocultural and semiotic terms) Pointing: Where language, culture, and cognition meet 171 215
N. WongC. Gutwin 2010 Where are you pointing?: the accuracy of deictic pointing in cves In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems 1029 1038 ACM
W. Wundt 1973 The language of gestures 6 Walter de Gruyter

Comments

Comment on this article

[1] D. AkkilP. Isokoski 2016 Accuracy of interpreting pointing gestures in egocentric view In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing 262 273 ACM

[2] M. W. AlibaliD. C. HeathH. J. Myers 2001 Effects of visibility between speaker and listener on gesture production: Some gestures are meant to be seen Journal of Memory and Language 44 2 169 188

[3] G. L. Allen 2003 Gestures accompanying verbal route directions: Do they point to a new avenue for examining spatial representations? Spatial cognition and computation 3 4 259 268

[4] J. AmoresX. BenavidesP. Maes 2015 Showme: A remote collaboration system that supports immersive gestural communication In Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems 1343 1348 ACM

[5] I. AvellinoC. FleuryM. Beaudouin-Lafon 2015 Accuracy of deictic gestures to support telepresence on wall-sized displays In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems 2393 2396 ACM

[6] A. D. BalakrishnanS. R. FussellS. Kiesler 2008 Do visualizations improve synchronous remote collaboration? In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems 1227 1236 ACM

[7] E. BatesB. O’ConnellC. Shore 1987 Language and communication in infancy

[8] Z. Bauman 2000 Liquid modernity 2000 Polity, Cambridge

[9] F. BioccaA. TangC. OwenF. Xiao 2006 Attention funnel: omnidirectional 3d cursor for mobile augmented reality platforms In Proceedings of the SIGCHI conference on Human Factors in computing systems 1115 1122 ACM

[10] R. A. Bolt 1980 Put-that-there: Voice and gesture at the graphics interface In Proceedings of the 7th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH ’80 New York, NY, USA 262 270 ACM

[11] F. Cairncross 2001 The death of distance: How the communications revolution is changing our lives Harvard Business Press

[12] G. Calbris 1990 The semiotics of French gestures 1900 Indiana Univ Pr

[13] M. CarpenterK. NagellM. TomaselloG. ButterworthC. Moore 1998 Social cognition, joint attention, and communicative competence from 9 to 15 months of age Monographs of the society for research in child development, i–174

[14] S. D’AngeloD. Gergle 2016 Gazed and confused: Understanding and designing shared gaze for remote collaboration In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, CHI ’16 New York, NY, USA 2492 2496 ACM

[15] H. Diessel 1999 Demonstratives: Form, function and grammaticalization 42 John Benjamins Publishing

[16] H. Diessel 2006 Demonstratives, joint attention, and the emergence of grammar Cognitive linguistics 17 4 463 489

[17] Economic Commission for Europe, Transport Division 2007 Convention on road signs and signals of 1968; European agreement supplementing the convention and protocol on road markings, additional to the European agreement ix 276

[18] D. Efron 1972 Gesture, race and culture JSTOR

[19] C. Egido 1988 Video conferencing as a technology to support group work: a review of its failures In Proceedings of the 1988 ACM conference on Computersupported cooperative work 13 24 ACM

[20] C. Egido 1990 Teleconferencing as a technology to support cooperative work: Its possibilities and limitations

[21] P. Eisert 2003 Immersive 3d video conferencing: challenges, concepts, and implementation In VCIP 69 79

[22] P. EkmanW. V. Friesen 1969 The repertoire of nonverbal behavior: Categories, origins, usage, and coding semiotica 1 1 49 98

[23] T. FechnerD. WilhelmC. Kray 2015 Ethermap: real-time collaborative map editing In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems 3583 3592 ACM

[24] H. Feldman 1986 A grammar of awtuw.(= pacific linguistics series b 94) Canberra: Australian National University

[25] F. FrancoG. Butterworth 1996 Pointing and social awareness: Declaring and requesting in the second year Journal of child language 23 2 307 336

[26] S. R. FussellL. D. SetlockJ. YangJ. OuE. MauerA. D. Kramer 2004 Gestures over video streams to support remote collaboration on physical tasks HumanComputer Interaction 19 3 273 309

[27] S. GauglitzC. LeeM. TurkT. Höllerer 2012 Integrating the physical environment into mobile remote collaboration In Proceedings of the 14th international conference on Human-computer interaction with mobile devices and services 241 250 ACM

[28] S. GauglitzB. NuernbergerM. TurkT. Höllerer 2014 World-stabilized annotations and virtual scene navigation for remote collaboration In Proceedings of the 27th annual ACM symposium on User interface software and technology 449 459 ACM

[29] A. Giddens 1984 The constitution of society: Outline of the theory of structuration Univ of California Press

[30] J. A. GrahamM. Argyle 1975 A cross-cultural study of the communication of extra-verbal meaning by gestures International Journal of Psychology 10 1 57 67

[31] F. HaqueM. NancelD. Vogel 2015 Myopoint: Pointing and clicking using forearm mounted electromyography and inertial motion sensors In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems 3653 3656 ACM

[32] D. Harvey 1989 The condition of postmodernity: An enquiry into the origins of social change Malden, MA: Blackwell

[33] M. HegartyA. E. RichardsonD. R. MontelloK. LovelaceI. Subbiah 2002 Development of a self-report measure of environmental spatial ability Intelligence 30 5 425 447

[34] G. W. Hewes 1981 Pointing and language Advances in Psychology 7 263 269

[35] G. W. Hewes 1996 A history of the study of language origins and the gestural primacy hypothesis Handbook of human symbolic evolution 571 595

[36] K. HirataY. HaradaT. TakadaN. YamashitaS. AoyagiY. ShiraiK. KajiJ. YamatoK. Nakazawa 2008 Video communication system supporting spatial cues of mobile users Proc. CollabTech 122 127

[37] E. A. IsaacsJ. C. Tang 1994 What video can and cannot do for collaboration: a case study Multimedia systems 2 2 63 73

[38] O. B. KaulM. PfeifferM. Rohs 2016 Follow the force: Steering the index finger towards targets using ems In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems EA ’16, New York, NY, USA 2526 2532 ACM

[39] A. KendonL. Versante 2003 Pointing by hand in neapolitan Pointing: where language, culture, and cognition meet 109 137

[40] S. Kern 1983 The culture of time and space Cambridge, MA: Harvard UP

[41] D. KirkT. RoddenD. S. Fraser 2007 Turn it this way: grounding collaborative action with remote gestures In Proceedings of the SIGCHI conference on Human Factors in Computing Systems 1039 1048 ACM

[42] S. Kita 2003 Pointing: Where language, culture, and cognition meet Psychology Press

[43] S. KitaA. Özyürek 2003 What does cross-linguistic variation in semantic coordination of speech and gesture reveal?: Evidence for an interface representation of spatial thinking and speaking Journal of Memory and language 48 1 16 32

[44] Y. LiuY. GuoC. Liang 2008 A survey on peer-topeer video streaming systems Peer-to-peer Networking and Applications 1 1 18 28

[45] S. MayerK. WolfS. SchneegassN. Henze 2015 Modeling distant pointing for compensating systematic displacements In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems 4165 4168 ACM

[46] D. McNeill 1992 Hand and mind: What gestures reveal about thought University of Chicago press

[47] D. McNeill 2008 Gesture and thought University of Chicago press

[48] S. MitraT. Acharya 2007 Gesture recognition: A survey IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 37 3 311 324

[49] O. MorikawaT. Maesako 1998 Hypermirror: toward pleasant-to-use video mediated communication system In Proceedings of the 1998 ACM conference on Computer supported cooperative work 149 158 ACM

[50] D. Morris 1978 Manwatching: A fold guide to human behaviour

[51] S. Navas MedranoM. PfeifferC. Kray 2017 Enabling remote deictic communication with mobile devices: An elicitation study In Proceedings of the 19th International Conference on HumanComputer Interaction with Mobile Devices and Services, MobileHCI ’17, New York, NY, USA 19:1 19:13 ACM

[52] D. J. PovinelliD. R. Davis 1994 Differences between chimpanzees (pan troglodytes) and humans (homo sapiens) in the resting state of the index finger: Implications for pointing Journal of Comparative Psychology 108 2134

[53] F. H. RauscherR. M. KraussY. Chen 1996 Gesture, speech, and lexical access: The role of lexical movements in speech production Psychological Science 7 4 226 231

[54] S. S. RautarayA. Agrawal 2015 Vision based hand gesture recognition for human computer interaction: a survey Artificial Intelligence Review 43 1 1 54

[55] L. Rolfe 1996 Theoretical stages in the prehistory of grammar Handbook of human symbolic evolution 776 792

[56] M. SeyfeddinipurS. KitaC. CaveI. GuaitellaS. Santi 2001 Gesture and dysfluency in speech Oralite et gesturalite: Interactions et comportemetns multimodaux dans la communication 266 270

[57] J. Sherzer 1973 Verbal and nonverbal deixis: The pointed lip gesture among the san blas cuna Language in Society 2 1 117 131

[58] J. Sherzer 1993 Pointed lips, thumbs up, and cheek puffs: Some emblematic gestures in social interactional and ethnographic context In Texas linguistic forum 33 196 211 University of Texas, Department of Linguistics

[59] K. TsukadaM. Yasumura 2004 Activebelt: Belttype wearable tactile display for directional navigation In UbiComp 3205 384 399 Springer

[60] D. Wilkins 2003 Why pointing with the index finger is not a universal (in sociocultural and semiotic terms) Pointing: Where language, culture, and cognition meet 171 215

[61] N. WongC. Gutwin 2010 Where are you pointing?: the accuracy of deictic pointing in cves In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems 1029 1038 ACM

[62] W. Wundt 1973 The language of gestures 6 Walter de Gruyter

Celebrating 65 years of The Computer Journal - free-to-read perspectives - bcs.org/tcj65