Beyond Questionnaires: Innovative Approaches to Evaluating Mixed Reality

Mixed Reality (MR) technologies, including augmented and virtual reality, are increasingly used in a number of sectors, thanks to their capabilities to immerse users in multisensory interaction environments. However, as indicated by some systematic reviews, questionnaire remains the main method for evaluating the interaction quality of MR. There is a lack of innovative approaches addressing unique features of MR. It can dampen the advances of MR, as evaluation feedback can inform its future development. In this workshop we aim to explore this issue by inviting participants to share their practical experiences or conceptual ideas of evaluating MR in various contexts, using different methods and tools. The ultimate aim is to produce a research agenda on this topic for the community to examine it further in the future.

Content

Author and article information

Contributors

Effie L-C. Law

Santawat Thanydait

Matthias Heintz

Abey Campbell

Fridolin Wild

Conference

Publication date (Print): July 2021

Pages: 1-5

Affiliations

[0001]University of Leicester, UK

[0002]HKUST, China

[0003]University College Dublin, Ireland

[0004]Open University, UK

Article

DOI: 10.14236/ewic/HCI2021-W2.0

SO-VID: 5104c2b3-79a1-44d2-9299-8cc433b53b50

License:

This work is licensed under a Creative Commons Attribution 4.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/

Conference name: 34th British HCI Workshop and Doctoral Consortium

Conference acronym: HCI2021-WDC

Conference number: 34

Conference location: London, UK

Conference date: 20th - 21st July 2021

Conference sponsor: Electronic Workshops in Computing (eWiC)

Conference theme: Post-pandemic HCI – Living Digitally

History

Product

1477-9358 BCS Learning & Development

Self URI (article page): https://www.scienceopen.com/hosted-document?doi=10.14236/ewic/HCI2021-W2.0

Self URI (journal page): https://ewic.bcs.org/

REFERENCES

M AkçayırG Akçayır 2017 Advantages and challenges associated with augmented reality for education: A systematic review of the literature Educational Research Review, 20 1 11
ARETE: Augmented Reality Interactive Educational Systems https://www.areteproject.eu/
R Azuma 1997 A Survey of Augmented Reality Presence: Teleoperators and Virtual Environments 6 4 August 1997 355 385
A BardeI GumilarA. F HayatiA DeyG LeeM Billinghurst 2020 A Review of Hyperscanning and Its Use in Virtual Environments Informatics 55
K ChowC CoyiutoC NguyenD Yoon 2019 Challenges and Design Considerations for Multimodal Asynchronous Collaboration in VR Proceedings of the ACM on Human-Computer Interaction 3 1 24
A DeyH ChenC ZhuangM BillinghurstR. W Lindeman 2018 Effects of sharing real-time multi-sensory heart rate feedback in different immersive collaborative virtual environments IEEE International Symposium on Mixed and Augmented Reality (ISMAR) 165 173
A DeyT PiumsomboonY LeeM Billinghurst 2017 Effects of sharing physiological states of players in a collaborative virtual reality gameplay Proceedings of Conference on Human Factors in Computing Systems (CHI’17) 4045 4056
Y. M KimI RhiuM. H Yun 2020 A systematic review of a virtual reality system from the perspective of user experience International Journal of Human–Computer Interaction 36 10 893 910
M KytöB EnsT PiumsomboonG.A LeeM Billinghurst 2018 Pinpointing: Precise head-and eye-based target selection for augmented reality Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems 1 14
E. L. C LawV RotoM HassenzahlA. P VermeerenJ Kort 2009 Understanding, scoping and defining user experience: a survey approach Proceedings of the SIGCHI Conference on Human Factors in Computing Systems 719 728
Y LeeK MasaiK KunzeM SugimotoM Billinghurst 2016 A Remote Collaboration System with Empathy Glasses IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct) Merida, Mexico 2016 342 343
K Leswing 2021 The tech industry is looking to replace the smartphone https://www.cnbc.com/2021/02/20/apple-facebook-microsoft-battle-to-replace-smartphone-with-ar.html 29/03/21
K MasaiK KunzeM SugimotoM Billinghurst 2016 Empathy Glasses Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems New York, NY, USA 2016 ACM 1257 1263
S MasneriA DominguezF WildJ PronkM HeintzJ TiedeA NistorG ChiazzeseE Mangina 2020 ARETE - An Interactive Educational System using Augmented Reality Proceedings of the 6th International Conference of the Immersive Learning Research Network (iLRN) 283 286
P MilgramH TakemuraA UtsumiF Kishino 1994 Augmented reality: A class of displays on the reality-virtuality continuum Telemanipulator and Telepresence Technologies 2351 [Cross Ref]
H NguyenC PontonnierS HiltT DuvalG Dumont 2017 VR-based operating modes and metaphors for collaborative ergonomic design of industrial workstations Journal on Multimodal User Interfaces 11 97 111
J. V. V ParrR. J FosterG WoodN. M ThomasM. A Hollands 2020 Children with developmental coordination disorder show altered visuomotor control during stair negotiation associated with heightened state anxiety Frontiers in Human Neuroscience 14 512
T PiumsomboonA DeyB EnsG LeeM Billinghurst 2019 The effects of sharing awareness cues in collaborative mixed reality Frontiers in Robotics and AI 6 5
C PontonnierT DuvalG Dumont 2014 Collaborative virtual environments for ergonomics: Embedding the design engineer role in the loop International Workshop on Collaborative Virtual Environments (3DCVE) 1 5
J RadiantiT. A MajchrzakJ FrommI Wohlgenannt 2020 A systematic review of immersive virtual reality applications for higher education: Design elements, lessons learned, and research agenda Computers & Education 147 103778
Y RahmanS. M AsishN. P FisherE. C BruceA. K KulshreshthC. W Borst 2020 Exploring eye gaze visualization techniques for identifying distracted students in educational VR IEEE Conference on Virtual Reality and 3D User Interfaces (VR) 868 877
S SamroseD McDuffR Sim 2021 MeetingCoach: An Intelligent Dashboard for Supporting Effective & Inclusive Meetings Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’21)
M. E. C SantosA ChenT TaketomiG YamamotoJ MiyazakiH Kato 2013 Augmented reality learning experiences: Survey of prototype design and evaluation IEEE Transactions on Learning Technologies 7 38 56
M SpeicherB HallM Nebeling 2019 What is Mixed Reality? CHI 2019 ACM New York, NY, USA [Cross Ref]
VOSTARS Video and Optical See Through Augmented Reality Surgical Systems https://www.vostars.eu/
Y YanC YuX MaS HuangH IqbalY Shi 2018 Eyes-free target acquisition in interaction space around the body for virtual reality Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems 1 13
B YoonH KimS. Y OhW Woo 2020 Evaluating remote virtual hands models on social presence in hand-based 3D remote collaboration IEEE International Symposium on Mixed and Augmented Reality (ISMAR) 520 532
H ZengX ShuY WangY WangL ZhangT. C PongH Qu 2020 EmotionCues: Emotion-oriented visual summarization of classroom videos IEEE Transactions on Visualization and Computer Graphics

Comments

Comment on this article

[1] M AkçayırG Akçayır 2017 Advantages and challenges associated with augmented reality for education: A systematic review of the literature Educational Research Review, 20 1 11

[2] ARETE: Augmented Reality Interactive Educational Systems https://www.areteproject.eu/

[3] R Azuma 1997 A Survey of Augmented Reality Presence: Teleoperators and Virtual Environments 6 4 August 1997 355 385

[4] A BardeI GumilarA. F HayatiA DeyG LeeM Billinghurst 2020 A Review of Hyperscanning and Its Use in Virtual Environments Informatics 55

[5] K ChowC CoyiutoC NguyenD Yoon 2019 Challenges and Design Considerations for Multimodal Asynchronous Collaboration in VR Proceedings of the ACM on Human-Computer Interaction 3 1 24

[6] A DeyH ChenC ZhuangM BillinghurstR. W Lindeman 2018 Effects of sharing real-time multi-sensory heart rate feedback in different immersive collaborative virtual environments IEEE International Symposium on Mixed and Augmented Reality (ISMAR) 165 173

[7] A DeyT PiumsomboonY LeeM Billinghurst 2017 Effects of sharing physiological states of players in a collaborative virtual reality gameplay Proceedings of Conference on Human Factors in Computing Systems (CHI’17) 4045 4056

[8] Y. M KimI RhiuM. H Yun 2020 A systematic review of a virtual reality system from the perspective of user experience International Journal of Human–Computer Interaction 36 10 893 910

[9] M KytöB EnsT PiumsomboonG.A LeeM Billinghurst 2018 Pinpointing: Precise head-and eye-based target selection for augmented reality Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems 1 14

[10] E. L. C LawV RotoM HassenzahlA. P VermeerenJ Kort 2009 Understanding, scoping and defining user experience: a survey approach Proceedings of the SIGCHI Conference on Human Factors in Computing Systems 719 728

[11] Y LeeK MasaiK KunzeM SugimotoM Billinghurst 2016 A Remote Collaboration System with Empathy Glasses IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct) Merida, Mexico 2016 342 343

[12] K Leswing 2021 The tech industry is looking to replace the smartphone https://www.cnbc.com/2021/02/20/apple-facebook-microsoft-battle-to-replace-smartphone-with-ar.html 29/03/21

[13] K MasaiK KunzeM SugimotoM Billinghurst 2016 Empathy Glasses Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems New York, NY, USA 2016 ACM 1257 1263

[14] S MasneriA DominguezF WildJ PronkM HeintzJ TiedeA NistorG ChiazzeseE Mangina 2020 ARETE - An Interactive Educational System using Augmented Reality Proceedings of the 6th International Conference of the Immersive Learning Research Network (iLRN) 283 286

[15] P MilgramH TakemuraA UtsumiF Kishino 1994 Augmented reality: A class of displays on the reality-virtuality continuum Telemanipulator and Telepresence Technologies 2351 [Cross Ref]

[16] H NguyenC PontonnierS HiltT DuvalG Dumont 2017 VR-based operating modes and metaphors for collaborative ergonomic design of industrial workstations Journal on Multimodal User Interfaces 11 97 111

[17] J. V. V ParrR. J FosterG WoodN. M ThomasM. A Hollands 2020 Children with developmental coordination disorder show altered visuomotor control during stair negotiation associated with heightened state anxiety Frontiers in Human Neuroscience 14 512

[18] T PiumsomboonA DeyB EnsG LeeM Billinghurst 2019 The effects of sharing awareness cues in collaborative mixed reality Frontiers in Robotics and AI 6 5

[19] C PontonnierT DuvalG Dumont 2014 Collaborative virtual environments for ergonomics: Embedding the design engineer role in the loop International Workshop on Collaborative Virtual Environments (3DCVE) 1 5

[20] J RadiantiT. A MajchrzakJ FrommI Wohlgenannt 2020 A systematic review of immersive virtual reality applications for higher education: Design elements, lessons learned, and research agenda Computers & Education 147 103778

[21] Y RahmanS. M AsishN. P FisherE. C BruceA. K KulshreshthC. W Borst 2020 Exploring eye gaze visualization techniques for identifying distracted students in educational VR IEEE Conference on Virtual Reality and 3D User Interfaces (VR) 868 877

[22] S SamroseD McDuffR Sim 2021 MeetingCoach: An Intelligent Dashboard for Supporting Effective & Inclusive Meetings Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’21)

[23] M. E. C SantosA ChenT TaketomiG YamamotoJ MiyazakiH Kato 2013 Augmented reality learning experiences: Survey of prototype design and evaluation IEEE Transactions on Learning Technologies 7 38 56

[24] M SpeicherB HallM Nebeling 2019 What is Mixed Reality? CHI 2019 ACM New York, NY, USA [Cross Ref]

[25] VOSTARS Video and Optical See Through Augmented Reality Surgical Systems https://www.vostars.eu/

[26] Y YanC YuX MaS HuangH IqbalY Shi 2018 Eyes-free target acquisition in interaction space around the body for virtual reality Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems 1 13

[27] B YoonH KimS. Y OhW Woo 2020 Evaluating remote virtual hands models on social presence in hand-based 3D remote collaboration IEEE International Symposium on Mixed and Augmented Reality (ISMAR) 520 532

[28] H ZengX ShuY WangY WangL ZhangT. C PongH Qu 2020 EmotionCues: Emotion-oriented visual summarization of classroom videos IEEE Transactions on Visualization and Computer Graphics

Studying business & IT? Drive your professional career forwards with BCS books - for a 20% discount click here: shop.bcs.org