Support Rather Than Assault – Cooperative Agents in Minecraft

With the dominant trope of the computer as adversary rather than enabler, reinforcement learning for games has mainly focused on the ability of agents to compete and win. Although cooperation is a product of learning, of understanding the player’s requirements and applying agents’ competences to fulfil them, there has been little investigation of reinforcement learning for cooperation in games. Reinforcement learning results in the agent adapting and changing, however, there are concerns that such adaptivity could alienate users if their cooperative agent outperforms them. To explore this, the paper outlines the development and training of cooperative agents reporting users’ positive response to adaptive cooperation in games.

Content

Author and article information

Contributors

Danielle Potts

Kate MacFarlane

Lynne Hall

Conference

Publication date: July 2021

Publication date (Print): July 2021

Pages: 133-138

Affiliations

[0001]Accenture UK

Sunderland, SR68JP

[0002]University of Sunderland

Sunderland, SR60DD

Article

DOI: 10.14236/ewic/HCI2021.13

SO-VID: 5ad47f0a-0782-4e8b-9001-ae9220d581f7

License:

This work is licensed under a Creative Commons Attribution 4.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/

Conference name: 34th British HCI Conference

Conference acronym: HCI2021

Conference number: 34

Conference location: London, UK

Conference date: 20th - 21st July 2021

Conference sponsor: Electronic Workshops in Computing (eWiC)

Conference theme: Post-pandemic HCI – Living Digitally

History

Product

1477-9358 BCS Learning & Development

Self URI (article page): https://www.scienceopen.com/hosted-document?doi=10.14236/ewic/HCI2021.13

Self URI (journal page): https://ewic.bcs.org/

REFERENCES

K. Arulkumaran, M. Deisenroth, M. Brundage and A, Bharath “Deep Reinforcement Learning: A Brief Survey”. IEEE Signal Processing Magazine, 34(6), 2017, pp.26-38, Doi: 10.1109/MSP.2017.2743240
P. Barros, A. Tanevska and A. Sciutti, 2020. “Learning from Learners: Adapting Reinforcement Learning Agents to be Competitive in a Card Game”, viewed 10 May 2021 https://arxiv.org/abs/2004.04000
J. Clement, 2021, Minecraft number of players worldwide. Statista. Available at: https://www.statista.com/statistics/680139/minecraf t-active-players-worldwide/ (accessed 5.10.21).
M. Dann, F. Zambetta and J. Thangarajah, 2018. “Integrating Skills and Simulation to Solve Complex Navigation Tasks in Infinite Mario”, IEEE Transactions on Games, 10(1), 2018, pp.101-106, doi: 10.1109/TCIAIG.2017.2696045
P. Daugherty and J. Euchner, 2020. “Human + Machine: Collaboration in the Age of AI”, Research-Technology Management, 63(2), 2020, pp.12-17, doi: [Cross Ref]
D. de Almeida Rocha and J. Cesar Duarte, 2019. “Simulating Human Behaviour in Games using Machine Learning”, 2019 18th Brazilian Symposium on Computer Games and Digital Entertainment (SBGames), 2019, doi:10.1109/SBGames.2019.00030
A. Efros, T. Griffiths, D. Pathak, P. Agrawal and R. Dubey, 2018. “Investigating Human Priors for Playing Video Games”, International Conference on Machine Learning, viewed 19 December 2020, https://arxiv.org/abs/1802.10217
C. Gough, 2020. Minecraft Unit Sales Worldwide 2020 | Statista. Available at: https://www.statista.com/statistics/680124/minecraf t-unit-sales-worldwide/#:~:text=Minecraft%20unit%20sales%20 worldwide%202016%2D2020&text=Sinc e%20its%20release%20in%202011,and%20Grand %20Theft%20Auto%20V (Accessed: 19 December 2020)
R. Hara, 2020. The Last of Us 2 Dev Reveals Major AI Improvements. Game Rant. Available at: https://gamerant.com/last-of-us-2-ai-improvements/ (Accessed: 10 May 2021).
T. Hester, M. Vecerik, O. Pietquin, M. Lanctot, T. Schaul, B. Piot, D. Horgan, J. Quan, A. Sendonaris, G. Dulac-Arnold, I. Osband, J. Agapiou, J. Leibo and A. Gruslys, 2017. “Deep Q-learning from Demonstrations”, Association for the Advancement of Artificial Intelligence, viewed 19 December 2020, <https://arxiv.org/abs/1704.03732>
M. Jaderberg, V. Mnih, W. Czarnecki, T. Schaul, J. Leibo, D. Silver and K. Kavukcuoglu, 2016. “Reinforcement Learning with Unsupervised Auxiliary Tasks”, DeepMind, viewed 19 December 2020, https://arxiv.org/abs/1611.05397
Home, 2021. The Elder Scrolls Online. Available at: https://www.elderscrollsonline.com/en-us/home (accessed 5.10.21).
J. Kahn, 2017. Bloomberg - Are You A Robot? Bloomberg.com Available at: https://www.bloomberg.com/news/articles/2017-03-31/video-games-without-the-players (Accessed 19 December 2020).
J. Kaliappan and K. Sundararajan, 2020. “ Machine Learning in Video Games. Handbook of Research on Emerging Trends and Applications of Machine Learning", 2020, pp.425-443, doi:10.4018/978-1-5225-9643-1
K Kunanusont, SM Lucas, D Perez-Liebana (2017). General Video Game AI: Learning from screen capture. DOI: 10.1109/CEC.2017.7969556
S. Milani, N. Topin, B. Houghton, W. Guss, S. Mohanty, K. Nakata, O. Vinyals and N. Kuno, 2020. “Retrospective Analysis of the 2019 MineRL Competition on Sample Efficient Reinforcement Learning”, viewed 19 December 2020 <https://arxiv.org/abs/2003.05012>
A. Nair, B. McGrew, M. Andrychowicz, W. Zaremba and P. Abbeel, 2018.” Overcoming Exploration in Reinforcement Learning with Demonstrations”, 2018 IEEE International Conference on Robotics and Automation (ICRA), 2018, doi:10.1109/ICRA.2018.8463162
K. Orland, 2013. How Forza 5 and the Xbox One Use the Cloud to Drive Machine-Learning AI. Ars Technica. Available at: https://arstechnica.com/gaming/2013/10/how-forza-5-and-the-xbox-one-use-the-cloud-to-drive-machine-learning-ai/ (Accessed 19 December 2020)
D Perez, J Liu, A Abdel Samea Khalifaet al. (2019). General Video Game AI: a Multi-Track Framework for Evaluating Agents, Games and Content Generation Algorithms. DOI:10.1109/tg.2019.2901021
M. Reynard, H. Kamper, H. Engelbrecht and B. Rosman, 2020. “Combining primitive DQNs for improved reinforcement learning in Minecraft”. 2020 International SAUPEC/RobMech/PRASA Conference, 2020, doi:10.1109/SAUPEC/RobMech/PRASA48453.2020.9041025
M. Riedmiller, R. Hafner, T. Lampe, M. Neunert, J. Degrave, T. Van de Wiele, V. Mnih, N. Heess and T. Springenberg, 2018. “Learning by Playing – Solving Sparse Reward Tasks from Scratch”, Google Deepmind, viewed 21 May 2020, https://arxiv.org/abs/1802.10567
C. Scheller, Y. Schraner and M. Vogel, 2020. “Sample Efficient Reinforcement Learning through Learning from Demonstrations in Minecraft”. NeurIPS2019 Competition & Demonstration Track, viewed 19 December 2020 <https://arxiv.org/abs/2003.06066>
D. Silver, A. Guez and H. Hasselt, 2016. “Deep Reinforcement Learning with Double Q- learning”, Association for the Advancement of Artificial Intelligence, viewed 19 December 2020, <https://arxiv.org/abs/1509.06461>
Star Wars: The Old Republic Available at: https://www.swtor.com/ (accessed 5.10.21)
R. Sutton, F. Bach and A. Barto, 2018. Reinforcement Learning. Massachusetts: MIT Press Ltd, ISBN: 978026203924
V. Velardo, 2019. The Reality of Red Dead Redemption 2’S AI (Part 1). Medium. Available at: https://medium.com/the-sound-of-ai/the-reality-of-red-dead-redemption-2s-ai-part-1-c276e9da2763 (Accessed 20 December 2020).
A. Walsh, 2018 Forza Motorsport 7 Drivatars Being Improved, Race Adjudication Pushed To 2019 – Fullthrottle Media. Available at: Fullthrottlemedia.co.uk (Accessed 19 March 2020)
L. Xu and Y. Chen, 2019. A Hierarchical Approach for MARLÖ Challenge. 2019 IEEE Conference on Games (CoG), 2019, doi: 10.1109/CIG.2019.8847943

Comments

Comment on this article

[1] K. Arulkumaran, M. Deisenroth, M. Brundage and A, Bharath “Deep Reinforcement Learning: A Brief Survey”. IEEE Signal Processing Magazine, 34(6), 2017, pp.26-38, Doi: 10.1109/MSP.2017.2743240

[2] P. Barros, A. Tanevska and A. Sciutti, 2020. “Learning from Learners: Adapting Reinforcement Learning Agents to be Competitive in a Card Game”, viewed 10 May 2021 https://arxiv.org/abs/2004.04000

[3] J. Clement, 2021, Minecraft number of players worldwide. Statista. Available at: https://www.statista.com/statistics/680139/minecraf t-active-players-worldwide/ (accessed 5.10.21).

[4] M. Dann, F. Zambetta and J. Thangarajah, 2018. “Integrating Skills and Simulation to Solve Complex Navigation Tasks in Infinite Mario”, IEEE Transactions on Games, 10(1), 2018, pp.101-106, doi: 10.1109/TCIAIG.2017.2696045

[5] P. Daugherty and J. Euchner, 2020. “Human + Machine: Collaboration in the Age of AI”, Research-Technology Management, 63(2), 2020, pp.12-17, doi: [Cross Ref]

[6] D. de Almeida Rocha and J. Cesar Duarte, 2019. “Simulating Human Behaviour in Games using Machine Learning”, 2019 18th Brazilian Symposium on Computer Games and Digital Entertainment (SBGames), 2019, doi:10.1109/SBGames.2019.00030

[7] A. Efros, T. Griffiths, D. Pathak, P. Agrawal and R. Dubey, 2018. “Investigating Human Priors for Playing Video Games”, International Conference on Machine Learning, viewed 19 December 2020, https://arxiv.org/abs/1802.10217

[8] C. Gough, 2020. Minecraft Unit Sales Worldwide 2020 | Statista. Available at: https://www.statista.com/statistics/680124/minecraf t-unit-sales-worldwide/#:~:text=Minecraft%20unit%20sales%20 worldwide%202016%2D2020&text=Sinc e%20its%20release%20in%202011,and%20Grand %20Theft%20Auto%20V (Accessed: 19 December 2020)

[9] R. Hara, 2020. The Last of Us 2 Dev Reveals Major AI Improvements. Game Rant. Available at: https://gamerant.com/last-of-us-2-ai-improvements/ (Accessed: 10 May 2021).

[10] T. Hester, M. Vecerik, O. Pietquin, M. Lanctot, T. Schaul, B. Piot, D. Horgan, J. Quan, A. Sendonaris, G. Dulac-Arnold, I. Osband, J. Agapiou, J. Leibo and A. Gruslys, 2017. “Deep Q-learning from Demonstrations”, Association for the Advancement of Artificial Intelligence, viewed 19 December 2020, <https://arxiv.org/abs/1704.03732>

[11] M. Jaderberg, V. Mnih, W. Czarnecki, T. Schaul, J. Leibo, D. Silver and K. Kavukcuoglu, 2016. “Reinforcement Learning with Unsupervised Auxiliary Tasks”, DeepMind, viewed 19 December 2020, https://arxiv.org/abs/1611.05397

[12] Home, 2021. The Elder Scrolls Online. Available at: https://www.elderscrollsonline.com/en-us/home (accessed 5.10.21).

[13] J. Kahn, 2017. Bloomberg - Are You A Robot? Bloomberg.com Available at: https://www.bloomberg.com/news/articles/2017-03-31/video-games-without-the-players (Accessed 19 December 2020).

[14] J. Kaliappan and K. Sundararajan, 2020. “ Machine Learning in Video Games. Handbook of Research on Emerging Trends and Applications of Machine Learning", 2020, pp.425-443, doi:10.4018/978-1-5225-9643-1

[15] K Kunanusont, SM Lucas, D Perez-Liebana (2017). General Video Game AI: Learning from screen capture. DOI: 10.1109/CEC.2017.7969556

[16] S. Milani, N. Topin, B. Houghton, W. Guss, S. Mohanty, K. Nakata, O. Vinyals and N. Kuno, 2020. “Retrospective Analysis of the 2019 MineRL Competition on Sample Efficient Reinforcement Learning”, viewed 19 December 2020 <https://arxiv.org/abs/2003.05012>

[17] A. Nair, B. McGrew, M. Andrychowicz, W. Zaremba and P. Abbeel, 2018.” Overcoming Exploration in Reinforcement Learning with Demonstrations”, 2018 IEEE International Conference on Robotics and Automation (ICRA), 2018, doi:10.1109/ICRA.2018.8463162

[18] K. Orland, 2013. How Forza 5 and the Xbox One Use the Cloud to Drive Machine-Learning AI. Ars Technica. Available at: https://arstechnica.com/gaming/2013/10/how-forza-5-and-the-xbox-one-use-the-cloud-to-drive-machine-learning-ai/ (Accessed 19 December 2020)

[19] D Perez, J Liu, A Abdel Samea Khalifaet al. (2019). General Video Game AI: a Multi-Track Framework for Evaluating Agents, Games and Content Generation Algorithms. DOI:10.1109/tg.2019.2901021

[20] M. Reynard, H. Kamper, H. Engelbrecht and B. Rosman, 2020. “Combining primitive DQNs for improved reinforcement learning in Minecraft”. 2020 International SAUPEC/RobMech/PRASA Conference, 2020, doi:10.1109/SAUPEC/RobMech/PRASA48453.2020.9041025

[21] M. Riedmiller, R. Hafner, T. Lampe, M. Neunert, J. Degrave, T. Van de Wiele, V. Mnih, N. Heess and T. Springenberg, 2018. “Learning by Playing – Solving Sparse Reward Tasks from Scratch”, Google Deepmind, viewed 21 May 2020, https://arxiv.org/abs/1802.10567

[22] C. Scheller, Y. Schraner and M. Vogel, 2020. “Sample Efficient Reinforcement Learning through Learning from Demonstrations in Minecraft”. NeurIPS2019 Competition & Demonstration Track, viewed 19 December 2020 <https://arxiv.org/abs/2003.06066>

[23] D. Silver, A. Guez and H. Hasselt, 2016. “Deep Reinforcement Learning with Double Q- learning”, Association for the Advancement of Artificial Intelligence, viewed 19 December 2020, <https://arxiv.org/abs/1509.06461>

[24] Star Wars: The Old Republic Available at: https://www.swtor.com/ (accessed 5.10.21)

[25] R. Sutton, F. Bach and A. Barto, 2018. Reinforcement Learning. Massachusetts: MIT Press Ltd, ISBN: 978026203924

[26] V. Velardo, 2019. The Reality of Red Dead Redemption 2’S AI (Part 1). Medium. Available at: https://medium.com/the-sound-of-ai/the-reality-of-red-dead-redemption-2s-ai-part-1-c276e9da2763 (Accessed 20 December 2020).

[27] A. Walsh, 2018 Forza Motorsport 7 Drivatars Being Improved, Race Adjudication Pushed To 2019 – Fullthrottle Media. Available at: Fullthrottlemedia.co.uk (Accessed 19 March 2020)

[28] L. Xu and Y. Chen, 2019. A Hierarchical Approach for MARLÖ Challenge. 2019 IEEE Conference on Games (CoG), 2019, doi: 10.1109/CIG.2019.8847943

Studying business & IT? Drive your professional career forwards with BCS books - for a 20% discount click here: shop.bcs.org