Designing A Musical Game of Shogi

This paper will discuss the design and implementation of the Shogi board, an interface built to allow members of the general public to interact with various Japanese instruments in a novel way. The Shogi board brings elements of Japanese culture to the attention of the general public in England ahead of the season of culture commencing in 2019. This interface utilizes maker technology to create an accessible method for members of the general public to compose music with traditional instruments in public settings.


INTRODUCTION
Announced in August 2017 1 , the months between Autumn 2019 and Summer 2020 will see a 'Season of Culture' between Japan and the United Kingdom curated by the British Council in partnership with the Japanese government.Two of the British Council's main themes for this cultural exchange are 'technology & innovation' and 'art in public spaces'.The British Council also promotes the development of 'Creative Music Education' as a bridge between the United Kingdom and Japan.The British Council is the UK's leading international organisation for cultural relations and educational opportunities and this 'Season of Culture' initiative is designed to promote the flow and exchange of ideas, and generation of trust, between the people of Japan and the UK.
In the build up to the cultural exchange between Japan and the United Kingdom, our research team are investigating different methods of introducing elements of Japanese culture into the eyes of the general public, specifically across the North West of England.As part of our team's established 'Make-It-Yourself: Building New Musical Experiences'' project, we ran co-design workshops, inviting members of the general public to learn different fundamentals of musical technology, specifically around interactions and designing accessible instruments.A main objective of these workshops was to design interactive, accessible musical interfaces as means of playing back and composing using a series of traditional Japanese instruments 1 https://www.britishcouncil.jp/en/uk-japan-2019-20that were collated as part of our digital sound archive.As part of these workshops, the authors designed and built a Tangible User Interface that incorporates various elements of traditional Japanese culture, as a means of demonstrating the concept behind the workshops to participants.This interface uses nine different instruments recorded as part of a Japanese sound archive, and presents them as a playful, innovative instrument based around the Japanese game of 'Shogi'.Shogi, similar to chess, is an ancient traditional strategy based board game dating back to the 10th century, that uses a variety of different pieces on a grid.The musical Shogi board utilizes a vision-led system based in the programming language 'Python', that tracks the position of both player's pieces as the game progresses.

RELATED WORK
Tangible User Interfaces are currently used for a wide range of purposes, including both education and entertainment (Shaer, 2009).Studies have suggested that Tangible User Interfaces have the potential ability to enhance both the teaching and learning of different materials in a wide range of contexts, most recently in the classrooms of a higher education institution (Raffaele, 2017).Developing musical experiences with either 'playful' and 'gameful' characteristics, or basing interfaces on existing games and pastimes is an area of interest within both the music technology and Human Computer Interaction community.By developing interfaces that use novel methods of input and interaction, developers can study the differing ways in which participants access and use the materials present, whether that be in a game, museum exhibition or musical interface.
Traditional tabletop games often use music and sound design as an extra element to enhance gameplay and immersion.Games such as 'Dungeons & Dragons' utilize music to intensify certain elements of the story or campaign being played out, which was developed further through the production of 'AmbiDice' (Berndt et al 2017).The AmbiDice were introduced into a regular Dungeons & Dragons game, and provided minimalist, ambient musical accompaniment to the gameplay, triggered every time a player rolled the dice to decide an action in the game.Consultations with the members of the D&D game suggested that the addition of a tangible, musical object into their regular gameplay provided an extra dimension to the game.Further to this, musical interfaces have also been integrated into digital versions of board games.A software system was developed which created a musical improvisation based on different representations of chess game states (Parson, 2009).'Music for 32 Chess Pieces' utilized mapping software within the chess game environment, to translate different positions and relationships between the 32 pieces of a chess game into varying musical structures, timbres and patterns.This system was produced as a software patch using MAX/MSP, and was intended to teach chess players, both novices and experts, about different fundamentals of western music theory.As the interface was solely developed as a computer patch rather than a physical tangible board, the musical experience was limited in its capabilities for public interaction, and was primarily experienced with a solo player.It is worth mentioning that participants often exhibit a higher level of engagement with tangible interfaces when in a collaborative setting; (Xie 2008) explores this through 'Are Tangibles More Fun?', where children's interactivity with physical, graphical, and tangible user interfaces were compared.Through the utilization of a toolkit such as "ToyVision" (Marco, 2013), there is scope to develop a project like 'Music for 32 Chess Pieces' into a physical, tangible board game.
Playful interactive musical interfaces often utilize the use of vision-led input, particular those built to track positions of tangible objects, and the distance or relationship between several objects.The Sound of Colour interface (Davenport, 2017) used a webcam placed overhead to track the position of multiple coloured balls, which were then mapped against an X/Y grid to control both the pitch and volume of a variety of synthesized software instruments.Similarly, the ReacTable (Jordà, 2017) is a step sequencer based tabletop interface that employs a video camera placed beneath the tabletop interface, which constantly tracks the position of the objects placed on the surface.The tangible objects placed on the tabletop surface controlled different elements of a modular synthesizer.
Step sequencer interfaces allow users to compose and create musical materials in a controlled environment, as the user is marginally restricted into the boundaries of pitch and rhythm, set by the step sequencer.The Tactus (Mann, 2009) step sequencer uses a vision-led system, and allows users to compose with a variety of musical materials, but led to a result which the authors describe as 'aleatoric music', which relates to the random element of using a step sequencer, commenting on the fact that an element of the composition is left up to chance.Often, interactive musical interfaces are not built for composition, but rather to play back musical files in more tangible ways, something that was addressed through the development of a novel camera based system, TMusic (Zhang, 2015).

DESIGN
In searching for ways to introduce members of the general public to different elements of Japanese culture, the authors first ran an 'ideation' session.Through this process, members of the team combined different ideas around the field of culture, tradition, maker technologies, music programming and tangible interfaces to form the idea of the Shogi board.This ideation session provided the bedrock from which concepts began to take form, combining ideas in new ways to create memory links between different aspects of culture and tradition with sonification of data.A purpose of this Shogi board was to act as a link between two cultures, bringing elements of Japanese culture to the general public of the North West of England, creating new synergies and embodying the season of culture.

Figure 1-Musical ideation session
Previously, several of the authors worked on the camera based musical interface the Sound of Colour, which took the form of an interactive tabletop device.Through this work, the authors were able to conclude that users preferred to use the tabletop interface as opposed to screen or button based controllers.It was decided that the Shogi board should be produced as a tabletop interface, to allow the system to be moved between location with ease, and allow members of the general public to have hands on interaction with the traditional sounds of Japan, which they would never come into contact with usually in their daily lives.Taking influence from 'Seiza', the Japanese custom of 'proper sitting' on the floor rather than chairs, it was decided that the Shogi board would be housed in wood box that could be situated on the floor and played easily.The Shogi board uses a Raspberry Pi 3 as its main controller.To detect the position of the game pieces, the Pi is connected to a Picamera and uses the OpenCV python library.OpenCV (Open Source Computer Vision Library) is an open source library used for real-time computer vision.The playing field itself consists of a sheet of transparent acrylic, allowing the camera to be situated beneath the playing surface and process the position of the pieces without obstruction from participant's hands.The pieces are 3D printed with the respective instrument engraved into it, which not only represents the instrument associated with the piece but also symbolizes the type of the piece in Shogi.In order to easily detect the pieces, each piece is marked using bespoke ArUco markers.These markers are generated and detected by the ArUco library, an open-source library based on OpenCV which is reliable, well documented and easy to use.Each of these markers are associated with an individual ID.This method of identifying pieces has been employed previously, with prosperous results (La Delfa, 2015), (Chewell, 2016).

Figure 3-Both sides of a shogi piece -featuring a Koto
For this project, 100 markers, with a size of 1cm, were generated using the DICT_6X6_100 dictionary.This dictionary was selected due to the fact that it uses close proximity to the amount of pieces required for the Shogi board.There are 40 basic pieces in Shogi which can be promoted individually to different ranks, so ultimately there could be 74 different pieces used throughout a game.The dictionary used allows the system to keep track of every individual piece.The last marker is also reserved for determining the playing field.To give additional visual feedback, there are LEDs placed beneath the playing field.These LEDs are placed in such a way that they do not obstruct the view of the board, but are placed to support the user by lighting up different pathways depending on the selected mode.When the system starts, it first initializes the playing field; this is done by by first locating the markers with ID '99'.It then finds the inner corners of these markers, which determine the corners of the playing field.

Figure 4-The view of the playing field
Once the playing field is initialized, the system waits until the user selects a mode of interaction.There are three possible modes for users to select between: shogi, step sequencer and musical theory mode.In musical theory mode, each square represents a note.The program checks each square separately in a snake-like motion and plays the note of a certain instrument, depending on which piece is present in that square.It allows the user to select a root note and show a scale, highlighted by the LEDs that make up the grid of the board.Step sequencer mode works as a normal step sequencer that plays back audio depending on where pieces are positioned in the rows of the board.As the 3D printed pieces represent different instruments, it is possible to compose with a combination of koto, shamisen, taiko, and shakuhachi notes.The instruments compiled in the archive have long been prominent in various elements of Japanese culture, particularly when accompanying visual performances or dramatised narratives, such as in the case of Bunraku; a traditional form of puppet theatre, and Kabuki; a form of classical dance and drama.Their diverse performance capabilities make them an incredibly emotive set of instruments, with the capabilities of supporting a broad range of visual mediums, whilst being equally competent and effective as part of a standalone musical ensemble.
The LEDs light up the row that the sequencer is currently playing.Shogi mode allows the user to play a traditional game of Shogi, and helps the player by giving visual feedback using the LEDs, highlighting possible moves that users can make, working as a basic tutorial to teach them the rules of Shogi.In both step sequencer and theory mode, during the update, the program only creates a list of the pieces it has found and their positions.These positions are then used to determine what piece to play.The update in shogi mode is more complicated as the software needs three variables to determine what moves has been made.These variables are the old and new position of the piece and it's last position on the board.During each update, the software loops over the pieces and checks if it found the marker corresponding with that piece and determines its coordinate.It then compares the positions, and if the positions have changed the piece gets added to a list holding all moved pieces.Pieces that are marked as 'picked up' are also added to this list.It is possible to promote a piece in a traditional Shogi game, by flipping the piece.The software recognizes the promotion of a piece when it detects a new marker with an ID of 40 or more on the position of an old piece.It will then check if that can be promoted, if so, it changes the marker associated with that piece.After looping over all pieces, the system uses the list to determine what move has been made based on a decision tree.During a capture, the last position on the board of the captured piece is set to 'None', this is done so the system understands when a captured piece is being played again.This system allows users to have an accessible way of interacting and composing with Japanese sound materials, showcasing a range of different instruments that members of the general public would never usually come across or work with.

FUTURE WORK & CONCLUSION
To best validate the design of this initial interface, the Shogi board was trialed at a co-design workshop for the 'Make-It-Yourself' project.The workshops involved participants learning the fundamentals of musical interactions, inputs, processes and outputs.The Shogi board was showcased alongside a number of other musical controllers, such as the Ableton Push and Novation Launchpads, before users designed their own musical interfaces using the BareConductive Touchboards.Users present at this workshop commented on the versatility of the board, and were interested in all three available modes.Participants also commented on the method of changing between the different modes, and how it could be altered to be more intuitive and work in a smoother way.Through observing how members of the public use the Shogi board, it was evidenced that the majority of the participants were interested in the step sequencer mode, as opposed to the Shogi gameplay mode or the music theory mode.It could be argued that this was the most accessible mode of the three, and that members of the general public are more likely to use the most entry-level method of interacting with these audio materials.The ages of participants ranged from late teens to early fifties, as the location of the workshop was a public library and local archive, and often attracts people of all ages.Collecting feedback from a wide range of ages will benefit the authors when it comes to refining the interface for future workshops.From these primary workshops, it could be suggested that the Shogi board, particularly in the step-sequencer mode, is a user friendly and accessible way of the general public interacting with traditional Japanese instrumentation.From this point, there will be a second and possibly third iteration of the Shogi board, using comments and ideas taken from the workshops with the general public, to truly make the interface as accessible and informative as possible.

Figure 2 -
Figure 2-Building the Shogi board

Figure 6 -
Figure 6-Getting hands on during a 'Make-It-Yourself' workshop