Adaptive VR Test in Music Harmony Based on Conditional Spiking GAN

Conference theme: Use of new and emerging technologies in Digital Art, Data, Scientific and Creative Visualisation, Digitally Enhanced Reality and Everyware, 2D and 3D Imaging, Display and Printing, Mobile Applications, Museums and Collections, Music, Performing arts, and Technologies, Open Source and Technologies, Preservation of Digital Visual Culture, Virtual Cultural Heritage, Ethical Issues, Historical Issues, Digital Culture, Artificial Intelligence, NFTs

Conference date: 4–8 July 2022

Keywords: Adaptive test, VR, AI, Conditional Spiking GAN, Semantic music generation, Graphs in music harmony

Bookmark

Abstract

This article proposes an adaptive VR test for the knowledge level control in music harmony. The core functioning relies on conditional semantic music generation strategy, using spiking conditional GAN architecture. The novel method of semantic music information encoding based on the system of graphs in music harmony, allowed two-dimensional data representation of harmonic sequences. which made possible considerable data augmentation and a transition to the specifics of training inherent to the visual domain. To our best knowledge, this is the first attempt of conditional spiking GAN implementation along with the application of the spiking neural networks in a domain of semantic music generation.

Content

Author and article information

Contributors

Anna Shvets

Samer Darkazanli

Conference

Publication date: July 2022

Publication date (Print): July 2022

Pages: 177-184

Affiliations

[0001]Fablab by Inetum

157 Boulevard MacDonald

75019 Paris, France

[0002]iMSA

Rue Clos Maury 82000 Montauban, France

Article

DOI: 10.14236/ewic/EVA2022.33

SO-VID: 045402f3-bfe1-4506-85ff-ff80327f64da

License:

This work is licensed under a Creative Commons Attribution 4.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/

Conference name: Proceedings of EVA London 2022

Conference acronym: EVA 2022

Conference number:

Conference location: London

Conference date: 4–8 July 2022

Conference sponsor: Electronic Workshops in Computing (eWiC)

History

Product

1477-9358 BCS Learning & Development

Self URI (article page): https://www.scienceopen.com/hosted-document?doi=10.14236/ewic/EVA2022.33

Self URI (journal page): https://ewic.bcs.org/

REFERENCES

K. Akbarzadeh-SherbafS. Safari, & A. H. Vahabie (2020). A digital hardware implementation of spiking neural networks with binary FORCE training. Neurocomputing, 412, pp. 129-142.
A. Artola, and W. Singer (1993). Long-term depression of excitatory synaptic transmission and its relationship to long-term potentiation. Trends in neurosciences, 16(11), pp. 480-487.
E. L. BienenstockL. N. Cooper, and P. W. Munro (1982). Theory for the development of neuron selectivity: orientation specificity and binocular interaction in visual cortex. Journal of Neuroscience, 2(1), 32-48.
R BogaczM. W. Brown, and C. Giraud-Carrier (2000). Frequency-based error backpropagation in a cortical network. In Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium (Vol. 2), pp. 211-216.
S.M. BohtéJ.N. Kok, and H.L. Poutré (2000). SpikeProp: backpropagation for networks of spiking neurons. ESANN.
R. Brette and W. Gerstner (2005), Adaptive Exponential Integrate-and-Fire Model as an Effective Description of Neuronal Activity, J. Neurophysiol. 94, pp. 3637-3642.
N. Caporale and Y. Dan (2008). Spike timing-dependent plasticity: a Hebbian learning rule. Annu. Rev. Neurosci., vol. 31, pp. 25–46.
A DelormeJ. GautraisR. Van Rullen and S. Thorpe (1999). Spikenet: A simulator for modeling large networks of integrate and fire neurons. Neurocomputing, vol. 26, pp. 989–996.
Y DemiragC. FrenkelM. Payvand, and G. Indiveri (2021). Online Training of Spiking Recurrent Neural Networks with Phase-Change Memory Synapses. arXiv preprint arXiv:2108.01804.
P. U. DiehlG. ZarrellaA. CassidyB. U. Pedroni, and E. Neftci (2016). Conversion of artificial recurrent neural networks to spiking neural networks for low-power neuromorphic hardware. In: 2016 IEEE International Conference on Rebooting Computing (ICRC) pp. 1-8. IEEE.
R. Fitzhugh (1961). Impulses and Physiological States in Theoretical Models of Nerve Membrane. Biophysical journal, 1(6), pp. 445–466.
R. V. Florian (2007). Reinforcement learning through modulation of spike-timing-dependent synaptic plasticity. Neural computation, 19(6), pp. 1468-1502.
X. Guan, and L. Mo (2020). Unsupervised conditional reflex learning based on convolutional spiking neural network and reward modulation. IEEE Access, 8, pp. 17673-17690.
H HazanD. SaundersD. T. SanghaviH. Siegelmann, and R. Kozma (2018). Unsupervised learning with self-organizing spiking neural networks. In: 2018 International Joint Conference on Neural Networks (IJCNN), pp. 1-6, IEEE.
D.O. Hebb (1949). The Organization of Behavior. New York: Wiley & Sons.
J. L. Hindmarsh, and R. M. Rose (1984). A model of neuronal bursting using three coupled first order differential equations. Proceedings of the Royal society of London. Series B. Biological sciences, 221(1222), pp. 87-102.
A. L. Hodgkin, and A. F. Huxley (1952). A quantitative description of membrane current and its application to conduction and excitation in nerve. The Journal of physiology, 117(4), pp. 500-544.
J. Jemielnik and A. Shvets (2015). Visualization impact on the effective-ness of music harmony knowledge assimilation. In: Art and education. Music art. (en polonais). Eds. A. BoguszewskaB. Niścior, Publisher: Department of Arts of Maria Curie-Sklodowska University in Lublin, pp. 47-61.
S.R. Kheradpisheh, and T. Masquelier (2020). S4NN: temporal backpropagation for spiking neural networks with one spike per neuron. International journal of neural systems, 2050027.
S.R. KheradpishehM. Mirsadeghi, and T. Masquelier (2021). BS4NN: Binarized Spiking Neural Networks with Temporal Coding and Learning. ArXiv, abs/2007.04039.
R KimY. Li, and T. J. Sejnowski (2019). Simple framework for constructing functional spiking recurrent neural networks. In: Proceedings of the national academy of sciences, 116(45), pp. 22811-22820.
V. Kotariya and U. Ganguly (2021). Spiking-GAN: A Spiking Generative Adversarial Network Using Time-To-First-Spike Coding. arXiv preprint arXiv:2106.15420.
S LattnerM. Grachten, and G. Widmer (2016). Imposing higher-level Structure in Polyphonic Music Generation using Convolutional Restricted Boltzmann Machines and Constraints. ArXiv, abs/1612.04742.
M. Lee, and J. Seok (2020). Regularization methods for generative adversarial networks: An overview of recent studies. arXiv preprint arXiv:2005.09165.
S. Li, and Y. Sung (2021). INCO-GAN: Variable-Length Music Generation Method Based on Inception Model-Based Conditional GAN. Mathematics 2021, 9(4), p. 387.
H. Liu, and Y. Yang (2018). Lead Sheet Generation and Arrangement by Conditional Generative Adversarial Network. In: 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA), pp. 722-727.
D. MakrisK.R. Agres, & D. Herremans (2021). Generating Lead Sheets with Affect: A Novel Conditional seq2seq Framework. 2021 International Joint Conference on Neural Networks (IJCNN), 1-8.
R MassaA MarchisioM. Martina, and M. Shafique (2020). An efficient spiking neural network for recognizing gestures with a dvs camera on the Loihi neuromorphic processor. In: 2020 International Joint Conference on Neural Networks (IJCNN) pp. 1-9. IEEE.
M. Minsky (1974). A framework for representing knowledge.
M. Molano-MazonA. OnkenE. Piasini, and S. Panzeri (2018). Synthesizing realistic neural population activity patterns using generative adversarial networks. arXiv preprint arXiv:1803.00338.
C. Morris, and H. Lecar (1981). Voltage oscillations in the barnacle giant muscle fiber. Biophysical journal, 35(1), pp. 193-213.
P. O'ConnorD. NeilS. C. LiuT. Delbruck, and M. Pfeiffer (2013). Real-time classification and sensor fusion with a spiking deep belief network. Frontiers in neuroscience, 7, p. 178.
R. C. O'Reilly (1996). The Leabra model of neural interactions and learning in the neocortex. Doctoral dissertation, Carnegie Mellon University.
A. Pati, & A. Lerch (2021). Is Disentanglement enough? On Latent Representations for Controllable Music Generation. ArXiv, abs/2108.01450.
A. PatiS. Gururani, & A. Lerch (2020). dMelodies: A Music Dataset for Disentanglement Learning. ISMIR.
P. Pistone and A. Shvets (2014). Investigation of the activity based teaching method in e-learning musical harmony course. In: Proceeding of EVA Florence 2014. Firenze University Press: Florence, ed. Vito Cappellini, 7th – 8th May 2014, pp. 107-112.
F. Ponulak, and A. Kasinski (2006). ReSuMe learning method for Spiking Neural Networks dedicated to neuroprostheses control. In: Proceedings of EPFL LATSIS Symposium 2006, Dynamical Principles for Neuroscience and Intelligent Biomimetic Devices (pp. 119-120).
W. PotjansA. Morrison, and M. Diesmann (2009). A spiking neural network model of an actor-critic learning agent. Neural computation, 21(2), pp. 301-339.
N. Rathi, and K. Roy (2020). Diet-snn: Direct input encoding with leakage and threshold optimization in deep spiking neural networks. arXiv preprint arXiv:2008.03658.
B. RosenfeldO. Simeone, and B. Rajendran (2021). Spiking Generative Adversarial Networks With a Neural Network Discriminator: Local Training, Bayesian Models, and Continual Meta-Learning. ArXiv, abs/2111.01750.
X SheP. SahaD. KimY. Long, and S. Mukhopadhyay (2020). Safe-DNN: A deep neural network with spike assisted feature extraction for noise robust inference. In: 2020 International Joint Conference on Neural Networks (IJCNN) pp. 1-8. IEEE.
A. Shvets and S. Darkazanli (2020). Graphs in harmony learning: AI assisted VR application. In: J. WeinelJ.P. BowenG. Diprose, and N. Lambert (eds) (2019) EVA London 2019: Electronic Visualisation and the Arts. London: British Computer Society, pp.104-105. doi: 10.14236/ewic/EVA2020.18
A. Shvets and S. Darkazanli (2021). Conditional GAN for Diatonic Harmonic Sequences Generation in a VR Context. In: J. WeinelJ.P. BowenA. Borda, and G. Diprose (eds) (2021) EVA London 2021: Electronic Visualisation and the Arts. London: British Computer Society, pp.97-100. doi: 10.14236/ewic/EVA2021.15
A. Shvets (2016), The system of graphs in music harmony: a user inter-face for mobile learning game development. In: J.P. BowenG. Diprose, and N. Lambert (eds) (2016) EVA London 2016: Electronic Visualisation and the Arts. British Computer Society (BCS): London, UK, 12th - 14th July 2016, pp.193-194.
A. Shvets (2019). Contemporary methods of functional harmony teaching in a high school context. In: Electronic Imaging & the Visual Arts (EVA Florence) 2019. Firenze University Press: Florence, ed. Vito Cappellini, 8th – 9th May 2019, pp. 142-150.
H.H. Tan, and D. Herremans (2020). Music FaderNets: Controllable Music Generation Based On High-Level Features via Low-Level Feature Modelling. ISMIR.
A. TavanaeiM. GhodratiS.R. KheradpishehT. Masquelier, and A. Maida (2019). Deep Learning in Spiking Neural Networks. Neural networks: the official journal of the International Neural Network Society, 111, 47-63.
H. C. Tuckwell (1988). Introduction to theoretical neurobiology: linear cable theory and dendritic structure (Vol. 1). Cambridge University Press.
Z. WangD. WangY. Zhang, and G. Xia (2020). Learning Interpretable Representation for Controllable Polyphonic Music Generation. ISMIR.
Y. Yu, and S. Canales (2021). Conditional LSTMGAN for Melody Generation from Lyrics. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 17, pp. 1-20.
M. ZhangJ. WangB. AmornpaisannonZ. ZhangV. MiriyalaA. BelatrecheH. QuJ. WuY. ChuaT.E. Carlson, and H. Li (2021). Rectified Linear Postsynaptic Potential Function for Backpropagation in Deep Spiking Neural Networks. In: IEEE transactions on neural networks and learning systems.

Comments

Comment on this article

[1] K. Akbarzadeh-SherbafS. Safari, & A. H. Vahabie (2020). A digital hardware implementation of spiking neural networks with binary FORCE training. Neurocomputing, 412, pp. 129-142.

[2] A. Artola, and W. Singer (1993). Long-term depression of excitatory synaptic transmission and its relationship to long-term potentiation. Trends in neurosciences, 16(11), pp. 480-487.

[3] E. L. BienenstockL. N. Cooper, and P. W. Munro (1982). Theory for the development of neuron selectivity: orientation specificity and binocular interaction in visual cortex. Journal of Neuroscience, 2(1), 32-48.

[4] R BogaczM. W. Brown, and C. Giraud-Carrier (2000). Frequency-based error backpropagation in a cortical network. In Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium (Vol. 2), pp. 211-216.

[5] S.M. BohtéJ.N. Kok, and H.L. Poutré (2000). SpikeProp: backpropagation for networks of spiking neurons. ESANN.

[6] R. Brette and W. Gerstner (2005), Adaptive Exponential Integrate-and-Fire Model as an Effective Description of Neuronal Activity, J. Neurophysiol. 94, pp. 3637-3642.

[7] N. Caporale and Y. Dan (2008). Spike timing-dependent plasticity: a Hebbian learning rule. Annu. Rev. Neurosci., vol. 31, pp. 25–46.

[8] A DelormeJ. GautraisR. Van Rullen and S. Thorpe (1999). Spikenet: A simulator for modeling large networks of integrate and fire neurons. Neurocomputing, vol. 26, pp. 989–996.

[9] Y DemiragC. FrenkelM. Payvand, and G. Indiveri (2021). Online Training of Spiking Recurrent Neural Networks with Phase-Change Memory Synapses. arXiv preprint arXiv:2108.01804.

[10] P. U. DiehlG. ZarrellaA. CassidyB. U. Pedroni, and E. Neftci (2016). Conversion of artificial recurrent neural networks to spiking neural networks for low-power neuromorphic hardware. In: 2016 IEEE International Conference on Rebooting Computing (ICRC) pp. 1-8. IEEE.

[11] R. Fitzhugh (1961). Impulses and Physiological States in Theoretical Models of Nerve Membrane. Biophysical journal, 1(6), pp. 445–466.

[12] R. V. Florian (2007). Reinforcement learning through modulation of spike-timing-dependent synaptic plasticity. Neural computation, 19(6), pp. 1468-1502.

[13] X. Guan, and L. Mo (2020). Unsupervised conditional reflex learning based on convolutional spiking neural network and reward modulation. IEEE Access, 8, pp. 17673-17690.

[14] H HazanD. SaundersD. T. SanghaviH. Siegelmann, and R. Kozma (2018). Unsupervised learning with self-organizing spiking neural networks. In: 2018 International Joint Conference on Neural Networks (IJCNN), pp. 1-6, IEEE.

[15] D.O. Hebb (1949). The Organization of Behavior. New York: Wiley & Sons.

[16] J. L. Hindmarsh, and R. M. Rose (1984). A model of neuronal bursting using three coupled first order differential equations. Proceedings of the Royal society of London. Series B. Biological sciences, 221(1222), pp. 87-102.

[17] A. L. Hodgkin, and A. F. Huxley (1952). A quantitative description of membrane current and its application to conduction and excitation in nerve. The Journal of physiology, 117(4), pp. 500-544.

[18] J. Jemielnik and A. Shvets (2015). Visualization impact on the effective-ness of music harmony knowledge assimilation. In: Art and education. Music art. (en polonais). Eds. A. BoguszewskaB. Niścior, Publisher: Department of Arts of Maria Curie-Sklodowska University in Lublin, pp. 47-61.

[19] S.R. Kheradpisheh, and T. Masquelier (2020). S4NN: temporal backpropagation for spiking neural networks with one spike per neuron. International journal of neural systems, 2050027.

[20] S.R. KheradpishehM. Mirsadeghi, and T. Masquelier (2021). BS4NN: Binarized Spiking Neural Networks with Temporal Coding and Learning. ArXiv, abs/2007.04039.

[21] R KimY. Li, and T. J. Sejnowski (2019). Simple framework for constructing functional spiking recurrent neural networks. In: Proceedings of the national academy of sciences, 116(45), pp. 22811-22820.

[22] V. Kotariya and U. Ganguly (2021). Spiking-GAN: A Spiking Generative Adversarial Network Using Time-To-First-Spike Coding. arXiv preprint arXiv:2106.15420.

[23] S LattnerM. Grachten, and G. Widmer (2016). Imposing higher-level Structure in Polyphonic Music Generation using Convolutional Restricted Boltzmann Machines and Constraints. ArXiv, abs/1612.04742.

[24] M. Lee, and J. Seok (2020). Regularization methods for generative adversarial networks: An overview of recent studies. arXiv preprint arXiv:2005.09165.

[25] S. Li, and Y. Sung (2021). INCO-GAN: Variable-Length Music Generation Method Based on Inception Model-Based Conditional GAN. Mathematics 2021, 9(4), p. 387.

[26] H. Liu, and Y. Yang (2018). Lead Sheet Generation and Arrangement by Conditional Generative Adversarial Network. In: 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA), pp. 722-727.

[27] D. MakrisK.R. Agres, & D. Herremans (2021). Generating Lead Sheets with Affect: A Novel Conditional seq2seq Framework. 2021 International Joint Conference on Neural Networks (IJCNN), 1-8.

[28] R MassaA MarchisioM. Martina, and M. Shafique (2020). An efficient spiking neural network for recognizing gestures with a dvs camera on the Loihi neuromorphic processor. In: 2020 International Joint Conference on Neural Networks (IJCNN) pp. 1-9. IEEE.

[29] M. Minsky (1974). A framework for representing knowledge.

[30] M. Molano-MazonA. OnkenE. Piasini, and S. Panzeri (2018). Synthesizing realistic neural population activity patterns using generative adversarial networks. arXiv preprint arXiv:1803.00338.

[31] C. Morris, and H. Lecar (1981). Voltage oscillations in the barnacle giant muscle fiber. Biophysical journal, 35(1), pp. 193-213.

[32] P. O'ConnorD. NeilS. C. LiuT. Delbruck, and M. Pfeiffer (2013). Real-time classification and sensor fusion with a spiking deep belief network. Frontiers in neuroscience, 7, p. 178.

[33] R. C. O'Reilly (1996). The Leabra model of neural interactions and learning in the neocortex. Doctoral dissertation, Carnegie Mellon University.

[34] A. Pati, & A. Lerch (2021). Is Disentanglement enough? On Latent Representations for Controllable Music Generation. ArXiv, abs/2108.01450.

[35] A. PatiS. Gururani, & A. Lerch (2020). dMelodies: A Music Dataset for Disentanglement Learning. ISMIR.

[36] P. Pistone and A. Shvets (2014). Investigation of the activity based teaching method in e-learning musical harmony course. In: Proceeding of EVA Florence 2014. Firenze University Press: Florence, ed. Vito Cappellini, 7th – 8th May 2014, pp. 107-112.

[37] F. Ponulak, and A. Kasinski (2006). ReSuMe learning method for Spiking Neural Networks dedicated to neuroprostheses control. In: Proceedings of EPFL LATSIS Symposium 2006, Dynamical Principles for Neuroscience and Intelligent Biomimetic Devices (pp. 119-120).

[38] W. PotjansA. Morrison, and M. Diesmann (2009). A spiking neural network model of an actor-critic learning agent. Neural computation, 21(2), pp. 301-339.

[39] N. Rathi, and K. Roy (2020). Diet-snn: Direct input encoding with leakage and threshold optimization in deep spiking neural networks. arXiv preprint arXiv:2008.03658.

[40] B. RosenfeldO. Simeone, and B. Rajendran (2021). Spiking Generative Adversarial Networks With a Neural Network Discriminator: Local Training, Bayesian Models, and Continual Meta-Learning. ArXiv, abs/2111.01750.

[41] X SheP. SahaD. KimY. Long, and S. Mukhopadhyay (2020). Safe-DNN: A deep neural network with spike assisted feature extraction for noise robust inference. In: 2020 International Joint Conference on Neural Networks (IJCNN) pp. 1-8. IEEE.

[42] A. Shvets and S. Darkazanli (2020). Graphs in harmony learning: AI assisted VR application. In: J. WeinelJ.P. BowenG. Diprose, and N. Lambert (eds) (2019) EVA London 2019: Electronic Visualisation and the Arts. London: British Computer Society, pp.104-105. doi: 10.14236/ewic/EVA2020.18

[43] A. Shvets and S. Darkazanli (2021). Conditional GAN for Diatonic Harmonic Sequences Generation in a VR Context. In: J. WeinelJ.P. BowenA. Borda, and G. Diprose (eds) (2021) EVA London 2021: Electronic Visualisation and the Arts. London: British Computer Society, pp.97-100. doi: 10.14236/ewic/EVA2021.15

[44] A. Shvets (2016), The system of graphs in music harmony: a user inter-face for mobile learning game development. In: J.P. BowenG. Diprose, and N. Lambert (eds) (2016) EVA London 2016: Electronic Visualisation and the Arts. British Computer Society (BCS): London, UK, 12th - 14th July 2016, pp.193-194.

[45] A. Shvets (2019). Contemporary methods of functional harmony teaching in a high school context. In: Electronic Imaging & the Visual Arts (EVA Florence) 2019. Firenze University Press: Florence, ed. Vito Cappellini, 8th – 9th May 2019, pp. 142-150.

[46] H.H. Tan, and D. Herremans (2020). Music FaderNets: Controllable Music Generation Based On High-Level Features via Low-Level Feature Modelling. ISMIR.

[47] A. TavanaeiM. GhodratiS.R. KheradpishehT. Masquelier, and A. Maida (2019). Deep Learning in Spiking Neural Networks. Neural networks: the official journal of the International Neural Network Society, 111, 47-63.

[48] H. C. Tuckwell (1988). Introduction to theoretical neurobiology: linear cable theory and dendritic structure (Vol. 1). Cambridge University Press.

[49] Z. WangD. WangY. Zhang, and G. Xia (2020). Learning Interpretable Representation for Controllable Polyphonic Music Generation. ISMIR.

[50] Y. Yu, and S. Canales (2021). Conditional LSTMGAN for Melody Generation from Lyrics. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 17, pp. 1-20.

[51] M. ZhangJ. WangB. AmornpaisannonZ. ZhangV. MiriyalaA. BelatrecheH. QuJ. WuY. ChuaT.E. Carlson, and H. Li (2021). Rectified Linear Postsynaptic Potential Function for Backpropagation in Deep Spiking Neural Networks. In: IEEE transactions on neural networks and learning systems.

Celebrating 65 years of The Computer Journal - free-to-read perspectives - bcs.org/tcj65