Cloud Hosting

Music can evoke strong emotions and affect human behaviour. We process music via a series of complex cognitive operations. Consequently, it can be a window to understanding higher brain functions, as well as being used as a diagnostic and therapeutic tool. So how can we understand the way music evokes emotions and effectively use this in healthcare technologies?

Recently PLOS ONE launched a collection on Affective Computing and Human-Computer Interactions and we discuss with Stefan Ehrlich from the Technische Universitt Mnchen and Kat Agres from the National University of Singapore their paper on a music-based brain-computer interface for emotion mediation.

PLOS In your paper A closed-loop, music-based brain-computer interface for emotion mediation you present a Brain-Computer Interface (BCI) pilot study that uses an automatic music generation system to both affect users emotional states and allows them to mediate the music via their emotions. What would you say are the key points of your work?

Stefan Ehrlich Our work focuses on the integration of music with healthcare technology to mediate and reinforce listeners emotional states. The key point we see is in providing a novel automatic music generation system that allows a listener to continuously interact with it via an emotion display. The system translates the listeners brain activity, corresponding to a specific emotional state, into a musical representation that seamlessly and continuously adapts to the listeners current emotional state. Whilst the user listens, they are made aware of their current emotional state by the type of generated music, and the feedback allows them to mediate or to regain control over the emotional state. Many of the neurofeedback applications that have been already proposed often only have one-dimensional feedback provided to the to the subject. For instance, a levitating ball is displayed on the screen, and the subject is asked to control it up or down. The advantage of using music is that its possible to map a relatively complex signal, in this case brain activity, in a multi-dimensional manner to a cohesive, seemingly only one- dimensional feedback. Its possible to embed different information in a single cohesive BCI feedback by using the different features of music, such as rhythm, tempo, the roughness of the rhythm or the harmonic structure.

PLOS Were there any particular health care applications that you had in mind when designing this pilot study?

Kat Agres I tend to think of music as being a sort of Swiss army knife where there are lots of features that can come in handy, depending on the scenario or the clinical population. For example, its social, its engaging, it often evokes personal memories, and it often lends itself to rhythmic entrainment. Its these properties or features of music that lend itself particularly well to health care applications. Our main focus is on mental health and emotional wellbeing, and teaching people how to control their own emotions. And I think thats the really interesting part about this study, that the music is a sonification of the listeners emotional state, as measured via their EEG. It is meant to influence their emotional state, and helps teach the listener how to mediate their emotional states as they interact with the music system. This sonification can show the listener both whats happening emotionally but it also allows them to mediate the sound of the music by affecting their own emotional state. The music is being created in real time based on the brain activity. Weve recently been awarded a fairly large grant in Singapore to develop a holistic BCI system that were actually calling a Brain-Computer-Brain Interface. The project will cover different aspects, e.g., motor skills, cognition and emotion. Weve already started developing the 2.0 version of the automatic generation system, and we are about to validate it with a listening study with both healthy adults and depressed patients. Once all these validation steps have been completed and we can effectively say that the system is flexible enough to induce different emotion states in a depressed population, we will be applying this to stroke patients who are battling depression.

PLOS What do you think the main differences will be in the ability of depressed and healthy populations to affect emotions with this system?

Kat Agres The number one reason people listen to music is to enhance or modify their emotion state or their mood. There is very significant literature now supporting the use of music for various mental health scenarios and for people who are struggling with various mental health conditions. I think that music is particularly well positioned to help people when other things are not helping them. The first group of depressed patients that we will be testing our system on is made up of many young people who actually think of their identity in part in terms of their music. Based on the literature and unique affordances of music, I think that we have a decent shot at reaching these individuals and helping them figure out how to gain better control of their motion states. In our pilot study, some individuals really got the hang of it and some had a harder time figuring out how to use the system. I think well find the same thing in this population of depressed patients. Im cautiously optimistic that this system will be effective for this population.

Stefan Ehrlich When using the system, different psychiatric and neurological populations will probably elicit different patterns of interaction. These will lead to the next steps in understanding how to modify the system in order to better help the patients. At the moment its a system that can help them gain awareness of their emotional state and that allows us to measure the variations between the different groups.

Kat Agres And one of the interesting directions we are exploring with the automatic music generation system is the trajectory of taking someone from a particular (current) emotional state to another, target emotional state. It will be interesting to compare whether the optimal trajectory through emotion space is similar for depressed patients and healthy adults.

PLOS Was there anything that particularly surprised you?

Stefan Ehrlich A surprise for me was that without telling the listeners how to gain control over the feedback, when asked, all of them reported that they self-evoked emotions by thinking about happy/sad moments in their life. I want to emphasise that the system triggered people to engage with their memories and with their emotions in order to make the music feedback change. I was surprised that all of the subjects chose this strategy.

PLOS What was the biggest challenge for you?

Stefan Ehrlich The most difficult part was developing the music generation system and the mapping with continuous changes of brain activity. In the beginning we wanted to map brain activity features with musical features and the idea of focusing on emotions as the target only came during the development of the system. Constraining the system to emotional features and target variables helped to reduce the dimensionality and the complexity, while clarifying the main objective (emotion mediation) of the eventual system.

Kat Agres Creating an automatic music generation system is not as easy as it might sound, especially when it has to be flexible to react to changes in brain state in real time. Theres a lot of structure and repetition in music. So when the participants try to push their emotion state up or down the music has to adapt in real time to their brain signals and sound continuous and musically cohesive.

Stefan Ehrlich Yes, and there cant be a big time-lag with the generated music, as this would compromise the sense of agency participants have over the system. If the system does not react or respond accordingly, people would lose faith that the system actually responds to their emotions.

PLOS This work is very interdisciplinary with researchers from many different backgrounds. What are your thoughts on interdisciplinary research?

Stefan Ehrlich I think it is more fun to work in an interdisciplinary setting. Im really excited to hear and learn about the insight or the perspective of the other side on a topic or problem. It can be occasionally challenging. You have to establish a common ground, values and methodological approaches to a problem. You need to be able to communicate and exchange in an efficient way so that you can learn from each other. Its important that all of the involved parties are willing to understand to a certain degree the mindset of the other side.

Kat Agres I feel quite passionately about interdisciplinary research, especially as a cognitive scientist working at a conservatory of music. One of the obvious things that comes to mind when youre working with people from different disciplines is how they use different terms, theoretical approaches, or methods. And yes, that can be a difficulty. But as long as everyone is clear on what the big challenges are, have the same high-level perspectives, values, and a shared sense of what the big goals are, it works well. In order to collaborate, you have to get on the same page about what you think is the most important issue, and then you can decide on the methods and how to get there.

PLOS Considering your original research backgrounds, how did you end up doing such interdisciplinary research?

Stefan Ehrlich I have a very non-interdisciplinary background in a way (electrical engineering and computer science). During my masters I attended a lecture called Introduction to computational neuroscience and it was really an eye opener for me. I realized that my background could contribute to research in neuroscience, engineering, and medicine. From then I started developing a strong interest in research at this intersection of topics.

Kat Agres I specifically chose an undergrad institution that allowed me to pursue two majors within one degree programme: cognitive psychology and cello performance. I found it really difficult to choose one over the other and eventually I realised that I could study the cognitive science of music. And then I did a PhD in music, psychology, and cognitive science. I consider health to be yet another discipline that Im interested in incorporating into a lot of my research. I am very grateful that recently Ive been able to do more research at the intersection of music, technology, and health.

PLOS In the field of affective computing and human-computer interactions, what do you think are the biggest challenges and opportunities?

Stefan Ehrlich I think one important aspect is the human in the loop. The human is at the centre of this technology, as important as the system itself. Often the transfer from the lab is very difficult to do due to the variables associated with humans. Ultimately, we want to see people using these technologies in the real world, and this is the main challenge.

Kat Agres I agree that human data can be messy. Physiological signals, like EEG, galvanic skin response, heart rate variability, etc., are all pretty noisy signals, and so its just difficult to work with the data in the first place. We see daily advancements in AI, medical technologies, and eHealth. I think the future is going to be about merging these computational and engineering technologies with the creative arts and music.

PLOS Do you see Open Science practices, like code and data sharing, as important for these fields?

Stefan Ehrlich Yes absolutely. When I started working in research there were not many data sets available that would have been useful for my work. I think researchers should upload everything from data to code to a public repository. I personally use GitHub, which currently has the limitation of not allowing very large files, e.g., EEG data. Its not an ideal repository for this kind of data at the moment, but there are many other platforms being developed and will hopefully be adopted in the future.

Kat Agres I wholeheartedly agree that Open Access is extremely important. I am glad that a discussion is happening around not all researchers having access to funds to make their work Open Access. Im lucky that Im attached to an academic institution where one can apply for funds for Open Access. My concern is that policies requiring authors to pay might create elitism in publication. Academic partnerships with journals like PLOS ONE can help researchers publish Open Access.

PLOS What would be your take home message for the general public?

Stefan Ehrlich & Kat Agres We think that the public currently perceives music predominantly as a medium for entertainment, but music has a much bigger footprint in human history than this. Historically, music served many important roles in society, from social cohesion, to mother-infant bonding, to healing. In ancient Greece, Apollo was the god of Music and Medicine. He could heal people by playing his harp. They used to think that music had healing properties. The same is found in Eastern cultures, where for example the Chinese character for medicine is derived from the character for music. There is a very long-standing connection between these areas. In more recent years music has taken this more limited role in our society, but now more and more people are beginning to realise that music serves many functions in society, including for our health and wellbeing. We hope that music interventions and technologies such as our affective BCI system will contribute to this evolving landscape and provide a useful tool to help people improve their mental health and well-being.

References:

1. Ehrlich SK, Agres KR, Guan C, Cheng G (2019) A closed-loop, music-based brain-computer interface for emotion mediation. PLOS ONE 14(3): e0213516. https://doi.org/10.1371/journal.pone.0213516

Author Biographies

Stefan Ehrlich is a postdoctoral fellow in the Dystonia and Speech Motor Control Laboratory at Harvard Medical School and Massachusetts Eye and Ear Infirmary, Boston, USA. His current research is focused on brain-computer interfaces (BCIs) for the treatment of focal dystonia using non-invasive neurofeedback and real-time transcranial neuromodulation. Formerly, he was a postdoctoral researcher at the Chair for Cognitive Systems at the Technical University of Munich, where he also obtained his PhD in electrical engineering and computer science in 2020. His contributions comprise research works on passive brain-computer interfaces (BCI) for augmentation of human-robot interaction as well as contributions to the domain of easy-to-use wearable EEG-based neurotechnology and music-based closed-loop neurofeedback BCIs for affect regulation.

ORCID ID 0000-0002-3634-6973.

Kat Agres is an Assistant Professor at the Yong Siew Toh Conservatory of Music (YSTCM) at the National University of Singapore (NUS), and has a joint appointment at Yale-NUS College. She was previously the Principal Investigator and founder of the Music Cognition group at the Institute of High Performance Computing, A*STAR. Kat received her PhD in Psychology (with a graduate minor in Cognitive Science) from Cornell University in 2013, and holds a bachelors degree in Cognitive Psychology and Cello Performance from Carnegie Mellon University. Her postdoctoral research was conducted at Queen Mary University of London, in the areas of Music Cognition and Computational Creativity. She has received numerous grants to support her research, including Fellowships from the National Institute of Health (NIH) and the National Institute of Mental Health (NIMH) in the US, postdoctoral funding from the European Commissions Future and Emerging Technologies (FET) program, and grants from various funding agencies in Singapore. Kats research explores a wide range of topics, including music technology for healthcare and well-being, music perception and cognition, computational modelling of learning and memory, automatic music generation and computational creativity. She has presented her work in over fifteen countries across four continents, and remains an active cellist in Singapore.

ORCID ID 0000-0001-7260-2447

Read the original:

Music based brain-computer interfaces an interview with Stefan Ehrlich and Kat Agres - EveryONE - PLoS Blogs