Teaching Monsters

Published on April 3, 2018

For his show La Fabrique des monstres, presented during the festival ManiFeste-2018, Jean-François Peyret revisits the myth of Frankenstein’s monster submitted to artificial intelligence (AI). Joining the composer Daniele Ghisi, he worked closely with the IRCAM teams and especially the researcher Philippe Esling… A look back on this sinuous and difficult process.

How did the project come about?
Jean-François Peyret: The project was born on the shores of Lake Geneva during a few discussions with Vincent Baudriller, the director of the théâtre de Vidy. And, to make a long story short, for several years my theater has centered on the artificalization of the living, my encounter with Frankenstein was surely inevitable. The fact that Mary Shelley started her book on the shores of the same lake was the true catalyst…. Victor Frankenstein and his creatures are two specters that haunt the lake. Then, after my last show, Citizen Jobs, Frank Madlener and I had the same desire to work together. I talked to him about La Fabrique des monstres and about the idea of a machine that fabricates music that escapes its creator, wondering to myself if this line of questioning would make sense to a composer… This is when Frank Madlener introduced me to Daniele Ghisi1.
Daniele Ghisi: In the beginning, the music had to “incarnate” the performance’s “monster”. From there, I imagined myself working on a project in which the mastery of the musical discourse would be given (a large part, anyway) to a “composing machine”. My role as a composer was to conceive and produce the machine and select the musical fragments the machine would compose. A more or less expected secondary effect of this work was a sort of “Turing Test” applied to the music produced by the machine…
Philippe Esling: Over the past few years, Daniele and I have talked a lot together and worked together on several projects. I have always had a lot of respect and admiration for his musical approach that combines equal portions of mathematics, computer tools, and composition. So, when we had our first meeting on this project, we immediately talked about an exploratory phase to discover how automatic learning could be used for a compositional purpose. However, his approach was the complete opposite of the scientific research in which learning is not a means for attaining an end (which is generation), making the project even more exciting. For me, focusing on the musical aspect of the learning process itself was an unsettling and fascinating idea. This project holds a special place in my work because it was both orthogonal and coherent with our research on creative intelligence.

How was the discourse created, not only on a subject, but on the role of artificial intelligence (AI) in the piece?
Jean-François Peyret: I was struck by the importance Mary Shelley gives to the creature’s learning in her novel. The creature is released into the wild with a nearly blank brain, as if the “reset” were finished. The creature follows a complete path of mental organization, from first feelings to the acquisition of high culture: at the end of this learning, he has read Goethe, Plutarch, Milton! This is systematically forgotten in the adaptations of the myth for film where the monster is represented like a crude beast or one implanted with the brain of a crazy person or an assassin. Yet Mary Shelley never says anything about the creature’s brain: it must therefore be a normal brain… This doesn’t stop him from committing crimes and completing his learning with a murder.  
Our conversations centered on learning and Daniele started to explain how his machine would learn. AI is not a proper character (having said this, in my theater, there are no characters); however, this question of machine learning was at the heart of our discussions. How the brain learns and how machines learn.  The ideas came from there, and sorting them out, well, that happens on its own. It’s a question of tact.


La Fabrique des monstres © Mathilde Olmi

Jean-François, you often say that you “expose” your work to science, like we are exposed to the sun or to risks. How did you “expose” artificial intelligence?  
Jean-François Peyret: I like that expression: expose my theater to science. To avoid any misunderstanding, my theater is not scientific vulgarization, nor a theater that imagines it is scientific. Still, my history with AI is fairly unique and goes back to my first meeting, now over 20 years ago, with Alan Turing. Turing has not stopped, like a ghost himself, to haunt my theater. I always thought that the theater with an actor—a creature for whom the boundary between the living and the artificial is blurred, troubled—is the ideal venue to question this artificial intelligence and try to imagine, if not understand, its operations. The stock-in-trade of theater is dialogue: it can not, in my opinion, disregard the man-machine dialogue… One of my ambitions would be, using the means of the theater, to understand how machines “think”. I read an article by Turing in Mind (1950), and I don’t know if they really think. But what we do know, is that we are led (condemned?) to think with extremely powerful machines, and therefore, maybe, more and more, to think like machines, or in a way they “want” us to think. Food for thought.

And you, Daniele, how did you see this project? You have already worked with databases, did this extension towards “machine learning” seem natural?
Daniele Ghisi: Yes, I’m used to working from databases. In this case, I didn’t used these databases to compose, but to teach a machine to compose.
What was interesting, was the learning process (seeing how a machine coached using a database of lieder for baritone and piano, for example, learned to sing and play at the same time…), and the results are sometimes “inappropriate”, bizarre, but intriguing (and good to listen to!).

In actual fact, for certain databases, the best results were never used in the piece because the machine learned so well that the musical fragments obtained didn’t disturb anyone! 

How did you design this machine? How does it work?  
Philippe Esling: After my meeting with Daniele, we started working in a studio with several researchers. An interesting detail: each member of this group went in their own direction. We pushed the experiment in completely opposite directions. Several emanations resulted from this reflection (some of which were the subject of publications), from mechanisms enabling audio hybridizations to spaces for non-supervised synthesis. But the most remarkable fact was that Daniele developed the solution that was used in the piece. From our scientific point of view, I think we wanted a “machine for composer” too much whereas Daniele was looking for a “machine composer”. We then tried to help Daniele as best we could in his research, but I think that he deserves all the credit for the development of the machine the audience will have the pleasure of hearing learn.
Daniele Ghisi: Philippe is exaggerating a little. I didn’t create the “machine” by myself. I absolutely do not have the skills necessary for the conception of this kind of algorithm starting with a blank slate. This “machine” is in fact a network of neurons trained with datasets, capable of reproducing motifs learned through the aforementioned datasets (to be precise, the model we started with is called SampleRNN and is available on GitHub2). I was also enormously helped by Robin Meier. My discussions with him, his work, and his critiques were precious.

It was essential for me that the machine be absolutely “ignorant”: in the beginning of the piece, it has no knowledge of musical syntax or lexis (notes, chords, harmony, intensity, reverberation, etc.). It learns little by little using the datasets we give it! It doesn’t know anything but the sequence of audio samples, the variations in pressure due to sound waves. Everything else, we have to teach it… What is surprising, is that, based on this, and only this, the machine learns to predict a sequence of samples that include a certain “creativity”, or it is us who interpret it this way.


La Fabrique des monstres © Mathilde Olmi

Daniele, how was the machine integrated in the compositional process? What music did it produce in the end?  
Daniele Ghisi: It produces months of music, in different degrees of its learning in relation to playing the original data. My role in the compositional process was limited to “selecting” the fragments generated by this machine that I thought were interesting, and “showing” them. For me, the real issue was making myself “listen” to the machine: listening, organizing, sorting, and finally, selecting. Independent of the result in this piece, the method is certainly radical, but interesting. Certain results pushed me to question my presuppositions on musical syntax and lexis, and the process has already taught me a lot. This raises the essential questions that touch intellectual property, paternity, without talking about legalities. Who wrote these pieces? The “machine” that generated them? Me, the person who selected and/or showed them? The people who created the algorithm? No law, neither moral or societal, tells us. I like to question this aspect of artistic creation, and, in many ways, this work of “anonymous composition” echoes the work of “collective composition” I defend in the group /nu/thing of which I am a part.

The process at work questions another of my obsessions: the mechanism I used for this piece is a new type of sound synthesis (and not sampling). And it is a type of synthesis that will be more present in the upcoming years. We already hear that samplers are replacing musicians. Here, we go even further, because these machines replace “certain” uses for which we call upon composers… We must confront this issue, without imagining the worst, by developing an in-depth reflection on the meaning we attribute to the concept of “creativity”, and defend our ideas and independence.

1. Music doctorate: research and composition (Sorbonne Université/IRCAM)
2. Developed by Soroush Mehri, Kundan Kumar, Ishaan Gulrajani, Rithesh Kumar, Shubham Jain, Jose Sotelo, Aaron Courville, and Yoshua Bengio. https://github.com/soroushmehr/sampleRNN_ICLR2017

By Jérémie Szpirglas, journalist and author