Apparatus of Euphonious Thinking
Music is the wallpaper of our minds
Jamie Wheals
Communication of inexpressible
We believe music has the capacity to communicate the ineffable, or inexpressible. And humans have this urge to language these inexpressible thoughts when the words don’t work. And these thoughts are instrumented into existence by artists through poems, visuals and music. Our innovations of doing so have been through instruments, mediated through our body (hands, mouth, feet) but WHAT IF WE COULD ACTUALIZE THE BRAIN IN CREATING MELODY DIRECTLY?
We attempted to communicate these very thoughts through a melody by translating the brain signals into a musically satisfying melody using machine learning.
– written by Vinay
Brain signals to music
Brain signals are electrical impulses that became the carriers of memory
During the experience, participants are free to think anything which they want to capture in the form of a melody. We encouraged them to think of a cherished memory, run wild with their imaginations, things that excite them. These thoughts give rise to series of ionic currents in the brain. These ionic currents are captured by the brain reader headband.
These analogue signals are further converted into digital musical notes, and a computer uses machine learning to transform them into a classical music style melody. This is then transferred to the music punching machine. The machine then transforms the melody into punch hole pattern which is then punched into a sheet of paper. This sheet of music can be played with a physical cranking music box. The music is also available for the user online by scanning a QR code which will led the user into a digital platform for them to download their unique melody.
Turning inside-out
With this experimental project, we attempted to actualize the human imagination or recall a memory in the form of music. Using machine learning, we blend the brain signals into a Bach style classical melody. This melody is further presented in a form of punched sheet to be played with a cranking music box. The process of the transformation of the electrical brain signals into digital pulses in the software run by machines, to a tangible sheet of music exhibits the fluidic flow of information from the physical to digital and then back to physical form. This truly is a phygital experience at play.
The ineffability of music experience from brain waves
What it feels like to listen to music created by your brain signals
Over hundred participants have experienced their brain melodies. And ironically, found it very difficult to express what they felt. Recording the brain signals being fascinated by the idea and intrigued to listen the final music, they seem to fall short of words while expressing how they felt in the end. It almost always pushed them to be an artist themselves, looking for metaphors, musical pieces, etc. to express the ineffable feelings.
As the creator of this experiment, we also witnessed the joy of our participants whenever they heard the music their brain had to sing. It was remarkable to see the curiosity during the process grow into discovering the final music as if it opened a new medium of self-expression.
The embark of a long journey
– written by Minjoo
Working as a creative technologist in the design firm requires relentless curiosity and willingness for a challenge. The beginning of the journey to the “Apparatus of Euphonious Thinking” began with the sudden invite from Karel: “Minjoo, to bring the momentum of the innovative culture of INDEED, I want you to think of the idea for an interactive installation using state-of-art technology with something mechanical.”
When I take such (sometimes seemingly abstract) requests, I took the following three criteria as the main considerations.
1) Learning experience
Rather than merely combining existing skills, we should leverage the opportunity to learn the newest technology that might be needed for future projects.
2) Representing Indeed
The installation should entail the core capability of INDEED – strong background in product design and engineering imbued with creativity.
3) Emotional touch points
What INDEED would like to deliver is an emotional wave deeply rooted in humanity by combining personal traits from the participants.
We have been keen to explore various applications of Artificial Intelligence and initiated our explorative study in 2017 with AI-powered algorithmic mesh distortion. (How AI will change you; Several other methods to be combined with AI-driven solutions, such as projection mapping, human body tracking, and brain interface were suggested in different concepts).
What we aimed for was the sharable experience among all participants. Another important consideration was the development resource. As the project needed to be developed merely with internal resources aside from our main client projects, the technology that has lots of resources on the web, or has open APIs were prioritized.
Considering Artificial Intelligence, I wanted to pursue the musical direction this time, not only because of my personal affinity to music (especially classical music) but also due to the emotional impact that music induces.
You may have experienced that you can remember a specific memory when you listen to specific phrases or melodies of one particular music, so can we replicate such thought processes by state-of-the-art technology?
Taking these factors in consideration, we decided to take a traditional music box as the initial concept of the installation. The music box, from human history, always evokes the personal memory of the individual. What if the Brain Interface, an emerging method of human-machine interface, then connects the human memory with the machine driven by Artificial Intelligence? The result of the human-machine collaboration will then be processed through the mechanical author and it will keep creating the music sheet with physical forms.
Sounds tricky? Believe me it was.
The dream team
When the idea was all set, I started to forage for people adding my skills to further develop the idea into reality. Having myself as the main system developer, I needed somebody to develop the machine to punch the results into the music sheets and support to consolidate the storyline and experience. Vinay and Sam joined to help with their expertise in mechanical engineering. Isabel and Stefanie jumped in for the experience and marketing part. I still remember the moment that every single member welcomed a rather ambitious idea with a supportive smile (There is no other ways to better explain the facial expression :-)).
The technical exploration during the initial phase
1) Music generation engine
I started to write a script interfacing MAGENTA’s music generation engine (written in Python) running in the backend with the frontend written in JavaScript. To ensure the quality of the generated music still sounds familiar to the bare listeners, I used Polyphony RNN engine that generates Bach’s polyphonic (having several notes played at the same time) out of the given monophonic (having single note at a time) melody as an input. The backend engine was decided to be hosted in Python Flask server.
Frontend Javascript code was made so that the Interface automatically transcribes the played MIDI notes running in the background. Onsets and Frames model hosted in MAGENTA.js has been integrated into our Javascript code.
2) Brain Interface
Neurosky is the pioneer company in the BI (Brain Interface) industry and provides various brain interfaces applicable to covering a broad range of project complexities. Among them, I chose to utilize the brain library written for the Arduino by hacking their low-end Mindflex model designed for the basic gaming experience.
To showcase the legacy of INDEED as a product design firm, we redesigned the interface body and started to integrate the circuit into it.
3) Punching Machine
We took the Music Box Auto Hole Punch Machine by Josh Seldon as the initial inspiration for the machine. The existing machine punches one hole at a time and does not provide enough speed to punch polyphonic notes to be punched for at least 50 bars. None of the participants will be patient enough to wait for 30 minutes to receive their result and it will eventually destroy the whole user experience. As a result, Vinay, Sam and me decided to take a rather audacious attempt than merely improving the existing machine to expect a dramatic increase in the punching speed.
The marathon
-written by Sam
Following the main ideation Sam and Vinay conceptualized a faster way to punch music notes by rearranging punches in 4 offset lines. This idea was then translated into 3D model to visualize the working and troubleshoot any new issues with fabricating and assembling.
1) Small reduced components work
Zooming in: From the big picture in mind, we all broke it into smaller jobs to be done. Step by step the components were made and tested. We started with hacking the brain reader to get a signal using Arduino nano. Then Sam, made the first punch block with a single servo which could punch holes reliably. Looking at small wins, we then made a mockup of the entire set-up using cardboard boxes to test the experience, usability and ergonomics.
Zooming out: With every small component being made and tested, our ambitions were piling up as well. We gathered some of the team-members to create an aesthetic vision of the installation with mood board as the starting point. With the vision of making the machine inside-out, exhibiting everything we could to users on how it works.
With the aesthetic vision and CAD model complete, we started to integrate the components to make bigger sub-assemblies. Minjoo took charge of scaling up the one servo punch block to controlling 30 servo blocks together. Sam built the core punching assembly with all the 30 punch blocks and Vinay started fabricating the mid-section. With many hours spent focused on our own tasks we came together after 3 weeks to integrate and assemble the top assembly.
2) Testing, failing, fixing
With integration we started to experience the first failures, soon we all turned into obsessed detectives trying to find the problems. Solving the problem was never an issue, we never ran out of ideas, but figuring out the problem (debugging) took the bigger chunk of our times. Up until the day before the exhibition we were stumbling upon issues and fixing them.
The evening before the exhibition, the punch motor burned on the venue and machine went “kaputt”. (“Kaputt” is by the way the most familiar German word within our international team. Cause it’s the best way to describe a broken thingy which needs fixing but you don’t now yet how. ) We went back to the workshop, still not at all in the mood to give up, calmed ourselves, started to fix the problem. The day of the exhibition, the machine ran with some cautionary breaks in between.
3) The big upgrade
We knew that the fixes we had were not enough. Two big events were coming up and we had to prepare the machine to run more reliably, therefore, we took a step back, and re-organized ourselves right from project management, to design and engineering tasks. Sam was given the baton of project manager; Vinay took charge of developing a circuit board with safety features integrated and Minjoo looked into software and integration of software and hardware.
Having a more scrum like approach helped us to keep focus. During the first project phase, we got sometimes lost in the problems and the overwhelming demands of the original concept. Stepping back and taking a break allowed us, to refocus and coordinate our work in more appropriate sprints with clear goals and mile stones. Controlling and evaluation become an integrative part of our work and approach.
We know where we were heading and each delivered mile stone could be celebrated as such.
Crossing the finish line
-written by Vinay Bhajantri
The first time the machine was exhibited in Digital Kindergarten, it was performing with some hiccups (with cautionary measures). But with the big upgrade in safety of electronics and a better motor, it worked like a charm in the next event- the Chamber of Beautiful Business.
This was the event where the whole team could recognize the first success. Multiple participants went through the complete experience and were awe-inspired by the music created by their own brain signals. With more education materials from Isabel and Steffi, we were able to better explain how it works and this elevated the experience even further.
House of Beautiful Business
Next few weeks were geared towards testing the machine for durability and quality of music. We adapted the platform to be more portable since we had to fly to Lisbon for the next event. In the middle of all the excitement of being there, setting it up we hit another bump the night before event.
We noticed during the test runs that the motor was not getting enough power, so we went back to module testing mode. We created a plan of action for troubleshooting and slowly started testing all the interfaces on the mechanics.
Combining the wit and hack ideas of all the team, from getting olive oil and lip balm for lubrication, getting a nail file for sharpening the punch pins, we managed to find and fix the gears on the motor. This must have happened during the transport. But the big news was, it started working well and we went back to sleep still skeptical about what new issue might come up next day.
The following days, the motor ran smooth as warm butter. The machine was working well, we were all super excited but very few people turned up. But slowly, through word of mouth we saw steep increase in people by the end of event.
This project was filled with many ups and downs down up until the last minute. One of the biggest gains from this project was deeper and authentic connection with each other. One of the most surreal moments was when after the motor burning incident, everyone took an hour off, came back and all of us wanted to work harder to fix the issues, at no point we thought of giving up.
We always dream radical and big, but we need the pragmatic thinking to deliver. Another learning comes in the form of handling such projects, on bringing pragmatic measures to very ambitious goals. With successive stages of project, we took measures to deliver the promised in modules that still acknowledged the ambitious initial idea.
This project provided us a real-world experience to experiment different hard skills, roles, and mindsets to evolve ourselves into better innovators.
And luckily, we are now at a point where we can take the installation on tour. So, like the ladies from the Fruchthof Campus let us know which event you like to infuse with our installation and we are happy to offer you an individual package filled with music, lovely operators and a brilliant machine.
A post by Minjoo Cho, Sam Swift and Vinay Bhajantri.
Photos shot by João Nogueira and Ana Torres at the House of Beautiful Business 2019, Lisbon
