May 27, 2020

Apparatus of Euphonious Thinking I

This blog post is part of a series taking you behind scenes of INDEED’s latest installation. In the first part, we deep dive into the ideation and development process.

Minjoo Cho

The Embark of a Long Journey - The beginning

Working as a creative technologist in the design firm requires relentless curiosity and willingness for a challenge. The beginning of the journey to the “Apparatus of Euphonious Thinking” began with the sudden invite from Karel: “Minjoo, to bring the momentum of the innovative culture of INDEED, I want you to think of the idea for an interactive installation using state-of-art technology with something mechanical.”

When I take such (sometimes seemingly abstract) requests, I took the following three criteria as the main considerations.

1) Learning experience

Rather than merely combining existing skills, we should leverage the opportunity to learn the newest technology that might be needed for future projects.

2) Representing Indeed

The installation should entail the core capability of INDEED – strong background in product design and engineering imbued with creativity.

3) Emotional touch points

What INDEED would like to deliver is an emotional wave deeply rooted in humanity by combining personal traits from the participants.

We have been keen to explore various applications of Artificial Intelligence and initiated our explorative study in 2017 with AI-powered algorithmic mesh distortion. (How AI will change you; Several other methods to be combined with AI-driven solutions, such as projection mapping, human body tracking, and brain interface were suggested in different concepts).

What we aimed for was the sharable experience among all participants. Another important consideration was the development resource. As the project needed to be developed merely with internal resources aside from our main client projects, the technology that has lots of resources on the web, or has open APIs were prioritized.

Considering Artificial Intelligence, I wanted to pursue the musical direction this time, not only because of my personal affinity to music (especially classical music) but also due to the emotional impact that music induces.

You may have experienced that you can remember a specific memory when you listen to specific phrases or melodies of one particular music, so can we replicate such thought processes by state-of-the-art technology?

Taking these factors in consideration, we decided to take a traditional music box as the initial concept of the installation. The music box, from human history, always evokes the personal memory of the individual. What if the Brain Interface, an emerging method of human-machine interface, then connects the human memory with the machine driven by Artificial Intelligence? The result of the human-machine collaboration will then be processed through the mechanical author and it will keep creating the music sheet with physical forms.

Sounds tricky? Believe me it was.

The Dream Team

When the idea was all set, I started to forage for people adding my skills to further develop the idea into reality. Having myself as the main system developer, I needed somebody to develop the machine to punch the results into the music sheets and support to consolidate the storyline and experience. Vinay and Sam joined to help with their expertise in mechanical engineering. Isabel and Stefanie jumped in for the experience and marketing part. I still remember the moment that every single member welcomed a rather ambitious idea with a supportive smile (There is no other ways to better explain the facial expression :-)).

The technical exploration during the initial phase

1) Music generation engine

I started to write a script interfacing MAGENTA’s music generation engine (written in Python) running in the backend with the frontend written in JavaScript. To ensure the quality of the generated music still sounds familiar to the bare listeners, I used Polyphony RNN engine that generates Bach’s polyphonic (having several notes played at the same time) out of the given monophonic (having single note at a time) melody as an input. The backend engine was decided to be hosted in Python Flask server.

Frontend Javascript code was made so that the Interface automatically transcribes the played MIDI notes running in the background. Onsets and Frames model hosted in MAGENTA.js has been integrated into our Javascript code.

2) Brain Interface

Neurosky is the pioneer company in the BI (Brain Interface) industry and provides various brain interfaces applicable to covering a broad range of project complexities. Among them, I chose to utilize the brain library written for the Arduino by hacking their low-end Mindflex model designed for the basic gaming experience.

To showcase the legacy of INDEED as a product design firm, we redesigned the interface body and started to integrate the circuit into it.

3) Punching Machine

We took the Music Box Auto Hole Punch Machine by Josh Seldon as the initial inspiration for the machine. The existing machine punches one hole at a time and does not provide enough speed to punch polyphonic notes to be punched for at least 50 bars. None of the participants will be patient enough to wait for 30 minutes to receive their result and it will eventually destroy the whole user experience. As a result, Vinay, Sam and me decided to take a rather audacious attempt than merely improving the existing machine to expect a dramatic increase in the punching speed.

Read in the next episode how the story continues…

Part II

Part III

Minjoo Cho

Creative Interaction Technology

Minjoo is Creative Interaction Technologist at INDEED. She loves to reach and combine ideas from different fields including design, technology and art, but always supported by firm humanity within.


click here for more: