Syphone to transmit Jitter video files to Madmapper. Import videos and Kinect mask to Max/MSP/Jitter, then use Jitter to combine them together. Kinect camera captures the silhouette of pianist. She prepared 3 pieces of music during the visualization testing, they are:ĭebussy – Etude (Pour les arpèges composés) We did a collaboration with Mannes The New School for Music and invited piano major student Shulin Guo from Mannes to play the piano. Collaboration with Mannes the New School of Music, New York
Syphon is a powerful open sources for sharing videos in realtime, connecting different creative VJ software for stunning audio visual performances.Ģ. Once it projected on the piano, the structure automatically appear on the screen, we manually draw masks and set positions for videos. Madmapper is an easygoing software for mapping project like ours. Using Macbook trackpad to control the directions of lines movements. OpenFrameworks is the main platform to generate and control animation. Volume of piano sound wave controls the speed of motion and brightness of lines. Interactive Audio Visualization + Projection Mapping We used iPad to control TouchOSC system to do real time manipulation Synapse to capture the Kinect image to Max/MSP/Jitter. We used Max/MSP/Jitter to do visual effects interpolation, connected with Kinect to capture the silhouette of pianist, record the body movement while creating life mask of pianist’s body. Lots of technique explored for this project, although not all of them were implemented for the final performance.Īfter all, there are four parts: live Input and output using Max Msp Jitter & Kinect, Projection Mapping using Madmapper, Visual design using After Effects, interactive audio visualization using OpenFrameworks. We analysed the natural body movement from piano performance, and created a Matrix that regarding to different possible gestures of a pianist. They also create visual illusions, providing ‘back-up’ (or canceling) the dramatic content of the performed work. That’s why the purpose of the piano gesture is not only to enhance the quality and the character of the sound, but also to offer a convincing ‘visual support’ for the musical image you’re trying to communicate to the public. The pianistic gestures influence not only the quality of our sound and the character of the objectively heard music. This whole project contains a lot of researches and regards to pianist’s gestures. Many pianists think that: ‘every gesture you make while playing should reflect the character of the music, its emotional and dramatic content’. Researches of music and gesture movements: