Out of the Ether Redux

Previous

Students

Bo Bell, Music

Jim Kleban, Elec & Comp Engineering

Dan Overholt, Media Arts & Tech

Lance Putnam, Media Arts & Tech
Bob Sturm, Elec & Comp Engineering

John Thompson, Postdoctoral Researcher

Faculty Advisors

JoAnn Kuchera-Morin, Media Arts & Tech
B.S. Manjunath, Elec & Comp Engineering

Next

Abstract

The use of computers in the arts is now wide spread. In music, the computer is a powerful instrument for performing real-time synthesis and transformation processes. However, many input devices used to interface with computers are designed for office-like work and not musical performance. The problem then arises: how does an expert musical performer apply their existing technique naturally to performance with a computer? Previous attempts involve sensor augmentation of traditional instruments, creating entirely new instruments, playing along with fixed computer accompaniment, and using external controls such as foot pedals. All of these solutions, however, force the musician to adopt new performance responsibilities sometimes with an obtrusive or constraining effect. Our approach is to use audio, video, and electric field sensors to more naturally inform the cueing and control of computer-generated musical material in the context of live performance. We hope to make the technology transparent to the audience while freeing the performer from the use of knobs and switches often used in musical human-computer interaction. In addition to providing a transparent manner of interacting with the computer performer, this system should open many musical possibilities to the musical performer and composer. We intend to examine mapping strategies for the gestural control of multimedia content using this system. An overarching theme in this examination is the mapping of gesture across sound and image.

Our system will consist of four main distributed components: acquisition, analysis, pattern recognition, and synthesis.

The first layer of the system will be responsible for acquiring data for analysis. A multi-modal music stand (MMMS) will be designed to house cameras, microphones, and other electronic sensors to allow the musician to perform in an unobtrusive, unconstrained performance space. The music stand will provide data to an analysis system that will track the natural gestures of the flutist and communicate that information to audio and visual synthesis and transformation processes. In addition, visual feedback will be provided to the flutist to keep him/her "in the loop" as to the effects of his/her actions.

The analysis system will consist of components for computer vision, audio analysis, and sensor analysis. The computer vision component will track the position of the flute, the position of the flutist's head, and the direction in which the flutist is looking (gaze tracking). The audio analysis component will extract various features from the flute audio, such as pitch, brightness, loudness, and harmonicity. Using a microphone array, we hope to determine additional information concerning the location of the flute audio. The electronic sensors will provide proximity information to give an estimate of the performer's position and stance.

These various analysis components will inform a middle classification layer that, based on its received data, identifies key gestures by the flutist. Classifiers will be trained on synchronous multi-modal data to detect head nods, flute positioning, and extended playing techniques apart from traditional technique. Timing and knowledge of the score will make the system more robust. The classifications will determine the cueing and control of the computer performer.

The final component of the system will be a real-time audio/visual synthesis engine. This component will record the human performer's playing and apply various spectral transformations to the sound according to a predetermined musical score. Visuals will be generated based on events in the score and directly with the characteristics of the sound.