Light Instrument

Light Instrument is about exploring synesthesic transformations between vision and hearing to produce tools for audio and music transcription, analysis and education, instrument practicing and artistic output.

The video above presents a branch of the project focused on a visual representation suitable for music analysis and instrument practice.
It offers a decomposition of sound in 10 octaves equally tempered in 5*12 notes/octave (600 channels), matching it closely to the widely adopted western tuning system, while still allowing for small frequency deviations on vibratos and bendings.
This provides a more clear visualization from a musical perspective, than a fourrier transform.

The audio processing lib was build so that different tunning systems can be easilly selected or customized.

Colors follow the cycle of 5ths, and provide a harmonic pallette for harmonics and mainly related chords, with higher color contrast on tritones and dissonances. This offers a quicker feedback response compared to classic musical notation.

Equal loudness correction to better weight the pitch spectrum.

Harmonics are still visible, and provide a handful feedback on the characteristics of timbre in wind/brass instruments for example.

Amplitude envelopes aid in representing dynamics such as vibrato, specially in bowed instruments.
The project also implements a protocol with a timeline description very different from the usual MIDI description, allowing better and more flexible timbre dynamics and sentences envelopes.
The cycle of the visualization provides reference for customizable bmp and time signature.

This version runs in realtime in a webbrowser using javascript and WebGL2. No external libraries are used. All coded from scratch.

There is also a cross-platform standalone application version developed in Rust +wgpu +gdal.

We are preparing an audiovisual demonstration and deeper explanation of the project, and a demo of both javascript and rust version should be released soon.
We are considering the possibilities for financial support of the project, but it is our desire to open-source its code.

Another branch allows more artistic output for audioreactive visualizations. You can enjoy a mix of a few extracts in the above video.
It makes use of auditory perceptions such as pitch, timbre, tone, harmony, loudness and dissonance to produce visual perceptions of color, luminance, opacity, saturation, length, decay and sharpness.

Project under development.
More updates soon...
follow us on instagram @lightinstrument

by @erick.vermot and @chicotdl