This week I have focused on building the software which synthesizes audio. The software I used last year was very rudimentary; it could only map the position and direction of the pulleys’ movements onto audio file playback position and volume.
What I’ve been trying to do at the beginning of this residency is get the software to a point where I feel it has adequate expressive capabilities so that I can begin to try out the different mappings — and do more drawings.
At the beginning of this week, I worked towards mapping the existing data from the pulleys onto several different parameters in the Reason software synthesis environment. As I wrote earlier, there just wasn’t enough variety in the sounds I was generating to hold my interest.
To find another way to move forward, I turned my attention to the data — and came up with a number of ways to derive dynamic values from my existing sensing system.
Properties of Lines
Instead of simply treating the coordinates I receive from the sensors as changes in position, I could begin to record data about the emerging line — and its relationship to previously drawn lines.
- Current line length – since I know how far the pen has moved horizontally and vertically, I know the distance it has travelled.
- Running average of past line lengths – maybe it will be useful to know how the lengths of the lines have been changing over time.
- Current line drawing time – it’s easy to figure out how long the pen was down. Once I know how long the pen has been down, I can also compute the speeds at which lines are being drawn. Maybe averaged data about these properties will also yield useful control values for generating audio.
- “Not drawing” time – how long did I pause between drawing lines. Is my drawing proceeding at a “furious” pace or am I stopping to think a bit between making marks?
- Line slopes – I want some way to tell if I’m drawing similar lines. I want visual repetition to translate into sonic repetition.
- Line shapes: Loops – Am I drawing over the same line in order to darken it on the paper? The software should be able to sense this. I don’t know of a good computational approach to this, but I had the idea to look at the current line (pen-down to pen-up) as video.
If I render my digitized lines in a transparent color, they would darken over time and I could probably find a computer vision algorithm which tests for an enclosed shape (not sure if this is blob tracking or something else).
- Line shapes: Similar forms – There are computer vision techniques for sensing the statistical similarity of an image to a reference image. I tried some experiments with this using cv.jit, but this didn’t yield useful results, yet.