Continuous Gesture-Sound Interaction
Designing Gestures for Continuous Sonic Interaction
Environment
Tested using Max 8.0.5
MacOS 10.14.5
Currently not working on Windows 10
Dependencies
- Mubu (install from package manager)
- CNMAT Externals: https://github.com/CNMAT/CNMAT-Externs/releases
- Wekinator
How to:
The max patch works in communication with Wekinator.
- Open Wekinator
- From Wekinator, open
../Wekinator2/sound-tracing-2/sound-tracing-2.wekproj, and press start listening. - open open /SCP BioMusic 2.maxproj in SCP BioMusic 2.
- Connect Myo, and stream data.
- Do calibration processed if prefered (optional).
Authoring
The authoring mode allows you to create static-sounds gesture relationships. You can set sound paramenters via the SCP synth, and then train the ANN model using weinator.
Sound-Tracing
The sound tracing mode allows you to create time-variant sounds-gesture reliationships.
In the envelopes sub-patch you find different evelopes, one per each parameters of the syntesiser. For each parameter you can establish the value of each of the four points of the envelope, numbered as in the range [0, 4]. Evelopes points can be edited by dragging each point within the table, or by selecting a point (Select Point) and editing the Point value. The duration of the envelope can be also edited by editing the Duration value. The enveloped can be played by pressing Play envelope.
In sound tracing mode the the sound gesture relationship can be established by using two algorithms, ANN and HMM.
The ANN implementation uses Wekinator, instead the HMM on uses the Mubu library.
When training the ANN model, it is possible to record the whole gesture or segments of it togheter with the time varying or static sound parameters, (Rec. Whole/ Rec. Segments).
When recording the whole gesture and evelope evolution, there is no need to take the Window parameter.
In the case of recording segments (Rec. Segments), then you might want to edit the length of the segments to record in ms (Segment). This number corresponds to the windows size of recorded samples.
In both modalyties, you have to enable the ANN (Enable ANN), then play the envelope (Play Envelope). The software will automatically record and train the model for you. Before running and playing sounds, please check that the training phase has been succesfuly completed.
To train the HMM, you have to adopt the same approach; enable HMM (Enable HMM) and then play the envelope (Play Envelope). The software will train the model at the when the audio file has been played.
After training the model, you can choose if scrubbing along the whole file (Scrub Envelope) or to scrub along points (Scrub Points). To access the HMM data, please open the HMM subpatch.
Moreover, the sound tracing mode can be also used as alternative to the authoring mode. After recalling any point of the envelope, it is possible to recod and train Wekinator with such parameters. When running wekinator, you will be able to explore the space between the choosen SCP paramenters-gestural associations.
Sound Design Mapping
The subpatch SoundDesign-Mappings contains all messaging system to interface the SCP synth with Wekinator.
| # | Wkinator ANN | Synth control | Parameter |
|---|---|---|---|
| 1 | string |
sendrcv |
string+;
|
| 2 | string |
disconnect+;
|
|
| 3 | string |
arg |
string or 0+;
|
| 4 | string |
min |
float+;
|
| 5 | string |
max |
float+;
|
| 6 | string |
speedlim |
int+;
|
| 7 | string |
time |
int+;
|
The first message is to specify the dataflow between the Wekinator Artificial Neural Network and a specific SCP synth's processor.
The ANN is named using the Wekinator project input port, the name Weki and the ANN number, i.e. 6448weki01; the string 'sendrev' stands for send/ receive, and the parameter is the processor name.
This link can be broken by streaming the message disconnect to the wekinator neural network.
Each ANN is to control a specific parameter of that processor with a message where is is specified the Wekinator ANN, followed by the string 'arg' and a string specifying the processor parameter, i.e. 6448weki01 arg stretch.
This mapping can be deleted by sending the same message with parameter 0.
It is possible to set the minumum and maximum range values indicating the Wekiantor ANN, the min or max strings, respectivelly for the minimum and maximum of the range, i.e. 6448weki01 min -1.;
6448weki01 max 3.;
The data stream from/to Wekinator can be limited and interpolated with a message that contains reference to the Wkinator ANN, the string speetlim for limiting and time for the interpolation and the time in milliseconds.
EMG calibration
Without any calibration, the output data are the extracted features, in this case the bayesan filter. the calibration process normalises data from each channel between a range [0, 1].
- min: press min and relax your arm until you dont see stable values
- max: press max and contract your muscles performing different movements until you dont see stable values
- run: after calibration hit run!
- reset: resets calibration values. NB: if you want only to update the min or max values of a specific emg channel you can just press min or max again, and relax/contract muscles relative to that channel. In this way you won't loose calibration data for other channels.
IMU calibration
- calibrate: set the origin of the three signals around the 0 value.
- reverse: do the inverse of the euler angle on the x, y, or z axis.
Gestural and audiovisual data recorder
In progress