README.md

Continuous Gesture-Sound Interaction

Environment

Tested using Max 8.0.5
MacOS 10.14.5
Currently not working on Windows 10

Dependencies

• Myo for Max
• Mubu (install from package manager)
• CNMAT Externals
• Wekinator
• Peter Elsea's LObjects
• Karlheinz Essl's RTC-lib

How to

The max patch works in communication with Wekinator.

1. Open Wekinator
2. From Wekinator, open ./Wekinator/gesture-sound-interaction.wekproj, and press start listening.
3. Open ./gesture-sound-interaction/gesture-sound-interaction.maxproj
4. Connect Myo, and stream data.

Authoring

The authoring mode allows you to create static-sounds gesture relationships. Using scripting functions and envelopes GUI.

Sound-Tracing

The sound tracing mode allows you to create time-variant, sound-gesture relationships. In the envelopes sub-patch there are different envelopes, one per each parameter of the syntesiser. For each parameter you can establish the value of each of the four points of the envelope, numbered in the range [0, 4]. Envelope points can be edited by dragging each point within the table, or by selecting a point (Select Point) and editing the Point value. The duration of the envelope can be also edited by editing the Duration value. The enveloped can be played by pressing Play envelope.

In sound tracing mode, the sound-gesture relationship can be established by using two algorithms, ANN and Hidden Markov Model (HMM). The ANN implementation uses Wekinator and the HMM uses the Mubu library. When training the ANN model, it is possible to record the whole gesture or segments of it togheter with the time-varying or static sound parameters, (Rec. Whole/ Rec. Segments). When recording the whole gesture and envelope evolution, there is no need to take the Window parameter. In the case of recording segments (Rec. Segments), then you might want to edit the length of the segments to record in ms (Segment). This number corresponds to the windows size of recorded samples. In both modalities, you have to enable the ANN (Enable ANN), then play the envelope (Play Envelope). The software will automatically record and train the model for you. Before running and playing sounds, please check that the training phase has been succesfuly completed. To train the HMM, you have to adopt the same approach; enable HMM (Enable HMM) and then play the envelope (Play Envelope). The software will train the model at the when the audio file has been played. After training the model, you can choose if scrubbing along the whole file (Scrub Envelope) or to scrub along points (Scrub Points). To access the HMM data, please open the HMM subpatch.

Moreover, the sound tracing mode can be also used as alternative to the authoring mode. After recalling any point of the envelope, it is possible to recod and train Wekinator with such parameters. When running wekinator, you will be able to explore the space between the choosen SCP paramenters-gestural associations.

Sound Design Mapping

The subpatch SoundDesign-Mappings contains all messaging system to interface the SCP synth with Wekinator.

#	Wekinator ANN	Synth control	Parameter
1	string	sendrcv	string+;
2	string	disconnect+;
3	string	arg	string or 0+;
4	string	min	float+;
5	string	max	float+;
6	string	speedlim	int+;
7	string	time	int+;

The first message is to specify the dataflow between the Wekinator Artificial Neural Network and a specific SCP synth's processor. The ANN is named using the Wekinator project input port, the name Weki and the ANN number, i.e. 6448weki01; the string 'sendrev' stands for send/ receive, and the parameter is the processor name. This link can be broken by streaming the message disconnect to the wekinator neural network.

Each ANN is to control a specific parameter of that processor with a message where is is specified the Wekinator ANN, followed by the string 'arg' and a string specifying the processor parameter, i.e. 6448weki01 arg stretch. This mapping can be deleted by sending the same message with parameter 0.

It is possible to set the minumum and maximum range values indicating the Wekiantor ANN, the min or max strings, respectivelly for the minimum and maximum of the range, i.e. 6448weki01 min -1.; 6448weki01 max 3.;