The Sonification of Human Motion for Motor Learning

Abstract

This thesis examines how sonification can be used to help a student emulate the complex motion of a teacher with increasing spatial and temporal accuracy. As an example scenario, the system captures a teacher's motion in real time and generates a 3- D motion path, which is recorded along with a reference sound. A student then attempts to perform the motion and thus recreate the teacher's reference sound. The student's synthesized sound will dynamically approach the teacher's sound as the student's movement becomes more accurate. Several types of sound mappings, which simultaneously represent time and space deviations are explored. For the experimental platform, a novel system that uses low-cost camera-based motion capture hardware and open source software has been developed. This work can be applied to diverse areas such as rehabilitation and physiotherapy, performing arts and aiding the visually impaired.

The contributions of this work include the following: 1) A methodology for capturing complex human motion in 3- D and mapping the motion to sound for in- context feedback of position and speed; 2) The development of low- cost software system which is being used as experimental platform for the research; and 3) Initial testing of the system using several different types of motion to demonstrate performance.