Compared to other populations, individuals with autism tend to have more challenges with processing sensory input as well as motor coordination. In general, children with autism exhibit numerous sensorimotor impairments and may have poor motor control defaulting their ability to appropriately redirect their movements to respond to sensory stimuli. For example, when children with autism move their body to respond to a stimuli, their reaction time is usually off reacting tardy or in advance; or they will use insufficient or a lot of strength. There is some evidence that indicates that music can help individuals with autism to regulate their sensorimotor system.

Neurological Music Therapy (NMT) is being increasingly used to support the sensorimotor regulation of individuals with autism [4]. NMT is the therapeutic use of music to improve individual’s cognitive, sensory, and motor dysfunctions due to a neurologic disease of the human nervous system . In a typical NMT session, musical instruments, not being played in a traditional manner, are arranged in different locations to guide patients’ movements. For example, when working with two musical instruments, a physiologist uses one musical instrument as the starting point of the movement and the second one as the ending point. Children with autism must practice different movements when playing on each musical instrument.

Current NMT sessions heavily rely on the use of traditional musical instruments to facilitate the interactions of children with autism with live music. Unfortunately, these children often lack of musical training, and their motor, sensory, and cognitive impairments limit their interactions with musical instruments. We hypothesize that interactive surfaces offer a natural and casual interaction, using multisensory stimuli to maintain the attention of children, and removing the complexity of the input interaction mechanism are appropriate tools to support NMT sessions of children with autism.

In this project, we have been developing  BendableSound: a fabric-based multi-touch surface aimed at supporting NMT sessions allowing children to practice movement patterns while playing music when touching the fabric.

BendableSound is a multi-touch system that uses the MS Kinect sensor and the TSPS library to detect users’ interactions. We developed the multimedia interfaces with Processing. Digital animations are displayed in a 2.3 x 1.8m spandex fabric acting as a digital screen. The JavaSounds library is being used as music synthesizer. Users can play sounds of the musical instrument selected by the therapist when pushing and touching the fabric or the digital elements appearing on top of the fabric. BendableSound displays a 3D background showing an animation of a nebula. On top of the background, translucent space-based elements, like stars and planets, are organized as a music sheet scattered throughout the canvas.

The major goal is to catapult a rocket to catch space-based elements representing the musical notes of a song –including a variety of nursery rhymes. BendableSound has five different activities.

Activity 1: Freely interaction with the canvas. Children must discover the space by erasing a black layer blocking the animation of nebulas by freely swiping throughout the fabric canvas. The amount of strength being used when pushing the fabric will vary the volume and the size of the “eraser”.

Activity 2: Understanding pressure. Children must use a rocket to collect one space-based element representing a music note.

A rocket will appear on top of each note in the music sheet. Children must push the rocket to catapult it’s flight and collect the note. The flight distance is determined by the amount of used strength when pushing the fabric.

Activity 3: Strength regulation (Figure 2c). Children’s strength when pushing the fabric should catapult the rocket’s flight to collect several space-based elements representing one music compass of a song. The amount of strength used should be enough to reach the next note but not too much to skip it.

For activities #2 and #3, sounds will be reproduced for each “push” or collected note. Children must follow the “movement routine” completing a number of repetitions specified by the therapist. One rocket flight will count as one repetition.

Activity 4: Timing control. Children’s must push the space-based elements of the music sheet according to the music tempo. The rocket provides children guidance on how to play a song by touching a blinking space-base element. The space-base element disappears according to the music tempo, but children are not penalized when missing a beat. A music pulse will sound like a metronome to give users guidance whenever the music speeds up or slows down.

Activity 5: Freely interaction with live music. Returning to an open-ended modality, and similar to activity #1, children can freely interact with sounds available in the fabric canvas and with the space-based elements representing notes from a song. By freely pushing anywhere children could create music combining musical notes and sound effects.

We are currently evaluating the use of BendableSound in supporting children with autism and toddlers. See some initial results …


Collaborators:  Nadir Weibel (UCSD), Gillian R. Hayes (UC Irvine)

Project participants

Deysi Helen Ortega Roman, M.Sc. (see more about deysi …)
e: dortega[at]cicese[dot]edu[dot]mx
Research interes …
Oscar Peña Ramirez, M.Sc. (see more about oscar …)
e: opena[at]cicese[dot]edu[dot]mx
Research interests:
Franceli Linney Cibrian, Ph.d. Student (see more about franceli linney ...)
e: franceli[at]cicese[dot]edu[dot]mx

Monica Tentori, Ph.D., Assistant Professor (see more about monica …)
e: mten …

Related publications to the project

  • Cibrian, F. L, Weibel, N., Tentori, M. 2016. Collective Use of a Fabric-based Interactive Surface to Support Early Development in Toddler Classrooms (2016). In Proceedings of Ubicomp (Ubicomp ’16).