private webpage: 

Building upon our prior work and using the results from participatory design sessions we present the design of two mechanisms enabling the multi-modal interaction with social networking sites.

Tlatoque + Ubisketch

UbiSketch, a ubiquitous real-time sketch-based communication tool, supporting information sharing and social communication based on natural interactions with paper-based supports. Our interactive system enables people to write text notes or draw sketches with digital pens and paper and share them in real time via their mobile phones. We are currently conducting a deployment study with an older adult using Ubisketch integrated to Tlatoque. The older adult uses UbiSketch to send written-based messages and poems to their relatives.


Tlatoque + Exergames

We designed two ubiquitous exergames: GuessMyCaption and TakeAPhoto. These games use family memoirs available in SNS and natural interfaces to encourage older adults to exercise.

Take [a] Photo

The objective of the game is to ask relatives to take artistic photographs, or photographs portraying the context of
their everyday life or memoirs, based on the older adult’s submitted “mission”. A “mission” might demand relatives to take a photograph of the context of their everyday life (e.g., when they are at work), an artistic photograph (e.g., a sunset), a photo of their activities based on their location or time of the day (e.g., take a photo at 10 o’clock) or from a memoir (e.g., take a photo of your childhood). Relatives will compete from the best photograph rated by the older adult. To avoid cognitive load to older adults for submitting “missions” the system might automatically submit “mission” if no new requests are detected. The system has two interfaces: a digital portrait the older adult uses to submit “missions” and consult photographs, and a capture tool that is synchronized with Facebook to post “missions” for relatives and to upload the photographs resulted from a “mission”. To initiate the game, the older adult has to use either “left hello”  or “right hello” . Once in the game, the older adult sends the “mission” to all her relatives who have 48 hours to upload their photo into the SNS. Relatives receive the “mission” in their Facebook account as a post in their timeline. Once the 48-hours period expires, the older adult will consult the photographs in her digital portrait (see Fig. 3). The older adult uses either “left arm raise” (see Fig. 1c) or “right arm raise” body action (see Fig. 1d) to navigate through the uploaded photographs. To rate the photographs, the older adult uses the “circular arms raise” body action (see Fig. 1e) to add a star rate for up to five stars.

So for a family of around 10 members, the older adult will execute a minimum of 20 exercises for around 30 minutes, three times a week.The older adult will be rewarded with a printout of the top rated photograph of the previous “mission”. The top three rated photographs are published on the older adults’ SNS timeline as feedback for relatives.

Guess my Caption

The other exergame employs the same approach to initiate the game and uses the content available in the Facebook’s family account to feed the content of the game. The objective of this game is to guess the photograph that better matches a description. Three photographs are randomly retrieved from the relatives’ accounts on Facebook, and are displayed to the user (Fig. 4). From this set of photos one is randomly selected and the description displayed at the bottom is retrieved from the photograph’s caption or associated comment in Facebook. The player has 45 seconds and two attempts to select the correct answer. He uses the body actions to select the desired photo; “left arm raise” (see Fig. 1c) to select the displayed photograph on the left of the screen, “right arm raise” (see Fig. 1d) to select the displayed photograph on the right of the screen and “circular arms raise” (see Fig. 1e) to select the photograph displayed at the top of the screen.

The exergame provides visual (e.g., glowing green rectangle around the correct photo) and audio feedback (e.g., cheers sound) when the older adult selects the correct answer. A puzzle of a family photograph on the top right corner of the screen is assembled based on the number of attempts that the user requires to get the correct answer. If the user deduces the correct photo on her first attempt, then the game assembles 2 pieces of the puzzle; if it’s the second attempt, then the game assembles 1 piece of the puzzle. Once the puzzle is complete, the older adult receives one virtual credit that can be exchanged for a printout of a photograph of her choice. A post is published on her SNS showing the photograph the older adult chose to print and the owner of this photograph.

The results of a 5-weeks deployment study with one older adult and 12 relatives showed the system maintained the older adult motivated to exercising while offering new opportunities for online and offline social encounters.

We are still recruiting so contact us if you are interested in participating in this project!

Project participants

 Raymundo Cornejo, Doctoral Student (see more about raymundo ...)
e: rcornejo[at]cicese[dot]edu[dot]mx

Monica Tentori, Ph.D., Assistant Professor (see more about monica …)
e: mten …

In collaboration with Jesus Favela (CICESE), Sergio F. Ochoa (U. of Chile) and Nadir Weibel (UCSD)

Related publications to the project

  • Cornejo, R., Tentori, M., and Favela, J. (in press) “Enriching in-person encounters through social media: A study on family connectedness for the elderly”. International Journal of Human-Computer Studies

  • Cornejo, R., Hernández, D., Favela, J. and Tentori, M (2012) “Persuading older adults to socialize and exercise trough ambient games” In Workshop of Wellness and HCI, PervasiveHealth 2012, San Diego California, May 21-24, 2012