CHARLIE: An Adaptive Robot Design with Hand and Face Tracking for Use in Autism Therapy

Laura Boccanfuso, Jason M. O'Kane
International Journal of Social Robotics


Basic turn-taking and imitation skills are imperative for effective communication and social interaction. Recently, research has demonstrated that interactive games using turn-taking and imitation have yielded positive results with autistic children who have impaired communication or social skills. This paper describes a robot that plays interactive imitation games using hand and face tracking. The robot is equipped with a head and two arms, each with two degrees of freedom, and a camera. We trained a human hands detector and subsequently, used this detector along with a standard face tracker to create two autonomous interactive games: single-player ("Imitate Me, Imitate You") and two-player ("Pass the Pose".) Additionally, we implemented a third setting in which the robot is teleoperated by remote control. In "Imitate Me, Imitate You", the robot has both passive and active game modes. In the passive mode, the robot waits for the child to initiate an interaction by raising one or both hands. In the second game mode, the robot initiates interactions. The "Pass the Pose" game engages two children in cooperative play by enlisting the robot as a mediator between two children alternately initiating and imitating poses. These games are designed to increase attention, promote turn-taking skills and encourage child-led verbal and non-verbal communication through simple imitative play. This research makes two specific contributions: (1) We present a low-cost robot design which measures and adapts to a child's actions during interactive games and, (2) we train, test and make freely available, a new hand detector, based on Haar-like features, which is usable in various kinds of human-robot interactions. We present proof-of-concept experiments with a group of typically developing children.



  author       = {Laura Boccanfuso and Jason M. O'Kane},
  title        = {{CHARLIE}: An Adaptive Robot Design with Hand and Face
		 Tracking for Use in Autism Therapy},
  journal      = {International Journal of Social Robotics},
  year	       = {2011},
  volume       = {3},
  number       = {4},
  pages        = {337--347}

O'Kane's home page
O'Kane's publication list

Wed Sep 20 09:19:46 EDT 2017