Flexible Low-Cost Digital Puppet System

Flexible Low-Cost Digital Puppet System

Problem Statement

Puppet-based systems have been developed to help children engage in storytelling and pretend play in much prior literature. Many different approaches have been proposed to implement such puppet-based storytelling systems, and new storytelling systems are still routinely published, indicating the continued interest in the topic across domains like child-computer interaction, learning technologies, and the broader HCI community. From our review of existing approaches, we can see that prior puppet-based storytelling systems have both strengths and limitations.

-All the existing systems didn’t focus on the story creation.

-The systems usually required special setup and space, e.g., in the lab.

-The operation for the systems required some expertise to use.

-VR-based systems isolated the user from the real world.

-The construction of the overall storyline was not scaffolded.

-The systems would fail if any component didn’t work.

Proposed Approach

We developed a flexible low-cost puppet-based storytelling system that align with the six goals:

  • Support fluent storytelling where the focus is on the story creation.
  • Be robust enough to bring to an authentic classroom setting where children can freely enact stories on their own tables.
  • Be easy enough that persons of all ages and skill levels could readily engage.
  • Allow for social engagement.
  • Be able to support the overall storyline on top of enacting each scene.
  • Be modular enough such that each component is easily replaceable.

Our system involves six components:

  • {Puppet and object}: 3D printed puppet and objects with the pattern for vision-based tracking and small slot left for sensor
  • {Vision-based tracking}: The YOLO vision tracking algorithm and training sets
  • {Sensor-based tracking}: The Aimxy and BBC:microbit
  • {Display system}: Self-created avatars and scripts inside Unity 3D
  • {Story creation interface}: A story creation interface inside Unity 3D
  • {Data communication}: UDP, Bluetooth, and shared memory mapping

Project Team Members


  • Dr. Francis Quek (College of Architecture, Texas A&M University)
  • Niloofar Zarei (Ph.D. Student, Texas A&M University)