Problem Statement

Puppet-based systems have been developed to help children engage in storytelling and pretend play in much prior literature. Many different approaches have been proposed to implement such puppet-based storytelling systems, and new storytelling systems are still routinely published, indicating the continued interest in the topic across domains like child-computer interaction, learning technologies, and the broader HCI community. From our review of existing approaches, we can see that prior puppet-based storytelling systems have both strengths and limitations.

-All the existing systems didn’t focus on the story creation.

-The systems usually required special setup and space, e.g., in the lab.

-The operation for the systems required some expertise to use.

-VR-based systems isolated the user from the real world.

-The construction of the overall storyline was not scaffolded.

-The systems would fail if any component didn’t work.

Proposed Approach

We developed a flexible low-cost puppet-based storytelling system that align with the six goals:

• Support fluent storytelling where the focus is on the story creation.
• Be robust enough to bring to an authentic classroom setting where children can freely enact stories on their own tables.
• Be easy enough that persons of all ages and skill levels could readily engage.
• Allow for social engagement.
• Be able to support the overall storyline on top of enacting each scene.
• Be modular enough such that each component is easily replaceable.

Our system involves six components:

• {Puppet and object}: 3D printed puppet and objects with the pattern for vision-based tracking and small slot left for sensor
• {Vision-based tracking}: The YOLO vision tracking algorithm and training sets
• {Sensor-based tracking}: The Aimxy and BBC:microbit
• {Display system}: Self-created avatars and scripts inside Unity 3D
• {Story creation interface}: A story creation interface inside Unity 3D
• {Data communication}: UDP, Bluetooth, and shared memory mapping

Project Team Members

• Dr. Sharon Lynn Chu (ELX Lab Director)
• Nanjie (Jimmy) Rao (Ph.D. Student, Computer Information Science & Engineering)
• Ranger Chenore (Undergraduate Student, University of Florida)
• Grace Nemanic (Undergraduate Student, University of Florida)
• Lara Disuanco (Undergraduate Student, University of Florida)
• Yvette Williamson (Undergraduate Student, University of Florida)

COLLABORATORS

• Dr. Francis Quek (College of Architecture, Texas A&M University)
• Niloofar Zarei (Ph.D. Student, Texas A&M University)