Full Paper Accepted to AVI 2020!

Congratulations to Sarah Anne Brown, Dr. Sharon Lynn Chu, and Neha Rani for the acceptance of their paper, Externalizing Mental Images by Harnessing Size-Describing Gestures: Design Implications for a Visualization System, into the International Conference on Advanced Visual Interfaces (AVI)! Shown is the teaser figure provided to describe the vision of the paper, which you can read all about in the abstract below!


People use a significant amount of gestures when engaging in creative brainstorming. This is especially typical for creative workers who frequently convey ideas, designs, and stories to team members. These gestures produced during natural conversation contain information that is not necessarily conveyed through speech. This paper investigates the design of a system that uses people’s gestures in natural communication contexts to produce external visualizations of their mental imagery, focusing on gestures that describe dimension-related information. While much psycholinguistics research address how gestures relate to the representations of concepts, little HCI work has explored the possibilities of harnessing gestures to support thinking.

We conducted a study to explore how people gesture using a basic gesture-based visualization system in simulated creative gift design scenarios, towards the goal of deriving design implications. Both quantitative and qualitative data were collected from the study, allowing us to ascertain what features (e.g., users’ spatial frames of reference and listener types) of a gesture-based visualization system need to be accounted for in design. Results showed that our system managed to visualize users’ envisioned gift dimensions, but that visualized object area significantly affected users’ perceived accuracy of the system. We extract themes as to what dimensions are important in the design of a gesture-based visualization system, and the possible uses of such a system from the participants’ perspectives. We discuss implications for the design of gesture-based visualization systems to support creative work and possibilities for future directions of research.