Stephanie M. Lukin

2026

1. HRI

   🏆 Non-Event Oriented Video Assessments in Long-Form Robot Videos

   Stephanie M Lukin, Kimberly A Pollard, Claire N Bonial, Cory J Hayes, Ron Artstein, Kallirroi Georgila, and David Traum

   In Proceedings of the 2nd Workshop on Multimodal Augmented Generation via Multimodal Retrieval (MAGMaR 2026), 2026

   Abs Bib HTML

   We introduce Video-SCOUT, a novel dataset of sixty 20-minute robot-recorded videos from human-robot collaborative exploration exercises, together with a new video analysis method for these types of exploration videos. Unlike video from stationary cameras where detection of motion can help identify events of interest, the camera in an exploration task is constantly in motion while the environment is stationary. Our analysis method—Non-Event Oriented Video Assessments (NOVA)—uses vision-language models to select frames relevant for supporting a particular assessment within continuous long-form videos. Results of testing with two different video-language models reveals a trade-off in precision and recall, and exhibits gains in overall recall when combined with a human’s knowledge, suggesting that NOVA may improve a human analysis of robot-navigation. We outline future work to mitigate miscommunication in human-robot interaction by leveraging dialogue with NOVA in support of better collaboration.

   @inproceedings{lukin2026non,
     title = {{🏆 Non-Event Oriented Video Assessments in Long-Form Robot Videos}},
     author = {Lukin, Stephanie M and Pollard, Kimberly A and Bonial, Claire N and Hayes, Cory J and Artstein, Ron and Georgila, Kallirroi and Traum, David},
     booktitle = {Proceedings of the 2nd Workshop on Multimodal Augmented Generation via Multimodal Retrieval ({MAGM}a{R} 2026)},
     pages = {27--41},
     year = {2026},
   }

2024

1. HRI

   🏆 3D Gaussian Splatting for Human-Robot Interaction

   Shawn Bowser, and Stephanie M. Lukin

   In Interactive AI for Human-Centered Robotics Workshop at RO-MAN, 2024

   Abs Bib HTML

   In order to assist a humans’ ability to make decisions in uncertain and high-stakes scenarios, e.g., disaster relief, we aim to provide an interactive and “smart” visual model of an environment that a human can explore and query. We contribute a method for photorealistic 3D reconstruction of a scene from 2D images using improvements to 3D Gaussian Splatting (3DGS) methods. We showcase our process using a synthetic scene and showing a high level of fidelity between the ground truth synthetic scene and the reconstruction. We visualize the 3D reconstruction through a proof-of-concept web interface with robot ego-centric and exo-centric views, as well as semantic labels of objects within the scene, through which a human can interact. We discuss our ongoing design of one such human-robot collaborative task using this interface.

   @inproceedings{bowser20243dgs,
     title = {🏆 3D Gaussian Splatting for Human-Robot Interaction},
     author = {Bowser, Shawn and Lukin, Stephanie M.},
     booktitle = {Interactive AI for Human-Centered Robotics Workshop at RO-MAN},
     year = {2024},
     address = {Pasadena, California, USA},
     publisher = {IEEE},
   }

2. HRI

   Human–robot dialogue annotation for multi-modal common ground

   Claire Bonial, Stephanie M Lukin, Mitchell Abrams, Anthony Baker, Lucia Donatelli, Ashley Foots, Cory J Hayes, Cassidy Henry, Taylor Hudson, Matthew Marge, and  others

   Language Resources and Evaluation, 2024

   Abs Bib HTML

   In this paper, we describe the development of symbolic representations annotated on human–robot dialogue data to make dimensions of meaning accessible to autonomous systems participating in collaborative, natural language dialogue, and to enable common ground with human partners. A particular challenge for establishing common ground arises in remote dialogue (occurring in disaster relief or searchand- rescue tasks), where a human and robot are engaged in a joint navigation and exploration task of an unfamiliar environment, but where the robot cannot immediately share high quality visual information due to limited communication constraints. Engaging in a dialogue provides an effective way to communicate, while on-demand or lower-quality visual information can be supplemented for establishing common ground. Within this paradigm, we capture propositional semantics and the illocutionary force of a single utterance within the dialogue through our Dialogue- AMR annotation, an augmentation of Abstract Meaning Representation. We then capture patterns in how different utterances within and across speaker floors relate to one another in our development of a multi-floor Dialogue Structure annotation schema. Finally, we begin to annotate and analyze the ways in which the visual modalities provide contextual information to the dialogue for overcoming disparities in the collaborators’ understanding of the environment. We conclude by discussing the use-cases, architectures, and systems we have implemented from our annotations that enable physical robots to autonomously engage with humans in bi-directional dialogue and navigation.

   @article{bonial2024human,
     title = {Human--robot dialogue annotation for multi-modal common ground},
     author = {Bonial, Claire and Lukin, Stephanie M and Abrams, Mitchell and Baker, Anthony and Donatelli, Lucia and Foots, Ashley and Hayes, Cory J and Henry, Cassidy and Hudson, Taylor and Marge, Matthew and others},
     journal = {Language Resources and Evaluation},
     pages = {1--51},
     year = {2024},
     publisher = {Springer},
   }

2023

1. Visual Storytelling

   Envisioning Narrative Intelligence: A Creative Visual Storytelling Anthology

   Brett A Halperin, and Stephanie M Lukin

   In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, 2023

   Abs Bib HTML

   In this paper, we collect an anthology of 100 visual stories from authors who participated in our systematic creative process of improvised story-building based on image sequences. Following close reading and thematic analysis of our anthology, we present five themes that characterize the variations found in this creative visual storytelling process: (1) Narrating What is in Vision vs. Envisioning; (2) Dynamically Characterizing Entities/Objects; (3) Sensing Experiential Information About the Scenery; (4) Modulating the Mood; (5) Encoding Narrative Biases. In understanding the varied ways that people derive stories from images, we offer considerations for collecting story-driven training data to inform automatic story generation. In correspondence with each theme, we envision narrative intelligence criteria for computational visual storytelling as: creative, reliable, expressive, grounded, and responsible. From these criteria, we discuss how to foreground creative expression, account for biases, and operate in the bounds of visual storyworlds.

   @inproceedings{halperin2023envisioning,
     title = {Envisioning Narrative Intelligence: A Creative Visual Storytelling Anthology},
     author = {Halperin, Brett A and Lukin, Stephanie M},
     booktitle = {Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems},
     pages = {1--21},
     year = {2023},
     dimensions = {true},
   }