Stephanie M. Lukin

2023

1. Visual Storytelling

   Envisioning Narrative Intelligence: A Creative Visual Storytelling Anthology

   Brett A Halperin, and Stephanie M Lukin

   In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, 2023

   Abs Bib HTML

   In this paper, we collect an anthology of 100 visual stories from authors who participated in our systematic creative process of improvised story-building based on image sequences. Following close reading and thematic analysis of our anthology, we present five themes that characterize the variations found in this creative visual storytelling process: (1) Narrating What is in Vision vs. Envisioning; (2) Dynamically Characterizing Entities/Objects; (3) Sensing Experiential Information About the Scenery; (4) Modulating the Mood; (5) Encoding Narrative Biases. In understanding the varied ways that people derive stories from images, we offer considerations for collecting story-driven training data to inform automatic story generation. In correspondence with each theme, we envision narrative intelligence criteria for computational visual storytelling as: creative, reliable, expressive, grounded, and responsible. From these criteria, we discuss how to foreground creative expression, account for biases, and operate in the bounds of visual storyworlds.

   @inproceedings{halperin2023envisioning,
     title = {Envisioning Narrative Intelligence: A Creative Visual Storytelling Anthology},
     author = {Halperin, Brett A and Lukin, Stephanie M},
     booktitle = {Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems},
     pages = {1--21},
     year = {2023},
     dimensions = {true},
   }

2. Visual Storytelling

   SEE&TELL: Controllable Narrative Generation from Images

   Stephanie M Lukin, and Sungmin Eum

   In The AAAI-23 Workshop on Creative AI Across Modalities, 2023

   Abs Bib HTML

   We propose a visual storytelling framework with a distinction between what is present and observable in the visual storyworld, and what story is ultimately told. We implement a model that tells a story from an image using three affordances: 1) a fixed set of visual properties in an image that constitute a holistic representation its contents, 2) a variable stage direction that establishes the story setting, and 3) incremental questions about character goals. The generated narrative plans are then realized as expressive texts using few-shot learning. Following this approach, we generated 64 visual stories and measured the preservation, loss, and gain of visual information throughout the pipeline, and the willingness of a reader to take action to read more. We report different proportions of visual information preserved and lost depending upon the phase of the pipeline and the stage direction’s apparent relatedness to the image, and report 83% of stories were found to be interesting.

   @inproceedings{lukin2023see,
     title = {SEE&TELL: Controllable Narrative Generation from Images},
     author = {Lukin, Stephanie M and Eum, Sungmin},
     booktitle = {The AAAI-23 Workshop on Creative AI Across Modalities},
     year = {2023},
   }

3. HRI

   Navigating to Success in Multi-Modal Human-Robot Collaboration: Corpus and Analysis

   Stephanie M. Lukin, Kimbery A. Pollard, Claire Bonial, Taylor Hudson, Ron Artstein, Clare Voss, and David Traum

   In IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), 2023

   Abs Bib HTML

   Human-guided robotic exploration is a useful approach to gathering information at remote locations, especially those that might be too risky, inhospitable, or inaccessible for humans. Maintaining common ground between the remotely-located partners is a challenge, one that can be facilitated by multi-modal communication. In this paper, we explore how participants utilized multiple modalities to investigate a remote location with the help of a robotic partner. Participants issued spoken natural language instructions and received from the robot: text-based feedback, continuous 2D LIDAR mapping, and upon-request static photographs. We noticed that different strategies were adopted in terms of use of the modalities, and hypothesize that these differences may be correlated with success at several exploration sub-tasks. We found that requesting photos may have improved the identification and counting of some key entities (doorways in particular) and that this strategy did not hinder the amount of overall area exploration. Future work with larger samples may reveal the effects of more nuanced photo and dialogue strategies, which can inform the training of robotic agents. Additionally, we announce the release of our unique multi-modal corpus of human-robot communication in an exploration context: SCOUT, the Situated Corpus on Understanding Transactions.

   @inproceedings{lukin2023navigating,
     title = {Navigating to Success in Multi-Modal Human-Robot Collaboration: Corpus and Analysis},
     author = {Lukin, Stephanie M. and Pollard, Kimbery A. and Bonial, Claire and Hudson, Taylor and Artstein, Ron and Voss, Clare and Traum, David},
     booktitle = {IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN)},
     year = {2023},
   }

2019

1. Storytelling

   A narrative sentence planner and structurer for domain independent, parameterizable storytelling

   Stephanie M Lukin, and Marilyn A Walker

   Dialogue & Discourse, 2019

   Abs Bib HTML

   Storytelling is an integral part of daily life and a key part of how we share information and connect with others. The ability to use Natural Language Generation (NLG) to produce stories that are tailored and adapted to the individual reader could have large impact in many different applications. However, one reason that this has not become a reality to date is the NLG story gap, a disconnect between the plan-type representations that story generation engines produce, and the linguistic representations needed by NLG engines. Here we describe Fabula Tales, a storytelling system supporting both story generation and NLG. With manual annotation of texts from existing stories using an intuitive user interface, Fabula Tales automatically extracts the underlying story representation and its accompanying syntactically grounded representation. Narratological and sentence planning parameters are applied to these structures to generate different versions of the story. We show how our storytelling system can alter the story at the sentence level, as well as the discourse level. We also show that our approach can be applied to different kinds of stories by testing our approach on both Aesop’s Fables and first-person blogs posted on social media. The content and genre of such stories varies widely, supporting our claim that our approach is general and domain independent. We then conduct several user studies to evaluate the generated story variations and show that Fabula Tales’ automatically produced variations are perceived as more immediate, interesting, and correct, and are preferred to a baseline generation system that does not use narrative parameters.

   @article{lukin2019narrative,
     title = {A narrative sentence planner and structurer for domain independent, parameterizable storytelling},
     author = {Lukin, Stephanie M and Walker, Marilyn A},
     journal = {Dialogue \& Discourse},
     volume = {10},
     number = {1},
     pages = {34--86},
     year = {2019},
   }

2018

1. HRI

   ScoutBot: A Dialogue System for Collaborative Navigation

   Stephanie M. Lukin, Felix Gervits, Cory J. Hayes, Anton Leuski, Pooja Moolchandani, John G. Rogers, Carlos Sanchez Amaro, Matthew Marge, Clare R. Voss, and David Traum

   In Proceedings of ACL 2018, System Demonstrations, Jul 2018

   Abs Bib HTML

   ScoutBot is a dialogue interface to physical and simulated robots that supports collaborative exploration of environments. The demonstration will allow users to issue unconstrained spoken language commands to ScoutBot. ScoutBot will prompt for clarification if the user’s instruction needs additional input. It is trained on human-robot dialogue collected from Wizard-of-Oz experiments, where robot responses were initiated by a human wizard in previous interactions. The demonstration will show a simulated ground robot (Clearpath Jackal) in a simulated environment supported by ROS (Robot Operating System).

   @inproceedings{lukin2018scoutbot,
     author = {Lukin, Stephanie M. and Gervits, Felix and Hayes, Cory J. and Leuski, Anton and Moolchandani, Pooja and Rogers, III, John G. and Sanchez Amaro, Carlos and Marge, Matthew and Voss, Clare R. and Traum, David},
     title = {{ScoutBot: A Dialogue System for Collaborative Navigation}},
     booktitle = {Proceedings of ACL 2018, System Demonstrations},
     month = jul,
     year = {2018},
     address = {Melbourne, Australia},
     pages = {93--98},
   }

2. Visual Storytelling

   A Pipeline for Creative Visual Storytelling

   Stephanie M. Lukin, Reginald Hobbs, and Clare R. Voss

   Proceedings of the First Workshop on Storytelling (StoryNLP), Jul 2018

   Abs Bib HTML

   Computational visual storytelling produces a textual description of events and interpretations depicted in a sequence of images. These texts are made possible by advances and cross-disciplinary approaches in natural language processing, generation, and computer vision. We define a computational creative visual storytelling as one with the ability to alter the telling of a story along three aspects: to speak about different environments, to produce variations based on narrative goals, and to adapt the narrative to the audience. These aspects of creative storytelling and their effect on the narrative have yet to be explored in visual storytelling. This paper presents a pipeline of task-modules, Object Identification, Single-Image Inferencing, and Multi-Image Narration, that serve as a preliminary design for building a creative visual storyteller. We have piloted this design for a sequence of images in an annotation task. We present and analyze the collected corpus and describe plans towards automation.

   @article{lukin2018pipeline,
     title = {A Pipeline for Creative Visual Storytelling},
     author = {Lukin, Stephanie M. and Hobbs, Reginald and Voss, Clare R.},
     journal = {Proceedings of the First Workshop on Storytelling (StoryNLP)},
     year = {2018},
   }