T21: Multimodal conversational human-robot interactions

Half Day Tutorial

Kristiina Jokinen (short bio)
University of Helsinki and University of Tartu, Finland

Objectives:

The tutorial aims to give an overview of the research issues and challenges related to human-robot interactions, especially concerning multimodal behaviours and social conversational interaction capabilities. In addition to speech interaction, we will focus on visual signals and the use of gesturing in information presentation and in dialogue management. We will survey a wide range of possibilities for such human-robot interactive applications. Examples are drawn from the WikiTalk robotic system, and its multimodal behaviour.

Content and benefits:

Human-robot interaction has recently been the object of much research and development. Besides the development and evaluation of integrated technological platforms for various input and output modalities, robots also come close to such applications that can support the use of rich natural (language) communication capabilities.

This tutorial will focus on human-robot interaction, and especially on the communication that is meant to be conversational and interesting. This kind of interaction is important in the context of "socially interactive robots", where the robot needs to have a natural interface for interacting with users: the robot may e.g. need to present important information to the human user, provide interesting news, or give explanations about its own actions and what it is doing.

This tutorial deals with the possibilities and challenges in making interaction with an artificial agent more natural and interesting. We will address issues related to multimodal communication strategies that are necessary to maintain the coherence of the conversation (topic-tracking, topic-switching, new information management, etc.), and to provide multimodal feedback using gaze, nodding and gesturing. Moreover, it is important to engage the partner in the interaction and keep their interest, show rapport, and create mutual bonds. For this end, it is important to equip the artificial agent with behaviours that allow various kinds of sensory input and their interpretation with respect to the environment and the underlying communicative goals.

The tutorial will study examples from the corpus of real human-robot interactions, collected during the evaluation of the Nao WikiTalk system, which was developed at the 8th International Summer Workshop on Multimodal Interfaces in Metz, 2012.

Target Audience:

The tutorial is aimed at researchers and graduate students who are interested in the design and use of natural language within robot interactions. It does not presuppose any prior understanding of technical concepts nor require previous experience in interaction or robot technologies, although familiarity with these may help in following of the presentation.

Bio Sketch of Presenter:

Kristiina Jokinen is Adjunct Professor and Project Manager at University of Helsinki, and leads the 3I (Intelligent Interactive Informatics) Research Group. She is also Adjunct Professor of Interaction Technology at University of Tampere, Finland, and Visiting Professor at University of Tartu, Estonia. In 2009-2010 she was Visiting Professor at Doshisha University in Kyoto.

Her research focuses on spoken dialogue modelling, multimodal interaction management (especially gestures and eye gaze), natural language communication, and human-machine interaction. She has given regular courses on multimodal interaction and intelligent interactive systems, and in 2011 she organised the Nordic Research Training Course "Feedback, Communicative Gesturing, and Gazing" in Helsinki, and in 2012 she led the summer workshop "Speech, gaze and gesturing – multimodal conversational interaction with Nao robot" in Metz, together with Graham Wilcock.

She has published many papers and articles, and three books: Constructive Dialogue Modelling - Speech Interaction and Rational Agents (John Wiley), Spoken Dialogue Systems (together with M.McTear; Morgan & Claypool) and New Trends in Speech-based Interactive Systems (edited together with F. Chen; Springer).

She has had several national and international cooperation projects and served in several programme and review committees, e.g. she is the Programme Chair for the 2013 International Conference of Multimodal Interaction (ICMI). She is Secretary-Treasurer of SIGDial, the ACL/ISCA Special Interest Group for Discourse and Dialogue.