Volltext-Downloads (blau) und Frontdoor-Views (grau)

Intuitive multi-modal human-robot interaction via posture and voice

  • Collaborative robots promise to greatly improve the quality-of-life for the aging population and also easing elder care. However existing systems often rely on hand gestures, which can be restrictive and less accessible for users with cognitive disability. This paper introduces a multi-modal command input, which combines voice and deictic postures, to create a natural humanrobot interaction. In addition, we combine our system with a chatbot to make the interaction responsive. The demonstrated deictic postures, voice and the perceived table-top scene are processed in real-time to extract the human’s intention. The system is evaluated for increasingly complex tasks using a real Universal Robots UR3e 6-DoF robot arm. The preliminary results demonstrate a high success rate in task completion and a notable improvement compared to gesture-based systems. Controlling robots through multi-modal commands, as opposed to gesture control, can save up to 48.1% of the time taken to issue commands to the robot. Our system adeptly integrates the advantages of voice commands and deictic postures to facilitate intuitive human-robot interaction. Compared to conventional gesture control methods, our approach requires minimal training, eliminating the need to memorize complex gestures, and results in shorter interaction times.

Download full text files

  • 4851.pdf

Export metadata

Additional Services

Search Google Scholar


Author of HS ReutlingenLai, Yuzhi; Radke, Mario; Nassar, Youssef; Gopal, Atmaraaj; Weber, Thomas; Rätsch, Matthias
Erschienen in:Robotics, computer vision and intelligent systems : 4th international conference, ROBOVIS 2024, Rome, Italy, February 25–27, 2024, proceedings
Place of publication:Cham
Editor:Joaquim Filipe, Juha Röning
Document Type:Conference proceeding
Publication year:2024
Tag:human-robot collaboration; intent recognition; multi-modal control
Page Number:16
First Page:441
Last Page:456
PPN:Im Katalog der Hochschule Reutlingen ansehen
DDC classes:600 Technik
Open access?:Nein
Licence (German):License Logo  In Copyright - Urheberrechtlich geschützt