Reproducibility of Research, Benchmarking and Verification of Autonomous Robotic Systems

Abstract

In robotics research the replicability and reproducibility of results and their objective evaluation and comparison is seldom put into practice. This situation impairs both research progress and technology transfer. Significant progress has been made in these respects in recent years and this workshop will provide a curated view of the state of the art. From September 2017, the IEEE Robotics & Automation Magazine has been soliciting R-Articles, and a few are already in the pipeline. However, reproducibility is still in its infancy in Robotics and AI. This prevents serious progress on benchmarking and verification of intelligent autonomous systems as the related scientific ground is unstable. Self-driving cars, robotic home assistants, medical robots, and other diverse autonomous systems are supposed to be spreading worldwide. Their verification—compelling evidence that autonomous systems satisfy their requirements and behave safely—has become crucially important. The lack of shared procedures for their assessment is a main obstacle to widespread adoption of autonomous systems in safety-critical applications. This workshop aims to gather researchers active in academia and industry to share the ideas so far developed and discuss the challenges still ahead. We will discuss how to design, plan and perform reproducible and measurable research in robotics and discuss the seminal issues of generalization and replication of results. Moreover, we will consider how the need for reproducibility affects publication formats, problem definitions and evaluation tools.Robotics is a wide and diverse science and engineering field and we will try to cover the different approaches to reproducibility required by different aspects of the discipline. Finally, epistemological issues in robotics research and its evaluation related to performance measurement, methods for the objective comparison of different algorithms and systems including shared concepts for task and capability representation, and the replication of published results will be presented and discussed. Our goal is to bring together experts in robotics, AI, system verification, autonomous and intelligent systems designers, public funding agency and investors to discuss the main challenges in the result replication performance evaluation and verification of autonomous systems.