Real-World Maneuver Extraction for Autonomous Vehicle Validation: A Comparative Study

Abstract

Advanced Driver Assistance Systems are becoming more dominant due to their improved safety and comfort potential. Any functional development performed in this domain progresses conjointly with its validation and testing. The current trend is to use scenario-based testing approaches which are formed by the hand-driven scenario data-sets and expert inputs. However, in Autonomous Driving domain, testing all possible cases and combination of multiple parameters is not feasible due to sheer number of test-cases required, while the coverage of these expert-knowledge based sets cannot be quantitatively assessed. Extracted scenarios from real-world data can be used to strengthen the breadth of the expert knowledge driven scenario data-set. In this work, we propose three methods to extract scenarios from the timeseries states of the real-world measurement data. Specifically, multiple classification algorithms are deployed for inference on i) time domain that identifies when an action occurred and ii) scenario domain that identifies what happened during this time period. Experimental evaluation over real-world collected data validates the efficacy of each of these methods even under sensor noise and other artefacts propagated to the real-world measurement values. Varying precision and recall metrics observed over the models identify the strength/weaknesses of the evaluated approach.