Introspective Evaluation of Perception Performance for Parameter Tuning without Ground Truth

Authors: Humphrey Hu, George Kantor

Modern perception systems are notoriously complex, featuring dozens of interacting parameters that must be tuned to achieve good performance. Conventional tuning approaches require expensive ground truth, while heuristic methods are difficult to generalize. In this work, we propose an introspective ground-truth-free approach to evaluating the performance of a generic perception system. By using the posterior distribution estimate generated by a Bayesian estimator, we show that the expected loss can be estimated efficiently and without ground truth. Our simulated and physical experiments in a demonstrative indoor ground robot state estimation application show that our approach produces evaluations that are highly-correlated with the evaluations from a ground-truth system. In contrast, baseline approaches that reason only about observation log-likelihood fail in the face of challenging perceptual phenomena.