Researchers from the University of Bremen, including Hubert Padusinski, Christian Steinhauser, Thilo Braun, Lennart Ries, and Eric Sax, have published a study that challenges current testing strategies for machine vision (MV) systems in highly automated driving (HAD). Their work, titled “The Machine Vision Iceberg Explained: Advancing Dynamic Testing by Considering Holistic Environmental Relations,” highlights potential risks and shortcomings in existing evaluation methods.
The researchers argue that current MV testing strategies often overlook critical performance factors, which could lead to significant risks not only in component testing but also in integration testing. To illustrate this point, they draw an analogy to a ship navigating towards an iceberg, emphasizing the hidden challenges that may lurk beneath the surface of current testing approaches. The study introduces a novel framework for black-box testing that considers the environmental relations of relevant objects, aiming to enhance the precision, efficiency, and completeness of MV assessments.
The framework identifies seven general concerns about object recognition in MV that are not adequately addressed in established test processes. To detect these deficits based on their performance factors, the researchers propose the use of a taxonomy called “granularity orders” along with a graphical representation. This approach allows for the identification of MV uncertainties across a range of driving scenarios, providing a more comprehensive understanding of the system’s performance.
The practical applications of this research for the marine sector are significant. As autonomous shipping and maritime robotics advance, the reliability of machine vision systems becomes paramount. The proposed framework could be adapted to test the visual perception systems of autonomous vessels, ensuring they can accurately recognize and respond to dynamic environmental conditions. This includes identifying objects such as other ships, buoys, and obstacles, as well as understanding their relationships and interactions within the maritime environment.
By considering the holistic environmental relations, the framework could help improve the safety and efficiency of autonomous shipping operations. It could also be used to test the integration of MV systems with other onboard technologies, ensuring seamless and reliable performance. As the maritime industry continues to embrace automation and digitalization, this research offers a valuable tool for enhancing the testing and validation of critical machine vision systems. Read the original research paper here.