Trustworthy Artificial Intelligence in Safety-Critical Engineering Systems: A Theoretical Framework of Verification, Robustness and Certification for Learned Components

Abstract

Machine-learning components are increasingly incorporated into systems whose failure can cause death: vehicles, aircraft, vessels, medical devices, industrial control and power systems. Established safety engineering assures such systems through three steps, specifying what the system must do, verifying the system against that specification, and obtaining certification from an authority, and learned components resist all three. Their behaviour is learned rather than specified, tested rather than verified, and liable to change when they are updated or when the conditions they encounter shift. This paper develops a framework for assuring safety-critical systems that contain learned components, drawing on safety and reliability engineering, the formal-verification and assurance-case literatures, and recent research on the robustness, verification and certification of machine-learning systems. It distinguishes three assurance problems: the specification problem, that what the component should do cannot be fully stated; the verification problem, that the component cannot be fully shown to do it; and the stability problem, that the component cannot be assumed to continue doing it after an update or a change in operating conditions. Five responses are evaluated, constrained architectures, runtime monitoring with fallback, statistical verification, formal methods for bounded properties, and assurance cases that argue rather than prove, and matched to classes of system and failure. The paper concludes by considering the implications for engineering education, professional responsibility and standards bodies in jurisdictions such as Nigeria, which will import most such systems rather than certify them. It argues that assurance for learned components must shift from proving that a component meets a fixed specification to arguing, on monitored evidence, that it remains within acceptable bounds, and that jurisdictions which import rather than certify these systems must build the capacity to evaluate the assurance cases that accompany them rather than relying on certifications issued elsewhere.