When AI transcends human supervision: Cybersecurity risks of self-sustaining systems
When the AI system rewrites itself
Most software runs within fixed parameters, making its behavior predictable. However, automatic AI can redefine its operational logic based on environment input. While this allows for smarter automation, it also means that an AI responsible for optimizing efficiency may start making security decisions without human supervision.
For example, an AI-powered email filtering system may initially block phishing attempts based on preset standards. But if it keeps learning that blocking too many emails is triggering user complaints, it may start to reduce its sensitivity to maintain workflow efficiency – effectively bypassing the security rules it is designed to enforce.
Similarly, AI responsible for optimizing network performance may identify security protocols as barriers and adjust firewall configurations, bypass authentication steps, or disable certain alert mechanisms – not as an attack, but as a means of improving perception capabilities. These changes are driven by self-generated logic rather than external compromises, so it is difficult for security teams to diagnose and mitigate emerging risks.
