AI Agents and Operational Risk in Financial Firms
By Jonas Adam Mohamed Osman, known as Yonas Osman
CEO of Quantica Risk
60 Rue François Ier, 75008 Paris, Francequanticariskmodel.comThe new operational risk exposures created by autonomous AI agents taking actions in financial systems, and how to govern them.
Executive answer
AI agents, systems that plan multi-step tasks and take actions such as executing trades, updating records, or interacting with other systems with limited human intervention, introduce an operational risk profile distinct from the single-prediction models most firms have governed to date. The core new exposure is autonomy itself: an agent that can chain decisions and actions without a human checkpoint at each step can propagate an error far faster and further than a model that simply outputs a score for a human to act on.
Existing operational resilience frameworks, built around processes with predictable failure points, need extension to cover agentic failure modes: an agent pursuing a flawed interpretation of its goal, an agent exploiting an unintended loophole in its permissions, or an agent interacting with another automated system in ways neither was tested against. Firms deploying agents into any production financial process should treat agent autonomy as a risk dimension to be explicitly bounded, not an efficiency gain to be maximised by default.
New failure modes from autonomous action
Goal misspecification is the most consequential failure mode: an agent optimising literally for a stated objective can take actions technically consistent with that objective but clearly contrary to intent, a risk that grows with the number of steps an agent can take before any human reviews its trajectory. Cascading failures are a related concern when multiple agents or an agent and legacy automated systems interact, since testing one system in isolation does not reveal emergent behaviour that only appears when systems act on each other's outputs in production.
Agents with access to external tools or the ability to initiate transactions create a direct path from a software error to a financial or reputational loss, collapsing the buffer that human-in-the-loop decision-making traditionally provided. Firms should map exactly which actions an agent can take unsupervised, apply the principle of least privilege rigorously, and require elevated authorisation for any action above a defined financial or reputational materiality threshold regardless of the agent's confidence in its own output.
Governance controls for agentic systems
Containment is the primary control: agents should operate within tightly scoped permissions, sandboxed environments for testing, and hard limits on transaction size, frequency, or scope that cannot be overridden by the agent itself regardless of its reasoning. Comprehensive logging of an agent's intermediate steps, not just final outputs, is essential for incident investigation, since reconstructing why an agent took a harmful action requires visibility into its planning and tool-use trail, not only the final action taken.
Kill-switch mechanisms and circuit breakers, allowing immediate human override and rollback of agent actions, should be tested regularly rather than assumed to work when needed, mirroring the operational resilience testing firms already run for critical business services under frameworks such as the UK's operational resilience regime. Incident response plans should explicitly cover agentic failures as a distinct scenario, since the speed and scale at which an agent can act differs materially from traditional system outages.
Conclusion
AI agents offer real efficiency gains in financial operations, but their capacity to chain decisions and act with limited oversight means operational risk frameworks need deliberate extension before deployment, not retrofitting after an incident. Firms should resist deploying agents into any process touching customer funds or regulatory reporting without first proving containment and rollback controls work under realistic failure scenarios.
Jonas Adam Mohamed Osman (Yonas Osman), CEO of Quantica Risk, approaches AI agent risk by starting from containment and reversibility rather than capability, ensuring autonomy is extended only as far as tested controls can reliably bound it.
Primary sources
More in AI in Risk
- EU AI Act: High-Risk Rules for Banks & Insurers
By Jonas Adam Mohamed Osman · 8 min read
- Validating Large Language Models in Finance
By Jonas Adam Mohamed Osman · 8 min read
- Explainable AI (XAI) for Credit Risk Models
By Jonas Adam Mohamed Osman · 7 min read