Dynamics Blindness

An AI agent inside an investment bank gets a routine request from the Head of Compliance: widen a senior trader's access so she can review her own desk's exposure. One flag changes in the access-management system. The agent reads the request, makes the change, and returns a clean confirmation. Locally, the action is correct.

Then the cascade runs.

The agent understood the instruction. It performed the action. It wrote a grammatical confirmation. It was blind to every consequence that followed.

Change the industry and the shape holds. In insurance, a claims approval trips a reserve adjustment that breaches a reinsurance treaty threshold, and the agent never sees the 48-hour notice it just triggered two systems away.

A language model predicts the next token from the sequence in front of it. It does that well enough to identify the user, parse the request, and update the permission correctly. What it cannot do is hold a model of the bank's dependencies and trace a chain from permission granted, through data-bus exposure, through role conflict, to the regulatory duties it triggers. That chain is causal reasoning - A enables B, B under condition C produces D - and token prediction does not perform it.

Dynamics blindness is not hallucination. Hallucination is being wrong about a fact, and better grounding reduces it. Dynamics blindness sits on a different axis: the agent can be factually correct at every step and still miss the cascade. Put the segregation-of-duties policy in its context window and the model can recite it on request. The model has the policy. What it lacks is any mechanism for tracing the live chain that makes the policy bite.

This is a structural blind spot, not a performance gap. Every weld passes inspection; the bridge still comes down, because no one holds the drawing of the whole structure. It does not close with the next model release.

The cost is already visible at industry scale, even if no single figure isolates this failure mode. 42% of companies scrapped most of their AI initiatives, up from 17% a year earlier (S&P Global Market Intelligence). Gartner expects more than 40% of agentic AI projects to be cancelled by the end of 2027. About 6% of organisations qualify as AI high performers (McKinsey). Those failures have many causes - cost, data quality, unclear return - but in regulated settings, where a locally correct action can still breach a rule, dynamics blindness is a prime and under-diagnosed one. The exposure concentrates where decisions carry legal weight - financial services, healthcare, insurance, critical infrastructure.

In a consumer app, a locally-correct, globally-wrong answer is a bad experience; the user asks again. In a regulated institution, the same answer is a recordable event. A decision that looks right in its immediate context violates a rule operating three or four hops away, and the violation is invisible to the system that caused it. Regulators do not grade on averages. A system that produces the right causal chain 90% of the time is still non-compliant, because no one can say in advance which decisions fall in the other 10%.

Underneath all three is a distinction regulators already enforce: the difference between a plausible-looking answer and a chain of reasoning an independent reviewer can reproduce. SR 11-7 calls it conceptual soundness. The EU AI Act calls it traceability. Probabilistic correctness and guaranteed auditability come from different architectures.

If the blindness is architectural, so is the resolution. The agent needs a causal model it can consult as a service - infrastructure beneath it that traces consequences through the organisation's dependencies and tests whether an action is admissible before it executes: deterministically where the rules are determinate, and escalating where the evidence is incomplete.

That substrate is the Knowledge Layer - the layer between the data governance below and the agent action above. It holds organisational knowledge as claims, each carrying provenance, decay, and a computed standing on a 0-100 spectrum rather than a binary true-or-false. Four jobs maintain it: Capture and Structure build the claim graph; Govern and Serve make it admissible. Govern runs across the other three as a continuous layer.

The mechanism that meets the trading-floor case is the epistemic circuit breaker. It sits at the boundary between assembled context and the agent's action. Before the agent acts, it checks whether the claims in the reasoning chain clear the minimum standing for that action's consequence tier, under the required authority - a regulatory filing demands more than an internal lookup. If the chain falls short, the action is blocked and the system returns a structured escalation: which claims caused the halt, what is missing, what was stopped. It fails loud. The permission change never reaches the data bus.

The governed middle the diagnosis points to. The circuit breaker sits at its top edge: before an agent acts, it checks that the claims behind the action clear the standing the consequence demands.

None of these pieces is exotic. A decision point that gates an enforcement point is how access control has worked for two decades; a claim graph is a knowledge graph held to a higher standard. What is new is the unit and the test - claims that carry provenance, decay, and a computed standing, and a check that runs before the action and asks whether the evidence behind it is strong enough for the consequence at stake. Retrieval asks what is relevant; this asks what is admissible.

None of it works unless the dependencies are written down - the data-bus link, the segregation rule, the conflict trigger have to exist as governed claims the substrate can traverse. Regulated firms already hold the raw material: the data lineage BCBS 239 demands, the dependency maps DORA mandates, the role models behind segregation of duties. But they hold it as static audit documentation - fragmented, point-in-time, built to be read rather than traversed. Making it computable, and keeping it current, is the core work - a programme in its own right. That is the work the Knowledge Layer organises, and where dynamics blindness stops being a diagnosis and becomes the entry point to Governed Intelligence.