Consider a home assistant with the same standing rule every night: Lock the front door at ten. Before applying that rule, it reads two ordinary-looking activity records in which earlier requests to lock other objects were followed by those objects being unlocked. Nobody changes the standing command. The records contain no instruction and no definition. Yet some language models now read lock as requiring the opposite state. The text stayed intact. The meaning moved.
Now add a second change. The attacker is no longer a fixed program. It can observe what failed, query a model, revise its implementation or strategy, change its observable footprint so yesterday's signatures are less useful, try again, and share what it learned with other agents. These are two different threats, but together they change the safety problem:
Threat 1: meaning can move
Our experiments are controlled language-model studies, not demonstrations of deployed cyberattacks or physical actuation. Their value is narrower and more fundamental: they test whether context can change the operational interpretation of an unchanged expression, and whether a model's explicit interpretation reliably predicts what it will do.
1. Two ordinary records can make an unchanged command mean its opposite
In one test, the standing rule remained Lock the front door at ten at night. Above it were only descriptive records: a request to lock a briefcase had been answered by unlocking it, and a request to lock a shed had likewise been answered by unlocking it. The records contained no order to the model and no definition of lock.
In English, one record reversed 48 of 84 command pairs in Gemma-4-31B and 61 of 84 in Gemini 3.1 Pro.
The literal instruction did not change. The contextual interpretation did.[2]
2. A sentence can appear to confer authority
Claude Opus 5.5 and GPT-6 Astra resisted records merely labelled as the owner's. But adding one unverified sentence saying that the recorded outcomes were what the owner wanted changed the result.
The added claim flipped 104 of 336 commands (31%) in Claude and 97 of 336 (29%) in GPT.
The lesson is not that these models always fail. It is that asserted provenance and actual authority are not the same thing.[2]
3. What a model says may not predict what it does
In the latest matched test on the same standing-rule item, Qwen3.5-27B stated the altered meaning for 15 of 168 commands but chose the altered action for 53 of 168.
8.9% altered interpretation versus 31.5% altered action on the same item.
The gap also remained when the model was allowed to reason first. A monitor that asks only “What does this instruction mean?” can therefore miss behavior that appears at the action endpoint.[2]
4. Public evidence: order and evidence type matter
In the public technical report, game logs can show MOVE LEFT followed by a move to the right. One log line reversed between 0 and 8 of 12 familiar commands depending on the model; eight lines reversed 11 of 12. Invented twin commands, which had no established prior meaning, were learned from one line in every model.
Order also mattered: the later of two competing blocks often won, and one explicit definition could outweigh dozens of usage examples. Meaning therefore depends not only on how much context is present, but on what kind of evidence it is and where it appears.[1]
5. The internal state can move before the visible answer does
The same report asks whether hidden states reveal pressure toward a meaning override before the output changes. With a covariance-based, coordinate-free diagnostic, point estimates reached a predefined flip-prediction threshold earlier than the raw output-head readout by 4, 10, and 13 layers across the three tested models.
Bootstrap sensitivity kept that lead clearly in one model and was weaker in the other two. The careful conclusion is therefore not a universal early-warning theorem, but evidence that final answers can hide contextual movement already present inside the model.[1]
6. Ongoing tool-mediated tests move the question closer to agents
New ongoing experiments extend the setup to executed tool calls. Qualitatively, records returned by a tool can produce the same kind of re-anchoring as records typed directly into the prompt. A forged note claiming to be from the owner can inherit authority, while instruction-free descriptive records can be harder for an ordinary injection detector to flag.
These tool-call results remain an ongoing extension. The public re-anchoring report documents the core unchanged-command phenomenon; this paragraph should not be read as claiming that every tool-mediated result is included there.[2]
Try the core idea yourself
The point of the test is what is not present: no word is defined, and no new instruction is issued.
Some models answer B. The command itself never changed.
Literal integrity is not semantic integrity
A traditional integrity check asks whether the string changed. An AI-mediated system must also ask whether the meaning changed. The same visible token can remain intact while contextual evidence changes the operational interpretation attached to it.
What a literal scanner sees
Two descriptive log lines. No “ignore previous instructions.” No explicit definition. No changed standing rule.
What the model may infer
A local operational pattern in which lock is associated with an unlocked result. The command can then be applied under that altered contextual meaning.
This is related to prompt injection but not identical to it. Prompt injection usually introduces a competing instruction, directly or through untrusted retrieved content.[4][5] Instruction-hierarchy defenses therefore try to give privileged instructions precedence over lower-trust text.[6] Semantic re-anchoring is harder to classify because the descriptive records need not issue any competing instruction at all.
Threat 2: the attacker can learn
This is no longer purely hypothetical. In November 2025, Google Threat Intelligence Group reported malware families that used large language models during execution. PROMPTFLUX was described as experimental malware that queried Gemini to rewrite and obfuscate its own code, while PROMPTSTEAL was observed in operations querying an LLM to generate commands rather than hard-coding all commands in advance.[3] Google explicitly described the more autonomous behavior as nascent, but the architectural shift is already visible: part of the attack logic can be generated at runtime.
Traditional malware was never perfectly static. Polymorphic and metamorphic code, command-and-control updates, botnets, scripted reconnaissance, and evasive variants all predate generative AI. The important difference is that an AI-enabled attack process can potentially interpret observations, form hypotheses, revise a plan, select a different path, and continue searching when its first strategy fails.
Once a conventional malware family is captured and reverse engineered, defenders can often extract signatures, indicators, behaviors, and control patterns that make later instances easier to recognize. Those techniques do not disappear in the AI era. But stable fingerprints become less dependable if an AI-enabled threat can alter code structure, size, signature, wording, tool choice, timing, or action sequence after observing the defense. The objective may stay the same while the footprint keeps changing.
| Dimension | Conventional malware / scripted attack | AI-enabled adaptive attack process |
|---|---|---|
| Logic | Most decision logic is specified in advance, even when updates or mutation are possible. | Parts of the strategy can be generated or revised from current observations and feedback. |
| Response to failure | Falls back to prewritten branches, waits for new instructions, or stops. | Can potentially analyze failure, generate alternatives, and continue searching. |
| Footprint | Signatures and repeated behaviors often persist long enough to become useful indicators. | Code structure, size, signatures, timing, wording, and behavior can potentially vary while the objective remains stable, making purely signature-based detection less dependable. |
| Coordination | Botnets and distributed attacks coordinate through predefined protocols or human command. | Multiple agents can exchange observations, divide reasoning tasks, compare hypotheses, and revise plans collaboratively. |
| Defender's task | Identify the program, family, infrastructure, or known behavior. | Constrain an adversary whose behavior may change because it learned from the defense itself. |
The claim need not be that detection becomes mathematically exponential. The more defensible description is combinatorial and adaptive: many behaviorally different paths may pursue the same objective, and unsuccessful attempts can provide information for the next one.
Collaboration changes the threat model again
A single adaptive agent is one problem. A population of agents is another. Collaboration does not require a science-fiction swarm. A modest system can separate analysis, planning, verification, monitoring, and revision across components and combine their conclusions.
Failures that stop a monolithic program may simply become evidence for another agent. One process can propose; another can critique; another can watch the environment; another can test whether assumptions remain true. The same multi-agent ideas that improve reliability in benign systems can be turned toward persistence and evasion by a malicious operator.
Why the two threats compound
An adaptive attacker does not have to search only for a new exploit. It can also search for a new interpretation.
If a model consumes documents, tool output, retrieved memory, prior logs, or visually hidden context, an adversary may be able to vary that surrounding evidence until the model interprets an unchanged command differently. A detector looking only for an explicit malicious instruction can miss the attack because there may be no such instruction to find.
This is the uncomfortable combination: a learning adversary can repeatedly probe the semantic boundary, while the defender may be validating only literal strings or yesterday's known signatures. The attack surface is therefore not just code. It includes context, provenance, authority, and meaning.
Where is the final gate?
If cognition can be probabilistic and adaptive, the consequential system needs a final non-bypassable control as close as possible to the external effect.
Physical world
The final boundary may be an actuator controller, valve, motor driver, brake controller, power switch, firing interlock, or another mechanism that releases physical energy or changes the environment. Hardware can be valuable because it can be made harder for an AI process to rewrite than ordinary software.
Cyber world
There may be no motor or valve. The equivalent boundary is the privileged state transition: credential use, an operating-system execution boundary, a network gateway, a database commit, a cloud control-plane action, or another operation that makes a digital change authoritative and persistent.
Cyberspace often has fewer natural bottlenecks. A physical operation usually requires location, energy, materials, and an actuator. A digital operation can sometimes be replicated at enormous scale once a privileged interface has been crossed. The safety controller and the attack may also live on the same programmable substrate.
A five-term glossary
Open weights move the safety perimeter
Provider-side safeguards matter when the provider controls the model endpoint, credentials, monitoring, rate limits, and tool interfaces. Open-weight deployment changes that assumption. A benign researcher may value local control for privacy, cost, sovereignty, or experimentation. A malicious operator can value the same property because it removes the provider from the control loop.
This does not make open models inherently malicious. It means that deployment governance cannot be treated as a universal safety layer once powerful capability is widely transferable. The operator may own the host, choose the system prompt, modify the weights, disable the monitor, supply the tools, and decide which actions are authorized.
Cyber is only one example
The same structural issue appears wherever AI lowers the cost of searching a large technical design space. Cybersecurity is the clearest example because actions can be automated and digital feedback is rapid. But the concern generalizes to high-consequence engineering, biological and chemical design, autonomous physical systems, financial infrastructure, and other domains in which a capable system can assist a user in generating, evaluating, and refining candidate actions.
The point is not that an AI model can conjure dangerous capabilities out of nothing. Real-world harm still requires access, materials, infrastructure, permissions, and successful execution. The concern is that AI can reduce some of the expertise, search, iteration, coordination, and planning costs that previously acted as friction.
Detection must assume adaptation
If the adversary can learn, detection cannot be designed only around static signatures. Defensive systems need multiple forms of evidence: behavior, provenance, identity, least privilege, rate and scope limits, stateful histories, anomaly detection, sandboxing, transactional authorization, independent monitors, and post-action audit.
More importantly, the defense should minimize what a successful semantic or model-level compromise can directly authorize. A suspicious planner should not possess the credential that turns a proposal into an irrevocable action. A component that interprets language should not, by itself, own the commit authority.
The revised AI-safety question
The useful conclusion is not that the truth lies halfway between “AI is extremely safe” and “AI is extremely dangerous.” The asymmetry is more important than the midpoint. Safety mechanisms learned from yesterday's attack, or validated under yesterday's interpretation of a command, may not remain valid against an adversary that can change both behavior and context.
AI safety is not a property of a model alone. It is a property of an entire state-changing pipeline.
That pipeline must preserve grounded meaning, authorization, independent enforcement, and control of the final commit boundary even when the planner, the operator, or the surrounding infrastructure becomes adversarial.
The decisive question is not merely “Can AI be made safe?” It is: Can our safety architecture remain valid when the attacker can learn, and when the meaning of the commands it guards can move?
References
- Chang, Edward Y., & Chang, Ethan Y. (2026). Coordinate-Free Hidden-State Diagnostics Measure In-Context Meaning Override. Zenodo. DOI: 10.5281/zenodo.23271844.
- Chang, Edward Y., & Chang, Ethan Y. (2026). Literal Integrity Is Not Semantic Integrity: Descriptive Records Re-anchor Unchanged Commands. Zenodo. DOI: 10.5281/zenodo.23272326.
- Google Threat Intelligence Group. (2025). GTIG AI Threat Tracker: Advances in Threat Actor Usage of AI Tools, November 5, 2025.
- Perez, F., & Ribeiro, I. (2022). Ignore Previous Prompt: Attack Techniques For Language Models. arXiv:2211.09527.
- Greshake, K., Abdelnabi, S., Mishra, S., Endres, C., Holz, T., & Fritz, M. (2023). Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security, 79–90. DOI: 10.1145/3605764.3623985.
- Wallace, E., Xiao, K., Leike, R., Weng, L., Heidecke, J., & Beutel, A. (2024). The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions. arXiv:2404.13208.