Katie Paxton-Fear's Experiment

Katie Paxton-Fear, who works in cybersecurity for Semgrep, spent less than an hour and under $100 to manipulate the behavior of an open-weight AI model. With just ten examples of "poisoned" material, she managed to convince the model to produce code susceptible to executing remote commands, a flaw that allows an attacker to run their code on the victim's machine. "I created a real backdoor," was her comment when sharing the result.

The demonstration highlights a problem that Semgrep outlined in a recent publication: even when the weights of a model are public, the ability to predict its behavior is almost nonexistent. This is a fundamental difference compared to traditional software: a regular program, even in binary form, can undergo reverse engineering to yield a complete description of what it does; with models, nothing like that exists, and mechanistic interpretability remains an open research problem. An open-weight model retains visible parameters, but not the training data or the lineage that produced it: it remains a black box.

Here’s the thing: You can reverse-engineer a suspicious binary. You cannot reverse-engineer a model. I co-wrote this at work on this topic last week with @0xine and @spacerog. Link to the original tweet — Katie Paxton-Fear (@InsiderPhD)

Semgrep's Findings

Semgrep recalls the research "Small Samples" by Anthropic, conducted in collaboration with the UK AI Security Institute and the Alan Turing Institute, which suggests that a few hundred "poisoned" documents are enough to embed hidden behavior in a model, and the number of samples required does not increase with the size of the model. Large datasets are not immune to manipulation: the origin of the data used for training the model matters. A backdoor can remain dormant and invisible until a trigger phrase activates it, and currently, there is no reliable way to detect it.

There's also a slippery observation regarding the concept of a "compromised model." Models are already widely oriented to refuse offensive requests or those related to cybersecurity, or to decline unhealthy relational requests from users, behavior commonly referred to as alignment. "The poisoning of a model for one person is someone else's alignment problem," Semgrep writes, extending the reasoning to geography: concern should be raised for any model whose lineage cannot be traced, not just those built in specific countries.

Security Barriers Preventing Defenders

That the warning is not academic was demonstrated by HuggingFace itself, the largest platform for distributing AI models. The company disclosed that it was breached last week by a system of autonomous AI agents that attacked its production infrastructure. The cumbersome detail emerged from the defensive phase: to conduct forensic analysis, the company turned to Z.ai's GLM 5.2, a Chinese open-weight model, after western cutting-edge models rejected requests because they contained real attack commands, exploit payloads, and command-and-control artifacts, with security barriers activating because those models could not differentiate an attacker from someone legitimately responding to an incident. "The attacker was not bound by any usage policy, while our forensic work was being blocked by the security barriers of the hosted models we had tried first," the company explained. The practical lesson drawn: have a capable model ready to run on your own infrastructure before an incident to avoid being cut off from its own protections and to prevent the attacker's data from escaping the perimeter.

Ransomware Targets Models

On the economic front, ML infrastructure has already become an explicit target. Cloud security company Sysdig documented JadePuffer, an autonomous AI agent capable of autonomously traversing all phases of a ransomware attack, equipped with tailor-made malware called EncForge, designed to encrypt machine learning assets: model checkpoints, vector databases, training datasets, and embedding indices. The binary, written in Go, targets around 180 file extensions covering much of today's AI stack; among the examples in its command-line help are LoRA adapters and GGML files, suggesting its targeted nature according to Sysdig.

The account of the intrusion illustrates the speed of this new way of operating. The attacker returned to a vulnerable Langflow instance related to CVE-2025-3248, found an exposed Docker socket that granted root-level control, and when the first attempt to download the payload failed, they developed and deployed six Python scripts in about five minutes, leading to a successful delivery. EncForge only encrypts selected portions of each file using a hybrid AES-256 scheme in counter mode with a key protected by RSA-2048, appending the .locked extension. Sysdig estimates that encrypting weights, datasets, and vector indices could cost an organization between $75,000 and $500,000 per model, depending on size and purpose, along with weeks or months of training lost.

Challenges of Inspecting What You're Building

The common thread uniting these three episodes is the difficulty of inspecting what you’re building on. Semgrep argues that benchmarks and marketing claims are insufficient because benchmarks can be circumvented and a model can be fine-tuned specifically to pass them. What is actually necessary is verifiable provenance, reproducibility, and independent evaluation, akin to the trusted third-party ecosystem expected for software, consisting of researchers, auditors, and CVE programs. The question with which Paxton-Fear closed her demonstration remains the most honest summary of the moment: can we trust open-weight models, fine-tuned online and sold as a remedy for token spending? "We probably need something better than benchmarks and a 'don’t write insecure code' approach."