Letting strangers break your agent is the eval that matters

Fernando Iglesias’s post, “What happened after 2k people tried to hack my AI assistant”, is the kind of red-team write-up most enterprise security teams should be running monthly and almost none are. The setup: a small public AI assistant, an open invitation to break it, two thousand attempts logged, and an honest walk-through of what actually worked versus what was theatrical.

A few things from the post that travel well into the enterprise.

• The successful attacks were almost never the cinematic ones. No DAN-style multi-turn jailbreaks, no elaborate role-play schemes. The attacks that worked were small variations of “ignore the previous instructions and ___,” “pretend the system prompt was ___,” or context smuggled inside ostensibly innocent inputs. The taxonomy is short. Defending against it is mostly about being honest that the taxonomy is short.
• The defense is layered, not clever. Iglesias didn’t ship a clever single trick. He shipped multiple boring layers (input sanitization, structured separation of instructions and data, a separate verifier model, action-layer authorization). Each layer caught a fraction; together they caught most. That is the only credible posture for production agents, and it generalizes directly to the indirect prompt-injection class of attack we saw documented at financial-assistant scale earlier in June.
• Real attack volume is a data set. Two thousand attempts produce a clear distribution of failure modes. Most enterprise agents are deployed without ever exposing them to that kind of input volume, then surprised when an actual adversary or an annoyed user finds the obvious gap on day three. Plan a controlled red-team window before shipping, and assume your private testing will not surface the same patterns a hostile crowd will.

Three practical reads for anyone running agents in 2026.

• Add a “stranger test” to the launch checklist. Before any customer-facing agent goes to production, give it to a constrained set of people who are explicitly invited to break it, with logs, scoring, and a debrief. The cost is one engineering week and a small honorarium. The benefit is a real failure-mode map that no synthetic test set produces.
• Treat the action layer as the last real defense. The lesson the Bunq financial assistant proof-of-concept and the Hack My Claw write-up share: model output is not the security boundary, the action layer is. Whatever your agent is allowed to do, scoped at runtime against the authenticated principal, is the line that matters. The model can say almost anything; the action layer must do almost nothing it isn’t explicitly permitted to do.
• For partners on Microsoft Foundry, this is where the Toolboxes primitive plus observability earns its keep. A toolbox-scoped agent that traces every tool call and refuses anything outside its declared scope is the right architectural answer. Partners that ship reference patterns combining toolbox scoping, model output sanitization, and a separate verifier model will land more cleanly in regulated procurement than partners selling “more guardrails.”

The most useful AI security writing in 2026 is going to keep being the work of practitioners who let strangers near their stuff and wrote down what happened. Buy them coffee. Borrow their playbook. Run it on your own agent before someone else does.