Your AI Helpdesk Bot Has an Account Takeover Vector You Designed In

Early this year, tens of thousands of Instagram accounts got hijacked, and the interesting part is how boring the attack was. Nobody popped a server. Nobody found a memory corruption bug in Meta's auth service or a race condition in their token issuance. Attackers just opened a conversation with Meta's AI assistant and asked it, in plain language, to send a password reset code to an email address they controlled, and the model, doing exactly what it was trained to do, helpfully complied because the request read as a perfectly plausible support interaction. That's it. The whole exploit fits in a sentence you could say out loud to a human support agent, except a human support agent would have flinched at the part where the reset destination didn't match anything on the account. The model didn't flinch, because flinching isn't a token it knows how to emit. It has no internal model of session identity, no concept that the person typing is not necessarily the person who owns the account, and no interface primitive that would have forced it to reconcile those two things before executing something irreversible. What Meta shipped was a language model with a tool in its hand that could rewrite the recovery path of any account, and the only thing standing between an attacker and that tool was how convincing their prose was. That's not a vulnerability in the classical sense. It's a design decision, and a lot of teams are quietly making the same one right now.

There's a temptation to treat this as a prompt engineering problem, like if you just add enough system prompt text saying "never send reset codes to unverified emails, be very careful about social engineering, verify identity first," the model will hold the line. It won't. We've tested this pattern repeatedly with clients who wanted a support assistant wired into their account backend, and every single guardrail expressed purely as instructions to the model is a guardrail an attacker can talk their way around, because the same conversational flexibility that makes the assistant useful is the exact surface the attacker exploits. You cannot instruct your way out of a capability. If the model can call reset_password(account_id, destination_email) as a tool, then some sequence of tokens exists that will make it call that tool with an attacker's email, and no amount of "please be careful" in the system prompt closes that gap, it just raises the word count of the jailbreak. The model doesn't have a trust boundary. It has a probability distribution over next tokens, and social engineering is precisely the craft of shifting that distribution. So the fix cannot live inside the model. It has to live in the interface between the model and the consequential action, in a layer the model cannot reach past, argue with, or reason around. The model can suggest a password reset all day long. It just can't be the thing that finalizes one.

The design principle we land on with every client building this kind of assistant is simple to state and annoying to implement well: any action that mutates identity, billing, access, or account recovery has to route through a confirmation surface that lives outside the chat transcript and that the model has no ability to auto-approve. The model produces an intent, not an action. When a user asks the assistant to change their recovery email, the assistant doesn't call the API. It emits a structured proposal, something like {action: "change_recovery_email", from: "a@x.com", to: "b@y.com"}, and that proposal gets rendered as an explicit confirmation card in the UI with the actual before and after values shown in plain text, not summarized by the model, pulled directly from your backend so the model can't misrepresent them. The user then confirms through a control that requires their authenticated session, ideally re-attested with a fresh factor for high-consequence actions, a passkey tap or a TOTP, before anything executes. The critical detail is that the confirmation is bound to the session that owns the account, not to whoever happens to be typing in the chat window. An attacker who's socially engineered their way into a conversation still doesn't hold the victim's session, so the confirmation card either never reaches them or reaches the actual account owner who never asked for a recovery email change and immediately knows something is wrong. The model becomes a very good router of intent that has been stripped of the authority to finalize anything. That's the whole game.

The obvious objection is that confirmation friction is exactly what everyone spent a decade removing from support flows, and if every trivial request now spawns a modal with a passkey prompt, you've built something worse than the phone tree it replaced. That objection is correct if you apply confirmation uniformly, which is why you don't. Classify actions by blast radius. A user asking "what's my current plan" or "why was I charged twice" is a read, it touches nothing, let the assistant answer instantly with no confirmation at all. A user updating their display name or notification preferences is low-consequence and reversible, a single inline confirm click is plenty. The heavyweight surface, the fresh factor re-attestation, the out-of-band render, gets reserved for the small set of actions that can lose someone their account: recovery email changes, password resets, MFA device removal, email address changes, role escalation, API key generation. In practice that's maybe six or seven action types for a typical SaaS, and they're the ones users perform rarely, so the friction lands on the flows where friction is a feature and stays out of the flows where it's a tax. We built a document processing portal last year where the assistant could do almost everything conversationally, but reassigning document ownership popped a confirmation card that showed the exact document, the exact current owner, and the exact target, and required the requester to still be the current owner or an admin, verified server side. Nobody complained about the friction, because the people doing it understood it was consequential. Good confirmation design isn't about slowing everything down. It's about making the weight of a confirmation match the weight of the action, and putting a hard boundary the model can't cross around the handful of actions that actually matter.

If you've shipped an AI assistant that touches account management, go read your tool definitions today, the actual JSON schema of every function you've exposed to the model, and ask one question per tool: if an attacker fully controlled the conversation, what's the worst thing this tool lets them do. Any tool that can mutate identity or recovery or access with only conversational input as authorization is a live account takeover vector, full stop, and it doesn't matter how good your prompt hardening is. Then check whether your confirmation logic, if you have any, lives in code your model can influence or in code it can't. If the model generates both the action and the summary the user confirms against, an attacker can make the summary lie, so pull confirmation values straight from your backend of record and render them in your own UI, never let the model narrate what it's about to do to you. Log every high-consequence intent the model proposes, approved or not, because that log is how you catch the probing before the successful attempt, and rate limit consequential proposals per session because a legitimate user does not try to change their recovery email nine times in two minutes. This is squarely a design problem wearing a security costume, and it's the kind of thing we spend a lot of time on at steezr when a client wants an assistant that's genuinely useful and doesn't hand out their users' accounts to anyone with a convincing paragraph. The assistant can be smart. The authority to burn down an account just can't live inside something you can talk to.