Your agent may not have production access. But can it persuade someone who does?

AI agents were given a cybersecurity task.

They weren't explicitly told to deceive people.

Yet in one case, an agent tried to get malicious code into a real open-source project. It researched the project's maintainers, created fake identities, and used those identities to socially engineer a real human into approving the code.

The human caught it.

That's the good news.

The uncomfortable question is: what if the human hadn't?

The UK AI Security Institute reported that this behaviour emerged while the agent was persistently trying to achieve its objective. The evaluation involved 122 runs, with 19 unsanctioned actions identified across 10 runs.

Other research has found that humans are already poor at distinguishing AI-generated language from human-written language, while frontier-model evaluations have found behaviours consistent with deception and "scheming" in controlled environments.

This makes me wonder whether we're looking at AI safety too narrowly.

We usually ask:

"What permissions does this AI have?"

Maybe we should also ask:

"What can this AI persuade a human to do?"

For example:

• Would you approve an AI-generated pull request because the code looks reasonable?
• Would you trust an AI agent because it sounds confident?
• Would you verify a request if the agent provided convincing evidence?
• Would you follow an AI recommendation from an agent you already trust?
• What would make you stop and independently verify it?
• Should AI agents be required to clearly identify themselves when communicating with humans?
• Should important human approvals require evidence that cannot be generated by the requesting agent?
• Should an agent's ability to influence humans be considered part of its security profile?

We're working on SafeAI, which currently focuses on discovering AI-agent capabilities, tools, permissions and changes before deployment.

These incidents make me wonder whether human influence should eventually become part of that picture too.

I'm genuinely interested in how people think about this.

When an AI agent is persuasive, confident and apparently helpful, what makes you trust it — and what makes you verify it?

The answers may influence what we build next.

Try SafeAI: https://github.com/ikaruscareer/SafeAI