What actually leaves your machine when you use an AI coding agent

It is a reasonable question and a bad starting point, because it collapses three completely separate things into one. What the provider's policy permits, what your organisation actually configured, and what your team does day to day are three different systems, and the third one is where nearly every real leak I have seen came from.

Here is how I would untangle them, with the sources worth reading rather than my paraphrase of them.

Layer 1: what the provider says it does

This is the layer everyone argues about and the one that is easiest to check, because it is written down.

The practical thing to know is that consumer and business tiers usually differ, and the difference is usually training. Anthropic publishes a plain-language explanation of how personal data is and is not used in model training, and the commercial terms cover the business side. GitHub documents the policies an organisation can set for Copilot, including suggestion matching and what admins can turn off.

Read the tier you are actually on, not the one on the marketing page. This is a ten-minute job that most teams never do, and it resolves the training question completely.

Layer 2: what you actually configured

Layer 1 tells you what is possible. Layer 2 is what someone on your team actually selected, probably a year ago, probably in a hurry.

Retention windows, whether admins can see prompts, whether telemetry is on, whether people are signed in with company accounts at all. None of this is exotic, it is a settings page, but it drifts, and nobody owns re-checking it.

If you want a structure for that review rather than inventing one, a few are worth knowing. The NIST AI Risk Management Framework is the general-purpose one, and it is free. The OWASP Top 10 for LLM Applications is the one I would actually hand a developer, because it is specific and short enough to be read. If you are in the UK or EU, the ICO guidance on AI and data protection is unusually readable for regulator output, and the Cloud Security Alliance AI work is worth a look if you are the person writing the internal policy.

Layer 3: what your team does when the policy is inconvenient

This is the one that matters, and the one nobody documents.

Policies describe intended behaviour. People route around them when the intended behaviour blocks work, and they do it quietly, because they are trying to ship something. Three patterns I have watched happen repeatedly:

Context gets pasted where it is convenient. Someone drops a stack trace into a chat window to get help. The stack trace has a customer identifier in it. That is not a model-training problem, it is a now-it-lives-in-three-systems problem, and no provider setting prevents it.

Screenshots go into tickets. A screenshot of a bug is also a screenshot of whatever else was on screen. Real customer names, other tabs, a token sitting in a terminal. Tickets get shared far more widely than the person taking the screenshot imagined.

Logins get shared. This is the big one. Someone needs a task done, the person who can do it does not have a seat, so credentials move, over Slack, verbally, or by leaving a session logged in. It usually breaks the provider's terms, and it destroys attribution: once two people use one identity, the audit trail stops telling you who did what.

That third pattern is worth attacking directly rather than with a policy reminder, because the underlying need is real. People are not sharing logins to be reckless. They are sharing them because the work needs to move and the tooling did not offer another way.

There are a few approaches. Buy more seats, which is the honest answer and often the right one. Restructure so fewer people need access, which works for some teams and not others. Or use something built for that specific handoff. Wagglet is the one I know best, where the task moves and each person runs it on their own authenticated agent, so a handoff is a disclosure of task context rather than a transfer of someone's login. Its security page is explicit that a handoff does not move an API key, subscription or token balance, which is the distinction that matters if you are the person who has to defend the setup later.

Whatever route you take, the useful reframe is that the credential and the work are separate things, and most tooling has historically forced you to move both.

The checklist I would actually run

Not exhaustive. Just the five that surface the most in the least time.

Which tier is each AI tool on? Consumer and business differ, usually on training and retention. Check the tier, not the brand.

When was the config last reviewed? If the answer is when we set it up, that is the finding.

Where does context get pasted? Ask people what they actually do when they need help fast. You will learn more from this than from any policy audit.

Is anyone sharing a login? Ask without consequences attached, or you will get a useless answer. Then fix the reason rather than the behaviour.

For anything running locally: what leaves the machine, can it update itself, and can someone send it instructions remotely? Those three cover most of the risk of local agent tooling.

The part worth keeping

Privacy with AI tooling is much less about cryptography than people expect, and much more about where data ends up because someone was in a hurry.

The provider policies are readable and mostly fine. The frameworks above will structure a review if you need one. But the leak, when it comes, will almost certainly be a pasted log, a screenshot, or a shared password, and none of those get fixed by picking a different model vendor.

Sources and further reading

Anthropic, how personal data is used in model training: https://privacy.anthropic.com/en/articles/10023548-how-do-you-use-personal-data-in-model-training

Anthropic commercial terms: https://www.anthropic.com/legal/commercial-terms

GitHub, managing Copilot policies for your organisation: https://docs.github.com/en/copilot/managing-copilot/managing-github-copilot-in-your-organization/managing-policies-for-copilot-in-your-organization

NIST AI Risk Management Framework: https://www.nist.gov/itl/ai-risk-management-framework

OWASP Top 10 for LLM Applications: https://owasp.org/www-project-top-10-for-large-language-model-applications/

ICO guidance on AI and data protection: https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/artificial-intelligence/

Cloud Security Alliance, artificial intelligence: https://cloudsecurityalliance.org/artificial-intelligence

Wagglet: https://wagglet.com

Wagglet security and privacy boundaries: https://wagglet.com/security

Disclosure: I work on Wagglet, one of the tools linked above. The other resources are independent and I have no relationship with them.