Reading 400 Million Git Pushes: GitHub OSINT at Full Population

TL;DR

I've rebuilt the entire 2025 Personal Public GitHub push stream (405M pushes, 32.8M repos, 2.41B file paths) into a single queryable BigQuery table and measured what actually leaks at full population rather than from a sample. The dataset is available to anyone with a valid Google Cloud account: https://console.cloud.google.com/bigquery?ws=!1m4!1m3!3m2!1sghscan-corpus-12520!2sghscan

Secrets that still work: after removing the scanner noise we were left with 9,965 credentials that still authenticated (a floor), 57% of which were database logins. In our focused cloud pass we found 449 live keys, 22 of which were AWS root keys and the average live cloud key had been public for over a year.

The blind spot: GitHub's secret scanning and push protection only apply to repos you own. The exposure that really matters is in the personal public repos of your developers, contractors, vendors and ex-employees. That exposure is completely invisible to the org by design. It's third party risk and your only lever is to revoke at the provider (not to rotate).

Scale of the hygiene gap: 639,806 repos have committed a .env path and corporate identities leak straight through the commit emails (tens of thousands of devs committing from their employer addresses into their personal side projects).

Cheap by design: the whole corpus is 196GB in BigQuery and the majority of the questions are literally cents to dollars per query. Pay per query OSINT at population scale.

Open-source intelligence usually means targeting one target across many sources. This is the inverse experiment: take the entire 2025 push stream to public personal repos on GitHub, bring it into one place, and ask what you can learn when you can query it all at once, then run the same queries against your own organization.

The "one place" is a single BigQuery table: 405,771,901 pushes, ~32.8M personal public repos, 2.41B file paths, 195.8GB. Each row is one push reconstructed from the public event feed (the GH Archive tells you a push occurred, it doesn't tell you what changed), plus one REST Compare API call to get the list of changed files:

event_id, actor, repository, created_at, commit_sha, author_email, author_name, ref_full, api_status, files (array), files_count, api_fetched_at

Small schema, serious instrument. Here's what it shows and what it shows about you.

The short version: a content pass across the paths this table maps has yielded approximately 9,965 credentials, 57% of which are database logins and each one verifiable for a few cents per query. Here's how you get there and how you can point it at your own organization.

Lesson 1: A commit is a name badge

Every git commit has an author email that you set at one point or another years ago and have since forgotten. At scale, those emails are a way of mapping code back to organisations.

SELECT REGEXP_EXTRACT(author_email, r'@(.+)$') AS domain, COUNT(*) AS pushes, COUNT(DISTINCT actor) AS devs FROM ghscan.commits_2025 WHERE author_email LIKE '%@%' AND NOT ENDS_WITH(author_email, 'noreply.github.com') AND REGEXP_EXTRACT(author_email, r'@(.+)$') NOT IN ('gmail.com','hotmail.com','qq.com','163.com','outlook.com', 'yahoo.com','icloud.com','126.com','foxmail.com') GROUP BY domain ORDER BY pushes DESC LIMIT 10;

Lesson 2: names are leads, not proof

The next instinct is to grep repository names for your own company. Seductive, and a trap:

SELECT SPLIT(repository, '/')[OFFSET(0)] AS owner, COUNT(DISTINCT repository) AS repos, COUNT() AS pushes FROM ghscan.commits_2025 WHERE REGEXP_CONTAINS(LOWER(repository), r'^[^/](nvidia|oracle|siemens)[^/]*/') GROUP BY owner ORDER BY pushes DESC LIMIT 8;

oracle-devrel and NVIDIA-NeMo are the companies' own public orgs. The others are completely unrelated projects or personal accounts whose handle just contains the string. The presence of the word "oracle" in the name tells you absolutely nothing about whether there's actually been a leak. That's the key takeaway from the dataset: name matching will generate you a list of leads, it will not generate you a finding. Of the 200 large companies, there are 14,675 namespace-bearing repos under non-org accounts. At that scale you're simply unable to tell the difference between a real leak and legitimate open source, forks and coursework. The query is only honest if its result is "to triage" rather than "exposed". 11GB at about $0.06.

Lesson 3: most of the signal is machines

Before trusting any ranking, look at who is loudest:

SELECT actor, COUNT(*) AS pushes, COUNT(DISTINCT repository) AS repos FROM ghscan.commits_2025 GROUP BY actor ORDER BY pushes DESC LIMIT 5;

cb22 pushed 466,579 times to six repos. Bots, mirrors, and deploy automation are at the top of the chart. If you just go to 'most active developers' without filtering out the bots then you're measuring CI not people. The most obvious indicator is a massive push count against a very small number of repos.

The expensive column: from paths to risk

The queries above were trivial, dealing with small string columns and costing nothing at all. The files array contains all the changed paths, and it's in that array that identity becomes a risk. It's the same table which tells you who it is that also tells you what they've left lying around:

SELECT repository, actor, f AS path FROM ghscan.commits_2025, UNNEST(files) AS f WHERE f LIKE '%.env' OR f LIKE '%id_rsa' OR f LIKE '%.pem' OR f LIKE '%.tfstate' OR f LIKE '%.npmrc' LIMIT 100;

Kept at the aggregate level, it simply sizes up the exposure without naming any individuals: 639,806 repositories have committed a path ending in .env, with tens of thousands more having .pem (63,972), .npmrc (64,453) or Terraform state.

That is the question. A separate content pass along exactly those paths under co-disclosure identified approximately 9,965 still-live credentials (57% of which were database logins). Take that as a floor not a census. Deleted repos, force pushes and the 300 file cap per push all result in undercounting and a secret in file 301 of a push is effectively invisible. The paths are queryable in this table, the content and liveness are live (with the requisite legal care) outside of this table.

The same column looks at the landscape. By extension we have JS (212.5M) and TS (211.2M) very much in it together, Python (135.1M) a long way behind. 74,351 repos now have a CLAUDE.md, well ahead of the vendor neutral AGENTS.md (26,829). A CLAUDE.md is a new surface in the supply chain as a poisoned one in a dependency is prompt injection you've committed yourself to.

The rule that emerges is that COUNT(*) is free, string columns are a penny a column and anything that involves files (at about $0.73 per scan) needs a — dry_run prior to the session. A single session over the array column can then quietly cost you more than a month of storage.

Now point it at yourself

Everything above becomes defensive the moment you swap in your own identifiers:

SELECT actor, ANY_VALUE(author_name) AS name, COUNT(DISTINCT repository) AS repos, COUNT(*) AS pushes FROM ghscan.commits_2025 WHERE ( ENDS_WITH(author_email, '@acme.com') OR REGEXP_CONTAINS(LOWER(repository), r'^[^/]acme[^/]/') ) AND NOT REGEXP_CONTAINS(LOWER(repository), r'^(acme-official|acme-labs)/') GROUP BY actor ORDER BY pushes DESC;

This is the one control GitHub's own tooling cannot give you. Secret scanning and push protection only cover repositories inside orgs and plans you administer; your corporate identity and code leaking through developers' personal public repos is precisely the blind spot, and 42 GB ($0.24) is how you see it.

What to do with the hit list:

· Commit-email hygiene. Default developers to the GitHub noreply address and clean history with .mailmap. A name badge you did not mean to wear is the cheapest leak to stop.

· Treat name-match output as a triage queue, never a breach list (Lesson 2). Hand it to whoever owns the attack surface, not to a panic channel.

· The live-credential floor is your rotation case. Pre-commit secret scanning, and revoke at the provider what you cannot un-leak, because a secret sitting in someone else's repo can be revoked but never rotated in place.

And make it continuous. GH Archive is a daily feed, so this is a scheduled query against yesterday's partition with a diff on new actors; staying on the cheap string columns keeps a daily job at cents. Branch names (ref_full) are another near-free column worth the same sweep, since they leak ticket IDs and codenames.

The line between measurement and surveillance is a column list. Everything here rolls up to domains, owners, and counts, and it should: you are one GROUP BY from profiling real people. The 195.8 GB sitting in BigQuery will tell you more about your own exposure, for cents a question, than any vendor dashboard, and it will not flatter you. Run the self-scan on your own domain before someone else runs it on yours.

Data & legal note

No passwords in the dataset. The BigQuery table holds only public metadata and file paths (which files a push changed), never file contents and never credential values.

Contents are not stored. When I needed file content, I fetched it directly from GitHub at analysis time and did not retain it in the dataset. Public sources only. All data comes from publicly available sources: the public GitHub Archive event feed and GitHub's public REST API. No private repositories, no authenticated access to anyone's account, no non-public data.