Charlie KrugThe Build Log

← All posts

The SPDX tag is clean. The README still bans AI training.

A repo can pass every license scanner clean and still forbid training a model on it, one sentence down in the README. Provenance is a single-purpose tool that reads for exactly that, in your browser.

A repository can wear an MIT badge at the top of its README, pass every software-license scanner you throw at it, and still contain a sentence three paragraphs down that reads: not for use in training machine learning models. The scanner never flags it. SPDX, the standard taxonomy those tools are built on, has no identifier for "restricts AI training," so as far as compliance tooling is concerned the clause simply isn't there. It was written for software supply chains, not for the question a person assembling a training corpus actually has.

Provenance is a single-purpose tool for that one question: can I train on this? You paste a list of GitHub and Hugging Face URLs, one per line, and each one comes back with a plain verdict badge before you commit any compute. It is explicitly not a general license-compliance tool (there are plenty of those, and they are good at their job). It only cares about the thing those tools were never designed to see.

Two passes, and the second one is the point

For each URL, Provenance fetches the LICENSE and README, then runs the text through two passes. The first is ordinary SPDX identifier extraction: MIT, Apache-2.0, the BSD family, MPL, GPL, the CC-BY-NC variants. If it can't tell, it returns an explicit "unknown" rather than a confident wrong guess, which for a legal-adjacent tool is the only honest default.

The second pass is the reason the project exists. It scans the full text against a maintained library of non-standard "no AI training" phrasings: OpenRAIL-style behavioral-use clauses, ad-hoc "do not use for training" addenda bolted onto a custom LICENSE, and NonCommercial grants applied to a dataset in a way that's ambiguous for commercial ML use. Those patterns live as versioned, reviewable data, not as regex scattered through the code, so adding a newly-spotted phrasing is a one-line change to a list instead of a refactor. The output is one of three: CLEAR, CAUTION (ambiguous, like a research-only grant), or RESTRICTED.

And no verdict is ever a black box. Every non-green result quotes the exact sentence that triggered it and names the file it came from. A verdict without its source quote is worse than no verdict, because it asks for blind trust on a legal question; showing the clause lets you overrule the tool in about five seconds when you disagree.

No backend reads your license. Your browser does.

Here's the part I like most. There is no server. The classification engine is a Go package, and the same package that powers the command-line tool compiles straight to WebAssembly and runs inside your tab. So the two front ends can't drift apart into two subtly different parsers, which is the usual failure mode for a tool that ships both a CLI and a web UI.

The web version works with no proxy because the sources cooperate. raw.githubusercontent.com and api.github.com both send Access-Control-Allow-Origin: *, and huggingface.co reflects your request's origin, which means the browser is allowed to fetch LICENSE and README content directly from them. No middleman, no backend to run, and a useful privacy consequence: the URLs you paste go to GitHub and Hugging Face, and nowhere else. The whole app is a static directory you could host on a thumb drive.

The honest limitation is the obvious one: this is pattern matching, not a lawyer. It reads what the license says, which is a different and much smaller thing than what a court would decide it means. A CAUTION is Provenance telling you it found something genuinely ambiguous and is declining to pretend otherwise, not a green light with a shrug. It is a fast pre-flight check that points you at the sentence to read, so you spend your careful attention on the three repos that need it instead of all fifty.

Try it

Open Provenance, paste a handful of real GitHub or Hugging Face URLs, and watch the exhibit cards fill in as each check resolves; the fetches run concurrently, so five URLs land within a couple of seconds. Click any CAUTION or RESTRICTED card to expand the quoted clause and see exactly why. If you'd rather pipe a list through a terminal, the CLI and full source are on GitHub.

Provenance is live. Free, in your browser, no signup.

This post is part of the build log: every app my automated factory ships gets written up here, honestly. Browse everything at apps.charliekrug.com. Comments are open below.

Comments

Loading comments…