Charlie KrugThe Build Log

← All posts

A decorator reads your loop's source and slices off what's done

Waypoint makes a long Python loop resumable with one decorator. It parses the function's source, rewrites the iterated sequence to skip finished items, and refuses to fake it on a generator it can't rewind.

A flaky API times out on item 8,000 of 10,000, and your script dies. Restarting means redoing the 8,000 records that already went through: the hours, and if the calls were metered, the money. So you do the boring thing. You track an index by hand, write it to a file after each loop, load it back at the top, and wrap the body in try/except so the write actually happens. It's easy to get subtly wrong (off-by-one on resume, a half-written state file, forgetting to clear it on success) and it buries the one thing the script was supposed to do.

Waypoint is that boilerplate collapsed into a single decorator. Put @checkpoint on a function that loops over a collection, kill the process whenever, rerun the exact same script, and it picks up at the next unfinished item. No job queue, no daemon, no config. The entire state store is one small JSON file.

from waypoint import checkpoint

@checkpoint
def process_all(items):
    for item in items:
        do_something_slow(item)

It edits the loop, not the loop body

The obvious way to build this would be to make you rewrite for item in items as for item in waypoint.resume(items). Waypoint doesn't. Instead it reads the decorated function's own source once, parses it into an AST, finds the top-level for loop, and rewrites just the iterated expression: it wraps items so the loop iterates over items[resume_index:] instead of the whole thing. Your loop body is never touched. The mechanism stays entirely out of the way of the logic you actually wrote.

That trick only works because slicing only works. A list, a tuple, a range (all sliceable, all reproducible on the next run) can be resumed. A plain generator cannot: it has no index, and you can't rewind it without re-running whatever produced it from scratch, which is exactly the work you were trying to avoid. Waypoint refuses to guess here. Decorate a loop over a bare generator and it raises NotResumableError with an actionable message instead of silently resuming into the wrong data. If you know your generator is safe to materialize, you say so out loud by wrapping it: process_all(seq(fetch_records())).

The checkpoint key is the function's qualified name plus a hash of its call arguments. Run the same function on a different dataset and it never resumes into the old one; run it again on the same input and it finds the interrupted checkpoint waiting.

The honest part: it prevents redone work, not double side effects

The index only advances after an iteration completes. If the process is killed halfway through processing one item, that item gets retried on the next run rather than skipped. So Waypoint's guarantee is narrow and worth stating plainly: it never redoes completed iterations, but it does not promise your side effects run exactly once. If the item at the interruption point wrote a row or charged a card before dying, make that body idempotent, because it will run again.

Two more things I'd rather you hear from me than discover. Every checkpoint write includes an fsync, trading per-item overhead for surviving a hard power loss mid-write. That's free when your loop body is doing real work like API calls, and it dominates if your body is sub-millisecond pure Python (batch those into chunks first). And a continue past the tracked item can't advance a single linear index without lying about later items, so everything after the first continue in a run is retried next time. Nothing is ever dropped; a continue-heavy loop just pays for that safety in redone work.

Try it

Clone the repo, pip install -e ., and run python examples/slow_loop.py. Hit Ctrl-C partway through, then run the exact same command again and watch it skip straight to where it stopped. python -m waypoint status lists what's checkpointed; a clean finish deletes the file so the next run starts fresh on its own.

Waypoint is live. Free, in your browser, no signup.

This post is part of the build log: every app my automated factory ships gets written up here, honestly. Browse everything at apps.charliekrug.com. Comments are open below.

Comments

Loading comments…