Context

LLM-generated artifacts (example sentences, summaries, labels) are written to persistent storage and reused for a long time, while the models and prompts that produce them keep improving. The prompt changes more often than the model, and can matter as much to output quality, sometimes more.

Examples

When the system generates example sentences for a word, it stamps each stored sentence with a created_by value naming the model and prompt version that produced it (for example, claude-opus / examples-v3). When the example-generation prompt is improved to v4, the stale sentences are exactly those still stamped v3, so a single query finds them and they are regenerated, without touching the rest of the store.

The stamp is not confined to example sentences. Simplified articles and their per-level summaries, audio lessons, and meaning-frequency labels all carry it, and they carry it by pointing at a shared generator row rather than each repeating a model name in a string column of its own.¹

That shared entity is what turns regeneration into a join. Asking which artifacts a given prompt version produced is then one lookup, answered across every kind of artifact at once. With the model name denormalized into each table it is four scans over four independently spelled strings, and each new kind of generated artifact adds a fifth.

Problem

When a prompt or model improves, how can exactly the stale artifacts be found and regenerated, without reprocessing the entire store?

Forces

• Selective regeneration needs to know how each artifact was made. Without a record of the model and prompt behind a stored artifact, applying an improved prompt means reprocessing everything. (pushes toward stamping provenance)
• A stamp is only useful if it stays truthful. Provenance has to be written on every artifact and kept in lockstep with the code: a prompt edited in place without bumping its version leaves the stamp naming the wrong one, silently lying, which is worse than no stamp at all. Keeping it honest is an ongoing discipline, not a one-off write. (pushes toward stamping precisely what drives regeneration, and versioning it rigorously)
• The prompt is the higher-churn axis: it changes more often than the model and can change the output more, so the stamp must capture the prompt version, not just the model.

Solution

Store the full provenance tuple alongside every LLM-generated artifact: model version, prompt version, generated output, timestamp. This turns regeneration into a targeted query, as broad as “re-run everything produced by prompt v2 with the improved v3” or as narrow as a single (model, prompt) pair, so when a prompt does well on one model and poorly on another, only that combination is redone.

Provenance as Zeeguu stores it. Each generated example sentence keeps its own timestamp and points at a shared `ai_generator` row naming the model and prompt version that produced it, rather than repeating them in every row. Finding what a given prompt produced is then a join on one table.

Consequences

• Selective regeneration becomes a query. Re-run everything a given prompt or model produced and leave the rest, and the same record doubles as a quality-audit trail.
• The guarantee lasts only as long as the discipline. Selective regeneration is trustworthy only while every write stamps accurately; once a stamp drifts from what actually produced the artifact, queries built on it quietly return the wrong rows.
• One identifier, shared with selection and validation. The stamped model identifier and the one used to select the model at call time should be the same central constant, kept in one place, and provenance pairs with LLM Content Validation Tracking: how an artifact was made, and whether it has been confirmed.

Known Uses

• Better attested in tooling than in production self-reports. The capability is productized: MLflow Prompt Registry and LangSmith version prompts and record which prompt/model produced each output, confirming the pattern’s core claim that the prompt deserves versioning as much as the model, but these are tools, not documented in-app deployments.
• Adjacent production pipelines. DoorDash stores versioned LLM-generated profiles and “treat[s] prompts as code”; Etsy stores Pydantic-validated LLM attribute extractions over 100M+ listings. Both store LLM artifacts at scale and version prompts, but neither publicly describes stamping each artifact with its prompt version to drive selective regeneration: the load-bearing mechanic here. We did not find a first-hand account of the full pattern, so Zeeguu is our instance.

Notes

• The key insight is that the prompt is at least as important to version as the model: a prompt change can completely alter output format, quality, or behaviour even with the same model.
• This is also critical for Rent, Then Build: when accumulating LLM-generated labels as training data for a classical replacement, provenance tracking lets one exclude data produced by a prompt version that was later found to be noisy or biased.
• Implicit provenance keeps model names and prompt versions as constants in code and, when the origin of a row is needed, correlates its created_at timestamp with git history to find which model/prompt was deployed then. This works only where a single model/prompt is active at a time. Once more than one is live at once (A/B tests, multi-model routing, or the second model of an Escalate to the LLM path), the timestamp no longer identifies a unique version and provenance must be stamped explicitly. Explicit tracking also makes analysis faster and keeps the data self-describing.
• Gateways log calls, not artifacts. An LLM gateway records each request and response, but selective regeneration needs the provenance to live on the stored artifact and reach the database, where a query can find the stale rows; request logs alone do not drive it.

War Story: The Nine Hundred Lessons

Provenance has to capture the dimension that actually varies. Zeeguu’s audio_lesson_meaning rows carry a created_by field naming the model ("Claude-Opus-Prompt1"), but the prompt templates were edited in place without bumping that identifier, so one value spanned two materially different prompt eras. When the ~900 lessons generated under the earlier, ambiguous prompt had to be found, no query could find them. The only route was a content regex on the generated script itself, asking whether it contained the ambiguous phrasing. A provenance field that does not move when the prompt moves is decorative, and selective regeneration degrades into forensics on the output. If prompts evolve by in-place edits, the field naming them must bump on every edit, through a versioned filename or a content hash.

Finding them was half the problem. The rows were also a cache, and they were referenced from lessons learners had already listened to, so they could be neither regenerated in bulk (expensive, and wasted on content never requested again) nor deleted (old daily lessons resolve them by id). Each affected row was given a deprecated_at timestamp instead, and the cache lookup was gated to skip deprecated rows: new lessons regenerate under the improved prompt while existing ones keep playing what the learner heard before, and regeneration is paid lazily, one row at a time, on next demand.

That fix contained a second bug. Each lesson’s audio file was named after the vocabulary item it taught, which does not change when the lesson is regenerated, so the first regeneration wrote its audio over the recording the deprecated row still needed. Files had to be keyed on the row’s own id. Where old artifacts must stay resolvable, every stored by-product has to be keyed by the artifact that produced it, not by the subject it is about.

Both halves trace to one root: the stored artifact did not carry enough of its own history.

---

← Reject and RepromptRent, Then Build →

💬 Open an issue about this pattern

1. The AIGenerator entity (model_name, prompt_version) and the tables referencing it in the zeeguu/api repository. ↩