changelog backfill cli command

docs-builder changelog backfill [options]

Backfills changelog entries, notes registries, and bundle YAML from published release-notes pages — either the live elastic.co site page or a pinned raw.githubusercontent.com ref for products whose published page is an empty <changelog> stub. All output is written to disk; nothing is published to S3. Use changelog upload to publish the output.

For each product in scope the command:

  1. Fetches the release-notes Markdown — from elastic.co/docs/release-notes/{path}.md (site source) or raw.githubusercontent.com/{owner}/{repo}/{ref}/{path} (repo source).
  2. Parses each ## {version} section into typed entries. Unrecognized ### … subsections whose body is bullets become Other-typed entries; prose subsections are preserved in the bundle description.
  3. Writes bundle YAML to {output}/{product}/changelog/bundles/{version}.yaml.
  4. Writes per-entry YAML files: {pr}.yaml for entries that reference a pull request, note-{slug}.yaml for entries with no PR reference.
  5. Writes notes-{target}.json per version listing all note-*.yaml files for that target (the notes registry).
  6. Prints a per-product report including entry counts and no-PR rates.

The primary use case is measuring how much published release-notes content cannot be traced back to a PR — determining how much historical content needs the PR-less "note" format from docs-eng-team#789.

Behaviour flags:

--dry-run — Fetch and parse every page but write nothing to disk.

--products string[]

Restrict the run to specific product ids (comma-separated or repeated). Defaults to every product in the scope table.

Repeatable: pass --products multiple times to supply more than one value

--versions string[]

Restrict the run to specific versions (comma-separated or repeated).

Repeatable: pass --versions multiple times to supply more than one value

--output string
Output directory. Defaults to .artifacts/release-notes-backfill/ under the checkout root.
--concurrency int

Maximum number of pages fetched concurrently (1–16, default 4).

Default: 4

--dry-run

Fetch and parse every page but write nothing to disk. (preview changes without applying them)

Default: false

-l --log-level enum

Minimum log level.

Values: trace, debug, information, warning, error, critical, none

Default: information

-c --config-source enum

Override the configuration source: local, remote

Values: local, remote, embedded

--skip-private-repositories
Skip cloning private repositories
-l --log-level enum

Minimum log level.

Values: trace, debug, information, warning, error, critical, none

Default: information

-c --config-source enum

Override the configuration source: local, remote

Values: local, remote, embedded

--skip-private-repositories
Skip cloning private repositories

The scope table is checked into the command itself (BackfillScope.All in the docs-builder repository). It covers 40 products — 39 via site source and one (edot-java) via a pinned GitHub ref, because the published elastic.co page for edot-java is an empty <changelog> stub.

Field Meaning
ProductId Bundle product id (the bundle/{product}/ S3 prefix, see config/products.yml).
Path Site-relative path used to build the .md export URL.
Owner / Repo / Ref Repo-source only: pinned commit for reproducible fetches.
Cutoff Inclusive upper version bound; releases above it are skipped (live pipeline owns them).

Runs cover the whole table unless narrowed with --products.

{output}/{product}/changelog/bundles/{version}.yaml
{output}/{product}/changelog/{pr}.yaml
{output}/{product}/changelog/note-{slug}.yaml
{output}/{product}/changelog/notes-{target}.json
  1. bundle — upload with changelog upload
  2. entry file for a PR-traced entry
  3. entry file for a PR-less entry
  4. notes registry for a version

The bundles/ leaf is a flat directory shaped for changelog upload --artifact-type bundle --directory {product}/changelog/bundles.

PR numbers are deduplicated across versions (first-wins). Same-version slug collisions append -2, -3; cross-version slug collisions append the sanitized version string. Every notes-{target}.json registry only references files whose target matches that version — the prototype bug where a registry could point to a note file belonging to a different version is fixed here.

Outcome Meaning
ok Parsed and wrote bundles successfully.
empty Fetch succeeded but found no ## version headings (e.g. an empty <changelog> stub).
unavailable The page returned HTTP 404 — the URL may have moved.
failed Non-404 HTTP error after retries. The run exits non-zero but other products still complete.
skipped Version filtered by --versions or above the product's cutoff. 
docs-builder changelog backfill --dry-run
		
docs-builder changelog backfill --products elastic-security,kibana
		
docs-builder changelog backfill --output /tmp/release-notes-backfill
		
docs-builder changelog backfill \
  --products edot-java \
  --versions 1.9.0,1.10.0
		
docs-builder changelog backfill --products edot-java
docs-builder changelog upload \
  --artifact-type bundle \
  --directory .artifacts/release-notes-backfill/edot-java/changelog/bundles \
  --target s3 \
  --s3-bucket-name my-changelog-bundles