essc — sourcing CLI

essc is an AOT-compiled .NET CLI for indexing elastic.co content into Elasticsearch. It lives in src/tooling/essc/ and shares the search document contract (Elastic.Documentation.Search.Contract) with the docs indexing pipeline, keeping every content source schema-consistent. Commands are namespaced by content source so new sources can be added without breaking the existing interface.

Current sources:

Pull the distroless container image:

docker pull ghcr.io/elastic/website-search-essc:latest
		

Available tags: latest, edge (latest main), and per-version tags.

Or run from source:

dotnet run --project src/tooling/essc -- --help
		

essc resolves credentials in this order of precedence:

1. CLI flags (--endpoint, --api-key)
2. Environment variables (see table below)
3. Dotnet user-secrets store docs-builder (local development) — the same store the rest of this repository uses, so aspire and essc share Parameters:ElasticsearchUrl and Parameters:ElasticsearchApiKey

The dotnet configuration section keys (Parameters:ElasticsearchUrl, ContentStack:ApiKey, etc.) are also accepted. For CI, use the flat environment variable names above.

For local development, populate the secrets store directly:

dotnet user-secrets set --id docs-builder Parameters:ElasticsearchUrl <url>
dotnet user-secrets set --id docs-builder Parameters:ElasticsearchApiKey <key>
dotnet user-secrets set --id docs-builder ContentStack:ApiKey <key>
dotnet user-secrets set --id docs-builder ContentStack:DeliveryToken <token>
		

For cross-cluster commands (indices copy, indices sync-remote, indices unify-incremental-sync), the destination cluster seeds from the same Parameters:ElasticsearchUrl / Parameters:ElasticsearchApiKey secret as the source — so configuring just one cluster in the secrets store is enough; pass --to-url/--to-api-key (or --from-url/--from-api-key) on the command to point only one side at a different cluster, or set DESTINATION_ELASTIC_URL/DESTINATION_ELASTIC_APIKEY to override the destination for every run.

If you previously used essc from the website-search-data repository, copy the Parameters:Elasticsearch* and ContentStack:* values from your old elastic-website-ai-search store into the docs-builder store (~/.microsoft/usersecrets/, on Windows %APPDATA%\Microsoft\UserSecrets\).

Fetches all published content from Contentstack and indexes it into Elasticsearch. Uses 5 parallel lanes. Cursors are saved after every page so interrupted runs resume automatically.

essc contentstack sync
		

--max-ai-docs uses DataAnnotations: it must be at least 1 when passed; 0 is invalid. For a large or unbounded batch without re-syncing, use contentstack ai-enrich instead.

The sync runs in two phases:

1. Fetch + index — pages through Contentstack in parallel, maps each item to a SiteDocument, and bulk-indexes into both the lexical and semantic indices.
2. Finalize — flushes the remaining buffer, reindexes the lexical index into the semantic index, publishes synonyms and query rules, then runs a bounded generative AI enrichment pass on the semantic write index (unless --no-ai).

Runs generative AI enrichment on existing site-* semantic indices without calling Contentstack. Same Elasticsearch overrides as contentstack sync.

essc contentstack ai-enrich
		

Lists all Contentstack content types and their document counts.

essc contentstack types
		

Dumps raw Contentstack entries for a given content type, useful for inspecting the source data structure.

essc contentstack samples <content-type>
		

Discovers labs URLs from sitemaps, crawls HTML, and bulk-ingests into labs-* indices. See essc labs sync --help for crawl flags (--dry-run, --force, --no-ai, --max-ai-docs, --max-ai-time, etc.).

Runs generative AI enrichment on existing labs-* semantic indices without re-crawling. Flags match contentstack ai-enrich (--max-ai-time, --max-ai-docs, --endpoint, --api-key).

essc labs ai-enrich
		

Copies one or more indices verbatim between clusters using the Elasticsearch server-side remote reindex API. Unlike indices sync-remote / indices unify-incremental-sync, this command is not docs-search specific — it copies no synonym/query-rule resources and runs no lexical/semantic inference phases. Useful for pulling standalone indices (e.g. AI-enrichment -ai-cache lookup indices) from production down to a local cluster.

essc indices copy docs-assembler.semantic-prod-ai-cache labs-public.semantic-prod-ai-cache \
  --from-url https://<prod-es> --from-api-key <key> \
  --to-url http://localhost:9200
		

• Takes one or more source index/alias names as positional arguments.
• --rename-from / --rename-to — regex rename applied to each source name to derive its destination name (back-references supported), e.g. --rename-from 'semantic-prod-' --rename-to 'semantic-local-'.
• --force — delete and recreate an existing destination index from the source's mapping and settings before reindexing. Without it, an existing destination index is reindexed into as-is.
• --strip-semantic-fields — strip _inference_fields metadata when the source has semantic_text mappings (see indices sync-remote).
• Shares --from-url / --from-api-key / --to-url / --to-api-key / --slices / --rps / --aliases with indices sync-remote and indices unify-incremental-sync. --slices has no effect on indices copy — every reindex it does is a remote reindex, and Elasticsearch rejects slices > 1 for reindex from a remote source.

When CI=true or stdout is not a TTY, essc switches from interactive progress widgets to plain log lines:

[Lane 1]    100%   blog — 1,240 fetched, 1,198 indexed
[Lane 2]     14%   customer_tile — 42 fetched, 40 indexed (2 skipped)
…      14%   secondary reindex — 1,240/8,500
✓     100%   secondary reindex — 8,500/8,500
		

essc ships as a distroless container only, published to ghcr.io/elastic/website-search-essc using the .NET SDK's native container support as part of the docs-builder release pipeline (./build.sh publishcontainers). Base image: mcr.microsoft.com/dotnet/nightly/runtime-deps:10.0-noble-chiseled. Tagged edge;latest;<version> on release, edge on every push to main.

AOT compilation is validated on every CI run by building the container image and running essc --help against it.