essc — sourcing CLI

essc is an AOT-compiled .NET CLI for indexing elastic.co content into Elasticsearch. It lives in src/tooling/essc/ and shares the search document contract (Elastic.Documentation.Search.Contract) with the docs indexing pipeline, keeping every content source schema-consistent. Commands are namespaced by content source so new sources can be added without breaking the existing interface.

Current sources:

Search Labs (/search-labs/* — blog, tutorials, notebooks, integrations) is sourced from Contentstack rather than crawled: it moved off the labs HTML crawler and is now indexed by contentstack sync alongside the rest of the marketing site. Security Labs and Observability Labs are still crawled by labs sync.

Pull the distroless container image:

docker pull ghcr.io/elastic/website-search-essc:latest
		

Available tags: latest, edge (latest main), and per-version tags.

Or run from source:

dotnet run --project src/tooling/essc -- --help
		

essc resolves credentials in this order of precedence:

1. CLI flags (--endpoint, --api-key)
2. Environment variables (see table below)
3. Dotnet user-secrets store docs-builder (local development) — the same store the rest of this repository uses, so aspire and essc share Parameters:ElasticsearchUrl and Parameters:ElasticsearchApiKey

The dotnet configuration section keys (Parameters:ElasticsearchUrl, ContentStack:ApiKey, etc.) are also accepted. For CI, use the flat environment variable names above.

For local development, populate the secrets store directly:

dotnet user-secrets set --id docs-builder Parameters:ElasticsearchUrl <url>
dotnet user-secrets set --id docs-builder Parameters:ElasticsearchApiKey <key>
dotnet user-secrets set --id docs-builder ContentStack:ApiKey <key>
dotnet user-secrets set --id docs-builder ContentStack:DeliveryToken <token>
		

For cross-cluster commands (indices copy, indices sync-remote, indices unify-incremental-sync), the destination cluster seeds from the same Parameters:ElasticsearchUrl / Parameters:ElasticsearchApiKey secret as the source — so configuring just one cluster in the secrets store is enough; pass --to-url/--to-api-key (or --from-url/--from-api-key) on the command to point only one side at a different cluster, or set DESTINATION_ELASTIC_URL/DESTINATION_ELASTIC_APIKEY to override the destination for every run.

If you previously used essc from the website-search-data repository, copy the Parameters:Elasticsearch* and ContentStack:* values from your old elastic-website-ai-search store into the docs-builder store (~/.microsoft/usersecrets/, on Windows %APPDATA%\Microsoft\UserSecrets\).

Fetches all published content from Contentstack and indexes it into Elasticsearch. Uses 5 parallel lanes. Cursors are saved after every page so interrupted runs resume automatically.

essc contentstack sync
		

--max-ai-docs uses DataAnnotations: it must be at least 1 when passed; 0 is invalid. For a large or unbounded batch without re-syncing, use contentstack ai-enrich instead.

The sync runs in two phases:

1. Fetch + index — pages through Contentstack in parallel, maps each item to a SiteDocument, and bulk-indexes into both the lexical and semantic indices.
2. Finalize — flushes the remaining buffer, reindexes the lexical index into the semantic index, publishes synonyms and query rules, then runs a bounded generative AI enrichment pass on the semantic write index (unless --no-ai).

Runs generative AI enrichment on existing site-* semantic indices without calling Contentstack. Same Elasticsearch overrides as contentstack sync.

essc contentstack ai-enrich
		

Lists all Contentstack content types and their document counts.

essc contentstack types
		

Dumps raw Contentstack entries for a given content type, useful for inspecting the source data structure.

essc contentstack samples <content-type>
		

Discovers labs URLs from sitemaps, crawls HTML, and bulk-ingests into labs-* indices. See essc labs sync --help for crawl flags (--dry-run, --force, --no-ai, --max-ai-docs, --max-ai-time, etc.).

Runs generative AI enrichment on existing labs-* semantic indices without re-crawling. Flags match contentstack ai-enrich (--max-ai-time, --max-ai-docs, --endpoint, --api-key).

essc labs ai-enrich
		

Copies one or more indices verbatim between clusters using the Elasticsearch server-side remote reindex API. Unlike indices sync-remote / indices unify-incremental-sync, this command is not docs-search specific — it copies no synonym/query-rule resources and runs no lexical/semantic inference phases. Useful for pulling standalone indices (e.g. AI-enrichment -ai-cache lookup indices) from production down to a local cluster.

essc indices copy docs-assembler.semantic-prod-ai-cache labs-public.semantic-prod-ai-cache \
  --from-url https://<prod-es> --from-api-key <key> \
  --to-url http://localhost:9200
		

• Takes one or more source index/alias names as positional arguments.
• --rename-from / --rename-to — regex rename applied to each source name to derive its destination name (back-references supported), e.g. --rename-from 'semantic-prod-' --rename-to 'semantic-local-'.
• --force — delete and recreate an existing destination index from the source's mapping and settings before reindexing. Without it, an existing destination index is reindexed into as-is.
• --strip-semantic-fields — strip _inference_fields metadata when the source has semantic_text mappings (see indices sync-remote).
• Shares --from-url / --from-api-key / --to-url / --to-api-key / --slices / --rps / --aliases with indices sync-remote and indices unify-incremental-sync. --slices has no effect on indices copy — every reindex it does is a remote reindex, and Elasticsearch rejects slices > 1 for reindex from a remote source.

When CI=true or stdout is not a TTY, essc switches from interactive progress widgets to plain log lines:

[Lane 1]    100%   blog — 1,240 fetched, 1,198 indexed
[Lane 2]     14%   customer_tile — 42 fetched, 40 indexed (2 skipped)
…      14%   secondary reindex — 1,240/8,500
✓     100%   secondary reindex — 8,500/8,500
		

essc ships as a distroless container only, published to ghcr.io/elastic/website-search-essc using the .NET SDK's native container support as part of the docs-builder release pipeline (./build.sh publishcontainers). Base image: mcr.microsoft.com/dotnet/nightly/runtime-deps:10.0-noble-chiseled. Tagged edge;latest;<version> on release, edge on every push to main.

AOT compilation is validated on every CI run by building the container image and running essc --help against it.