Integrating Elasticsearch as the Search Backend

The in-memory search engine that ships with Backstage rebuilds its index in the backend’s own memory on every restart, holds a separate copy per replica, and searches it linearly — fine for a demo, unworkable once the catalog is real. This how-to replaces it with Elasticsearch (or OpenSearch, which uses the same client), so the index lives out of process, is shared across replicas, and survives restarts. It is the scaling step described in Search & Discovery in the Portal within Plugin Ecosystem & Custom Extensions.

Prerequisites

  • Backstage 1.20+ with the @backstage/plugin-search-backend already wired and working on the in-memory engine.
  • A reachable Elasticsearch 8.x or OpenSearch 2.x cluster — managed or self-hosted — with credentials.
  • The @backstage/plugin-search-backend-module-elasticsearch module installed in the backend.
  • Connection details injected at runtime as ${ES_NODE}, ${ES_USER}, ${ES_PASS} — never committed.
Prerequisites for the Elasticsearch backend A working search backend, a reachable cluster, the ES module, and injected credentials enable the swap. search backend works ES/OpenSearch cluster ES module installed credentials injected Shared index out of process
Prove search works on the in-memory engine first; swapping the engine should change where the index lives, not whether search functions.

Exact Configuration

  1. Install the module and register it in the backend.

    # Requires Backstage >= 1.20
yarn --cwd packages/backend add @backstage/plugin-search-backend-module-elasticsearch

    // packages/backend/src/index.ts
backend.add(import('@backstage/plugin-search-backend-module-elasticsearch'));

  2. Point search at the Elasticsearch cluster in config. The elasticsearch block replaces the default engine.

    # app-config.yaml
# Requires @backstage/plugin-search-backend-module-elasticsearch >= 1.5.0
search:
  elasticsearch:
    node: ${ES_NODE}
    auth:
      username: ${ES_USER}
      password: ${ES_PASS}
    # batchSize controls how many docs are indexed per bulk request
    batchSize: 1000

  3. Schedule collation so the index is refreshed on a cadence rather than only at startup.

    # app-config.yaml
search:
  collators:
    catalog:
      schedule:
        frequency: { minutes: 10 }
        timeout: { minutes: 15 }
Collation into Elasticsearch Scheduled collators pull documents and bulk-index them into Elasticsearch, which every backend replica queries. scheduled collator every 10 min bulk index (1000) Elasticsearch shared index replica A replica B
One shared index means a document indexed once is searchable from every replica — the core reason to leave the in-memory engine.

Validation

# Requires curl, jq
# 1. The Backstage index exists and has documents
curl -s -u "${ES_USER}:${ES_PASS}" "${ES_NODE}/_cat/indices?v" | grep backstage
# Expected: a backstage-* index with a non-zero docs.count

# 2. A query returns hits from the cluster
curl -s -u "${ES_USER}:${ES_PASS}" "${ES_NODE}/backstage-*/_search?q=payment" | jq '.hits.total.value'
# Expected: > 0 for a term you know is indexed

# 3. The portal search API returns results end to end
curl -s "${PORTAL_URL}/api/search/query?term=payment" | jq '.results | length'
# Expected: > 0
Elasticsearch validation checks A populated index, a direct query with hits, and an end-to-end portal query confirm the integration. Index populated docs.count > 0 Direct query hits > 0 Portal query end to end
Check the Elasticsearch cluster directly and through the portal — a populated index that the portal cannot reach points at credentials or config, not indexing.

Edge Cases & Troubleshooting

Symptom Root Cause Resolution
Index stays empty Collators never ran or errored on startup Check backend logs for collation errors; confirm a schedule is set
Search works locally, empty in prod Prod points at a different or unreachable cluster Verify ${ES_NODE} and network policy from the backend pods
Indexing times out on large catalogs batchSize too large for the Elasticsearch cluster Lower batchSize; raise the collator timeout
Auth errors from Elasticsearch Wrong credentials or missing index privileges Grant the user create/write on backstage-*
Old documents linger after deletion Index not pruned between collation runs Confirm the engine replaces the index per run rather than only upserting
Elasticsearch pitfalls by class Empty and stale indexes are collation issues; unreachable clusters and auth errors are connectivity issues. Collation empty index stale documents → check schedule + prune Connectivity unreachable node auth error → verify node + grants
An empty index almost always traces to collation, not the Elasticsearch cluster — confirm a collator ran before you suspect Elasticsearch itself.

Frequently Asked Questions

Can I use OpenSearch instead of Elasticsearch?

Yes. The same module speaks to OpenSearch through the compatible client — set the node to your OpenSearch endpoint and provide its credentials. Teams already running OpenSearch for logging often reuse that cluster for portal search rather than standing up Elasticsearch separately.

Does swapping engines change how I write collators?

No. Collators emit engine-agnostic documents; the engine is what stores and queries them. That separation is deliberate — you can move from in-memory to Elasticsearch to OpenSearch without touching a single collator, and any custom collator you have written keeps working unchanged.