Developer Experience Metrics

Quantifying developer experience requires correlating delivery-pipeline signals with how engineers actually use your portal, so that platform investment decisions rest on evidence rather than anecdote.

Platform teams are routinely asked to justify their roadmap, and “developers feel faster” is not a defensible answer in a budget review. This capability defines what to instrument across two complementary surfaces — the DORA delivery metrics that measure throughput and stability, and the portal usage signals that reveal whether self-service features are adopted or abandoned. Combined, they tell you whether a golden path is being followed and whether your service onboarding automation actually shortens time-to-first-deploy.

Prerequisites & Environment Baseline

Instrumenting developer experience touches the delivery pipeline, the portal frontend, and a metrics backend. Four things must be in place first: the portal analytics API, a metrics store, read access to CI/CD events, and a catalog that defines the deployable-unit boundary every metric attaches to.

Pin these versions before wiring anything together:

• Backstage >= 1.20.0 with the new backend system. The analytics API (@backstage/core-plugin-api) and the useAnalytics hook are stable here.
• A metrics store: either a managed Prometheus/Grafana stack (Prometheus >= 2.45) for delivery metrics, or a product-analytics provider for usage events. This guide assumes both.
• CI/CD event access: a service account token with read scope on your pipeline provider (GitHub Actions, GitLab CI) to ingest deployment events.
• A defined service boundary: DORA metrics are only meaningful per deployable unit, so the catalog must already model components and systems. Align this with your catalog integration patterns so every metric maps to a real entity.

# Verify the analytics API surface is available in your app
# Requires Backstage >= 1.20.0
grep -r "useAnalytics" packages/app/src | head -n 5

# Confirm Prometheus is scraping your delivery exporter
curl -sSf "${PROMETHEUS_URL}/api/v1/query?query=up{job=\"deploy-exporter\"}" | jq '.data.result[0].value'

Step-by-Step Configuration & Plugin Architecture

Instrumentation proceeds in four steps: define the metric contract, wire the portal analytics API, emit explicit DevEx events on the actions you care about, and ingest delivery events tagged by entity. The diagram sequences them.

1. Define the metric contract

Decide what each metric means before collecting anything. The four DORA metrics — deployment frequency, lead time for changes, change failure rate, and mean time to restore — are detailed in Measuring Developer Experience with DORA Metrics. On the usage side, define a small set of events: template execution, catalog search, doc views, and plugin route hits.

2. Wire the portal analytics API

Register an analytics implementation in the app so every navigation and custom event is captured. The deep mechanics — choosing a provider, batching, and PII scrubbing — live in Instrumenting Portal Usage Analytics.

// packages/app/src/apis.ts
// Requires @backstage/core-plugin-api >= 1.9.0
import {
  analyticsApiRef,
  configApiRef,
  createApiFactory,
} from '@backstage/core-plugin-api';
import { GenericAnalytics } from './analytics/GenericAnalytics';

export const apis = [
  createApiFactory({
    api: analyticsApiRef,
    deps: { configApi: configApiRef },
    factory: ({ configApi }) =>
      GenericAnalytics.fromConfig(configApi, {
        endpoint: process.env.ANALYTICS_ENDPOINT ?? '',
      }),
  }),
];

3. Emit a custom DevEx event

Beyond automatic page views, emit explicit events on the actions you care about — for example, when an engineer completes a self-service template run.

// packages/app/src/components/onboarding/OnboardComplete.tsx
// Requires @backstage/core-plugin-api >= 1.9.0
import { useAnalytics } from '@backstage/core-plugin-api';

export const OnboardComplete = ({ templateRef }: { templateRef: string }) => {
  const analytics = useAnalytics();
  // Fired once when onboarding finishes; correlate with lead-time later
  analytics.captureEvent('onboard_complete', templateRef, {
    attributes: { surface: 'scaffolder' },
  });
  return null;
};

4. Ingest delivery events

Run a lightweight exporter that turns deployment webhooks into Prometheus counters keyed by component. Tag every series with the catalog entityRef so dashboards join cleanly to ownership data.

# app-config.yaml
# Requires Backstage >= 1.20.0
proxy:
  endpoints:
    /devex-metrics:
      target: ${DEVEX_METRICS_API_URL}
      headers:
        Authorization: Bearer ${DEVEX_METRICS_TOKEN}

Validation & Health Checks

Confirm both data planes are flowing before building dashboards on top of them: portal events reach the collector, a lead-time series exists per component, and custom DevEx events are recorded.

# Requires Backstage >= 1.20.0
# 1. Confirm portal events reach the collector (expect HTTP 202)
curl -s -o /dev/null -w "%{http_code}\n" \
  -X POST "${ANALYTICS_ENDPOINT}/collect" \
  -H "Content-Type: application/json" \
  -d '{"event":"healthcheck","ts":"2026-06-19T00:00:00Z"}'
# Expected: 202

# 2. Confirm lead-time series exists per component
curl -s "${PROMETHEUS_URL}/api/v1/query?query=dora_lead_time_seconds" \
  | jq '.data.result | length'
# Expected: >= 1 (one series per deployable component)

# 3. Confirm custom DevEx events are recorded
curl -s "${ANALYTICS_ENDPOINT}/query?event=onboard_complete" | jq '.count'
# Expected: a non-zero integer once onboarding has run

Maintenance & Lifecycle Management

Maintenance centers on two ongoing risks: provider churn and metric cardinality. Pin the analytics provider, keep a no-op fallback for instant rollback, watch label cardinality monthly, and govern event retention like audit data.

• Upgrade path: the analytics API is stable, but provider plugins move faster. Pin the provider package and review its changelog on each Backstage minor bump before rolling to production.
• Rollback: analytics is non-critical to portal function. If a provider misbehaves, swap analyticsApiRef back to the no-op NullAnalytics factory and redeploy — the portal keeps working with zero collection.
• Debug commands: enable analytics.debug: true in app-config.yaml to log every event to the browser console without sending it, which isolates “events not firing” from “events not delivered.”
• Metric hygiene: review cardinality monthly. Per-user attributes on Prometheus series explode storage; keep high-cardinality dimensions in the product-analytics store, not the time-series DB.
• Retention: align event retention with your audit logging and compliance policy, since usage events can contain user identifiers.

Common Pitfalls & Mitigation Strategies

The metrics pitfalls split into measurement mistakes (individual scorecards, vanity counts) and data-hygiene mistakes (unbounded cardinality, orphaned series, ungoverned PII). The diagram groups them so the fix is either a framing change or a data control.

• Treating DORA as an individual scorecard — Root cause: misreading team-level delivery metrics as productivity rankings. Fix: aggregate strictly at the team/service level and pair every DORA number with a qualitative signal.
• Vanity page-view counts — Root cause: instrumenting only raw views. Fix: model funnels and retention so you can tell adoption from accidental clicks.
• Unbounded label cardinality — Root cause: tagging time-series with user IDs or commit SHAs. Fix: keep identifiers in the event store; reserve Prometheus labels for low-cardinality dimensions like team and lifecycle.
• Orphaned metrics — Root cause: series that do not map to a catalog entity. Fix: enforce an entityRef label at the exporter and drop unmatched samples.
• Collecting PII without governance — Root cause: capturing emails or full URLs by default. Fix: scrub on the client and document retention against your compliance baseline.

Frequently Asked Questions

Should DORA metrics or portal usage metrics come first?

Start with portal usage analytics if your self-service platform is new — it tells you whether anyone is using what you built, which is the prerequisite question. Layer DORA metrics once adoption is non-trivial, because DORA only moves when enough teams flow through your paved roads to shift the delivery aggregate.

How do we avoid metrics becoming a surveillance tool?

Report at the team and service level, never the individual. Make dashboards transparent and accessible to the engineers being measured, frame every metric as a question about the platform rather than the person, and govern any user-level event data under the same retention rules as audit logs.

Can we compute DORA without a dedicated tool?

Yes. Deployment frequency and change failure rate come from your CI/CD provider’s API; lead time comes from joining commit timestamps to deploy timestamps. A small exporter feeding Prometheus is sufficient for most organizations before a commercial product is justified.

Related

• Up to Developer Experience & Self-Service Platforms
• Measuring Developer Experience with DORA Metrics
• Instrumenting Portal Usage Analytics
• Service Onboarding Automation
• Tracking Golden Path Adoption — measure whether teams actually take the paved road.