Streaming Audit Events to a SIEM with Kafka

Batch-exporting audit logs on a nightly job means a security team learns about a suspicious action hours after it happened. Streaming portal audit events through Kafka into a SIEM closes that gap to seconds, and gives you a durable, replayable buffer that survives a SIEM outage. This how-to emits structured audit events from Backstage, ships them over Kafka, and lands them in your SIEM. It extends Audit Logging & Compliance within Authentication, RBAC & Security Governance.

Prerequisites

  • Backstage 1.20+ already producing structured audit events for sensitive actions.
  • A Kafka cluster (or compatible stream) the portal backend can produce to.
  • A SIEM — Splunk, Elastic Security, or similar — with a Kafka consumer or connector.
  • A defined audit event schema so downstream consumers can parse every event the same way.
Prerequisites for streaming audit events Structured events, a Kafka cluster, a SIEM consumer, and a shared schema enable real-time audit streaming. structured events Kafka cluster SIEM consumer shared schema Real-time audit durable
Kafka sits between the portal and the SIEM as a durable buffer — if the SIEM is down, events wait on the topic rather than vanishing.

Exact Configuration

  1. Define a stable audit event schema so every event is parseable.

    {
  "timestamp": "2026-07-22T10:15:00Z",
  "actor": "user:default/asmith",
  "action": "catalog.entity.update",
  "target": "component:default/payments-api",
  "outcome": "allow",
  "sourceIp": "10.2.3.4"
}

  2. Produce audit events to Kafka from the backend audit hook.

    // packages/backend/src/audit/kafkaSink.ts
// Requires kafkajs >= 2.2
import { Kafka } from 'kafkajs';
const producer = new Kafka({ brokers: [process.env.KAFKA_BROKER!] }).producer();

export async function emitAudit(event: AuditEvent) {
  await producer.send({
    topic: 'portal.audit',
    messages: [{ key: event.actor, value: JSON.stringify(event) }],
  });
}

  3. Land events in the SIEM via its Kafka connector, mapping the topic to an index.

    # kafka-connect: splunk sink (illustrative)
connector.class=com.splunk.kafka.connect.SplunkSinkConnector
topics=portal.audit
splunk.hec.uri=${SPLUNK_HEC_URI}
splunk.indexes=portal_audit
Audit event stream The portal produces audit events to a Kafka topic, which a SIEM connector consumes into a searchable index. portal backend audit hook Kafka topic portal.audit SIEM connector consume SIEM index searchable
Keying each message by actor keeps one user's events ordered on a partition — useful when reconstructing a session in the SIEM.

Validation

# Requires kafka CLI, jq
# 1. Events land on the topic
kafka-console-consumer --bootstrap-server "${KAFKA_BROKER}" \
  --topic portal.audit --max-messages 1 | jq '.action'
# Expected: an action like "catalog.entity.update"

# 2. Every event matches the schema (no missing required fields)
kafka-console-consumer --bootstrap-server "${KAFKA_BROKER}" --topic portal.audit --max-messages 20 \
  | jq 'select((.actor and .action and .timestamp and .outcome) | not)'
# Expected: no output — all events well-formed

# 3. The SIEM shows the event within seconds
#    (query the SIEM for the action just performed)
echo "Confirm the test action appears in portal_audit within the SLA."
Audit streaming validation checks Events land on the topic, match the schema, and appear in the SIEM within seconds. Events land on the topic Schema valid no missing fields In SIEM fast within seconds
Validate schema conformance on the topic, not just in the SIEM — a malformed event that the SIEM silently drops is a blind spot.

Edge Cases & Troubleshooting

Symptom Root Cause Resolution
Events missing in the SIEM Connector lag or wrong topic Check consumer lag; confirm the topic name
Portal blocks on Kafka down Synchronous produce with no fallback Produce async; buffer or spool locally on failure
Duplicate events At-least-once delivery on retry Deduplicate in the SIEM on an event id
Unparseable events Schema drift over time Version the schema; reject events that fail validation
Sensitive data in events Secrets or PII in the payload Redact before emit; log identifiers, not values
Audit streaming pitfalls by class Missing events and blocking on Kafka are delivery issues; drift and leaked data are payload issues. Delivery missing events portal blocks → async + spool Payload schema drift leaked PII → version + redact
Never let audit emission block a user action — produce asynchronously and spool on failure, so a Kafka hiccup slows nothing.

Frequently Asked Questions

Why Kafka instead of writing straight to the SIEM?

The buffer. A direct write couples portal availability to SIEM availability — if the SIEM is slow or down, the portal either blocks or drops events. Kafka absorbs bursts and outages: events sit durably on the topic and the SIEM catches up when it can, and you can add a second consumer (a data lake, an alerting rule) without touching the portal.

How long should audit events be retained?

On the topic, long enough to survive a SIEM outage and any replay you need — often days. Long-term retention belongs in the SIEM or an archive tier governed by your compliance requirements. Keep the topic as a durable transport, not the system of record, and set retention against your recovery window.