Resolving OOM Errors and Build Timeouts When Optimizing Static Site Generation for 10k+ Pages

When scaling internal documentation platforms beyond 10,000 markdown files, static site generators frequently encounter heap exhaustion (FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed) or CI/CD pipeline timeouts. This guide provides a targeted resolution path for platform engineers facing these bottlenecks. By adjusting runtime memory allocation, implementing incremental compilation, and restructuring asset pipelines, teams can reduce build times by 60–80% while stabilizing memory consumption. This approach aligns with established patterns in modern Developer Portal Architecture & Frameworks and ensures predictable deployment cycles for enterprise-scale knowledge bases.

Full rebuild versus incremental hash-based build A full rebuild loads the entire document graph into memory and risks OOM, while a hash-cached incremental build re-renders only changed pages. Full rebuild 10k-page graph All in heap GC thrash OOM risk Incremental build Changed files Hash cache .ssg-cache < 10s rebuild Bounded memory peak RSS < 3.5GB capped workers
Hash-based caching turns a memory-bound full rebuild into a fast, bounded incremental build.

Context: Identifying the SSG Bottleneck at Scale

Static site generation (SSG) pipelines typically load the entire document graph into memory before rendering. At 10k+ pages, AST traversal, link validation, and template hydration phases trigger garbage collection thrashing. Symptoms include exponential build time growth, OOM crashes on standard CI runners (4GB–8GB), and failed incremental deployments.

Before applying fixes, isolate the bottleneck rather than guessing: trace garbage collection, watch peak resident memory through the render phase, and confirm the thrash is in template hydration before you touch a single flag.

Isolating the build bottleneck before tuning Trace GC, monitor peak RSS, and locate the thrashing phase before changing any configuration. Trace GC --trace-gc Watch peak RSS render phase Locate thrash hydration phase Then tune
Measure first: tuning a flag before locating the thrash phase usually moves the bottleneck rather than removing it.
NODE_OPTIONS=--trace-gc npm run build

Monitor peak RSS during the compilation phase. For teams standardizing on Python-based generators, reviewing MkDocs for Internal Docs performance profiles reveals similar memory mapping constraints that require parallelized processing and strict cache invalidation rules.

Exact Fix & Configuration

Apply the following three-tier configuration to stabilize builds and enforce memory boundaries. The tiers stack: runtime limits cap memory, framework hash-caching avoids full rebuilds, and asset pre-bundling removes the last I/O bottleneck.

Three configuration tiers Runtime memory tuning, framework hash-caching, and asset pipeline optimization stack to stabilize the build. Tier 1 — runtime limits max-old-space-size · capped concurrency Tier 2 — framework hash-cache workers · .ssg-cache · no source maps Tier 3 — asset pipeline pre-bundle · CDN · routing manifest
Each tier removes a distinct bottleneck; applied together they cut build time 60–80% and cap peak memory.

1. Runtime Memory & Concurrency Tuning

Override default V8 limits to prevent premature OOM while capping allocation to avoid host machine thrashing.

export NODE_OPTIONS="--max-old-space-size=4096 --max-semi-space-size=64"
export CI=true
npm run build -- --incremental --concurrency=4

2. Framework Configuration

Disable full-graph rebuilds. Configure your SSG to track file hashes and only re-render modified nodes. Enable worker threads for template hydration and disable source map generation in production builds to reduce I/O overhead.

{
  "build": {
    "workers": 4,
    "sourceMaps": false,
    "cacheDir": ".ssg-cache",
    "linkValidation": "warn",
    "parallelHydration": true
  }
}

3. Asset Pipeline Optimization

Pre-bundle static assets and serve them via CDN. Flatten nested directory structures to reduce path resolution overhead. Implement a routing manifest instead of dynamic route generation at build time to decouple compilation from filesystem traversal.

Validation & Performance Metrics

After applying the configuration, validate the pipeline using deterministic benchmarks. Three numbers tell you whether the fix held: full-build wall time, peak resident memory, and incremental rebuild time after touching one file.

Three thresholds that confirm the optimization Full build under eight minutes, peak RSS under 3.5 gigabytes, and incremental rebuild under ten seconds. Full build < 8 min · 4 cores Peak RSS < 3.5 GB Incremental < 10 s · 1 file
If any of the three drifts, the earlier configuration regressed — measure them on every pipeline change.
  1. Capture Build Metrics:
    time npm run build
# After the build completes, check peak memory from system monitoring
# On Linux: check /proc/<pid>/status or use 'time -v npm run build'
  2. Target Thresholds:
    • Build time: < 8 minutes on a 4-core runner
    • Peak RSS: < 3.5GB
    • GC Pauses: 0 pauses exceeding 500ms
  3. Incremental Verification: Run a link validator against the output directory. Touch a single markdown file and trigger a rebuild. A properly configured pipeline should complete in < 10 seconds and only output modified HTML files.
  4. Rollback & Audit: If metrics deviate, audit the worker thread pool size and ensure the cache directory resides on a high-throughput SSD or tmpfs mount. Revert to --concurrency=1 temporarily to isolate CPU contention.

Edge Cases & CI/CD Constraints

Large-scale SSG pipelines often fail due to runner timeouts or network I/O limits, not the generator itself. The diagram pairs each infrastructure constraint with the mitigation that keeps the build inside it.

CI constraints paired with mitigations Strict timeouts, disk I/O limits, and monorepo contention each map to a specific mitigation. Strict 15-min timeout Disk I/O latency Monorepo contention Matrix split by prefix tmpfs for artifacts Isolate doc build
The generator rarely is the limit; the runner's clock and disk are — mitigate those directly.
  • Strict CI Timeouts (e.g., 15 min): Split the build into parallel matrix jobs by directory prefix.
  • Orphaned Links: Configure the SSG to log warnings instead of failing the pipeline during migration; switch to errors once the doc set is clean.
  • Monorepo Contention: Isolate documentation builds from application builds to prevent resource contention.
  • Disk I/O Bottlenecks: Monitor write latency on ephemeral runners; switch to tmpfs for intermediate build artifacts if latency exceeds 50ms.
  • CDN Staleness: Configure cache headers to bypass stale-while-revalidate during deployment windows to prevent partial HTML delivery.

Common Pitfalls

Every pitfall below is a default left unchanged — Node’s heap limit, source maps, full rebuilds. The diagram shows the two categories the five pitfalls fall into: memory defaults and build-strategy defaults.

Scaling pitfalls grouped by the default that causes them Memory defaults and build-strategy defaults are the two roots behind the common large-site pitfalls. Memory defaults 1.5GB heap limit source maps on disk I/O ignored Build-strategy defaults full rebuild always assets not pre-bundled
Change the defaults on both sides — raise the heap and cache by hash — and the five pitfalls stop recurring.
  • Leaving source maps enabled in production, which doubles I/O and memory overhead during asset bundling.
  • Relying on default Node.js heap limits (approximately 1.5GB on 64-bit systems) for large graph traversals.
  • Running full rebuilds on every commit instead of implementing hash-based incremental caching.
  • Ignoring CI runner disk I/O limits when writing thousands of HTML files simultaneously.
  • Failing to pre-compile or CDN-host static assets, causing template hydration bottlenecks.

Frequently Asked Questions

Why does my SSG crash with ‘Ineffective mark-compacts near heap limit’ at 10k pages? The default V8 heap limit (~1.5GB) is insufficient for loading and traversing a 10k+ node AST. The garbage collector cannot reclaim memory fast enough during template hydration, triggering an OOM. Increase --max-old-space-size and enable incremental builds to reduce peak memory pressure.

Can I run parallel builds on a standard GitHub Actions runner? Yes, but you must cap concurrency to 2–4 workers on a 2-core runner to avoid CPU thrashing. Use matrix strategies to split documentation directories across separate jobs if build times exceed runner timeouts.

How do I verify that incremental builds are actually working? Enable verbose logging and check the cache directory for hash mismatches. Touch a single markdown file and run the build command. A properly configured incremental pipeline should complete in under 10 seconds and only output modified HTML files.