The dev server got slower with every save

Writing a static site generator is the easy part. Markdown goes in, HTML comes out, and there are good libraries for both halves. The hard part is the dev server, because a dev server runs for hours, and anything that leaks, drifts or piles up will find you around the twentieth save.

Stati's dev server had two of those, and after the second one I built the tools to catch the next one before I feel it.

Save, wait, save, wait longer

The first one showed up on Windows, which is where I do most of my work.

Edit layout.eta, save, and the dev server rebuilds every page. The first rebuild took about 1900 ms on a site with forty-odd pages. The fifth took 2200. The tenth, 2800. It never came back down. Restart the server and it was 1900 again, so it wasn't the site getting bigger, it was the server getting tired.

I timed the phases, and it wasn't the rendering. Rendering was flat. Everything that touched the disk was climbing:

Releasing the lock is deleting one small file. Six hundred milliseconds to delete one file is not Node being slow. And it didn't reproduce on macOS or Linux at all.

My best explanation, and I never got a better one, is Windows Defender. Real-time protection scans a file every time it's written, and it seems to get more suspicious of a file the more often you rewrite it, which is exactly what a dev server does to dist/. NTFS journaling probably adds its share.

You can't fix an antivirus from an npm package, so the fix was to give it less to do:

• Don't write what didn't change. The search index and the Tailwind inventory are hashed, and when the hash matches the last write, nothing is written.
• Write the HTML in one batch, in parallel, instead of one file at a time between renders. Fewer, denser bursts of I/O.
• Cache what the build reads: the markdown-it instance, the compiled templates, file hashes, the config. A rebuild shouldn't re-read what it already knows.
• Skip TypeScript in dev when the bundles already exist.
• One dev server per folder. A lock file, so two servers can't fight over the same dist/. I had done that to myself more than once.

That was core 1.22.0, published on Christmas Day, which tells you how much it annoyed me. The HTML report of the metrics system (more on that below) now has a line under the file-write phase saying that antivirus scanning can increase this time significantly. Every word of it was earned.

The watcher that watched too much

The second slowdown was subtler, and I found it in Stati's own docs site. Same test: edit layout.eta twenty-odd times with a short pause between saves. Rebuilds went from 664 ms to 2563 ms, and the process's memory from 164 MB to 987 MB. Not Windows this time, and none of the Defender fixes touched it.

So I did it properly. The build function in a plain loop, same site, same edit: flat at about 520 ms. The build with the dev server's flags: flat. The build with the TypeScript watcher running next to it: flat. A synthetic dev server with 40 pages and a dozen partials: flat. The real dev server against the real docs site: creeps. Heap snapshots between rebuild 3 and rebuild 17: nothing retained. File watchers, child processes, sockets: stable. Memory was climbing, the heap was not, and nothing was leaking in the JavaScript sense of the word.

Then I switched things off one at a time with environment variables until the creep went away.

The CSS watcher. I had added it so that Tailwind or Sass writing a stylesheet into dist/ reloads the browser without a full rebuild. It watched dist/**/*.css, a recursive glob over the output folder, while every rebuild wrote forty-one files into that same folder. The watcher did a little more work on every cycle, and a little more, and the work never went away.

The fix was embarrassing in its size: find the CSS files once at startup and watch exactly those, not the folder. Early median 618 ms, late median 553 ms. Flat, slightly improving, which is what a warm cache should look like.

Along the way I also stopped throwing the whole Eta engine away when one template changed, and evict just that template from its cache instead. That was a real improvement, and I was sure it was the cause. It wasn't. The thing I added to make dev nicer was the thing making dev slower. It usually is.

Measuring instead of feeling

Both of these I found by noticing. I don't want to find the next one by noticing, so Stati now carries its own instruments.

Build metrics. stati build --metrics writes a JSON file per build into .stati/metrics/: the time of every phase, how many pages were rendered and how many came from the cache, peak memory, the git commit, the Node version and the platform. --metrics-detailed adds a timing for every page, with the number of templates it loaded. --metrics-html draws it as a report with a timeline and a waterfall. STATI_METRICS=1 turns it on without touching the build script, which is how it runs in CI.

Benchmarks with a baseline. Four scenarios on a generated 100-page site: a cold build, a warm build, one changed file, and a build with nested components. One warm-up and seven measured runs, and the median is compared to a baseline file in the repo. A benchmark fails when its median is more than 30% or 40 ms over the baseline, whichever is larger, so a noisy CI runner doesn't fail a 50 ms build for taking 60. The workflow comments the table on the pull request.

// perf/baselines/benchmark.json, one of the four
"warmBuild": {
  "medianMs": 50,
  "p95Ms": 80,
  "tolerance": 0.3,
  "minHeadroomMs": 40,
  "minCacheHitRate": 0.95
}

A leak test. A hundred rebuilds of a 40-page site, run in a worker with --expose-gc so the test can force a collection and compare the live heap before and after. It fails if the heap grows more than 25 MB over the session, or if the late rebuilds take more than 1.6 times the early ones.

The slowdown check. The exact experiment from above, as a script: boot the real dev server against the docs site, edit the layout twenty times with 700 ms of idle in between, compare the early median with the late one, and fail over a ratio of 1.25. It runs on every pull request as its own workflow and comments the numbers.

npm run test:dev-rebuild-slowdown

None of this makes Stati faster, but a slowdown now shows up on the pull request before it ships.

Rocks along the way

1. Everything that touched the disk got slower on Windows, and the fix was to touch it less.
2. The dev server crept and the build didn't, so the problem was in the wiring around the build.
3. Memory that climbs while the heap stays flat is accumulated work. Look at what runs on every cycle rather than at what's retained.
4. A convenience watcher over the output folder is a loop waiting to happen.

The dev server is flat now. I checked. Twenty times, with 700 ms in between.