Skip to content

Reproduce the synthetic content-loading benchmark

The repository benchmark measures how long the CLI’s content loader takes to read, schema-parse, and build course graphs for a known synthetic fixture. It is not an Astro build benchmark, browser performance test, or measurement of real courses or creators.

Build the CLI first, then run the benchmark from the repository root:

Terminal window
pnpm build
pnpm benchmark:content

The command creates temporary content trees and writes the raw, machine-readable record to the ignored release/content-benchmark.json. Re-run it to compare method and output shape; elapsed times can vary between runs and machines.

The fixture creates 10, 100, and 1,000 lessons. Each scale has ten lessons per course, one module per course, one shared synthetic instructor, and the same draft/free course and lesson frontmatter and body shape. Every run verifies that all courses, modules, lessons, schemas, and course graphs load without issues before it records a sample.

Each scale has five timing samples in one process. The record reports every sample, the first in-process run, later repeat samples, their median and range, and the median across all samples. The first call can be slower due to process and filesystem activity, but the operating-system file cache is not cleared or controlled. Later calls are therefore warm in-process observations, not a controlled comparison against a true cold cache.

The record also includes the Git source revision and whether tracked source was clean, a SHA-256 digest of pnpm-lock.yaml, Node and operating-system details, CPU model and count, available memory, and the fixture definition. If you change the fixture or measurement method, update its definition in the script and interpret old records using their stored schema version and source revision.

The 30-second per-sample ceiling is a coarse regression guard, not a desired target and not a cross-machine service-level objective. To investigate a possible regression, compare runs from the same candidate, lockfile, operating system, hardware, fixture, and command. Look at all samples and the repeat-run range; do not infer a stable result from a single millisecond value. Record any environment difference before comparing.

This benchmark contains no production lesson data, learner identifiers, deployment results, or user counts. A passing local sample does not qualify an adapter build, accessibility, live payment, CDN behavior, or a deployed site; those checks have separate evidence in the maintainer-only owner qualification runbook.