← Module 8/Job systems
RU
Module 8 · Technical deep dive

Job systems

A modern CPU is wide, not fast: 8–16 cores, not one quick one. A single-threaded game wastes ~90% of the machine, and single-threaded gameplay is AAA's main bottleneck. A job system saturates the cores with a task graph — but parallelism is bounded (Amdahl's law) and dangerous (data races).
~17 min🛠 parallelism + 🔬 Amdahl
The gist in 30 seconds
A 2026 CPU has 8–16 cores; a single-threaded game occupies one and leaves ~90% idle (this is AAA's main performance bottleneck). Engines saturate the cores with a job system: the frame is split into small tasks with explicit input/output dependencies → the scheduler builds a DAG → worker threads pull tasks from queues, and idle ones steal from busy ones (work stealing). Two flavours: job-based (Unity Jobs+Burst, Bevy) and fiber-based (Naughty Dog). But speedup runs into Amdahl's law: a serial fraction s sets a ceiling of 1/s — 10% serial → at most 10× no matter how many cores. And the price is correctness: shared mutable state → data races (the worst bugs). So you design for data parallelism (independent work with no shared writes — hello ECS/SoA) and profile ruthlessly (Tracy: find frames >16 ms, dig into the worst zone). The frame budget — 16.67 ms at 60 fps — is a hard deadline everything has to fit inside.

The mechanism: a task graph under a hard deadline

The frame budget and why to parallelize

60 fps = 16.67 ms per frame (8.3 ms at 120): input, gameplay, physics, animation, culling and submitting render commands all have to fit inside. Miss it and you get a dropped frame. And a 2026 CPU has 8–16 logical cores; a single-threaded game uses one and leaves 90% of the compute on the table. That is why modern engines are built around a job system: scene updates, physics, AI and building render commands all run as job graphs across cores. Single-threaded gameplay code inside a multi-threaded engine is AAA's most common bottleneck.

The job system: DAG plus work stealing

Work is split into small jobs, each declaring what it reads and what it writes. The scheduler builds a dependency graph (task B waits on A if it reads A's output) and runs independent tasks in parallel across worker threads. Balancing is done by work stealing: each worker has its own queue, and an idle one steals a task from the tail of someone else's. Two dominant approaches:

Amdahl's law — the ceiling on speedup

Adding cores does not mean speeding up linearly. If a fraction s of the work is inherently serial (it does not parallelize — sync points, dependencies, the main thread), the speedup on N cores is:

speedup(N)= 1 s+1−sN

Example: s=0.1 (10% serial), N=8 → 1/(0.1 + 0.9/8) ≈ 4.7×, not 8×. And as N→∞ the ceiling is 1/s=10×, however many cores you throw at it. The takeaway: the goal is to shrink the serial fraction (usually the main thread and synchronization), not just to pile on cores. Amdahl is brutal: 5% serial caps you at 20×.

The price is correctness: data races

Parallel tasks writing into shared mutable state produce data races — two threads write the same thing and the result is undefined; these are the worst bugs (non-deterministic, intermittent, vanishing under a debugger). Synchronization (mutexes, atomics) fixes correctness but adds contention — threads wait on each other, and waiting means serialization, which makes Amdahl worse. The clean solution is data parallelism with no shared writes: partition the data so each task owns its own slice. That is the whole point of ECS / data-oriented design: systems read and write non-overlapping components, the scheduler sees the independence and parallelizes safely. A subtle trap is false sharing: two threads write different data on the same cache line → it ping-pongs between cores, killing speed without any logical race at all.

Profiling: measure, don't guess

You cannot optimize what you have not measured. The industry standard of the 2020s is Tracy (FOSS, used by id/CDPR/Naughty Dog and hundreds of indies): you mark functions with the ZoneScoped macro → Tracy draws a per-frame Gantt chart of CPU+GPU+locks on one timeline. The loop: instrument → run → find frames >16 ms → drill into the worst zone → optimize → measure again. Guessing "where it is slow" is the classic mistake: the bottleneck is almost never where it seems to be.

🕹 What to switch on — and what to notice

Parallelism is visible in core load and in the profiler — and in where a game's "sim bottleneck" sits.

Factorio / Cities: Skylines simulation · CPU/thread-bound

Simulation-heavy games hit the CPU: grow the number of entities (factories, agents) and FPS/UPS falls, because the simulation is thread-bound rather than GPU-bound. The perfect bench for "a wide CPU against a serial simulation".

🎮 Do: in Factorio/Cities, blow up your base or city and open the task manager: are all the cores loaded, or just one? Many simulations were single-threaded for a long time (Amdahl: a serial simulation does not parallelize) — you will see one core pegged while the rest idle. That is exactly "90% of the machine left on the table".

The engine profiler / Tracy a thread timeline

In Tracy/Unity Profiler/UE Insights you can see the main thread and the workers on one timeline: where jobs run in parallel, and where everything waits on a sync point (the serial "isthmus" — Amdahl's fraction s).

🎮 Do: open the profiler of any game or engine and find the sync points — the moments where all the workers idle waiting on one thread. That is the serial fraction capping your speedup. Estimate it: remove that, and how much would parallelism improve?

CPU-bound vs GPU-bound diagnosing the bottleneck

The same game can stutter for different reasons: FPS drops in "crowded" scenes (many entities/physics → CPU/jobs) or at high resolution (→ GPU/fill). The diagnosis changes what you optimize.

🎮 Do: in a game, halve the resolution. FPS jumps a lot → you were GPU-bound. Barely changes → CPU-bound (jobs/gameplay/draw calls). It is a 30-second test that tells you where to look at all — in the job system or in rendering.

Deep end · theory: Amdahl, Gustafson and the serial isthmusskippable

Why the serial fraction caps you so painfully

Amdahl: speedup=1/(s+(1−s)/N). The parallel part shrinks as N grows while the serial part does not, so in the limit it dominates: at s=0.05 the ceiling is 20×, at 0.2 only 5×. Hence the engineering priority: not "how many cores" but "what is the serial fraction" — sync points, global locks, everything-waits-on-the-main-thread dependencies. Removing 1% of serial work is often worth more than doubling the cores.

Gustafson — the other side

Amdahl fixes the size of the problem. Gustafson observes that in practice, with bigger hardware we solve bigger problems (more entities, more detailed physics), and there the parallel part grows with N while the serial part barely does — so speedup by volume is better than Amdahl predicts. In games that means more cores buys not "the same game faster" but "more agents/particles within the same budget". Both laws are true: Amdahl is about a fixed problem, Gustafson about a growing one.

Work stealing and balancing

The naive "hand each worker 1/N of the tasks" breaks when tasks are unequal in length (one worker finishes while others are still grinding). Work stealing: you take your own tasks from the head of your queue (cache-local), and when idle you steal from the tail of someone else's (less conflict with the owner). That gives dynamic balancing without a central scheduler bottleneck and lands close to optimal utilization — the standard in Cilk, TBB, Rayon, Unity/Bevy.

Deep end · engineering: races, false sharing, and job vs fiberskippable

Why races are the worst bugs

A data race is non-deterministic: it depends on the exact timing of threads, so it reproduces once in a thousand runs, vanishes under a debugger (which changes timing — a heisenbug) and behaves differently on another machine. Sanitizers (TSan) and a "no shared writes" model are the only reliable defense. The rule: shared mutable state is the source of every race; remove the mutability (immutability) or the sharing (partition the data) and there are no races by construction. That is why ECS and Rust (the borrow checker) get along so well with parallelism.

False sharing

The cache works in lines (~64 bytes). If two threads write different variables that happen to land on the same line, each write invalidates that line on the other core → it ping-pongs across the bus, and "independent" threads slow each other down with no logical race. The cure is aligning/padding hot per-thread data to a cache-line boundary. The classic "I parallelized it and it got slower" trap.

Job vs fiber

Job: a task is a short function with no blocking; dependencies are expressed as a graph and the scheduler runs whatever is ready. Simple, predictable, friendly to data-oriented design. Fiber: a task can yield in the middle of itself while waiting on a dependency and resume later on any thread — convenient for complex chains of waits (Naughty Dog), but harder (manual management of fiber stacks, subtleties with thread-local storage). The choice is simplicity/data-oriented (job) versus flexibility for complex dependencies (fiber).

Analogy
Parallelizing a frame is like cooking a banquet with N cooks. You cut a dish into tasks (chop, sear, plate) with dependencies (you cannot plate before searing). A cook who frees up grabs the next ready task (work stealing). But some steps are inherently sequential — reducing a sauce takes as long as it takes, and a second cook will not speed it up; that serial node is what caps how much N cooks help at all (Amdahl). And if two cooks grab the same pan (shared state) — chaos (a race); which is why each gets their own station (data parallelism). More cooks help only until the sequential steps and the jostling coordination start to dominate.
Why it matters
Modern CPUs are wide, not fast: 8–16 cores, not one quick one. A single-threaded game wastes the machine, and single-threaded gameplay is AAA's main performance bottleneck. Job systems saturate the cores, but parallelism is bounded (Amdahl: the serial fraction caps speedup) and dangerous (races are the worst bugs). So the craft is to minimize the serial fraction, design for data parallelism with no shared writes (ECS/SoA) and profile ruthlessly (Tracy). This is the core of systems engineering, and it transfers one-to-one to any parallel workload — including distributed ML.
🔁 Beyond games — where this transfers
The lesson is saturating a parallel machine under a deadline: a task DAG, Amdahl, races, balancing.

ML / AI (your domain): this is literally distributed training and inference. Amdahl's law caps training scale-out: the serial/communication fraction (all-reduce sync, the straggler — the slowest worker gating the sync) prevents linear gains from adding GPUs — why "2× the GPUs" ≠ "2× faster". Data parallelism with no shared writes = data-parallel training (each GPU gets its own shard of the batch, gradients synced) — the same principle of non-overlapping data as in a job system. Work stealing/balancing = dynamic balancing and the straggler problem. Races/synchronization = consistency in async SGD (staleness, lock-free). The frame budget (a hard deadline) = the latency budget of real-time inference. And "measure, don't guess" is ML performance discipline: profile the training loop — the bottleneck is often not the matmul but data loading / host→device (the same "serial isthmus"). "The machine is wide, not fast" is the whole essence of accelerators.

Backend / distributed systems: DAG schedulers (Airflow, task graphs), worker pools and work stealing, races and locks, a critical section as the serial isthmus; what scales is not what is parallel but what has a small serial fraction.

Performance in general: profile before optimizing; speed up the serial path (Amdahl) rather than only adding parallelism; avoid shared mutable state.

Principle: adding cores or GPUs helps only as far as the serial fraction and coordination are small. Minimize the serial path, partition the data, measure — then parallelism pays off.

🔧 Run it and poke at it — on your home machine
What to play is above (🕹). Here — measuring parallelism.
🔧 Tinker (Tracy / a profiler) ~40 min, Bevy or an engine
Take a Bevy example (or your own engine) with Tracy: put ZoneScoped in a couple of systems, run it, find a frame >16 ms and drill into the worst zone. Find the sync points (all workers idle) — that is the serial fraction. Work out via Amdahl what speedup your current s gives on 8 cores, and what happens if the serial chunk goes away.
🧪 Test (diagnosis) ~15 min
For a stuttering scene: CPU-bound or GPU-bound (the "half resolution" test)? If CPU — is it jobs, gameplay or draw calls? Then find shared mutable state between threads in your own code — a potential race; how do you remove the sharing (partition it) or the mutability (immutability)?
Checklist: instrumented zones in Tracy and found the worst one; found a sync point / the serial fraction; computed Amdahl for your own s and N; identified CPU/GPU-bound; found shared state and a way to remove the race.
Connections
foundation
The game loop — the frame budget (16.67 ms) and the fixed step that all the parallel work has to fit inside.
foundation
ECS and data-oriented design — partitioning data by component makes parallelism safe (no shared writes → no races).
adjacent
The render pipeline — GPU parallelism (warps/occupancy); here it is the CPU parallelism that feeds it commands.
next
Audio and DSP — the audio mix also runs on its own thread under a hard buffer deadline.
Questions worth asking
Why is single-threaded gameplay "the main bottleneck" rather than a slow shader or physics?
Because it leaves most of the machine idle and becomes the serial isthmus (Amdahl's fraction s) for the whole frame. An engine can parallelize rendering, physics and culling across 16 cores, but if gameplay logic runs on one thread and everything waits on it at a sync point, the frame is bound by that — 15 cores idle. A slow shader hits the GPU budget locally; single-threaded gameplay caps the whole frame per Amdahl and does not scale with hardware. So it is a systemic rather than a local problem — and why engines push so hard on data-oriented gameplay that parallelizes.
What actually limits the speedup from adding cores?
The serial fraction s (Amdahl's law): part of the work inherently does not parallelize — sync points, dependencies, critical sections, a single main thread. The parallel part shrinks as N grows while the serial part does not, so 1/s is the ceiling in the limit: 10% serial → at most 10× no matter how many cores. Plus overhead: synchronization (threads waiting = serialization), lock contention, false sharing, task imbalance. Hence the engineering priority is to cut the serial fraction and the coordination rather than pile on cores: removing 1% of serial work is often worth more than doubling them.
Why are data races the "worst" bugs, and how does ECS avoid them?
Because they are non-deterministic: they depend on the exact timing of threads, so they reproduce rarely, vanish under a debugger (which changes the timing — a heisenbug) and behave differently on different machines. Ordinary debugging is powerless. The root is shared mutable state: two threads writing the same thing. ECS removes the sharing by construction: data is laid out by component and systems declare which components they read and write — the scheduler sees that two systems touch non-overlapping data and parallelizes them safely, serializing the overlapping ones. No shared writes → no races, and that is a guarantee of the structure rather than of programmer discipline (the same idea as Rust's borrow checker).
Why work stealing, if you can just hand each worker 1/N of the tasks?
Because a static "1/N" breaks when tasks are unequal in length: updating one heavy boss takes longer than hundreds of simple particles. Split evenly by count and one worker grinds away while three finished long ago and idle (imbalance). Work stealing balances dynamically: finish your queue and steal a task from the tail of someone else's. Everyone takes their own from the head (cache-local) and steals from the tail (less conflict with the owner), with no central scheduler bottleneck. That gets close to optimal utilization without knowing durations in advance — which is why it is the standard (Cilk, TBB, Rayon, Unity/Bevy). Balancing by fact, not by plan.
CPU-bound or GPU-bound — why is that the first question in optimization?
Because it determines where to look at all and stops you optimizing the wrong thing. If you are GPU-bound (stuck on fill/shaders/geometry), no amount of speeding up the job system will move the FPS — the GPU is still the chokepoint. And vice versa. The 30-second test: halve the resolution — a big FPS jump means GPU-bound (fewer pixels = less GPU work), barely any change means CPU-bound (pixels are irrelevant, the bottleneck is on the CPU: jobs, gameplay, draw calls). Only then do you pick the tool — a CPU profiler (Tracy) or a GPU one (RenderDoc/Nsight). Optimizing without knowing what you are bound by is fixing the wrong end.
Further reading