Infrastructure for AI.

Why AI infrastructure is its own discipline

Classic enterprise infrastructure optimizes for isolation: thousands of independent workloads, each protected from its neighbors. AI training is the opposite — one job, thousands of accelerators, moving in lockstep. Every optimizer step, every GPU exchanges gradients with the others; the whole machine advances at the pace of its slowest participant. That single fact drives everything else on this page.

Anatomy of a GPU node

A modern training node is a small datacenter in a box: eight accelerators, a private interconnect between them, hundreds of gigabytes of HBM, multiple RDMA NICs, and local NVMe. The hierarchy of bandwidths is the whole story — each hop away from the GPU die costs roughly an order of magnitude:

The three networks (never mix them)

A serious training cluster runs three physically or logically separate networks — the same discipline as a SAN shop that never let backup traffic touch the production fabric:

The vocabulary that matters: collectives

Distributed training speaks NCCL. Three verbs cover most of it: all-reduce (every GPU ends with the sum of everyone's gradients — the heartbeat of data parallelism), all-gather (everyone ends with everyone's shard — how ZeRO/FSDP reassembles weights), and reduce-scatter (the sum, but sharded — the other half of FSDP). Ring and tree algorithms trade latency against bandwidth; the fabric's job is to make either work at full line rate on every rail simultaneously.

InfiniBand vs RoCE — the decision, not the religion

RoCE tuning checklist — the shortlist that separates working clusters from mystery stalls: enable and verify PFC on the RDMA traffic class end-to-end (one unmarked hop poisons the path); configure ECN/DCQCN thresholds per vendor guidance and confirm marking actually occurs under load; isolate RDMA in its own queue; watch for PFC storms and pause propagation — the RoCE failure mode that looks exactly like a hung filesystem; and validate ECMP hashing spreads flows across rails rather than pinning elephants to one link. Then run nccl-tests (all_reduce_perf) at every scale step and keep the numbers — the baseline is your future 2 a.m. evidence.

Storage for AI — the layer that starves clusters quietly

Training storage has two personalities, and sizing for the wrong one is the classic mistake. The read side streams the dataset: high aggregate sequential bandwidth, but with the small-file trap — vision and multimodal datasets can be millions of tiny files, which murders filesystems that were benchmarked on large sequential reads. The write side is checkpointing: near-idle for an hour, then every node writing gigabytes at once in a burst that must finish fast, because (unless checkpoints are asynchronous) the GPUs wait.

GPUDirect Storage (GDS) extends the RDMA idea to disk: NVMe/NVMe-oF reads DMA straight into GPU memory, skipping the CPU bounce buffer. Where it applies, it lifts throughput and frees CPU cycles for the data loader — which is often the actual bottleneck, spending its life decoding and augmenting samples.

Streaming datasets without starving GPUs (or the budget)

Checkpoint lifecycle mirrors backup tiering exactly: hot — latest N checkpoints on the parallel filesystem or node-local NVMe for instant restart; warm — recent history in object storage for rollback and evaluation forks; cold — milestone checkpoints archived (and the rest deleted — a 70B run checkpointing hourly writes ~24 TB/day, and someone has to say what survives). Retention policy is a training-infra decision, not an afterthought; you have written this exact policy before, it was called backup rotation.

Checkpointing — DR discipline at training speed

A checkpoint is a point-in-time copy of the training state: weights, optimizer state, and bookkeeping. If that sounds like replication and RPO thinking, it is — the same math wearing a lab coat. The state is large: in mixed-precision Adam training, weights + gradients + optimizer state commonly land around ~16 bytes per parameter of live memory, and the persisted checkpoint (weights + optimizer) around ~12–14 bytes per parameter before any sharded-format overhead. A 70B-parameter run checkpoints roughly a terabyte. Every interval.

Two levers change the math entirely: asynchronous checkpointing (snapshot to host/local memory, persist in the background — the GPU stall shrinks from the full write time to the snapshot time) and differential/sharded formats (each data-parallel rank persists only its shard; distributed checkpoint formats reassemble on restore). Profile before optimizing: time-to-safe vs time-to-stall-end, per-rank write throughput, and metadata ops/sec tell you which of the three walls — bandwidth, metadata, or serialization — you're actually hitting.

Design questions, straight from DR practice: how much progress can you afford to lose (checkpoint interval = RPO)? How fast must the write finish (synchronous checkpoint stall = RTO cousin)? Where do copies land (local NVMe → async to PFS → archive to object = tiered replication)? The calculator below turns those into required bandwidth.

Slurm vs Kubernetes — two cultures, one cluster

The Kubernetes GPU story just changed: DRA

Dynamic Resource Allocation graduated to GA in Kubernetes 1.34 (September 2025) and replaces the count-only device-plugin model ("nvidia.com/gpu: 1") with attribute-aware claims — request "a GPU with ≥80 GB and NVLink" and let the scheduler find it. OpenShift 4.21 ships the same GA framework, which matters if your platform team lives there. Two operational realities survive the upgrade: MIG fragmentation — carving GPUs into slices strands capacity when slice-shaped holes don't match slice-shaped requests, exactly like thin-pool fragmentation on an array — and gang scheduling, which vanilla K8s still delegates to Kueue/Volcano while Slurm has it natively: an all-or-nothing job that gets 7 of 8 nodes deadlocks politely forever.

Inference infrastructure — a memory business

Training is a compute problem; serving is a memory-bandwidth and memory-capacity problem. Generating each token requires streaming the model weights through the GPU, plus reading and appending the KV cache — the attention keys/values for every token already in the context. The KV cache is why long contexts and big batches collide: it grows linearly with tokens × layers × heads, eating the HBM you wanted for batch size.

Choosing a serving engine (mid-2026 map)

Power & cooling — the constraint that outranks the rest

The numbers moved fast: an H100 SXM module is rated 700 W; an 8-GPU DGX H100 system is ~10.2 kW; a GB200 NVL72 rack is on the order of ~120 kW and is liquid-cooled by design. Traditional datacenter rows were built for 5–15 kW racks on air. That gap — not chip supply, not budget — is what actually limits many AI buildouts.

Architecture as arithmetic

Four calculators, all client-side, each encoding a rule of thumb the field actually uses — with its assumptions printed, because a number you can't defend is worse than no number.

For storage engineers making this move

The premise of this whole site: your instincts transfer better than the job boards suggest. The mapping, concretely:

The shortlist