Tech
SkyPilot's Grace Blackwell scheduler lifts GR00T N1.7 pre-training 14.4%, models 27% more usable nodes on 2,304-node clusters.
skypilot.ai · Rohan Sonecha · 2026-10-29 · Source ↗
SkyPilot's Grace Blackwell scheduler lifts GR00T N1.7 pre-training 14.4%, models 27% more usable nodes on 2,304-node clusters.
skypilot.ai · Rohan Sonecha · 2026-10-29 · Source ↗
Rohan Sonecha, Member of Technical Staff at SkyPilot, published a 1,000-word post on Thursday arguing that the GB200 and GB300 NVL72 generation needs a scheduler built around the rack. Each NVL72 rack binds 72 GPUs behind a single NVLink fabric with roughly an order of magnitude more bandwidth than InfiniBand, so a job whose workers all land in one rack runs much faster than one that spans several. SkyPilot's Topology-Aware Scheduler, TAS, enforces that rule. Single-rack jobs land on a single rack; multi-rack jobs cut into 16-node slices placed on dedicated racks, with two spare nodes per rack reserved for in-rack swaps when GPU Manager detects a failure. Frontier Trainer either moves a failed worker to a spare in the same rack or, if none are left, relocates the slice to a healthy rack.
The 14.4 percent GR00T measurement, and the 27 percent capacity model
The only measured throughput number in the article comes from NVIDIA's GR00T team. They pre-trained GR00T N1.7 on 512 GB300 GPUs across 128 nodes, comparing a naive scheduler that spread nodes across arbitrary racks with the rack-aware policy that used exactly 8 racks. Rack-aware placement delivered 14.4 percent higher throughput, cutting iteration time from 1.59 seconds to 1.39 seconds. Sonecha extrapolates that gap to roughly $16,700 per training hour of savings on a 10,000-GPU cluster. The 27 percent figure in the headline is a modeled capacity comparison. Applying NVIDIA's model of a 128-rack (2,304-node) cluster to the roughly 2 percent node unavailability SkyPilot reports on production NVL72 clusters, 16-node slices yield 2,037 usable nodes against 1,602 for 18-node slices, a 27 percent gain. The article also measures 16-node slices beating 9-node slices by 4 percent.
"NVL72 puts all 72 GPUs in a rack on one NVLink fabric, but only rack-local jobs get the speedup." — Rohan Sonecha, Member of Technical Staff, SkyPilot, 2026-10-29
Automatic Defragmentation and interactive workloads
When blocked jobs cannot find a contiguous rack, Automatic Defragmentation preempts workloads that are safe to restart, such as managed jobs that resume from a checkpoint, in order to free a complete rack. The scheduler leaves interactive environments alone unless an admin opts them in, which keeps developer sessions untouched by background rearrangement. Sonecha's second quoted line, "sometimes less is more: using less nodes in each rack significantly improves reliability," is the summary justification for the 16-node slice choice over the natural 18 that would fill each rack completely.