Feels AI-generated, so much text real estate is spent explaining the basics in sufficient detail then just two paragraphs on their scheduler with a lot of hand-waving.

This blog's goal is to introduce the problem around the new NVL72 system with GPU failures in production, which not all people outside of xAI are aware of. We have more blogs coming up for what we built to handle topology-aware scheduling, defragmentation and hot swapping during failures. The short version of the scheduling piece: jobs are decomposed into 16-node slices placed atomically on a single rack, with two spares per rack reserved for hot swaps.

Seems very relevant to what you are building. Quoting what Ali said: "Quick PoC?"

Why do you target millisecond-level cycle time? Are you looking at very fine-grained tasks like elastic sandboxes, in addition to beefy pre- and post-training jobs?

> O(millions)

Big-O doesn't include constant factors for a reason (never mind just being a constant entirely), but maybe I am misunderstanding what you are trying to say?

> making it more of a fluff/marketing piece than an actually interesting technical blog in this area.

/genuine do you recommend any other blog?

open source the scheduler

Yes, SkyPilot Platform built the scheduler on top of Kueue. There is some shortcoming of Kueue's TAS that you have to resolve, e.g., you cannot do TAS with a Elastic job, which makes the hot swapping not possible for large-scale jobs on NVL72 racks: https://kueue.sigs.k8s.io/docs/concepts/elastic_workload/#li...

"No Topology-Aware Scheduling (TAS) support."

There is another blog talking about this coming soon.

The danger of being excited about old techniques is giving the impression that you are years behind everyone else.

A coworker once said of a particular breed of manager who is an emphatic late adopter: it’s as if every time he goes to the dentist he reads an old trade journal in the waiting room and comes back to work with a “new” idea that everyone else is already busy replacing.

Are there any actual newer techniques? For instance, putting 11% of your hardware on reserve for hardware failures seems a bit generous to me for such an expensive cluster. Even if it’s down from some much larger number. One solution to this I’ve seen for handling cpu and memory variability with bin packing is to pool the spare capacity between tasks. For instance a data center with four zones might have 7 backup generators, instead of eight or twelve, counting on clusters of generator failures not to happen frequently.

And of course there’s always running low priority tasks on the spare hardware and evicting them on a fault. However those low priority tasks tend to develop their own shadow priorities when they are in house. But that’s a management problem not a technical one.