A rate card is not a price
The published hourly rate is the smallest term in the cost of running a campaign somewhere new. Four other terms decide whether the move was worth making, and three of them are routinely omitted from the comparison.
Ask a compute buyer what an H100-hour costs and you will get a rate card number. Ask what a campaign costs and the honest answer takes longer, because the rate is multiplied by a duration that depends on the machine, and then added to a set of costs that the rate card does not mention.
We call the resulting figure delivered cost: all-in dollars per chip-hour, including everything you will actually be invoiced for and everything you will actually waste. It is the only basis on which two suppliers can be compared, and constructing it is most of the analytical work in a dispatch audit.
The four terms that are not on the rate card
Egress and transfer time. This is the one that most often reverses a decision — but it has to be priced at your rate, and this is where the standard version of the argument cheats.
Published internet egress runs near $0.09/GB, which puts an 18 TB corpus at roughly $1,600. Almost no enterprise at this scale pays that; over dedicated interconnect the rate is typically $0.02–0.05, which puts the same corpus at $360–900.
Two corrections to the version of this argument we published first. The 2024 regulatory changes do not help here: those waivers apply to customers exiting a provider and closing the account, are application-gated, and explicitly exclude ongoing operational transfer. Continuous multi-supplier dispatch is the definitional opposite of migrating away, so modelling zero egress on that basis is wrong and we have removed it.
The second correction is about scale, and it undercuts our own headline. A 512-GPU week at $2.50/GPU-hr is roughly $215,000. Against that, a one-off $360–1,600 corpus move is 0.2–0.7% — it reverses nothing. Choosing the corpus as the example made the term look decisive while quietly proving it trivial.
The egress exposure that actually matters is recurring: checkpoints and artifacts, not the training set. A large model's checkpoint with optimizer state runs to multiple terabytes; written every thirty minutes across a week that is hundreds of writes, and any fraction of it crossing a provider boundary — replicated for durability, pulled to a second supplier for evaluation, or moved on re-dispatch — is measured in tens or hundreds of thousands of dollars, not hundreds. That is the term that reverses decisions, and it is why "keep the checkpoints where the compute is" is usually a stronger recommendation than "move to the cheaper supplier."
Transfer time survives even when the bytes are cheap. 18 TB is 144,000 gigabits; at an effective 2–4 Gbps that is 10–20 hours, and on a well-provisioned dedicated interconnect under two. During that window the destination is either idle and rented, or not yet rented and therefore not yet reserved. Either way it is a cost that appears on no invoice.
The practical consequence is that data gravity beats price more often than the arbitrage story suggests. "Don't move the data" has to be a first-class option the optimizer can select, and a supplier that only looks cheap because the dataset has been assumed into place is not actually cheap.
And checkpointing is not free either. Halving the checkpoint interval halves the work at risk from a preemption — and doubles a multi-terabyte synchronous write that stalls every rank and consumes storage bandwidth you are paying for. A rerun model that prices interruption without pricing the checkpointing that mitigates it implies more frequent checkpoints are costless, which would be an odd thing to believe. The real optimization trades the two against each other.
Expected rerun. Interruptible capacity is priced at a discount because it is interruptible, and the discount is only real if you finish. A preemption on a large distributed job does not cost the remaining minutes; it costs the work since the last checkpoint across every rank, plus the restart.
Take a 512-GPU job checkpointing every 30 minutes, preempted at the midpoint of an interval. Lost compute is 15 minutes × 512 GPUs = 128 GPU-hours. Restart is not free either: re-provisioning the gang, pulling images, initialising collectives and warming the dataloader is realistically 10–30 minutes of wall clock, another 85–256 GPU-hours. And in a capacity-constrained market the gang may not be immediately re-formable at all, so requeue latency can dominate both. A single preemption event on that job is therefore 213–384 GPU-hours, not the ~85 figure we previously quoted, which covered restart alone and understated the total by several times.
The two inputs that actually determine this are your checkpoint interval and the instrument's preemption rate. Any rerun estimate that does not state both is decoration.
Idle and fit. A chip-hour you rent and do not saturate costs the same as one you do, so instance shape versus job shape belongs in the comparison. We used to quote a widely-circulated single-digit "average GPU utilization" figure here and have removed it: that measurement is Kubernetes provisioning efficiency across general managed clusters — allocated versus requested — not streaming-multiprocessor occupancy on AI training fleets, where well-run fleets run far higher. Using it as a proxy for AI-fleet waste is the same category error we accuse price indices of making, and it would also argue for a product we don't sell: if your utilization genuinely is that low, bin-packing and all-or-nothing scheduling will return more than any supplier arbitrage.
Storage and the long tail. Checkpoint storage, retained artifacts, and the object-store operations behind them are small per unit and unbounded in duration. They are the reason a campaign that finished in March is still generating charges in September.
The fifth term: the machine is not the same machine
The four terms above are accounting. This one is physics, and it is the reason the energy analogy for compute has a genuine hole in it.
A megawatt-hour is identical regardless of which generator produced it. An H100-hour is not. Delivered throughput on the same nominal silicon varies with interconnect fabric — InfiniBand versus tuned Ethernet — with storage IOPS, with PCIe topology, with driver and collective-library versions, and with whoever else is on the box. Two 512-GPU clusters, one on NDR InfiniBand and one on RoCE, are not two prices for the same product. They are different machines that happen to contain the same accelerator.
For an embarrassingly parallel batch inference sweep this barely matters. For a communication-bound distributed train it can decide the trade. But the arithmetic has to be done properly, and the version of this argument we published first was wrong in a way worth showing.
Suppose a supplier is 20% cheaper per chip-hour on a fabric whose collectives run 35% slower. The tempting conclusion is that the campaign costs more. Work it through: if collective time rises 35% and collectives were all of wall clock, duration multiplies by 1.35 and cost by 0.80 × 1.35 = 1.08 — 8% worse, not the 15% we originally asserted. And collectives are never all of wall clock. On a real training job with overlapped communication they are typically 10–30% of step time, so by Amdahl a 35% collective regression is a 3.5–10.5% wall-clock regression, and 0.80 × 1.10 = 0.88. The cheaper, slower fabric wins comfortably.
That is the honest answer, and it cuts against the simple story. Fabric quality matters most when the communication fraction is high — large-scale synchronous training, small batch sizes, heavy all-reduce — and matters far less than the marketing suggests for the batch and fine-tune workloads that make up most of a work that can move. You cannot know which case you are in without measuring your own communication fraction.
Price per chip-hour is only meaningful once it is divided by the throughput that particular fabric delivers for that particular workload — and multiplied by how much of that workload is actually communication-bound.
Two defensible responses, and a serious system needs both. The first is constraint: express the requirement as a hard scheduling constraint so unqualified suppliers are ineligible rather than disfavored. The honest caveat is provenance — minimum bisection bandwidth is not a figure most providers publish or contractually guarantee, so for a supplier you have not run on, that constraint is an assertion until you have measured it once. The second is measurement: run nccl-tests all_reduce_perf across the allocation at provisioning, before the real job starts, and compare against the spec it was sold at.
Two limits on that remedy, since a probe is often oversold. It is not free — billing starts at instance start on every provider, so the acceptance test costs real billed minutes, cheap against a week on the wrong machine but not zero. And teardown is only a remedy on on-demand and spot: you cannot un-buy a reserved block or an assigned term contract, so on exactly the instruments where fabric misrepresentation is most expensive, the probe feeds a service-credit claim and the next qualification decision rather than a refund.
Putting it together
The delivered-cost calculation for a campaign is unglamorous:
- Quoted rate × expected duration at that supplier's measured throughput, not at nominal throughput
- Plus egress in and out, plus the transfer hours during which nothing is computing
- Plus checkpoint and artifact storage over the retention period
- Plus expected rerun — the interruption probability for that instrument, times the work at risk between checkpoints
- Plus the amortised vendor-management cost of that supplier: security review, data-processing agreement, subprocessor and export checks, and the ongoing management of one more counterparty
- Divided by useful chip-hours actually consumed, not chip-hours rented
Two consequences follow that are worth stating plainly, because they cut against the interests of anyone selling arbitrage.
First, the last term frequently kills the trade. Adding a supplier is not free. If the incremental saving is a few percent on a modest volume, the security review alone can exceed it. This is why a serious answer is not "use ten suppliers" but "qualify two or three well and route between them continuously."
Second, the honest number is much smaller than the headline. The published gap between hyperscaler and independent rates is roughly 2×. Against your negotiated rate it is more like 1.3–1.6× before any of the deductions above. After fit, egress, rerun and vendor overhead, a mid-to-high-teens percentage net saving on the movable portion of an fleet is a good result — and since the movable portion is rarely more than a third of total spend, that is a low-to-mid single-digit percentage of the whole bill. We treat 12% net on the work that can move as the threshold below which the exercise is not worth doing, and we publish the model so you can test it against your own numbers rather than ours.
A vendor quoting you materially more than that, measured against list price, has either not done this arithmetic or is hoping you won't.
Sources: published hyperscaler egress and interconnect pricing; distributed-training checkpoint and restart cost modelling; Novation modelling of a representative 18 TB campaign. Figures are illustrative of magnitude and will vary by fleet, region, contract and instrument. This paper was revised after buyer review: the fabric worked example and the preemption and egress figures were corrected, and a utilization statistic was withdrawn.