Cloud GPUs are the right answer for some AI workloads — and a genuinely expensive mistake for others. The decision comes down to utilization. If your GPUs run more than 40% of the year, owned hardware in colocation beats cloud on total cost of ownership by 40–60%. Below 20% utilization, cloud flexibility wins. The hard part is knowing which bucket you're actually in.
Why Cloud GPUs Feel Cheaper Than They Are
The hourly rate is visible. The full cost isn't.
An H100 on AWS (p4de.24xlarge or p5.48xlarge) runs $30–$98/hour depending on instance type and reservation level. That sounds manageable until you're three months into a training run and you do the math. Thirty dollars an hour is $720/day. A sustained training workload running 80% of the time costs you over $17,000/month — per node.
Cloud providers have also gotten good at making egress costs invisible until they're not. You pull your training data in, you push checkpoints out, you sync results to your object store, and suddenly you've got a $4,000 line item you didn't model. It's not malicious. It's just how the pricing works, and it's easy to miss when you're focused on getting your training job running.
The other hidden cost is time. Cloud GPU availability isn't guaranteed. If you've ever tried to spin up a p5 cluster on short notice, you know what I mean. Reserved capacity helps, but reservations are 1- or 3-year commitments — at which point you've already made the financial case for owning hardware.
When Cloud Is Actually the Right Call
I want to be clear: cloud GPUs aren't always wrong. There are workloads where they're the correct answer.
Proof of concept and early research. If you're not sure whether a model architecture is going to work, you don't want to own the hardware. Spin up a cloud instance, run your experiments, kill it when you're done. The flexibility premium is worth it when you're still figuring out what you're building.
Irregular training bursts. Some teams train quarterly — a major model refresh every few months, with relatively light GPU usage in between. If your cluster would sit idle 70% of the year, cloud spot instances (AWS Spot, GCP Spot VMs) can cut your cost significantly. You accept interruption risk, but for non-time-critical training jobs, that's often acceptable.
Geographic or compliance flexibility. If you need to spin up capacity in a specific region for a short-term project, cloud gives you that without a facility contract.
Inference at variable scale. Early-stage inference serving with unpredictable traffic is a legitimate cloud use case. You can autoscale, you don't commit to hardware, and you pay for what you use. Once your inference traffic stabilizes and you can predict it, the calculus changes.
Where Colocation Wins: The TCO Math
Here's a concrete scenario. A team running continuous fine-tuning and inference for a production LLM application. They need four H100 SXM5 nodes — roughly a 320-GPU cluster at 40kW total draw.
| Cost Component | AWS (p5.48xlarge, 1-yr reserved) | GPU Colo (owned hardware) |
|---|---|---|
| Compute/instance cost | ~$22/hr per node, ~$770K/yr for 4 nodes | $0 (owned) |
| Hardware amortization (3yr) | — | ~$180K/yr (4x DGX H100) |
| Power (40kW at $250/kW/mo) | Included in instance price | ~$120K/yr |
| Network/egress | $0.09/GB, variable | Flat, included |
| Total (Year 1) | ~$770K | ~$300K |
| Total (Year 3) | ~$2.31M | ~$660K |
That's not a rounding error. Over three years, the owned-hardware-in-colo scenario saves roughly $1.65M on this configuration. The hardware is fully amortized, and you're into year four at $120K/year in infrastructure cost.
The numbers above use IDACORE East's $250/kW/month all-in pricing. That rate includes power, cooling, connectivity, and facility — no line-item surprises.
What Makes GPU Colocation Actually Work
The financial case is clear. The operational case is what trips people up.
Standard colocation wasn't designed for GPU density. A traditional 2kW-per-cabinet facility can't handle a DGX H100, which draws 10.2kW by itself. You need a facility that was engineered for this from the start — not one that retrofitted a few high-density cabinets into a legacy air-cooled build.
The cooling architecture matters more than almost anything else. Air cooling tops out around 30–40kW per cabinet in practice. Direct-to-chip liquid cooling handles 120kW per cabinet without thermal throttling, which is what dense GPU configurations require to run at full utilization. If your GPUs are thermally throttling because the facility can't keep up, you're not getting the compute you paid for.
Power redundancy is the other one. GPU training jobs are long-running. A 72-hour training run interrupted by a power event doesn't just lose an hour — it loses the entire run if you haven't checkpointed aggressively. True 2N power (independent grid source plus gas generation, not generator backup) means both sources can carry full load independently. That's a different architecture than "we have a generator."
Network is less often the bottleneck for training, but it matters for inference. Multi-carrier connectivity with diverse fiber routes means you're not dependent on a single provider's maintenance window.
The Hybrid Pattern That Actually Works
The teams I've seen get this right don't treat it as binary. They split the workload.
Inference serving — the stuff that runs 24/7 once a model ships — goes on owned hardware in colocation. Predictable utilization, predictable cost, no egress surprises. Training and fine-tuning, especially experimental work, uses cloud spot instances. You accept the interruption risk in exchange for not paying for idle hardware.
This pattern also gives you a natural migration path. Start in cloud, validate your architecture, understand your actual utilization patterns. Once you know what you need, move the steady-state workload to colo and keep cloud for overflow and experiments.
The mistake is running everything in cloud indefinitely because it was easier to start there. That's not a technical decision at that point — it's inertia.
Frequently Asked Questions
Is GPU colocation cheaper than cloud for AI training?
For sustained workloads running more than a few weeks per quarter, GPU colocation is typically 40–60% cheaper than cloud. An H100 in AWS costs $30–$98/hour depending on instance type. Colocating your own H100 server at a facility like IDACORE East runs closer to $2–$4/hour in all-in infrastructure cost once you amortize hardware. The break-even point for most teams is around 30–40% utilization annually.
What GPU utilization rate justifies buying hardware over renting cloud?
If your GPU cluster runs above 40% utilization averaged across the year, owned hardware in colocation almost always wins on TCO. Below 20%, cloud flexibility is hard to beat. Between 20–40%, it depends on your team's operational capacity and whether your workload is predictable enough to plan around. Spiky, unpredictable inference traffic often stays in cloud longer than training workloads.
How much power does an H100 server require in colocation?
A standard 8x H100 SXM5 server (DGX H100) draws approximately 10.2kW at full load. A four-node cluster is roughly 40kW. Most traditional colocation facilities cap cabinets at 10–20kW, which is why high-density AI colocation matters. IDACORE East supports up to 120kW per cabinet with direct-to-chip liquid cooling, which is what dense GPU configurations actually require.
Can I use colocation for AI inference as well as training?
Yes, and inference is often the better long-term case for colocation. Training runs are finite — you can time-bound them and use cloud spot instances. Inference serving runs continuously once a model is in production, which means sustained GPU utilization and predictable cost. Owning the inference infrastructure in colocation and using cloud for training bursts is a common and cost-effective hybrid pattern.
What should I look for in a GPU colocation facility?
Four things matter most: power density per cabinet (you need at least 40–80kW for serious GPU work, ideally 100kW+), cooling type (liquid cooling handles dense GPU configs that air cooling can't), network (low-latency, multi-carrier with diverse fiber paths), and operational support. A facility that treats your rack as a ticket number will cost you more in downtime than you saved on rent.
If you're running sustained AI workloads and your cloud bill is climbing faster than your model accuracy, it's worth running the actual numbers. IDACORE East was built specifically for this — 120kW per cabinet, true 2N power, $250/kW/month all-in, with dark fiber interconnects to our Boise facility for teams that need both AI compute and general infrastructure under one operator. We're pre-leasing Phase 1 now. Talk to us about your GPU infrastructure requirements before you sign another cloud reservation.
