Cattle, Not Pets: Spreading GPU Inference Across Small Depots in 2026

November 16, 2026 · 11 MIN READ

Cattle, not pets means treating every GPU site as replaceable. IDACORE Depots apply it to inference: small, roughly 1MW air-cooled sites on power the utility already has, so losing one shrinks capacity instead of causing an outage. Depot 01 in Northeast Washington is live now: up to 50kW per rack, powered in under two weeks, 99.9% uptime SLA.

What Does "Cattle, Not Pets" Mean for GPU Infrastructure?

Most of us learned this lesson with servers a decade ago. Pets have names. You nurse them back to health at 2 a.m. Cattle have numbers. When one gets sick, you replace it and the herd keeps moving. We rebuilt our software around that idea with autoscaling groups, immutable images, and health-checked load balancers.

Facilities never got the memo. Ask the industry how to make AI infrastructure reliable and the answer is still a bigger, more redundant building: hundreds of megawatts, 2N everything, a new substation, and a multi-year timeline. That's a pet. A very expensive pet the size of a warehouse district.

Here's the thing: inference isn't training. Training wants thousands of GPUs on one tightly coupled fabric in one place, and a site failure mid-run hurts. Inference is request-scoped. A request lands on one node, the model weights are read-only, and the state lives and dies with the request. Inference doesn't care which building it runs in, as long as your latency budget holds and something answers.

So make the building the unit of failure. That's the IDACORE Depot model. A Depot is a small, purpose-built compute site: 5MW or less, typically about 1MW of IT load, built from two roughly 500kW Compute Modules. We scale out across many Depots instead of building one giant campus. No single site is gold-plated. The platform is resilient as a whole, and you pay for the resilience you need.

Why Do Small Depots Energize Faster Than Big Campuses?

Power is the bottleneck for GPU deployments in 2026. Not chips. Not buildings. Power. Large campuses sit in interconnection queues and wait on substation projects measured in years.

A Depot skips that line. We site Depots where the utility already has power on a primary feed, set our own transformer, and deliver 415V to the racks. A load around 1MW fits on existing distribution in places a 200MW campus never could. That's why Depot 01 in Northeast Washington powers racks in under two weeks from signing. You aren't waiting on a utility capital project. You're waiting on us to land your gear.

The building side is just as lean. Each Compute Module runs cross-flow free-air cooling: outside air comes in through louvers and MERV 13 filtration, crosses the servers, and exits the far side. A direct evaporative wet wall handles summer. No compressors. No chillers. Louvers, recirculation, and the wet wall are fully automated. Design-target PUE is about 1.1–1.2, and because you pay for power at cost, that efficiency lands directly on your bill.

Inside, you get 48-inch-deep cabinets with A+B PDUs (C13/C19) and air-cooled capacity up to 50kW per rack. That's room for three 8-GPU HGX B200-class servers in a single cabinet. H100-class and RTX PRO 6000-class fleets fit comfortably too.

Why Do Small Sites Win Community Support?

Small towns have watched hyperscale projects show up asking for tax abatements and grid upgrades the whole service territory pays for. Depots don't work that way. Small footprint. No new substation. No tax-break asks. We bring community contributions, training, and events, and we create local jobs and modern infrastructure without straining the grid. We already host IdahoMesh MeshCore repeaters, because being a good neighbor is part of how we operate.

That matters to you as a customer. A site the community wants is a site that gets built and stays built.

What Does a Local Operator Change?

We've run infrastructure for 30 years, including founding an ISP with our own ASN and peering at the Seattle Internet Exchange. When you call, you reach people who know the site and your rack. Decisions don't climb four layers of account management. If you need a cabinet reconfigured or a handoff upgraded, we sort it out quickly, because we're the operator.

How Do You Pick the Resilience You Need?

This is where cattle thinking pays off. The base Depot offering is utility-fed: no UPS, no generator. It's the lowest-cost tier, and for a well-architected inference fleet it's often the right call. You don't need every site to survive every event. You need the fleet to.

When a single site does need to ride through utility events, you add it. Nothing is bundled in that you don't use.

Resilience option What it covers Best fit
Utility-fed (base) Lowest-cost tier, 99.9% uptime SLA Inference fleets spread across sites, batch and async jobs, marketplace capacity
UPS + generator (add-on) Site rides through utility outages Single-site deployments with stateful or latency-pinned workloads
Battery energy storage, BESS (add-on) Battery-backed ride-through at the site, duration set by sizing Workloads that need to bridge utility events at one site
Multiple Depots Loss of an entire site for any reason: utility, network, weather Production inference with a load balancer that drains failed sites

Notice the last row. A UPS protects you from a utility blip. It doesn't protect you from a fiber cut, a cooling fault, or a regional storm. Spreading across sites covers all of them. That's the "pick the resilience you need" approach. It isn't a lesser product. It's paying for redundancy where it does the most work.

How Should You Architect Inference Across Multiple Sites?

Start with what's real today. Depot 01 in Northeast Washington is live. The next Depots are planned for the Treasure Valley and Eastern Oregon, connected back to our Boise interconnect hub over metro fiber. Dark fiber directly between Depots is a long-term plan, not something that exists now, so design for sites that reach each other over IP transit.

IDACORE IP transit runs on our own network, AS36824, with BGP, IPv4/IPv6, and 1G/10G/100G handoffs on burstable 95th-percentile billing. If you hold your own address space, announce it. Your routing stays under your control.

The rules I'd follow:

  • Pre-stage weights at every site. Never pull a 400GB checkpoint from a central bucket during a failover. Every site boots from local storage.
  • Keep cross-site state out of the request path. KV cache stays local. Session affinity is fine, as long as a lost session degrades to a retry and not an error.
  • Health-check the model, not the port. A listening socket says nothing about whether the GPUs loaded the weights.
  • Size N+1 at the site level. With three sites, losing one leaves two to carry peak. Each site needs headroom for 50% of peak traffic, not 33%.
  • Stop paging humans for single nodes. Drain it, power-cycle it yourself from the IDACORE portal, and file a remote hands ticket during business hours if hardware needs touching. Remote hands is $150/hr in business hours, billed in 15-minute increments, and $250/hr after hours with a one-hour minimum. Cattle math says most failures can wait for morning.

Here's a minimal HAProxy backend that spans two sites and drains a failed node or site in about six seconds:

backend inference
    balance leastconn
    option httpchk GET /ready
    http-check expect status 200
    default-server inter 2s fall 3 rise 5 slowstart 30s
    # Depot site A
    server siteA-gpu01 203.0.113.10:8000 check
    server siteA-gpu02 203.0.113.11:8000 check
    # Site B
    server siteB-gpu01 198.51.100.20:8000 check
    server siteB-gpu02 198.51.100.21:8000 check

The key detail is /ready. Have it return 200 only after weights are loaded and a warmup inference completes. The slowstart keeps a recovering node from drowning under a full share of traffic the instant it comes back. Run the gateway itself in at least two places, either behind anycast or DNS with short TTLs, so the load balancer doesn't become the new pet.

What Does a Depot Rack Cost to Run?

Let's run a real scenario. A GPU marketplace host deploys one 40kW rack at Depot 01 on a 36-month term.

Facility (NNN): 40kW × $175/kW/month = $7,000/month. That covers space, cooling, and facility infrastructure.

Power, at cost: 40kW of metered IT load × 1.2 PUE (the conservative end of the design target) = 48kW at the meter. Over a 730-hour month, that's 35,040 kWh, billed at the utility rate with no markup. If your fleet averages 34kW instead of 40kW, the power line follows your meter down.

Separately billed: IP transit, any transport or waves, backup power if you add it, and remote hands.

So your total is $7,000 plus power at cost plus the network and services you choose. Every line is published and traceable. Compare that with hyperscaler GPU instances, where egress is billed per GB and grows with your success. A popular inference endpoint can double its egress bill in a month and you won't see it coming. Committed transit at the 95th percentile turns that into a number you can forecast.

Terms are flexible enough for a startup. There's no minimum deployment size, so one rack is fine. Deposit and first month are due at signing. 12- and 24-month terms are available at a higher rate. For current availability, check idacore.com/depots.

Frequently Asked Questions

Is IDACORE Depot 01 available for GPU colocation right now?
Yes. Depot 01 in Northeast Washington is live and energized. It offers air-cooled capacity up to 50kW per rack, enough for three 8-GPU HGX B200-class servers, with racks powered in under two weeks from signing and a 99.9% uptime SLA. IP transit from IDACORE's own network, AS36824, is delivered on-site with 1G, 10G, and 100G handoffs. For current rack availability, check idacore.com/depots rather than relying on any published figure.

How much does GPU colocation cost at an IDACORE Depot?
Depot pricing is $175/kW/month NNN on a 36-month term, covering space, cooling, and facility infrastructure. Power bills on top at cost: metered IT load times PUE at the utility rate, with no markup. Shorter 12- and 24-month terms are available at a higher rate. There's no minimum deployment size. Deposit and first month are due at signing. Backup power, IP transit, transport, and remote hands are billed separately. Remote hands runs $150/hr in business hours and $250/hr after hours.

Can an air-cooled rack handle 8-GPU B200 servers?
At an IDACORE Depot, yes. Depot racks support up to 50kW each on air cooling, which leaves room for three 8-GPU HGX B200-class servers. The cooling is cross-flow free air: outside air through louvers and MERV 13 filtration, across the servers, and out the far side, with a direct evaporative wet wall for summer and no compressors or chillers. Cabinets are 48 inches deep with A+B PDUs (C13/C19), fed at 415V from IDACORE's own transformer.

What happens to my inference workload if a Depot loses utility power?
On the base utility-fed tier, there's no UPS or generator, so a utility outage takes that site's racks down. That's the lowest-cost option and the right fit when your fleet spans multiple sites and your load balancer drains a failed site in seconds. If you need ride-through at a single site, UPS plus generator or battery energy storage (BESS) is available as an add-on. Pick the resilience your workload needs instead of paying for gold-plating it doesn't.

Where will IDACORE build its next GPU Depots?
The next Depots are planned for the Treasure Valley and Eastern Oregon, connected back to IDACORE's Boise interconnect hub over metro fiber. Each follows the same standard: roughly 500kW Compute Modules, two per ~1MW Depot, sited where the utility already has power. Dark fiber directly between Depots is a long-term plan, not something that exists today, so cross-site traffic runs over IP transit. Depot 01 in Northeast Washington is the live site you can deploy in now.

If your inference roadmap is stuck waiting on power, don't wait on a campus that's years from energizing. Depot 01 puts your GPUs on 50kW air-cooled racks in Northeast Washington in under two weeks, at $175/kW/month NNN plus power at cost. Reserve your Depot 01 rack and get your GPUs racked this month.

Tags

GPU inference colocationmodular GPU colocationIDACORE DepotsNortheast Washington GPU colocationAI inference site redundancy
IDACORE

IDACORE

Infrastructure Architect

Specializes in AI/HPC infrastructure design, power density planning, and liquid cooling deployments.

Ready to Implement These Strategies?

Our team of experts can help you apply these gpu & hpc colocation techniques to your infrastructure. Contact us for personalized guidance and support.

Get Expert Help