Infrastructure · Intermediate
Run a local AI lab that never stops working
Why some workloads belong on hardware you own, and how to set up a machine that runs jobs continuously without a per-token bill.
Renting frontier intelligence for every task is the right default and the wrong extreme. Some workloads run continuously, tolerate a small quality drop, and would bankrupt you on per-token pricing.
This playbook covers setting up hardware you own to absorb that class of work, with a routing rule that keeps the decision honest.
The playbook
01 Identify the always-on work
Look for jobs that are high-volume, latency-tolerant, and not commercially sensitive to run slightly worse: bulk classification, transcription backlogs, document pre-processing, overnight enrichment.
These are the workloads where a frontier model's per-token price dominates and its quality advantage does not.
02 Size the machine to the workload
Match VRAM to the largest model you actually need in the pipeline, then add headroom for the context length you will use in practice.
Buy for the model class you need, not the one you might want. Requirements move faster than depreciation.
03 Set up the serving layer
Run an OpenAI-compatible server so your application code does not care whether it is talking to local hardware or a hosted API.
That compatibility is the whole point: it lets you move a workload between local and hosted based on cost and quality without touching application code.
04 Build the queue
Put a durable job queue in front of it. The machine should always have work, and a failed job should retry rather than disappear.
Log per-job cost in both directions (local compute time and the hosted price you avoided), so the payback period is measurable rather than assumed.
05 Decide the routing rule
Write an explicit rule for what runs locally and what escalates to a frontier model. Keep it in code, not in someone's head.
Review the rule monthly. Hosted prices move, and the boundary moves with them.
Start before the gap gets expensive.
Tell us where the work is stuck. We will tell you, in plain language, what AI can and cannot fix, and what it would take to do it properly.