Most teams buy their first AI machine for the wrong number. Speed matters, but memory decides what you can run at all.
Start with the model, not the card
A rough rule: a model needs about 2 bytes per parameter at 16 bit precision, and about half a byte at 4 bit. So a 7B model fits easily almost anywhere, while a 70B model needs around 40 GB even when compressed to 4 bit, before you leave room for long prompts.
Three sensible starting points
- GeForce RTX 5090, 32 GB. The cheapest serious option. Runs 7B to 30B models well, and small fine tunes. Draws up to 575 W, so plan the power supply.
- RTX PRO 6000 Blackwell, 96 GB. One card that holds a 70B model in memory with room to spare. ECC memory, built to run all day.
- DGX Spark, 128 GB unified memory. A small desktop machine for prototyping and fine tuning models up to around 200B parameters, quietly, on a desk.
Buy for the largest model you will actually run in the next year, not the one in the headline.
Do not forget the boring parts
- A UPS, so a power cut does not lose a twelve hour run. See the Smart-UPS SRT.
- Fast storage for model weights and datasets, such as the 990 PRO.
- Airflow. A 600 W card in a small cabinet will slow itself down.
Not sure which fits? Talk to an engineer, or browse compute.