PRIVATE AI / HARDWARE & SIZING
Start with the task, model, and number of people it must serve. Hardware is the consequence of those choices, not the first slide in the conversation.
A PRACTICAL FIRST EXPERIMENT
Use an existing workstation or server to explore smaller quantized models, document workflows, and low-concurrency assistants. Memory capacity helps, but memory bandwidth and prompt length can make response times the limiting factor.
llama.cpp supports CPU execution and CPU/GPU combinations. Measure prefill time and token generation on the exact machine before making a user-facing latency commitment.
COMPACT, QUIET, LOCAL
Unified memory can support a useful local experimentation environment in a small physical footprint. Current Mac mini configurations reach 64GB unified memory; the OS, applications, weights, and context share that pool.
Evaluate compatible models with MLX LM or a Metal-enabled runtime. A 64GB machine is not equivalent to a discrete GPU with 64GB dedicated VRAM, and memory fit is not a speed guarantee.
Apple Mac mini specifications ↗ · MLX LM: inference and fine-tuning on Apple silicon ↗
A CAPABLE LOCAL WORKSTATION
A high-memory GPU can serve a quantized model with responsive local inference. For reference, NVIDIA specifies 32GB GDDR7 on the GeForce RTX 5090. A 27B model at four bits has a theoretical 13.5GB raw weight footprint before overhead.
Leave room for KV cache, vision inputs, runtime allocations, and concurrent requests. Review power, cooling, driver support, and reliability needs before choosing a production workstation.
NVIDIA GeForce RTX 5090 specifications ↗ · Weight figure is arithmetic, not a measured deployment.
SHARED ENTERPRISE SERVICE
Larger checkpoints and sustained multi-user load call for dedicated serving infrastructure. Plan memory per accelerator, interconnect, scheduling, monitoring, failover, and a tested recovery process.
DeepSeek V4 Flash illustrates this tier. Its large total weights require substantial memory even though only part of the model activates per token. Quantization, offloading, and supported serving architectures must be evaluated together.
DeepSeek-V4-Flash-0731 official model card ↗ · NVIDIA DeepSeek-V4-Flash deployment recipes ↗
AN INTUITION FOR MEMORY
Move between model sizes and precision to see the theoretical raw weight storage. This deliberately simple calculation is a starting point for sizing, not a compatibility checker.
About 12.6 GiB before any overhead.
Formula: parameters × bits ÷ 8. Decimal GB and binary GiB are shown separately. Real formats add scales, metadata, and sometimes mixed-precision tensors. This excludes KV cache, activations, vision components beyond the chosen count, runtime buffers, operating system memory, and concurrency. Selecting a precision does not imply a compatible checkpoint or runtime is available.
BUILD THE DEPLOYMENT AROUND THE WORK
The interaction. A short internal answer, a long document analysis, and an agent making tool calls stress the system differently.
The load. Concurrent users, request length, peak arrivals, and acceptable waiting time determine serving requirements.
The operation. Identity, encrypted storage, backups, monitoring, updates, retention, and recovery belong in the infrastructure plan.
The evidence. We test the proposed precision and runtime using your evaluation set, then document observed quality and performance.