Self-Hosting Open-Weight LLMs in Production: FP8 vs. AWQ vs. GGUF Quantization & vLLM Tuning

1. Choosing the Right Quantization Format for Your GPU Architecture

When self-hosting open-weight Large Language Models (Llama-3.1/3.3, Qwen-2.5, Mistral, DeepSeek) on dedicated GPU servers, choosing the wrong quantization format for your specific GPU generation can cut token throughput in half or degrade reasoning accuracy.

The key distinction is between Weight-Only Quantization (W4A16) and Weight-and-Activation Quantization (W8A8 FP8):

| Quantization Format | Precision (Weights / Activations) | Required Hardware Support | Best Target Scenarios |
| :--- | :--- | :--- | :--- |
| BF16 / FP16 (Unquantized) | 16-bit / 16-bit | All modern GPUs (Ampere, Ada, Hopper, Blackwell, MI300X) | Maximum baseline accuracy when VRAM is abundant |
| FP8 (W8A8 E4M3) | 8-bit Weights / 8-bit Activations | NVIDIA Hopper (H100/H200), Ada (L4/L40S/RTX 4090), Blackwell (B200) | High-concurrency production serving (vLLM / TensorRT-LLM) — accelerates both compute-bound Prefill (via native FP8 Tensor Cores) and memory-bound Decode (halves VRAM vs. BF16) with near-zero perplexity loss |
| AWQ / Marlin (W4A16) | 4-bit Weights / 16-bit Activations | NVIDIA Ampere (A100, A10G, RTX 3090), Ada, Hopper | VRAM-constrained deployments & low-batch serving — shrinks a 70B model to ~38 GB so it fits on a single 48 GB L40S or A6000. Weights are dequantized on-the-fly in registers to FP16 for compute |
| GGUF (llama.cpp / Ollama) | 4-bit to 8-bit (Q4_K_M, Q5_K_M, Q8_0) | CPU (AVX-512 / Apple Silicon Unified Memory) & Consumer GPUs | Local developer workstations, edge appliances, and CPU+GPU hybrid offloading |

Why Ampere (A100) Should Use AWQ (Marlin) While Hopper (H100/H200) Should Use FP8

- NVIDIA A100 (Ampere, Compute Capability 8.0) does not have native hardware FP8 Tensor Cores. Running an FP8 model on an A100 requires software weight-only emulation. Instead, use 4-bit AWQ or GPTQ with Marlin kernels on A100 when memory-bound, or native BF16. - NVIDIA H100, H200, L40S, and B200 (Compute Capability 8.9 / 9.0 / 10.0) include native hardware FP8 Tensor Cores that double dense compute throughput (1,978.9 TFLOPS Dense FP8 vs. 989.5 TFLOPS Dense BF16 on H100/H200). On these GPUs, W8A8 FP8 is the gold standard for production high-batch inference.

2. Preventing KV-Cache Preemption & Latency Spikes in vLLM

Even with PagedAttention in vLLM, if too many concurrent requests with long prompts arrive simultaneously, the KV-cache blocks fill up (GPU KV cache usage: 100%). When that happens, vLLM prints a warning (Sequence group ... is preempted by recomputation mode) and evicts in-flight requests—causing sudden multi-second P99 tail-latency stalls.

Four Production Flags to Stabilize Tail Latency (TTFT and ITL)

vllm serve Qwen/Qwen2.5-72B-Instruct-AWQ \
  --quantization awq_marlin \
  --tensor-parallel-size 2 \
  --max-model-len 32768 \
  --max-num-seqs 64 \
  --enable-chunked-prefill \
  --max-num-batched-tokens 8192 \
  --enable-prefix-caching \
  --gpu-memory-utilization 0.90 \
  --disable-log-requests \
  --port 8000

1. --enable-chunked-prefill: Without chunked prefill, a single incoming 24,000-token RAG prompt monopolizes the GPU for several hundred milliseconds during prefill, freezing token generation (Inter-Token Latency) for all other streaming users. Chunked prefill slices long prompts into smaller chunks (--max-num-batched-tokens) and interleaves them with active decode steps.
2. --enable-prefix-caching (Automatic Prefix Caching in vLLM): Caches KV blocks of shared system prompts and shared RAG documents in GPU HBM using content-addressed hashes, skipping prefill computation entirely on cache hits.
3. --max-model-len Capped to Actual Application Need: Many models declare a 131072 (128k) context limit in config.json. If your application only sends up to 16384 or 32768 tokens, explicitly pass --max-model-len 16384 or 32768 so vLLM does not reserve oversized activation buffers.
4. Prometheus Observability (/metrics): Scrape vllm:gpu_cache_usage_perc, vllm:num_preemptions_total, and vllm:time_to_first_token_seconds into Prometheus/Grafana, and trigger horizontal autoscaling whenever gpu_cache_usage_perc sustains above 0.80 (80%) for more than 2 minutes.

3. Sizing Multi-GPU Tensor Parallelism (TP) vs. Pipeline Parallelism (PP)

When serving models larger than a single GPU's VRAM (Llama-3.3-70B, Qwen-2.5-72B, or DeepSeek-R1-Distill-70B), choosing between Tensor Parallelism (--tensor-parallel-size) and Pipeline Parallelism (--pipeline-parallel-size) depends on your physical GPU interconnect:

| Hardware Interconnect | Recommended vLLM Parallelism Strategy | Latency & Throughput Behavior |
| :--- | :--- | :--- |
| NVLink (H100/H200 SXM or 2x PCIe with NVLink Bridge) | Tensor Parallelism (--tensor-parallel-size 2 or 4) | Splits matrix multiplications across GPUs within every layer, reducing single-request latency (TTFT and ITL). |
| Standard PCIe Gen4 without NVLink (2x L40S or 4x RTX 4090) | Pipeline Parallelism (--pipeline-parallel-size 2) or Custom All-Reduce Disable | Avoids saturating PCIe lanes with per-layer all-reduce synchronization; pass --disable-custom-all-reduce if P2P DMA is unsupported by the motherboard IOMMU. |

4. Production systemd Service & Prometheus Telemetry for vLLM

In production, wrap vLLM in a resource-governed container or systemd unit and scrape its native /metrics endpoint (vllm:num_requests_waiting, vllm:gpu_cache_usage_perc, vllm:time_to_first_token_seconds) with Prometheus:

# Inspect real-time vLLM KV-cache utilization and queued request pressure
curl -s http://127.0.0.1:8000/metrics | grep -E "vllm:gpu_cache_usage_perc|vllm:num_requests_running|vllm:num_requests_waiting"
# Run an automated 200-request concurrency stress test against your local vLLM endpoint
vllm bench serve \
  --backend vllm \
  --model Qwen/Qwen2.5-72B-Instruct-AWQ \
  --dataset-name random \
  --random-input-len 2048 \
  --random-output-len 512 \
  --num-prompts 200 \
  --max-concurrency 32

Compare GPU hourly rental rates across H100, H200, B200, L40S, and A100 using our GPU Cluster Cost Calculator.