Mistral

Mistral Small 4

Mistral's docs say 119B total and 6.5B active. The launch post says about 6B active per token, and about 8B if embeddings are counted.

Apache-2.0 · 119B total, 6.5B active · context 262,144

The product docs say 256k context. config.json max_position_embeddings is 1,048,576. This page uses the 256k figure from the docs.

From the model card

Strengths

  • The docs describe a multimodal model (text and image) with tool use, a 256k context, and Apache-2.0 weights.
  • Mixture-of-experts: 128 routed experts in config.json, 4 experts per token.

Weaknesses

  • Weight memory follows the 119B total, not the active count.
  • The KV figure uses the latent sizes in config.json (kv_lora_rank and the rope dim). It is not a measured allocation. If a runtime stores more than that latent, real KV is higher. The weight term dominates.
  • No Frankenstein (HP ProLiant DL380p Gen8) row yet.

Sources: Mistral Small 4 docs · Launch post · config.json

Memory at 8,192 tokens

Computed, not measured. Weights use llama.cpp’s whole-file averages (Q4_K_M = 4.8944 bits/weight, Q8_0 = 8.5008) on the total parameter count. KV is fp16 and does not depend on the quant. The formula, including the sliding-window and latent-KV cases, is on the methodology page.

Computed memory for Mistral Small 4 at 8192 context
Q4_K_MQ8_0
Weights67.8 GiB117.77 GiB
KV at 8,1920.18 GiB0.18 GiB
Total67.98 GiB Computed117.94 GiB Computed

36 layers. Latent KV uses kv_lora_rank 256 plus a rope dim of 64, not a full KV head times head dim.

Tokens per second

Not yet measured. This lab has no Frankenstein (HP ProLiant DL380p Gen8) row for Mistral Small 4, so there is no estimated tok/s either.

Hardware tiers

Minimum is the smallest RAM that holds Q4_K_M plus 8k KV. Suggested does not name a second machine or a speed this lab has not run. Performance means a CUDA-class card with 12 GB or more of GDDR6 or GDDR6X and the weights fully on the card. The Tesla P40 is 24 GB of GDDR5, so a P40 speed is not the performance-tier number.

TierExample hardwareQuantFittok/s
MinimumCPU-only, with at least 69.98 GiB of system RAM for the Q4_K_M weights, 8k KV, and a 2 GiB planning allowance.Q4_K_MCPU-only, if the RAM is thereCPU: not yet measured
SuggestedA machine you are willing to wait on, or a GPU that can hold the quant. This lab has not measured a separate “comfortable” machine, so this row does not invent one.Q4_K_M until a Q8 run existsWhichever of the machines above actually holds the weightsnot yet measured
PerformanceA CUDA-class card with enough GDDR6 or GDDR6X to hold about 68.98 GiB (Q4_K_M weights, 8k KV, and a 1 GiB planning allowance). 12 GB is not enough for that total.Q4_K_MFull VRAM only if the card is larger than 12 GBnot yet measured

Where it sits

Best-for is a judgment from the card’s stated jobs and the model’s size. It is not a speed ranking. The matching build is CPU-only used Xeon. Score it against a machine you have on What can my rig run?