A downloaded model fits on the GPU and answers several short questions. Then someone submits a long document while two colleagues start conversations. Memory runs out, or the response arrives too late. Even if execution survives, another question remains: does the four-bit version still get the important answers right?
The bit-width label cannot answer those questions. A deployment decision needs three results together: task quality, peak runtime memory and latency under the intended load. This is ZharfAI's proposed method for making that comparison, not a benchmark of a particular model or a hardware buying recommendation. Every numerical example below is an explicitly hypothetical calculation.
Start with what four bits actually describe
Model weights are numbers learned during training. Quantization represents them using fewer numerical states. Unlike lossless file compression, expanding the representation does not necessarily recover the original numbers. The operational question is which answers change when numerical detail is discarded, not whether a compact artifact can be loaded.
In a linear scheme, a scale connects stored values to approximate numerical values. Those scales occupy storage too. Four-bit is also not one interchangeable format: NVIDIA's TensorRT documentation distinguishes weight-only INT4 from floating-point FP4. Two artifacts carrying the same bit-width label need not have identical computations, hardware support or quality.
Identify the exact base-model revision, conversion method, weight grouping, unconverted components and execution software before comparison. Without that information, you can evaluate an artifact as a product, but cannot confidently attribute a difference to bit width. A different model revision, chat template or generation policy changes the experiment.
This distinction matters when selecting a community conversion. A convenient filename is not a reproducible configuration. Preserve the artifact digest and enough conversion information to explain what was changed; do not infer a numerical recipe from a short marketing label.
Separate storage, computation and cache precision
Four-bit weight storage does not require every operation to use four-bit arithmetic. The Hugging Face bitsandbytes guide exposes a separate computation data type and describes modules that are not converted into low-bit linear layers. “The whole model is four-bit” therefore leaves important implementation details unspecified.
Record three independent settings: weight representation, computation representation and cache representation. The key-value cache, or KV cache, retains information about preceding tokens for subsequent generation. Quantizing weights does not establish that this cache has also been quantized.
Use these distinctions when debugging. If long conversations still exhaust memory after weight storage shrinks, first identify where the memory goes. Repeating the weight conversion may target the wrong component. If execution slows down, inspect the actual computation path and data types; the artifact's filename cannot tell you which instructions the processor executed.
Keep the configuration explicit even when a runtime chooses defaults automatically. A later software update may select a different path without changing the model's familiar display name.
A four-gigabyte calculation is only about weights
Consider a hypothetical model containing eight billion weights. If every weight were packed uniformly with no overhead, its raw payload would be:
| Bits per weight | Raw weight payload, decimal GB |
|---|
| 16 | 16 GB |
| 8 | 8 GB |
| 4 | 4 GB |
The arithmetic is parameter count multiplied by bits per parameter, divided by eight. It is not a prediction of a real product's download size or required GPU memory. Scales, unconverted components, cache, intermediate activations and runtime workspace are not included. Reserved memory can also exceed the memory occupied by active data.
Measure several moments: after model loading, during processing of a long input, near the end of generation and during concurrent requests. Where instrumentation provides both allocated and reserved memory, retain both. A screenshot of an idle, newly loaded model is not a peak-capacity measurement.
On a unified-memory machine, distinguish available memory from nominal device capacity; the operating system and other applications share that resource. Our Ollama installation and local-model guide covers getting a first model running. That successful first conversation is the beginning of capacity testing, not its conclusion.
The Transformers cache guide describes a dynamic cache that grows during generation, with bounded growth in sliding-window or chunked-attention layers. It also warns that cache quantization can hurt latency for short contexts when sufficient memory is already available. Choose and test cache precision separately from weight precision.
Here is an illustrative full-attention architecture: 32 layers, eight key-value heads per layer, 128 values per head, 8,192 cached tokens and two bytes per value. If all layers retain all these tokens, raw key-plus-value storage for one sequence is:
2×32×8×128×8192×2=1,073,741,824 bytes=1 GiB
The leading two accounts for keys and values. Four independent sequences without cache sharing require four GiB in this example, separately from model weights. The token count means everything retained at that moment, not merely the user's original question; generated tokens must enter the budget too.
This is an estimate for a declared architecture, not a universal model-sizing formula. Prefix sharing, sliding windows, distribution across devices and allocation strategy change the result. Its purpose is to identify what to measure. Test the service's maximum permitted context and concurrency together, and leave headroom for observed variation. There is no defensible universal headroom percentage for every runtime and workload.
Do not mix decimal GB with binary GiB when reconciling a capacity report. Keep the raw byte counts alongside the human-readable values so that a unit conversion does not look like unexplained runtime overhead.
Keep calibration separate from the final examination
Some conversion methods use example inputs to choose numerical ranges. PyTorch's static-quantization tutorial explains ranges calibrated on similar inputs before inference, contrasting them with dynamic ranges computed for new input batches. Not every four-bit method needs the same calibration procedure; follow the selected method's requirements.
Our recommendation is to separate calibration data, development examples and the final evaluation set. If you repeatedly inspect final-test answers and adjust the recipe to fix them, that test is no longer independent evidence for the chosen conversion. Retain the origin and assignment of each example rather than relying on a folder called “test.”
For a Persian assistant, a few clean English paragraphs do not represent the intended workload. Include actual task shapes: short and long questions, product names, negation, Persian and Latin numerals, tables and structured responses. The aim is not universal language coverage; it is evidence for the uses you intend to approve.
Related documents and rewritten variants can leak across seemingly separate files. Our customer- and time-separated evaluation guide explains why separating rows alone does not necessarily separate the underlying examples. Apply that reasoning when constructing conversion and evaluation data.
One percentage point can conceal three important regressions
Suppose both versions complete the same 100 artificial tasks. The reference gets 94 right; the four-bit candidate gets 93 right. The aggregate difference is one percentage point, but paired outcomes reveal more:
| Task group | Reference correct | Candidate correct | Regressions | Improvements |
|---|
| 20 critical tasks | 19 | 16 | 3 | 0 |
| 80 ordinary tasks | 75 | 77 | 1 | 3 |
| All 100 tasks | 94 | 93 | 4 | 3 |
A regression means reference-correct and candidate-incorrect; an improvement means the reverse. Three improvements largely conceal four regressions in the aggregate. Within the critical group—perhaps detecting a prohibition in an internal contract—the change is from 19 correct answers to 16. These are invented counts, not measurements of any model, and twenty cases cannot precisely establish a future error rate.
Define unacceptable errors before examining results. A modest decline may be tolerable for editable drafts but not for extracting an amount that enters an operation without review. Valid JSON and syntactically valid numbers are not evidence of semantic correctness. Both versions can return perfectly valid objects while only one contains the right amount.
Retain outputs under stable case identifiers. For qualitative answers, blind reviewers to the version label so expectations about compression do not determine the verdict. If generation is stochastic, use sufficient repetitions and consistent settings. One good answer and one bad answer do not establish a quantization effect. The table illustrates a reporting method, not a statistical certification procedure.
Report sample counts with each task slice and investigate the changed answers. An aggregate quality floor alone cannot express the cost of a rare but consequential error.
Measure speed on the actual execution path
ONNX Runtime's documentation ties speed gains to the model, hardware and conversion overhead. For quality losses, it describes comparing original and quantized weights and activations, then changing the method or excluding problematic tensors. Fewer bits are neither a speed guarantee nor proof that every task must become worse.
Start a performance comparison with two questions: how long until the first token, and when is the task finished? Fast generation after a long initial wait may still be unsuitable for conversation. For offline work, correct completed tasks per interval may matter more. Our inference-latency guide separates input processing, subsequent generation and queueing.
Use the same inputs, output limits, concurrency and generation settings for the reference and candidate. Report cold and warm runs separately, and distinguish requests with reusable cache from requests without it. Retain total completion time and upper latency percentiles rather than only average tokens per second. Out-of-memory failures and rejected requests belong in the results; omitting them overstates usable capacity.
If the reference cannot fit on the target device, evaluate its quality on another suitable machine and disclose that difference. Timings from different machines do not isolate the effect of quantization. Either perform a compatible same-hardware comparison or explicitly describe the experiment as a comparison between two deployment options.
The useful result is the workload that finishes correctly within its constraints. A higher token rate is not a saving if shorter prompts, extra retries or human corrections are required to obtain that result.
Turn measurements into a concrete choice
The following is ZharfAI's decision framework, not a ranking of quantization methods. It does not approve a candidate merely because it uses four or eight bits.
| Observed result | Defensible next decision |
|---|
| Critical-task quality, memory headroom and response time all pass | Accept four-bit inference only for the tested context, concurrency and use cases |
| Aggregate quality passes but critical regressions increase | Reject for now; retest higher precision or selectively unconverted components |
| Answers remain correct but long requests exhaust memory | Examine cache, context limits and concurrency before further reducing weight precision |
| Memory improves but task completion slows | Inspect implementation and execution path; retain the previous version if the needed benefit is absent |
| No configuration meets the requirements | Separately evaluate a smaller model, a changed use case or different hardware |
Do not borrow a universal “two percent degradation is acceptable” threshold from another article. The limit must reflect error cost, review arrangements and the strength of your evidence. Sometimes a smaller model at higher precision is the better choice; sometimes a larger low-bit model is. Test those specific alternatives instead of deciding from parameter count alone.
Changing the allowed workload is also a product decision. A lower concurrency cap or shorter document limit may make a configuration viable, but it must be disclosed and enforced rather than silently reducing the test to an easier problem.
Revisit the decision when the workload or runtime changes
Preserve the accepted artifact and digest, runtime and driver versions, chat template, generation settings, conversion recipe, calibration data and evaluation results together. Record who reviewed critical answers and which cases still need human checking. This record explains behavior changes more effectively than an endpoint named “lightweight.”
After adoption, monitor input and output lengths, concurrent requests, peak memory, response time and human corrections. More long documents or an additional language can invalidate the earlier decision without any change to weights. A runtime update or different cache strategy is another reason to repeat the comparison.
A smaller file creates an option, not unconditional permission to deploy. Four-bit inference is valuable when it performs the important work correctly within the available time and memory. If those three requirements do not hold together, a smaller download is not enough.
Source notes
Sources checked on November 19, 2026. The documentation supports technical mechanisms and caveats; the decision table, test design and numerical examples are ZharfAI analysis. This article reports no model benchmark.
- ONNX Runtime: Quantize ONNX models: living documentation without an explicit update date on the page.
- PyTorch: Static Quantization: creation and last-update date shown as March 25, 2026.
- Hugging Face: bitsandbytes: development-branch documentation; verify compatibility with the installed release separately.
- Hugging Face: Transformers 5.23.0 cache strategies: version-specific library documentation.
- NVIDIA: Working with Quantized Types: page last updated November 18, 2026.