How Many Bits Does an AI Need?

You decide to download an eight-billion-parameter AI model.

You have a computer, a few tasks in mind, and the optimistic belief that downloading something is an activity with a beginning and an end.

Then you reach the model page.

There is an FP16 version. An eight-bit version. A six-bit version. A five-bit version. Several four-bit versions. And something called Q4_K_M, which looks like a replacement part you should photograph before visiting the hardware store.

They are all versions of the same trained model. They contain essentially the same collection of learned parameters. Yet their memory requirements are very different.

Your first instinct is understandable: choose the largest one. Surely it contains more of whatever makes the model useful.

Your graphics card, with its eight gigabytes of VRAM, would like to participate in that decision.

The difference between these versions comes largely from weight quantization: representing the model’s learned numerical weights with fewer bits, using carefully chosen approximations.

Reducing the precision of those numbers can dramatically reduce the memory they occupy.

That sounds like a bargain, followed immediately by a reason to be suspicious.

What has been lost? Will the model still understand your documents? Can it still help with code? Does a four-bit model know that Boston is in Massachusetts, or was geography removed to make room for grammar?

There is no department deciding which cities survive the compression.

What changes is how faithfully the model’s learned numbers are preserved. Those numbers influence its calculations, so changing them can affect how well it uses what it learned.

But preserving every available bit has a cost. And for the work you actually want to do, the extra precision may or may not justify the additional video memory.

That is the question underneath the entire download menu:

How much of the original model’s numerical precision do I need—and what can I do with the VRAM I save?

The Model Has Already Been to School

Before we compare four bits with sixteen, it helps to separate learning from storing what was learned.

During training, a model adjusts enormous collections of numerical values called weights. Together, these values influence how it recognizes patterns, follows instructions, and produces answers.

By the time you download an ordinary pretrained model, that training has already happened.

The four-bit version has not attended four years of school while the sixteen-bit version has attended sixteen. They are different numerical representations of the same trained model.

The education came first. Quantization changes how its results are preserved.

Imagine that training produced this weight:

To understand numerical precision, compare these representations:

They describe roughly the same value, but they preserve different amounts of detail.

The first retains all the digits shown in our example. The second rounds the value to two decimal places. The third is a coarser approximation.

That is the everyday idea behind numerical precision: how closely the stored number preserves the value we wanted to keep.

When we move toward lower-bit representations of the same weights, we generally accept coarser approximations in exchange for lower memory use.

The decimal example is an illustration. Computers use binary formats, so four bits does not mean four decimal places. Different formats distribute their bits differently, and practical quantization is more sophisticated than simply cutting digits off the end.

But the underlying trade is real.

More bits give us more room to preserve the trained numbers faithfully. Fewer bits save memory, but discard some numerical detail.

Because the model’s abilities depend on those numbers, discarding detail can affect its ability to use what it learned.

Sometimes the difference is difficult to notice. Sometimes it matters.

The job is to find out which situation applies to your work.

Your Graphics Card Has a Smaller Apartment

Your SSD may have room for several versions of a model. A two-terabyte drive can accommodate quite a collection.

Your graphics card is a less generous landlord.

For the following examples, suppose you want the model to run entirely on a dedicated GPU. Its weights must fit into that graphics card’s video memory, or VRAM, with additional room for processing your requests.

This is the memory budget we are interested in here.

A bit has two possible states: zero or one. Several bits together can form more patterns:

Four bits provide sixteen possible patterns. Six bits provide sixty-four. Eight bits provide 256.

A numerical format uses these patterns to represent values. More representational room costs more actual memory.

Eight bits make one byte. Sixteen bits make two bytes.

Consider a simplified model containing exactly eight billion weights. At sixteen bits per weight, its raw weight data requires approximately:

That is approximately 16 GB of weight data to accommodate in VRAM, before adding working memory.

An 8 GB graphics card cannot supply that space. It does not become more flexible when you explain that the model has excellent reviews.

At four bits per weight, the idealized calculation becomes:

Practical formats add supporting information and may keep some components at higher precision. But the underlying difference is substantial.

The same collection of learned weights can occupy much less VRAM.

These equations use decimal gigabytes. Software may display GiB, a slightly larger unit. More importantly, they describe the weight data alone, not the complete memory requirement of a running model.

Four-bit therefore does not mean “requires a four-gigabyte graphics card.”

The useful relationship is that, for the same model, fewer bits per weight generally leave more video memory available for everything else.

Eight billion parameters are still present. They have received a revised housing allowance.

What Four Bits Actually Store

Let us return to our imaginary weight:

How can a four-bit code represent a number like that?

A four-bit code can identify sixteen possibilities. In a deliberately simple example, we could let those codes represent sixteen values spaced 0.1 apart:

Our original value is not on the list. The closest available value is 0.4.

The system can store the compact code 4, along with shared information saying that each step represents 0.1:

The step size is called a scale.

The difference between the original weight and its approximation is:

That difference is quantization error.

This is a teaching example, not the literal recipe used by Q4_K_M. Real weights can be negative, and practical methods choose their representations more carefully.

Many methods divide weights into groups, with each group using an appropriate scale or other interpretation information. Consequently, the same four-bit code can represent different approximate values in different groups. The whole model is not restricted to one universal list of sixteen numbers. Hugging Face’s quantization guide explains these mappings.

Think of a ruler whose markings can be spaced to suit the measurements in front of you.

A group of closely packed values benefits from fine spacing. A group spread across a wider range needs a different arrangement.

You still have a limited number of markings. You can position them intelligently, but you cannot create additional ones through enthusiasm.

Computers remain disappointingly resistant to enthusiasm.

How Rounding Reaches the Answer

Our original weight was 0.3726. Its approximation is 0.4.

Suppose an incoming numerical value is 2. Using the original weight gives:

Using the approximation gives:

The operation is the same. The result has shifted.

A model performs enormous numbers of calculations involving its weights. Their results feed into later calculations across its layers.

When you ask about Boston, your text is divided into tokens—words or pieces of words—and converted into numerical representations. Those representations change as they move through the model.

These changing numerical values are called activations. The learned weights help determine how they are transformed.

One common operation can be written compactly as:

Here, $X$ contains incoming numerical information, $W$ contains learned weights, and $Y$ contains the transformed result.

Behind that little equation can be an enormous collection of multiplications and additions. While the model appears to be calmly discussing Boston architecture, the hardware is doing arithmetic with an intensity that would concern a human accountant.

Changing the weights can therefore change the calculations—and eventually the answer.

But the effects are not as simple as adding every rounding error together. Some differences cancel. Some have little practical importance. Others affect parts of the computation that matter more.

Good quantization methods try to preserve useful behavior despite the approximations.

Researchers also consider how the weights are actually used. AWQ, or Activation-aware Weight Quantization, uses information about activations to protect important channels through scaling. The AWQ research describes this approach.

Precision matters unevenly.

Apparently, even inside a neural network, some numbers have negotiated better contracts.

Boston Survives. The Itinerary Requires More Attention.

Suppose you ask several versions of the same capable model:

Is Boston in Massachusetts?

They may all answer correctly.

Boston survives four-bit quantization. Massachusetts remains where we left it. The Charles River has not been relocated to Ohio.

Now try a more demanding request:

I am staying near Back Bay. Plan a day with a museum, an independent bookstore, a seafood dinner, and two free hours in the afternoon. I have no car, want to avoid long walks, and need to reach South Station before my evening train.

The model must coordinate several requirements. It also needs trustworthy information about opening hours and transportation.

There are more opportunities for weaknesses to become visible.

A good four-bit version may handle the task very well. A higher-precision version may better preserve the original model’s ability to coordinate those requirements. Neither outcome follows automatically from the bit count.

The same issue can arise when comparing proposals with complicated exclusions, modifying code without breaking related behavior, or working through a problem with several dependent steps.

These tasks give us reasons to care about preserving the model’s learned behavior.

But numerical precision and factual accuracy are different things.

A model can preserve its weights very precisely and still lack the information needed to answer your question. More bits cannot supply an updated timetable that the model has never received.

Sixteen bits do not persuade a closed restaurant to reopen.

And a more detailed answer is not necessarily a more accurate answer. A model can be wrong briefly or provide an impressively organized explanation of why it is wrong.

Confidence, regrettably, requires very little storage.

Then Why Keep the Sixteen-Bit Version?

If a four-bit version can answer correctly, why would anyone use the larger one?

Because “works equally well on this task” is not the same as “preserves all of the original model’s behavior across every task.”

A sixteen-bit checkpoint often serves as the high-precision version from which smaller variants are made. Keeping it avoids the additional approximation introduced by low-bit quantization.

That matters when preserving the original model’s behavior is more important than reducing its memory requirements.

Perhaps your workload exposes a weakness in the quantized version. Perhaps you are evaluating the model across a broad set of tasks. Perhaps you have sufficient VRAM for the high-precision weights, the context, and the necessary workspace, so there is little reason to accept further numerical loss.

Higher precision does not guarantee a correct answer. It removes one additional source of change.

The original version is also valuable as a starting point for further work: making new quantizations, comparing methods, or preparing the model for additional training.

Once numerical detail has been discarded, simply converting the smaller representation back into a larger format does not recover it.

Writing 0.400000 instead of 0.4 does not bring back 0.3726. It gives the approximation a longer suit.

Further training deserves a separate distinction. Installing a sixteen-bit model does not cause it to continue learning automatically during ordinary conversation. Training requires a process that updates parameters.

Higher-precision representations are useful for that work, although low-bit approaches also exist. QLoRA, for example, keeps a four-bit base model frozen while training additional components called adapters. The QLoRA paper explains this method.

So sixteen-bit models have practical purposes. They preserve a more faithful starting point and can help avoid degradation that matters for a particular workload.

What they do not provide is an automatic upgrade from “knows Boston” to “has completed postgraduate research on Massachusetts.”

And Why You May Not Need Sixteen Bits

Most people downloading a local model are trying to accomplish something.

They want to organize notes, understand a document, draft an email, compare products, work through code, or explore a subject.

They are usually not trying to preserve every numerical detail for its own sake.

If a good four-, five-, or six-bit version performs their tasks well, the additional memory required by a higher-precision version may buy little practical improvement for that particular use.

This is the distinction between a measurable difference and a difference that changes your experience.

Both matter, but they answer different questions.

Suppose two versions produce equally useful summaries of the documents you actually read. One occupies substantially more VRAM. The other leaves room for longer documents and more conversation history.

Choosing the smaller representation can be a sensible engineering decision.

You are not required to reserve a presidential suite for every decimal place.

At the same time, there is no universal rule saying that four-bit quantization loses only two percent of quality—or retains a fixed percentage of intelligence.

A reported difference belongs to a particular model, quantization method, evaluation, and setup. Another workload may show a smaller difference, a larger one, or little useful difference at all.

Research such as GPTQ has demonstrated three- and four-bit quantization with small losses on its reported evaluations, including very large models. That is evidence that the trade can work extremely well. It is not a guarantee for every model and every task. The GPTQ paper describes the results.

The useful question is therefore personal and testable:

Does this smaller version still do my work well enough—and is the saved VRAM worth more to me than the remaining difference?

For many everyday workloads, the answer can be yes.

For some demanding workloads, higher precision may earn its place.

Neither decision requires joining a faction on the internet.

Four Bits and a MacBook

The idea of reducing neural-network precision existed long before local language models became a popular subject.

What changed was the scale of the opportunity.

As language models grew to billions of parameters, reducing the number of bits per weight could turn an impractical memory requirement into something a personal computer could accommodate.

In March 2023, that possibility acquired a particularly tangible expression through Georgi Gerganov’s llama.cpp.

On March 11, developer Simon Willison documented running the seven-billion-parameter LLaMA model on his M2 MacBook Pro. His computer had 64 GB of memory; the model file was about 4 GB after four-bit quantization. His firsthand account is still available.

A trained language model was answering questions locally on a laptop.

For people accustomed to hearing about data centers and immense training budgets, that was an important change in perspective. You could experiment with a capable model on a personal computer without also acquiring an industrial cooling system and a person named Facilities.

Quantization did not make the original training free. It helped make the trained result practical to use.

The models were expensive to educate and surprisingly accommodating about the condition of their notes.

And once very low precision proved useful, researchers continued exploring how far they could push it.

Programmers occasionally reach philosophy by repeatedly subtracting one.

Q4_K_M and Other Forms of Small Print

You return to the download list.

You understand four bits now. Unfortunately, Q4_K_M still appears to contain several additional opinions.

The label identifies a particular quantization recipe. The four points toward its low-bit weight representation, but practical formats need more than the compact codes themselves.

They may also store scales, other interpretation information, and some model components at higher precision.

Consequently, a “four-bit” format can average more than four effective bits per weight.

Two files carrying a four-bit label can differ in size and behavior. The recipe matters, not just the headline number.

A four-bit model resembles a five-minute software installation in one limited respect: the number is useful, but you should remain on speaking terms with the footnotes.

You may also see GGUF, which names the file format carrying the model and its metadata. It is a different kind of label from Q4_K_M, which describes a quantization recipe.

And FP16 and BF16 are both sixteen-bit floating-point formats, but they divide their available bits differently between numerical range and precision.

You can understand the practical choice without memorizing every internal layout.

What matters is that the bit count does not completely identify the format, and your runtime—the software that runs the model—must support the particular version you choose.

The abbreviations can remain acquaintances. There is no obligation to invite them into the family.

Smaller Weights Still Have to Move

A reasonable question now arises: if the weights are stored as compact codes, does the computer have to expand the entire model back into a huge high-precision copy before using it?

A suitable implementation does not need to keep such a permanent expanded copy.

Optimized routines can read compact weight blocks and interpret their approximate values as part of the calculation. Parts of the arithmetic may still use higher precision.

That does not restore information lost during quantization. Multiplying 0.4 very accurately does not reveal that the original weight was 0.3726.

Keeping the weights compact also reduces the amount of weight data that must move through the memory system.

That movement takes time. Memory bandwidth describes how quickly data can be transferred. When moving data is a major bottleneck, smaller representations can help the processor spend less time waiting.

But decoding compact representations can require additional work. Performance depends on the hardware, runtime, format, and task.

A format that speeds up generating an answer may behave differently while processing a long prompt.

So “four-bit is always faster” is not a reliable rule. Neither is “the smallest download will produce the fastest answer.”

The smaller file will generally download sooner under the same conditions. It is comforting when at least one part of the discussion behaves itself.

For actual inference speed, measurements on the intended setup remain valuable.

The Weights Fit. Now the Model Would Like to Work.

Here is where the saved VRAM becomes especially useful.

The weights are only part of a running model’s memory requirements. It also needs space for temporary calculations and runtime buffers.

Many language models maintain a KV cache, which stores intermediate information associated with previously processed tokens. Reusing that information avoids repeating some of the same calculations while generating the next token. Hugging Face explains how this caching works.

Your instructions, supplied documents, conversation history, and the developing answer contribute to the model’s context.

A longer context commonly requires more cache memory, although the exact relationship depends on the architecture and runtime. Some designs limit or change how that memory grows.

The weights themselves do not expand because you paste in another chapter. The additional demand comes from handling the longer context.

There is another easily missed distinction: four-bit weights do not automatically mean a four-bit KV cache. Weight precision and cache precision are separate choices.

For a simple GPU memory plan, we can think in these terms:

The working allowance covers things such as computation buffers and room reserved for other GPU uses.

Let us put numbers into that idea.

The llama.cpp documentation lists a Q4_K_M file for Llama 3.1 8B at approximately 4.58 GiB, and a Q8_0 file at approximately 7.95 GiB. These published weight-file sizes provide a starting point for an estimate; they are not measurements of complete runtime VRAM usage. Source: llama.cpp’s quantization comparison.

Suppose, for our planning example, we use those figures for the weight allocation and allow 1 GiB for the conversation cache and another 1 GiB for workspace and other uses.

With the smaller weight representation:

With the larger one:

On a graphics card with an 8 GiB VRAM budget, those are very different situations.

The one-gigabyte allowances are assumptions for this example. Actual requirements depend on the runtime and workload.

But the reason the complete calculation matters is now visible.

The larger weight representation nearly fits by itself. The planned workload does not.

The smaller representation leaves room to do the work.

This is why the most faithful representation is not automatically the most useful configuration on a particular computer.

A model that consumes almost all available VRAM is rather like a magnificent sofa occupying almost the entire living room. The sofa fits. You are now eating dinner vertically.

Quantization can make room for the rest of the evening.

Can System RAM Help?

Yes, when the runtime and model support the arrangement.

A model can run on the CPU using system RAM. It can run on a GPU using VRAM. Or the runtime can divide work between them, allowing part of the model to use system RAM while another part uses the GPU. Projects such as llama.cpp support CPU execution and mixed CPU–GPU operation.

On unified-memory systems, the processor and graphics hardware share a memory pool, which changes the placement picture again.

These approaches can make models usable that would not fit entirely into a dedicated graphics card’s VRAM. Performance depends on the arrangement.

However, 8 GB of dedicated VRAM plus 32 GB of system RAM does not become 40 GB of equally fast GPU memory.

The runtime must place data and computation appropriately. Moving data and performing part of the work on the CPU can affect speed.

Your operating system and other applications also need RAM. They have not agreed to move out merely because you have developed an interest in artificial intelligence.

Likewise, an agent’s entire database does not automatically move into VRAM. The database runs in its own environment; passages retrieved and sent to the model become part of the model’s context.

The GPU calculations above assume the model’s weights are placed entirely on the graphics card. Changing that arrangement changes the memory plan.

Back to the Download Button

By now, the file list should look less like an admission examination.

The sixteen-bit version preserves a high-precision representation of the trained weights. Eight-bit quantization generally makes a less aggressive approximation than lower-bit alternatives. Five- and six-bit formats offer intermediate trade-offs. Four-bit formats can make a capable model practical within a much smaller VRAM budget.

Across those choices, the underlying question remains the same: how much numerical fidelity does your task benefit from?

For short classification tasks, a modest context and a good low-bit version may be enough. For a coding assistant examining several files, or an agent carrying instructions and task history, working memory can become a substantial part of the requirement.

Some tasks expose a useful advantage from higher precision. Others produce equally satisfactory results with less.

A model that fits with a short greeting may still struggle with the document you actually wanted it to read. A high-precision model that cannot accommodate that document has preserved its numbers beautifully while missing the assignment.

This is why testing should resemble your work.

If you want help comparing print orders, give it representative orders with known answers. If you want help with code, use code you can evaluate. If you want a Boston itinerary, check whether it respects the walking limits and whether the opening hours came from a trustworthy source.

Successfully locating Boston in Massachusetts is a promising start. It does not qualify the model to organize your entire weekend.

When comparing quantizations, keep the underlying model and task as consistent as possible. That makes the effect of the changed representation easier to judge.

This is also the purpose of exploring memory budgets in Green Zone: to see how the selected weights, context, memory placement, and available capacity might work together.

An estimate helps you choose a plausible configuration. Trying it on representative tasks tells you whether it is useful for you.

You may discover that four bits preserves enough of the model’s abilities for everything you need. You may prefer five or six. Or your workload may benefit from higher precision, with enough VRAM available to support it comfortably.

There is no prize for using every available byte on the weights alone.

And there is no obligation to discard precision when it provides a useful benefit.

The model has already learned. Your decision is how faithfully to preserve that learning while giving it enough room to work on your computer.

You return to the download page. The labels are unchanged, but you can now read them.

You choose a version that suits your runtime, your available working memory, and the job you actually have in mind.

Boston remains in Massachusetts. Your computer has room to work. You can finally close the comparison thread.

Someone has just posted a detailed argument about two-bit quantization.

You do not have to open it.