What Are Model Weights?

A modern AI model can write Python, recognize a dog, translate Japanese, summarize a contract, imitate Shakespeare, and occasionally tell you something completely wrong with the confidence of a man who has never once considered checking Google.

Underneath all of this sophistication is something surprisingly ordinary:

numbers.

Billions of them.

When you see a model described as 8B, 30B, or 70B, the B means billion parameters.

Those parameters are learned numerical values inside the neural network. Most of them live inside enormous mathematical structures called weight matrices.

These numbers are what people usually mean when they talk about a model’s weights.

They are not little sentences stored somewhere inside the computer.

There is no hidden row saying:

France → Paris

There is no folder called:

Things I Know About Dogs

And no individual weight has a job title such as:

Senior Vice President of Sarcasm.

Knowledge in a neural network is distributed across enormous patterns of numbers.

That sounds mysterious.

The basic idea is not.

In fact, the story begins more than eighty years ago.

Before AI Was Cool — or Even Electronic

In 1943, neurophysiologist Warren McCulloch and mathematician Walter Pitts published an influential mathematical model of an artificial neuron.

This was before modern AI.

Before GPUs.

Before Python.

Before the transistor had even been invented.

Their question was audaciously simple:

Could something inspired by a biological neuron be represented mathematically as a computational unit?

Their artificial neuron was extremely primitive by modern standards. It received inputs and produced an output according to mathematical rules.

But an important idea had arrived:

perhaps complicated behavior could emerge from large numbers of simple computational units connected together.

Modern neural networks are vastly more sophisticated.

But that ancestral idea is still recognizable underneath them.

Start With One Artificial Neuron

Forget eight billion parameters for a moment.

Let us build one tiny artificial neuron.

Suppose it receives three numerical inputs:

The letter $x$ simply means input.

Those inputs could ultimately represent almost anything: numbers from a financial dataset, pieces of an image, or internal numerical representations of language.

Each input is multiplied by another number:

The letter $w$ stands for weight.

A weight determines how strongly that particular input contributes to the neuron's calculation.

The neuron combines the values like this:

The formula looks more intimidating than the operation.

In ordinary English it says:

Multiply every input by its weight, add the results together, and then add one additional adjustable number called the bias.

The bias is:

It allows the neuron to shift its result up or down independently of the incoming values.

After calculating the weighted sum, a neural network will normally pass the result through a nonlinear function:

Here $f$ simply means:

Apply another mathematical transformation to the result.

Why bother?

Because without nonlinear functions, piling many layers on top of one another would still behave mathematically like one giant linear transformation.

Nonlinearity allows the network to represent far more complicated relationships.

So the basic journey is:

inputs → weights → weighted sum → nonlinear transformation → next layer

One artificial neuron will not write your quarterly report.

Billions of learned parameters arranged in sophisticated layers become considerably more interesting.

The Perceptron Arrives — and So Does AI Hype

In the late 1950s, psychologist Frank Rosenblatt developed the perceptron, an early trainable neural-network system.

The important word is trainable.

Instead of a programmer explicitly choosing every useful internal value, the machine could adjust its own connection strengths based on examples.

Those connection strengths were weights.

The basic idea sounded remarkably modern:

Make a prediction.
Find out whether it was wrong.
Adjust the numbers.
Try again.

Newspapers became extremely excited.

Which proves that AI hype was not invented by social media.

The computers were somewhat less convenient.

Today you can run models containing billions of parameters on hardware sitting under a desk.

Rosenblatt worked in an era when computers could occupy an impressive portion of a room while accomplishing considerably less than your phone.

Progress is occasionally cruel to old hardware.

The Giant Mixing Desk

A useful way to imagine weights is an enormous audio mixing console.

A signal comes in.

One control makes part of it stronger.

Another makes something weaker.

Another affects how several signals combine.

Now imagine not twenty controls.

Not twenty thousand.

Billions.

The final behavior depends on their combined settings.

But this analogy comes with an important warning.

A particular weight does not normally correspond neatly to one human concept.

You cannot open an 8B model, find parameter number 4,827,112,391, and announce:

“Excellent. We found the dog's left ear.”

Modern neural networks use distributed representations.

A concept such as a dog, irony, JavaScript, gravity, or a polite way of telling a customer that his artwork is terrible may depend on patterns involving enormous numbers of parameters and temporary internal states.

The meaning is distributed.

The intelligence is in the pattern.

Not in one magic knob.

Before Training: A Very Expensive Idiot

When a neural network is first initialized, its weights generally begin as small numerical values chosen according to an initialization method.

At that point, the network has architecture.

It has billions of places where knowledge could eventually be represented.

But it has not yet learned useful relationships.

You could build an enormous model and still have something largely useless.

This is worth remembering whenever people become hypnotized by parameter counts.

A warehouse containing ten billion blank notebooks does not automatically contain more knowledge than a library.

The notebooks need something written in them.

Training is the process that changes the numbers.

Not by writing sentences into them.

By creating useful numerical relationships.

Let One Neuron Make a Mistake

Now let us watch a tiny training example with numbers small enough to follow by hand.

Suppose our neuron receives two inputs.

The first input is:

The second is:

So the first signal has a numerical value of one, while the second has a value of one-half.

The current first weight is:

The second is:

And the bias is:

Now use our weighted-sum calculation:

Rather than staring at the formula, let us take it apart.

The first input contributes:

The second input contributes:

And the bias contributes another:

Add the three pieces:

So our neuron produces:

The little hat over $y$ means:

the model's prediction

Now suppose the correct target in this training example is:

The model predicted:

The correct result was:

So the prediction is low by:

The neuron is wrong.

This is excellent news.

Mistakes are basically the educational system of neural networks.

Without an error, the training process has no signal telling it what needs improvement.

Loss: A Mathematical Embarrassment Meter

The training system now needs a formal way to answer:

How wrong was that prediction?

That is the job of a loss function.

For our toy example, we can use a simple squared-error loss:

Let us translate every symbol.

$L$ means loss.

$\hat y$ means the model's prediction.

$y$ means the correct target.

So the formula says:

Subtract the correct answer from the prediction, then square the difference.

Our prediction is:

The target is:

Subtract them:

Now square that result:

So our loss is:

Why square the error?

Among other reasons, we do not want mistakes in opposite directions to cancel each other.

Being 0.30 too low is still a mistake.

Squaring gives us a positive quantity that optimization can try to reduce.

Think of loss as a mathematical embarrassment meter.

High loss:

That went badly.

Lower loss:

Less embarrassing.

A modern language model usually uses a different loss function for next-token prediction, commonly a form of cross-entropy loss.

We will meet that later.

For now the essential idea is:

prediction → measure the mistake

The next question is harder.

Our little example has only two weights.

A real model may have billions.

Which weights were responsible for the error?

And in which direction should they move?

1986: Mathematics Learns How to Assign Blame

One of the central tools used to answer that question is backpropagation.

The mathematical ideas behind it have a longer history, but influential work by David Rumelhart, Geoffrey Hinton, and Ronald Williams in 1986 helped make backpropagation central to modern neural-network training.

The problem is straightforward to state.

The output was wrong.

But many layers and many parameters participated in producing it.

How do we determine which changes would make the output better?

Backpropagation works backward through the network and calculates how sensitive the final error is to each trainable parameter.

You could think of it as an extraordinarily efficient accounting department.

A mistake happened.

Now everybody's contribution gets calculated.

Nobody gets to say:

“Pretty sure that was Layer 17.”

Calculus has receipts.

Backpropagation: What Does the Formula Mean?

One of the expressions you will encounter is:

If you studied calculus twenty years ago and have spent the intervening period successfully avoiding it, the notation may produce unnecessary emotional damage.

The idea is considerably simpler than the symbol.

It asks:

If I changed this particular weight by a tiny amount, how would the loss change?

That value is part of what we call the gradient.

Suppose increasing a weight slightly makes the error larger.

Useful information.

Suppose increasing it makes the error smaller.

Also useful.

The derivative therefore tells the optimizer something about both direction and sensitivity.

Backpropagation uses the chain rule from calculus to calculate these relationships efficiently through many layers.

The network is not thinking:

“I confused the dog with a cat because I focused too heavily on the fur.”

Nothing so literary is happening.

It is calculating how changes in numbers affect other numbers.

A lot of numbers.

Gradient Descent: Now Move Downhill

Once backpropagation has calculated the gradients, the optimizer has something actionable.

A simplified weight update looks like this:

Again, let us translate before using it.

$w_{\text{old}}$ is the weight we have now.

$w_{\text{new}}$ is the weight after the training step.

The gradient:

tells us how the loss changes when that weight changes.

And the Greek letter:

pronounced eta, is the learning rate.

The learning rate determines how large a step the optimizer should take.

So the whole formula means:

Start with the current weight.
Determine which direction increases the error.
Move a controlled distance in the opposite direction.

Why the opposite direction?

Because the objective is to make loss go down.

Imagine standing on a mountain in thick fog.

You cannot see the valley.

But you can determine which direction slopes uphill.

If the goal is to descend, walk the other way.

That is the basic intuition behind gradient descent.

The real optimization landscape of a modern neural network may have billions of dimensions.

This makes the mountain slightly difficult to sketch on a cocktail napkin.

If you can comfortably visualize a seventy-billion-dimensional landscape, you may have other issues.

Modern training commonly uses more sophisticated optimizers such as Adam or AdamW, which track additional information and adjust the update process intelligently.

But the central idea remains:

calculate how the parameters affect the error, then move them in a direction expected to reduce it.

Now let us actually do that.

One Tiny Update — With the Actual Math

Our toy neuron previously received:

and:

Its weights were:

and:

The bias was:

The neuron predicted:

The desired result was:

For this demonstration, we will use the weighted sum directly as the prediction so that every part of the arithmetic remains visible. A real network may also contain an activation function, whose derivative becomes part of backpropagation.

Our signed prediction error is:

The negative sign tells us:

The model's prediction is below the target.

Our squared-error loss was:

so:

Now we calculate how this loss responds to each weight.

Finding the Gradient for the First Weight

For this simple neuron, the gradient of the loss with respect to the first weight is:

It looks technical.

But every value is already known.

The error is:

The first input is:

Substitute those values:

Multiply them:

What does negative 0.60 mean?

It tells us that, at this point, increasing $w_1$ should reduce the loss.

The gradient points toward increasing error.

Gradient descent moves in the opposite direction.

Finding the Gradient for the Second Weight

Now do the same thing for the second weight:

The error is still:

But the second input is only:

Substitute:

which gives:

Notice something useful.

The magnitude of the second gradient is only half as large as the first.

Why?

Because the second input:

is half the size of the first:

In this tiny example, changing the first weight has a stronger effect on the final result.

This is exactly the sort of dependency backpropagation calculates automatically.

Choosing a Learning Rate

Now we know the direction.

We still need to decide how large a step to take.

For our demonstration, choose:

That means the optimizer will move by one-tenth of the gradient magnitude in this simplified step.

This is a convenient round number for teaching.

Real training systems choose and schedule learning rates much more carefully.

Too small, and learning can become painfully slow.

Too large, and the optimizer may leap past good solutions and begin bouncing around the landscape like someone attempting to descend a mountain in ski boots.

Updating the First Weight

The update rule is:

For the first weight:

First multiply the learning rate by the gradient:

So:

Subtracting a negative number means adding:

And now we know exactly where 0.26 came from.

Nobody selected it because it looked nice.

No engineer typed:

Let's try 0.26.

The gradient and the learning rate produced it.

Updating the Second Weight

Now repeat the exact same process for $w_2$:

Multiply:

So:

which gives:

That is where 0.43 came from.

For this teaching example, we will leave the bias unchanged at:

In a real network, a trainable bias would normally have its own gradient and could be updated by the same general process.

Run the Neuron Again

The inputs have not changed.

The first is still:

But its weight is now:

Its contribution is therefore:

The second input remains:

Its new weight is:

So its contribution becomes:

The bias remains:

Add everything:

The neuron's new prediction is:

Before the training step it predicted:

Now it predicts:

The target remains:

It is still wrong.

But notice what happened.

Before the update, the distance from the target was:

After the update:

The error became smaller.

We can also calculate the new squared loss:

First:

Then square it:

So the loss has fallen from:

to:

In this deliberately simple example, one gradient-descent step reduced the loss substantially.

That does not mean a real Large Language Model improves by the same percentage every training step.

Real optimization is noisy, complicated, and enormously higher-dimensional.

But our tiny example has now shown one complete learning step without hiding anything:

In words:

The neuron made a prediction.
The loss measured the mistake.
Calculus determined how the weights contributed to the mistake.
The optimizer changed the weights.
The new prediction moved closer to the target.
The loss became smaller.

Nothing magical happened.

The mathematics changed the numbers.

Then the system does it again.

And again.

And again.

Our example uses two weights because two weights still fit comfortably on a page.

A real neural network may adjust millions or billions of trainable parameters.

That is where industrial-scale computation enters the story.

How Does a Language Model Learn From This?

A Large Language Model is not usually trained by receiving a gigantic table of explicit rules such as:

Paris = capital of France

Dog = mammal

Python = indentation matters

Instead, much of modern language-model pretraining uses a deceptively simple objective:

Predict the next token.

So let us stay with Paris and follow the example all the way through.

Suppose the training text contains:

The capital of France is Paris.

At one point during training, the model receives:

The capital of France is

and must predict the next token.

For simplicity, assume Paris is represented by one token. Real tokenization can split some words into several tokens, but that detail does not change the principle.

Early in training, the model might produce probabilities something like:

London — 18%
Paris — 5%
Lyon — 4%
Rome — 3%
Berlin — 2%

The remaining probability is spread across thousands of other possible tokens.

But the training text tells us the actual next token is:

Paris

The model therefore made a poor prediction.

It gave Paris only five percent probability while favoring London.

Now training needs to measure how poor that prediction was.

Cross-Entropy: The Language Model's Loss

For next-token prediction, language models commonly use a form of cross-entropy loss.

For a single correct token, the simplified expression can be written as:

Again, translate before calculating.

$P_{\text{correct}}$ means:

the probability the model assigned to the token that actually came next

In our example:

Five percent written as a decimal is:

Substitute it:

The result is approximately:

You do not need to remember 2.996.

The important thing is what the logarithm accomplishes.

A low probability for the correct answer produces a relatively high loss.

Now suppose, much later in training, the same kind of context gives Paris a seventy-percent probability:

Then:

which is approximately:

So:

Paris at 5% → relatively high loss

Paris at 70% → much lower loss

The loss function does not know what Paris is.

It does not know what France is.

It does not understand the political concept of a capital city.

It simply says:

The probability assigned to the correct token should have been higher.

That numerical signal is enough.

Backpropagation can now do the same kind of work we saw in our toy neuron.

How Does “Paris” Actually Reach the Weights?

The prediction did not come from one weight.

The words:

The capital of France is

were first converted into numerical token representations.

Those representations moved through many Transformer layers.

Inside those layers, attention mechanisms and feed-forward networks repeatedly transformed the numbers using learned weight matrices.

Eventually the model produced raw scores for possible next tokens.

These raw scores are called logits.

Imagine a drastically simplified version:

These are not yet probabilities.

They are raw preference scores.

A mathematical operation called softmax converts the logits into a probability distribution whose values add up to one hundred percent.

Perhaps after softmax we get:

London — 18%
Paris — 5%
Lyon — 4%
Rome — 3%

The training data tells us:

Paris should have won.

Cross-entropy gives us the loss.

Backpropagation then works backward through the network, calculating how small changes to the participating parameters would affect that loss.

The optimizer updates them.

Not one giant change.

Huge numbers of microscopic changes.

Then another piece of text arrives.

One Sentence Does Not Teach Geography

The model does not see:

The capital of France is Paris.

once and immediately announce:

“Geography complete.”

Its training data may also contain:

Paris is the capital of France.

France's capital city is Paris.

The French government is based in Paris.

Paris is a major European city.

France is a country in Western Europe.

The Seine flows through Paris.

And enormous numbers of other contexts involving France, Paris, cities, governments, countries, languages, travel, history, and geography.

Each example produces predictions.

Each prediction produces a loss.

Each loss generates gradients.

Each training step slightly modifies parameters.

Over time, overlapping relationships are gradually woven into the network.

That word—woven—is useful.

The fact is not placed into a box marked:

France → Paris

It becomes part of a distributed mathematical fabric.

After Enough Training

Eventually the model sees:

The capital of France is

and perhaps its output distribution now looks more like:

Paris — 92%
Lyon — 1.2%
Marseille — 0.7%
London — 0.3%

Paris now wins easily.

So where is the fact stored?

Not in one weight.

Not in one neuron.

Not in one hidden text record.

The relationship is distributed through the network.

And this is why the model may also answer:

Which city serves as the capital of France?

or:

What is France's capital city?

The exact sentence has changed.

But the underlying relationships among the concepts are still useful.

The model is not simply searching for the precise training sentence.

It is computing through a learned network of numerical relationships.

This is also why the apparently simple task of next-token prediction can lead to surprisingly rich behavior.

To predict language well, it helps to learn grammar.

Meaning.

Facts.

Relationships.

Style.

Structure.

Patterns of reasoning.

The objective is simple:

Predict what comes next.

What the model must learn to become exceptionally good at that task is not.

And ultimately, those learned relationships are reflected in its weights.

Dogs, Cats, and Why “Feature Importance” Can Mislead You

A common explanation of neural-network weights uses image classification.

Suppose a model is trained to distinguish dogs from cats.

It might discover that certain visual patterns involving muzzle shape, ears, face structure, or body shape are more useful than others.

That is useful intuition.

But it should not be taken literally.

A modern network does not normally contain:

Ear weight = 0.84

Fur-color weight = 0.17

Early layers may become sensitive to relatively simple patterns such as edges.

Later layers can combine them into textures and shapes.

Still later representations can respond to increasingly complicated structures.

The final classification emerges from interactions among many layers and many parameters.

So when we say:

the model learned that ear shape matters

we are translating a complicated distributed numerical process into human language.

There is no knob labeled EAR.

2012: The GPU Enters the Story

Neural networks had existed for decades.

Why did their capabilities suddenly accelerate?

One major reason was hardware.

In 2012, Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton demonstrated what became known as AlexNet, a deep image-recognition network containing roughly sixty million parameters.

A crucial ingredient was the use of Graphics Processing Units, or GPUs.

GPUs had largely evolved because computer graphics require enormous numbers of similar mathematical operations to happen in parallel.

Video games needed to process pixels, triangles, lighting, and textures rapidly.

Neural networks looked at that hardware and effectively said:

“That multiplication machine looks useful.”

The gaming industry had accidentally spent years helping develop infrastructure for machine learning.

Somewhere, a virtual dragon deserves partial credit for modern AI.

From Individual Weights to Tensors

Our tiny neuron used individual values such as:

and:

A real network cannot conveniently manage billions of parameters this way.

Instead, the numbers are organized into structures called tensors.

A simple list of numbers can be thought of as a one-dimensional tensor.

A rectangular table of numbers is a two-dimensional tensor, commonly called a matrix.

Higher-dimensional collections are tensors as well.

Large parts of neural-network computation can be expressed in compact forms such as:

Let us translate this one too.

$X$ represents a collection of input numbers.

$W$ represents a matrix of learned weights.

Multiplying:

means combining many input values with many learned weights according to the rules of matrix multiplication.

The result:

is a new collection of transformed numerical values.

The notation looks tiny.

The actual computation may contain millions or billions of individual multiply-and-add operations.

That is why GPUs are so valuable.

They can perform enormous quantities of this arithmetic in parallel.

While the AI appears to be calmly discussing Dostoevsky, the GPU is performing industrial quantities of multiplication.

The conversation looks intellectual.

The hardware sees arithmetic.

2017: Attention Changes the Architecture

In 2017, the paper “Attention Is All You Need” introduced the Transformer architecture.

Transformers became the foundation for most modern Large Language Models.

The architecture introduced a powerful way for different parts of a sequence to interact through attention.

But the arrival of attention did not abolish weights.

Quite the opposite.

Transformers contain large learned matrices throughout the architecture.

Attention uses learned projection matrices.

Feed-forward layers use learned matrices.

Embedding systems contain learned parameters.

The architecture became dramatically more sophisticated.

The basic currency remained numbers.

2020: 175 Billion Becomes a Headline

Then came scale.

In 2020, GPT-3 was introduced with:

175 billion parameters.

Compare that with AlexNet's roughly sixty million parameters only eight years earlier.

The field had moved from millions of learned values to hundreds of billions.

The fundamental idea was still recognizable:

Learn useful numerical parameters from data.

Researchers improved architecture.

Training methods improved.

Datasets grew.

GPU clusters became enormous.

Then, as computing often does, we added an unreasonable number of zeros.

What Does an 8B Model Physically Contain?

Now parameter counts stop being abstract.

They become a physical storage problem.

Suppose a model contains:

parameters.

That means roughly eight billion learned numerical values.

Suppose every parameter is stored using sixteen bits.

Sixteen bits equals:

So the approximate raw parameter storage is:

which gives:

or roughly:

Where did those sixteen gigabytes come from?

Nothing mysterious.

We have eight billion numbers.

Each requires two bytes.

Eight billion times two gives roughly sixteen billion bytes.

Now consider a seventy-billion-parameter model at the same precision.

The calculation is:

which gives approximately:

just for raw parameter storage.

Suddenly:

“70B model”

is no longer merely a model name.

It is a hardware problem.

Your GPU becomes interested.

Your system RAM may become interested.

Eventually your wallet becomes interested.

Quantization: What Are We Actually Shrinking?

Suppose sixteen bits per parameter requires too much memory.

Can we represent the same learned values using fewer bits?

That is the basic idea behind quantization.

Take our hypothetical 8B model.

At sixteen bits per parameter, its idealized raw parameter storage is approximately:

Now suppose we represent many weights using roughly four bits instead.

Four bits is half a byte:

So:

gives approximately:

The important point is what did not happen.

We did not turn an eight-billion-parameter model into a two-billion-parameter model.

The model still has roughly the same number of parameters.

We changed how much numerical precision is used to represent them.

A simple intuition is numerical rounding.

You might store:

or:

The second number contains less precision.

Real neural-network quantization is considerably more sophisticated than simply rounding decimal places. Practical schemes use scaling factors, groups, specialized encodings, and other techniques intended to preserve useful behavior while reducing memory.

But the central physical trade-off remains:

fewer bits per weight → less memory

That is why local-AI users spend surprising portions of their lives debating Q4, Q5, Q6, and Q8.

From a distance it sounds like motor oil.

It is usually about fitting billions of numbers into a graphics card.

Weights Are Not Activations

Another distinction matters enormously when talking about hardware.

Weights are persistent.

They are stored as part of the model.

Turn off the machine.

Load the same model tomorrow.

The learned weights remain.

Activations are temporary.

When an input travels through the network, every layer produces intermediate numerical states.

Those values depend on the current input.

Change the prompt and the activations change.

Think of a factory.

The weights are the installed machinery.

The activations are the material currently moving through that machinery.

This explains why loading model weights into GPU memory is not the entire memory requirement.

The computation itself needs workspace.

Language models may also use substantial memory for a Key-Value Cache, or KV cache, which stores useful intermediate information from previously processed tokens.

As context grows, that cache can grow too.

So:

The weights fit in VRAM

does not automatically mean:

The model can work comfortably in that VRAM.

The sofa may fit through the apartment door.

You would still like somewhere to stand.

Do the Weights Change While You Chat With the Model?

Normally:

No.

When a trained model answers a prompt, it performs inference.

Inference means:

Use the existing learned parameters to calculate an output.

The weights are read.

They are not normally rewritten because you asked a question.

This is fortunate.

Otherwise one sufficiently strange conversation could permanently remodel the network.

A conversation creates temporary state: context, activations, KV cache, tool results, and other working information.

Those are different from the persistent trained parameters.

The simplest summary is:

Training changes weights. Inference uses weights.

That sentence explains an enormous amount of AI.

Fine-Tuning: Changing the Numbers Again

A pretrained model can later receive additional training.

That process is called fine-tuning.

Imagine a general model that already understands language.

You might fine-tune it on:

legal material,

medical terminology,

programming examples,

customer-service conversations,

financial analysis,

or another specialized domain.

Training once again modifies parameters so the model becomes more adapted to the new task.

Full fine-tuning can update huge numbers of parameters and require substantial computing resources.

Other methods, such as Low-Rank Adaptation, or LoRA, train much smaller additional parameter structures while leaving most of the original model unchanged.

The practical distinction is important.

Giving a model a PDF during a conversation normally changes its context.

Fine-tuning changes learned parameters.

One provides temporary information.

The other changes the trained model.

The Model Is Not the Whole System

Suppose two people download exactly the same model.

Same checkpoint.

Same weights.

Same quantization.

One builds an excellent research system.

The other produces unreliable answers with beautiful formatting.

How can that happen?

Because the model is only one component.

Around it may be prompts, retrieval systems, databases, memory, tools, browsers, routing, permissions, validation, and application code.

This surrounding software is sometimes called a harness or an agent environment.

The harness normally does not rewrite the model's weights every time a request arrives.

It controls what information reaches the model, which tools the model can use, and what happens to the result.

Same weights.

Different environment.

Different behavior.

The engine may be identical.

The vehicle does not have to be.

What Is an Open-Weight Model?

During training, parameters are constantly changing.

At useful points, developers save their current state.

That saved state is called a checkpoint.

A checkpoint is essentially a snapshot of the model's learned numerical state at a particular point.

If those trained parameter files are made available for others to download, the model is commonly described as open-weight.

Depending on the license, people may be able to run it locally, quantize it, fine-tune it, study it, integrate it into products, or redistribute derived versions.

But there is an important terminology distinction.

Open-weight does not automatically mean open-source.

A developer may release weights while keeping the training dataset, parts of the training code, the complete training recipe, or other components private.

Licenses may also impose restrictions.

So these are two different questions:

Can I download the trained weights?

and:

Is the entire model-development process open?

The answers may not be the same.

Words matter.

Especially after lawyers arrive.

Why Open Weights Changed the Game

Training a powerful foundation model from scratch can require enormous resources:

data, GPU clusters, electricity, engineering, time, and money.

Downloading trained weights allows somebody else to begin after that expensive training has already happened.

You receive the learned numerical state.

You do not reproduce the entire education.

For researchers, that enables study.

For developers, it enables local deployment and experimentation.

For organizations, it can provide more control.

And for hobbyists and technically curious users, it creates something that would have sounded extraordinary not long ago:

A model trained using industrial-scale infrastructure can eventually arrive as files on a computer in your house.

The training cluster may occupy a data center.

The finished checkpoint may end up on an SSD under your desk beside three USB cables nobody remembers buying.

That is quite a journey for a tensor.

What Weights Are Not

After all of this, several misconceptions become easier to remove.

Weights are not a database.

There is no row containing:

capital_of_france = Paris

A single weight is not normally a human-readable concept.

Finding:

inside a tensor tells you almost nothing by itself.

Weights normally do not change during ordinary inference.

Training modifies them.

Inference uses them.

One weight does not generally equal one feature.

Meaning is distributed across many parameters and layers.

Parameter count does not equal intelligence.

Architecture matters.

Training data matters.

Optimization matters.

Post-training matters.

Context handling matters.

The surrounding environment matters.

An eighty-billion-parameter idiot would still be an idiot.

It would simply require a more expensive GPU.

Weights vs. Parameters

People often use the words weight and parameter almost interchangeably.

For ordinary conversation, that is usually fine.

Technically, parameter is the broader term.

Weights are parameters.

Biases, normalization values, and other trainable values may also be parameters.

So when somebody says:

“This is a 70B model,”

the more precise meaning is:

The model contains roughly seventy billion learned parameters.

Most of those parameters exist inside large learned matrices.

People will continue calling them weights.

Civilization will probably survive.

What Does It Mean When We Say a Model “Knows” Something?

Now mathematics begins touching philosophy.

A language model does not contain a miniature librarian searching shelves of facts.

There is no tiny SQL database hidden inside the Transformer.

Instead, training shapes billions of numerical relationships so that particular input patterns tend to produce useful output patterns.

Ask:

The capital of France is—

and the learned network strongly favors:

Paris.

Ask for Python code and another pattern of internal activity appears.

Ask about medieval history and another.

The same persistent weights participate in all of these behaviors.

So when we say a model knows something, mathematically we mean something closer to:

Its learned parameters have reached a configuration that produces behavior consistent with having learned that information or capability.

Whether that deserves the same word knowledge that we use for human beings is a much larger philosophical question.

Probably best attempted after dinner.

The Whole Story in Two Diagrams

For readers who want the technical skeleton without repeating the entire journey, inference can be summarized as:

In ordinary language:

Give the trained network an input.
Pass it through layers containing learned weights.
Generate temporary internal states.
Produce an output.

Training adds a second loop:

In ordinary language:

Make a prediction.
Measure how wrong it was.
Calculate which parameter changes should reduce the error.
Update those parameters.

Then repeat.

And repeat.

And repeat.

Or, in the shortest technically useful version:

Inference uses the numbers. Training changes the numbers.

Everything else is architecture, data, optimization, hardware, and scale.

Admittedly, those four subjects have kept quite a few researchers employed.

Why You Should Care Even If You Never Train Anything

Most people running local AI will never calculate a gradient by hand.

That is fine.

Most people driving cars do not machine their own crankshafts either.

But understanding weights explains many of the things a technically curious AI user eventually encounters.

Why is an 8B model several gigabytes?

Because billions of learned numerical parameters must physically be stored.

Why does Q4 require much less memory than sixteen-bit precision?

Because fewer bits are used to represent many of those parameters.

Why can quantization slightly change behavior?

Because numerical precision changed.

Why can fine-tuning change a model?

Because additional training changes parameters.

Why does uploading a PDF not permanently teach the model?

Because the PDF changes the current context, not the persistent weights.

Why can two systems using identical model weights behave very differently?

Because the model is only one part of the environment around it.

And why can somebody download a trained AI model at all?

Because much of what training produced can be preserved as enormous collections of learned numerical parameters.

The idea began with tiny mathematical neurons.

Then came trainable perceptrons.

Backpropagation made multilayer learning practical.

GPUs supplied extraordinary amounts of parallel arithmetic.

Transformers reorganized how language could be processed.

Parameter counts grew from millions to billions and beyond.

The scale changed almost beyond recognition.

The central loop remained surprisingly recognizable:

Make a prediction.

Measure the mistake.

Calculate what should change.

Change the numbers a little.

Try again.

Do this enough times, with enough data, enough mathematics, enough hardware, and enough electricity—

and eventually the numbers start talking back.