Open weights run on your own machine

Here you put a model on your own machine and turn it into your first agent: an agent with personality, four sentences typed into a running model, one saying who you are, three giving it a character and a job. It is prompting at its simplest, and it is meant to be played with, one sentence at a time, until you meet the limits a personality alone cannot cross. Our next lesson picks up exactly there, with an agent given a purpose and written in named blocks. The plumbing beneath all of it, how one configuration line routes a request to your own processor, is kept whole in Getting started with an open model for whenever you need it.

One statement about this module’s numbers and pictures, made once. The transcript in this module is verbatim from qwen3:8b run on the laptop the lesson was written on. The memory figures (about 5 GB on disk, 5 to 6 GB to run an 8B model, about 10 GB for 14B, and 16 GB at sixteen bits) are approximate and depend on the quantization chosen. The token counts and percentages in the final figure (~100, ~900, and ~3,000 tokens read; 60%, 7%, and 2%) are shaped to show a fixed prompt’s share of the window falling, and are measured from no run.

open weights run on your own machine

a model is a downloadable file of trained weights

A model is a large set of trained numerical parameters called weights, fixed the day training ended. Those numbers are the whole of the model: its language ability, its apparent knowledge, its habits of phrasing, all of it held in one file on your disk.

An open-weight model is that set of weights, published as a file you can download.

Take the concreteness seriously, because it is the whole argument of this module. Eight billion weights, at the sixteen bits apiece the weights are published in, fill about 16 GB, which is more memory than most laptops will hand to one program. Local runtimes store each number on a coarser grid instead, near four bits rather than sixteen, a step called quantization, and the same model arrives as a file of about five gigabytes. The rounding introduces a small error into every weight, and it is the reason the model fits on your laptop at all. Four bits is an average across the file: the common schemes hold some values at higher precision and store a scale for each block of weights. That file sits on your disk next to your photos. You can copy it to a USB stick, keep it in a drawer for ten years, and run it on a laptop with no internet connection.

What exactly is being published? The trained weights, usually with a license governing use. That is what “open weights” names. The training data and the full training code are typically kept back, and “open source” would mean those were published too. You get the finished model, and you do not get the recipe. The distinction is worth having before someone corrects you in a meeting.

Three things follow, and each one is practical rather than ideological:

• Your data stays put. When an agent reads patient files, internal code, or unreleased financials, those tokens go into memory on your own machine and nowhere else. For a great many organizations this one property decides whether an AI project is allowed to run at all. The guarantee is exactly as wide as the model line: the moment any agent in the system names a cloud model, whatever it sends leaves your network with it.
• Iteration carries no token fees. Prompt engineering is a loop: run, read, adjust one thing, and run again. On a metered API every turn of that loop is billed per token, so people run fewer tests than they should. On local weights you pay for the electricity and the hardware you run it on, with no metered tokens, so you can put one prompt through two hundred awkward inputs and read every result.
• The model does not move under you. The weights file on your disk holds the same weights next year that it holds tonight, and nobody can replace or retire it but you. Cloud models are updated and retired on the vendor’s schedule, and an update that improves average quality can still change behavior your prompt depended on.

choosing a model balances capability against memory

An 8-billion-parameter model needs roughly 5 to 6 GB of memory to run at a comfortable speed, in RAM or in the VRAM on a graphics card; a 14B model needs around 10 GB. Frontier cloud models are hundreds of billions of parameters and will not fit on your laptop under any arrangement, and a local 8B model has genuinely less reasoning capacity than they do. For learning this craft that gap is an asset: a smaller model does what you told it and little more, so a vague instruction produces a visibly wrong result instead of being quietly repaired by a larger model’s tolerance. Prompts that hold on a small local model have survived a hard reviewer.

The instinct is to reach for the largest model that fits. Memory is finite, every extra weight is arithmetic on every token, and the task has a short list of things it needs the model to do. Once a model can do everything on that list, each further parameter adds arithmetic to every token, slows generation and takes memory, while the answers stay where they were:

The capability floor: deploy the smallest model that clears the minimum capability the task requires. Parameters beyond that threshold consume memory and add latency without improving the outcome.

The floor is a list of abilities, not a size. Does the agent need to emit a structured request your code can execute (tool calling)? Does it need to read images? Does it need to hold a long document in one pass? The smallest model clearing those requirements leaves the most memory for everything else you are running.

Model	Capabilities	Practical role
qwen3:8b	text generation, tool calling	The standard local workhorse. Tool calling lets it participate in the agent loop and reach databases and APIs.
qwen3-vl:8b	text, vision, tool calling	The visual specialist, for agents that must read charts, screenshots, or interface diagrams.
qwen3:14b	text, tool calling, larger capacity	More headroom for multi-step logic, at roughly 10 GB of memory.