Choose local AI by the job, not the download size

The model downloads, loads, and answers a question. That is a useful first milestone. It does not yet tell you whether it can complete the job you need, whether the surrounding application stays offline, or who will maintain the setup next month.

Local AI attracts several different goals: control over data, operation without a dependable connection, predictable access, customization, and experimentation. Those goals can point toward different systems. A private server, a laptop model, and an on-device mobile feature are not interchangeable answers.

The best starting question is narrow: what task should run here, for whom, under which constraints? Once that is written, model size becomes one variable in a test instead of the organizing principle of the project.

Separate the four meanings of open and local

Downloadable weights mean you can obtain model parameters. Open-source AI is a broader claim about freedoms and the materials needed to study and modify a system. The Open Source Initiative’s definition includes requirements concerning code, parameters, and data information. Attribute that definition clearly rather than assuming everyone uses “open” the same way.

Local inference means computation runs on infrastructure you control or on the user’s device. Offline operation means the required workflow can complete without external connectivity. A local model connected to cloud search is local inference inside an online workflow.

Licensing is another separate check. Record the exact model and license terms that apply to your intended use and distribution. A permissive interface application does not automatically grant the same permissions for every model it can load.

These distinctions prevent a common planning error: selecting one attractive property and assuming it supplies all the others. Write the properties you actually require, then verify them individually.

Define the workload in ordinary language

A workload card should describe the input, output, frequency, user, and consequence of failure. “An AI assistant for our team” is difficult to test. “Draft an internal issue summary from one short maintenance note” is concrete.

Specify the largest expected input and the normal input. The extreme case matters for capacity, but a system designed only around the largest file may be needlessly cumbersome for everyday use.

Describe what a good output contains. For a maintenance summary, that might be equipment identifier, observed symptom, actions already attempted, and unresolved questions. The model should preserve uncertainty and avoid inventing a diagnosis.

Also specify how a person uses the output. A draft reviewed by the original author permits a different level of automation from a record sent directly to an external party. DWKD’s lesson on choosing managed or self-run infrastructure helps connect technical control to the responsibility it creates.

Size memory for the running job

A model file’s size is not the application’s total memory requirement. The runtime also needs space for working data, context, and other application components. The operating system and the user’s ordinary applications still need room.

Quantization stores model weights at lower precision to reduce memory requirements, with a possible effect on task quality. The Hugging Face quantization overview describes that tradeoff and shows that methods differ in supported hardware and capabilities.

Do not assume two quantized versions behave identically because their files are similar sizes. Compare the exact candidate, runtime, and settings you intend to use. A model that performs well on a short example may behave differently with a long document or several simultaneous requests.

Record the configuration with the measurements: device, available memory, runtime version, model identifier, quantization, context setting, and concurrency. Without those details, a speed claim is difficult to reproduce and may tell a prospective user little about their own device.

Work through a local drafting decision

Consider an illustrative maintenance team that wants to draft issue summaries from short notes while working in locations with unreliable connectivity. The draft remains on the device until a person reviews it. The task does not require web research.

The team collects synthetic examples that resemble its note structure. Some include missing equipment IDs, uncertain observations, or conflicting dates. A useful model must preserve those gaps rather than complete them from a plausible pattern.

Three options enter the comparison: a simple form with a fixed summary template, a small local model, and a larger local model on more capable hardware. The form is included because it may meet the requirement with fewer moving parts.

The team tests the same cases and records completeness, unsupported details, correction time, and waiting time. This is a proposed evaluation, not a reported result. The winner is whichever option meets the required quality and operating constraints with a maintainable setup.

Create a decision record that survives the demo

A decision table makes the tradeoffs visible. Fill it with measured results or explicit unknowns. Do not replace missing evidence with a favorable adjective such as “fast” or “secure.”

Include a decision date and a review trigger. A new workload, operating-system update, model replacement, or change in data sensitivity may invalidate part of the original evidence.

Inspect the whole data boundary

A local model does not make every connected component local. The interface may synchronize history. A document parser may call a hosted service. A search feature may send a query outside the device. Backups and diagnostic logs can create additional copies.

Ollama’s FAQ distinguishes local operation from its cloud features and documents controls for disabling those cloud features. Read the current behavior for the runtime you actually deploy, then inspect the application around it as well.

Draw a simple inventory: input location, preprocessing, inference, output storage, logs, backups, and optional tools. Assign a destination and an access rule to each. This reveals whether “stays on the laptop” is an observed property or merely an assumption about one component.

Test offline behavior after the setup and model download are complete. A successful offline run is evidence about that tested path. It does not prove every optional feature is offline or that future updates will preserve the same behavior. Document what the test covered.

Measure the experience people will use

Measure startup as well as generation. A rarely used tool that takes a long time to become ready can feel slow even if it generates text quickly once loaded. A frequently used tool has a different balance.

Time the whole task: opening the input, generating, reviewing, correcting, and saving. Tokens per second may help diagnose the runtime, but minutes to an acceptable result determine whether the workflow helps its user.

Test normal working conditions. Leave the ordinary browser, editor, or business application open. Try a longer input and a second request while the first is running if that reflects real use. Record whether the application queues, rejects, slows, or crashes.

Check quality after every configuration change that could affect output. A memory-saving setting can be useful, but its value depends on preserving the required task behavior. Keep a small regression set, such as the structure in DWKD’s evaluation-set resource, so those tradeoffs remain visible.

Set acceptance criteria before selecting a favorite

For the illustrative maintenance-summary tool, proposed quality criteria could require every provided equipment ID to be copied correctly, every missing ID to remain explicitly missing, and no unsupported diagnosis to appear in the draft.

Operational criteria could require the complete workflow to run without external connectivity, recover from an interrupted generation without losing the source note, and remain responsive within a waiting-time limit chosen by the actual users. Measure that limit with those users rather than borrowing a generic benchmark.

Keep an explicit failure policy. If the input is too large, reject it with a useful explanation or ask the user to select a smaller scope. Do not silently discard the end of a note and produce a summary that appears complete.

Report results with the exact test scope. “All twenty synthetic examples preserved the equipment ID” is a claim someone can inspect. “Safe for maintenance work” is far broader than that evidence supports. A pilot decision should match the narrower claim.

Budget for ownership and know when to stop

Local inference can remove some service dependencies while adding responsibilities elsewhere. Someone owns installation, model files, application updates, disk space, compatibility, and user support. Those tasks exist even when the model download is free.

Calculate cost around the expected workload. Include hardware already available versus hardware purchased for the project, setup time, ongoing maintenance, review effort, and downtime. Avoid treating a cloud invoice as the entire cloud cost or a one-time device purchase as the entire local cost.

A local approach may be a poor fit when the required capability exceeds available hardware, the team cannot maintain the stack, or the workload depends heavily on hosted data and services anyway. A managed service may meet the actual requirements more simply if its data handling and availability are acceptable.

A hybrid design can also be reasonable, but make routing explicit. Users should understand which tasks stay local and which require another environment. A hidden fallback that sends a difficult document elsewhere can violate the very reason the local workflow was chosen.

Show the thing

Share the workload card, configuration, synthetic test cases, and decision record. Demonstrate one successful run, one ambiguous input, and one recovery from interruption. If offline operation is a requirement, show the tested path operating with connectivity unavailable.

Publish the limitations beside the result. State which device was tested, which input sizes were covered, and which features were disabled. Another person should be able to determine whether your evidence applies to their environment.

If the simple template wins, show that result too. The useful learning is discovering what the job needs, and the useful build is the smallest system that meets those needs reliably.

Local AI becomes a practical capability when its boundaries are visible. You know what runs where, what quality it achieves on your cases, what happens when it fails, and who can keep it working after the first successful demo.

Source note: OSI, Hugging Face, and Ollama documentation was reviewed on September 23, 2026. Model licenses, runtime behavior, and hardware support can change. The maintenance workflow and evaluation criteria are illustrative; no hardware benchmark or completed deployment is claimed.

Continue the field guides: Build an AI cost model that survives contact with production or The permission boundary belongs outside the AI prompt.

Try this

Write down one task you might run locally, including its typical input, required output, and the longest acceptable wait. List every component the task needs: interface, model, document parser, search index, logs, and optional network tools. Mark where each component runs and whether it can send data elsewhere. Then choose five synthetic examples that would expose a bad result, including one with missing information. You now have a workload card and a small test set. Use them to compare options before buying hardware or assuming that a local model makes the entire workflow private.