Which local AI models should you use with Stellary?

A local AI model can prepare cards, summarize documents or contribute to a coding mission. Its usefulness depends on hardware, the execution engine, available tools and the quality of its result on your work.

Select a complete configuration, then test a representative mission. A model name, parameter count and benchmark score do not establish that an agent will work in Stellary.

A local model and a local application differ

A local model generates responses on a machine or server you control. An application installed on your computer may call a remote API. Keep those choices separate in your configuration.

Stellary Desktop can connect engines on a computer and make their capabilities available to agents according to configuration. The workspace, shared documents and collaboration history can still depend on online services. Local inference does not establish entirely offline operation.

The local, cloud and hybrid guide covers processing location. This article focuses on choosing a model and verifying its use in a mission.

Candidates worth comparing

These candidates cover different sizes and specializations. They are not a list of validated Stellary configurations. Engine version, quantization and the selected tool contract need checking during the trial.

Large open-weight models such as Mistral Large 4 belong in the wider self-hosting discussion too. Open weights do not imply suitability for an ordinary computer. Hardware and operation remain deployment decisions; consult the weight release for that model.

Start with useful work

Choose something you can review. Classify requests against a known set of examples. Compare an extraction with its source documents. Check a code change through an actual journey and appropriate technical checks.

Include an easy input and one that exposes a real difficulty: incomplete material, conflicting facts, long text or an unavailable tool. A useful configuration should either complete the task correctly or explain what prevents completion.

A first Stellary trial could draft a project status report from selected cards without changing or distributing it. Review facts and sources. A coding trial requires a different environment and scope.

Check hardware and quantization

Available memory needs to accommodate weights, context, the inference engine and other work. Context length and concurrent executions affect requirements. Download size alone does not describe total memory use.

Quantization reduces weight size with a tradeoff to evaluate on actual outcomes. Two files carrying the same family name can behave and perform differently. Record the exact variant instead of recommending a family without configuration.

On a shared workstation, observe responsiveness of the other applications. Inference that slows the rest of the team's work can erase the benefit. Measure representative use before deciding on dedicated hardware.

Separate text generation from tool-enabled agency

A server answering a chat request does not prove it can apply mission permissions and use the required tools. An engine may support some operations while refusing others.

Stellary's LM Studio, Ollama and other runner paths have their own capabilities. A tool-enabled mission needs a contract compatible with the selected engine and model. A configuration refusal is not an invitation to remove controls.

The local AI apps guide describes connection. Verify a simple configuration first, then a mission with the necessary tools. Agent permissions remain part of what the trial must establish.

Prove the complete Stellary path

1. The local engine is reachable on the connected machine and the model is identified.
2. The model is shared through the intended Desktop flow.
3. It appears in the workspace catalog and can be selected for the agent.
4. The mission actually uses that configuration with the permitted access.
5. The result returns to Stellary for review from its card.

Successful discovery in Desktop alone does not establish the following steps. If the model is missing from agent configuration, inspect sharing and the catalog before treating local setup as complete.

Preserve the exact model identity used in the trial. Changing a server or configuration can make a selection refer to something different. Check the available execution record and mission outcome.

Compare configurations consistently

Provide the same inputs and criteria to each configuration. Record time to an accepted result, errors, retries and human effort. Generation speed alone does not indicate useful work.

Local operation includes hardware and upkeep. It may be worthwhile for recurring tasks, but its cost does not become zero because each response lacks a separate API bill.

Use a hybrid workflow when it helps

A local model might handle a documentary step while another engine performs a difficult investigation. Keep the handoff explicit: which information leaves the computer and which service receives it?

This division depends on verified capabilities. It should not be described as universal automatic routing. The supervisor workflow guide explains how to define role inputs and outputs.

Give users a clear outcome when the engine is unavailable. Quietly switching from local inference to a remote service changes a central condition of the choice. People should understand what can be retried and under which configuration.

Choose something the team can maintain

Document the installation, updates and person responsible for troubleshooting. Keep reference tasks to check behavior after a model or engine changes.

A useful local configuration completes particular missions at acceptable quality on suitable hardware, with behavior the team understands. Compare that demonstrated usefulness against the cloud option rather than assuming every remote model can be replaced.

Keep a small quality reference

Preserve inputs from a few accepted missions and the criteria used in review. After an update, repeat them without simultaneously changing model and source material. This helps distinguish engine behavior from changes in project content.

Include a task that should remain blocked when information is missing. A model inventing an answer to finish quickly can look efficient on ordinary cases while creating expensive correction work. Asking a precise question belongs in the evaluation.

For team use, inspect what happens when the machine shuts down or the model is unloaded. The work queue and user message should make waiting understandable. An individual workstation has different availability constraints from a shared service.

Match an engine to the actual mission

Start with the least demanding configuration that completes your reference tasks correctly. A documentary mission may need only read access and text output. A coding mission needs a repository, commands and a check of the resulting behavior. Treat those as different trials.

If an engine supports chat but cannot enforce the required mission tools, choose a compatible path or reduce the task to its actual capability. Do not present text generation as a successful end-to-end agent integration.

Record configuration changes alongside results. This makes recommendations useful for another person who wants to reproduce them and prevents a later update from silently inheriting evidence collected under an older setup.

Keep the observations from a first trial

Download the local or cloud evaluation worksheet and record hardware, quantisation, runtime and exact model. The candidates in this article come from vendor model cards; they are not a list of configurations all tested in Stellary.

For an external client, the LM Studio guide describes an MCP connection; that differs from sharing a model through Stellary Desktop. Check which path your task uses before interpreting its result.