Run local AI models with Ollama (and use them from your code)

Install Ollama, pull an open model, chat with it, call it over the REST and OpenAI-compatible APIs, and customize it with a Modelfile. Commands verified against the official Ollama docs.

Cloud models are brilliant, but every prompt leaves your machine and every token costs money. Ollama runs open models locally with one command, then exposes them through an API your apps (and agents) can call. This playbook follows the Ollama README and docs.ollama.com.

Step 1: Pick the right model size for your machine

Model size (measured in billions of parameters, “B”) drives quality, speed and memory use. Rough guidance that holds across hardware:

• 8 GB RAM laptops: stick to small models (a few billion parameters). Fine for summaries and classification.
• 16–32 GB RAM or Apple Silicon with unified memory: mid-size models become comfortable.
• Dedicated GPUs with lots of VRAM: larger models and longer contexts.

Browse ollama.com/library; each model page lists available sizes (tags). Start small. You can always pull a bigger tag later, and a fast small model you actually use beats a slow big one you don’t.

Step 2: Install Ollama

From the official README:

# macOS and Linux
curl -fsSL https://ollama.com/install.sh | sh

# Windows (PowerShell)
irm https://ollama.com/install.ps1 | iex

Manual downloads (a .dmg for Mac, an installer .exe for Windows) and an official Docker image, ollama/ollama, are also available. As with any piped install script, read it first if you’re cautious.

Step 3: Run and chat with a model

Running ollama on its own opens an interactive menu that can run a model or connect Ollama to coding tools. To go straight to a chat, the README’s example is:

ollama run gemma4

The first run downloads the model, then drops you into a chat prompt. Try a real task: paste a messy meeting transcript and ask for decisions and action items. Type /bye to exit. Useful commands: ollama list (installed models), ollama pull <model> (download without chatting), and ollama rm <model> (free disk space).

Step 4: Call the local REST API

While Ollama is running it serves an API on localhost:11434:

curl http://localhost:11434/api/chat -d '{
  "model": "gemma4",
  "messages": [{ "role": "user", "content": "Why is the sky blue?" }],
  "stream": false
}'

"stream": false returns one JSON object instead of a token stream, which is easier for scripts. There are official libraries too: pip install ollama for Python and npm i ollama for JavaScript:

from ollama import chat
response = chat(model="gemma4", messages=[{"role": "user", "content": "Summarize: ..."}])
print(response.message.content)

Step 5: Use the OpenAI-compatible endpoint

This is the trick that makes Ollama useful for builders: it supports a subset of the OpenAI API, so most code written for OpenAI can switch to local models by changing two lines:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11434/v1/",
    api_key="ollama",  # required by the client, ignored by Ollama
)
reply = client.chat.completions.create(
    model="gemma4",
    messages=[{"role": "user", "content": "Say this is a test"}],
)
print(reply.choices[0].message.content)

That means frameworks that accept a custom OpenAI base URL can often run against your laptop for development, then against a hosted provider (OpenRouter, Groq or OpenAI) in production. It’s a “subset”, so test features like tool calling and structured output with your chosen model before relying on them.

Step 6: Customize behavior with a Modelfile

A Modelfile is a small blueprint for a customized model. FROM (required) sets the base; PARAMETER tunes runtime settings; SYSTEM sets a system message. Example for a house-style summarizer:

FROM gemma4
PARAMETER temperature 0.2
SYSTEM You summarize internal documents into 5 bullet points, then list action items with owners. Never invent names.

Build and run it:

ollama create summarizer -f Modelfile
ollama run summarizer

Now your team can call summarizer by name from any script, with the instructions baked in.

Step 7: Tune context length and networking

Two settings trip up most people:

• Context window. Ollama’s FAQ states the default is 4,096 tokens, roughly a few pages. Long documents get silently truncated. Raise it globally with OLLAMA_CONTEXT_LENGTH=8192 ollama serve, inside a chat with /set parameter num_ctx 8192, or per API request with "options": {"num_ctx": 8192}. Bigger contexts use more memory.
• Network access. Ollama binds to 127.0.0.1:11434, so only your machine can reach it. To serve other devices, set OLLAMA_HOST=0.0.0.0:11434 (the FAQ shows how per OS). Only do this on a trusted network or behind a firewall or reverse proxy with authentication, since the API has no login of its own.

Step 8: Decide what stays local

A practical split many teams use:

• Local: classifying and tagging, summarizing internal docs, redacting PII before anything goes to the cloud, offline drafts, development and test runs.
• Cloud: hard reasoning, complex coding, very long contexts, and anything customer-facing where top quality matters.

Ollama also offers optional cloud-hosted models through the same tooling; those send data off your machine, so treat them like any cloud provider.

Pitfalls

• Pulling a model too big for your RAM. It will crawl or fail. Drop to a smaller tag.
• Forgetting the context limit. If summaries ignore the end of a document, raise num_ctx.
• Exposing port 11434 to the internet. There’s no built-in auth. Keep it on localhost or put it behind a proxy with authentication.
• Assuming full OpenAI parity. The compatibility layer is a subset; test your exact features.
• Ignoring licenses. Open models carry their own licenses; check commercial-use terms on each model’s page or on Hugging Face.

Where to go next

Plug your local model into an agent: the OpenAI Agents SDK playbook works the same way once you point it at a compatible endpoint. Or use a local model as the classifier in our n8n inbox triage playbook so email content never leaves your server.

Worked example: a private document summarizer in 15 lines

A consultant wanted to summarize client interview notes without uploading them anywhere. After Step 6 created the summarizer model, this script summarizes every .txt file in a folder:

from pathlib import Path
from ollama import chat

for f in Path("interviews").glob("*.txt"):
    text = f.read_text()
    r = chat(
        model="summarizer",
        messages=[{"role": "user", "content": text}],
        options={"num_ctx": 8192},
    )
    Path(f.with_suffix(".summary.md")).write_text(r.message.content)
    print("done:", f.name)

The num_ctx option matters here: interview notes were longer than the 4,096-token default, and early summaries ignored the second half of each file until it was raised. On a recent laptop, each summary took well under a minute with a small model. Nothing left the machine.

Choosing between local, hosted open models and frontier APIs

Many teams use all three: local for development and sensitive preprocessing, hosted open models for cheap high-volume jobs, frontier models for the hardest steps.

FAQ

Is Ollama free?

Yes. Ollama is open source and running models locally is free apart from your hardware and electricity. Ollama also offers optional cloud models that require signing in.

What port does Ollama use?

By default Ollama listens on 127.0.0.1 port 11434. Change the bind address with the OLLAMA_HOST environment variable.

Can I use the OpenAI Python library with Ollama?

Yes. Point the client at base_url http://localhost:11434/v1/ and pass any api_key value such as 'ollama'; the key is required by the client but ignored locally. Ollama supports a subset of the OpenAI API.

What is the default context window?

Ollama's FAQ states a default of 4096 tokens. Raise it with the OLLAMA_CONTEXT_LENGTH environment variable or the num_ctx parameter, at the cost of more memory.

Are local models as good as GPT or Claude?

Usually not for the hardest reasoning or coding tasks, but small open models are very capable for summarizing, classifying, extraction and drafting, and they keep data on your machine.

Last checked against official docs: October 2026 (Ollama README, docs.ollama.com OpenAI compatibility, FAQ and Modelfile reference). Prices and UI labels change; confirm on each vendor's site.