Ollama Local Setup and Cheat Sheet / survival guide
ollama <command> [model]
ollama run llama3.2
Ollama runs large language models (LLMs) on your own computer. Download a model once, then chat
with it offline, for free, with your data staying on your machine. It handles model downloads,
GPU use and memory for you, and exposes a local web API (and an OpenAI-compatible one) so your
own programs and tools can use it too. It runs on Linux, macOS and Windows.
Local models are smaller and slower than the big hosted ones, but they are private, cheap
and yours. Model names and sizes change fast. Browse ollama.com/library for what exists today.
What you need
- A computer with 8 GB of RAM or more (16 GB is comfortable). More memory means bigger, smarter models
- Free disk space: about 2 to 5 GB for a small model, 40 GB or more for a large one
- Optional but faster: a GPU (NVIDIA, AMD, or Apple Silicon). Without one, models run on the CPU, only slower
- Linux, macOS 12+ or Windows 10/11 (or WSL2)
Rough guide (4-bit quantized models):
1-3B parameters about 1 to 2.5 GB Any modern PC, even a Raspberry Pi 5. Fast, basic answers
7-8B about 4.5 to 6 GB 8 GB RAM minimum. A good general starting point
12-14B about 8 to 10 GB 16 GB RAM
27-34B about 17 to 22 GB 32 GB RAM or a 24 GB GPU
70B about 40+ GB 64 GB RAM or several GPUs. Slow on CPU
Memory needed is the model file size plus room for the context (conversation) and your operating system.
Install
Linux
curl -fsSL https://ollama.com/install.sh | sh
Read the script first if you prefer: curl -fsSL https://ollama.com/install.sh | less
It installs the program and creates a systemd service that starts at boot.
ollama --version
systemctl status ollama # is the background service running?
Manual install (no script): download the tarball from ollama.com/download, extract it to /usr, then create a service.
macOS
Download the app from ollama.com/download and open it, or: brew install ollama
The menu bar app runs the server. With Homebrew: brew services start ollama (or run: ollama serve)
Apple Silicon (M1 and newer) uses the GPU automatically and shares memory with the system.
Windows
Download the installer from ollama.com/download and run it (or: winget install Ollama.Ollama)
It runs in the system tray and starts the server. Use PowerShell or Command Prompt for the commands.
Or use WSL2 and follow the Linux steps.
Docker
docker run -d --name ollama -v ollama:/root/.ollama -p 11434:11434 ollama/ollama
docker run -d --name ollama --gpus=all -v ollama:/root/.ollama -p 11434:11434 ollama/ollama # NVIDIA GPU (needs the NVIDIA Container Toolkit)
docker exec -it ollama ollama run llama3.2
Run your first model
ollama run llama3.2 # downloads it the first time (about 2 GB), then starts a chat
>>> Explain what a load balancer does in two sentences.
>>> /bye # leave the chat
The first run pulls the model. After that it starts in seconds. Type your question at the >>> prompt.
Everyday commands
ollama run MODEL Chat with a model (downloads it if missing)
ollama run MODEL "your prompt" One-shot: print the answer and exit
ollama pull MODEL Download or update a model
ollama list Models on your disk (also: ollama ls)
ollama ps Models loaded in memory now, and whether they use GPU or CPU
ollama stop MODEL Unload a model from memory
ollama rm MODEL Delete a model from disk
ollama show MODEL Details: size, parameters, quantization, context length, license
ollama show MODEL --modelfile Print the Modelfile it was built from
ollama cp MODEL NEWNAME Copy a model (to customize it)
ollama create NAME -f Modelfile Build a custom model
ollama serve Start the server by hand (if it is not already running as a service)
ollama help / ollama COMMAND --help
Pick a size with a tag: ollama run llama3.1:8b ollama run qwen2.5:14b ollama run llama3.2:1b
Pick a quantization: ollama run llama3.1:8b-instruct-q4_K_M (smaller = q4, higher quality = q8)
Use a file or pipe as input:
ollama run llama3.2 "Summarize this file" < notes.txt
cat error.log | ollama run llama3.2 "What is wrong here?"
ollama run llama3.2 "Write a bash script that backs up /etc" > backup.sh.txt
Choosing a model (examples; check ollama.com/library for the current list)
llama3.2 (1B, 3B) Small, fast, runs anywhere. Good for simple tasks and low-memory machines
llama3.1 (8B) A solid all-round model for 8 to 16 GB machines
qwen2.5 / qwen3 Strong at reasoning and multilingual work, many sizes
gemma3 Google's open models in several sizes, good quality per size
mistral A popular efficient 7B model
phi4 / phi3 Small Microsoft models, good for their size
deepseek-r1 Reasoning-style model that "thinks" before answering (slower)
gpt-oss (20b) OpenAI's open-weight model. Needs about 16 GB of memory
qwen2.5-coder / codellama Code-focused models for programming help
llava / llama3.2-vision Models that can look at images
nomic-embed-text Embeddings for search and RAG (turns text into vectors, not a chat model)
Start small (3B to 8B). If the answers are not good enough and you have the memory, step up to a larger one.
Read a model's page for its size, license and recommended settings.
Chat commands (inside ollama run)
/? Help /bye Exit
/set parameter temperature 0.2 Change a setting for this chat (lower = more focused, higher = more creative)
/set parameter num_ctx 8192 Change the context window (how much it can remember)
/set system "You are a Linux expert. Be brief." Set the system prompt
/show info Model details /show system Show the system prompt
/save mychat Save this session as a model /load mychat Load it later
/clear Clear the conversation context
""" Start and end a multi-line message with triple quotes
Press Ctrl+D to send EOF and exit, and Ctrl+C to stop a long answer.
Where the API lives
Ollama listens on http://localhost:11434 (only on your own computer by default).
curl http://localhost:11434/api/tags # list models
curl http://localhost:11434/api/generate -d '{
"model": "llama3.2",
"prompt": "Why is the sky blue? Answer in one sentence.",
"stream": false
}'
curl http://localhost:11434/api/chat -d '{
"model": "llama3.2",
"messages": [
{"role": "system", "content": "You are a helpful sysadmin."},
{"role": "user", "content": "How do I check disk usage by folder?"}
],
"stream": false
}'
curl http://localhost:11434/api/embed -d '{"model": "nomic-embed-text", "input": "hello world"}'
curl http://localhost:11434/api/pull -d '{"model": "llama3.2"}'
Without "stream": false you get a stream of JSON lines, one per chunk. Pipe to jq: ... | jq -r '.response'
OpenAI-compatible endpoint (many apps and libraries can use it unchanged):
curl http://localhost:11434/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "llama3.2",
"messages": [{"role": "user", "content": "Hello!"}]
}'
Use base URL http://localhost:11434/v1 and any dummy API key (for example "ollama") in tools that ask for an OpenAI-style setup.
Use it from Python
pip install ollama
import ollama
r = ollama.chat(model="llama3.2", messages=[{"role": "user", "content": "Give me 3 Linux tips."}])
print(r["message"]["content"])
Using the OpenAI library:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
r = client.chat.completions.create(model="llama3.2", messages=[{"role": "user", "content": "Hello"}])
print(r.choices[0].message.content)
JavaScript: npm install ollama then import ollama from "ollama"; await ollama.chat({ model: "llama3.2", messages: [...] })
Customize with a Modelfile
# Modelfile
FROM llama3.2
PARAMETER temperature 0.3
PARAMETER num_ctx 8192
SYSTEM """You are a concise Linux sysadmin assistant. Give commands first, then a one-line explanation."""
ollama create sysadmin -f Modelfile
ollama run sysadmin
Useful parameters: temperature (creativity), num_ctx (context length), top_p, top_k, repeat_penalty, num_predict (max tokens), stop.
Bigger num_ctx uses more memory. If long conversations get "forgotten", raise num_ctx (the default context is fairly small).
Add a friendly web interface (Open WebUI)
docker run -d --name open-webui -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data ghcr.io/open-webui/open-webui:main
Open http://localhost:3000, create the first (admin) account, and pick a model. If it cannot find Ollama, set the Ollama URL to
http://host.docker.internal:11434 in its settings. Other clients: Msty, Chatbox, LM Studio-style apps, and editor plugins such as Continue.
Never expose Open WebUI to the internet without accounts, HTTPS and a firewall.
Settings (environment variables)
OLLAMA_HOST=127.0.0.1:11434 Address and port to listen on. 0.0.0.0 makes it reachable from other computers (see security)
OLLAMA_MODELS=/data/ollama/models Where models are stored
OLLAMA_KEEP_ALIVE=30m How long a model stays loaded after use (default 5 minutes; -1 keeps it forever)
OLLAMA_NUM_PARALLEL=2 Requests handled at the same time per model
OLLAMA_MAX_LOADED_MODELS=2 Models kept in memory at once
OLLAMA_CONTEXT_LENGTH=8192 Default context length (newer versions)
OLLAMA_DEBUG=1 Verbose logs
OLLAMA_ORIGINS=... Allowed browser origins for web apps that call the API
Set them for the Linux service:
sudo systemctl edit ollama
[Service]
Environment="OLLAMA_HOST=127.0.0.1:11434"
Environment="OLLAMA_MODELS=/data/ollama/models"
Environment="OLLAMA_KEEP_ALIVE=30m"
sudo systemctl daemon-reload && sudo systemctl restart ollama
macOS: launchctl setenv OLLAMA_HOST "127.0.0.1:11434" then restart the app. Windows: set the variable in System Properties > Environment Variables, then restart Ollama.
If you move OLLAMA_MODELS, the "ollama" service user must be able to read and write that folder.
GPU support
NVIDIA Install the current driver (check with nvidia-smi). CUDA libraries come with Ollama
AMD Supported on many cards through ROCm; the list of supported GPUs is on the Ollama docs
Apple Metal is used automatically on Apple Silicon
Intel Some support through Vulkan or community builds; check the docs
Check what is happening:
ollama ps # the PROCESSOR column shows 100% GPU, 100% CPU, or a split
nvidia-smi # GPU memory and load while a model runs
journalctl -u ollama -f # the log says which GPU was found, or why not
If a model is larger than your GPU memory, part of it runs on the CPU and gets slower. Use a smaller model or a smaller quantization.
Files and storage
Linux (service) /usr/share/ollama/.ollama/models (runs as the "ollama" user)
Linux (user) ~/.ollama/models
macOS ~/.ollama/models
Windows C:\Users\YOU\.ollama\models
Models are big. Check space: du -sh ~/.ollama/models Remove ones you do not use: ollama rm MODEL
Import your own model files (GGUF): put "FROM /path/model.gguf" in a Modelfile, then ollama create mymodel -f Modelfile
Run it on a server (and stay safe)
Ollama has no login. Anyone who can reach the port can use your GPU and your models.
Keep it on 127.0.0.1 and reach it through an SSH tunnel:
ssh -L 11434:localhost:11434 user@server then use http://localhost:11434 on your own computer
If you must listen on the network (OLLAMA_HOST=0.0.0.0):
- Firewall the port to trusted addresses: sudo ufw allow from 10.0.0.0/24 to any port 11434 proto tcp
- Or put a reverse proxy in front (nginx, Caddy, Traefik) with authentication and HTTPS
- Never publish port 11434 on a public IP
Free-tier cloud VMs (about 1 GB of RAM) are too small for useful models. Use a machine with at least 8 GB of RAM, or a GPU instance, and stop it when finished so it does not keep costing money.
Practical uses
# Explain a command or an error
ollama run llama3.2 "Explain this command line: tar -xzvf a.tar.gz -C /opt --strip-components=1"
# Summarize a log file
tail -200 /var/log/syslog | ollama run llama3.2 "Summarize any problems in these logs. Be brief."
# Draft a script and review it before you run it
ollama run qwen2.5-coder "Write a bash script to rotate logs older than 7 days. Explain each line."
# A shell helper function (add to ~/.bashrc)
ask() { ollama run llama3.2 "$*"; }
ask how do I find the largest files on this disk
# Summarize a document
ollama run llama3.2 "Summarize this in 5 bullet points" < README.md
# Keep a model loaded for quick answers, or free the memory again
ollama run llama3.2 --keepalive 1h "hi"
ollama stop llama3.2
# Try a different model on the same question and compare
for m in llama3.2 qwen2.5:7b gemma3; do echo "== $m"; ollama run $m "What is DNS in one sentence?"; done
# List what is loaded and how much memory it uses
ollama ps
Update and uninstall
Update models: ollama pull llama3.2 Update Ollama: run the install script again (Linux), or use the app's update (macOS, Windows)
Docker: docker pull ollama/ollama && docker rm -f ollama && (run the docker run command again; the volume keeps your models)
Uninstall on Linux:
sudo systemctl disable --now ollama
sudo rm /etc/systemd/system/ollama.service
sudo rm $(which ollama)
sudo rm -r /usr/share/ollama # removes downloaded models
sudo userdel ollama; sudo groupdel ollama
macOS / Windows: remove the app, and delete the .ollama folder if you want the models gone too.
Common problems
"Error: could not connect to ollama app" The server is not running: systemctl start ollama, open the app, or run ollama serve
Port 11434 already in use Another Ollama is running: ps aux | grep ollama, or change OLLAMA_HOST
Very slow answers It is running on the CPU. Check ollama ps and nvidia-smi. Use a smaller model or quantization
"model requires more system memory" The model is too big for your RAM or GPU. Choose a smaller one (for example 3B instead of 8B)
Computer freezes or swaps a lot Not enough memory: use a smaller model, close other programs, and lower num_ctx
GPU not detected Update the driver, restart the service, and read journalctl -u ollama. Docker needs the container toolkit
Download stalls or fails Check the network and disk space. Run ollama pull again: it resumes
Forgets earlier messages / cuts long input Raise num_ctx (with /set parameter num_ctx 8192 or a Modelfile). This uses more memory
Answers are poor Small models have limits: try a larger model, a lower temperature, and a clearer prompt
Web app cannot reach the API Set OLLAMA_ORIGINS, and check OLLAMA_HOST and the firewall
Docker container cannot reach Ollama on host Use http://host.docker.internal:11434, or run both on the same Docker network
Models directory ran out of space Move it with OLLAMA_MODELS to a bigger disk, and fix folder permissions
Tips
Start with a small model (llama3.2 3B or an 8B model) and move up only if you need to.
Run ollama ps to see whether the model is on the GPU. GPU is far faster than CPU.
More RAM or VRAM beats a faster CPU. Model size is the main limit, so pick a quantization that fits.
Raise num_ctx only when you need long inputs, because it costs memory.
Keep the server on localhost. It has no password, so use an SSH tunnel or an authenticated proxy for remote use.
Put your favorite settings and system prompt in a Modelfile so you can recreate them anywhere.
Local models can be wrong just like hosted ones. Check anything important, and never run a script you have not read.
Use local models for private material (notes, logs, client files) and a hosted AI for the latest facts and the hardest problems.
Stop models you are not using (ollama stop) and remove ones you do not need (ollama rm) to save memory and disk space.
Model names, sizes and settings change quickly. See ollama.com/library and github.com/ollama/ollama for the current details.