--- title: "How to Install Ollama and Run LLMs on Ubuntu 24.04" description: "Install Ollama on Ubuntu 24.04, download and run open language models, call the REST API, tune the systemd service and add a web chat interface with Open WebUI." category: "AI and ML Infrastructure" language: en source: https://cubepath.com/docs/ai-and-ml-infrastructure/ollama-installation-local-llm-deployment updated: 2026-09-25 --- # How to Install Ollama and Run LLMs on Ubuntu 24.04 Ollama is an open source runtime that downloads, manages and serves large language models such as Llama, Gemma, Qwen and Mistral on your own hardware. It exposes a simple CLI and an HTTP API, including an OpenAI-compatible endpoint, so existing tools can talk to it with minimal changes. In this tutorial you will install Ollama on Ubuntu 24.04, run your first model, use the API, configure the service, and add Open WebUI as a private chat interface. ## Prerequisites To follow this guide you need: - A server running Ubuntu 24.04 LTS, for example a CubePath VPS or bare metal server. - A non-root user with `sudo` privileges. - Enough memory for the models you plan to run. As a rough guide for 4-bit models: 8 GB of RAM for 3B to 8B models, 16 GB for 12B to 14B models, and 48 GB or more for 70B models. - Disk space for the models: a 3B model takes about 2 GB, an 8B model about 5 GB, a 70B model about 40 GB. - Optional: an NVIDIA GPU with the driver installed (`nvidia-smi` works). Ollama runs on CPU without one, only more slowly. - Optional, for Step 6: Docker Engine installed. Want your models and agents running 24/7? Get a [VPS for AI agents](/vps-ai-agents). ## Step 1 - Installing Ollama Ollama's supported Linux installation is its install script. It downloads the binaries, creates an `ollama` system user and a systemd service, and detects NVIDIA or AMD GPUs. Download the script first so you can read what it does before running it: ```bash curl -fsSL https://ollama.com/install.sh -o ollama-install.sh less ollama-install.sh ``` When you are satisfied, run it: ```bash sh ollama-install.sh ``` The script uses `sudo` for the steps that need root. At the end it should report that the API is available at `127.0.0.1:11434`, and whether it found a GPU. Verify the binary and the service: ```bash ollama -v systemctl status ollama --no-pager ``` ```output ollama version is 0.12.3 ● ollama.service - Ollama Service Loaded: loaded (/etc/systemd/system/ollama.service; enabled; preset: enabled) Active: active (running) since Thu 2026-09-25 11:02:40 UTC; 20s ago ``` The API should answer on localhost: ```bash curl http://127.0.0.1:11434 ``` ```output Ollama is running ``` ## Step 2 - Running your first model Models are pulled from the Ollama library at [ollama.com/library](https://ollama.com/library), where each model has tags for sizes and quantizations. Start with a small model that runs well even on CPU: ```bash ollama run llama3.2:3b ``` The first run downloads the model, then opens an interactive prompt: ```output pulling manifest pulling dde5aa3fc5ff: 100% ▕████████████████▏ 2.0 GB ... success >>> Explain what a reverse proxy is in two sentences. ``` Type `/bye` or press `Ctrl+D` to leave. You can also pass a prompt directly, which is useful in scripts: ```bash ollama run llama3.2:3b "Write a one-line bash command that shows the 5 largest files in /var/log" ``` ## Step 3 - Managing models Download a model without starting a chat: ```bash ollama pull gemma3:4b ``` List the models on disk and their sizes: ```bash ollama list ``` ```output NAME ID SIZE MODIFIED gemma3:4b a2af6cc3eb7f 3.3 GB 10 seconds ago llama3.2:3b a80c4f17acd5 2.0 GB 3 minutes ago ``` See which models are loaded in memory right now, and whether they run on the CPU or the GPU: ```bash ollama ps ``` ```output NAME ID SIZE PROCESSOR CONTEXT UNTIL llama3.2:3b a80c4f17acd5 3.1 GB 100% GPU 4096 4 minutes from now ``` `100% GPU` means the whole model fits in VRAM. A split such as `45%/55% CPU/GPU` means part of it runs on the CPU, which is much slower; use a smaller model or quantization if speed matters. Inspect a model's parameters, template and license with `ollama show llama3.2:3b`, and delete one you no longer need with `ollama rm gemma3:4b`. ### Creating a custom model with a Modelfile A Modelfile builds a new model on top of an existing one with your own system prompt and parameters. Create one: ```bash nano Modelfile ``` ```text FROM llama3.2:3b SYSTEM """ You are a concise Linux system administration assistant. Answer with the exact commands needed and a one-line explanation of each. """ PARAMETER temperature 0.3 PARAMETER num_ctx 8192 ``` Build and run it: ```bash ollama create sysadmin -f Modelfile ollama run sysadmin "How do I find which process is listening on port 443?" ``` ## Step 4 - Using the REST API Everything the CLI does goes through the HTTP API on port 11434, so any application on the server can use the models. Generate a single completion: ```bash curl http://127.0.0.1:11434/api/generate -d '{ "model": "llama3.2:3b", "prompt": "What is a VPS? Answer in one sentence.", "stream": false }' ``` The response is a JSON object; the generated text is in the `response` field, along with timing data such as `eval_count` and `eval_duration`. For multi-turn conversations, use the chat endpoint and send the message history: ```bash curl http://127.0.0.1:11434/api/chat -d '{ "model": "llama3.2:3b", "messages": [ {"role": "system", "content": "You answer in one short paragraph."}, {"role": "user", "content": "How does a container differ from a virtual machine?"} ], "stream": false }' ``` Ollama also implements the OpenAI Chat Completions API under `/v1`, so SDKs and tools built for OpenAI work by changing the base URL: ```bash curl http://127.0.0.1:11434/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "llama3.2:3b", "messages": [{"role": "user", "content": "Say hello"}] }' ``` With the official `openai` Python package, point the client at Ollama. The API key is required by the library but ignored by Ollama: ```python from openai import OpenAI client = OpenAI(base_url="http://127.0.0.1:11434/v1", api_key="ollama") reply = client.chat.completions.create( model="llama3.2:3b", messages=[{"role": "user", "content": "List three uses of the ss command."}], ) print(reply.choices[0].message.content) ``` ## Step 5 - Configuring the Ollama service The service is configured through environment variables. Set them in a systemd override so they survive Ollama upgrades: ```bash sudo systemctl edit ollama ``` Add the settings between the comment lines at the top of the file: ```ini [Service] Environment="OLLAMA_KEEP_ALIVE=30m" Environment="OLLAMA_CONTEXT_LENGTH=8192" Environment="OLLAMA_NUM_PARALLEL=2" Environment="OLLAMA_FLASH_ATTENTION=1" ``` What each variable does: - `OLLAMA_KEEP_ALIVE`: how long a model stays in memory after the last request (default 5 minutes). Longer values avoid reload delays. - `OLLAMA_CONTEXT_LENGTH`: the default context window in tokens. Larger windows use more memory. - `OLLAMA_NUM_PARALLEL`: how many requests each loaded model handles at once. Each parallel slot needs its own context memory. - `OLLAMA_FLASH_ATTENTION`: enables flash attention on supported GPUs, which reduces memory use for long contexts. To store models on a different disk, add `Environment="OLLAMA_MODELS=/data/ollama/models"` and give the `ollama` user ownership of that directory with `sudo chown -R ollama:ollama /data/ollama`. The default location for the service is `/usr/share/ollama/.ollama/models`. Apply the changes and check the logs: ```bash sudo systemctl restart ollama sudo journalctl -u ollama -n 30 --no-pager ``` The startup log lists the configuration in effect and the GPUs Ollama detected, which is the quickest way to confirm that GPU acceleration is active. ### Access from other machines By default Ollama listens only on `127.0.0.1`. The API has **no authentication**, so anyone who can reach port 11434 can use your models and download or delete them. The safest way to use it from your workstation is an SSH tunnel: ```bash ssh -N -L 11434:127.0.0.1:11434 your_user@your_server_ip ``` While the tunnel is open, `http://127.0.0.1:11434` on your workstation reaches the server. If other servers must call the API directly, add `Environment="OLLAMA_HOST=0.0.0.0:11434"` to the override, restart the service, and allow only those hosts with UFW: ```bash sudo ufw allow from 203.0.113.10 to any port 11434 proto tcp ``` Replace `203.0.113.10` with the IP of the client server, and never open the port to everyone. ## Step 6 - Adding a web interface with Open WebUI Open WebUI is a self-hosted chat interface for Ollama with user accounts, conversation history and document upload. Run it in Docker using host networking, so the container can reach Ollama on `127.0.0.1` without exposing Ollama: ```bash docker run -d --network=host \ -v open-webui:/app/backend/data \ -e OLLAMA_BASE_URL=http://127.0.0.1:11434 \ --name open-webui --restart always \ ghcr.io/open-webui/open-webui:main ``` With host networking, Open WebUI listens on port 8080 of the server. Follow its startup until it reports that it is ready: ```bash docker logs -f open-webui ``` Do not open port 8080 in the firewall. Reach it through an SSH tunnel from your workstation instead: ```bash ssh -N -L 8080:127.0.0.1:8080 your_user@your_server_ip ``` Open `http://localhost:8080` in your browser and create an account right away: **the first account created becomes the administrator**. Your Ollama models appear in the model selector. For access by a team, publish Open WebUI behind Nginx with HTTPS instead of the tunnel. ## Troubleshooting **`Error: model requires more system memory`.** The model does not fit in the available RAM. Pick a smaller size or quantization tag from the model's page, or reduce `OLLAMA_CONTEXT_LENGTH` and `OLLAMA_NUM_PARALLEL`, which both increase memory use. **The GPU is not used (`ollama ps` shows `100% CPU`).** Confirm `nvidia-smi` works, then look at the startup log with `sudo journalctl -u ollama -b | grep -iE 'gpu|cuda'`. If the driver was installed after Ollama, restart the service with `sudo systemctl restart ollama`. For more detail, add `Environment="OLLAMA_DEBUG=1"` to the override temporarily. **`could not connect to ollama app, is it running?`.** The CLI cannot reach the server. Check `systemctl status ollama`. If you changed `OLLAMA_HOST` in the service, the CLI still defaults to `127.0.0.1:11434`, which is fine for `0.0.0.0`; for any other address, export the same `OLLAMA_HOST` in your shell. **Open WebUI shows no models.** Check that Ollama answers with `curl http://127.0.0.1:11434/api/tags` on the server and that the container was started with `--network=host` and the `OLLAMA_BASE_URL` above. ## Conclusion Ollama now runs as a systemd service on Ubuntu 24.04, serving local models through its native and OpenAI-compatible APIs, tuned through a service override and kept private on localhost, with Open WebUI as a chat front end. Next, you can try larger models on a GPU server, connect coding assistants or automation tools to the `/v1` endpoint, or put Open WebUI behind Nginx with a Let's Encrypt certificate for your team.