Ollama

Large Language Model API

Get up and running with Llama 3.2, Mistral, Gemma 2, and other large language models.

Links:

GPU override for Ollama — AMD iGPU inference via the Vulkan backend. Enable by setting GPU_COMPOSE_SUFFIX=amdgpu in config/docker//.env NOTE: deliberately the STANDARD image, not :rocm — colony’s Vega iGPU (Ryzen 7 5825U, gfx90c) has no rocblas kernels in ROCm builds (“dropping ROCm device”), while the standard image ships the ggml Vulkan backend (the :rocm image does not), which supports Vega iGPUs via RADV. OLLAMA_IGPU_ENABLE opts the iGPU into scheduling on Linux. References:

Vulkan backend for AMD/Intel GPUs: Add Vulkan GPU Backend for AMD/Intel Support · Issue #11247 · ollama/ollama
Hardware support matrix (Vulkan path for iGPUs): Hardware support - Ollama

name: ollama
services:
  ollama:
    # GPU acceleration: set GPU_COMPOSE_SUFFIX=amdgpu in host .env to load ollama-amdgpu.yaml (ROCm/AMD)
    image: ollama/ollama:0.34.4
    container_name: ollama
    restart: unless-stopped
    # https://docs.docker.com/config/containers/resource_constraints/#configure-the-default-cfs-scheduler
    # https://batey.info/cgroup-cpu-shares-for-docker.html
    cpu_shares: 512
    volumes:
      - ${DOCKER_VOLUMES}/ollama:/root/.ollama
    networks:
      - proxy
    labels:
      traefik.enable: true
      traefik.http.routers.ollama.middlewares: localaccess@file
      traefik.http.services.ollama.loadbalancer.server.port: 11434
      homepage.group: AI
      homepage.name: "Ollama"
      homepage.icon: ollama.png
      homepage.href: https://ollama.${MYDOMAIN}/
      homepage.description: "Large Language Model API"

networks:
  proxy:
    external: true