LiteLLM LiteLLM

LLM Gateway to call 100+ LLM APIs in OpenAI format

Python SDK, Proxy Server (LLM Gateway) to call 100+ LLM APIs in OpenAI format - [Bedrock, Azure, OpenAI, VertexAI, Cohere, Anthropic, Sagemaker, HuggingFace, Replicate, Groq]

Authentication

Set a strong, randomly generated LITELLM_MASTER_KEY in the ignored config/docker/<hostname>/.env.litellm file and restrict its permissions to 0600. LiteLLM refuses to start without a safe master key. Keep this key separate from the upstream provider API keys and never commit it.

Configure gateway clients with this key as their LiteLLM API key; requests use Authorization: Bearer <key>. Recreate ai/litellm after changing the environment. The local /health/liveliness check does not require authentication.

References

name: litellm
services:
  litellm:
    image: litellm/litellm:v1.106.6@sha256:37e2dafb088a7723beb998590682957386dfc83ffea1848486f8e0e5cfac0b06
    command:
      - --config=/app/config.yaml
      - --telemetry=false
    container_name: litellm
    restart: unless-stopped
    volumes:
      - ./config/config.yaml:/app/config.yaml
    networks:
      - proxy
    environment:
      LITELLM_MASTER_KEY: ${LITELLM_MASTER_KEY:?Set LITELLM_MASTER_KEY in the service environment file}
      ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY:-""}
      OPENAI_API_KEY: ${OPENAI_API_KEY:-""}
      GEMINI_API_KEY: ${GEMINI_API_KEY:-""}
      OPENROUTER_API_KEY: ${OPENROUTER_API_KEY:-""}
      LOCAL_OLLAMA_API_BASE: https://ollama.${MYDOMAIN}
      REMOTE_OLLAMA_API_BASE: http://localhost:11444
    healthcheck:  # Defines the health check configuration for the container
      test:
        - CMD-SHELL
        - python3 -c "import urllib.request; urllib.request.urlopen('http://localhost:4000/health/liveliness')"  # Command to execute for health check
      interval: 30s  # Perform health check every 30 seconds
      timeout: 10s   # Health check command times out after 10 seconds
      retries: 3     # Retry up to 3 times if health check fails
      start_period: 40s  # Wait 40 seconds after container start before beginning health checks
    labels:
      traefik.enable: true
      traefik.http.routers.litellm.middlewares: localaccess@file
      traefik.http.services.litellm.loadbalancer.server.port: 4000
      homepage.group: AI
      homepage.name: "LiteLLM"
      homepage.icon: chatgpt.png
      homepage.href: https://litellm.${MYDOMAIN}/
      homepage.description: "LLM Gateway to call 100+ LLM APIs in OpenAI format"

networks:
  proxy:
    external: true