Skip to content
CodeLifter
Search/ Browse prompts
CodeLifter Forum

How do you run an LLM on your own server for coding?

Staff answerWritten by CodeLifter editors · MCP and tools0 replies

Short answer

To run an LLM on your own server for coding, install an inference server such as vLLM, llama.cpp's llama-server or Ollama on a machine with enough GPU memory, serve an open-weight model through its OpenAI-compatible API, keep that API on your private network behind a key or proxy, then point your coding agent's base URL at it.

Written by CodeLifter editors · Staff, CodeLifter ·
CiteCodeLifter Forum, "How do you run an LLM on your own server for coding?", https://codelifter.com/forum/posts/how-do-you-run-an-llm-on-your-own-server-for-coding/, accessed 2026-11-16.
  1. Staff, CodeLifter ·

    What does running a model "on premise" mean?

    An LLM (large language model) is the program that writes the code; an inference server is the software that loads its weights into memory and answers requests over HTTP. Running it on premise means both live on hardware you control: a workstation under a desk, a rack server or a private cloud machine. Your coding agent, the client that reads files and runs commands, can stay on each developer's laptop and send its requests to that server.

    This differs from the single-laptop setup in the local model thread in one way that matters: the server now listens on a network. Everything in this thread is about doing that safely. If you only need a model on your own machine, the Ollama thread is the shorter path.

    Which server software can you use?

    The three below all offer an HTTP API in the OpenAI format, which is what most coding agents expect. Licences are from the CodeLifter local models page ledger, checked 15 October 2026; behaviour is from each project's documentation, read 16 October 2026.

    Server Licence Start command Listens on by default Built-in API key
    vLLM Apache-2.0 vllm serve <model> http://localhost:8000 --api-key or VLLM_API_KEY
    llama.cpp llama-server MIT llama-server -m model.gguf 127.0.0.1 (set with --host) --api-key or LLAMA_API_KEY
    Ollama MIT ollama serve 127.0.0.1:11434 Not described in its FAQ; use a proxy

    The vLLM quickstart says its server "currently hosts one model at a time" and accepts --host and --port. The llama-server README notes that OpenAI-style function calling, which agents use to edit files and run commands, "is supported with the --jinja flag". The Ollama FAQ changes its address with the OLLAMA_HOST environment variable and shows Nginx as a proxy in front of it.

    How big a server do you need?

    The model must fit in memory, and so must its working space. Use the memory estimate on the local models page for the weights. Then add room for context, the text the model holds per request: the Ollama FAQ warns that parallel requests multiply it, so "a 2K context with 4 parallel requests will result in an 8K context and additional memory allocation." A team server handles several developers at once, so plan memory for the number of parallel requests you allow, not for one.

    Step by step: serve a model with vLLM and connect Codex

    1. Pick a model whose licence allows your use. The local models page lists the licence of each model it covers. Check that it supports tool calling; a model without it can chat but cannot act.
    2. Start the server on a private address. Set the key in the service's environment, not on the command line, where other users of the machine can see it in the process list:
    export VLLM_API_KEY="<long random value>"
    vllm serve <model> --host 10.0.0.5 --port 8000
    
    1. Fence the port. Allow port 8000 only from your developers' network. vLLM's security page says the key protects only paths under /v1, /v2, /inference and /cohere, and lists others, such as /metrics and /pause, that need no key at all.
    2. Test it from a developer machine: curl http://10.0.0.5:8000/v1/models -H "Authorization: Bearer $VLLM_API_KEY".
    3. Point Codex at it. OpenAI's advanced configuration page defines custom providers in ~/.codex/config.toml:
    model = "<model>"
    model_provider = "team_vllm"
    
    [model_providers.team_vllm]
    name = "Team vLLM"
    base_url = "http://10.0.0.5:8000/v1"
    env_key = "VLLM_API_KEY"
    

    The same page documents a wire_api setting for which request format a provider speaks; vLLM's endpoint list includes both /v1/chat/completions and /v1/responses, so check which your Codex version expects. 6. Try it on a throwaway branch and read every diff before you merge.

    Can Claude Code use your own server?

    Not with a model you host. Claude Code can route through a gateway set with ANTHROPIC_BASE_URL, but Anthropic's LLM gateway page says Anthropic "doesn't support routing Claude Code to non-Claude models through any gateway." Some third parties document such setups anyway, but you would be outside what Anthropic supports, and the same page warns that a gateway which does not forward new capabilities "breaks the corresponding features". For a self-hosted model, use an agent that supports custom providers, such as the Codex CLI.

    What does running it yourself not fix?

    Your code stays on your network, which is often the reason to do this. But an open-weight model that fits one server is usually smaller than the hosted models these agents were built around, so expect more mistakes on long, multi-file tasks. You also take on patching, access control and logging, the jobs a hosted vendor did for you. Treat the server's key like any other credential, as the API keys thread describes.

    Short version

    • Use an inference server (vLLM, llama-server or Ollama) that speaks the OpenAI API format.
    • Bind it to a private address, set an API key from the environment, and firewall the port.
    • vLLM's key protects only some paths; put a reverse proxy in front for anything wider.
    • Point Codex at it with a [model_providers.<id>] entry; Claude Code does not support non-Claude models.
    • Plan memory for the model plus context times parallel requests.
    0
    ReplyLink

Plain text or simple Markdown. Links are nofollow. Your email is never shown.

CodeLifter Forum. Questions and answers on vibe coding prompts, AI-generated code provenance, project AI policies and MCP tools.