How do you run an LLM on your own server for coding?
To run an LLM on your own server for coding, install an inference server such as vLLM, llama.cpp's llama-server or Ollama on a machine with enough GPU memory, serve an open-weight model through its OpenAI-compatible API, keep that API on your private network behind a key or proxy, then point your coding agent's base URL at it.
Cite
CodeLifter Forum, "How do you run an LLM on your own server for coding?", https://codelifter.com/forum/posts/how-do-you-run-an-llm-on-your-own-server-for-coding/, accessed 2026-11-16.- CodeLifter editorsStaffStaff, CodeLifter ·
What does running a model "on premise" mean?
An LLM (large language model) is the program that writes the code; an inference server is the software that loads its weights into memory and answers requests over HTTP. Running it on premise means both live on hardware you control: a workstation under a desk, a rack server or a private cloud machine. Your coding agent, the client that reads files and runs commands, can stay on each developer's laptop and send its requests to that server.
This differs from the single-laptop setup in the local model thread in one way that matters: the server now listens on a network. Everything in this thread is about doing that safely. If you only need a model on your own machine, the Ollama thread is the shorter path.
Which server software can you use?
The three below all offer an HTTP API in the OpenAI format, which is what most coding agents expect. Licences are from the CodeLifter local models page ledger, checked 15 October 2026; behaviour is from each project's documentation, read 16 October 2026.
Server Licence Start command Listens on by default Built-in API key vLLM Apache-2.0 vllm serve <model>http://localhost:8000--api-keyorVLLM_API_KEYllama.cpp llama-serverMIT llama-server -m model.gguf127.0.0.1(set with--host)--api-keyorLLAMA_API_KEYOllama MIT ollama serve127.0.0.1:11434Not described in its FAQ; use a proxy The vLLM quickstart says its server "currently hosts one model at a time" and accepts
--hostand--port. The llama-server README notes that OpenAI-style function calling, which agents use to edit files and run commands, "is supported with the--jinjaflag". The Ollama FAQ changes its address with theOLLAMA_HOSTenvironment variable and shows Nginx as a proxy in front of it.How big a server do you need?
The model must fit in memory, and so must its working space. Use the memory estimate on the local models page for the weights. Then add room for context, the text the model holds per request: the Ollama FAQ warns that parallel requests multiply it, so "a 2K context with 4 parallel requests will result in an 8K context and additional memory allocation." A team server handles several developers at once, so plan memory for the number of parallel requests you allow, not for one.
Step by step: serve a model with vLLM and connect Codex
- Pick a model whose licence allows your use. The local models page lists the licence of each model it covers. Check that it supports tool calling; a model without it can chat but cannot act.
- Start the server on a private address. Set the key in the service's environment, not on the command line, where other users of the machine can see it in the process list:
export VLLM_API_KEY="<long random value>" vllm serve <model> --host 10.0.0.5 --port 8000- Fence the port. Allow port 8000 only from your developers' network. vLLM's security page says the key protects only paths under
/v1,/v2,/inferenceand/cohere, and lists others, such as/metricsand/pause, that need no key at all. - Test it from a developer machine:
curl http://10.0.0.5:8000/v1/models -H "Authorization: Bearer $VLLM_API_KEY". - Point Codex at it. OpenAI's advanced configuration page defines custom providers in
~/.codex/config.toml:
model = "<model>" model_provider = "team_vllm" [model_providers.team_vllm] name = "Team vLLM" base_url = "http://10.0.0.5:8000/v1" env_key = "VLLM_API_KEY"The same page documents a
wire_apisetting for which request format a provider speaks; vLLM's endpoint list includes both/v1/chat/completionsand/v1/responses, so check which your Codex version expects. 6. Try it on a throwaway branch and read every diff before you merge.Can Claude Code use your own server?
Not with a model you host. Claude Code can route through a gateway set with
ANTHROPIC_BASE_URL, but Anthropic's LLM gateway page says Anthropic "doesn't support routing Claude Code to non-Claude models through any gateway." Some third parties document such setups anyway, but you would be outside what Anthropic supports, and the same page warns that a gateway which does not forward new capabilities "breaks the corresponding features". For a self-hosted model, use an agent that supports custom providers, such as the Codex CLI.What does running it yourself not fix?
Your code stays on your network, which is often the reason to do this. But an open-weight model that fits one server is usually smaller than the hosted models these agents were built around, so expect more mistakes on long, multi-file tasks. You also take on patching, access control and logging, the jobs a hosted vendor did for you. Treat the server's key like any other credential, as the API keys thread describes.
Short version
- Use an inference server (vLLM,
llama-serveror Ollama) that speaks the OpenAI API format. - Bind it to a private address, set an API key from the environment, and firewall the port.
- vLLM's key protects only some paths; put a reverse proxy in front for anything wider.
- Point Codex at it with a
[model_providers.<id>]entry; Claude Code does not support non-Claude models. - Plan memory for the model plus context times parallel requests.
Write something first.
Give people something to work with: at least 30 words on what happened and what you tried.
That is too long. Keep it under 6,000 characters.
Write the question as the title, 15 to 140 characters, no links.
Pick a category.
Add a name (2 to 40 characters, no links).
That email address does not look right.
That was quick. Read the thread, then try again.
The form expired. Reload the page and post again.
Something went wrong with the form. Reload and try again.
Please complete the check and post again.
Limit reached for now. Try again later.
This thread is closed to new replies.
This thread no longer accepts replies.
Something went wrong with the form. Reload and try again.
Post a reply or question first (name and email), then this browser can vote, edit and accept answers.
You cannot vote on your own post.
Only the author (within 30 days) or the forum team can do that.
That email already has an account. Use your sign-in link instead.
New accounts are paused for the moment. Try again later.
That was already posted.