What is an LLM, and how does it write code?

What does "large language model" actually mean?

Google's machine learning crash course defines a language model as something that "estimates the probability of a token or sequence of tokens occurring within a longer sequence of tokens." A token, it says, "could be a word, a subword (a subset of a word), or even a single character."

The "large" part refers to size. The course's page on LLMs says they predict better than older language models because they "contain far more parameters" and "gather far more context." Parameters are the numbers inside the network that training adjusts; the same page says Transformers contain "hundreds of billion or even trillions" of them. Context means the text around the token being predicted, which is how a model tells "orange" the fruit from "orange" the colour.

How did language models get here?

The crash course walks through three generations. The differences matter for code, where a variable defined on line 3 can decide what line 300 should say.

Model type	How it uses context	Main limit
N-gram model	Looks only at the previous few words (two words for a 3-gram)	Too little context; longer N-grams become too rare to be useful
Recurrent neural network	Reads "token by token", carrying some context forward	Useful context is "still relatively limited"
Transformer-based LLM	Evaluates "the whole context at once" with self-attention	Cost grows with context length; still makes mistakes

The Transformer came from a 2017 paper, Attention Is All You Need, which proposed an architecture "based solely on attention mechanisms, dispensing with recurrence and convolutions entirely." Self-attention, in the crash course's words, asks for each token: "How much does each other token of input affect the interpretation of this token?" The crash course calls the Transformer "the most successful and widely used architecture for building LLMs."

How is an LLM trained?

According to the crash course, training starts with "a phenomenal amount of training data (text)". The model learns by predicting tokens that have been hidden from it and adjusting its parameters when it is wrong. Repeated across an enormous number of examples, it picks up grammar, facts, and the structure of code. An optional later step, instruction tuning, "can improve an LLM's ability to follow instructions", which is what turns a raw text predictor into something that answers requests.

So how does it write code?

The crash course is blunt: "LLMs are essentially autocomplete mechanisms that can automatically predict (complete) thousands of tokens." When you type "write a function that validates an email address", the model treats your request as the start of a document and predicts the most plausible continuation, token by token. Code is text, so the same process produces Python or JavaScript.

It works well for code because models are trained on a lot of it. OpenAI's 2021 paper Evaluating Large Language Models Trained on Code introduced Codex, "a GPT language model fine-tuned on publicly available code from GitHub", and said a "distinct production version of Codex powers GitHub Copilot" at the time. On the paper's HumanEval test, which checks whether generated Python functions pass unit tests, Codex solved 28.8% of problems while GPT-3, a general model without that code training, solved 0%. Fine-tuning means extra training on a narrower dataset, here code.

The same paper lists limits that still explain many bad suggestions: "difficulty with docstrings describing long chains of operations and with binding operations to variables." In plain terms, long multi-step instructions and keeping track of which value goes in which variable are where the model slips.

Why does an LLM write code that looks right but is wrong?

Because it predicts what is plausible, not what is true. The crash course says LLMs "hallucinate, meaning their predictions often contain mistakes." For code that shows up as functions that do not exist, wrong arguments, outdated APIs, and package names nobody published. The thread on invented packages covers the research on that last one.

A model also knows nothing about your project unless the text reaches it. A plain chat model sees only what you paste. Coding tools such as Claude Code, Codex and Copilot wrap an LLM in a program that reads your files and runs commands, which the coding agents thread compares.

How do you use that knowledge when you prompt?

1. Give the model the context it cannot guess: language, version, framework, the existing function signatures, and the error message in full.
2. Ask for one small, testable piece at a time instead of a long chain of steps, the weakness the Codex paper names.
3. Run the code. Passing tests is the evidence; a confident explanation is not.
4. Check every import and API call against the official documentation before you install or ship anything.
5. Treat the output as a draft written by someone who has read a lot of code but has never seen yours.

The CodeLifter prompt guide turns those steps into a reusable prompt structure, and the models page lists current models with their makers and licences.

Short version

• An LLM is a neural network that predicts the next token from the tokens before it.
• Most current LLMs use the Transformer design from 2017, which weighs the whole context at once.
• It writes code by predicting plausible code, helped by training on public code.
• Plausible is not correct: run it, test it, and check imports against the docs.