Skip to content
CodeLifter
Search/ Browse prompts
CodeLifter Forum

Why does a coding assistant invent functions that do not exist?

Staff answerWritten by CodeLifter editors · Vibe coding prompts0 replies

Short answer

A coding assistant invents functions because the language model behind it predicts code that looks right from patterns, and does not look up the library it is calling. A method name that fits the library's style is likely text, so it gets written. A type checker, the library's own reference and one test run catch most of them.

Written by CodeLifter editors · Staff, CodeLifter ·
CiteCodeLifter Forum, "Why does a coding assistant invent functions that do not exist?", https://codelifter.com/forum/posts/why-does-a-coding-assistant-invent-functions-that-do-not-exist/, accessed 2026-11-17.
  1. Staff, CodeLifter ·

    What is an API hallucination?

    An API hallucination is a call to a function, method, option or flag that the library you are using does not have, or has with a different signature (the list of arguments it accepts). API stands for application programming interface: the set of names a library exposes for your code to call. The code reads fluently and often compiles in your head; it fails when it runs, or worse, it calls something real that does a different job.

    They come in a few recognisable shapes:

    What you see What usually happened Quickest check
    TypeError: x.getAllItems is not a function An invented method that matches the library's naming style Search the library's reference for the exact name
    Right function, wrong arguments The signature from another version or a similar library Read the signature in the docs for your installed version
    A function that existed once A deprecated or removed API from older training data Check the library's changelog or migration guide
    An option the library silently ignores An invented config key or CLI flag The library's options reference; unknown keys are often not errors
    An import of a whole package that does not exist A package hallucination See the package hallucination thread

    The last row is the dangerous cousin, because a missing package name can be registered by someone else. An invented method on a real library just breaks.

    Why do models make up functions?

    A language model is trained to predict the next piece of text. OpenAI's research summary Why language models hallucinate (5 September 2025) explains the split: spelling and brackets "follow consistent patterns, so errors there disappear with scale", but "arbitrary low-frequency facts... cannot be predicted from patterns alone and hence lead to hallucinations." The exact name of a method in a niche library is that kind of fact. The same post argues that training and evaluation "reward guessing over acknowledging uncertainty", so a model that is unsure tends to produce a confident call rather than say it does not know.

    The effect is measurable for code. The paper On Mitigating Code LLM Hallucinations with API Documentation (Jain and colleagues, arXiv 2407.09726, July 2024) built a benchmark, CloudAPIBench, from cloud service APIs and reports that "Code LLMs struggle with low frequency APIs": GPT-4o produced valid calls for only 38.58% of the rarely seen APIs it tested. Giving the model the documentation raised that to 47.94%, and checking against an index of real APIs before retrieving docs improved results further.

    Which code is most likely to contain invented calls?

    • Rarely used libraries and rarely used corners of popular ones. That is the low-frequency case the paper measured.
    • Libraries newer than the model, or a major version that renamed things. The model saw the old names far more often than the new ones.
    • Your own project's code that is not in the conversation. If the assistant has not read utils/date.js, it guesses what is in it. What the assistant can see at once is its context window, covered in the context window thread.
    • Configuration files and command-line flags, where a wrong key may be ignored instead of raising an error.

    How do you catch an invented function before it ships?

    1. Run the type checker or compiler. In a TypeScript project, npx tsc --noEmit type-checks without writing output files; the noEmit option exists so TypeScript can act purely as "a source code type-checker". A call to a method a type does not have is a compile error, not a runtime surprise.
    2. Look up each unfamiliar call in the library's own reference, for the version in your lockfile (package-lock.json, poetry.lock or similar), not the latest version on the website.
    3. Ask Python itself. For a Python function, python -c "import inspect, json; print(inspect.signature(json.dumps))" prints the real parameters. inspect.signature (Python 3.15 docs, read 16 October 2026) returns the signature of any callable, so swap in the module and function the assistant used.
    4. Run the code path. A test that calls the new code once turns a silent guess into a visible failure. The thread on getting an assistant to write tests for its own code, published tomorrow, covers how to make those tests useful.
    5. Ask for the source, then open it. "Link the documentation page for every library call you added" is a fair request. A link the model writes can be invented too, so the check is that you open it.

    The general review routine in how to review AI-generated code before you merge it puts these checks alongside security and licensing.

    How do you prompt so it invents less?

    Prompts lower the rate; they do not remove it.

    • Paste or attach the docs. The paper's result is the practical lesson: documentation in the context helps most for rare APIs. Paste the relevant reference section, or point an agent at the docs URL.
    • Pin the version in your rules file. A line such as "This project uses React Router 7; do not use v5 APIs" in AGENTS.md or CLAUDE.md is read every session.
    • Give it permission to be unsure. "If you are not certain a function exists in this version, say so and show how you would check" invites the abstention that the OpenAI post says evaluations usually discourage.
    • Let the tool see errors. Claude Code's best practices page recommends giving the agent "a check it can run: tests, a build", and says a code intelligence plugin gives "automatic error detection after edits" in typed languages.

    CodeLifter's guide on how to write a coding prompt covers the rest of the prompt structure.

    Short version

    • Models predict plausible names; they do not look up the library unless a tool makes them.
    • Rare APIs are the weak spot: one study found GPT-4o made valid calls for 38.58% of low-frequency cloud APIs, 47.94% with docs supplied.
    • Type-check (npx tsc --noEmit), check signatures (inspect.signature), and run the code once.
    • Supply the docs and the version, and invite the assistant to say when it is unsure.
    0
    ReplyLink

Plain text or simple Markdown. Links are nofollow. Your email is never shown.

CodeLifter Forum. Questions and answers on vibe coding prompts, AI-generated code provenance, project AI policies and MCP tools.