LLM Fundamentals for Agent Builders

Models do not read words, they read tokens. A token is roughly four characters of English, so a thousand words is about 1,300 tokens. Code, identifiers, and non-English text tokenize less efficiently, sometimes twice as heavily, which is why a JSON-heavy agent costs more than its word count suggests.

Both directions are billed, and usually at different rates, with output several times the price of input. This has a design consequence founders miss: a verbose system prompt is charged on every single call forever, while a verbose answer is charged once. Trimming the prompt pays a dividend on every request.

Get a token counter into your project on day one and log input and output counts per run. Without it you are guessing at your own margin, and the guess is always optimistic. With it, the expensive step in your pipeline becomes obvious within a day of real traffic.

The context window is the total budget for system prompt, tool definitions, retrieved documents, conversation history, and the answer combined. Large advertised windows tempt teams into filling them, which is usually a mistake: cost rises linearly, latency rises with it, and accuracy does not improve past the point where the relevant material is present.

Reserve room for the output before you spend anything. An agent given a window packed with input has nowhere to put its answer, and truncation reads to the user as stupidity rather than as a configuration error. A few thousand tokens held back is cheap insurance.

Quality beats quantity every time. Ten well-chosen chunks with the right metadata outperform a hundred loosely related ones, because the model has less to disambiguate. If your retrieval returns forty chunks, the fix is better filtering, not a bigger window.