tokens

Think of tokens as the fundamental units a computer breaks down text into when it's trying to understand or create language. They aren't always whole words, but can be parts of words, punctuation marks, or even just single characters. For example, the word "unbelievable" might be broken into tokens like "un-", "believe", and "-able". This method helps models handle a massive vocabulary more efficiently and deal with new or complex words.

When you interact with an AI model, like asking it a question or giving it a prompt, your input is first converted into these tokens. The model then processes these tokens to generate its response, which is also made up of tokens. These output tokens are then reassembled into human-readable text. Understanding tokens is key to knowing how much information an AI can process at once, as models often have a limit on the number of tokens they can handle in a single conversation or request. This limit is often referred to as the 'context window'.