Post cover image

August 24, 2026

The Hidden Engine Room: What Actually Happens Between You Hitting “Send” and an AI Typing Back

A field guide to the unglamorous, brilliant plumbing that makes modern chatbots fast, cheap, and (barely) affordable to run

By DhanushKumar

21 min read

Every time you send a message to ChatGPT, Claude, or any other AI chatbot, you're triggering a small industrial process. Somewhere in a data center, a GPU the size of a paperback book lights up, hunts through hundreds of gigabytes of numbers, and starts spitting out your answer one word at a time.

We tend to talk about large language models like they're magic — "the AI thought about it and answered." But underneath that sentence is a genuinely fascinating engineering discipline, one most people (even a lot of software engineers) have never looked at closely. It's called inference engineering, and it's the reason a chatbot response costs a fraction of a cent instead of several dollars.

This is the story of what's actually happening in that gap between "send" and "typing…" — told the way I wish someone had told it to me.

First, a distinction that matters more than it sounds like it should

There are two totally different phases in an AI model's life, and mixing them up is where most confusion starts.

Training is the process of teaching the model — showing it a huge slice of the internet, adjusting billions of internal numbers over weeks, spending millions of dollars, and doing it maybe a handful of times a year.

Inference is using that already-trained, now-frozen model to actually answer people. It happens billions of times a day, it has to happen in under a second, and it has to be cheap enough that a company can give it away for $20/month and still turn a profit.

Training is a marathon you run once. Inference is a sprint you run forever. And it turns out optimizing "run this sprint a billion times a day, as cheaply as possible" is its own entire field. That field is what this post is about.

The trip a sentence takes

Let's follow one message — "Write a haiku about the ocean" — from your keyboard to your screen.

Step 1: Your words become numbers. The model can't read English. It reads integers. A process called tokenization chops your sentence into small chunks (roughly ⅔ of a word each) and looks each one up in a giant dictionary, turning your sentence into something like [8144, 264, 6520, 27678, 922, 279, 18246, 13].

Step 2: The model reads your whole prompt at once. This is called prefill, and it's the one part of the process that happens in parallel — every word of your prompt gets processed simultaneously, all at the same time, across every layer of the network. Think of eight people speed-reading the same page together instead of one person reading it eight times.

Step 3: It starts writing, one word at a time — forever. This is decode, and it's fundamentally different from Step 2. The model can't generate the whole poem in one shot, because word 5 depends on word 4, which depends on word 3, all the way back. So it writes "Waves," looks at everything so far (including the word it just wrote), writes "crash," looks again, writes "upon," and so on — one token, one full pass through the model, one token, one full pass through the model — until the poem is done.

Step 4: It decides when to stop. Either the model itself signals "I'm done" with a special token, it hits a length limit, or it happens to generate a string the developers told it to treat as a stop sign.

Step 5: The numbers become words again, streamed to your screen as they're produced — which is why you see the response "type itself out" instead of appearing all at once. It's not a special effect. It's literally being computed live, in front of you.

That's the whole loop. Everything else in this post is about making that loop fast enough and cheap enough to run at planetary scale.

Two Regimes You Need to Hold in Your Head

  • Prefill = reading your prompt. It happens once, all tokens at once, in parallel. The GPU's math units are the bottleneck (compute-bound).
  • Decode = writing the response. It happens token by token, one at a time, forever until it stops. Moving the model's weights and cached data through memory is the bottleneck (memory-bound), not the math itself.
None

Prefill

Once tokenized, the entire prompt is pushed through every layer of the network in one parallel pass. For each token, at each layer, the model computes a Query, a Key, and a Value vector , the three ingredients of the attention mechanism, which lets every token "look back" at every earlier token to build contextual understanding (e.g., the model figuring out that "ocean" is part of a poem-writing request because "haiku" appeared earlier).

This step is compute-bound: because all tokens are processed simultaneously, the GPU's arithmetic units are the limiting factor, and GPUs are extremely good at exactly this kind of massively parallel matrix math — which is why even long prompts (thousands of tokens) tend to prefill quickly relative to how long decode will take afterward.

The Keys and Values computed here aren't thrown away — they're written into the KV cache, which is where the next section picks up.

The single most important trick: don't repeat yourself

Here's a question that should bother you about the next step : if the model has to "look at everything so far" for every single word it writes, isn't that insanely wasteful? Wouldn't generating the 500th word of a long response mean re-reading all 499 words before it, every single time?

Yes. That's exactly the problem. And the fix is one of the most important ideas in this whole field: the KV cache.

When the model reads a word, it computes two little summary vectors for it — a "Key" and a "Value" — which capture what that word means for attention purposes.

The trick is: once you've computed those, you never have to compute them again. You just save them. Every future step just glances at the saved summaries instead of re-processing the original words from scratch.

I think of it like a detective's corkboard. A bad detective re-reads the entire case file every time a new clue comes in. A good detective keeps an index card for every clue they've already processed and just pins a new card up when something new happens. The KV cache is the corkboard.

The catch: that corkboard isn't free. For a genuinely large model, storing the Keys and Values for a single 4,000-word conversation can eat over a gigabyte of GPU memory — and that's for one user. Multiply that by the thousands of people using a chatbot at the same second, and you start to see why "memory management" is basically the whole ballgame in this field.

Attention Head Variants — MHA vs. MQA vs. GQA

The size of the KV cache is directly controlled by how many independent Key/Value "heads" the attention mechanism keeps. The original design, Multi-Head Attention (MHA), gives every attention head its own full set of Keys/Values, which maximizes quality but makes the cache large. Multi-Query Attention (MQA) goes to the opposite extreme — every head shares one single set of Keys/Values — shrinking the cache dramatically but costing some output quality. Grouped-Query Attention (GQA) splits the difference: heads are bundled into small groups that each share one Key/Value set, cutting cache size 4–8x versus MHA while staying nearly indistinguishable in quality.

None None

Renting rooms instead of floors

Given that the KV cache is expensive, how do you avoid wasting it? Early inference systems made an embarrassingly simple mistake: they'd reserve the maximum possible amount of memory for every single request, just in case it turned out to be a long one. So a two-word question and a ten-thousand-word document upload would get allocated the exact same chunk of memory — meaning the two-word question wasted almost all of it.

The fix, called PagedAttention, is a straight-up rip of an idea from 1960s operating systems: instead of handing out one giant contiguous slab of memory per request, chop memory into small interchangeable pages, and hand pages out only as they're actually needed.

It's the difference between renting an entire floor of an apartment building to every tenant (wasteful, if most tenants are singles) versus renting individual rooms and giving people more as their families grow. This one idea, more than almost anything else, is why a single GPU today can serve dozens of simultaneous users instead of just a handful.

And it has a lovely side effect: if two people happen to share the exact same starting text — like, say, every user of a customer support bot sharing the same system prompt — the system can just point both of their conversations at the same physical pages instead of duplicating them. That's called prefix caching, and it means the shared boilerplate only ever gets processed once, no matter how many people are using it.

Use this whenever you're serving many concurrent requests of wildly varying length — which is essentially every real production chat workload — since it's what lets a single GPU serve far more simultaneous users without wasting memory on padding.

None

Prefix Caching

Because pages are independently addressable, two requests that begin with an identical sequence of tokens — most commonly, the same system prompt reused by every user — can literally point to the same physical pages instead of duplicating the computation. The engine hashes chunks of the prompt, and if an identical chunk (and everything before it) was already computed and is still resident, its pages are reused directly, skipping prefill for that portion entirely.

When to use it: any application with a long, static, shared prefix — a support bot's system prompt, a fixed set of few-shot examples, a shared document — where recomputing that prefix on every single request would otherwise be pure waste.

Sliding Window / Cache Eviction

For very long-running conversations, keeping every single token's Key/Value forever is eventually unaffordable, and distant tokens are often less relevant than recent ones anyway. A sliding window caps the cache at the most recent N tokens, dropping the oldest entries as new ones arrive — trading long-range memory for bounded, predictable memory usage. Reach for this in long-running chat or agent sessions where unbounded growth would eventually exhaust GPU memory.

None

The bakery analogy that explains half of modern AI infrastructure

Okay, here's the one that I think is actually the most underrated idea in the entire field, and almost nobody outside of ML infra knows about it.

Imagine a bakery with 8 checkout counters, and a rule: you can't let a new customer into any counter until every single customer at every single counter has finished their entire order. If 7 people finish in 2 minutes and 1 person is still deciding on pastries 20 minutes later, those 7 counters just… sit there. Empty. Earning nothing.

That's how early AI serving systems batched requests together a fixed group would go in, and the whole group was locked together until the last one finished, no matter how much faster everyone else was.

The fix is called continuous batching, and it's beautifully simple once you see it: check after literally every single word generated whether any customer (request) just finished, and if so, immediately let a new one sit down. No more waiting for the whole table to clear.

This single change is one of the biggest reasons AI got so much cheaper to run between 2023 and today. It's not a smarter model. It's a smarter waiter.

Continuous batching (also called in-flight batching) fixes this by re-evaluating batch membership after every single decode step: the instant any request finishes, it's evicted and a new waiting request is slotted into that freed capacity immediately, without waiting for the rest of the batch. This requires the KV cache to support per-request dynamic allocation and deallocation — which is exactly what paged attention provides, which is why the two techniques are almost always deployed together.

None

Analogy: a bakery with 8 checkout counters. Static batching won't let a new customer into any counter until all 8 finish, even if 7 finished two minutes ago. Continuous batching slots a new customer into counter #3 the moment it's free.

When to use continuous batching: essentially always, in any production multi-user serving scenario — it is the single biggest lever for total throughput

class Request:
    def __init__(self, request_id, prompt_tokens, max_new_tokens):
        self.id = request_id
        self.tokens = prompt_tokens       # Contains prompt + generated tokens so far
        self.max_tokens = max_new_tokens
        self.is_prefilled = False
        self.is_finished = False

class ContinuousBatchingEngine:
    def __init__(self, model, kv_cache_allocator, max_batch_size=8):
        self.model = model
        self.kv_cache = kv_cache_allocator  # PagedAttention block manager
        self.max_batch_size = max_batch_size

        self.waiting_queue = []            # New incoming requests
        self.active_batch = []             # Requests currently in-flight

    def add_request(self, request):
        self.waiting_queue.append(request)

    def step(self):
        """Executes a single decode/prefill iteration."""

        # 1. EVICTION & CLEANUP: Remove finished requests immediately
        for req in self.active_batch[:]:
            if req.is_finished:
                self.kv_cache.free_pages(req.id)  # Deallocate pages (PagedAttention)
                self.active_batch.remove(req)

        # 2. IMMEDIATE RE-SLOTTING: Fill vacant slots from waiting queue
        while len(self.active_batch) < self.max_batch_size and self.waiting_queue:
            new_req = self.waiting_queue.pop(0)
            if self.kv_cache.can_allocate(new_req):
                self.kv_cache.allocate_pages(new_req.id)
                self.active_batch.append(new_req)
            else:
                # Memory full, leave in waiting queue
                self.waiting_queue.insert(0, new_req)
                break

        if not self.active_batch:
            return  # No active requests to process

        # 3. INTERLEAVED EXECUTION: Perform 1 pass through GPU
        # Separate incoming prefill requests from existing decode requests
        prefill_reqs = [r for r in self.active_batch if not r.is_prefilled]
        decode_reqs = [r for r in self.active_batch if r.is_prefilled]

        # Execute Prefill for new requests (compute-bound pass)
        for req in prefill_reqs:
            kv_pages = self.kv_cache.get_pages(req.id)
            self.model.forward_prefill(req.tokens, kv_pages)
            req.is_prefilled = True

        # Execute Decode for all ready requests (memory-bound, single token step)
        if decode_reqs:
            next_tokens = self.model.forward_decode_batch(decode_reqs, self.kv_cache)

            # 4. TOKEN EVALUATION: Update status after single decode step
            for req, next_token in zip(decode_reqs, next_tokens):
                req.tokens.append(next_token)

                # Check for completion (EOS token or max token limit reached)
                if next_token == EOS_TOKEN or len(req.tokens) >= req.max_tokens:
                    req.is_finished = True

    def run_loop(self):
        """Main event loop running continuous batching."""
        while self.active_batch or self.waiting_queue:
            self.step()

Rounding numbers to save your GPU's life

Here's a fact that sounds almost too simple to matter: the numbers that make up a language model — its "weights" — are usually stored with way more precision than they actually need. A 70-billion-parameter model stored at full precision needs about 140 gigabytes just to exist in memory, before it's even done anything.

Quantization is the practice of rounding those numbers to a coarser scale — instead of 16 bits of precision per number, using 8, or even 4. It's the exact same idea as describing a paint color as "olive green" instead of "RGB(126, 202, 88)." You lose a little precision. For almost every practical purpose, nobody notices.

But the payoff is enormous: that same 70-billion-parameter model, quantized down to 4 bits, shrinks to roughly 35–40 gigabytes — small enough to run on a single beefy consumer GPU instead of a multi-GPU server rack. This is the entire reason the open-source "run a real AI model on your gaming PC" community exists.

None

Model weights are normally stored as 16-bit floating point numbers, so a 70-billion-parameter model needs roughly 140 GB just for its weights. Quantization represents those same weights using fewer bits — commonly 8-bit or 4-bit integers — by rounding each weight's value onto a coarser, discrete set of representable values, using a per-layer or per-group scale factor chosen to minimize the resulting error. The payoff is two-fold: a smaller memory footprint (fits on cheaper or fewer GPUs) and less data that needs to be moved during the memory-bound decode phase, directly attacking the bottleneck described in Section 1.

Analogy: describing a color as the exact value RGB(126, 202, 88) versus rounding it to "olive green" — you lose some precision but gain speed and a much smaller footprint, and it's usually good enough.

None

Method differences:

GPTQ (Generalized Post-Training Quantization)

  • Core Idea (Layer-by-Layer Reconstruction): GPTQ looks at the model layer by layer and quantizes the weights one column at a time.
  • How it Works: When a weight is rounded to 4-bit, it introduces a small quantization error. GPTQ instantly updates the remaining unquantized weights in that layer to compensate for that error using second-order information (Hessian matrices).
  • Strength: Extremely fast execution speeds during inference because it uses a flat, uniform scale across large weight blocks.
  • Weakness: It treats all weights relatively equally, meaning critical high-magnitude weights can get corrupted, occasionally leading to slight accuracy drops or edge-case hallucinations.

AWQ (Activation-aware Weight Quantization)

  • Core Idea (Protecting Important Weights): AWQ observes the activation channels when running a small calibration dataset through the model. It discovers that not all weights are equally important — roughly 1% of the weights carry most of the model's critical features.
  • How it Works: Instead of quantizing all weights uniformly, AWQ identifies these "salient" weight channels and applies a per-channel scaling factor to protect them from precision loss before rounding the rest of the matrix.
  • Strength: Preserves model accuracy and reasoning capability noticeably better than GPTQ (especially at lower bitrates like 3-bit or 4-bit) because it protects the most vital neural paths.
  • Weakness: Requires fine-grained group scaling, which traditionally had slightly higher metadata overhead, though modern inference engines like vLLM optimize both exceptionally well
None None

Quantization is the default first lever precisely because it's the least invasive of the three — no retraining required.

Why the model gets slower, not faster, once it starts talking

Here's a genuinely counterintuitive fact: reading your prompt is compute-hungry but fast. Writing the response is compute-cheap but comparatively slow, word by word. Why?

Reading your prompt (prefill) processes everything in parallel — it's a workload that keeps every math unit on the GPU busy simultaneously. It's "compute-bound," meaning the bottleneck is literally how much arithmetic the chip can crunch per second, and modern GPUs are monsters at that.

Writing the response (decode) is different in a way that surprised me the first time I really understood it: generating one single word still requires the GPU to haul the model's entire set of weights — tens of gigabytes — out of memory, just to compute one tiny new number. The actual math is trivial. The data movement is enormous. This is "memory-bound" — the bottleneck isn't how fast the chip can think, it's how fast it can fetch.

Picture a librarian who has to walk to a warehouse and drag back an entire bookshelf just to answer one two-second question. The walking, not the answering, eats all the time.

And this is exactly why the bakery trick (continuous batching) matters so much. If you're going to pay the cost of hauling that bookshelf out anyway, you might as well answer 32 people's questions while it's out instead of just one. That single insight — that the expensive part of decoding doesn't get more expensive when you serve more people at once — is the economic foundation the entire AI chatbot industry is quietly built on.

Chunked Prefill

A very long prompt's prefill can hog the GPU for an extended, uninterrupted stretch, delaying decode steps for everyone else mid-conversation. Chunked prefill splits that long prefill into smaller pieces and interleaves them with other users' decode steps, smoothing out latency for the whole population. Use this when your workload mixes very long documents with many short, latency-sensitive chats at the same time.

None

Disaggregated Prefill/Decode Serving

At larger scale, some systems run prefill and decode on entirely separate pools of GPUs, since one is compute-bound and burst while the other is memory-bound and sustained — mixing them on the same hardware means each interferes with the other's performance profile. Splitting them lets each pool be tuned and scaled independently. This is a scale-driven choice, appropriate once a single shared GPU pool can no longer serve both workloads efficiently.

None

A cheat code: let a smaller model do the guessing

One more idea, because it's too clever not to mention. Since generating text word-by-word is fundamentally slow (each word needs its own full pass through a giant model), what if you let a much smaller, much faster model guess several words ahead, and then have the big model just check the guesses instead of generating from scratch?

That's speculative decoding. A lightweight "draft" model rattles off, say, 4 guesses. The big model checks all 4 in a single parallel pass — remember, checking things in parallel is the fast, compute-bound kind of work — and accepts whichever guesses turn out to be right. Wrong guesses get corrected, and everything after the first mistake gets thrown out and retried.

It's a senior editor who doesn't write every sentence themselves — a junior assistant drafts a paragraph, and the editor just skims and approves big chunks at once, only slowing down to fix the spot where the assistant guessed wrong. Done well, this can nearly double or triple how fast a single response streams back to you.

Because decode is slow and strictly sequential, speculative decoding uses a small, cheap "draft" model to guess several tokens ahead; the big model then verifies all of them in a single parallel (compute-bound, not memory-bound) pass. Correctly guessed tokens are accepted for free; the first wrong guess is corrected and everything after it is discarded and retried.

Analogy: a senior editor who doesn't write every sentence themselves — a junior assistant drafts a paragraph, and the editor quickly approves whole chunks at once or fixes the first mistake and has the rest redone.

When to use it: whenever single-request latency matters more than raw aggregate throughput, and a smaller, faster draft model with similar behavior to the main model is available.

None

Sampling — Turning Probabilities Into an Actual Word

At every decode step the model doesn't hand you "the answer" — it hands you a probability across its entire vocabulary (often 100,000+ possible tokens), and something has to pick one actual token from that distribution.

  • Greedy decoding always takes the single highest-probability token. It's fully deterministic, which makes it the right choice when you need reproducibility — code generation, math, anything where you want the same input to reliably produce the same output — at the cost of sounding repetitive.
  • Temperature reshapes the whole distribution before anything else happens: low temperature sharpens it toward the safest, most likely tokens; high temperature flattens it, giving less-likely tokens more of a chance. It's a "creativity knob," not a selection method on its own.
  • Top-k restricts the candidate pool to a fixed number (k) of the most likely tokens before sampling.
  • Top-p (nucleus) instead keeps the smallest set of tokens whose combined probability crosses a threshold p — a pool size that adapts automatically to how confident the model is at that step, which is why it's the default in most chat products over fixed top-k.
  • Beam search explores several candidate continuations in parallel and keeps the overall best-scoring full sequence. It suits tasks like translation, where there's roughly one "correct" answer, and is a poor fit for open-ended creative generation where diversity is the goal, not a defect.

Temperature and top-p are almost always combined, because they solve different problems: temperature decides how confident vs. flat the distribution should be, and top-p decides how many of those reshaped options are even considered.

None

Compute-Bound vs. Memory-Bound — The Core Performance Lens

Prefill and decode aren't just different steps — they stress completely different parts of the hardware, and understanding this explains why nearly every technique above exists.

Prefill is compute-bound: all prompt tokens are processed in parallel, the GPU's math units run near full capacity, and performance is limited by raw arithmetic throughput. Picture a factory assembly line running at full throttle — every worker (compute core) busy simultaneously.

Decode is memory-bound: generating a single token still requires loading the model's entire set of weights (potentially tens of gigabytes) from memory, even though the actual math for one token is tiny — so the compute units sit mostly idle, waiting on data to arrive. Picture a librarian who has to walk to the storage room and haul back an entire bookshelf just to answer one small question — the fetching, not the answering, dominates the time.

Concrete numbers: a GPU that moves 3,000 GB/sec of data, serving a 140 GB model, takes roughly 140 ÷ 3000 ≈ 47 milliseconds just to load the weights for one decode step — whether it's serving 1 user or 32 batched users in that same step, since the weight-loading cost is paid once and shared. Serving 32 users per step costs the same ~47ms but produces 32x the output. This single fact is why continuous batching (Section 6) is the highest-leverage optimization available: it converts idle, memory-bound waiting time into shared, productive work.

None

Scaling Across Multiple GPUs

When a model doesn't fit on or can't be served fast enough by a single GPU, work gets split using one of three strategies:

  • Tensor parallelism slices individual weight matrices across GPUs, with each device computing part of every matrix multiplication and synchronizing results constantly. Use it when the model literally can't fit on one GPU and you have very fast interconnects (like NVLink) to support the frequent communication.
  • Pipeline parallelism assigns different layers to different GPUs, so data flows through them like an assembly line. It needs less communication than tensor parallelism but can create idle "bubbles" while later stages wait on earlier ones.
  • Data parallelism just runs full copies of the model on separate GPUs, each handling different users. Use it once the model already fits on one GPU/node and you simply need more total capacity.
None

Analogy: tensor parallelism is 4 chefs each cooking one quarter of every dish together, constantly coordinating. Pipeline parallelism is an assembly line — prep, cook, plate, each in a different station. Data parallelism is 4 fully independent kitchens each serving different customers.

None

Serving Frameworks at a Glance

None

Stopping Criteria and Detokenization

The decode loop ends when one of three things happens: the model itself emits a special end-of-sequence token, a system-enforced maximum token count is reached, or a developer-defined stop string appears in the output (e.g., stopping if the model starts generating "\n\nUser:" in a chat app, which would mean it's hallucinating the next turn). Once stopped, the full sequence of output token IDs is detokenized — mapped back to text via the tokenizer's vocabulary — and streamed to the screen, usually as each token is produced, which is what creates the "typing" effect.

Pseudocode — The Whole Engine, Tied Together

# ============================================================
# CORE LLM INFERENCE ENGINE (simplified, end-to-end pseudo-code)
# ============================================================

class InferenceRequest:
    def __init__(self, prompt_tokens, max_new_tokens, stop_sequences,
                 temperature, top_p):
        self.prompt_tokens = prompt_tokens
        self.generated_tokens = []
        self.max_new_tokens = max_new_tokens
        self.stop_sequences = stop_sequences
        self.temperature = temperature
        self.top_p = top_p
        self.kv_cache = KVCache()     # paged, per-request cache (Sec. 5.2)
        self.finished = False

# ---------------- PREFILL (compute-bound, Sec. 4) ----------------
def prefill(model, request):
    # all prompt tokens processed in ONE parallel pass
    hidden_states, keys, values = model.forward_parallel(request.prompt_tokens)
    request.kv_cache.append(keys, values)          # fills the KV cache (Sec. 5)
    logits = model.lm_head(hidden_states[-1])       # only need the last position
    return logits

# ---------------- SAMPLING (Sec. 8) ----------------
def sample_next_token(logits, temperature, top_p):
    scaled_logits = logits / max(temperature, 1e-5)         # temperature knob
    probs = softmax(scaled_logits)

    sorted_probs, sorted_ids = sort_descending(probs)
    cumulative = cumulative_sum(sorted_probs)
    cutoff_index = first_index_where(cumulative >= top_p)    # nucleus filter
    allowed_ids = sorted_ids[: cutoff_index + 1]
    allowed_probs = renormalize(sorted_probs[: cutoff_index + 1])

    return weighted_random_choice(allowed_ids, allowed_probs)

# ---------------- DECODE STEP (memory-bound, Sec. 9) ----------------
def decode_step(model, request):
    last_token = (request.generated_tokens[-1]
                  if request.generated_tokens else request.prompt_tokens[-1])

    # only ONE new token's Query/Key/Value computed; rest reused from cache
    hidden_state, new_key, new_value = model.forward_single(
        token=last_token,
        cached_keys=request.kv_cache.keys,
        cached_values=request.kv_cache.values,
    )
    request.kv_cache.append(new_key, new_value)

    logits = model.lm_head(hidden_state)
    return sample_next_token(logits, request.temperature, request.top_p)

# ---------------- SPECULATIVE DECODING (optional, Sec. 9.1) ----------------
def speculative_decode_step(big_model, draft_model, request, k=4):
    draft_tokens = draft_model.propose_tokens(request, count=k)   # cheap, fast

    # big model verifies all k draft tokens in ONE parallel (compute-bound) pass
    verified_logits = big_model.forward_parallel_verify(request, draft_tokens)

    accepted = []
    for i, draft_tok in enumerate(draft_tokens):
        true_tok = sample_next_token(verified_logits[i], request.temperature, request.top_p)
        if true_tok == draft_tok:
            accepted.append(true_tok)       # matched — accept for free
        else:
            accepted.append(true_tok)       # correct the mismatch
            break                           # discard everything after it
    return accepted

# ---------------- STOPPING CRITERIA (Sec. 11) ----------------
def should_stop(request, next_token, eos_token_id):
    if next_token == eos_token_id:
        return True
    if len(request.generated_tokens) + 1 >= request.max_new_tokens:
        return True
    if ends_with_any(request.generated_tokens + [next_token], request.stop_sequences):
        return True
    return False

# ---------------- CONTINUOUS BATCHING SERVER LOOP (Sec. 6) ----------------
def inference_server_loop(model, waiting_queue, max_batch_size):
    active_batch = []

    while True:
        # Backfill any free slot immediately — the essence of continuous batching
        while len(active_batch) < max_batch_size and waiting_queue.has_next():
            new_request = waiting_queue.pop()
            prefill(model, new_request)
            active_batch.append(new_request)

        if not active_batch:
            sleep_briefly()
            continue

        # ONE batched decode step across every active request —
        # amortizes the memory-bound cost of loading model weights (Sec. 9)
        next_tokens = model.forward_batch_single_step(
            [r.kv_cache for r in active_batch],
            [r.generated_tokens[-1] if r.generated_tokens else r.prompt_tokens[-1]
             for r in active_batch]
        )

        still_active = []
        for request, raw_logits in zip(active_batch, next_tokens):
            next_token = sample_next_token(raw_logits, request.temperature, request.top_p)
            request.generated_tokens.append(next_token)
            stream_token_to_user(request, next_token)          # the "typing" effect

            if should_stop(request, next_token, eos_token_id=EOS_ID):
                request.finished = True
                free_kv_cache_pages(request)      # pages return to shared pool (Sec. 5.2)
                notify_user_complete(request)
            else:
                still_active.append(request)

        active_batch = still_active   # freed slots get backfilled next loop iteration

So what's actually happening, one more time

Next time you watch a chatbot response type itself out, here's the real picture behind it:

Your words get chopped into number-tokens. The model reads all of them in one parallel burst, saving little summary notes (Keys and Values) as it goes so it never has to re-read them. Those notes live in small, reusable memory "rooms" instead of giant pre-reserved "floors," so the system can serve far more people at once. A scheduler is constantly swapping people in and out of GPU capacity the moment they finish, like a well-run bakery counter, never leaving a seat empty. The model itself has probably been shrunk down using quantization so it fits on the hardware at all. And it's writing to you one word at a time, in the genuinely slow part of the whole process, possibly sped up by a smaller model guessing ahead on its behalf.

None of this is what people picture when they imagine "AI." There's no mysterious reasoning ritual, no ghost in the machine deliberating over your question. There's a very well-engineered assembly line, built by people who cared enormously about milliseconds and gigabytes, running underneath a chat window that looks effortless entirely because of how much invisible work went into making it feel that way.

That, to me, is the actual magic trick. Not that the model is smart. That someone made "smart" affordable.