vLLM's beautiful architecture, explained

I usually dont geek over software designs, because I feel many are just overengineered, and some even just made fascinating just for the looks. But vLLM just takes my eyeballs because it's so cleanly designed. I wanted to talk about how and what I love about it, and it is just an appreciation post for the designers of this software. Lets talk!

Basics

A language model is a function from a list of token ids to one probability distribution over the next token. The loop, is generation. Think of it like walking across stepping stones, you can only choose the next stone after you’ve landed on the current one. An LLM works the same way, right?

It predicts one token, appends it, then uses the whole sequence again to predict the next. So generating 1,000 tokens means making that next-step decision about 1,000 times. Simple, neat.

Prefill and Decode

Two most important parts of LLM inference. Prefill is the part when the model dumps everything all at once into the GPU, all your input tokens. So it is compute bound, as it does a lot of operations in parallel. Decode, on the other hand, is quite opposite. It goes on, does small operations, while waiting for another token. So it reads all the memory (kv cache) and weights, just to produce one token. Decode is bandwidth-bound because for each token, the GPU must move/read a huge amount of weights + KV-cache data, while doing comparatively little computation. And then theirs the KV cache, and because it grows by every token the model produces, which again in autoregressive models, is unknown - you do not know how much tokens the model will produce because thats the nature of these models. You might say batching might work, but static batching does not. It waits for the longest request in the batch to finish, and thats.. game over.

AI made

The Problem

In a production system, like chatgpt for example, requests arrive with different prompt lengths, generate different numbers of tokens, and their KV caches keep growing at weirdly unpredictable rates. If you reserve large contiguous chunks of GPU memory for every request, you waste memory. If you pack things too tightly, moving and resizing them becomes painful. Meanwhile, the GPU wants large batches to stay busy, but requests are constantly entering and leaving the batch at different times.

This is basically the mess vLLM is designed to clean up.

AI made

The ideas that are vLLM

Paged Attention

This is where engineering gets its due - imagine there are two puzzles, one has already been solved - and another one, has just one puzzle piece waiting for it to be finished. The thing is, the solved one is hilariously similar to the unsolved one, so we see how did they solve the first one, borrow the idea, and boom, the puzzle pieces fit together. Same happens in computer science - specially here in vllm - they borrowed the operating system's trick: split each request's KV cache into fixed-size blocks, and keep a per-request table from logical blocks to physical ones.

AI made

The interesting part is the block table. It is exactly mapped to the operating system concepts, and it works almost flawlessly. I heard that they reduced wastage to almost 3% which is just fragmentation.

Awesome thing to note here is that this is not some sorcery. This is just neat engineering. PagedAttention is just paging applied to the KV cache. Continuous batching is just refusing to let finished requests hold everyone else back. Scheduling is just constantly asking: what work can I do right now to keep the GPU useful?

I feel that vllm got that right - in the manner that we can just use the working systems, borrow the concepts that have been working at scale, experiment, and then see if it works. Most of the times, good news, in CS, it works.

I think thats a lot of information about vllm, as a starter, I will more about it if this post gets traction. Sharing a nice picture chatgpt made for understanding the next interesting topic. Give it a read, or ask chatgpt about the image!

AI made