Home

From Scratch I

AI infra from scratch (kinda)

I recently started building AI infra from scratch. It has been an interesting forray into both learning the concepts in a way that's interesting and continually testing how powerful models are today.

  1. I started by building a Rust equivalent to vLLM, Temper. Helped me actually learn a bit about the challenges with actually hosting large models (they need to be split across GPUs, which I didn't actually do, etc.). While originally I wanted to the relative end to end latencies of different hosting providers like Cloudflare/Fly/Vercel to get the lowest latency experience, it's clear that the latency of a response comes from techniques around utilizing the GPU most effectively. This lead me down the path of focusing on those strategies rather than minimizing time to first token for an initial request. Also a lot of AI workloads are async so I don't care if they're slow.
  2. I then sort of tacked on a desktop app and local inference using on-device GPUs on Mac. Reducing cost is in vogue and walking around with a laptop with a bunch of RAM and a GPU feels silly if you're not going to use it. Quantization, model routing, etc.
  3. Finally, all this helped re-spark my interest in the fundamentals. I picked back up "Building an LLM from scratch".

Takeaways

  • This was the first time I built something (quickly) that was far beyond my current understanding. Just as quickly as I incorporated the concepts of chunked prefill, continuous batching, etc. I forgot them. I had to go back and look at the plans the AI had crafted to even remind myself of the concepts introduced. I understood at a high level the tradeoffs and stages of our implementation, but the normal process of understanding something through building it was less effective (as building it was often simply saying "what should the next plan be?" or "our performance is slower than vllm when benchmarked, why?").
  • I did most of this on my phone while on vacation. I often chose Claude because Codex on mobile periodically died and it feels like their infra is less reliable, would occassionally die.
  • AI at its current levels of intelligence can already be a completely automated software engineer, it's simply incumbered by permissions, capabilities (what tools it has access to), and direction. This last point is the most interesting given this exercise, and how I imagine many people will interact with AI in the coming years. I didn't actually know what the next steps were or really have a sense for what the ideal outcome of a phase of the project should be. Many people when building a business, project, etc. won't either. Much like a junior employee, AI today won't always fill in the gaps. It'll stop early, sit around and wait for instruction. Much of today's hubbub surrounds pausing or slowing AI research - I wonder, will this sort of improvement/closing of the feedback look in direction be considered AI research or is this simply run of the mill engineering/product improvement? If the latter, then AI research might be considered simply increasing the intelligence of models; pausing AI research today might have no effect on what I really needed: self-directed AI with the current intelligence level.
  • Books like "Building an LLM from scratch" are simultaneously timeless and immediately out of date. On the one hand, it's examples and sections are all outdated, referencing GPT 3 a lot. On the other hand, I'm able to read a section like "Working with text data" and understand years of background in ~an hour. Tack on a follow on conversation with AI about how these methods have changed / evolved in the time since publish and you have a rich foundation and landscape. Every day feels like a lifetime in AI wonderland, with new discoveries by the hour. But there's something to learning that feels evergreen.
  • Finally, just the overwhelming since of me not contributing much to this process! I guess it's similar to any learning process / building process when you're not an expert. You're really just distilling learning / information from other sources but you're not really contributing much other than cludgely putting it into code. With the value of code-writing / distilling ideas into code gone to zero, all that remains is just ideas.