0

๐Ÿ“˜ The Complete Guide to LLMs and AI Agents ๐Ÿค– - Everything from how a word becomes a token to how an agent books your flight - Part 2 ๐Ÿš€

Who is this for? Anyone who wants to understand modern AI deeply โ€” not just use it. Engineers, curious learners, and interview candidates who want the why behind the buzzwords, laid out in one place, in plain English.


Table of Contents

  1. ๐Ÿง  The Big Picture: What is an LLM?
  2. ๐Ÿ”ค Step 0: Tokenization โ€” Turning Words into Numbers
  3. ๐Ÿ“ Step 1: Embeddings โ€” Giving Numbers Meaning
  4. ๐Ÿ“ Step 2: Positional Encoding โ€” Teaching the Model Word Order
  5. ๐Ÿ‘๏ธ Step 3: The Attention Mechanism โ€” How Words Talk to Each Other
  6. ๐Ÿ‘€ Step 4: Multi-Head Attention โ€” Multiple Perspectives at Once
  7. ๐Ÿ—๏ธ Step 5: Multiple Layers โ€” Going Deeper
  8. ๐Ÿงฎ Step 6: The Feed-Forward Network โ€” Where Knowledge Lives
  9. ๐ŸŽฏ Step 7: Decoding โ€” Turning Numbers Back into Words
  10. โšก The KV Cache โ€” The Speed Trick That Makes Everything Practical
  11. ๐Ÿ”ง The Transformer: Putting It All Together
  12. ๐ŸŽ“ How LLMs Are Trained
  13. ๐ŸŽ›๏ธ Fine-Tuning: Teaching an Old Model New Tricks
  14. โœ๏ธ Prompt Engineering: Talking to the Model Intelligently
  15. ๐Ÿ“š RAG: Giving the Model a Memory
  16. ๐Ÿ—„๏ธ Vector Databases: The Filing Cabinet for Meaning
  17. ๐Ÿค– AI Agents: From Answering Questions to Taking Action
  18. ๐Ÿค Multi-Agent Systems: Teamwork Among AIs
  19. ๐Ÿ“Š Evaluation: How Do You Know It's Actually Working?
  20. ๐Ÿš€ Production Engineering: Shipping AI That Doesn't Break
  21. ๐Ÿ›ก๏ธ Safety and Security
  22. ๐Ÿ—บ๏ธ The Mental Model: Everything in One Map
  23. ๐Ÿญ The End-to-End Lifecycle: From Raw Files to a Production API Call
  24. ๐Ÿ“ˆ Scaling Laws โ€” Why Model Size Isn't Everything
  25. ๐Ÿ–ผ๏ธ Multimodality โ€” When Tokens Aren't Just Words
  26. ๐Ÿงช Knowledge Distillation โ€” Teaching Small Models to Punch Above Their Weight
  27. ๐Ÿ“‹ Structured Output Generation โ€” Guaranteeing the Format
  28. ๐Ÿ“ Long-Context Challenges
  29. ๐Ÿ”’ Guardrails as Infrastructure
  30. ๐Ÿ† Benchmarks โ€” How to Actually Read Them
  31. โš–๏ธ Constitutional AI & RLAIF โ€” AI Teaching AI

Section 1 -> 22: Read Part 1 here https://viblo.asia/p/the-complete-guide-to-llms-and-ai-agents-everything-from-how-a-word-becomes-a-token-to-how-an-agent-books-your-flight-part-1-gjLN0YbW432

23. ๐Ÿญ The End-to-End Lifecycle: From Raw Files to a Production API Call

Section 22 mapped the field conceptually. This section maps it physically โ€” the actual files that get created, transformed, and shipped, from a folder of messy documents to a JSON response landing in a user's app. Three diagrams, one continuous journey.

Not to be confused with RAG ingestion (Section 15). RAG's "Ingest โ†’ Chunk โ†’ Embed โ†’ Index" pipeline feeds an external vector database that the model reads at query time โ€” no weights change. The pipeline below feeds training โ€” the documents are baked directly into the model's weights via backpropagation. Same-looking file formats, completely different destination.

Diagram 1: Raw Files โ†’ Training-Ready Tensors (the Data Pipeline)

Two different sources feed this pipeline at two very different scales, but the shape is the same:

  • Foundation-model pretraining: web-scale โ€” Common Crawl (.warc/.wet), GitHub code, Wikipedia dumps, books, arXiv papers. Terabytes to petabytes.
  • Domain fine-tuning or a RAG knowledge base: your own files โ€” .pdf, .docx, .xlsx, .html, .csv. Megabytes to gigabytes.
[ Raw Sources ]                    [ 1. Extraction ]                  [ 2. Cleaning & Dedup ]
 Web-scale: Common Crawl,            Unstructured / PyPDF /              Language-ID + quality
  GitHub, Wikipedia, books    โ”€โ”€โ”€>    LlamaParse / python-docx    โ”€โ”€โ”€>    filters, then MinHash
 Your own docs: .pdf .docx             pull plain text / Markdown          + LSH near-dup removal
  .xlsx .html .csv                     (tables โ†’ MD tables or JSON)                โ”‚
                                                                                   โ–ผ
[ 5. Packed Training Shards ] <โ”€โ”€ [ 4. Pack (pretrain) /  <โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ [ 3. Tokenizer Training + Tokenization ]
 .bin/.idx, Arrow, or Parquet          Pad (fine-tune) ]              BPE/Unigram learns a FIXED vocab once
 shards of int32/int64 token IDs    fixed-length blocks,               โ†’ tokenizer.model / tokenizer.json
 (100s of GB โ€“ many TB)             no padding needed for              then every document is converted to
                                    pretrain; attention                an integer ID sequence using it
                                    masks for fine-tune/inference
Stage Tool / Algorithm Output Artifact Typical Size
Extraction Unstructured, PyPDF, python-docx, LlamaParse Plain text / Markdown; tables โ†’ MD or JSON Varies with source
Cleaning & dedup Language-ID + quality classifiers, MinHash + LSH for near-duplicate removal Filtered JSONL/Parquet shards Pretraining corpora: TBs; a company's doc set: MBsโ€“GBs
Tokenizer training (done once) BPE (GPT family) or SentencePiece/Unigram (Llama, T5) tokenizer.model, tokenizer.json, vocab.json + merges.txt 1โ€“10 MB (scales with vocab size: 32K vs. 128K+ tokens)
Tokenization (applied to everything) The vocab trained above Integer token-ID sequences โ€”
Packing / padding Fixed-length blocks (2Kโ€“128K tokens) for pretraining; padding + attention masks for fine-tuning/inference batches .bin+.idx (Megatron-style), WebDataset/Arrow shards Full training set: 100s of GB โ€“ TBs

Important note: embedding lookup is not a pipeline step you run once and save to disk โ€” it's a trainable weight matrix that lives inside the model itself (Section 3), looked up fresh on every forward pass. Everything this pipeline produces is just integers (token IDs). The model is what turns those integers into meaning.

Diagram 2: Training Run โ†’ Model Artifacts (what actually ships)

The compute path itself โ€” embeddings โ†’ attention โ†’ FFN ร— N layers โ€” is Section 11's job. Here's what lands on disk when a training run finishes and gets pushed to a model repository:

[ Packed Tensors ] โ”€โ”€> [ Pre-train โ†’ SFT โ†’ RLHF/DPO ] โ”€โ”€> [ Checkpoint saved ] โ”€โ”€> [ Model Repository ]
    (Diagram 1)              (Section 12; weeks on                                   (what actually ships)
                               1,000s of GPUs)
File What It Is 7B Model 70B Model
model-0000X-of-0000N.safetensors The weights โ€” billions of numbers, sharded across files โ‰ˆ 14 GB (FP16) โ‰ˆ 140 GB (FP16)
model.safetensors.index.json Manifest mapping tensor names โ†’ shard file KBs KBs
config.json Architecture hyperparameters (layers, heads, hidden size, vocab size) < 50 KB < 50 KB
generation_config.json Default sampling settings, stop tokens < 5 KB < 5 KB
tokenizer.json / tokenizer.model The vocabulary from Diagram 1 1โ€“10 MB 1โ€“10 MB
special_tokens_map.json Reserved token names (<bos>, <eos>, <pad>) < 5 KB < 5 KB
README.md (model card) Who built it, intended use, benchmark scores, license tens of KB tens of KB
*.gguf (optional export) Single-file, quantized weights for CPU/local engines (llama.cpp, Ollama) โ‰ˆ 4โ€“5 GB (Q4 quant) โ‰ˆ 35โ€“40 GB (Q4 quant)

Important note: an LLM does not produce an .exe. Weights are arrays of numbers with zero attached logic โ€” model.safetensors cannot execute anything by itself. .safetensors also replaced the older pytorch_model.bin (a Python pickle file) specifically because pickle can run arbitrary code on load โ€” a real supply-chain risk when downloading weights from an untrusted source. The actual executable is the inference engine (PyTorch, vLLM, llama.cpp, TensorRT-LLM) that reads these number arrays and runs the Section 11 transformer math on them.

Diagram 3: Model Files โ†’ Live Inference Server (Production Serving)

This is where Section 10's prefill/decode split and Section 20's cost/latency levers become concrete engineering:

                     โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
[ Model Repo ]       โ”‚    INFERENCE ENGINE (vLLM / TGI / TensorRT-LLM).    โ”‚
 (S3 / GCS /   โ”€โ”€โ”€>  โ”‚  1. Stream .safetensors shards into GPU VRAM        โ”‚
  HF Hub)            โ”‚  2. Reserve remaining VRAM for KV cache,            โ”‚
                     โ”‚     split into fixed-size "pages" (PagedAttention)  โ”‚
                     โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
                                          โ”‚
                     โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
                     โ–ผ                                             โ–ผ
            [ Prefill (Section 10) ]                  [ Decode (Section 10) ]
            whole prompt at once,                       one new token per step,
            compute-bound โ†’ first token                 memory-bound โ†’ tokens/sec
            (this delay = TTFT)                                    โ”‚
                     โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
                                          โ–ผ
                     [ Continuous-batching scheduler ]
                      new requests join the running GPU batch every
                      step โ€” nobody waits for a free full-batch slot
                                          โ”‚
                                          โ–ผ
                     [ Detokenizer ] โ†’ [ SSE / chunked HTTP ] โ†’ [ Client ]

What actually crosses the wire, both directions:

Client โ†’ server (HTTP POST body):

{
  "model": "my-custom-llm-7b",
  "messages": [{ "role": "user", "content": "Summarize the Q3 trading trend." }],
  "temperature": 0.7,
  "max_tokens": 200,
  "stream": true
}

Server โ†’ client, one small JSON chunk per token while streaming (SSE):

data: {"choices":[{"index":0,"delta":{"content":"The"},"finish_reason":null}]}

data: {"choices":[{"index":0,"delta":{"content":" trading"},"finish_reason":null}]}

data: [DONE]

Once streaming ends, this is the shape a non-streaming call returns directly:

{
  "model": "my-custom-llm-7b",
  "choices": [{
    "index": 0,
    "message": { "role": "assistant", "content": "The trading trend shows..." },
    "finish_reason": "stop"
  }],
  "usage": { "prompt_tokens": 512, "completion_tokens": 42, "total_tokens": 554 }
}
Component Mechanism Why It Matters
Weight loading .safetensors shards streamed into VRAM; split across GPUs (tensor parallelism) if the model doesn't fit one A 70B model at FP16 (~140GB) needs 2ร— H100 (80GB) minimum
KV cache management PagedAttention โ€” allocates cache in fixed-size blocks like OS virtual-memory pages instead of one contiguous buffer per request Eliminates fragmentation; lets many concurrent conversations share GPU memory
Request scheduling Continuous batching โ€” injects/removes sequences from the active GPU batch every decode step instead of waiting for a fixed batch to finish Short requests never queue behind long ones; GPU stays saturated
Output delivery Detokenizer converts IDs back to text; wrapped as SSE (text/event-stream) or one final JSON blob Streaming = users see words in ~0.5s instead of waiting for the whole answer

In practice, applications never talk to the engine's raw HTTP directly โ€” they go through an SDK (OpenAI's, Anthropic's, or an internal wrapper) that constructs exactly the request JSON shown above and parses exactly that response shape. And the model repository itself typically lives in object storage (S3/GCS) or a model registry (Hugging Face Hub, an internal registry), pulled onto GPU nodes โ€” often a Kubernetes GPU node pool โ€” when the serving deployment starts or autoscales.


24. ๐Ÿ“ˆ Scaling Laws โ€” Why Model Size Isn't Everything

In 2022, DeepMind published the Chinchilla paper with a simple, important finding: most models were undertrained. They were made bigger, but fed too little data.

The key insight: model size and training tokens should scale together. The optimal rule of thumb is roughly 20 tokens of training data per model parameter:

A 7B parameter model โ†’ trained on ~140B tokens (optimal)
GPT-3 (175B params) โ†’ was trained on 300B tokens (undertrained by this rule)

Training a smaller model on more data often beats training a larger model on less data โ€” at lower cost and with faster inference.

What this means in practice:

  • A well-trained 7B model can outperform a poorly-trained 70B model
  • Model cards now report both parameter count and training token count โ€” read both
  • Leaderboard rankings are meaningless without knowing training compute budget

Test-Time Compute Scaling is a newer companion insight: you can also trade inference cost for quality. Instead of always using the biggest model, let a model "think longer" on hard problems (more reasoning tokens, more self-reflection steps). OpenAI's o-series, DeepSeek-R1, and Claude's extended thinking all do this. The implication: for hard tasks, it's sometimes cheaper to run a mid-sized model for 30 seconds than a giant model for 1 second.


25. ๐Ÿ–ผ๏ธ Multimodality โ€” When Tokens Aren't Just Words

Modern LLMs don't have to be text-only. Multimodal models (GPT-4o, Gemini, Claude 3+) can process images, audio, and video alongside text.

The trick is the same as always: convert everything into tokens.

Images: a vision encoder (typically a ViT โ€” Vision Transformer) divides the image into a grid of fixed-size patches (e.g., 16ร—16 pixels each), converts each patch into a vector, and feeds those patch-vectors into the LLM just like word embeddings. The model learns that certain patch patterns correspond to objects, edges, and scenes.

Image โ†’ grid of 256 patches โ†’ 256 "image tokens" โ†’ fed into Transformer alongside text tokens

Audio: similarly converted into spectrogram frames, which become tokens.

Why this matters: a multimodal model can answer "what's in this screenshot?" or "find the bug in this error image" without any special architecture โ€” it's the same attention mechanism, just with a richer token vocabulary.

The main limitation: image tokens are expensive. A single 1024ร—1024 image can consume 1,000+ tokens, making multimodal prompts much pricier than text-only ones.


26. ๐Ÿงช Knowledge Distillation โ€” Teaching Small Models to Punch Above Their Weight

The problem: large models are expensive to run. The solution: train a small model to mimic a large one.

This is knowledge distillation:

  1. Run a big "teacher" model on a large dataset
  2. Collect its outputs (not just the final answer โ€” the full probability distribution over tokens)
  3. Train a small "student" model to match those distributions

The student learns from soft targets (probability distributions) rather than hard labels. The teacher's probabilities contain rich information: when GPT-4 says "Paris" with 90% confidence and "Lyon" with 8%, the student learns that both cities are plausible French capitals โ€” not just that "Paris" is correct.

Real-world examples: Microsoft's Phi series, Meta's smaller Llama variants, and most "efficient" models are trained this way. A distilled 3B model can match GPT-3 (175B) on many benchmarks.

Takeaway: don't assume you need the biggest model. A well-distilled small model is often faster, cheaper, and surprisingly capable.


27. ๐Ÿ“‹ Structured Output Generation โ€” Guaranteeing the Format

Telling the model "respond in JSON" in the system prompt usually works. But LLMs are probabilistic โ€” sometimes they add explanation text before the JSON, sometimes they forget a required field, sometimes they produce malformed output.

For production pipelines, "usually" isn't good enough. Structured output generation guarantees the format at the token level.

How It Works

Instead of letting the model sample from the full vocabulary at each step, constrain the sampling to only tokens that are valid continuations of the target schema.

If the schema requires "name": "<string>", after the model outputs "name": ", the only valid next tokens are string content โ€” not }, not a number, not a newline.

Tools

Tool How It Works
OpenAI Structured Outputs Pass a JSON schema; the API guarantees schema compliance
instructor library Wraps any LLM API; uses Pydantic schemas; retries on validation failure
Outlines Grammar-constrained decoding; works at the serving layer; supports JSON, regex, and context-free grammars
Guidance Interleaves LLM calls and structured constraints in a single template

Default recommendation: use instructor + Pydantic for most API-based work. Use Outlines if you're self-hosting and need the constraint enforced at the GPU level.

The pattern:

from pydantic import BaseModel
import instructor

class MovieReview(BaseModel):
    title: str
    sentiment: Literal["positive", "negative", "neutral"]
    score: int  # 1-10

client = instructor.from_openai(openai.OpenAI())
review = client.chat.completions.create(
    model="gpt-4o",
    response_model=MovieReview,
    messages=[{"role": "user", "content": "Review Inception"}]
)
# review.sentiment is guaranteed to be one of three values โ€” no parsing needed

28. ๐Ÿ“ Long-Context Challenges

LLMs now offer 128K, 200K, even 1M token context windows. That sounds like a magic solution to memory limits. It isn't โ€” at least not yet.

The "Lost in the Middle" Problem

Research consistently shows that LLMs are best at attending to information at the very beginning and very end of a long context. Information buried in the middle gets underweighted, even if it's the most relevant.

Position:    [Start]  [Middle]  [End]
Attention:   โ–ˆโ–ˆโ–ˆโ–ˆ     โ–ˆโ–ˆ        โ–ˆโ–ˆโ–ˆโ–ˆ   โ† middle is weakest

Implication: if you stuff 200 pages into the context, the answer from page 150 may be ignored. Always put the most critical information at the start or end, not the middle.

Attention Gets Expensive

Attention is O(nยฒ) in sequence length โ€” doubling the context length quadruples the computation. Very long contexts are slow and expensive.

Strategies used in production:

Strategy What It Does
Sliding window attention Each token only attends to a fixed local window, not the full sequence (used in Mistral)
Context compression Summarize older parts of the conversation before they overflow the window
Selective retrieval Don't dump everything in the context; use RAG to fetch only relevant chunks
Hierarchical summarization Recursively summarize long docs into shorter representations

Practical rule: long context is great for occasional large inputs (one big document). It's not a substitute for RAG when you have many documents or need to update knowledge frequently.


29. ๐Ÿ”’ Guardrails as Infrastructure

Putting safety logic inside the prompt ("please don't say anything harmful") is fragile. A determined user or a prompt injection can bypass it.

Production safety is a layered system, not a single instruction:

[Input] โ†’ [Input Classifier] โ†’ [LLM] โ†’ [Output Classifier] โ†’ [User]
              โ†“ (block if harmful)            โ†“ (block if harmful)
           Reject early                    Catch what slipped through

The Three Layers

1. Input guardrails โ€” check the user's message before sending to the LLM:

  • PII detection (mask phone numbers, emails, SSNs)
  • Jailbreak/injection detection
  • Topic/intent classification (is this off-topic for this use case?)

2. LLM-level โ€” the model's own safety training (RLHF alignment) plus your system prompt instructions.

3. Output guardrails โ€” check the LLM's response before showing to the user:

  • Hallucination/faithfulness check
  • Toxicity/safety classifiers
  • Schema validation for structured outputs
  • Sensitive content detection (e.g., don't output a phone number from the retrieved docs)

Tools

  • LlamaGuard (Meta): open-source safety classifier, runs fast, good for input/output checking
  • NeMo Guardrails (NVIDIA): programmable guardrail framework with conversation flow control
  • Perspective API (Google): toxicity detection
  • Custom Pydantic validators: for output schema enforcement

Key principle: each layer catches different things. Don't rely on any single layer. Defense in depth.


30. ๐Ÿ† Benchmarks โ€” How to Actually Read Them

Model leaderboards rank models by benchmark scores. Here's what you need to know to not be misled.

Common Benchmarks

Benchmark Tests Watch Out For
MMLU Knowledge across 57 subjects (science, law, history...) Multiple-choice; doesn't test generation quality
HumanEval / MBPP Coding: write Python to pass unit tests Only tests small, self-contained problems
GPQA PhD-level science questions High-signal for true reasoning ability
MATH / GSM8K Math word problems GSM8K is now nearly saturated (models score 95%+)
HELM Holistic battery of 42 scenarios Broad coverage, slower to run
MT-Bench Multi-turn conversation quality, judged by GPT-4 Subjective; biased toward GPT-4's preferences

Benchmark Contamination

The biggest caveat: if a benchmark's questions appeared in the training data, the model has "memorized" the answers. Its score reflects memory, not understanding.

With models training on trillions of tokens scraped from the internet โ€” and benchmarks published openly โ€” contamination is common and hard to detect.

Red flags that a score might be contaminated:

  • The model scores dramatically higher than peers on one specific benchmark
  • A new model "beats GPT-4" on benchmarks but underperforms in practice
  • No decontamination methodology is described in the model card

Best practice: supplement public benchmark scores with your own eval on your own data. A model that scores 5% lower on MMLU but performs 20% better on your specific task is the right model for you.


31. โš–๏ธ Constitutional AI & RLAIF โ€” AI Teaching AI

The problem with RLHF: it requires human raters to review thousands of responses. Humans are expensive, slow, and inconsistent. Scaling to bigger models means scaling the human labeling effort too.

Constitutional AI (CAI), developed by Anthropic, offers an alternative:

  1. Define a constitution โ€” a set of principles (e.g., "be helpful, harmless, honest; don't help with illegal activity")
  2. Have the model critique its own responses against the constitution
  3. Have the model revise its response to better satisfy the principles
  4. Use those revised responses as training data

No human labels required for the safety-tuning phase.

RLAIF (Reinforcement Learning from AI Feedback) extends this: instead of a human reward model, use a strong AI (e.g., GPT-4, Claude) as the preference judge. The AI scores pairs of responses; those scores train the reward model.

Why it matters:

  • Dramatically reduces human labeling cost
  • Can scale with model size (bigger model = better judge)
  • Already used in production by most major labs alongside RLHF
  • Raises a philosophical concern: if AI judges AI, the resulting model reflects the judge model's biases, not independent human values

Practical takeaway: when you see a model described as "trained with AI feedback" or "self-improved," this is the mechanism. It's not magic โ€” it's the judge model's values being distilled into the student.


๐Ÿ’ก Closing Thoughts

The field moves fast โ€” new models, new techniques, new papers every week. But the fundamentals move slowly.

If you deeply understand:

  • How attention works (Q, K, V, multi-head, layers)
  • How generation works (autoregressive decoding, KV cache)
  • How RAG works (chunking, hybrid retrieval, faithfulness)
  • How agents work (tool use, memory, failure modes)
  • How to evaluate (golden sets, LLM-as-judge, regression testing)
  • How to ship (cost, latency, reliability, security)
  • How scaling works (Chinchilla laws, test-time compute, distillation)
  • How to stay safe (guardrail layers, structured outputs, benchmark contamination)

...then you can learn any new framework, any new model, any new tool within days. The abstractions on top change constantly; these foundations don't.

The best AI engineers aren't the ones who memorized the most API calls. They're the ones who understand why each piece of the system works the way it does, can reason about failure modes before they happen, and measure everything they ship.


๐Ÿ“– Companion Reads

This guide covers the foundations. These companion pieces go deeper on specific areas:

๐Ÿค– Agents & Agentic Systems

Title What It Covers
๐Ÿ—๏ธ Building High-Quality AI Agents โ€” A Comprehensive, Actionable Field Guide ๐Ÿ“š Production patterns for reliable, well-tested agents
๐Ÿค– The Agentic Loop ๐Ÿ”„ Loop Engineering: A Practical Field Guide ๐Ÿ“˜ Deep dive into the observe-reason-act loop and its variants
โš ๏ธ Common Issues with LLMs & AI Agents โ€” and How to Fix Them ๐Ÿ› ๏ธ Practical debugging guide for the failure modes listed in ยง17
๐Ÿ—๏ธ Harness Engineering: The Emerging Discipline of Making AI Agents Reliable ๐Ÿค– How to build reliable agent harnesses from first principles
๐Ÿ› ๏ธ Harness Engineering โ€” Quick Actionable Guide ๐Ÿค– Concise reference: tools, patterns, and guardrails for agent harnesses

๐Ÿ”ฌ Agent Framework Deep Dives

Title What It Covers
๐Ÿค– SWE-agent โ€” Deep Dive & Build-Your-Own Guide ๐Ÿ“˜ How SWE-agent navigates codebases autonomously
๐Ÿ™Œ OpenHands โ€” Deep Dive & Build-Your-Own Guide ๐Ÿ“š OpenHands architecture and sandboxed code execution
๐Ÿ”ฎ Hermes Agent โ€” Deep Dive & Build-Your-Own Guide ๐Ÿ“˜ Hermes self-improving agent orchestration patterns
๐Ÿค– nanobot: A Comprehensive Build-Your-Own Guide ๐Ÿ“š Minimal, composable agent design from scratch
๐Ÿค– Multica Deep Dive โ€” How to Build a Managed-Agents Platform ๐ŸŒ Multi-agent coordination and managed platform architecture
๐Ÿ“Ž Paperclip Deep Dive โ€” A Build Guide for an "AI Company" Control Plane Task-planning and goal decomposition in agents
๐ŸฆŠ GoClaw Deep Dive โ€” A Builder's Guide to a Multi-Tenant AI Agent Platform ๐Ÿ“˜ Multi-tenant agent platform design in Go
๐Ÿฆ€ PicoClaw Deep Dive โ€” A Field Guide to Building an Ultra-Light AI Agent in Go ๐Ÿน Minimal-footprint agent implementation in Go

๐Ÿ—๏ธ Building with AI

Title What It Covers
๐Ÿ—๏ธ Building Production-Grade Fullstack Products with AI Coding Agents โ€” A Practical Playbook ๐Ÿ“˜ End-to-end guide: backend, frontend, and agents working together
๐Ÿ—๏ธ Building Agents Like Claude Code โ€” A Source-Derived Blueprint ๐Ÿ“˜ Architecture patterns derived from Claude Code's source
๐Ÿค– The AI SaaS Playbook ๐Ÿ“˜ (Practical Edition) How to build and ship an AI-powered SaaS product
๐Ÿค– GPT-5.4 vs Claude Sonnet 4.6 vs Gemini 3.1 Pro โ€” Agent Coding Behavior in Four Test Scenarios ๐Ÿ“Š Side-by-side comparison of frontier models on real coding tasks

๐Ÿ’ผ Career & Leadership

Title What It Covers
๐ŸŽฏ The AI Engineer Interview Playbook ๐Ÿ“– 80 core interview questions from 4,894 job descriptions
๐Ÿ› ๏ธ The Senior Software Engineer Playbook: From Good Coder to High-Impact Engineer ๐Ÿš€ Engineering skills for senior+ roles in AI-heavy teams
๐Ÿ‘จโ€๐Ÿ’ป The CTO Playbook: From Best Builder to Best Bet โ™Ÿ๏ธ Technical leadership, architecture decisions, and AI strategy
๐Ÿ›๏ธ The Solution Architect Playbook: From Best Designer to Best Bridge ๐ŸŒ‰ System design with AI components at scale
๐Ÿง‘โ€๐Ÿ’ป The Tech Lead Playbook: From Best IC to Multiplier ๐Ÿš€ How tech leads amplify team output with AI

๐Ÿ› ๏ธ Coding Agent & Tooling

Title What It Covers
๐Ÿš€ Claude Code: From Zero to Pro ๐Ÿค– From setup to advanced agentic workflows
๐Ÿ’ป Vibe Coding Interview Guide: Ace AI-Assisted Coding Assessments ๐Ÿค– How to use AI assistants well in technical interviews
๐Ÿ“˜ Spec Kit vs. Superpowers โšก โ€” A Comprehensive Comparison & Practical Guide to Combining Both ๐Ÿš€ When to use Spec Kit vs Superpowers and how to combine them
๐ŸŒฑ Supspec Orchestration โ€” From Spec to Evidenced Draft PRs, Autonomously Autonomous spec-to-PR orchestration with verified evidence

If you found this helpful, let me know by leaving a ๐Ÿ‘ or a comment!, or if you think this post could help someone, feel free to share it! Thank you very much! ๐Ÿ˜ƒ


All Rights Reserved

Viblo
Let's register a Viblo Account to get more interesting posts.