Skip to content

AI 工程从零开始

Main Navigation 首页开始学习 ← 返回 learn.traeai.com

主题

Sidebar Navigation

00. 环境搭建与工具

Dev Environment

Git & Collaboration

GPU Setup & Cloud

APIs & Keys

Jupyter Notebooks

Python Environments

Docker for AI

Editor Setup

Data Management

Terminal & Shell

Linux for AI

Debugging and Profiling

01. 数学基础

Linear Algebra Intuition

Vectors, Matrices & Operations

Matrix Transformations

Calculus for Machine Learning

Chain Rule & Automatic Differentiation

Probability and Distributions

Bayes' Theorem

Optimization

Information Theory

Dimensionality Reduction

Singular Value Decomposition

Tensor Operations

Numerical Stability

Norms and Distances

Statistics for Machine Learning

Sampling Methods

Linear Systems

Convex Optimization

Complex Numbers for AI

The Fourier Transform

Graph Theory for Machine Learning

Stochastic Processes

02. 机器学习基础

What Is Machine Learning

Linear Regression

Logistic Regression

Decision Trees and Random Forests

Support Vector Machines

K-Nearest Neighbors and Distances

Unsupervised Learning

Feature Engineering & Selection

Model Evaluation

Bias-Variance Tradeoff

Ensemble Methods

Hyperparameter Tuning

ML Pipelines

Naive Bayes

Time Series Fundamentals

Anomaly Detection

Handling Imbalanced Data

Feature Selection

03. 深度学习核心

The Perceptron

Multi-Layer Networks and Forward Pass

Backpropagation from Scratch

Activation Functions

Loss Functions

Optimizers

Regularization

Weight Initialization and Training Stability

Learning Rate Schedules and Warmup

Build Your Own Mini Framework

Introduction to PyTorch

Introduction to JAX

Debugging Neural Networks

04. 计算机视觉

Image Fundamentals — Pixels, Channels, Color Spaces

Convolutions from Scratch

CNNs — LeNet to ResNet

Image Classification

Transfer Learning & Fine-Tuning

Object Detection — YOLO from Scratch

Semantic Segmentation — U-Net

Instance Segmentation — Mask R-CNN

Image Generation — GANs

Image Generation — Diffusion Models

Stable Diffusion — Architecture & Fine-Tuning

Video Understanding — Temporal Modeling

3D Vision — Point Clouds & NeRFs

Vision Transformers (ViT)

Real-Time Vision — Edge Deployment

Build a Complete Vision Pipeline — Capstone

Self-Supervised Vision — SimCLR, DINO, MAE

Open-Vocabulary Vision — CLIP

OCR & Document Understanding

Image Retrieval & Metric Learning

Keypoint Detection & Pose Estimation

3D Gaussian Splatting from Scratch

Diffusion Transformers & Rectified Flow

SAM 3 & Open-Vocabulary Segmentation

Vision-Language Models — The ViT-MLP-LLM Pattern

Monocular Depth & Geometry Estimation

Multi-Object Tracking & Video Memory

World Models & Video Diffusion

05. 自然语言处理

Text Processing — Tokenization, Stemming, Lemmatization

Bag of Words, TF-IDF, and Text Representation

Word Embeddings — Word2Vec from Scratch

GloVe, FastText, and Subword Embeddings

Sentiment Analysis

Named Entity Recognition

POS Tagging and Syntactic Parsing

CNNs and RNNs for Text

Sequence-to-Sequence Models

Attention Mechanism — The Breakthrough

Machine Translation

Text Summarization

Question Answering Systems

Information Retrieval and Search

Topic Modeling — LDA and BERTopic

Text Generation Before Transformers — N-gram Language Models

Chatbots — Rule-Based to Neural to LLM Agents

Multilingual NLP

Subword Tokenization — BPE, WordPiece, Unigram, SentencePiece

Structured Outputs & Constrained Decoding

Natural Language Inference — Textual Entailment

Embedding Models — The 2026 Deep Dive

Chunking Strategies for RAG

Coreference Resolution

Entity Linking & Disambiguation

Relation Extraction & Knowledge Graph Construction

LLM Evaluation — RAGAS, DeepEval, G-Eval

Long-Context Evaluation — NIAH, RULER, LongBench, MRCR

Dialogue State Tracking

06. 语音与音频

Audio Fundamentals — Waveforms, Sampling, Fourier Transform

Spectrograms, Mel Scale & Audio Features

Audio Classification — From k-NN on MFCCs to AST and BEATs

Speech Recognition (ASR) — CTC, RNN-T, Attention

Whisper — Architecture & Fine-Tuning

Speaker Recognition & Verification

Text-to-Speech (TTS) — From Tacotron to F5 and Kokoro

Voice Cloning & Voice Conversion

Music Generation — MusicGen, Stable Audio, Suno, and the Licensing Earthquake

Audio-Language Models — Qwen2.5-Omni, Audio Flamingo, GPT-4o Audio

Real-Time Audio Processing

Build a Voice Assistant Pipeline — The Phase 6 Capstone

Neural Audio Codecs — EnCodec, SNAC, Mimi, DAC and the Semantic-Acoustic Split

Voice Activity Detection & Turn-Taking — Silero, Cobra, and the Flush Trick

Streaming Speech-to-Speech — Moshi, Hibiki, and Full-Duplex Dialogue

Voice Anti-Spoofing & Audio Watermarking — ASVspoof 5, AudioSeal, WaveVerify

Audio Evaluation — WER, MOS, UTMOS, MMAU, FAD, and the Open Leaderboards

07. Transformer 深入

Why Transformers — The Problems with RNNs

Self-Attention from Scratch

Multi-Head Attention

Positional Encoding — Sinusoidal, RoPE, ALiBi

The Full Transformer — Encoder + Decoder

BERT — Masked Language Modeling

GPT — Causal Language Modeling

T5, BART — Encoder-Decoder Models

Vision Transformers (ViT)

Audio Transformers — Whisper Architecture

Mixture of Experts (MoE)

KV Cache, Flash Attention & Inference Optimization

Scaling Laws

Build a Transformer from Scratch — The Capstone

Attention Variants — Sliding Window, Sparse, Differential

Speculative Decoding — Draft, Verify, Repeat

08. 生成式 AI

Generative Models — Taxonomy & History

Autoencoders & Variational Autoencoders (VAE)

GANs — Generator vs Discriminator

Conditional GANs & Pix2Pix

StyleGAN

Diffusion Models — DDPM from Scratch

Latent Diffusion & Stable Diffusion

ControlNet, LoRA & Conditioning

Inpainting, Outpainting & Image Editing

Video Generation

Audio Generation

3D Generation

Flow Matching & Rectified Flows

Evaluation — FID, CLIP Score, Human Preference

Visual Autoregressive Modeling (VAR): Next-Scale Prediction

09. 强化学习

MDPs, States, Actions & Rewards

Dynamic Programming — Policy Iteration & Value Iteration

Monte Carlo Methods — Learning from Complete Episodes

Temporal Difference — Q-Learning & SARSA

Deep Q-Networks (DQN)

Policy Gradient — REINFORCE from Scratch

Actor-Critic — A2C and A3C

Proximal Policy Optimization (PPO)

Reward Modeling & RLHF

Multi-Agent RL

Sim-to-Real Transfer

RL for Games — AlphaZero, MuZero, and the LLM-Reasoning Era

10. 从零构建 LLM

Tokenizers: BPE, WordPiece, SentencePiece

Building a Tokenizer from Scratch

Data Pipelines for Pre-Training

Pre-Training a Mini GPT (124M Parameters)

Scaling: Distributed Training, FSDP, DeepSpeed

Instruction Tuning (SFT)

RLHF: Reward Model + PPO

DPO: Direct Preference Optimization

Constitutional AI and Self-Improvement

Evaluation: Benchmarks, Evals, LM Harness

Quantization: Making Models Fit

Inference Optimization

Building a Complete LLM Pipeline

Open Models: Architecture Walkthroughs

Speculative Decoding and EAGLE-3

Differential Attention (V2)

Native Sparse Attention (DeepSeek NSA)

Multi-Token Prediction (MTP)

DualPipe Parallelism

DeepSeek-V3 Architecture Walkthrough

Jamba — Hybrid SSM-Transformer

Async and Hogwild! Inference

Speculative Decoding and EAGLE

Gradient Checkpointing and Activation Recomputation

11. LLM 工程

Prompt Engineering: Techniques & Patterns

Few-Shot, Chain-of-Thought, Tree-of-Thought

Structured Outputs: JSON, Schema Validation, Constrained Decoding

Embeddings & Vector Representations

Context Engineering: Windows, Budgets, Memory, and Retrieval

RAG (Retrieval-Augmented Generation)

Advanced RAG (Chunking, Reranking, Hybrid Search)

Fine-Tuning with LoRA & QLoRA

Function Calling & Tool Use

Evaluation & Testing LLM Applications

Caching, Rate Limiting & Cost Optimization

Guardrails, Safety & Content Filtering

Building a Production LLM Application

Model Context Protocol (MCP)

Prompt Caching and Context Caching

LangGraph — State Machines for Agents

Agent Framework Tradeoffs — LangGraph vs CrewAI vs AutoGen vs Agno

12. 多模态 AI

Vision Transformers and the Patch-Token Primitive

CLIP and Contrastive Vision-Language Pretraining

From CLIP to BLIP-2 — Q-Former as Modality Bridge

Flamingo and Gated Cross-Attention for Few-Shot VLMs

LLaVA and Visual Instruction Tuning

Any-Resolution Vision: Patch-n'-Pack and NaFlex

Open-Weight VLM Recipes: What Actually Matters

LLaVA-OneVision: Single-Image, Multi-Image, Video in One Model

Qwen-VL Family and Dynamic-FPS Video

InternVL3: Native Multimodal Pretraining

Chameleon and Early-Fusion Token-Only Multimodal Models

Emu3: Next-Token Prediction for Image and Video Generation

Transfusion: Autoregressive Text + Diffusion Image in One Transformer

Show-o and Discrete-Diffusion Unified Models

Janus-Pro: Decoupled Encoders for Unified Multimodal Models

MIO and Any-to-Any Streaming Multimodal Models

Video-Language Models: Temporal Tokens and Grounding

Long-Video Understanding at Million-Token Context

Audio-Language Models: the Whisper to Audio Flamingo 3 Arc

Omni Models: Qwen2.5-Omni and the Thinker-Talker Split

Embodied VLAs: RT-2, OpenVLA, π0, GR00T

Document and Diagram Understanding

ColPali and Vision-Native Document RAG

Multimodal RAG and Cross-Modal Retrieval

Multimodal Agents and Computer-Use (Capstone)

13. 工具与协议

The Tool Interface — Why Agents Need Structured I/O

Function Calling Deep Dive — OpenAI, Anthropic, Gemini

Parallel Tool Calls and Streaming with Tools

Structured Output — JSON Schema, Pydantic, Zod, Constrained Decoding

Tool Schema Design — Naming, Descriptions, Parameter Constraints

MCP Fundamentals — Primitives, Lifecycle, JSON-RPC Base

Building an MCP Server — Python + TypeScript SDKs

Building an MCP Client — Discovery, Invocation, Session Management

MCP Transports — stdio vs Streamable HTTP vs SSE Migration

MCP Resources and Prompts — Context Exposure Beyond Tools

MCP Sampling — Server-Requested LLM Completions and Agent Loops

Roots and Elicitation — Scoping and Mid-Flight User Input

Async Tasks (SEP-1686) — Call-Now, Fetch-Later for Long-Running Work

MCP Apps — Interactive UI Resources via `ui://`

MCP Security I — Tool Poisoning, Rug Pulls, Cross-Server Shadowing

MCP Security II — OAuth 2.1, Resource Indicators, Incremental Scopes

MCP Gateways and Registries — Enterprise Control Planes

MCP Auth in Production — DCR, JWKS Rotation, Audience-Pinned Tokens on iii Primitives

A2A — Agent-to-Agent Protocol

OpenTelemetry GenAI — Tracing Tool Calls End-to-End

LLM Routing Layer — LiteLLM, OpenRouter, Portkey

Skills and Agent SDKs — Anthropic Skills, AGENTS.md, OpenAI Apps SDK

Capstone — Build a Complete Tool Ecosystem

14. Agent 工程

The Agent Loop: Observe, Think, Act

ReWOO and Plan-and-Execute: Decoupled Planning

Reflexion: Verbal Reinforcement Learning

Tree of Thoughts and LATS: Deliberate Search

Self-Refine and CRITIC: Iterative Output Improvement

Tool Use and Function Calling

Memory: Virtual Context and MemGPT

Memory Blocks and Sleep-Time Compute (Letta)

Hybrid Memory: Vector + Graph + KV (Mem0)

Skill Libraries and Lifelong Learning (Voyager)

Planning with HTN and Evolutionary Search

Anthropic's Workflow Patterns: Simple Over Complex

LangGraph: Stateful Graphs and Durable Execution

AutoGen v0.4: Actor Model and Agent Framework

CrewAI: Role-Based Crews and Flows

OpenAI Agents SDK: Handoffs, Guardrails, Tracing

Claude Agent SDK: Subagents and Session Store

Agno and Mastra: Production Runtimes

Benchmarks: SWE-bench, GAIA, AgentBench

Benchmarks: WebArena and OSWorld

Computer Use: Claude, OpenAI CUA, Gemini

Voice Agents: Pipecat and LiveKit

OpenTelemetry GenAI Semantic Conventions

Agent Observability: Langfuse, Phoenix, Opik

Multi-Agent Debate and Collaboration

Failure Modes: Why Agents Break

Prompt Injection and the PVE Defense

Orchestration Patterns: Supervisor, Swarm, Hierarchical

Production Runtimes: Queue, Event, Cron

Eval-Driven Agent Development

Agent Workbench Engineering: Why Capable Models Still Fail

The Minimal Agent Workbench

Agent Instructions as Executable Constraints

Repo Memory and Durable State

Initialization Scripts for Agents

Scope Contracts and Task Boundaries

Runtime Feedback Loops

Verification Gates

Reviewer Agent: Separate Builder from Marker

Multi-Session Handoff

The Workbench on a Real Repo

Capstone: Ship a Reusable Agent Workbench Pack

15. 自主系统

The Shift from Chatbots to Long-Horizon Agents

STaR, V-STaR, Quiet-STaR — Self-Taught Reasoning

AlphaEvolve — Evolutionary Coding Agents

Darwin Godel Machine — Open-Ended Self-Modifying Agents

AI Scientist v2 — Workshop-Level Autonomous Research

Automated Alignment Research (Anthropic AAR)

Recursive Self-Improvement — Capability vs Alignment

Bounded Self-Improvement Designs

The Autonomous Coding Agent Landscape (2026)

Claude Code as an Autonomous Agent: Permission Modes and Auto Mode

Browser Agents and Long-Horizon Web Tasks

Long-Running Background Agents: Durable Execution

Action Budgets, Iteration Caps, and Cost Governors

Kill Switches, Circuit Breakers, and Canary Tokens

Human-in-the-Loop: Propose-Then-Commit

Checkpoints and Rollback

Constitutional AI and Rule Overrides

Llama Guard and Input/Output Classification

Anthropic Responsible Scaling Policy v3.0

OpenAI Preparedness Framework and DeepMind Frontier Safety Framework

METR Time Horizons and External Capability Evaluation

CAIS, CAISI, and Societal-Scale Risk

16. 多 Agent 与集群

Why Multi-Agent?

Heritage of FIPA-ACL and Speech Acts

Communication Protocols

The Multi-Agent Primitive Model

Supervisor / Orchestrator-Worker Pattern

Hierarchical Architecture and Its Failure Mode

Society of Mind and Multi-Agent Debate

Role Specialization — Planner, Critic, Executor, Verifier

Parallel / Swarm / Networked Architectures

Group Chat and Speaker Selection

Handoffs and Routines — Stateless Orchestration

A2A — The Agent-to-Agent Protocol

Shared Memory and Blackboard Patterns

Consensus and Byzantine Fault Tolerance for Agents

Voting, Self-Consistency, and Debate Topology

Negotiation and Bargaining

Generative Agents and Emergent Simulation

Theory of Mind and Emergent Coordination

Swarm Optimization for LLMs (PSO, ACO)

MARL — MADDPG, QMIX, MAPPO

Agent Economies, Token Incentives, Reputation

Production Scaling — Queues, Checkpoints, Durability

Failure Modes — MAST, Groupthink, Monoculture, Cascading Errors

Evaluation and Coordination Benchmarks

Case Studies and the 2026 State of the Art

17. 基础设施与生产

Managed LLM Platforms — Bedrock, Vertex AI, Azure OpenAI

Inference Platform Economics — Fireworks, Together, Baseten, Modal, Replicate, Anyscale

GPU Autoscaling on Kubernetes — Karpenter, KAI Scheduler, Gang Scheduling

vLLM Serving Internals: PagedAttention, Continuous Batching, Chunked Prefill

EAGLE-3 Speculative Decoding in Production

SGLang and RadixAttention for Prefix-Heavy Workloads

TensorRT-LLM on Blackwell with FP8 and NVFP4

Inference Metrics — TTFT, TPOT, ITL, Goodput, P99

Production Quantization — AWQ, GPTQ, GGUF K-quants, FP8, MXFP4/NVFP4

Cold Start Mitigation for Serverless LLMs

Multi-Region LLM Serving and KV Cache Locality

Edge Inference — Apple Neural Engine, Qualcomm Hexagon, WebGPU/WebLLM, Jetson

LLM Observability Stack Selection

Prompt Caching and Semantic Caching Economics

Batch APIs — the 50% Discount as Industry Standard

Model Routing as a Cost-Reduction Primitive

Disaggregated Prefill/Decode — NVIDIA Dynamo and llm-d

vLLM Production Stack with LMCache KV Offloading

AI Gateways — LiteLLM, Portkey, Kong AI Gateway, Bifrost

Shadow Traffic, Canary Rollout, and Progressive Deployment for LLMs

A/B Testing LLM Features — GrowthBook, Statsig, and the Vibes Problem

Load Testing LLM APIs — Why k6 and Locust Lie

SRE for AI — Multi-Agent Incident Response, Runbooks, Predictive Detection

Chaos Engineering for LLM Production

Security — Secrets, API Key Rotation, Audit Logs, Guardrails

Compliance — SOC 2, HIPAA, GDPR, PCI-DSS, EU AI Act, ISO 42001

FinOps for LLMs — Unit Economics and Multi-Tenant Attribution

Self-Hosted Serving Selection — llama.cpp, Ollama, TGI, vLLM, SGLang

18. 伦理、安全与对齐

Instruction-Following as Alignment Signal

Reward Hacking and Goodhart's Law

The Direct Preference Optimization Family

Sycophancy as RLHF Amplification

Constitutional AI and RLAIF

Mesa-Optimization and Deceptive Alignment

Sleeper Agents — Persistent Deception

In-Context Scheming in Frontier Models

Alignment Faking

AI Control — Safety Despite Subversion

Scalable Oversight and Weak-to-Strong Generalization

Red-Teaming: PAIR and Automated Attacks

Many-Shot Jailbreaking

ASCII Art and Visual Jailbreaks

Indirect Prompt Injection — Production Attack Surface

Red-Team Tooling — Garak, Llama Guard, PyRIT

WMDP and Dual-Use Capability Evaluation

Frontier Safety Frameworks — RSP, PF, FSF

Anthropic's Model Welfare Program

Bias and Representational Harm in LLMs

Fairness Criteria — Group, Individual, Counterfactual

Differential Privacy for LLMs

Watermarking — SynthID, Stable Signature, C2PA

Regulatory Frameworks — EU, US, UK, Korea

EchoLeak and the Emergence of CVEs for AI

Model, System, and Dataset Cards

Data Provenance and Training-Data Governance

Alignment Research Ecosystem — MATS, Redwood, Apollo, METR

Moderation Systems — OpenAI, Perspective, Llama Guard

Dual-Use Risk — Cyber, Bio, Chem, Nuclear Uplift

19. 毕业项目

Capstone 01 — Terminal-Native Coding Agent

Capstone 02 — RAG over Codebase (Cross-Repo Semantic Search)

Capstone 03 — Real-Time Voice Assistant (ASR to LLM to TTS)

Capstone 04 — Multimodal Document QA (Vision-First PDF, Tables, Charts)

Capstone 05 — Autonomous Research Agent (AI-Scientist Class)

Capstone 06 — DevOps Troubleshooting Agent for Kubernetes

Capstone 07 — End-to-End Fine-Tuning Pipeline (Data to SFT to DPO to Serve)

Capstone 08 — Production RAG Chatbot for a Regulated Vertical

Capstone 09 — Code Migration Agent (Repo-Level Language / Runtime Upgrade)

Capstone 10 — Multi-Agent Software Engineering Team

Capstone 11 — LLM Observability & Eval Dashboard

Capstone 12 — Video Understanding Pipeline (Scene, QA, Search)

Capstone 13 — MCP Server with Registry and Governance

Capstone 14 — Speculative-Decoding Inference Server

Capstone 15 — Constitutional Safety Harness + Red-Team Range

Capstone 16 — GitHub Issue-to-PR Autonomous Agent

Capstone 17 — Personal AI Tutor (Adaptive, Multimodal, with Memory)

Agent Harness Loop Contract

Tool Registry with Schema Validation

JSON-RPC 2.0 Over Newline-Delimited Stdio

Function Call Dispatcher

Plan-Execute Control Flow

Capstone Lesson 25: Verification Gates and the Observation Budget

Capstone Lesson 26: Sandbox Runner with Denylist and Path Jail

Capstone Lesson 27: Eval Harness with Fixture Tasks

Capstone Lesson 28: Observability with OTel GenAI Spans and Prometheus Metrics

Capstone Lesson 29: End-to-End Coding Agent on the Harness

BPE Tokenizer From Scratch

Tokenized Dataset with Sliding Window

Token and Positional Embeddings

Multi-Head Self-Attention

Transformer Block from Scratch

GPT Model Assembly

Training Loop and Evaluation

Loading Pretrained Weights

Capstone Lesson 38: Classifier Fine-Tuning by Head Swap

Capstone Lesson 39: Instruction Tuning by Supervised Fine-Tuning

Capstone Lesson 40: Direct Preference Optimization from Scratch

Capstone Lesson 41: Full Evaluation Pipeline

Large Corpus Downloader

HDF5 Tokenized Corpus

Cosine LR with Linear Warmup

Gradient Clipping and Mixed Precision

Gradient Accumulation

Checkpoint Save and Resume

Distributed Data Parallel and FSDP from Scratch

Language Model Evaluation Harness

Hypothesis Generator

Literature Retrieval

Experiment Runner

Result Evaluator

Paper Writer

Critic Loop

Iteration Scheduler

End-to-End Research Demo

目录

Phase 03: 深度学习核心

本阶段包含 13 课时。

原始课程来源：AI Engineering from Scratch (MIT License)

The Perceptron
Multi-Layer Networks and Forward Pass
Backpropagation from Scratch
Activation Functions
Loss Functions
Optimizers
Regularization
Weight Initialization and Training Stability
Learning Rate Schedules and Warmup
Build Your Own Mini Framework
Introduction to PyTorch
Introduction to JAX
Debugging Neural Networks

Pager

下一课Dev Environment

基于 AI Engineering from Scratch (MIT License) 整理

© 2025 learn.traeai.com