RLHF 是如何把大模型对齐到人类偏好的？
How does RLHF align an LLM with human preferences?
先让人对模型输出排序，训练一个奖励模型学习“什么是更好的回答”；再用强化学习（如 PPO）优化语言模型，使其在保持流畅的同时最大化奖励模型给出的分数，从而贴近人类期望。
Humans rank model outputs to train a reward model of 'what is better', then reinforcement learning (e.g. PPO) optimizes the language model to maximize that reward while staying fluent, nudging it toward human-preferred behavior.
难度 专业 · 适合 大众
CC BY 4.0 · 已核验 · Verified 2026-08-23 · 2 条独立证据
来源 · Source
Russell & Norvig,《Artificial Intelligence: A Modern Approach》第4版,Pearson,2020
S. Russell and P. Norvig, Artificial Intelligence: A Modern Approach, 4th ed., Pearson, 2020
独立核验依据 · Independent verification: 核验依据 2 ARCHIVE 标识 isbn_9788177583670：Artificial Intelligence: A
#RLHF #对齐
自回归语言建模与掩码语言建模在训练目标上的本质差异,如何造成二者在生成与理解任务上的分化?BERT 的独立重建假设又牺牲了什么?
How does the fundamental difference between autoregressive and masked language modeling objectives split the two families between generation and understanding tasks, and what does BERT's independent-reconstruction assumption sacrifice?
自回归模型按链式法则从左到右分解联合概率,天然适配文本生成;掩码模型用双向上下文恢复被遮蔽的词,学到更适合分类、抽取等理解任务的表示。BERT 对每个掩码位独立预测,忽略了被遮蔽词之间的联合依赖,这正是 XLNet 采用置换语言建模试图修复的缺陷。
Autoregressive models factor the joint probability left-to-right via the chain rule and fit generation naturally; masked models reconstruct corrupted tokens from bidirectional context, learning representations better suited to classification and extraction. BERT predicts each masked position independently, ignoring joint dependencies among masked tokens — the very shortcoming XLNet's permutation language model set out to repair.
难度 专业 · 适合 高中(15-18)
CC BY 4.0 · 来源待核验 · Source pending review
来源 · Source
出处备注：著录含书名，但缺版次 / 出版社 / 年份中的一部分，需编辑补全后才能做独立比对。
Z. Yang 等,《XLNet:广义自回归预训练》,NeurIPS 2019;Devlin 等的 BERT,NAACL 2019
Z. Yang et al., XLNet: Generalized Autoregressive Pretraining, NeurIPS 2019; Devlin et al., BERT, NAACL 2019
#自然语言处理 #预训练目标
Transformer 大模型自回归解码时为什么要缓存历史的 Key 和 Value（KV cache）？它省去了哪些重复计算，代价是什么？
Why do Transformer LLMs cache past Keys and Values (the KV cache) during autoregressive decoding? Which repeated computation does it save, and at what cost?
自回归解码时，每个新 token 都要对之前全部 token 做自注意力。若无缓存，每一步都要重算历史位置在各层的 Key/Value 投影，单步开销随已生成长度上升。缓存后，每步只为新 token 计算一份 Q/K/V 并追加，历史 K/V 直接复用；代价是缓存显存随上下文线性膨胀，长上下文时反成吞吐瓶颈。
At each decoding step the new token must attend to every earlier token. Without a cache, the Key/Value projections for all previous positions would be recomputed layer by layer on every step, so per-step work grows with the length generated so far. With a KV cache, only the new token's Q/K/V are computed once and appended, while historical K/V pairs are simply reused. The trade-off is that cache memory grows linearly with context length and becomes the throughput bottleneck for long contexts.
难度 专业 · 适合 高中(15-18)
CC BY 4.0 · 已核验 · Verified 2026-11-04
来源 · Source
Kwon, Li, Zhuang, Sheng,《Efficient Memory Management for Large Language Model Serving with PagedAttention》,Proceedings of the 29th Symposium on Operating Systems Princ,2023 (DOI: 10.1145/3600006.3613165)
Kwon, Li, Zhuang, Sheng, 'Efficient Memory Management for Large Language Model Serving with PagedAttention', Proceedings of the 29th Symposium on Operating Systems Princ, 2023, DOI: 10.1145/3600006.3613165
独立核验依据 · Independent verification: 核验依据 1 Crossref DOI 10.1145/3600006.3613165
#注意力机制 #自回归解码 #推理优化
旋转位置编码（RoPE）解决了自注意力的什么问题？
What does Rotary Position Embedding (RoPE) solve for attention?
Transformer 本身不自带词序信息，RoPE 把位置编码成旋转矩阵乘到 Q、K 上，使两个词元的内积只依赖它们的相对位置与夹角，既注入顺序又便于外推到更长的序列。
Transformers have no built-in word order; RoPE rotates Q and K by position-dependent angles so their dot product depends mainly on relative position, injecting order while helping extrapolation to longer sequences.
难度 进阶 · 适合 大众
CC BY 4.0 · 已核验 · Verified 2026-08-23 · 2 条独立证据
来源 · Source
Russell & Norvig,《Artificial Intelligence: A Modern Approach》第4版,Pearson,2020
S. Russell and P. Norvig, Artificial Intelligence: A Modern Approach, 4th ed., Pearson, 2020
独立核验依据 · Independent verification: 核验依据 2 ARCHIVE 标识 isbn_9788177583670：Artificial Intelligence: A
#位置编码 #RoPE
模型量化（如 INT8、INT4）为什么能缩小模型体积并提速？
Why does quantization (e.g. INT8, INT4) shrink models and speed them up?
量化把原本用 16 位或 32 位浮点存储的权重改用 8 位甚至 4 位整数表示，数据量成倍减少、内存占用下降，且整数运算通常比浮点更快，代价是少量精度损失。
Quantization stores weights as 8-bit or even 4-bit integers instead of 16/32-bit floats, cutting size and memory bandwidth while integer math is often faster—at the cost of slight accuracy loss.
难度 进阶 · 适合 大众
CC BY 4.0 · 已核验 · Verified 2026-08-23 · 2 条独立证据
来源 · Source
Russell & Norvig,《Artificial Intelligence: A Modern Approach》第4版,Pearson,2020
S. Russell and P. Norvig, Artificial Intelligence: A Modern Approach, 4th ed., Pearson, 2020
独立核验依据 · Independent verification: 核验依据 2 ARCHIVE 标识 isbn_9788177583670：Artificial Intelligence: A
#量化 #压缩
微调（fine-tuning）和提示工程（prompt engineering）有什么不同？
How do fine-tuning and prompt engineering differ?
提示工程不改模型权重，只通过设计输入文字来引导已有模型；微调则用新数据更新部分或全部模型参数，让能力更持久地“固化”下来。前者快且零训练成本，后者更适配特定任务但需算力与数据。
Prompt engineering leaves weights untouched and only shapes the input text to steer the model; fine-tuning updates some or all parameters with new data to bake in a capability. The former is cheap and instant, the latter fits a task better but needs compute and data.
难度 进阶 · 适合 幼儿(3-6)
CC BY 4.0 · 已核验 · Verified 2026-08-23 · 2 条独立证据
来源 · Source
Russell & Norvig,《Artificial Intelligence: A Modern Approach》第4版,Pearson,2020
S. Russell and P. Norvig, Artificial Intelligence: A Modern Approach, 4th ed., Pearson, 2020
独立核验依据 · Independent verification: 核验依据 2 ARCHIVE 标识 isbn_9788177583670：Artificial Intelligence: A
#微调 #提示
混合专家模型（MoE）怎样在不大改算力的情况下扩大参数规模？
How do Mixture-of-Experts (MoE) models grow parameters without scaling compute proportionally?
MoE 把前馈层拆成多个“专家”子网络，每个词元只由门控网络选中的少数专家处理，因此总参数量很大，但每次推理激活的参数量很小，从而兼顾容量与效率。
MoE splits feed-forward layers into many expert sub-networks and, per token, a gating network activates only a few experts. Total parameters are huge, but per-inference active parameters stay small, balancing capacity and efficiency.
难度 进阶 · 适合 大众
CC BY 4.0 · 已核验 · Verified 2026-08-23 · 2 条独立证据
来源 · Source
Russell & Norvig,《Artificial Intelligence: A Modern Approach》第4版,Pearson,2020
S. Russell and P. Norvig, Artificial Intelligence: A Modern Approach, 4th ed., Pearson, 2020
独立核验依据 · Independent verification: 核验依据 2 ARCHIVE 标识 isbn_9788177583670：Artificial Intelligence: A
#MoE #稀疏
上下文窗口（context window）为什么是有限的？
Why is the context window limited?
上下文窗口是模型一次能“看到”的词元数量上限，受注意力计算随长度平方增长、显存中 KV Cache 占用以及训练时见过的最大长度所限。超出窗口的内容会被丢弃或需要分段处理。
The context window is the max number of tokens a model can see at once, bounded by quadratic attention cost, the KV cache memory, and the longest length seen in training. Anything beyond is dropped or must be chunked.
难度 进阶 · 适合 大众
CC BY 4.0 · 已核验 · Verified 2026-11-03
来源 · Source
A. Vaswani 等,《Attention Is All You Need》,NeurIPS 2017
A. Vaswani et al., Attention Is All You Need, NeurIPS 2017
独立核验依据 · Independent verification: 核验依据 1 ARCHIVE 标识 sseymens-attention-is-all-you-need：Attention
#上下文 #长度
BERT 这类模型为什么不按词或按字切分,而要采用 WordPiece 这样的子词方案?
What motivates subword schemes like WordPiece over word-level or character-level segmentation in models such as BERT?
按词建词表会让开放词汇无限膨胀,低频词与新造词一律沦为 UNK,形态丰富的语言受害最深;纯字符序列又太长且单字符几乎不带语义,训练与推理成本都高。子词切分把高频词保持完整、把低频词拆成可复用片段,例如 transformer 可拆为 transform 与 er,BERT 的三万词表因此几乎不产生 UNK。Sennrich 等在机器翻译中引入的 BPE 是这一思想的代表实现,WordPiece 本质上是按似然增益挑选合并的变体。
A word vocabulary balloons toward unbounded size, dumping rare words and neologisms into UNK—a penalty morphologically rich languages pay hardest—while pure character streams stretch sequence length and carry almost no meaning per token. Subwords keep frequent words intact and split rare ones into reusable pieces, so transformer factors into transform and er, letting the 30k BERT vocabulary cover its corpus with almost no UNK tokens. Byte-pair encoding, introduced for machine translation by Sennrich et al., is the canonical implementation, and WordPiece is essentially a variant that merges whichever pair maximizes likelihood gain.
难度 进阶 · 适合 初中(12-15)
CC BY 4.0 · 已核验 · Verified 2026-11-03
来源 · Source
Sennrich 等《Neural Machine Translation of Rare Words with Subword Units》,ACL 2016
R. Sennrich et al., "Neural Machine Translation of Rare Words with Subword Units", ACL 2016
独立核验依据 · Independent verification: 核验依据 1 ARCHIVE 标识 arxiv-1508.07909：Neural Machine Translation o
#文本切分 #子词模型
Transformer 的自注意力机制是如何让模型聚焦到句子中相关词语上的？
How does the self-attention mechanism in a Transformer let the model focus on the relevant words in a sentence?
每个词向量经三组可学习矩阵投影为查询、键、值三种向量。某词的查询向量与全句各词的键向量做点积，缩放并归一化后得到注意力权重，权重越大代表关联越强；再以这些权重对所有值向量加权求和，得到该位置的新表示。于是代词能从远处的名词聚合信息，且各位置同时计算，训练可以完全并行。
Learned matrices project each token vector into query, key, and value vectors. A token's query is dotted with every key in the sequence, scaled, and softmax-normalized into attention weights—the better the match, the larger the weight. The new representation at that position is the weighted sum of all value vectors under these weights. This lets a pronoun gather information from a distant noun, and since all positions compute at once, training parallelizes completely.
难度 进阶 · 适合 大众
CC BY 4.0 · 来源待核验 · Source pending review
来源 · Source
出处备注：著录含书名，但缺版次 / 出版社 / 年份中的一部分，需编辑补全后才能做独立比对。
Vaswani 等，《Attention Is All You Need》，Advances in Neural Information Processing Systems 30（NeurIPS 2017）。
A. Vaswani et al., "Attention Is All You Need," Advances in Neural Information Processing Systems 30 (NeurIPS 2017).
#自注意力 #Transformer #注意力权重
什么是大语言模型的“幻觉”？
What is a 'hallucination' in a large language model?
幻觉是指模型生成看似通顺却与事实不符或根本不存在的内容。它源于模型在预测下一个词时追求流畅，而非核对真实世界的事实。
A hallucination is when the model produces fluent text that is false or fabricated. It happens because the model predicts plausible next words rather than verifying real-world facts.
难度 入门 · 适合 大众
CC BY 4.0 · 已核验 · Verified 2026-08-23 · 2 条独立证据
来源 · Source
Russell & Norvig,《Artificial Intelligence: A Modern Approach》第4版,Pearson,2020
S. Russell and P. Norvig, Artificial Intelligence: A Modern Approach, 4th ed., Pearson, 2020
独立核验依据 · Independent verification: 核验依据 2 ARCHIVE 标识 isbn_9788177583670：Artificial Intelligence: A
#大模型 #幻觉
为什么大模型的推理要用 KV Cache 来加速？
Why does LLM inference use a KV cache to speed up?
生成时每输出一个新词元都要让它与之前所有词元做注意力计算。KV Cache 把已算好的历史 K、V 缓存起来，避免每步重复计算，使每步开销从与序列长度平方相关降到近似线性。
When generating, each new token attends to all previous ones. The KV cache stores the already-computed past keys and values so they are not recomputed, dropping per-step cost from roughly quadratic to near-linear in sequence length.
难度 入门 · 适合 大众
CC BY 4.0 · 已核验 · Verified 2026-08-23 · 2 条独立证据
来源 · Source
Russell & Norvig,《Artificial Intelligence: A Modern Approach》第4版,Pearson,2020
S. Russell and P. Norvig, Artificial Intelligence: A Modern Approach, 4th ed., Pearson, 2020
独立核验依据 · Independent verification: 核验依据 2 ARCHIVE 标识 isbn_9788177583670：Artificial Intelligence: A
#推理 #KV缓存
自注意力机制里查询（Q）、键（K）、值（V）各起什么作用？
What do query (Q), key (K) and value (V) do in self-attention?
每个词元生成 Q、K、V 三个向量：用某个词的 Q 与其余词的 K 做点积得到相似度，再softmax成权重，最后用这些权重对 V 加权求和，从而得到融合了上下文信息的表示。
Each token produces Q, K and V vectors: a token's Q is compared with others' K via dot product to get similarity, softmaxed into weights, then used to take a weighted sum of the V vectors, yielding a context-aware representation.
难度 入门 · 适合 大众
CC BY 4.0 · 已核验 · Verified 2026-08-23 · 2 条独立证据
来源 · Source
Russell & Norvig,《Artificial Intelligence: A Modern Approach》第4版,Pearson,2020
S. Russell and P. Norvig, Artificial Intelligence: A Modern Approach, 4th ed., Pearson, 2020
独立核验依据 · Independent verification: 核验依据 2 ARCHIVE 标识 isbn_9788177583670：Artificial Intelligence: A
#Transformer #注意力
什么是词元（token），为什么说 token 数量影响使用成本？
What is a token, and why does token count affect cost?
词元是模型处理文本时切分出的基本单位，可能是一个字、词或标点。模型按输入和输出的词元数计费与计算力，因此同样一段话，切得越碎、生成越长，花费就越高。
A token is the basic unit a model splits text into—a character, word piece or punctuation. Models bill and compute by the number of input and output tokens, so more (or longer) tokens cost more.
难度 入门 · 适合 大众
CC BY 4.0 · 已核验 · Verified 2026-08-23 · 2 条独立证据
来源 · Source
Russell & Norvig,《Artificial Intelligence: A Modern Approach》第4版,Pearson,2020
S. Russell and P. Norvig, Artificial Intelligence: A Modern Approach, 4th ed., Pearson, 2020
独立核验依据 · Independent verification: 核验依据 2 ARCHIVE 标识 isbn_9788177583670：Artificial Intelligence: A
#词元 #计费