混合专家模型（MoE）如何在几乎不增加推理计算量的情况下扩展参数规模？
How do mixture-of-experts (MoE) models scale parameter count while keeping per-token inference compute roughly constant?
MoE 在前馈层中放置多个专家子网络，并用一个门控网络为每个词元只激活少数专家，从而在总参数量很大的情况下保持每词元的实际计算量较小。这以通信与负载均衡为代价换取参数效率，是稀疏激活扩展的关键范式。
MoE places multiple expert sub-networks in the feed-forward layer and uses a gating network to activate only a few experts per token, so total parameters can be huge while actual per-token compute stays small. It trades communication and load-balancing cost for parameter efficiency and is a key sparse-activation scaling paradigm.
难度 专业 · 适合 高中(15-18)
CC BY 4.0 · 来源待核验 · Source pending review
来源 · Source
出处备注：此条未指向带版次的可引用著作（通识性内容、机构文件或未展开的英文书名），本库不提供独立书目出处；如需精确引用，需由编辑补全具体篇目与版次。
Shazeer et al., Outrageously Large Neural Networks: The Sparsely-Gated MoE Layer, ICLR 2017
Shazeer et al., Outrageously Large Neural Networks: The Sparsely-Gated MoE Layer, ICLR 2017
#MoE #稀疏激活 #专家模型
什么是大语言模型的「叠加现象」（superposition），它给可解释性研究带来什么困难？
What is superposition in large language models, and what difficulty does it create for interpretability research?
叠加指模型用远少于特征数的神经维度以稀疏叠加方式表示大量特征，从而超越线性表示容量。它使单个神经元不再对应单一可解释概念，必须用稀疏自编码器等方法才能分离出单体特征，显著增加了机制可解释性的难度。
Superposition means the model represents far more features than it has neurons by sparsely combining them, exceeding linear representational capacity. It breaks the one-neuron-one-concept assumption, so researchers must use sparse autoencoders to recover monosemantic features, greatly complicating mechanistic interpretability.
难度 专业 · 适合 高中(15-18)
CC BY 4.0 · 来源待核验 · Source pending review
来源 · Source
出处备注：此条未指向带版次的可引用著作（通识性内容、机构文件或未展开的英文书名），本库不提供独立书目出处；如需精确引用，需由编辑补全具体篇目与版次。
Elhage et al., Toy Models of Superposition, Transformer Circuits Thread, 2022 (Anthropic)
Elhage et al., Toy Models of Superposition, Transformer Circuits Thread, 2022 (Anthropic)
#可解释性 #叠加 #稀疏特征
从概率建模和训练目标的角度，如何解释大语言模型产生「幻觉」（生成事实错误内容）的机理，有哪些主流缓解思路？
From the perspectives of probabilistic modeling and training objectives, how can we explain why large language models hallucinate (produce factually wrong content), and what are the main mitigations?
语言模型以自回归方式最大化下一个词的条件概率，目标是语言连贯而非事实真实，因此会生成似是而非但无事实支撑的内容。缓解方法包括检索增强生成（RAG）、在对齐时引入事实性奖励、以及推理时的自我一致性校验。这些方法各有取舍，目前仍无法彻底消除幻觉。
Language models maximize the conditional probability of the next token autoregressively, optimizing for linguistic coherence rather than factual truth, so they can emit plausible but unsupported content. Mitigations include retrieval-augmented generation (RAG), adding factuality rewards during alignment, and self-consistency checks at inference. Each trades off something, and hallucination is not yet fully solved.
难度 专业 · 适合 高中(15-18)
CC BY 4.0 · 来源待核验 · Source pending review
来源 · Source
出处备注：此条未指向带版次的可引用著作（通识性内容、机构文件或未展开的英文书名），本库不提供独立书目出处；如需精确引用，需由编辑补全具体篇目与版次。
arXiv（Huang 等, A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions, 2023）
arXiv (Huang et al., A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions, 2023)
#幻觉 #检索增强 #事实性
什么是「上下文学习」（in-context learning），它为什么不需要更新模型权重？
What is in-context learning, and why does it not require updating model weights?
上下文学习指大模型仅凭提示中给出的若干示例即可在新任务上表现，而不进行任何梯度更新。研究认为这是一种隐式的贝叶斯推断或算法模仿，模型在预训练阶段已习得在上下文中「遵循模式」的能力。
In-context learning means a large model performs a new task from a few examples given in the prompt, without any gradient updates. It is viewed as implicit Bayesian inference or algorithm imitation, a capability the model already acquired during pre-training.
难度 进阶 · 适合 大众
CC BY 4.0 · 已核验 · Verified 2026-11-03
来源 · Source
Brown et al., Language Models are Few-Shot Learners (GPT-3), NeurIPS 2020
Brown et al., Language Models are Few-Shot Learners (GPT-3), NeurIPS 2020
独立核验依据 · Independent verification: 核验依据 1 ARCHIVE 标识 language-models-are-few-shot-learners-gpt3：La
#上下文学习 #少样本 #提示
为什么 Transformer 需要位置编码，旋转位置编码（RoPE）相对原始正弦编码有何改进？
Why do Transformers need positional encoding, and how does Rotary Position Embedding (RoPE) improve over the original sinusoidal encoding?
自注意力本身对词序不敏感，位置编码用于注入序列顺序信息。RoPE 把位置信息以旋转矩阵形式融入查询和键，使相对位置可通过内积自然表达，在长上下文和长度外推上表现更优。
Self-attention is order-agnostic, so positional encoding injects sequence order. RoPE encodes position as a rotation applied to queries and keys, so relative position is expressed naturally through the dot product, yielding better behavior on long contexts and length extrapolation.
难度 进阶 · 适合 大众
CC BY 4.0 · 来源待核验 · Source pending review
来源 · Source
出处备注：此条未指向带版次的可引用著作（通识性内容、机构文件或未展开的英文书名），本库不提供独立书目出处；如需精确引用，需由编辑补全具体篇目与版次。
Su et al., RoFormer: Enhanced Transformer with Rotary Position Embedding, ACL 2021 (arXiv:2104.09864)
Su et al., RoFormer: Enhanced Transformer with Rotary Position Embedding, ACL 2021 (arXiv:2104.09864)
#位置编码 #RoPE #Transformer
Transformer 中的自注意力机制是如何计算输入序列中词元之间的依赖关系的？
How does self-attention in a Transformer compute dependencies between tokens in the input sequence?
自注意力将每个词元映射为查询、键、值三个向量，通过计算查询与所有键的点积并归一化得到注意力权重，再用权重对值向量加权求和。它让每个位置都能直接聚合序列中任意位置的信息，克服了 RNN 难以并行和长程依赖衰减的问题。
Self-attention maps each token to query, key, and value vectors, computes attention weights by normalizing the dot products of queries with all keys, then forms a weighted sum of the value vectors. This lets every position directly aggregate information from anywhere in the sequence, enabling parallelism and mitigating long-range dependency decay that RNNs suffer from.
难度 进阶 · 适合 大众
CC BY 4.0 · 已核验 · Verified 2026-11-03
来源 · Source
Vaswani et al., Attention Is All You Need, NeurIPS 2017 (arXiv:1706.03762)
Vaswani et al., Attention Is All You Need, NeurIPS 2017 (arXiv:1706.03762)
独立核验依据 · Independent verification: 核验依据 1 ARCHIVE 标识 sseymens-attention-is-all-you-need：Attention
#Transformer #自注意力 #注意力机制
大语言模型训练中的 RLHF（基于人类反馈的强化学习）起什么作用？
What role does RLHF (reinforcement learning from human feedback) play in training large language models?
RLHF 先用人类偏好数据训练一个奖励模型，再用强化学习（如 PPO）优化语言模型使其输出更符合人类期望。它能显著提升模型的有用性、安全性和指令遵循能力，是 ChatGPT 等对话模型对齐的关键步骤。
RLHF first trains a reward model on human preference data, then uses reinforcement learning such as PPO to optimize the language model toward outputs that better match human expectations. It markedly improves helpfulness, safety, and instruction following, and is a key step in aligning conversational models like ChatGPT.
难度 进阶 · 适合 初中(12-15)
CC BY 4.0 · 已核验 · Verified 2026-10-04
来源 · Source
Ouyang et al., Training language models to follow instructions with human feedback, NeurIPS 2022
Ouyang et al., Training language models to follow instructions with human feedback, NeurIPS 2022
独立核验依据 · Independent verification: 核验依据 1 DOI 10.52202/068431-2011
#RLHF #对齐 #强化学习
大语言模型的子词分词（如 BPE）解决了哪些问题？
What problems does subword tokenization such as BPE solve for large language models?
子词分词把词拆分为高频整体与低频片段，既控制了词表规模，又让模型能处理未登录词与多语言文本。BPE 等算法通过合并高频字符对逐步构建词表，在覆盖率和计算效率之间取得平衡。
Subword tokenization splits words into frequent wholes and rare fragments, keeping the vocabulary small while letting the model handle unseen words and multilingual text. Algorithms like BPE merge frequent character pairs to build the vocabulary, balancing coverage with computational efficiency.
难度 进阶 · 适合 初中(12-15)
CC BY 4.0 · 已核验 · Verified 2026-10-04
来源 · Source
Sennrich et al., Neural Machine Translation of Rare Words with Subword Units, ACL 2016
Sennrich et al., Neural Machine Translation of Rare Words with Subword Units, ACL 2016
独立核验依据 · Independent verification: 核验依据 1 DOI 10.18653/v1/p16-1162
#分词 #BPE #词表