语言模型沿最高概率路径做贪心或束搜索解码时,生成的文本为何反而常常重复、平淡甚至前后矛盾？
When a language model is decoded greedily or by beam search along the highest-probability path, why is the resulting text often repetitive, bland, even self-contradictory?
因为最大化似然等于追逐概率分布的众数,而人类文本的分布是重尾的、天然带不可预测性,一路选最高概率往往落到最安全也最平庸的表达上。生成一旦绕回相似的前缀,相同的条件概率会再次选出相同的后缀,重复便自我强化成循环,束宽越大越明显。这证明似然高低并不等价于文本质量,需要引入采样的随机性或针对性的重复惩罚来打破确定性路径。
Maximizing likelihood amounts to chasing the mode of the distribution, whereas human text is heavy-tailed and inherently unpredictable, so always picking the top probability lands on the safest and blandest phrasing. Once generation loops back to a similar prefix, identical conditional probabilities select identical continuations again, and repetition self-reinforces into a cycle that wider beams worsen. This shows likelihood is not the same as quality: sampling randomness or targeted repetition penalties are needed to break the deterministic path.
难度 专业 · 适合 高中(15-18)
CC BY 4.0 · 来源待核验 · Source pending review
来源 · Source
出处备注：此条未指向带版次的可引用著作（通识性内容、机构文件或未展开的英文书名），本库不提供独立书目出处；如需精确引用，需由编辑补全具体篇目与版次。
A. Holtzman 等,The Curious Case of Neural Text Degeneration,ICLR 2020
A. Holtzman et al., The Curious Case of Neural Text Degeneration, ICLR 2020
#文本退化 #束搜索 #似然最大化
大语言模型仅凭提示词里两三个示例就能应对全新任务，权重一个都没更新，这种上下文学习能力究竟从何而来？
A large language model handles brand-new tasks given just a couple of examples in its prompt, with no weight updates at all — where does this in-context learning ability come from?
上下文学习并非运行中的参数更新，而是预训练养成的元学习能力：模型在前向传播内部隐式完成识别任务、归纳规律、迁移到新输入。预训练语料里问答对、清单、代码配注释这类自带结构的文本俯拾皆是，模型要压低损失就必须学会从上文推断当前文档隐含的任务规则。实验表明打乱示例顺序甚至替换真实标签对结果影响有限，真正起作用的是示例形式与任务分布，这与把上下文学习理解为隐式贝叶斯推断的解释相吻合。
In-context learning involves no runtime weight updates; it is meta-learning acquired during pretraining, with the model implicitly identifying the task, inducing its rule and applying it to new inputs inside the forward pass itself. Pretraining corpora teem with structured text such as question-answer pairs, lists and code with comments, so driving the loss down forces a model to infer whatever task rules the current document implies. Experiments show that shuffling demonstrations or even replacing their true labels barely hurts performance: format and task distribution do the heavy lifting, which fits viewing in-context learning as implicit Bayesian inference.
难度 专业 · 适合 小学高段(9-12)
CC BY 4.0 · 已核验 · Verified 2026-11-03
来源 · Source
T. Brown 等，Language Models are Few-Shot Learners，NeurIPS 2020；S. Min 等，Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?，EMNLP 2022
T. Brown et al., Language Models are Few-Shot Learners, NeurIPS 2020; S. Min et al., Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?, EMNLP 2022
独立核验依据 · Independent verification: 核验依据 1 ARCHIVE 标识 language-models-are-few-shot-learners-gpt3：La
#上下文学习 #元学习 #隐式贝叶斯推断
大语言模型的参数动辄上千亿，回答事实性问题时仍会一本正经地编造，检索增强生成先查外部文档再作答，为什么能明显缓解这种幻觉？
Large language models with hundreds of billions of parameters still confidently invent facts, so why does retrieval-augmented generation — querying external documents before answering — markedly curb such hallucination?
关键在于答题依据从参数记忆换成了当场检索到的真实文本：模型只需在看得见的证据上组织语言，而不必凭模糊记忆复述训练语料。流程上，问题先被编码成向量，从文档库中召回相关段落，连同原问题一起放进提示词；注意力机制让生成的每个词都能直接参照这些段落，事实细节有了明确出处，内部记忆的偏差会被眼前原文纠正，答案还可附带引用供人核查。知识过时时只需更换文档库而无需重训模型，幻觉因此大幅减少。
The trick is that answering grounds in freshly retrieved text rather than parameterized memory: the model only composes language over evidence it can literally see, instead of reciting training data from fuzzy recall. In practice the query is encoded as a vector, relevant passages are pulled from a document store, and everything goes into the prompt together; attention lets every generated token consult those passages, giving facts explicit provenance. Biased internal memories get overridden by the retrieved sources, answers can ship with citations for checking, and stale knowledge is fixed by swapping the corpus instead of retraining — hallucination drops sharply.
难度 进阶 · 适合 初中(12-15)
CC BY 4.0 · 来源待核验 · Source pending review
来源 · Source
出处备注：此条未指向带版次的可引用著作（通识性内容、机构文件或未展开的英文书名），本库不提供独立书目出处；如需精确引用，需由编辑补全具体篇目与版次。
P. Lewis 等，Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks，NeurIPS 2020；V. Karpukhin 等，Dense Passage Retrieval for Open-Domain Question Answering，EMNLP 2020
P. Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, NeurIPS 2020; V. Karpukhin et al., Dense Passage Retrieval for Open-Domain Question Answering, EMNLP 2020
#检索增强生成 #幻觉 #向量检索
评判语言模型时常提到困惑度这个指标,它究竟衡量什么？数值越低说明了什么？
Language models are often judged by a metric called perplexity. What does it actually measure, and what does a lower value indicate?
困惑度刻画模型在读到验证文本的每个位置时预测下一个词的平均犹豫程度,大致等于模型平均要在多少个等概率候选词里做选择。比如困惑度为20,可以粗略理解为模型每一步都像在20个词里猜。它由交叉熵损失指数化而来,数值越低说明模型对真实文本的拟合越好;但它只考核逐词命中,反映不了回答是否实用、事实是否正确。
Perplexity captures how uncertain a model is, on average, about the next word at each position of held-out text, roughly how many equally likely candidates it chooses among. A perplexity of 20 can be read loosely as the model guessing every next word from about twenty options. It is the exponentiated cross-entropy loss, so lower values mean a better fit to real text; however it only rewards word-by-word hits and cannot tell whether answers are useful or factually correct.
难度 入门 · 适合 大众
CC BY 4.0 · 已核验 · Verified 2026-10-04
来源 · Source
D. Jurafsky 与 J. H. Martin,《Speech and Language Processing》,第2版,Prentice Hall,2009年
D. Jurafsky and J. H. Martin, Speech and Language Processing, 2nd edition, Prentice Hall, 2009
独立核验依据 · Independent verification: 核验依据 1 ISBN 9780131873216 (2008)
#困惑度 #交叉熵 #语言模型评测