香农熵的公式 H=-Σ p_i log p_i 是怎么来的，为什么它能度量平均不确定性？
Where does Shannon's entropy formula H=-Σ p_i log p_i come from, and why does it measure average uncertainty?
直觉要求：概率越小的事件信息越多，独立试验的信息量应相加。取概率的对数作单事件信息量恰满足可加性，加负号使总量非负，按概率加权平均即得 H=-Σ p_i log p_i。香农进一步证明：任何满足连续性、对均匀分布的单调性与可加性的合理度量必取此形式；对数以 2 为底时单位是比特，这奠定了数据压缩极限的理论基础。
Intuition demands that rarer events carry more information and that independent experiments contribute additively. Taking the logarithm of a probability as an event's information is precisely the additive choice; a minus sign keeps the total nonnegative, and averaging under the distribution gives H=-Σ p_i log p_i. Shannon proved more: any reasonable measure obeying continuity, monotonicity for uniform distributions, and additivity must take this form. With base-2 logs the unit is the bit, grounding the theoretical limits of data compression.
难度 进阶 · 适合 初中(12-15)
CC BY 4.0 · 已核验 · Verified 2026-11-04
来源 · Source
Shannon,《A Mathematical Theory of Communication》,Bell System Technical Journal,1948 (DOI: 10.1002/j.1538-7305.1948.tb01338.x)
Shannon, 'A Mathematical Theory of Communication', Bell System Technical Journal, 1948, DOI: 10.1002/j.1538-7305.1948.tb01338.x
独立核验依据 · Independent verification: 核验依据 1 Crossref DOI 10.1002/j.1538-7305.1948.tb01338.x
#信息论 #香农熵 #比特
电话线路上总有噪声，为什么香农断言：只要传输速率低于信道容量 C，就能做到几乎无差错的通信？
A telephone line always carries noise, so why did Shannon assert that as long as the transmission rate stays below the channel capacity C, communication can be made almost error-free?
诀窍是主动加入冗余。发送端把长串消息编成大码块；噪声虽会翻转部分比特，但只要速率低于容量 C 且码块足够长，码字之间就被安排得足够远，少量翻转无法让一个码字冒充另一个，接收端照样认出原块——这就是有噪信道编码定理。反之速率超过 C，则显著差错不可避免。该定理给出可靠通信的速度上限及其达成路径，是数字通信的基石。
The trick is deliberate redundancy. The sender packs long message strings into large code blocks before they enter the channel; noise flips some bits, yet as long as the rate stays below the capacity C and blocks are long enough, codewords are spaced so far apart that a few flips cannot make one block impersonate another, and the receiver still recognizes the original — this is the noisy-channel coding theorem. Above C, by contrast, no encoding escapes a significant error rate. The theorem sets the speed limit of reliable communication and shows how to reach it.
难度 进阶 · 适合 大众
CC BY 4.0 · 已核验 · Verified 2026-10-04 · 2 条独立证据
来源 · Source
T. M. Cover 与 J. A. Thomas《Elements of Information Theory》（第2版），Wiley，2006，第7章
T. M. Cover & J. A. Thomas, Elements of Information Theory, 2nd ed., Wiley, 2006, Chapter 7
独立核验依据 · Independent verification: 核验依据 1 ISBN 9780471241959 (2006) · 核验依据 2 ISBN 9780665590955 (2002)
#信息论 #信道容量 #信道编码
对一个有 n 种可能结果的实验，为什么所有结果等概率时香农熵达到最大？这一事实如何支撑“熵是不确定性的度量”这一解释？
For an experiment with n possible outcomes, why does Shannon entropy peak when all outcomes are equally likely, and how does this support reading entropy as a measure of uncertainty?
把熵写成 H = E[log₂(1/p_i)]，即平均惊喜度。对数为凹，Jensen 不等式给出 H ≤ log₂ Σ p_i(1/p_i) = log₂ n，等号当且仅当每个 p_i = 1/n。分布越均匀越难猜中；偏爱某些结果时其余几乎注定，熵随之下降。上界 log₂ n 比特是无法区分 n 个结局的最坏不确定度。
Write the entropy as H = E[log₂(1/p_i)], the average surprise of an outcome. The logarithm is concave, so Jensen's inequality gives H ≤ log₂ Σ p_i(1/p_i) = log₂ n, with equality exactly when 1/p_i is constant, i.e., every p_i = 1/n. Intuition agrees: the more uniform the distribution, the harder the outcome is to guess, while favoring some outcomes makes others nearly certain and lowers uncertainty. The ceiling log₂ n bits marks maximal doubt when all n outcomes are equally indistinguishable.
难度 进阶 · 适合 大众
CC BY 4.0 · 已核验 · Verified 2026-10-04 · 2 条独立证据
来源 · Source
T. M. Cover 与 J. A. Thomas《Elements of Information Theory》（第2版），Wiley，2006，第2章
T. M. Cover & J. A. Thomas, Elements of Information Theory, 2nd ed., Wiley, 2006, Chapter 2
独立核验依据 · Independent verification: 核验依据 1 ISBN 9780471241959 (2006) · 核验依据 2 ISBN 9780665590955 (2002)
#信息论 #香农熵 #詹森不等式 #均匀分布
信息量为什么偏偏用比特来计量？而且一条消息越是出人意料，它携带的信息量为何反而越大？
Why is information measured in bits, and why does the more surprising a message strikes us, the more information it turns out to carry?
因为信息量度量的正是不确定性的消除：概率为p的事件，其自信息定义为−log₂p，概率越小、发生越意外，信息量就越大；掷均匀硬币这类二选一恰好对应1比特，单位由此得名。对整个信源取平均便得到香农熵H=−Σp log₂p，它刻画平均意外程度，而信源编码定理进一步证明：熵就是无损压缩长度的下限，数据越接近均匀随机，越难再压短。
Because information quantifies the removal of uncertainty: an event with probability p carries self-information −log₂p, so the smaller the probability, the greater the surprise and the more information the occurrence delivers. A fair coin flip, the archetypal binary choice, maps to exactly one bit, whence the unit takes its name. Averaging over the source gives Shannon's entropy H = −Σp log₂p, the mean surprise per message, and the source coding theorem adds that entropy is the lower bound of lossless compression: the closer data approach pure randomness, the harder they are to shrink.
难度 入门 · 适合 大众
CC BY 4.0 · 来源待核验 · Source pending review
来源 · Source
出处备注：著录含书名，但缺版次 / 出版社 / 年份中的一部分，需编辑补全后才能做独立比对。
香农,《A Mathematical Theory of Communication》,Bell System Technical Journal 第27卷,1948
C. E. Shannon, A Mathematical Theory of Communication, Bell System Technical Journal 27, 1948
#香农熵 #自信息 #信源编码定理