吉布斯不等式 吉布斯不等式说明: 约西亚·吉布斯 若 ∑ i = 1 n p i = ∑ i = 1 n q i = 1 {\displaystyle \sum _{i=1}^{n}p_{i}=\sum _{i=1}^{n}q_{i}=1} ,且 p i , q i ∈ ( 0 , 1 ] {\displaystyle p_{i},q_{i}\in (0,1]} ,则有: − ∑ i = 1 n p i log p i ≤ − ∑ i = 1 n p i log q i {\displaystyle -\sum _{i=1}^{n}p_{i}\log p_{i}\leq -\sum _{i=1}^{n}p_{i}\log q_{i}} ,等号成立当且仅当 p i = q i ∀ i {\displaystyle p_{i}=q_{i}\forall i} 在信息论和概率论,它能应用在法诺不等式和讯号源编码定理的证明。 约西亚·吉布斯在19世纪提出它。 证明 吉布斯不等式等价于: 0 ≥ ∑ i = 1 n p i log q i − ∑ i = 1 n p i log p i = ∑ i = 1 n p i log ( q i / p i ) = − D K L ( P ‖ Q ) {\displaystyle 0\geq \sum _{i=1}^{n}p_{i}\log q_{i}-\sum _{i=1}^{n}p_{i}\log p_{i}=\sum _{i=1}^{n}p_{i}\log(q_{i}/p_{i})=-D_{\mathrm {KL} }(P\|Q)} (见相对熵)证明最右的项小于或等于0的方法有几种: 已知 ln ( x ) ≤ x − 1 {\displaystyle \ln(x)\leq x-1} ,等号成立当且仅当 x = 1 {\displaystyle x=1} : ∑ i = 1 n p i log ( q i / p i ) ≤ ∑ i = 1 n p i ( q i / p i − 1 ) = ∑ i = 1 n ( q i − p i ) = ∑ i = 1 n q i − ∑ i = 1 n p i = 0 {\displaystyle \sum _{i=1}^{n}p_{i}\log(q_{i}/p_{i})\leq \sum _{i=1}^{n}p_{i}(q_{i}/p_{i}-1)=\sum _{i=1}^{n}(q_{i}-p_{i})=\sum _{i=1}^{n}q_{i}-\sum _{i=1}^{n}p_{i}=0} 根据对数求和不等式或延森不等式: ∑ i p i log q i p i ≤ log ∑ i p i q i p i = log ∑ i q i ≤ 0 {\displaystyle \sum _{i}p_{i}\log {\frac {q_{i}}{p_{i}}}\leq \log \sum _{i}p_{i}{\frac {q_{i}}{p_{i}}}=\log \sum _{i}q_{i}\leq 0} 引理 对于n个变量的概率分布P,其熵的最大值是: H ( p 1 , … , p n ) ≤ log n {\displaystyle H(p_{1},\ldots ,p_{n})\leq \log n}