Conditional Relative Entropy

Conditional Relative Entropy

• 𝐷_𝑄(𝑃(𝑋|𝑌)) = 𝐄_𝑃[ 𝑙𝑔 [𝑃(𝑋|𝑌)/𝑄(𝑋|𝑌)] ]
• 𝐷_𝑄(𝑃(𝑋|𝑌)) = ∑_𝑦∊𝑌𝑃(𝑌=𝑦)·𝐷_𝑄(𝑃(𝑋|𝑌=𝑦))

Relative Entropy Chain Rule

let 𝑄 and 𝑃 be distribution over 𝑋₁, ..., 𝑋_𝑛

• 𝐷_𝑄(𝑃) = 𝐷_𝑄(𝑃(𝑋₁)) + 𝐷_𝑄(𝑃(𝑋₂|𝑋₁)) + 𝐷_𝑄(𝑃(𝑋₃|𝑋₂, 𝑋₁)) + ... + 𝐷_𝑄(𝑃(𝑋_𝑛|𝑋_𝑛-1, ..., 𝑋₁))

Properties

• 𝐷_𝑄(𝑃(𝑋)) ≤ 𝐷_𝑄(𝑃(𝑋,𝑌))
• 𝐷_𝑄(𝑃(𝑋₁, ..., 𝑋_𝑘)) ≤ 𝐷_𝑄(𝑃(𝑋₁, ..., 𝑋_𝑛)) for 𝑘 ≤ 𝑛