Search tools and guides ⌘K
I · Information theory

Shannon entropy

What it is

H(X) = −Σ p(x) log₂ p(x). The expected number of bits needed to encode a sample from X.

Where it lives

Compression bounds, password strength, ML loss functions, Huffman coding.

The key insight

Higher entropy = more uncertainty = more bits to encode. A fair coin: 1 bit. A biased coin (90/10): ~0.47 bits.

More in Information theory
Mutual information I(X;Y) = H(X) − H(X∣Y). How much knowing Y reduces uncertainty about X… Channel capacity (Shannon-Hartley) C = B · log₂(1 + S/N). Bits per second for a noisy channel of bandwidt… KL divergence D_KL(P‖Q) = Σ p(x) log(p(x)/q(x)). The "extra bits" cost of using Q to…
Across the foundations
II Birthday paradox Probability & randomness II Poisson process Probability & randomness II Concentration inequalities Probability & randomness II Bayes theorem Probability & randomness III Shortest paths Graph theory III Min-cut / Max-flow Graph theory III Spanning trees Graph theory III Connectivity & cycles Graph theory
← All foundations