The Bit
Shannon formalized the concept of the bit (binary digit) as the basic unit of information — the answer to a yes/no question. He defined information mathematically: the information content of a message is the logarithm (base 2) of the inverse of its probability. Surprising messages carry more information than expected ones. This definition — elegant, abstract, and precise — became the foundation of all subsequent information theory.
Shannon Entropy
Shannon entropy H = -Σ p(x) log₂ p(x) measures the average information content of a source. For a fair coin: H = 1 bit. For a biased coin (70% heads): H ≈ 0.88 bits — less uncertainty, less information per flip. Entropy is also the theoretical minimum number of bits needed to represent a source without loss. It sets the limit for lossless compression.
Channel Capacity
Shannon's channel capacity theorem: every channel has a maximum information rate C (in bits per second), determined by its bandwidth and signal-to-noise ratio (C = B log₂(1 + S/N), the Shannon-Hartley theorem). Below capacity, communication can be made arbitrarily reliable. Above capacity, errors are inevitable. This theorem was startling — engineers had assumed noisy channels limited reliability in proportion to noise. Shannon proved you can transmit reliably up to capacity regardless of noise level, with the right encoding.
The Eccentric Genius
Shannon juggled and unicycled through Bell Labs corridors, built a flame-throwing trumpet, invented the first wearable computer (for roulette), and designed a maze-solving mechanical mouse called Theseus. He retired early from MIT, became obsessively wealthy through stock options, and spent his final decades playing with gadgets. He died of Alzheimer's in 2001, largely unknown to the public whose digital world rested entirely on his mathematics.
Frequently Asked Questions
What did Claude Shannon invent?
Shannon invented information theory in 1948, defining the bit as the unit of information, proving the Shannon channel capacity theorem (maximum reliable information rate over a noisy channel), establishing the mathematical limits of data compression, and founding the field that underpins all digital communication and computing.
What is Shannon entropy?
Shannon entropy H = -Σ p(x) log₂ p(x) measures the average information content (uncertainty) of a source. Higher entropy means more uncertainty and more information per symbol. It sets the theoretical minimum for lossless data compression — you cannot compress a source below its entropy without losing information.
What is the Shannon limit?
The Shannon limit (or Shannon capacity) C = B log₂(1 + S/N) is the maximum rate at which information can be transmitted reliably over a channel with bandwidth B and signal-to-noise ratio S/N. It is a hard theoretical ceiling — no encoding scheme can reliably exceed it. Modern communications engineering is largely the pursuit of getting close to this limit.
