Information is surprise. Shannon defined it precisely: the information content of an event is -log(p), where p is its probability.
Three consequences:
- Certain events carry zero information (you already knew).
- Rare events carry a lot (a coin landing on its edge tells you something).
- The average information of a distribution is its entropy.
This frames data compression: encode common symbols in few bits, rare symbols in many. Huffman coding falls out of this directly.