What Is Shannon Entropy?
Shannon entropy is H = −Σ pᵢ log pᵢ. It is zero when one outcome is certain and largest when all outcomes are equally likely. The unit depends on the logarithm base: bits for base 2, nats for base e and hartleys for base 10.
The calculator shows the maximum possible entropy for the number of outcomes, the normalised entropy H ÷ Hmax, the min-entropy, the Gini impurity and the perplexity. Reference points such as a fair coin (1 bit) and a certain outcome (0 bits) help you judge the value.
Below you will find the formulas, a variables table, a worked example and answers to common questions such as entropy in decision trees and how to convert between units.
How to Use the Shannon Entropy Calculator
- Choose whether you are entering probabilities or raw counts.
- Enter one value per outcome. Counts are converted to probabilities automatically.
- Pick the base: 2 for bits, e for nats.
- Compare your entropy with the maximum to see how close the distribution is to uniform.
Shannon Entropy Formula
Shannon entropy
Maximum entropy
Gini impurity
Perplexity
Shannon Entropy Variables Explained
| Symbol | Meaning | Notes |
|---|---|---|
| pᵢ | Probability of outcome i | Must sum to 1 |
| H | Shannon entropy | −Σ pᵢ log pᵢ |
| Hmax | Maximum entropy | log of the number of outcomes |
| Gini | Gini impurity | 1 − Σ pᵢ² |
| Perplexity | Effective number of outcomes | base^H |
Shannon Entropy Worked Example
With the example values loaded in the calculator above, the steps are:
- Normalise to probabilities p = value ÷ 14.
- Entropy H = −Σ p·log₂(p), skipping any p = 0.
- H = 0.940286 bits; the maximum for 2 outcomes is 1 bits.
Result: Shannon entropy H (bits) = 0.940286; Maximum possible entropy (log 2) = 1.
How to Read Shannon Entropy
Entropy measures how uncertain an outcome is. A fair coin has two equally likely outcomes and entropy of exactly 1 bit. A coin that lands heads 90% of the time has less uncertainty, so its entropy is lower, about 0.47 bits.
Compare the result with the maximum for the same number of outcomes, which is log₂ of the number of outcomes. The ratio H ÷ Hmax tells you how close the distribution is to uniform. Zero entropy means one outcome is certain.
How to interpret entropy
Entropy is the average surprise you feel when you learn an outcome. Rare outcomes carry high surprise (−log p is large) but occur rarely; entropy weights surprise by probability. It reaches its maximum when all outcomes are equally likely and drops to zero when one outcome is certain.
| Distribution | Entropy (bits) |
|---|---|
| Fair coin (½, ½) | 1.000 |
| Biased coin (0.9, 0.1) | 0.469 |
| Fair die (6 × 1/6) | 2.585 |
| Four outcomes (½, ¼, ⅛, ⅛) | 1.750 |
| Certain outcome (1) | 0.000 |
Entropy in decision trees and machine learning
Decision tree algorithms such as ID3 and C4.5 choose splits that reduce entropy the most. Information gain is the entropy before a split minus the weighted entropy of the child nodes. CART uses Gini impurity, a closely related measure that is cheaper to compute. Cross-entropy, used as the loss in classification, is the entropy of the true distribution measured with the model’s probabilities.
Bits, nats and hartleys
The only difference is the logarithm base. Base 2 gives bits, natural logs give nats, and base 10 gives hartleys. Convert with 1 nat = 1.4427 bits and 1 hartley = 3.3219 bits.
Where Shannon Entropy Is Used
Information content
Measure the uncertainty in a set of probabilities or counts.
Comparing distributions
Use normalised entropy to compare distributions with different numbers of outcomes.
Decision trees
See Gini impurity next to entropy for a split.
Choosing a unit
Report in bits, nats or another base.
Common Shannon Entropy Mistakes to Avoid
- Probabilities must add to 1. Counts are converted to probabilities for you.
- Outcomes with zero probability contribute nothing to entropy.
- Bits use log base 2 and nats use the natural log.
- Entropy depends only on the probabilities, not on the labels of the outcomes.
Shannon Entropy Terms and Definitions
- Information gain
- Reduction in entropy achieved by splitting a data set on an attribute.
- Perplexity
- The effective number of equally likely outcomes, b^H.
- Min-entropy
- Entropy based only on the most likely outcome; a conservative measure used in security.
Shannon Entropy FAQs
What is Shannon entropy?
A measure of the average uncertainty in a probability distribution, calculated as −Σ p log p.
What is the entropy of a fair coin?
One bit. Two equally likely outcomes need one yes/no question to resolve.
Can entropy be negative?
For discrete distributions no; entropy is zero or positive. Differential entropy of continuous variables can be negative.
What is the difference between entropy and Gini impurity?
Both measure how mixed a distribution is. Entropy uses logarithms; Gini uses squared probabilities. They usually lead to similar decision trees.
How do I calculate entropy from counts?
Divide each count by the total to get probabilities, then compute −Σ p log₂ p.
Why does maximum entropy occur for uniform distributions?
When all outcomes are equally likely, you have no information favouring any of them, so uncertainty is at its highest.