What Is Cross-Entropy Loss?
Cross-entropy loss is the average negative log of the probability the model gave to the true outcome. It punishes confident wrong predictions heavily, which is why classifiers are trained with it.
For binary labels, enter the 0 or 1 labels and the predicted probabilities. For several classes, enter the true distribution and the predicted distribution. You get the loss in nats and in bits.
In the multiclass case the calculator also shows the entropy of the true distribution and the KL divergence, since cross-entropy equals the two added together.
How to Use the Cross-Entropy Loss Calculator
- Choose binary or multiclass.
- Enter the true labels or distribution and the predicted probabilities in the same order.
- Read the loss in nats and bits.
Cross-Entropy Loss Formula
Binary
Multiclass
Cross-Entropy Loss Variables Explained
| Symbol | Meaning | Notes |
|---|---|---|
| y or q | True label or true probability | |
| p | Predicted probability | |
| L | Cross-entropy loss | −Σ q ln p |
| KL | Kullback-Leibler divergence | H(q, p) − H(q) |
Cross-Entropy Loss Worked Example
With the example values loaded in the calculator above, the steps are:
- For each example, loss = −[y·ln(p) + (1 − y)·ln(1 − p)].
- Average over 5 examples: 0.260273 nats, or 0.375495 bits.
Result: Mean binary cross-entropy (nats) = 0.260273; In bits (log base 2) = 0.375495.
Worked example
Take labels 1, 0, 1, 1, 0 with predictions 0.9, 0.2, 0.7, 0.6, 0.1. The per-example losses are 0.1054, 0.2231, 0.3567, 0.5108 and 0.1054.
Their mean is 0.2603 nats, or 0.3755 bits.
Link to entropy and KL divergence
Cross-entropy equals the entropy of the true distribution plus the KL divergence from the true to the predicted distribution. Minimising cross-entropy therefore pushes the prediction toward the truth.
Where Cross-Entropy Loss Is Used
Machine learning
Check the log loss of a classifier by hand.
Comparing models
Compare probability forecasts with a proper scoring rule.
Information theory coursework
Relate cross-entropy, entropy and KL divergence.
Common Cross-Entropy Loss Mistakes to Avoid
- Check which log base your library uses.
- Avoid predictions of exactly 0 or 1 in training.
Cross-Entropy Loss Terms and Definitions
- Log loss
- Another name for binary cross-entropy.
- Nat
- The unit of information when using natural logarithms.
Cross-Entropy Loss FAQs
What is cross-entropy loss?
The average negative log probability the model gave to the true outcomes.
What is a good cross-entropy value?
Lower is better. For binary balanced data, predicting 0.5 always gives 0.693 nats, so a useful model should beat that.
What is the difference between cross-entropy and KL divergence?
Cross-entropy equals entropy plus KL divergence. For one-hot labels the entropy is zero, so they match.
Why does it go to infinity?
If the model gives probability 0 to something that happens, the log of 0 is minus infinity.