Mathematically, Focal Loss is defined as a modified binary cross-entropy:
$$ FL(p_t) = -\alpha_t (1 — p_t)^\gamma \log(p_t) $$
Let us break down the components of this formula:
$p_t$ (Modulated Probability):
For the positive class (object is present): $p_t = p$
For the negative class (no object): $p_t = 1 — p$
Here, $p$ is the model’s predicted probability for the correct class. If the model is confident in the correct answer, $p_t$ is close to 1. If the model makes a mistake, $p_t$ is close to 0.
Focusing parameter $\gamma$ (Gamma): This is the key element of the function. The modulating factor $(1 — p_t)^\gamma$ automatically down-weights the loss for well-classified examples.
If the model is confident ($p_t \to 1$), then $(1 — p_t)$ approaches 0. Raising this to the power of $\gamma$ makes this factor vanish.
If the model is not confident or is wrong ($p_t \to 0$), then $(1 — p_t)$ is close to 1, and the factor hardly affects the loss value.
A typical value in papers: $\gamma = 2$. This means that for an example with $p_t = 0.9$ (an easy example), the loss will be scaled down by a factor of $(1 — 0.9)^2 = 0.01$ (100 times). For a hard example with $p_t = 0.1$, the loss will remain almost unchanged.
Balancing parameter $\alpha_t$ (Alpha): This is a classic way to address class imbalance. $\alpha$ is a weighting factor that assigns more weight to the rare class.
$\alpha$ for the positive class (object) is usually set higher (e.g., 0.25), and for the negative class (background) — lower (e.g., 0.75).
In the simplest case, if the imbalance is not extreme, $\alpha$ can be set to 1.
How to Properly Apply Focal Loss
To ensure Focal Loss brings benefit rather than harm, several practical aspects must be considered:
When to use: Apply Focal Loss only when there is a significant class imbalance (e.g., a ratio of 1:100 or 1:1000) or when easy examples dominate the dataset. On balanced datasets (e.g., CIFAR-10), Focal Loss may perform worse than standard cross-entropy because it will excessively suppress the useful signal from easy examples.
Hyperparameter tuning:
$\gamma$ (Gamma): Start with $\gamma = 2$. If $\gamma = 0$, the function becomes weighted cross-entropy. Too high a value for $\gamma$ (e.g., 5+) can make training unstable, as the model will ignore the vast majority of examples.
$\alpha$ (Alpha): It can be set inversely proportional to the class frequencies in the dataset. If the positive class occurs 10 times less frequently, set $\alpha_{pos} \approx 0.9, \alpha_{neg} \approx 0.1$. $\alpha$ can also be tuned as a hyperparameter.
Architecture compatibility: Focal Loss works ideally in single-stage detector tasks (RetinaNet), instance segmentation, and keypoint detection. When transferring to other tasks (e.g., image classification), ensure that your model actually suffers from «easy-example domination.»
Types and Variations of Focal Loss
Since the appearance of the original version, several modifications have been proposed:
Binary Focal Loss: Described above, used for binary classification.
Categorical Focal Loss: Applied for multiclass classification using Softmax. It sums the binary losses across all classes, excluding cross-terms (analogous to Sparse Categorical Cross-Entropy).
Adaptive Focal Loss: A variation where the parameter $\gamma$ is not fixed but dynamically changes during training depending on the difficulty of the current batch.
Focal Modulation (HamburgerNet): A more modern concept where «focusing» is used not as a loss function, but as a feature modulation mechanism inside the neural network itself, replacing the attention mechanism (Self-Attention).
PolyLoss: A mathematically similar loss function that also modifies cross-entropy with a polynomial term to accelerate convergence, having proven effective in segmentation tasks.
Where Focal Loss is Applied
Object Detection: This is the «native» environment for Focal Loss. The RetinaNet architecture became popular precisely because of this loss function, allowing single-stage networks to surpass two-stage networks (e.g., Mask R-CNN) in accuracy on the COCO dataset.
Object Segmentation (Instance & Semantic Segmentation): When outlining contours of rare objects in medical scans (searching for tumors, pathologies) or satellite imagery (searching for rare building types).
Anomaly and Defect Detection: In industry, where the number of high-quality parts on a conveyor belt exceeds the number of defective items by hundreds of times.
Biometrics and Security: Recognizing rare faces in a stream or detecting fraudulent transactions.
Natural Language Processing (NLP): Text classification in tasks where one class occurs extremely rarely (e.g., detecting single instances of data leakage in a mass of corporate correspondence).
Focal Loss is not a universal replacement for cross-entropy, but a surgically precise tool for combating class imbalance. It changes the very philosophy of training: instead of forcing the model to learn everything indiscriminately, we force it to focus on what it does poorly.
Contact our experts to select the optimal hardware configuration and calculate rental pricing for your AI workloads.