Focal Loss - term AI

Mathematically, Focal Loss is defined as a modified binary cross-entropy:

$$ FL(p_t) = -\alpha_t (1 — p_t)^\gamma \log(p_t) $$

Let us break down the components of this formula:

$p_t$ (Modulated Probability):

For the positive class (object is present): $p_t = p$
For the negative class (no object): $p_t = 1 — p$
Here, $p$ is the model’s predicted probability for the correct class. If the model is confident in the correct answer, $p_t$ is close to 1. If the model makes a mistake, $p_t$ is close to 0.

Focusing parameter $\gamma$ (Gamma): This is the key element of the function. The modulating factor $(1 — p_t)^\gamma$ automatically down-weights the loss for well-classified examples.

If the model is confident ($p_t \to 1$), then $(1 — p_t)$ approaches 0. Raising this to the power of $\gamma$ makes this factor vanish.
If the model is not confident or is wrong ($p_t \to 0$), then $(1 — p_t)$ is close to 1, and the factor hardly affects the loss value.
A typical value in papers: $\gamma = 2$. This means that for an example with $p_t = 0.9$ (an easy example), the loss will be scaled down by a factor of $(1 — 0.9)^2 = 0.01$ (100 times). For a hard example with $p_t = 0.1$, the loss will remain almost unchanged.

Balancing parameter $\alpha_t$ (Alpha): This is a classic way to address class imbalance. $\alpha$ is a weighting factor that assigns more weight to the rare class.

$\alpha$ for the positive class (object) is usually set higher (e.g., 0.25), and for the negative class (background) — lower (e.g., 0.75).
In the simplest case, if the imbalance is not extreme, $\alpha$ can be set to 1.
How to Properly Apply Focal Loss

To ensure Focal Loss brings benefit rather than harm, several practical aspects must be considered:

When to use: Apply Focal Loss only when there is a significant class imbalance (e.g., a ratio of 1:100 or 1:1000) or when easy examples dominate the dataset. On balanced datasets (e.g., CIFAR-10), Focal Loss may perform worse than standard cross-entropy because it will excessively suppress the useful signal from easy examples.
Hyperparameter tuning:
$\gamma$ (Gamma): Start with $\gamma = 2$. If $\gamma = 0$, the function becomes weighted cross-entropy. Too high a value for $\gamma$ (e.g., 5+) can make training unstable, as the model will ignore the vast majority of examples.
$\alpha$ (Alpha): It can be set inversely proportional to the class frequencies in the dataset. If the positive class occurs 10 times less frequently, set $\alpha_{pos} \approx 0.9, \alpha_{neg} \approx 0.1$. $\alpha$ can also be tuned as a hyperparameter.
Architecture compatibility: Focal Loss works ideally in single-stage detector tasks (RetinaNet), instance segmentation, and keypoint detection. When transferring to other tasks (e.g., image classification), ensure that your model actually suffers from «easy-example domination.»
Types and Variations of Focal Loss

Since the appearance of the original version, several modifications have been proposed:

Binary Focal Loss: Described above, used for binary classification.
Categorical Focal Loss: Applied for multiclass classification using Softmax. It sums the binary losses across all classes, excluding cross-terms (analogous to Sparse Categorical Cross-Entropy).
Adaptive Focal Loss: A variation where the parameter $\gamma$ is not fixed but dynamically changes during training depending on the difficulty of the current batch.
Focal Modulation (HamburgerNet): A more modern concept where «focusing» is used not as a loss function, but as a feature modulation mechanism inside the neural network itself, replacing the attention mechanism (Self-Attention).
PolyLoss: A mathematically similar loss function that also modifies cross-entropy with a polynomial term to accelerate convergence, having proven effective in segmentation tasks.
Where Focal Loss is Applied
Object Detection: This is the «native» environment for Focal Loss. The RetinaNet architecture became popular precisely because of this loss function, allowing single-stage networks to surpass two-stage networks (e.g., Mask R-CNN) in accuracy on the COCO dataset.
Object Segmentation (Instance & Semantic Segmentation): When outlining contours of rare objects in medical scans (searching for tumors, pathologies) or satellite imagery (searching for rare building types).
Anomaly and Defect Detection: In industry, where the number of high-quality parts on a conveyor belt exceeds the number of defective items by hundreds of times.
Biometrics and Security: Recognizing rare faces in a stream or detecting fraudulent transactions.
Natural Language Processing (NLP): Text classification in tasks where one class occurs extremely rarely (e.g., detecting single instances of data leakage in a mass of corporate correspondence).

Focal Loss is not a universal replacement for cross-entropy, but a surgically precise tool for combating class imbalance. It changes the very philosophy of training: instead of forcing the model to learn everything indiscriminately, we force it to focus on what it does poorly.

Let's start a conversation

Start Scaling Today

Contact our experts to select the optimal hardware configuration and calculate rental pricing for your AI workloads.

Location: Moscow, ext.ter.g., sh. Ochakovskoe, 28, p.2
GPU Servers High-density compute
Облачная аренда Быстрый старт ИИ моделей
Enterprise SLA Guaranteed availability
Low Latency Optimal network routes