Introduction to Dynamic Tanh
In a recent presentation at CVPR2025, researchers Kaiming He, Yann LeCun, and their team challenged a long-held assumption in deep learning: that normalization layers are indispensable. Their new approach introduces the Dynamic Tanh (DyT) function, an element-wise operation that can replace traditional normalization layers in Transformer models.
What are Normalization Layers?
Normalization layers — such as Batch Normalization, Layer Normalization, Instance Normalization, and Group Normalization — have been a staple in neural network design for over a decade. They are known for accelerating convergence, stabilizing training, and improving overall performance. In Transformer models, Layer Normalization (LN) is nearly ubiquitous. However, the team behind DyT observed that the input-output mapping of these normalization layers, particularly in deeper layers of Transformers, often follows an S-shaped curve reminiscent of the tanh function.
The Concept of Dynamic Tanh
This observation led to a bold idea: if the non-linear “squashing” effect of normalization is the key to its success, why not replicate it directly with a simpler, element-wise operation? DyT is designed to mimic the effect of normalization layers without computing statistics across tokens or batches. Instead, it applies a learnable scaling factor followed by a tanh-like nonlinearity to each element in the input.
How Dynamic Tanh Works
Here’s a simplified pseudocode to illustrate the concept:
# input x has the shape of [B, T, C]
# B: batch size, T: tokens, C: dimension
class DyT(Module):
def __init__(self, C, init_α):
super().__init__()
self.α = Parameter(ones(1) * init_α)
self.γ = Parameter(ones(C))
self.β = Parameter(zeros(C))
def forward(self, x):
x = tanh(self.alpha * x)
return self.γ * x + self.βIn this code, alpha is a learnable scalar parameter that allows the input to be scaled differently according to its range. The vector gamma is simply initialized as an all-ones vector, and beta as an all-zero vector. For the scaler parameter alpha, apart from LLM training, a default initialization of 0.5 is usually sufficient.
Benefits of Dynamic Tanh
DyT shouldn’t be considered a new type of normalization layer because it processes each input element individually during the forward pass without calculating any statistics or aggregating data. Instead, it still achieves the normalization effect by non-linearly squashing extreme values, while almost linearly transforming the input’s central range. This design eliminates the need to compute per-token or per-batch mean and variance, potentially streamlining both training and inference.
Experimental Results
The researchers conducted a wide range of experiments to compare DyT with traditional normalization layers across several tasks. The results show that DyT can achieve comparable or even better performance than traditional normalization layers in many cases.
Conclusion
In conclusion, the Dynamic Tanh function is a promising new approach to normalization in deep learning. By mimicking the effect of traditional normalization layers with a simpler, element-wise operation, DyT has the potential to streamline training and inference in Transformer models. While more research is needed to fully explore the capabilities of DyT, the initial results are exciting and suggest that this new approach may become a valuable tool in the field of deep learning.
