GeGLU
public
–
2 min read
GeGLU is a powerful activation function that enhances deep learning models in neural networks. It is a variant of the…
Rational Activation Function
public
–
2 min read
Rational Activation Function: An Introduction
Activation functions are an integral part of a deep neural network. They define how the…
SwiGLU
public
–
2 min read
What is SwiGLU?
SwiGLU is an activation function used in deep neural networks that is a variant of GLU (Gated…
Stochastically Scaling Features and Gradients Regularization
public
–
2 min read
What is SSFG Regularization?
SSFG regularization is a method of data analysis that is used to solve a common problem…
StarReLU
public
–
2 min read
StarReLU: An Overview
The Rectified Linear Unit (ReLU) function is a common activation function used in deep learning models. It…
DELU
public
–
3 min read
The DELU activation function is a type of activation function that uses trainable parameters and employs the complex linear and…
Shifted Rectified Linear Unit
public
–
3 min read
Understand ShiLU: A Modified ReLU Activation Function with Trainable Parameters
If you're familiar with machine learning or deep learning, you…
Smish
public
–
2 min read
The World of Smish and Deep Learning Methods
Smish is a relatively new activation function that has been introduced to…
Linear Combination of Activations
public
–
2 min read
What is LinComb?
LinComb, short for Linear Combination of Activations, is a type of activation function commonly used in machine…
Squared ReLU
public
–
2 min read
The Squared ReLU activation function is a nonlinear mathematical function used in the Primer architecture within the Transformer layer. It…
Serf
public
–
2 min read
Serf: Understanding Log-Softplus ERror Activation Function
When it comes to artificial neural networks and their deep learning algorithms, activation functions…
Tanh Exponential Activation Function
public
–
1 min read
When it comes to real-time computer vision tasks, lightweight neural networks are often used because they have fewer parameters than…
Normalized Linear Combination of Activations
public
–
2 min read
The Normalized Linear Combination of Activations, also known as NormLinComb, is a type of activation function commonly used in machine…
Collapsing Linear Unit
public
–
2 min read
CoLU is a cleverly crafted activation function that has numerous unique properties favorable to the performance of deeper neural networks.…
Cosine Linear Unit
public
–
2 min read
What is CosLU?
CosLU, short for Cosine Linear Unit, is an activation function used in Artificial Neural Networks. It uses…
Hermite Polynomial Activation
public
–
2 min read
The Hermite Activations are a type of activation function used in artificial neural networks. They differ from the widely used…
Hardtanh Activation
public
–
2 min read
A Hardtanh activation function is a mathematical formula that is used in artificial neural networks. It is an updated version…
Padé Activation Units
public
–
3 min read
Parametrized learnable activation function, based on the Padé approximant, or PAU, is a type of activation function used in machine…
EvoNorms
public
–
2 min read
EvoNorms are a new type of computation layer used in designing neural networks. Neural networks are a type of artificial…
Rectified Linear Unit N
public
–
3 min read
Understanding ReLUN: A Modified Activation Function
When it comes to training neural networks, the activation function is an essential component.…
ReGLU
public
–
1 min read
In the world of machine learning, one important mathematical concept is activation functions. Activation functions are used to transform a…
Gumbel Cross Entropy
public
–
1 min read
The Gumbel activation function is a mathematical formula used for transforming the unnormalized output of a model to probability. This…
Growing Cosine Unit
public
–
3 min read
Overview of GCU
If you're interested in artificial intelligence and machine learning, you've probably heard of the GCU. It stands…
Phish: A Novel Hyper-Optimizable Activation Function
public
–
2 min read
Phish: A Novel Activation Function That Could Revolutionize Deep-Learning Models
Deep-learning models have become an essential part of modern technology.…
Sinusoidal Representation Network
public
–
3 min read
What is Siren?
Siren, also known as Sinusoidal Representation Network, is a new type of periodic activation function used for…
Lecun's Tanh
public
–
2 min read
Understanding LeCun's Tanh Activation Function
In the field of artificial neural networks, an activation function is an important component of…
nlogistic-sigmoid function
public
–
2 min read
The Nlogistic-sigmoid function (NLSIG) is a mathematical equation used to model growth or decay processes. The function uses two metrics,…
ScaledSoftSign
public
–
2 min read
Overview of ScaledSoftSign
The ScaledSoftSign is an alteration of the SoftSign activation function that can be trained with parameters. The…
Hard Swish
public
–
2 min read
Hard Swish is a type of activation function that is based on a concept called Swish. Swish is a mathematical…
Exponential Linear Squashing Activation
public
–
2 min read
The Exponential Linear Squashing Activation Function, or ELiSH, is a type of activation function commonly used in neural networks. It…
HardELiSH
public
–
2 min read
HardELiSH is a mathematical equation used as an activation function for neural networks. This particular equation is a combination of…
SERLU
public
–
2 min read
Introduction to SERLU Activation Function
As technology continues to evolve, the need for faster, more efficient computing grows. One area…
Randomized Leaky Rectified Linear Units
public
–
2 min read
In the world of machine learning, there is a concept called activation functions. These functions help to determine the output…
S-shaped ReLU
public
–
3 min read
The S-shaped Rectified Linear Unit, or SReLU, is an activation function used in neural networks. This function can learn both…
Hard Sigmoid
public
–
2 min read
Neural networks are used for a wide range of applications, including image and speech recognition, predictive modeling, and more. One…
modReLU
public
–
2 min read
ModReLU is a type of activation function, used in machine learning and artificial neural networks, that modifies the Rectified Linear…
CReLU
public
–
2 min read
Introduction to CReLU
CReLU, or Concatenated Rectified Linear Units, is an activation function used in deep learning. It involves concatenating…
Gaussian Error Linear Units
public
–
2 min read
The Gaussian Error Linear Unit, or GELU, is an activation function that is commonly used in artificial neural networks. It…
Sigmoid Linear Unit
public
–
2 min read
SiLU, short for Sigmoid Linear Units, is an activation function used in neural networks to help improve their accuracy and…
Mish
public
–
2 min read
When it comes to neural networks, activation functions are a fundamental component. They are responsible for determining whether a neuron…
Kernel Activation Function
public
–
3 min read
If you've ever heard of the term "Kernel Activation Function" or KAF, you might be wondering what it is and…
Parametric Exponential Linear Unit
public
–
2 min read
Parameterized Exponential Linear Units, also known as PELU, is an activation function that is commonly used in neural networks. It…
Softsign Activation
public
–
2 min read
The Softsign Activation function is one of the many activation functions that researchers have developed for use in neural networks.…
ReLU6
public
–
2 min read
ReLU6: A Modified Version of Rectified Linear Unit
Machine learning algorithms are rapidly changing the computational landscape of artificial intelligence.…
Parameterized ReLU
public
–
2 min read
Parametric Rectified Linear Unit, commonly known as PReLU, is an activation function that enhances the traditional rectified unit with a…
Scaled Exponential Linear Unit
public
–
2 min read
SELU Overview: The Self-Normalizing Activation Function
If you've ever heard of neural networks, you might have come across the term…
Exponential Linear Unit
public
–
1 min read
In machine learning, an activation function is applied to the output of each neuron in a neural network. The exponential…
Maxout
public
–
1 min read
The Maxout Unit is a mathematical function used in deep learning. It is a generalization of the ReLU and the…
Swish
public
–
2 min read
Swish is an activation function used in machine learning that was introduced in 2017. It is comprised of a simple…
Adaptive Richard's Curve Weighted Activation
public
–
2 min read
Deep Neural Networks (DNNs) are ubiquitous in modern machine learning tasks like image and speech recognition. They take in input…