An activation function is an integral part of a neural network architecture. It takes the input data, performs some mathematical calculations and contributes in the final results. There are different types of activation functions available like Sigmoid, ReLU, tanh, Softmax etc. They all have different properties and are suitable for different types of problems. Their choice impacts the neural network's performance in terms of accuracy and efficiency.
Each neuron in a neural network performs two tasks:
- Calculating linear sum of weighted inputs and bias
- Applying activation function to that linear sum
The main workings of an activation function are given below:
Introducing non-linearity
An activation function introduces non-linearity and helps neural network to capture non-linear complex patterns in the data. In the absence of an activation function, a neural network behaves like a linear regression model only and fails to capture non-linear relationships which limits its application to complex problems like NLP, Computer Vision etc.
Filtering important information
An activation function acts as a decision maker while passing information from one neuron to another. They decide whether a neuron should be activated or not based on the input it receives. This process makes sure that only useful information is transferred for further processing.
Providing support for estimating weights and biases
Backpropagation is a method to estimate weights and biases in a neural network. Gradient based methods are used to perform backpropagation. Activation functions, being differentiable by nature, plays an important role in adjusting and estimating these parameters during training process.
In the next post, we will discuss about different types of activation functions.
— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —
I enjoy understanding Machine Learning/Deep Learning algorithms at conceptual level and sharing my knowledge with simple explanations/diagrams. I believe having a clear understanding of the basics helps in building a strong foundation in these emerging areas. Stay tuned for more of such updates🙂