Difference between Gradient descent and Normal equation

Gradient Descent is an optimization algorithm used to find the values of parameters of a function that minimizes a cost function. It is an iterative algorithm. We use gradient descent to update the parameters of the model. Parameters refer to coefficients in Linear Regression and weights in neural networks. 

Gradient descent can also converge even if the learning rate is kept fixed. 

Normal Equation is an analytical approach used for optimization. It is an alternative for Gradient descent. Normal equation performs minimization without iteration. 
Normal equations are equations obtained by setting equal to zero the partial derivatives of the sum of squared errors or cost function; normal equations allow one to estimate the parameters of multiple linear regression. 

Where 
X = input features value 
y = output value 

If the term X is non-invertible or singular then we can use regularization. 

Difference between Gradient Descent and Normal Equation. 

Gradient Descent	Normal Equation
In gradient descent, we need to choose The algorithm.	In normal equation, no need to choose the learning rate.
It is an iterative algorithm.	It is an analytical approach.
Gradient descent works well with large number of features.	Normal equation works well with small number of features.
Feature scaling can be used.	No need for feature scaling.
No need to handle non-invertibility case.	If (X) is non-invertible , regularization can be used to handle this.
Algorithm complexity is O().  n is the number of features.	Algorithm complexity is O().  n is the number of features.

My Personal Notes arrow_drop_up

Save