Mean squared error gradient

In this post, I show you how to implement the Mean Squared Error (MSE) Loss/Cost function as well as its derivative for Neural networks in Python. The function is meant for working with a batch of inputs, i.e., a batch of samples is provided at once.

The mathematical definition of the MSE loss function is

where are the expected or target outputs (known beforehand), are the predicted outputs from the neural network, and is the number of samples.

The derivative of the MSE loss function is:

Please note, that in the above definitions we are only considering a single output node. That is there is one output per sample.
But it is very common for neural networks to have more than one output node (multiple outputs) for a given input. In that case one has to average the squared error not only over the samples but also over the number of output nodes.

In this post I will only focus on this general case which means we would expect the predicted and target outputs to be 2D arrays, with nRows=nSamples, and nColumns=nOutput_Nodes. In other words, the rows correspond to outputs for different set of input samples, and the columns correspond to the outputs from the different output nodes.

Based on this we can implement the MSE loss function in Python as:

MSE loss simplest implementation

def MSE_loss(predictions, targets):
    """
    Computes Mean Squared error/loss between targets
    and predictions. 
    Input: predictions (N, k) ndarray (N: no. of samples, k: no. of output nodes)
          targets (N, k) ndarray        (N: no. of samples, k: no. of output nodes)
    Returns: scalar
    Note: The averaging is only done over the output nodes and not over the samples in a batch.
    Therefore, to get an answer similar to PyTorch, one must divide the result by the batch size.
    """
    
    return np.sum((predictions-targets**2)/predictions.shape[1]

MSE loss derivative simplest implementation

def MSE_loss_grad(predictions, targets):
    """
    Computes mean squared error gradient between targets 
    and predictions. 
    Input: predictions (N, k) ndarray (N: no. of samples, k: no. of output nodes)
          targets (N, k) ndarray        (N: no. of samples, k: no. of output nodes)
    Returns: (N,k) ndarray
    Note: The averaging is only done over the output nodes and not over the samples in a batch.
    Therefore, to get an answer similar to PyTorch, one must divide the result by the batch size.
    """
    
    return 2*(predictions-targets)/predictions.shape[1]

The above functions return the MSE loss and its derivative respectively, only averaged over the output nodes. Averaging over the samples can then be done by simply dividing the result by nSamples as shown

average_MSE = MSE_loss(predictions, target)/predictions.shape[0]
average_MSE_grad = MSE_loss_grad(predictions, target)/predictions.shape[0]

These implementations can be further accelerated (sped-up) by using Numba (https://numba.pydata.org/). Numba is a Just-in-time (JIT) compiler that

translates a subset of Python and NumPy code into fast machine code.

To use numba, install it as:

pip install numba

Also, make sure that your numpy is compatible with Numba or not, although usually pip takes care of that. You can get the info here: https://pypi.org/project/numba/

Accelerating the above functions using Numba is quite simple. Just modify them in the following manner:

MSE loss NUMBA implementation

from numba import njit 
@njit(cache=True,fastmath=True)
def MSE_loss(predictions, targets):
    """
    Computes Mean Squared error/loss between targets
    and predictions. 
    Input: predictions (N, k) ndarray (N: no. of samples, k: no. of output nodes)
          targets (N, k) ndarray        (N: no. of samples, k: no. of output nodes)
    Returns: scalar
    Note: The averaging is only done over the output nodes and not over the samples in a batch.
    Therefore, to get an answer similar to PyTorch, one must divide the result by the batch size.
    """
    
    return np.sum((predictions-targets)**2)/predictions.shape[1]

NOTE: I noticed that the above implementation can be unstable and return erroneous results for very large arrays. Therefore, you can use the following even faster implementation with parallelization:

from numba import njit 
@njit(cache=True,fastmath=False, parallel=True)
def MSE_loss(predictions, targets):
    """
    Computes Mean Squared error/loss between targets
    and predictions. 
    Input: predictions (N, k) ndarray (N: no. of samples, k: no. of output nodes)
          targets (N, k) ndarray        (N: no. of samples, k: no. of output nodes)
    Returns: scalar
    Note: The averaging is only done over the output nodes and not over the samples in a batch.
    Therefore, to get an answer similar to PyTorch, one must divide the result by the batch size.
    """
    loss = 0.0
    for i in prange(predictions.shape[0]):
        for j in range(predictions.shape[1]):
            loss = loss + (predictions[i,j] - targets[i,j])**2
    # Average over number of output nodes
    loss = loss / predictions.shape[1]
    return loss

MSE loss derivative NUMBA implementation

from numba import njit 
@njit(cache=True,fastmath=True)
def MSE_loss_grad(predictions, targets):
    """
    Computes mean squared error gradient between targets 
    and predictions. 
    Input: predictions (N, k) ndarray (N: no. of samples, k: no. of output nodes)
          targets (N, k) ndarray        (N: no. of samples, k: no. of output nodes)
    Returns: (N,k) ndarray
    Note: The averaging is only done over the output nodes and not over the samples in a batch.
    Therefore, to get an answer similar to PyTorch, one must divide the result by the batch size.
    """
    
    return 2*(predictions-targets)/predictions.shape[1]

This is quite fast and competitive with Tensorflow and PyTorch.

It is in fact also used in the CrysX-Neural Network library (crysx_nn)

Furthermore, the above implementations can also be accelerated using Cupy (CUDA).

CuPy is an open-source array library for GPU-accelerated computing with Python. CuPy utilizes CUDA Toolkit libraries to make full use of the GPU architecture.

The Cupy implementations look as follows:

def MSE_loss_cupy(predictions, targets):
    """
    Computes Mean Squared error/loss between targets
    and predictions. 
    Input: predictions (N, k) ndarray (N: no. of samples, k: no. of output nodes)
          targets (N, k) ndarray        (N: no. of samples, k: no. of output nodes)
    Returns: scalar
    Note: The averaging is only done over the output nodes and not over the samples in a batch.
    Therefore, to get an answer similar to PyTorch, one must divide the result by the batch size.
    """
    
    return cp.sum((predictions-targets)**2)/predictions.shape[1]

def MSE_loss_grad_cupy(predictions, targets):
    """
    Computes mean squared error gradient between targets 
    and predictions. 
    Input: predictions (N, k) ndarray (N: no. of samples, k: no. of output nodes)
          targets (N, k) ndarray        (N: no. of samples, k: no. of output nodes)
    Returns: (N,k) ndarray
    Note: The averaging is only done over the output nodes and not over the samples in a batch.
    Therefore, to get an answer similar to PyTorch, one must divide the result by the batch size.
    """
    
    return 2*(predictions-targets)/predictions.shape[1]

The above code is also used in the crysx_nn library.
I hope you found this information useful.

If you did, then don’t forget to check out my other posts on Machine Learning and efficient implementations of activation/loss functions in Python.

[wpedon id=»7041″ align=»center»]

Last Updated on June 21, 2020 by

Author(s): Pratik Shukla

Machine Learning

With Step-By-Step Mathematical Derivation

Index:

• Basics Of Gradient Descent.
• Basic Rules Of Derivation.
• Gradient Descent With One Variable.
• Gradient Descent With Two Variables.
• Gradient Descent For Mean Squared Error Function.

What is Gradient Descent?

Gradient Descent is a machine learning algorithm that operates iteratively to find the optimal values for its parameters. It takes into account, user-defined learning rate, and initial parameter values.

How does it work?

• Start with initial values.
• Calculate cost.
• Update values using the update function.
• Returns minimized cost for our cost function

Why do we need it?

Generally, what we do is, we find the formula that gives us the optimal values for our parameter. But in this algorithm, it finds the value by itself! Interesting, isn’t it?

Formula:

Some Basic Rules For Derivation:

( A ) Scalar Multiple Rule:

( B ) Sum Rule:

( C ) Power Rule:

( D ) Chain Rule:

Let’s have a look at various examples to understand it better.

Gradient Descent Minimization — Single Variable:

We’re going to be using gradient descent to find θ that minimizes the cost. But let’s forget the Mean Squared Error (MSE) cost function for a moment and take a look at gradient descent function in general.

Now what we generally do is, find the best value of our parameters using some sort of simplification and make a function that will give us minimized cost. But here what we’ll do is take some default or random for our parameters and let our program run iteratively to find the minimized cost.

Let’s Explore It In-Depth:

Let’s take a very simple function to begin with: J(θ) = θ², and our goal is to find the value of θ which minimizes J(θ).

From our cost function, we can clearly say that it will be minimum for θ= 0, but it won’t be so easy to derive such conclusions while working with some complex functions.

( A ) Cost function: We’ll try to minimize the value of this function.

( B ) Goal: To minimize the cost function.

( C ) Update Function: Initially we take a random number for our parameters, which are not optimal. To make it optimal we have to update it at each iteration. This function takes care of it.

( D ) Learning rate: The descent speed.

( E ) Updating Parameters:

( F ) Table Generation:

Here we are stating with θ = 5.

keep in mind that here θ = 0.8*θ, for our learning rate and cost function.

Here we can see that as our θ is decreasing the cost value is also decreasing. We just have to find the optimal value for it. To find the optimal value we have to do perform many iterations. The more the iterations, the more optimal value we get!

( G ) Graph: We can plot the graph of the above points.

Cost Function Derivative:

Why does the gradient descent use the derivative of the cost function? We want our cost function to be minimum, right? Minimizing the cost function simply gives us a lower error rate in predicting values. Ideally, we take the derivative of a function to 0 and find the parameters. Here we do the same thing but we start from a random number and try to minimize it iteratively.

The learning rate / ALPHA:

The learning rate gives us solid control over how large of steps we make. Selecting the right learning rate is a very critical task. If the learning rate is too high then you might overstep the minimum and diverge. For example, in the above example if we take alpha =2 then each iteration will take us away from the minimum. So we use small alpha values. But the only concern with using a small learning rate is we have to perform more iteration to reach the minimum cost value, this increases training time.

Convergence / Stopping gradient descent:

Note that in the above example the gradient descent will never actually converge to a minimum of theta= 0. Methods for deciding when to stop our iterations are beyond my level of expertise. But I can tell you that while doing assignments we can take a fixed number of iterations like 100 or 1000.

Gradient Descent — Multiple Variables:

Our ultimate goal is to find the parameters for MSE function, which includes multiple variables. So here we will discuss a cost function which as 2 variables. Understanding this will help us very much in our MSE Cost function.

Let’s take this function:

When there are multiple variables in the minimization objective, we have to define separate rules for update function. With more than one parameter in our cost function, we have to use partial derivative. Here I simplified the partial derivative process. Let’s have a look at this.

( A ) Cost Function:

( B ) Goal:

( C ) Update Rules:

( D ) Derivatives:

( E ) Update Values:

( F ) Learning Rate:

( G ) Table:

Starting with θ1 =1 ,θ2 =1. And then updating the value using update functions.

( H ) Graph:

Here we can see that as we increase our number of iterations, our cost value is going down.

Note that while implementing the program in python the new values must not be updated until we find new values for both θ1 and θ2. We clearly don’t want to use the new value of θ1 to be used in the old value of θ2.

Gradient Descent For Mean Squared Error:

Now that we know how to perform gradient descent on an equation with multiple variables, we can return to looking at gradient descent on our MSE cost function.

Let’s get started!

( A ) Hypothesis function:

( B ) cost function:

( C ) Find partial derivative of J(θ0,θ1) w.r.t to θ1:

( D ) Simplify a little:

( E ) Define a variable u:

( F ) Value of u:

( G ) Finding partial derivative:

( H ) Rewriting the equations:

( I ) Merge all the calculated data:

( J ) Repeat the same process for derivation of J(θ0,θ1) w.r.t θ1:

( K ) Simplified calculations:

( L ) Combine all calculated Data:

One Half Mean Squared Error :

We multiply our MSE cost function with 1/2, so that when we take the derivative the 2s cancel out. Multiplying the cost function by a scalar does not affect the location of the minimum, so we can get away with this.

Final :

( A ) Cost Function: One Half Mean Squared Error:

( B ) Goal:

( C ) Update Rule:

( D ) Derivatives:

So, that’s it. We finally made it!

Conclusion:

We are going to use the same method in various applications of machine learning algorithms. But at that time we are not going to go in this depth, we’re just going to use the final formula. But it’s always good to know how it’s derived!

Final Formula:

Is the concept lucid to you now? Please let me know by writing responses. If you enjoyed this article then hit the clap icon.

If you have any additional confusions, feel free to contact me. [email protected]

Gradient Descent Algorithm Explained was originally published in Towards AI — Multidisciplinary Science Journal on Medium, where people are continuing the conversation by highlighting and responding to this story.

Published via Towards AI

Функции активации

сигмовидная

σ(x) = 1/(1+e⁻ˣ)

• классическая функция активации для логистической регрессии
• сжать значения до [0, 1]
• Устарело в Pytorch 1.8.1: torch.nn.functional.sigmoid
• Вместо этого используйте torch.sigmoid()

import torch

# import torch.nn.functional as F  # deprecated for sigmoid

# sigmoid
print("sigmoid function")
z = torch.linspace(-100, 100, 10)
print(z)
print(torch.sigmoid(z))  # value range [0,1]

• Нахождение градиента для сигмовидной функции: градиент может быть представлен самим σ, поэтому он требует меньше вычислительных ресурсов.

Преимущества:

• σ известно, поэтому не нужно искать производную, более эффективную

Недостаток:

• Когда x->∞, градиент ->0, очень небольшое обновление до w и b

Тан

tanh(x) = (eˣ-e⁻ˣ)/(eˣ+e⁻ˣ)=2*сигмоид(2x)-1

• Обычно используется в РНН
• Это преобразование сигмовидной функции
• сжать значение до [-1, 1]
• Градиент tanh(x) = 1-tanh²(x)

# tanh
print("ntanh function")
z = torch.linspace(-1, 1, 10)
print(torch.tanh(z))  # value range [-1, 1]

Релу

• Самый популярный для нейронной сети
• когда x‹0, градиент=0, когда x›0, градиент=x

# relu
print("nrelu function")
z = torch.linspace(-1, 1, 9)
print(z)
print(torch.relu(z))  # when z=0, torch.relu() gives 0

Функции потерь

Среднеквадратическая ошибка (MSE)

MSE=∑(y-(x*w+b))²

L2-норма = √(∑(y-(x*w+b))²) (RMS)

MSE = (L2-норма)²

import torch
import torch.nn.functional as F

# MSE
x=torch.ones(1)
w=torch.full([1], 2.)
mse=F.mse_loss(x*w, torch.ones(1))
print(mse)

Нахождение градиента для функции MSE

• если w не был определен правильно, torch.autograd.grad() будет иметь ошибку
• запустите w.requires_grad_() и снова определите mse для обновления с новым w, прежде чем использовать autograd.
• Последний _ в w.requires_grad_() означает модификацию w на месте

# torch.autograd.grad(mse, [w])  # error

w.requires_grad_()  # need to add this line. The last _ means inplace
# torch.autograd.grad(mse, [w])  # error again

mse=F.mse_loss(x*w, torch.ones(1))  # need to update the loss function with updated x
print(mse)  # now mse contains information about gradient function

print(torch.autograd.grad(mse, [w]))

• Другой способ расчета градиента с помощью mse.backward()

# another way to calculate gradient
mse=F.mse_loss(x*w, torch.ones(1))
mse.backward()
print(w.grad)   # gradient is now saved in w

Софтмакс

• Отличие от сигмовидной: сжать значения до 0–1 И сумма всех членов = 1
• Это увеличит большую разницу и сожмет небольшую разницу между значениями:

b=torch.tensor([2,1,0.5])  # original ratio is [4:2:1]
b.requires_grad_()
p=F.softmax(b, dim=0)
print(p)
# OUTPUT: tensor([0.6285, 0.2312, 0.1402], grad_fn=<SoftmaxBackward>)
# ratio after softmax: [4.48:1.65:1]
bigger difference for big difference: 4.48/1.65=2.7
smaller difference for small difference: 1.65/1=1.65

Поиск градиента для функции softmax

• можно вычислить градиент только для скаляра, т.е. каждый элемент в softmax
• градиент для одного и того же термина +ve, для разных терминов -ve

# gradient for softmax
a=torch.rand(3)
a.requires_grad_()  
print(a)

p=F.softmax(a, dim=0)
# p.backward()   # error, grad can only calculate for scalar outputs
print(torch.autograd.grad(p[1], [a], retain_graph=True))
print(torch.autograd.grad(p[2], [a])) # need to include retain_graph=True in previous step for running autograd again
# print(torch.autograd.grad(p[0], [a]))  # error here since previous step has no retain_graph

Чаевые для unknown

• для последовательного запуска torch.autograd необходимо установить retain_graph=True, чтобы сохранить предыдущую информацию о градиенте
• eg. torch.autograd.grad(p[1], [a], retain_graph=True)

Ссылка

ссылка1