Squared error loss function

From Wikipedia, the free encyclopedia

In mathematical optimization and decision theory, a loss function or cost function (sometimes also called an error function) ^[1] is a function that maps an event or values of one or more variables onto a real number intuitively representing some «cost» associated with the event. An optimization problem seeks to minimize a loss function. An objective function is either a loss function or its opposite (in specific domains, variously called a reward function, a profit function, a utility function, a fitness function, etc.), in which case it is to be maximized. The loss function could include terms from several levels of the hierarchy.

In statistics, typically a loss function is used for parameter estimation, and the event in question is some function of the difference between estimated and true values for an instance of data. The concept, as old as Laplace, was reintroduced in statistics by Abraham Wald in the middle of the 20th century.^[2] In the context of economics, for example, this is usually economic cost or regret. In classification, it is the penalty for an incorrect classification of an example. In actuarial science, it is used in an insurance context to model benefits paid over premiums, particularly since the works of Harald Cramér in the 1920s.^[3] In optimal control, the loss is the penalty for failing to achieve a desired value. In financial risk management, the function is mapped to a monetary loss.

Examples[edit]

Regret[edit]

Leonard J. Savage argued that using non-Bayesian methods such as minimax, the loss function should be based on the idea of regret, i.e., the loss associated with a decision should be the difference between the consequences of the best decision that could have been made had the underlying circumstances been known and the decision that was in fact taken before they were known.

Quadratic loss function[edit]

The use of a quadratic loss function is common, for example when using least squares techniques. It is often more mathematically tractable than other loss functions because of the properties of variances, as well as being symmetric: an error above the target causes the same loss as the same magnitude of error below the target. If the target is t, then a quadratic loss function is

for some constant C; the value of the constant makes no difference to a decision, and can be ignored by setting it equal to 1. This is also known as the squared error loss (SEL). ^[1]

Many common statistics, including t-tests, regression models, design of experiments, and much else, use least squares methods applied using linear regression theory, which is based on the quadratic loss function.

The quadratic loss function is also used in linear-quadratic optimal control problems. In these problems, even in the absence of uncertainty, it may not be possible to achieve the desired values of all target variables. Often loss is expressed as a quadratic form in the deviations of the variables of interest from their desired values; this approach is tractable because it results in linear first-order conditions. In the context of stochastic control, the expected value of the quadratic form is used.

0-1 loss function[edit]

In statistics and decision theory, a frequently used loss function is the 0-1 loss function

where is the indicator function.
Meaning if the input is evaluated to true, then the output is 1. Otherwise, if the input is evaluated to false, then the output will be 0.

Constructing loss and objective functions[edit]

In many applications, objective functions, including loss functions as a particular case, are determined by the problem formulation. In other situations, the decision maker’s preference must be elicited and represented by a scalar-valued function (called also utility function) in a form suitable for optimization — the problem that Ragnar Frisch has highlighted in his Nobel Prize lecture.^[4]
The existing methods for constructing objective functions are collected in the proceedings of two dedicated conferences.^[5][6]
In particular, Andranik Tangian showed that the most usable objective functions — quadratic and additive — are determined by a few indifference points. He used this property in the models for constructing these objective functions from either ordinal or cardinal data that were elicited through computer-assisted interviews with decision makers.^[7][8]
Among other things, he constructed objective functions to optimally distribute budgets for 16 Westfalian universities^[9]
and the European subsidies for equalizing unemployment rates among 271 German regions.^[10]

Expected loss[edit]

In some contexts, the value of the loss function itself is a random quantity because it depends on the outcome of a random variable X.

Statistics[edit]

Both frequentist and Bayesian statistical theory involve making a decision based on the expected value of the loss function; however, this quantity is defined differently under the two paradigms.

Frequentist expected loss[edit]

We first define the expected loss in the frequentist context. It is obtained by taking the expected value with respect to the probability distribution, P_θ, of the observed data, X. This is also referred to as the risk function^{[11][12][13][14]} of the decision rule δ and the parameter θ. Here the decision rule depends on the outcome of X. The risk function is given by:

Here, θ is a fixed but possibly unknown state of nature, X is a vector of observations stochastically drawn from a population, is the expectation over all population values of X, dP_θ is a probability measure over the event space of X (parametrized by θ) and the integral is evaluated over the entire support of X.

Bayesian expected loss[edit]

In a Bayesian approach, the expectation is calculated using the posterior distribution π^* of the parameter θ:

One then should choose the action a^* which minimises the expected loss. Although this will result in choosing the same action as would be chosen using the frequentist risk, the emphasis of the Bayesian approach is that one is only interested in choosing the optimal action under the actual observed data, whereas choosing the actual frequentist optimal decision rule, which is a function of all possible observations, is a much more difficult problem.

Examples in statistics[edit]

• For a scalar parameter θ, a decision function whose output is an estimate of θ, and a quadratic loss function (squared error loss)

  

  the risk function becomes the mean squared error of the estimate,

  

  An Estimator found by minimizing the Mean squared error estimates the Posterior distribution’s mean.

• In density estimation, the unknown parameter is probability density itself. The loss function is typically chosen to be a norm in an appropriate function space. For example, for L² norm,

  

  the risk function becomes the mean integrated squared error

  

Economic choice under uncertainty[edit]

In economics, decision-making under uncertainty is often modelled using the von Neumann–Morgenstern utility function of the uncertain variable of interest, such as end-of-period wealth. Since the value of this variable is uncertain, so is the value of the utility function; it is the expected value of utility that is maximized.

Decision rules[edit]

A decision rule makes a choice using an optimality criterion. Some commonly used criteria are:

• Minimax: Choose the decision rule with the lowest worst loss — that is, minimize the worst-case (maximum possible) loss:

  

• Invariance: Choose the decision rule which satisfies an invariance requirement.
• Choose the decision rule with the lowest average loss (i.e. minimize the expected value of the loss function):

  

Selecting a loss function[edit]

Sound statistical practice requires selecting an estimator consistent with the actual acceptable variation experienced in the context of a particular applied problem. Thus, in the applied use of loss functions, selecting which statistical method to use to model an applied problem depends on knowing the losses that will be experienced from being wrong under the problem’s particular circumstances.^[15]

A common example involves estimating «location». Under typical statistical assumptions, the mean or average is the statistic for estimating location that minimizes the expected loss experienced under the squared-error loss function, while the median is the estimator that minimizes expected loss experienced under the absolute-difference loss function. Still different estimators would be optimal under other, less common circumstances.

In economics, when an agent is risk neutral, the objective function is simply expressed as the expected value of a monetary quantity, such as profit, income, or end-of-period wealth. For risk-averse or risk-loving agents, loss is measured as the negative of a utility function, and the objective function to be optimized is the expected value of utility.

Other measures of cost are possible, for example mortality or morbidity in the field of public health or safety engineering.

For most optimization algorithms, it is desirable to have a loss function that is globally continuous and differentiable.

Two very commonly used loss functions are the squared loss, , and the absolute loss, . However the absolute loss has the disadvantage that it is not differentiable at . The squared loss has the disadvantage that it has the tendency to be dominated by outliers—when summing over a set of ‘s (as in ), the final sum tends to be the result of a few particularly large a-values, rather than an expression of the average a-value.

The choice of a loss function is not arbitrary. It is very restrictive and sometimes the loss function may be characterized by its desirable properties.^[16] Among the choice principles are, for example, the requirement of completeness of the class of symmetric statistics in the case of i.i.d. observations, the principle of complete information, and some others.

W. Edwards Deming and Nassim Nicholas Taleb argue that empirical reality, not nice mathematical properties, should be the sole basis for selecting loss functions, and real losses often are not mathematically nice and are not differentiable, continuous, symmetric, etc. For example, a person who arrives before a plane gate closure can still make the plane, but a person who arrives after can not, a discontinuity and asymmetry which makes arriving slightly late much more costly than arriving slightly early. In drug dosing, the cost of too little drug may be lack of efficacy, while the cost of too much may be tolerable toxicity, another example of asymmetry. Traffic, pipes, beams, ecologies, climates, etc. may tolerate increased load or stress with little noticeable change up to a point, then become backed up or break catastrophically. These situations, Deming and Taleb argue, are common in real-life problems, perhaps more common than classical smooth, continuous, symmetric, differentials cases.^[17]

See also[edit]

• Bayesian regret
• Loss functions for classification
• Discounted maximum loss
• Hinge loss
• Scoring rule
• Statistical risk

References[edit]

Further reading[edit]

• Aretz, Kevin; Bartram, Söhnke M.; Pope, Peter F. (April–June 2011). «Asymmetric Loss Functions and the Rationality of Expected Stock Returns» (PDF). International Journal of Forecasting. 27 (2): 413–437. doi:10.1016/j.ijforecast.2009.10.008. SSRN 889323.
• Berger, James O. (1985). Statistical decision theory and Bayesian Analysis (2nd ed.). New York: Springer-Verlag. Bibcode:1985sdtb.book…..B. ISBN 978-0-387-96098-2. MR 0804611.

• Cecchetti, S. (2000). «Making monetary policy: Objectives and rules». Oxford Review of Economic Policy. 16 (4): 43–59. doi:10.1093/oxrep/16.4.43.

• Horowitz, Ann R. (1987). «Loss functions and public policy». Journal of Macroeconomics. 9 (4): 489–504. doi:10.1016/0164-0704(87)90016-4.

• Waud, Roger N. (1976). «Asymmetric Policymaker Utility Functions and Optimal Policy under Uncertainty». Econometrica. 44 (1): 53–66. doi:10.2307/1911380. JSTOR 1911380.

A loss function is a parameter estimation function which represents the error (loss) of a machine learning (ML) model. The main goal of every ML model is to minimize this error. Loss basically represents some numerical value that tells us how poor the prediction is, considering only one example. In case the prediction is ideal, the loss is going to be 0. In other cases, the prediction is not that good, so the model’s loss/error is (much) higher. A loss function is also called «cost function», «objective function», «optimization score function» or just «error function». So don’t get confused! 😀

A loss function is a useful tool with the means of which we can estimate weights and biases that suit the model in the best way. By finding the most appropriate parameters for a ML model, the model will then have a low error in respect to all data samples in the dataset.

The function is called «loss» because it penalizes the model if it produces a false or not optimal prediction. A loss function «punishes» the model according to its predictions in a way that the model receives a «feedback» to some extent from the loss function and can improve on it further.

In short, the major task of a loss function is to determine the most optimal gradients (vectors with partial derivatives) in a ML model, so it can help to minimize the error. This is our «goal», our «ideal outcome» we actually want to have for our model and we try to make what we have as an output as close to that «ideal outcome» as possible with the help of a loss function. We can achieve this optimal, desired outcome for our model by adjusting weights and biases: our model’s parameters. Therefore, we call loss fucntions «parameter estimation functions».

There are many loss functions out there. You can look at some of them, already implemented in Tensorflow: tf.losses .

Homer Simpson Math 😃

And now we will go through some frequently used loss functions.

Squared Error

As an example of loss functions, we will look at the squared error function, which is widely used in linear regression. The error will be larger in case, when the prediction is far away from the desired goal.

Likelihood (L) = Σ ^N_i=1 (desired_output_i — predicted_output_i)²

or to make it shorter:

= Σ ^N_i=1 (d_i — p_i)²

Mean Squared Error (MSE)

MSE determines how close some line (e.g. a regression line) is to some data sample set. MSE measures the distances from those data samples up to the line. These distances are called the errors, which MSE squares in the end. We have to square the errors (distances) because we might encounter negative values otherwise. The larger the differences between a data point and its projection on the line, the more importance it gains.

Mean Squared Error has the «mean» part in its name because we find the average of some error set (the distances) in the end of the computation. MSE again operates as estimator for model’s performance. Look again at the Squared Error formula above and proceed with this one:

Notice that this function is a quadratic one. That means this function is a convex function. A convex function has one global minimum. Such quality is rather helpful because our computation won’t get stuck in a local minimum.

If the value of MSE is small, it denotes that we are close to find the best fit for the line. So, the smaller the value, the better fit it means. Because of the quadratic element in the formula, one should always remember that MSE highlights the extremes in its calculations.

The algorithm of the MSE:

1. find the regression line for the data set



2. use the formula of MSE: find the error (the distances), square the errors, sum the errors up, find the mean.

Maximum Likelihood Estimation

or MLE for short, is a function applied for parameter estimation. In MLE the loss function is represented as the log-likelihood function. Log-likelihood is used to derive the maximum likelihood of some parameter Θ (theta). We use log here because the asymptotic curve of the logarithm prevents from exploding or shirking values. Thus the sums of the values are easier to analyze.

In models, based on probability, MLE has a nice application in relation to loss functions as MLE can be used to figure out the loss function for such models. Thus, maximum likelihood estimation can be applied to select a loss function for a ML model.

You can read more about Maximum Likelihood Estimation in the end of this article .

Cross Entropy

Cross Entropy Loss (sometimes named «Log Loss» or «Negative Log Likelihood») is one of the loss functions in ML. It judges a probability output of a ML classification model, that lies between [0;1]. We normally use cross entropy as a loss function when we have the softmax activation function in the last output layer in a multi layered neural network. In case that the predicted probability continuously varies from the actual label, the cross entropy will increase. In the ideal situation the cross entropy (or log loss) is equal to zero, so the difference from the actual (ideal) label is zero (or very small).

As a rule of thumb, use the cross entropy loss with softmax.

The idea of cross entropy and entropy itself roots in the information theory which deals with the problem of reliable message transmission with zero or minimum amount of information loss. We also want only useful information, with which we can make further predictions.

In other words, entropy is a measure of practical information we receive from one data point. The entropy becomes large if the variation in the data becomes large.

Binary Cross Entropy vs Categorical Cross Entropy

Binary Cross Entropy Function :

If we have to classify only two classes, we chose binary cross entropy.

L = — Σ ^N_n=1 {d_n log p_n + (1 — d_n) log(1 — p_n)}

We said before that we use softmax as an activation function in the last layer. With binary cross entropy, it is advisable to use the sigmoid activation function. Feel free to read more about activation functions.

The code demonstration will be done with the help of the
Keras
ML library.

from keras.preprocessing import sequence
from keras.models import Sequential
from keras.layers import Dense, Embedding, LSTM, Bidirectional


#  this function builds, trains and evaluates a keras LSTM model 

def build_and_evaluate_model(x_train, y_train, x_develop, y_develop):

    x_train = sequence.pad_sequences(x_train, maxlen = 100)
    x_dev = sequence.pad_sequences(x_develop, maxlen = 100)
    
    model = Sequential()
    model.add(Embedding(input_dim = 10000, output_dim = 50))  #input_dim or the size of the data set, e.g vocabulary 
    model.add(Bidirectional(LSTM(units = 25)))
    model.add(Dense(1,activation='sigmoid'))
    model.predict(x_develop)
    model.compile(loss = 'binary_crossentropy', optimizer = 'adam', metrics=['accuracy'])
    model.fit(x = x_train, y = y_train, batch_size = 32, epochs = 10, validation_data = (x_develop, y_develop))
    
    score, acc = model.evaluate(x_develop, y_develop)
 
    return score, acc, model 

Consider the whole code following this link to: Train a recurrent convolutional network on the IMDB sentiment classification task.

To find out more about the LSTM model used in the code above, read this article.

Categorical Cross Entropy Function :

If we have to classify multiple classes, we chose categorical cross entropy.

Categorical Cross Entropy is sometimes referred to as the softmax loss. The softmax loss is basically the softmax activation function combined with the cross entropy loss function. Essentially, it will output some probability over some N classes for each data sample.

Categorical Cross Entropy is already implemented in the Keras ML library:

keras.losses.categorical_crossentropy(y_true, y_pred, from_logits=False, label_smoothing=0) 

We can also use already implemented in Keras mean squared error (MSE) as our loss function. As a general rule, use a linear activation function and only one node in the output layer if MSE is used :

model.compile(loss="mse")  #instead of writing "mse", "mean_squared_error" is also acceptable and less cryptic 

model.add(Dense(1, activation="linear"))

Feel free to consult the Keras official documentation .

Conclusion

To sum up, loss functions are not always easy to comprehend. In some ML cases, it is more relevant to measure and present the accuracy of classification Ml models.

Sometimes we should select other metrics than the loss functions. An important thought: one may assume that the ML model which has the minimum loss, would be the best pick, but it doesn’t have to be like this. The model with the lowest loss still can be the model with the worst metric after all.

A sufficient practice would be to apply a loss function for evaluation of the model’s learning progress. Shortly, use loss functions for optimization: analyze whether there are typical problems such as: slow convergence or over/underfitting in the model. Chose the proper metric according to the task the ML model have to accomplish and use a loss function as an optimizer for model’s performance.

Further recommended readings:

Recurrent Neural Networks (RNNs)

Partial Derivatives and the Jacobian Matrix

Usage of loss functions Keras

Machine learning: an introduction to mean squared error and regression lines

In this post, I show you how to implement the Mean Squared Error (MSE) Loss/Cost function as well as its derivative for Neural networks in Python. The function is meant for working with a batch of inputs, i.e., a batch of samples is provided at once.

The mathematical definition of the MSE loss function is

where are the expected or target outputs (known beforehand), are the predicted outputs from the neural network, and is the number of samples.

The derivative of the MSE loss function is:

Please note, that in the above definitions we are only considering a single output node. That is there is one output per sample.
But it is very common for neural networks to have more than one output node (multiple outputs) for a given input. In that case one has to average the squared error not only over the samples but also over the number of output nodes.

In this post I will only focus on this general case which means we would expect the predicted and target outputs to be 2D arrays, with nRows=nSamples, and nColumns=nOutput_Nodes. In other words, the rows correspond to outputs for different set of input samples, and the columns correspond to the outputs from the different output nodes.

Based on this we can implement the MSE loss function in Python as:

MSE loss simplest implementation

def MSE_loss(predictions, targets):
    """
    Computes Mean Squared error/loss between targets
    and predictions. 
    Input: predictions (N, k) ndarray (N: no. of samples, k: no. of output nodes)
          targets (N, k) ndarray        (N: no. of samples, k: no. of output nodes)
    Returns: scalar
    Note: The averaging is only done over the output nodes and not over the samples in a batch.
    Therefore, to get an answer similar to PyTorch, one must divide the result by the batch size.
    """
    
    return np.sum((predictions-targets**2)/predictions.shape[1]

MSE loss derivative simplest implementation

def MSE_loss_grad(predictions, targets):
    """
    Computes mean squared error gradient between targets 
    and predictions. 
    Input: predictions (N, k) ndarray (N: no. of samples, k: no. of output nodes)
          targets (N, k) ndarray        (N: no. of samples, k: no. of output nodes)
    Returns: (N,k) ndarray
    Note: The averaging is only done over the output nodes and not over the samples in a batch.
    Therefore, to get an answer similar to PyTorch, one must divide the result by the batch size.
    """
    
    return 2*(predictions-targets)/predictions.shape[1]

The above functions return the MSE loss and its derivative respectively, only averaged over the output nodes. Averaging over the samples can then be done by simply dividing the result by nSamples as shown

average_MSE = MSE_loss(predictions, target)/predictions.shape[0]
average_MSE_grad = MSE_loss_grad(predictions, target)/predictions.shape[0]

These implementations can be further accelerated (sped-up) by using Numba (https://numba.pydata.org/). Numba is a Just-in-time (JIT) compiler that

translates a subset of Python and NumPy code into fast machine code.

To use numba, install it as:

pip install numba

Also, make sure that your numpy is compatible with Numba or not, although usually pip takes care of that. You can get the info here: https://pypi.org/project/numba/

Accelerating the above functions using Numba is quite simple. Just modify them in the following manner:

MSE loss NUMBA implementation

from numba import njit 
@njit(cache=True,fastmath=True)
def MSE_loss(predictions, targets):
    """
    Computes Mean Squared error/loss between targets
    and predictions. 
    Input: predictions (N, k) ndarray (N: no. of samples, k: no. of output nodes)
          targets (N, k) ndarray        (N: no. of samples, k: no. of output nodes)
    Returns: scalar
    Note: The averaging is only done over the output nodes and not over the samples in a batch.
    Therefore, to get an answer similar to PyTorch, one must divide the result by the batch size.
    """
    
    return np.sum((predictions-targets)**2)/predictions.shape[1]

NOTE: I noticed that the above implementation can be unstable and return erroneous results for very large arrays. Therefore, you can use the following even faster implementation with parallelization:

from numba import njit 
@njit(cache=True,fastmath=False, parallel=True)
def MSE_loss(predictions, targets):
    """
    Computes Mean Squared error/loss between targets
    and predictions. 
    Input: predictions (N, k) ndarray (N: no. of samples, k: no. of output nodes)
          targets (N, k) ndarray        (N: no. of samples, k: no. of output nodes)
    Returns: scalar
    Note: The averaging is only done over the output nodes and not over the samples in a batch.
    Therefore, to get an answer similar to PyTorch, one must divide the result by the batch size.
    """
    loss = 0.0
    for i in prange(predictions.shape[0]):
        for j in range(predictions.shape[1]):
            loss = loss + (predictions[i,j] - targets[i,j])**2
    # Average over number of output nodes
    loss = loss / predictions.shape[1]
    return loss

MSE loss derivative NUMBA implementation

from numba import njit 
@njit(cache=True,fastmath=True)
def MSE_loss_grad(predictions, targets):
    """
    Computes mean squared error gradient between targets 
    and predictions. 
    Input: predictions (N, k) ndarray (N: no. of samples, k: no. of output nodes)
          targets (N, k) ndarray        (N: no. of samples, k: no. of output nodes)
    Returns: (N,k) ndarray
    Note: The averaging is only done over the output nodes and not over the samples in a batch.
    Therefore, to get an answer similar to PyTorch, one must divide the result by the batch size.
    """
    
    return 2*(predictions-targets)/predictions.shape[1]

This is quite fast and competitive with Tensorflow and PyTorch.

It is in fact also used in the CrysX-Neural Network library (crysx_nn)

Furthermore, the above implementations can also be accelerated using Cupy (CUDA).

CuPy is an open-source array library for GPU-accelerated computing with Python. CuPy utilizes CUDA Toolkit libraries to make full use of the GPU architecture.

The Cupy implementations look as follows:

def MSE_loss_cupy(predictions, targets):
    """
    Computes Mean Squared error/loss between targets
    and predictions. 
    Input: predictions (N, k) ndarray (N: no. of samples, k: no. of output nodes)
          targets (N, k) ndarray        (N: no. of samples, k: no. of output nodes)
    Returns: scalar
    Note: The averaging is only done over the output nodes and not over the samples in a batch.
    Therefore, to get an answer similar to PyTorch, one must divide the result by the batch size.
    """
    
    return cp.sum((predictions-targets)**2)/predictions.shape[1]

def MSE_loss_grad_cupy(predictions, targets):
    """
    Computes mean squared error gradient between targets 
    and predictions. 
    Input: predictions (N, k) ndarray (N: no. of samples, k: no. of output nodes)
          targets (N, k) ndarray        (N: no. of samples, k: no. of output nodes)
    Returns: (N,k) ndarray
    Note: The averaging is only done over the output nodes and not over the samples in a batch.
    Therefore, to get an answer similar to PyTorch, one must divide the result by the batch size.
    """
    
    return 2*(predictions-targets)/predictions.shape[1]

The above code is also used in the crysx_nn library.
I hope you found this information useful.

If you did, then don’t forget to check out my other posts on Machine Learning and efficient implementations of activation/loss functions in Python.

[wpedon id=»7041″ align=»center»]

Before we discuss different kinds of loss functions used in deep learning, let’s talk about why we need loss functions in the first place. To do that, we first need to learn about what’s happening inside a neural network.

More From Artem OppermannArtificial Intelligence vs. Machine Learning vs. Deep Learning

Why Do We Need Loss Functions in Deep Learning?

There are two possible mathematical operations happening inside a neural network:

• Forward propagation

• Backpropagation with gradient descent

While forward propagation refers to the computational process of predicting an output for a given input vector x, backpropagation and gradient descent describe the process of improving the weights and biases of the network in order to make better predictions. Let’s look at this in practice. 

For a given input vector x the neural network predicts an output, which is generally called a prediction vector y.

The equations describing the mathematics happening during the prediction vector’s computation looks like this:

We must compute a dot-product between the input vector x and the weight matrix W1 that connects the first layer with the second. After that, we apply a non-linear activation function to the result of the dot-product. 

The prediction vector can represent a number of things depending on the task we want the network to do. For regression tasks, which are basically predictions of continuous variables (e.g. stock price, expected demand for products, etc.), the output vector y contains continuous numbers. 

Regardless of the task, we somehow have to measure how close our predictions are to the ground truth label.        

On the other hand, for classification tasks, such as customer segmentation or image classification, the output vector y represents probability scores between 0.0 and 1.0.

This is where the concept of a loss function comes into play.

Mathematically, we can measure the difference (or error) between the prediction vector y and the label y_hat by defining a loss function whose value depends on this difference.

An example of a general loss function is the quadratic loss:

The value of this loss function depends on the difference between the label y_hat and y. A higher difference means a higher loss value while (you guessed it) a smaller difference means a smaller loss value. Minimizing the loss function directly leads to more accurate predictions of the neural network as the difference between the prediction and the label decreases.

The neural network’s only objective is to minimize the loss function.

In fact, the neural network’s only objective is to minimize the loss function. This is because minimizing the loss function automatically causes the neural network model to make better predictions regardless of the exact characteristics of the task at hand.

A neural network solves tasks without being explicitly programmed with a task-specific rule. This is possible because the goal of minimizing the loss function is universal and doesn’t depend on the task or circumstances.

3 Key Types of Loss Functions in Neural Networks

That said, you still have to select the right loss function for the task at hand. Luckily there are only three loss functions you need to know to solve almost any problem.

1. Mean Squared Error Loss Function

Mean squared error (MSE) loss function is the sum of squared differences between the entries in the prediction vector y and the ground truth vector y_hat.

You divide the sum of squared differences by N, which corresponds to the length of the vectors. If the output y of your neural network is a vector with multiple entries then N is the number of the vector entries with y_i being one particular entry in the output vector.

The mean squared error loss function is the perfect loss function if you’re dealing with a regression problem. That is, if you want your neural network to predict a continuous scalar value.

An example of a regression problem would be predictions of . . . 

• the number of products needed in a supply chain.

• future real estate prices under certain market conditions.

• a stock value.

Here is a code snippet where I’ve calculated MSE loss in Python:

import numpy as np

# The prediction vector of the neural network
y_pred=[0.6, 1.29, 1.99, 2.69, 3.4]

# The ground truth label
y_hat= [1, 1, 2, 2, 4]

# Mean squared error
MSE = np.sum(np.square(np.subtract(y_hat, y_pred)))/len(y_hat)

print(MSE) # The result is 0.21606

Learn More From Our ExpertsWhat Is Linear Regression?

2. Cross-Entropy Loss Function

Regression is only one of two areas where feedforward networks enjoy great popularity. The other area is classification.

In classification tasks, we deal with predictions of probabilities, which means the output of a neural network must be in a range between zero and one. A loss function that can measure the error between a predicted probability and the label which represents the actual class is called the cross-entropy loss function.

One important thing we need to discuss before continuing with the cross-entropy is what exactly the ground truth vector looks like in the case of a classification problem.

The label vector y_hat is one hot encoded which means the values in this vector can only take discrete values of either zero or one. The entries in this vector represent different classes. The values of these entries are zero, except for a single entry which is one. This entry tells us the class into which we want to classify the input feature vector x.

The prediction y, however, can take continuous values between zero and one.

Given the prediction vector y and the ground truth vector y_hat you can compute the cross-entropy loss between those two vectors as follows:

First, we need to sum up the products between the entries of the label vector y_hat and the logarithms of the entries of the predictions vector y. Then we must negate the sum to get a positive value of the loss function.

One interesting thing to consider is the plot of the cross-entropy loss function. In the following graph, you can see the value of the loss function (y-axis) vs. the predicted probability y_i. Here y_i takes values between zero and one.

We can see clearly that the cross-entropy loss function grows exponentially for lower values of the predicted probability y_i. For y_i=0 the function becomes infinite, while for y_i=1 the neural network makes an accurate probability prediction and the loss value goes to zero.

Here’s another code snippet in Python where I’ve calculated the cross-entropy loss function:

import numpy as np

# The probabilities predicted by the neural network
y_pred = [0.1, 0.3, 0.4, 0.2]

# one-hot-encoded ground truth label
y_hat =[0, 1, 0, 0]

cross_entropy = - np.sum(np.log(y_pred)*y_hat)

print(cross_entropy) # The Result is 1.20

Read More From Our Machine Learning ExpertsMachine Learning for Beginners

3. Mean Absolute Percentage Error

Finally, we come to the Mean Absolute Percentage Error (MAPE) loss function. This loss function doesn’t get much attention in deep learning. For the most part, we use it to measure the performance of a neural network during demand forecasting tasks.

First thing first: what is demand forecasting?

Demand forecasting is the area of predictive analytics dedicated to predicting the expected demand for a good or service in the near future. For example:

• In retail, we can use demand forecasting models to determine the amount of a particular product that should be available and at what price.

• In industrial manufacturing, we can predict how much of each product should be produced, the amount of stock that should be available at various points in time, and when maintenance should be performed.

• In the travel and tourism industry, we can use demand forecasting models to assess optimal price points for flights and hotels, in light of available capacity, what price should be assigned (for hotels, flights), which destinations should be spotlighted, or, what types of packages should be advertised.

Although demand forecasting is also a regression task and the minimization of the MSE loss function is an adequate training goal, this type of loss function to measure the performance of the model during training isn’t suitable for demand forecasting.

Why is that?

Well, imagine the MSE loss function gives you a value of 100. Can you tell if this is generally a good result? No, because it depends on the situation. If the prediction y of the model is 1000 and the actual ground truth label y_hat is 1010, then the MSE loss of 100 would be in fact a very small error and the performance of the model would be quite good.

However in the case where the prediction would be five and the label is 15, you would have the same loss value of 100 but the relative deviation to the ground-truth value would be much higher than in the previous case.

This example shows the shortcoming of the mean squared error function as the loss function for the demand forecasting models. For this reason, I strongly recommend using mean absolute percentage error (MAPE).

The mean absolute percentage error, also known as mean absolute percentage deviation (MAPD) usually expresses accuracy as a percentage. We define it with the following equation:

In this equation, y_i is the predicted value and y_hat is the label. We divide the difference between y_i and y_hat by the actual value y_hat again. Finally, multiplying by 100 percent gives us the percentage error.

Applying this equation to the example above gives you a more meaningful understanding of the model’s performance. In the first case, the deviation from the ground truth label would be only one percent, while in the second case the deviation would be 66 percent:

We see that the performance of these two models is very different. Meanwhile, the MSE loss function would indicate that the performance of both models is the same.

The following Python code snipped shows how we can calculate MAPE:

ximport numpy as np

# The prediction vector of the neural network
y_pred=[0.6, 1.29, 1.99, 2.69, 3.4]

# The ground truth label
y_hat= [1, 1, 2, 2, 4]

MAPE = (100.0/len(y_hat))*np.sum(np.abs(y_pred-y_hat)/y_hat)

print(MAPE) # The result is mean percentage error of 23.79%

Best Practices for Loss Functions and Neural Networks

When you’re working with loss functions, just remember these key principles:

• A loss function measures how good a neural network model is in performing a certain task, which in most cases is regression or classification.

• We must minimize the value of the loss function during the backpropagation step in order to make the neural network better.

• We only use the cross-entropy loss function in classification tasks when we want the neural network to predict probabilities.

• For regression tasks, when we want the network to predict continuous numbers, we must use the mean squared error loss function.

• We use mean absolute percentage error loss function during demand forecasting to keep an eye on the performance of the network during training time.

In this post, you will be learning the difference between two common types of loss functions: Cross-Entropy Loss and Mean Squared Error (MSE) Loss. These are both used in machine learning for classification & regression tasks, respecitively, to measure how well a model performs on unseen dataset. Both these losses are ways of measuring how well the predictions are made by classification and regression algorithms, and they both provide different information about the performance of models. As a data scientist, it is very important for you to understand the difference between loss functions in a great manner.

What is cross-entropy loss?

Cross entropy loss is used in classification tasks where we are trying to minimize the probability of a negative class by maximizing an expected value of some function on our training data, also called as “loss function”. Simply speaking, it is used to measure the difference between two probabilities that a model assigns to classes. Cross entropy loss is also called as ‘softmax loss’ after the predefined function in neural networks. It is also used for multi-class classification problems. An example of the usage of cross-entropy loss for multi-class classification problems is training the model using MNIST dataset. The idea of this loss function is to give a high penalty for wrong predictions and a low penalty for correct classifications. It calculates a probability that each sample belongs to one of the classes, then it uses cross-entropy between these probabilities as its cost function. The more confident model is about prediction, the less penalty it incurs.

Cross-entropy loss is very similar to cross entropy. They both measure the difference between an actual probability and predicted probability, but cross entropy uses log probabilities while cross-entropy loss uses negative log probabilities (which are then multiplied by -log(p)) . Log probabilities can be converted into regular numbers for ease of computation using a softmax function.

Cross-entropy loss is also used in time series classification problems such as forecasting weather or stock values. An example would be comparing the forecast of a short term vs long term prediction model making cross-entropy loss incurring large penalty when one class has much higher probability.

A common example used to understand cross-entropy loss is comparing apples and oranges where each fruit has a certain probability of being chosen out of three probabilities (apple, orange, or other). Apple would have an 80% chance while Oranges will only get 20%. In order for our model to make correct predictions in this example, it should assign a high probability to apple and a low for orange. If cross-entropy loss is used, we can compute the cross-entropy loss for each fruit and assign probabilities accordingly. We will then want to choose apple with a higher probability as it has less cross entropy lost than oranges.

How do you calculate cross-entropy loss?

Cross-entropy loss is calculated by taking the difference between our prediction and actual output. We then multiply that value with `-y * ln(y)`. This means we take a negative number, raise it to the power of the logarithm of y (which will be positive), and then subtract this from our original calculation.

Here is the formula for calculating cross entropy loss:

The cross entropy loss is then calculated by using the cross-entropy formula and adding up all the losses: C = sum_{i=0}^{m} cross_entropy(X, Y)

What is mean squared error (MSE) loss?

Mean squared error (MSE) loss is used in regression tasks where we are trying to minimize an expected value of some function on our training data, which is also called as “loss function”.

How do you calculate mean squared error loss?

Mean squared error (MSE) loss is calculated by taking the difference between `y` and our prediction, then square those values. We take these new numbers (square them), add all of that together to get a final value, finally divide this number by y again. This will be our final result.

The formula for calculating mean squared error loss is as follows:

This will give us a loss value between 0 and infinity with larger values indicating mean squared error.

Root mean square error (RMSE) is a mean square error loss function that is normalized between 0 and infinity. The root mean squared error (RMSE) can be written as follows:

$$RMSE = sqrt{frac{ mean_squared_error}{m}}$$