Keras error functions

You’ve created a deep learning model in Keras, you prepared the data and now you are wondering which loss you should choose for your problem. 

We’ll get to that in a second but first what is a loss function?

In deep learning, the loss is computed to get the gradients with respect to model weights and update those weights accordingly via backpropagation. Loss is calculated and the network is updated after every iteration until model updates don’t bring any improvement in the desired evaluation metric. 

So while you keep using the same evaluation metric like f1 score or AUC on the validation set during (long parts) of your machine learning project, the loss can be changed, adjusted and modified to get the best evaluation metric performance.

You can think of the loss function just like you think about the model architecture or the optimizer and it is important to put some thought into choosing it. In this piece we’ll look at:

• loss functions available in Keras and how to use them,
• how you can define your own custom loss function in Keras,
• how to add sample weighing to create observation-sensitive losses,
• how to avoid nans in the loss,
• how you can monitor the loss function via plotting and callbacks.

Let’s get into it!

Keras loss functions 101

In Keras, loss functions are passed during the compile stage, as shown below. 

In this example, we’re defining the loss function by creating an instance of the loss class. Using the class is advantageous because you can pass some additional parameters. 

from tensorflow import keras
from tensorflow.keras import layers

model = keras.Sequential()
model.add(layers.Dense(64, kernel_initializer='uniform', input_shape=(10,)))
model.add(layers.Activation('softmax'))

loss_function = keras.losses.SparseCategoricalCrossentropy(from_logits=True)
model.compile(loss=loss_function, optimizer='adam')

If you want to use a loss function that is built into Keras without specifying any parameters you can just use the string alias as shown below:

model.compile(loss='sparse_categorical_crossentropy', optimizer='adam')

You might be wondering how does one decide on which loss function to use?

There are various loss functions available in Keras. Other times you might have to implement your own custom loss functions. 

Let’s dive into all those scenarios.

Which loss functions are available in Keras?

Binary Classification

Binary classification loss function comes into play when solving a problem involving just two classes. For example, when predicting fraud in credit card transactions, a transaction is either fraudulent or not. 

Binary Cross Entropy

The Binary Cross entropy will calculate the cross-entropy loss between the predicted classes and the true classes. By default, the sum_over_batch_size reduction is used. This means that the loss will return the average of the per-sample losses in the batch.

y_true = [[0., 1.], [0.2, 0.8],[0.3, 0.7],[0.4, 0.6]]
y_pred = [[0.6, 0.4], [0.4, 0.6],[0.6, 0.4],[0.8, 0.2]]
bce = tf.keras.losses.BinaryCrossentropy(reduction='sum_over_batch_size')
bce(y_true, y_pred).numpy()

The sum reduction means that the loss function will return the sum of the per-sample losses in the batch.

bce = tf.keras.losses.BinaryCrossentropy(reduction='sum')
bce(y_true, y_pred).numpy()

Using the reduction as none returns the full array of the per-sample losses.

bce = tf.keras.losses.BinaryCrossentropy(reduction='none')
bce(y_true, y_pred).numpy()
array([0.9162905 , 0.5919184 , 0.79465103, 1.0549198 ], dtype=float32)

In binary classification, the activation function used is the sigmoid activation function. It constrains the output to a number between 0 and 1. 

Multiclass classification

Problems involving the prediction of more than one class use different loss functions. In this section we’ll look at a couple:

Categorical Crossentropy

The CategoricalCrossentropy also computes the cross-entropy loss between the true classes and predicted classes. The labels are given in an one_hot format. 

cce = tf.keras.losses.CategoricalCrossentropy()
cce(y_true, y_pred).numpy()

Sparse Categorical Crossentropy

If you have two or more classes and  the labels are integers, the SparseCategoricalCrossentropy should be used. 

y_true = [0, 1,2]
y_pred = [[0.05, 0.95, 0], [0.1, 0.8, 0.1],[0.1, 0.8, 0.1]]
scce = tf.keras.losses.SparseCategoricalCrossentropy()
scce(y_true, y_pred).numpy()

The Poison Loss

You can also use the Poisson class to compute the poison loss. It’s a great choice if your dataset comes from a Poisson distribution for example the number of calls a call center receives per hour. 

y_true = [[0.1, 1.,0.8], [0.1, 0.9,0.1],[0.2, 0.7,0.1],[0.3, 0.1,0.6]]
y_pred = [[0.6, 0.2,0.2], [0.2, 0.6,0.2],[0.7, 0.1,0.2],[0.8, 0.1,0.1]]
p = tf.keras.losses.Poisson()
p(y_true, y_pred).numpy()

Kullback-Leibler Divergence Loss

The relative entropy can be computed using the KLDivergence class. According to the official docs at PyTorch:

KL divergence is a useful distance measure for continuous distributions and is often useful when performing direct regression over the space of (discretely sampled) continuous output distributions. 

y_true = [[0.1, 1.,0.8], [0.1, 0.9,0.1],[0.2, 0.7,0.1],[0.3, 0.1,0.6]]
y_pred = [[0.6, 0.2,0.2], [0.2, 0.6,0.2],[0.7, 0.1,0.2],[0.8, 0.1,0.1]]
kl = tf.keras.losses.KLDivergence()
kl(y_true, y_pred).numpy()

In a multi-class problem, the activation function used is the softmax function.

Object Detection

The Focal Loss

In classification problems involving imbalanced data and object detection problems, you can use the Focal Loss. The loss introduces an adjustment to the cross-entropy criterion. 

It is done by altering its shape in a way that the loss allocated to well-classified examples is down-weighted. This ensures that the model is able to learn equally from minority and majority classes.

The cross-entropy loss is scaled by scaling the factors decaying at zero as the confidence in the correct class increases. The factor of scaling down weights the contribution of unchallenging samples at training time and focuses on the challenging ones.

import tensorflow_addons as tfa

y_true = [[0.97], [0.91], [0.03]]
y_pred = [[1.0], [1.0], [0.0]]
sfc = tfa.losses.SigmoidFocalCrossEntropy()
sfc(y_true, y_pred).numpy()
array([0.00010971, 0.00329749, 0.00030611], dtype=float32)

Generalized Intersection over Union

The Generalized Intersection over Union loss from the TensorFlow add on can also be used. The Intersection over Union (IoU) is a very common metric in object detection problems. IoU is however not very efficient in problems involving non-overlapping bounding boxes. 

The Generalized Intersection over Union was introduced to address this challenge that IoU is facing. It ensures that generalization is achieved by maintaining the scale-invariant property of IoU, encoding the shape properties of the compared objects into the region property, and making sure that there is a strong correlation with IoU in the event of overlapping objects. 

gl = tfa.losses.GIoULoss()
boxes1 = tf.constant([[4.0, 3.0, 7.0, 5.0], [5.0, 6.0, 10.0, 7.0]])
boxes2 = tf.constant([[3.0, 4.0, 6.0, 8.0], [14.0, 14.0, 15.0, 15.0]])
loss = gl(boxes1, boxes2)

Regression

In regression problems, you have to calculate the differences between the predicted values and the true values but as always there are many ways to do it.

Mean Squared Error

The MeanSquaredError class can be used to compute the mean square of errors between the predictions and the true values. 

y_true = [12, 20, 29., 60.]
y_pred = [14., 18., 27., 55.]
mse = tf.keras.losses.MeanSquaredError()
mse(y_true, y_pred).numpy()

Use Mean Squared Error when you desire to have large errors penalized more than smaller ones. 

Mean Absolute Percentage Error

The mean absolute percentage error is computed using the function below.

It is calculated as shown below.

y_true = [12, 20, 29., 60.]
y_pred = [14., 18., 27., 55.]
mape = tf.keras.losses.MeanAbsolutePercentageError()
mape(y_true, y_pred).numpy()

Consider using this loss when you want a loss that you can explain intuitively. People understand percentages easily. The loss is also robust to outliers. 

Mean Squared Logarithmic Error

The mean squared logarithmic error can be computed using the formula below:

Here’s an implementation of the same:

y_true = [12, 20, 29., 60.]
y_pred = [14., 18., 27., 55.]
msle = tf.keras.losses.MeanSquaredLogarithmicError()
msle(y_true, y_pred).numpy()

Mean Squared Logarithmic Error penalizes underestimates more than it does overestimates. It’s a great choice when you prefer not to penalize large errors, it is, therefore, robust to outliers. 

Cosine Similarity Loss

If your interest is in computing the cosine similarity between the true and predicted values, you’d use the CosineSimilarity class. It is computed as:

The result is a number between  -1 and 1 . 0 indicates orthogonality while values close to -1 show that there is great similarity.

y_true = [[12, 20], [29., 60.]]
y_pred = [[14., 18.], [27., 55.]]
cosine_loss = tf.keras.losses.CosineSimilarity(axis=1)
cosine_loss(y_true, y_pred).numpy()

LogCosh Loss

The LogCosh class computes the logarithm of the hyperbolic cosine of the prediction error.

Here’s its implementation as a stand-alone function. 

y_true = [[12, 20], [29., 60.]]
y_pred = [[14., 18.], [27., 55.]]
l = tf.keras.losses.LogCosh()
l(y_true, y_pred).numpy()

LogCosh Loss works like the mean squared error, but will not be so strongly affected by the occasional wildly incorrect prediction. — TensorFlow Docs

Huber loss

For regression problems that are less sensitive to outliers, the Huber loss is used. 

y_true = [12, 20, 29., 60.]
y_pred = [14., 18., 27., 55.]
h = tf.keras.losses.Huber()
h(y_true, y_pred).numpy()

Learning Embeddings

Triplet Loss

You can also compute the triplet loss with semi-hard negative mining via TensorFlow addons. The loss encourages the positive distances between pairs of embeddings with the same labels to be less than the minimum negative distance. 

import tensorflow_addons as tfa

model.compile(optimizer='adam',
              loss=tfa.losses.TripletSemiHardLoss(),
              metrics=['accuracy'])

Creating custom loss functions in Keras

Sometimes there is no good loss available or you need to implement some modifications. Let’s learn how to do that.

A custom loss function can be created by defining a function that takes the true values and predicted values as required parameters. The function should return an array of losses. The function can then be passed at the compile stage. 

def custom_loss_function(y_true, y_pred):
   squared_difference = tf.square(y_true - y_pred)
   return tf.reduce_mean(squared_difference, axis=-1)

model.compile(optimizer='adam', loss=custom_loss_function)

Let’s see how we can apply this custom loss function to an array of predicted and true values.

import numpy as np

y_true = [12, 20, 29., 60.]
y_pred = [14., 18., 27., 55.]
cl = custom_loss_function(np.array(y_true),np.array(y_pred))
cl.numpy()

Use of Keras loss weights

During the training process, one can weigh the loss function by observations or samples. The weights can be arbitrary, but a typical choice is class weights (distribution of labels). Each observation is weighted by the fraction of the class it belongs to (reversed) so that the loss for minority class observations is more important when calculating the loss.  

One of the ways to do this is to pass the class weights during the training process. 

The weights are passed using a dictionary that contains the weight for each class. You can compute the weights using Scikit-learn or calculate the weights based on your own criterion. 

weights = { 0:1.01300017,1:0.88994364,2:1.00704935, 3:0.97863318,      4:1.02704553, 5:1.10680686,6:1.01385603,7:0.95770152, 8:1.02546573,
               9:1.00857287}
model.fit(x_train, y_train,verbose=1, epochs=10,class_weight=weights)

The second way is to pass these weights at the compile stage.

weights = [1.013, 0.889, 1.007, 0.978, 1.027,1.106,1.013,0.957,1.025, 1.008]

model.compile(optimizer=tf.keras.optimizers.SGD(),
              loss=tf.keras.losses.SparseCategoricalCrossentropy(),
              loss_weights=weights,
              metrics=['accuracy'])

How to monitor Keras loss function [example]

It is usually a good idea to monitor the loss function on the training and validation set as the model is training. Looking at those learning curves is a good indication of overfitting or other problems with model training.

There are two main options of how this can be done.

Monitor Keras loss using console logs 

The quickest and easiest way to log and look at the losses is simply printing them to the console. 

import tensorflow as tf

mnist = tf.keras.datasets.mnist
(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0

model = tf.keras.models.Sequential([
   tf.keras.layers.Flatten(input_shape=(28, 28)),
   tf.keras.layers.Dense(512, activation='relu'),
   tf.keras.layers.Dropout(0.2),
   tf.keras.layers.Dense(10, activation='softmax')
])

model.compile(optimizer='sgd',
             loss='sparse_categorical_crossentropy',
             metrics=['accuracy'])

model.fit(x_train, y_train,verbose=1, epochs=10)

The problem with this approach is that those logs can be easily lost, it is difficult to see progress, and when working on remote machines, you may not have access to it.

Monitor Keras loss using a callback

Another cleaner option is to use a callback that will log the loss somewhere on every batch and epoch ended. 

You need to decide where and what you would like to log, but it is really simple. 

For example, logging Keras loss to neptune.ai could look like this:

from keras.callbacks import Callback

class NeptuneCallback(Callback):
    def on_batch_end(self, batch, logs=None):
        for metric_name, metric_value in logs.items():
            neptune_run[f"{metric_name}"].log(metric_value)

    def on_epoch_end(self, epoch, logs=None):
        for metric_name, metric_value in logs.items():
            neptune_run[f"{metric_name}"].log(metric_value)

You can create the monitoring callback yourself or use one of the many available Keras callbacks both in the Keras library and in other libraries that integrate with it, like neptune.ai, TensorBoard, and others.

Once you have the callback ready, you simply pass it to the model.fit(...):

pip install neptune-tensorflow-keras

# the same as above
import neptune.new as neptune
from neptune.new.integrations.tensorflow_keras import NeptuneCallback
 
 
run = neptune.init_run()
 
neptune_callback = NeptuneCallback(run=run)
 
model.fit(
    x_train,
    y_train,
    validation_split=0.2,
    epochs=10,
    callbacks=[neptune_callback],
)

And monitor your experiment learning curves in the web app: 

Note: For the most up-to-date code examples, please refer to the Neptune-Keras integration docs.

Why Keras loss nan happens

Most of the time, losses you log will be just some regular values, but sometimes you might get nans when working with Keras loss functions. 

When that happens, your model will not update its weights and will stop learning, so this situation needs to be avoided.

There could be many reasons for nan loss but usually, what happens is:

• nans in the training set will lead to nans in the loss,
• NumPy infinite in the training set will also lead to nans in the loss,
• Using a training set that is not scaled,
• Use of very large l2 regularizers and a learning rate above 1,
• Use of the wrong optimizer function,
• Large (exploding) gradients that result in a large update to network weights during training.

So in order to avoid nans in the loss, ensure that:

• Check that your training data is properly scaled and doesn’t contain nans;
• Check that you are using the right optimizer and that your learning rate is not too large;
• Check whether the l2 regularization is not too large;
• If you are facing the exploding gradient problem, you can either: re-design the network or use gradient clipping so that your gradients have a certain “maximum allowed model update”.

Final thoughts

Hopefully, this article gave you some background into loss functions in Keras.

We’ve covered:

• Built-in loss functions in Keras,
• Implementation of your own custom loss functions,
• How to add sample weighing to create observation-sensitive losses,
• How to avoid loss nans,
• How you can visualize loss as your model is training.

For more information, check out the Keras Repository and the TensorFlow Loss Functions documentation.

Built-in loss functions.

Classes

class BinaryCrossentropy: Computes the cross-entropy loss between true labels and predicted labels.

class BinaryFocalCrossentropy: Computes the focal cross-entropy loss between true labels and predictions.

class CategoricalCrossentropy: Computes the crossentropy loss between the labels and predictions.

class CategoricalHinge: Computes the categorical hinge loss between y_true & y_pred.

class CosineSimilarity: Computes the cosine similarity between labels and predictions.

class Hinge: Computes the hinge loss between y_true & y_pred.

class Huber: Computes the Huber loss between y_true & y_pred.

class KLDivergence: Computes Kullback-Leibler divergence loss between y_true & y_pred.

class LogCosh: Computes the logarithm of the hyperbolic cosine of the prediction error.

class Loss: Loss base class.

class MeanAbsoluteError: Computes the mean of absolute difference between labels and predictions.

class MeanAbsolutePercentageError: Computes the mean absolute percentage error between y_true & y_pred.

class MeanSquaredError: Computes the mean of squares of errors between labels and predictions.

class MeanSquaredLogarithmicError: Computes the mean squared logarithmic error between y_true & y_pred.

class Poisson: Computes the Poisson loss between y_true & y_pred.

class Reduction: Types of loss reduction.

class SparseCategoricalCrossentropy: Computes the crossentropy loss between the labels and predictions.

class SquaredHinge: Computes the squared hinge loss between y_true & y_pred.

Functions

KLD(...): Computes Kullback-Leibler divergence loss between y_true & y_pred.

MAE(...): Computes the mean absolute error between labels and predictions.

MAPE(...): Computes the mean absolute percentage error between y_true & y_pred.

MSE(...): Computes the mean squared error between labels and predictions.

MSLE(...): Computes the mean squared logarithmic error between y_true & y_pred.

binary_crossentropy(...): Computes the binary crossentropy loss.

binary_focal_crossentropy(...): Computes the binary focal crossentropy loss.

categorical_crossentropy(...): Computes the categorical crossentropy loss.

categorical_hinge(...): Computes the categorical hinge loss between y_true & y_pred.

cosine_similarity(...): Computes the cosine similarity between labels and predictions.

deserialize(...): Deserializes a serialized loss class/function instance.

get(...): Retrieves a Keras loss as a function/Loss class instance.

hinge(...): Computes the hinge loss between y_true & y_pred.

huber(...): Computes Huber loss value.

kl_divergence(...): Computes Kullback-Leibler divergence loss between y_true & y_pred.

kld(...): Computes Kullback-Leibler divergence loss between y_true & y_pred.

kullback_leibler_divergence(...): Computes Kullback-Leibler divergence loss between y_true & y_pred.

log_cosh(...): Logarithm of the hyperbolic cosine of the prediction error.

logcosh(...): Logarithm of the hyperbolic cosine of the prediction error.

mae(...): Computes the mean absolute error between labels and predictions.

mape(...): Computes the mean absolute percentage error between y_true & y_pred.

mean_absolute_error(...): Computes the mean absolute error between labels and predictions.

mean_absolute_percentage_error(...): Computes the mean absolute percentage error between y_true & y_pred.

mean_squared_error(...): Computes the mean squared error between labels and predictions.

mean_squared_logarithmic_error(...): Computes the mean squared logarithmic error between y_true & y_pred.

mse(...): Computes the mean squared error between labels and predictions.

msle(...): Computes the mean squared logarithmic error between y_true & y_pred.

poisson(...): Computes the Poisson loss between y_true and y_pred.

serialize(...): Serializes loss function or Loss instance.

sparse_categorical_crossentropy(...): Computes the sparse categorical crossentropy loss.

squared_hinge(...): Computes the squared hinge loss between y_true & y_pred.

Setup

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

Introduction

This guide covers training, evaluation, and prediction (inference) models
when using built-in APIs for training & validation (such as Model.fit(),
Model.evaluate() and Model.predict()).

If you are interested in leveraging fit() while specifying your
own training step function, see the
Customizing what happens in fit() guide.

If you are interested in writing your own training & evaluation loops from
scratch, see the guide
«writing a training loop from scratch».

In general, whether you are using built-in loops or writing your own, model training &
evaluation works strictly in the same way across every kind of Keras model —
Sequential models, models built with the Functional API, and models written from
scratch via model subclassing.

This guide doesn’t cover distributed training, which is covered in our
guide to multi-GPU & distributed training.

API overview: a first end-to-end example

When passing data to the built-in training loops of a model, you should either use
NumPy arrays (if your data is small and fits in memory) or tf.data Dataset
objects. In the next few paragraphs, we’ll use the MNIST dataset as NumPy arrays, in
order to demonstrate how to use optimizers, losses, and metrics.

Let’s consider the following model (here, we build in with the Functional API, but it
could be a Sequential model or a subclassed model as well):

inputs = keras.Input(shape=(784,), name="digits")
x = layers.Dense(64, activation="relu", name="dense_1")(inputs)
x = layers.Dense(64, activation="relu", name="dense_2")(x)
outputs = layers.Dense(10, activation="softmax", name="predictions")(x)

model = keras.Model(inputs=inputs, outputs=outputs)

Here’s what the typical end-to-end workflow looks like, consisting of:

• Training
• Validation on a holdout set generated from the original training data
• Evaluation on the test data

We’ll use MNIST data for this example.

(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()

# Preprocess the data (these are NumPy arrays)
x_train = x_train.reshape(60000, 784).astype("float32") / 255
x_test = x_test.reshape(10000, 784).astype("float32") / 255

y_train = y_train.astype("float32")
y_test = y_test.astype("float32")

# Reserve 10,000 samples for validation
x_val = x_train[-10000:]
y_val = y_train[-10000:]
x_train = x_train[:-10000]
y_train = y_train[:-10000]

We specify the training configuration (optimizer, loss, metrics):

model.compile(
    optimizer=keras.optimizers.RMSprop(),  # Optimizer
    # Loss function to minimize
    loss=keras.losses.SparseCategoricalCrossentropy(),
    # List of metrics to monitor
    metrics=[keras.metrics.SparseCategoricalAccuracy()],
)

We call fit(), which will train the model by slicing the data into «batches» of size
batch_size, and repeatedly iterating over the entire dataset for a given number of
epochs.

print("Fit model on training data")
history = model.fit(
    x_train,
    y_train,
    batch_size=64,
    epochs=2,
    # We pass some validation for
    # monitoring validation loss and metrics
    # at the end of each epoch
    validation_data=(x_val, y_val),
)

Fit model on training data
Epoch 1/2
782/782 [==============================] - 3s 3ms/step - loss: 0.3387 - sparse_categorical_accuracy: 0.9050 - val_loss: 0.1957 - val_sparse_categorical_accuracy: 0.9426
Epoch 2/2
782/782 [==============================] - 2s 3ms/step - loss: 0.1543 - sparse_categorical_accuracy: 0.9548 - val_loss: 0.1425 - val_sparse_categorical_accuracy: 0.9593

The returned history object holds a record of the loss values and metric values
during training:

history.history

{'loss': [0.3386789858341217, 0.1543138176202774],
 'sparse_categorical_accuracy': [0.9050400257110596, 0.9548400044441223],
 'val_loss': [0.19569723308086395, 0.14253544807434082],
 'val_sparse_categorical_accuracy': [0.9426000118255615, 0.9592999815940857]}

We evaluate the model on the test data via evaluate():

# Evaluate the model on the test data using `evaluate`
print("Evaluate on test data")
results = model.evaluate(x_test, y_test, batch_size=128)
print("test loss, test acc:", results)

# Generate predictions (probabilities -- the output of the last layer)
# on new data using `predict`
print("Generate predictions for 3 samples")
predictions = model.predict(x_test[:3])
print("predictions shape:", predictions.shape)

Evaluate on test data
79/79 [==============================] - 0s 2ms/step - loss: 0.1414 - sparse_categorical_accuracy: 0.9569
test loss, test acc: [0.14140386879444122, 0.9569000005722046]
Generate predictions for 3 samples
predictions shape: (3, 10)

Now, let’s review each piece of this workflow in detail.

The compile() method: specifying a loss, metrics, and an optimizer

To train a model with fit(), you need to specify a loss function, an optimizer, and
optionally, some metrics to monitor.

You pass these to the model as arguments to the compile() method:

model.compile(
    optimizer=keras.optimizers.RMSprop(learning_rate=1e-3),
    loss=keras.losses.SparseCategoricalCrossentropy(),
    metrics=[keras.metrics.SparseCategoricalAccuracy()],
)

The metrics argument should be a list — your model can have any number of metrics.

If your model has multiple outputs, you can specify different losses and metrics for
each output, and you can modulate the contribution of each output to the total loss of
the model. You will find more details about this in the Passing data to multi-input,
multi-output models section.

Note that if you’re satisfied with the default settings, in many cases the optimizer,
loss, and metrics can be specified via string identifiers as a shortcut:

model.compile(
    optimizer="rmsprop",
    loss="sparse_categorical_crossentropy",
    metrics=["sparse_categorical_accuracy"],
)

For later reuse, let’s put our model definition and compile step in functions; we will
call them several times across different examples in this guide.

def get_uncompiled_model():
    inputs = keras.Input(shape=(784,), name="digits")
    x = layers.Dense(64, activation="relu", name="dense_1")(inputs)
    x = layers.Dense(64, activation="relu", name="dense_2")(x)
    outputs = layers.Dense(10, activation="softmax", name="predictions")(x)
    model = keras.Model(inputs=inputs, outputs=outputs)
    return model


def get_compiled_model():
    model = get_uncompiled_model()
    model.compile(
        optimizer="rmsprop",
        loss="sparse_categorical_crossentropy",
        metrics=["sparse_categorical_accuracy"],
    )
    return model

Many built-in optimizers, losses, and metrics are available

In general, you won’t have to create your own losses, metrics, or optimizers
from scratch, because what you need is likely to be already part of the Keras API:

Optimizers:

• SGD() (with or without momentum)
• RMSprop()
• Adam()
• etc.

Losses:

• MeanSquaredError()
• KLDivergence()
• CosineSimilarity()
• etc.

Metrics:

• AUC()
• Precision()
• Recall()
• etc.

Custom losses

If you need to create a custom loss, Keras provides two ways to do so.

The first method involves creating a function that accepts inputs y_true and
y_pred. The following example shows a loss function that computes the mean squared
error between the real data and the predictions:

def custom_mean_squared_error(y_true, y_pred):
    return tf.math.reduce_mean(tf.square(y_true - y_pred))


model = get_uncompiled_model()
model.compile(optimizer=keras.optimizers.Adam(), loss=custom_mean_squared_error)

# We need to one-hot encode the labels to use MSE
y_train_one_hot = tf.one_hot(y_train, depth=10)
model.fit(x_train, y_train_one_hot, batch_size=64, epochs=1)

782/782 [==============================] - 2s 2ms/step - loss: 0.0162
<keras.callbacks.History at 0x7ff8881ba250>

If you need a loss function that takes in parameters beside y_true and y_pred, you
can subclass the tf.keras.losses.Loss class and implement the following two methods:

• __init__(self): accept parameters to pass during the call of your loss function
• call(self, y_true, y_pred): use the targets (y_true) and the model predictions
  (y_pred) to compute the model’s loss

Let’s say you want to use mean squared error, but with an added term that
will de-incentivize prediction values far from 0.5 (we assume that the categorical
targets are one-hot encoded and take values between 0 and 1). This
creates an incentive for the model not to be too confident, which may help
reduce overfitting (we won’t know if it works until we try!).

Here’s how you would do it:

class CustomMSE(keras.losses.Loss):
    def __init__(self, regularization_factor=0.1, name="custom_mse"):
        super().__init__(name=name)
        self.regularization_factor = regularization_factor

    def call(self, y_true, y_pred):
        mse = tf.math.reduce_mean(tf.square(y_true - y_pred))
        reg = tf.math.reduce_mean(tf.square(0.5 - y_pred))
        return mse + reg * self.regularization_factor


model = get_uncompiled_model()
model.compile(optimizer=keras.optimizers.Adam(), loss=CustomMSE())

y_train_one_hot = tf.one_hot(y_train, depth=10)
model.fit(x_train, y_train_one_hot, batch_size=64, epochs=1)

782/782 [==============================] - 2s 2ms/step - loss: 0.0388
<keras.callbacks.History at 0x7ff8882130d0>

Custom metrics

If you need a metric that isn’t part of the API, you can easily create custom metrics
by subclassing the tf.keras.metrics.Metric class. You will need to implement 4
methods:

• __init__(self), in which you will create state variables for your metric.
• update_state(self, y_true, y_pred, sample_weight=None), which uses the targets
  y_true and the model predictions y_pred to update the state variables.
• result(self), which uses the state variables to compute the final results.
• reset_state(self), which reinitializes the state of the metric.

State update and results computation are kept separate (in update_state() and
result(), respectively) because in some cases, the results computation might be very
expensive and would only be done periodically.

Here’s a simple example showing how to implement a CategoricalTruePositives metric
that counts how many samples were correctly classified as belonging to a given class:

class CategoricalTruePositives(keras.metrics.Metric):
    def __init__(self, name="categorical_true_positives", **kwargs):
        super(CategoricalTruePositives, self).__init__(name=name, **kwargs)
        self.true_positives = self.add_weight(name="ctp", initializer="zeros")

    def update_state(self, y_true, y_pred, sample_weight=None):
        y_pred = tf.reshape(tf.argmax(y_pred, axis=1), shape=(-1, 1))
        values = tf.cast(y_true, "int32") == tf.cast(y_pred, "int32")
        values = tf.cast(values, "float32")
        if sample_weight is not None:
            sample_weight = tf.cast(sample_weight, "float32")
            values = tf.multiply(values, sample_weight)
        self.true_positives.assign_add(tf.reduce_sum(values))

    def result(self):
        return self.true_positives

    def reset_state(self):
        # The state of the metric will be reset at the start of each epoch.
        self.true_positives.assign(0.0)


model = get_uncompiled_model()
model.compile(
    optimizer=keras.optimizers.RMSprop(learning_rate=1e-3),
    loss=keras.losses.SparseCategoricalCrossentropy(),
    metrics=[CategoricalTruePositives()],
)
model.fit(x_train, y_train, batch_size=64, epochs=3)

Epoch 1/3
782/782 [==============================] - 2s 3ms/step - loss: 0.3404 - categorical_true_positives: 45217.0000
Epoch 2/3
782/782 [==============================] - 2s 3ms/step - loss: 0.1588 - categorical_true_positives: 47606.0000
Epoch 3/3
782/782 [==============================] - 2s 3ms/step - loss: 0.1168 - categorical_true_positives: 48278.0000
<keras.callbacks.History at 0x7ff8880a3610>

Handling losses and metrics that don’t fit the standard signature

The overwhelming majority of losses and metrics can be computed from y_true and
y_pred, where y_pred is an output of your model — but not all of them. For
instance, a regularization loss may only require the activation of a layer (there are
no targets in this case), and this activation may not be a model output.

In such cases, you can call self.add_loss(loss_value) from inside the call method of
a custom layer. Losses added in this way get added to the «main» loss during training
(the one passed to compile()). Here’s a simple example that adds activity
regularization (note that activity regularization is built-in in all Keras layers —
this layer is just for the sake of providing a concrete example):

class ActivityRegularizationLayer(layers.Layer):
    def call(self, inputs):
        self.add_loss(tf.reduce_sum(inputs) * 0.1)
        return inputs  # Pass-through layer.


inputs = keras.Input(shape=(784,), name="digits")
x = layers.Dense(64, activation="relu", name="dense_1")(inputs)

# Insert activity regularization as a layer
x = ActivityRegularizationLayer()(x)

x = layers.Dense(64, activation="relu", name="dense_2")(x)
outputs = layers.Dense(10, name="predictions")(x)

model = keras.Model(inputs=inputs, outputs=outputs)
model.compile(
    optimizer=keras.optimizers.RMSprop(learning_rate=1e-3),
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
)

# The displayed loss will be much higher than before
# due to the regularization component.
model.fit(x_train, y_train, batch_size=64, epochs=1)

782/782 [==============================] - 2s 2ms/step - loss: 2.4545
<keras.callbacks.History at 0x7ff87c53f310>

You can do the same for logging metric values, using add_metric():

class MetricLoggingLayer(layers.Layer):
    def call(self, inputs):
        # The `aggregation` argument defines
        # how to aggregate the per-batch values
        # over each epoch:
        # in this case we simply average them.
        self.add_metric(
            keras.backend.std(inputs), name="std_of_activation", aggregation="mean"
        )
        return inputs  # Pass-through layer.


inputs = keras.Input(shape=(784,), name="digits")
x = layers.Dense(64, activation="relu", name="dense_1")(inputs)

# Insert std logging as a layer.
x = MetricLoggingLayer()(x)

x = layers.Dense(64, activation="relu", name="dense_2")(x)
outputs = layers.Dense(10, name="predictions")(x)

model = keras.Model(inputs=inputs, outputs=outputs)
model.compile(
    optimizer=keras.optimizers.RMSprop(learning_rate=1e-3),
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
)
model.fit(x_train, y_train, batch_size=64, epochs=1)

782/782 [==============================] - 2s 2ms/step - loss: 0.3461 - std_of_activation: 0.9929
<keras.callbacks.History at 0x7ff87c3d5bd0>

In the Functional API,
you can also call model.add_loss(loss_tensor),
or model.add_metric(metric_tensor, name, aggregation).

Here’s a simple example:

inputs = keras.Input(shape=(784,), name="digits")
x1 = layers.Dense(64, activation="relu", name="dense_1")(inputs)
x2 = layers.Dense(64, activation="relu", name="dense_2")(x1)
outputs = layers.Dense(10, name="predictions")(x2)
model = keras.Model(inputs=inputs, outputs=outputs)

model.add_loss(tf.reduce_sum(x1) * 0.1)

model.add_metric(keras.backend.std(x1), name="std_of_activation", aggregation="mean")

model.compile(
    optimizer=keras.optimizers.RMSprop(1e-3),
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
)
model.fit(x_train, y_train, batch_size=64, epochs=1)

782/782 [==============================] - 2s 3ms/step - loss: 2.4647 - std_of_activation: 0.0017
<keras.callbacks.History at 0x7ff87c216f90>

Note that when you pass losses via add_loss(), it becomes possible to call
compile() without a loss function, since the model already has a loss to minimize.

Consider the following LogisticEndpoint layer: it takes as inputs
targets & logits, and it tracks a crossentropy loss via add_loss(). It also
tracks classification accuracy via add_metric().

class LogisticEndpoint(keras.layers.Layer):
    def __init__(self, name=None):
        super(LogisticEndpoint, self).__init__(name=name)
        self.loss_fn = keras.losses.BinaryCrossentropy(from_logits=True)
        self.accuracy_fn = keras.metrics.BinaryAccuracy()

    def call(self, targets, logits, sample_weights=None):
        # Compute the training-time loss value and add it
        # to the layer using `self.add_loss()`.
        loss = self.loss_fn(targets, logits, sample_weights)
        self.add_loss(loss)

        # Log accuracy as a metric and add it
        # to the layer using `self.add_metric()`.
        acc = self.accuracy_fn(targets, logits, sample_weights)
        self.add_metric(acc, name="accuracy")

        # Return the inference-time prediction tensor (for `.predict()`).
        return tf.nn.softmax(logits)

You can use it in a model with two inputs (input data & targets), compiled without a
loss argument, like this:

import numpy as np

inputs = keras.Input(shape=(3,), name="inputs")
targets = keras.Input(shape=(10,), name="targets")
logits = keras.layers.Dense(10)(inputs)
predictions = LogisticEndpoint(name="predictions")(logits, targets)

model = keras.Model(inputs=[inputs, targets], outputs=predictions)
model.compile(optimizer="adam")  # No loss argument!

data = {
    "inputs": np.random.random((3, 3)),
    "targets": np.random.random((3, 10)),
}
model.fit(data)

1/1 [==============================] - 0s 414ms/step - loss: 0.9889 - binary_accuracy: 0.0000e+00
<keras.callbacks.History at 0x7ff87c0848d0>

For more information about training multi-input models, see the section Passing data
to multi-input, multi-output models.

Automatically setting apart a validation holdout set

In the first end-to-end example you saw, we used the validation_data argument to pass
a tuple of NumPy arrays (x_val, y_val) to the model for evaluating a validation loss
and validation metrics at the end of each epoch.

Here’s another option: the argument validation_split allows you to automatically
reserve part of your training data for validation. The argument value represents the
fraction of the data to be reserved for validation, so it should be set to a number
higher than 0 and lower than 1. For instance, validation_split=0.2 means «use 20% of
the data for validation», and validation_split=0.6 means «use 60% of the data for
validation».

The way the validation is computed is by taking the last x% samples of the arrays
received by the fit() call, before any shuffling.

Note that you can only use validation_split when training with NumPy data.

model = get_compiled_model()
model.fit(x_train, y_train, batch_size=64, validation_split=0.2, epochs=1)

625/625 [==============================] - 2s 3ms/step - loss: 0.3682 - sparse_categorical_accuracy: 0.8957 - val_loss: 0.2276 - val_sparse_categorical_accuracy: 0.9301
<keras.callbacks.History at 0x7ff81c680890>

Training & evaluation from tf.data Datasets

In the past few paragraphs, you’ve seen how to handle losses, metrics, and optimizers,
and you’ve seen how to use the validation_data and validation_split arguments in
fit(), when your data is passed as NumPy arrays.

Let’s now take a look at the case where your data comes in the form of a
tf.data.Dataset object.

The tf.data API is a set of utilities in TensorFlow 2.0 for loading and preprocessing
data in a way that’s fast and scalable.

For a complete guide about creating Datasets, see the
tf.data documentation.

You can pass a Dataset instance directly to the methods fit(), evaluate(), and
predict():

model = get_compiled_model()

# First, let's create a training Dataset instance.
# For the sake of our example, we'll use the same MNIST data as before.
train_dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train))
# Shuffle and slice the dataset.
train_dataset = train_dataset.shuffle(buffer_size=1024).batch(64)

# Now we get a test dataset.
test_dataset = tf.data.Dataset.from_tensor_slices((x_test, y_test))
test_dataset = test_dataset.batch(64)

# Since the dataset already takes care of batching,
# we don't pass a `batch_size` argument.
model.fit(train_dataset, epochs=3)

# You can also evaluate or predict on a dataset.
print("Evaluate")
result = model.evaluate(test_dataset)
dict(zip(model.metrics_names, result))

Epoch 1/3
782/782 [==============================] - 2s 3ms/step - loss: 0.3372 - sparse_categorical_accuracy: 0.9047
Epoch 2/3
782/782 [==============================] - 2s 3ms/step - loss: 0.1596 - sparse_categorical_accuracy: 0.9523
Epoch 3/3
782/782 [==============================] - 2s 3ms/step - loss: 0.1171 - sparse_categorical_accuracy: 0.9655
Evaluate
157/157 [==============================] - 0s 2ms/step - loss: 0.1211 - sparse_categorical_accuracy: 0.9648
{'loss': 0.12107347697019577,
 'sparse_categorical_accuracy': 0.9648000001907349}

Note that the Dataset is reset at the end of each epoch, so it can be reused of the
next epoch.

If you want to run training only on a specific number of batches from this Dataset, you
can pass the steps_per_epoch argument, which specifies how many training steps the
model should run using this Dataset before moving on to the next epoch.

If you do this, the dataset is not reset at the end of each epoch, instead we just keep
drawing the next batches. The dataset will eventually run out of data (unless it is an
infinitely-looping dataset).

model = get_compiled_model()

# Prepare the training dataset
train_dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train))
train_dataset = train_dataset.shuffle(buffer_size=1024).batch(64)

# Only use the 100 batches per epoch (that's 64 * 100 samples)
model.fit(train_dataset, epochs=3, steps_per_epoch=100)

Epoch 1/3
100/100 [==============================] - 1s 3ms/step - loss: 0.7937 - sparse_categorical_accuracy: 0.7894
Epoch 2/3
100/100 [==============================] - 0s 3ms/step - loss: 0.3699 - sparse_categorical_accuracy: 0.8938
Epoch 3/3
100/100 [==============================] - 0s 3ms/step - loss: 0.3155 - sparse_categorical_accuracy: 0.9061
<keras.callbacks.History at 0x7ff81c587e90>

Using a validation dataset

You can pass a Dataset instance as the validation_data argument in fit():

model = get_compiled_model()

# Prepare the training dataset
train_dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train))
train_dataset = train_dataset.shuffle(buffer_size=1024).batch(64)

# Prepare the validation dataset
val_dataset = tf.data.Dataset.from_tensor_slices((x_val, y_val))
val_dataset = val_dataset.batch(64)

model.fit(train_dataset, epochs=1, validation_data=val_dataset)

782/782 [==============================] - 3s 3ms/step - loss: 0.3380 - sparse_categorical_accuracy: 0.9035 - val_loss: 0.2015 - val_sparse_categorical_accuracy: 0.9405
<keras.callbacks.History at 0x7ff81c30e450>

At the end of each epoch, the model will iterate over the validation dataset and
compute the validation loss and validation metrics.

If you want to run validation only on a specific number of batches from this dataset,
you can pass the validation_steps argument, which specifies how many validation
steps the model should run with the validation dataset before interrupting validation
and moving on to the next epoch:

model = get_compiled_model()

# Prepare the training dataset
train_dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train))
train_dataset = train_dataset.shuffle(buffer_size=1024).batch(64)

# Prepare the validation dataset
val_dataset = tf.data.Dataset.from_tensor_slices((x_val, y_val))
val_dataset = val_dataset.batch(64)

model.fit(
    train_dataset,
    epochs=1,
    # Only run validation using the first 10 batches of the dataset
    # using the `validation_steps` argument
    validation_data=val_dataset,
    validation_steps=10,
)

782/782 [==============================] - 3s 3ms/step - loss: 0.3369 - sparse_categorical_accuracy: 0.9036 - val_loss: 0.2953 - val_sparse_categorical_accuracy: 0.9187
<keras.callbacks.History at 0x7ff81c30e310>

Note that the validation dataset will be reset after each use (so that you will always
be evaluating on the same samples from epoch to epoch).

The argument validation_split (generating a holdout set from the training data) is
not supported when training from Dataset objects, since this feature requires the
ability to index the samples of the datasets, which is not possible in general with
the Dataset API.

Other input formats supported

Besides NumPy arrays, eager tensors, and TensorFlow Datasets, it’s possible to train
a Keras model using Pandas dataframes, or from Python generators that yield batches of
data & labels.

In particular, the keras.utils.Sequence class offers a simple interface to build
Python data generators that are multiprocessing-aware and can be shuffled.

In general, we recommend that you use:

• NumPy input data if your data is small and fits in memory
• Dataset objects if you have large datasets and you need to do distributed training
• Sequence objects if you have large datasets and you need to do a lot of custom
  Python-side processing that cannot be done in TensorFlow (e.g. if you rely on external libraries
  for data loading or preprocessing).

Using a keras.utils.Sequence object as input

keras.utils.Sequence is a utility that you can subclass to obtain a Python generator with
two important properties:

• It works well with multiprocessing.
• It can be shuffled (e.g. when passing shuffle=True in fit()).

A Sequence must implement two methods:

• __getitem__
• __len__

The method __getitem__ should return a complete batch.
If you want to modify your dataset between epochs, you may implement on_epoch_end.

Here’s a quick example:

from skimage.io import imread
from skimage.transform import resize
import numpy as np

# Here, `filenames` is list of path to the images
# and `labels` are the associated labels.

class CIFAR10Sequence(Sequence):
    def __init__(self, filenames, labels, batch_size):
        self.filenames, self.labels = filenames, labels
        self.batch_size = batch_size

    def __len__(self):
        return int(np.ceil(len(self.filenames) / float(self.batch_size)))

    def __getitem__(self, idx):
        batch_x = self.filenames[idx * self.batch_size:(idx + 1) * self.batch_size]
        batch_y = self.labels[idx * self.batch_size:(idx + 1) * self.batch_size]
        return np.array([
            resize(imread(filename), (200, 200))
               for filename in batch_x]), np.array(batch_y)

sequence = CIFAR10Sequence(filenames, labels, batch_size)
model.fit(sequence, epochs=10)

Using sample weighting and class weighting

With the default settings the weight of a sample is decided by its frequency
in the dataset. There are two methods to weight the data, independent of
sample frequency:

• Class weights
• Sample weights

Class weights

This is set by passing a dictionary to the class_weight argument to
Model.fit(). This dictionary maps class indices to the weight that should
be used for samples belonging to this class.

This can be used to balance classes without resampling, or to train a
model that gives more importance to a particular class.

For instance, if class «0» is half as represented as class «1» in your data,
you could use Model.fit(..., class_weight={0: 1., 1: 0.5}).

Here’s a NumPy example where we use class weights or sample weights to
give more importance to the correct classification of class #5 (which
is the digit «5» in the MNIST dataset).

import numpy as np

class_weight = {
    0: 1.0,
    1: 1.0,
    2: 1.0,
    3: 1.0,
    4: 1.0,
    # Set weight "2" for class "5",
    # making this class 2x more important
    5: 2.0,
    6: 1.0,
    7: 1.0,
    8: 1.0,
    9: 1.0,
}

print("Fit with class weight")
model = get_compiled_model()
model.fit(x_train, y_train, class_weight=class_weight, batch_size=64, epochs=1)

Fit with class weight
782/782 [==============================] - 2s 3ms/step - loss: 0.3708 - sparse_categorical_accuracy: 0.9032
<keras.callbacks.History at 0x7ff80c7ddd10>

Sample weights

For fine grained control, or if you are not building a classifier,
you can use «sample weights».

• When training from NumPy data: Pass the sample_weight
  argument to Model.fit().
• When training from tf.data or any other sort of iterator:
  Yield (input_batch, label_batch, sample_weight_batch) tuples.

A «sample weights» array is an array of numbers that specify how much weight
each sample in a batch should have in computing the total loss. It is commonly
used in imbalanced classification problems (the idea being to give more weight
to rarely-seen classes).

When the weights used are ones and zeros, the array can be used as a mask for
the loss function (entirely discarding the contribution of certain samples to
the total loss).

sample_weight = np.ones(shape=(len(y_train),))
sample_weight[y_train == 5] = 2.0

print("Fit with sample weight")
model = get_compiled_model()
model.fit(x_train, y_train, sample_weight=sample_weight, batch_size=64, epochs=1)

Fit with sample weight
782/782 [==============================] - 2s 3ms/step - loss: 0.3806 - sparse_categorical_accuracy: 0.9000
<keras.callbacks.History at 0x7ff80c650350>

Here’s a matching Dataset example:

sample_weight = np.ones(shape=(len(y_train),))
sample_weight[y_train == 5] = 2.0

# Create a Dataset that includes sample weights
# (3rd element in the return tuple).
train_dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train, sample_weight))

# Shuffle and slice the dataset.
train_dataset = train_dataset.shuffle(buffer_size=1024).batch(64)

model = get_compiled_model()
model.fit(train_dataset, epochs=1)

782/782 [==============================] - 3s 3ms/step - loss: 0.3588 - sparse_categorical_accuracy: 0.9070
<keras.callbacks.History at 0x7ff80c51cb50>

Passing data to multi-input, multi-output models

In the previous examples, we were considering a model with a single input (a tensor of
shape (764,)) and a single output (a prediction tensor of shape (10,)). But what
about models that have multiple inputs or outputs?

Consider the following model, which has an image input of shape (32, 32, 3) (that’s
(height, width, channels)) and a time series input of shape (None, 10) (that’s
(timesteps, features)). Our model will have two outputs computed from the
combination of these inputs: a «score» (of shape (1,)) and a probability
distribution over five classes (of shape (5,)).

image_input = keras.Input(shape=(32, 32, 3), name="img_input")
timeseries_input = keras.Input(shape=(None, 10), name="ts_input")

x1 = layers.Conv2D(3, 3)(image_input)
x1 = layers.GlobalMaxPooling2D()(x1)

x2 = layers.Conv1D(3, 3)(timeseries_input)
x2 = layers.GlobalMaxPooling1D()(x2)

x = layers.concatenate([x1, x2])

score_output = layers.Dense(1, name="score_output")(x)
class_output = layers.Dense(5, name="class_output")(x)

model = keras.Model(
    inputs=[image_input, timeseries_input], outputs=[score_output, class_output]
)

Let’s plot this model, so you can clearly see what we’re doing here (note that the
shapes shown in the plot are batch shapes, rather than per-sample shapes).

keras.utils.plot_model(model, "multi_input_and_output_model.png", show_shapes=True)

At compilation time, we can specify different losses to different outputs, by passing
the loss functions as a list:

model.compile(
    optimizer=keras.optimizers.RMSprop(1e-3),
    loss=[keras.losses.MeanSquaredError(), keras.losses.CategoricalCrossentropy()],
)

If we only passed a single loss function to the model, the same loss function would be
applied to every output (which is not appropriate here).

Likewise for metrics:

model.compile(
    optimizer=keras.optimizers.RMSprop(1e-3),
    loss=[keras.losses.MeanSquaredError(), keras.losses.CategoricalCrossentropy()],
    metrics=[
        [
            keras.metrics.MeanAbsolutePercentageError(),
            keras.metrics.MeanAbsoluteError(),
        ],
        [keras.metrics.CategoricalAccuracy()],
    ],
)

Since we gave names to our output layers, we could also specify per-output losses and
metrics via a dict:

model.compile(
    optimizer=keras.optimizers.RMSprop(1e-3),
    loss={
        "score_output": keras.losses.MeanSquaredError(),
        "class_output": keras.losses.CategoricalCrossentropy(),
    },
    metrics={
        "score_output": [
            keras.metrics.MeanAbsolutePercentageError(),
            keras.metrics.MeanAbsoluteError(),
        ],
        "class_output": [keras.metrics.CategoricalAccuracy()],
    },
)

We recommend the use of explicit names and dicts if you have more than 2 outputs.

It’s possible to give different weights to different output-specific losses (for
instance, one might wish to privilege the «score» loss in our example, by giving to 2x
the importance of the class loss), using the loss_weights argument:

model.compile(
    optimizer=keras.optimizers.RMSprop(1e-3),
    loss={
        "score_output": keras.losses.MeanSquaredError(),
        "class_output": keras.losses.CategoricalCrossentropy(),
    },
    metrics={
        "score_output": [
            keras.metrics.MeanAbsolutePercentageError(),
            keras.metrics.MeanAbsoluteError(),
        ],
        "class_output": [keras.metrics.CategoricalAccuracy()],
    },
    loss_weights={"score_output": 2.0, "class_output": 1.0},
)

You could also choose not to compute a loss for certain outputs, if these outputs are
meant for prediction but not for training:

# List loss version
model.compile(
    optimizer=keras.optimizers.RMSprop(1e-3),
    loss=[None, keras.losses.CategoricalCrossentropy()],
)

# Or dict loss version
model.compile(
    optimizer=keras.optimizers.RMSprop(1e-3),
    loss={"class_output": keras.losses.CategoricalCrossentropy()},
)

Passing data to a multi-input or multi-output model in fit() works in a similar way as
specifying a loss function in compile: you can pass lists of NumPy arrays (with
1:1 mapping to the outputs that received a loss function) or dicts mapping output
names to NumPy arrays.

model.compile(
    optimizer=keras.optimizers.RMSprop(1e-3),
    loss=[keras.losses.MeanSquaredError(), keras.losses.CategoricalCrossentropy()],
)

# Generate dummy NumPy data
img_data = np.random.random_sample(size=(100, 32, 32, 3))
ts_data = np.random.random_sample(size=(100, 20, 10))
score_targets = np.random.random_sample(size=(100, 1))
class_targets = np.random.random_sample(size=(100, 5))

# Fit on lists
model.fit([img_data, ts_data], [score_targets, class_targets], batch_size=32, epochs=1)

# Alternatively, fit on dicts
model.fit(
    {"img_input": img_data, "ts_input": ts_data},
    {"score_output": score_targets, "class_output": class_targets},
    batch_size=32,
    epochs=1,
)

4/4 [==============================] - 2s 9ms/step - loss: 5.6917 - score_output_loss: 0.1031 - class_output_loss: 5.5886
4/4 [==============================] - 0s 6ms/step - loss: 4.4108 - score_output_loss: 0.0999 - class_output_loss: 4.3109
<keras.callbacks.History at 0x7ff80c3b4110>

Here’s the Dataset use case: similarly as what we did for NumPy arrays, the Dataset
should return a tuple of dicts.

train_dataset = tf.data.Dataset.from_tensor_slices(
    (
        {"img_input": img_data, "ts_input": ts_data},
        {"score_output": score_targets, "class_output": class_targets},
    )
)
train_dataset = train_dataset.shuffle(buffer_size=1024).batch(64)

model.fit(train_dataset, epochs=1)

2/2 [==============================] - 0s 21ms/step - loss: 4.2451 - score_output_loss: 0.0993 - class_output_loss: 4.1458
<keras.callbacks.History at 0x7ff80c3ed450>

Using callbacks

Callbacks in Keras are objects that are called at different points during training (at
the start of an epoch, at the end of a batch, at the end of an epoch, etc.). They
can be used to implement certain behaviors, such as:

• Doing validation at different points during training (beyond the built-in per-epoch
  validation)
• Checkpointing the model at regular intervals or when it exceeds a certain accuracy
  threshold
• Changing the learning rate of the model when training seems to be plateauing
• Doing fine-tuning of the top layers when training seems to be plateauing
• Sending email or instant message notifications when training ends or where a certain
  performance threshold is exceeded
• Etc.

Callbacks can be passed as a list to your call to fit():

model = get_compiled_model()

callbacks = [
    keras.callbacks.EarlyStopping(
        # Stop training when `val_loss` is no longer improving
        monitor="val_loss",
        # "no longer improving" being defined as "no better than 1e-2 less"
        min_delta=1e-2,
        # "no longer improving" being further defined as "for at least 2 epochs"
        patience=2,
        verbose=1,
    )
]
model.fit(
    x_train,
    y_train,
    epochs=20,
    batch_size=64,
    callbacks=callbacks,
    validation_split=0.2,
)

Epoch 1/20
625/625 [==============================] - 2s 3ms/step - loss: 0.3725 - sparse_categorical_accuracy: 0.8939 - val_loss: 0.2314 - val_sparse_categorical_accuracy: 0.9321
Epoch 2/20
625/625 [==============================] - 2s 3ms/step - loss: 0.1805 - sparse_categorical_accuracy: 0.9471 - val_loss: 0.2012 - val_sparse_categorical_accuracy: 0.9379
Epoch 3/20
625/625 [==============================] - 2s 3ms/step - loss: 0.1346 - sparse_categorical_accuracy: 0.9603 - val_loss: 0.1651 - val_sparse_categorical_accuracy: 0.9505
Epoch 4/20
625/625 [==============================] - 2s 3ms/step - loss: 0.1065 - sparse_categorical_accuracy: 0.9684 - val_loss: 0.1510 - val_sparse_categorical_accuracy: 0.9571
Epoch 5/20
625/625 [==============================] - 2s 3ms/step - loss: 0.0884 - sparse_categorical_accuracy: 0.9734 - val_loss: 0.1505 - val_sparse_categorical_accuracy: 0.9538
Epoch 6/20
625/625 [==============================] - 2s 3ms/step - loss: 0.0746 - sparse_categorical_accuracy: 0.9778 - val_loss: 0.1508 - val_sparse_categorical_accuracy: 0.9575
Epoch 00006: early stopping
<keras.callbacks.History at 0x7ff80c64cad0>

Many built-in callbacks are available

There are many built-in callbacks already available in Keras, such as:

• ModelCheckpoint: Periodically save the model.
• EarlyStopping: Stop training when training is no longer improving the validation
  metrics.
• TensorBoard: periodically write model logs that can be visualized in
  TensorBoard (more details in the section
  «Visualization»).
• CSVLogger: streams loss and metrics data to a CSV file.
• etc.

See the callbacks documentation for the complete list.

Writing your own callback

You can create a custom callback by extending the base class
keras.callbacks.Callback. A callback has access to its associated model through the
class property self.model.

Make sure to read the
complete guide to writing custom callbacks.

Here’s a simple example saving a list of per-batch loss values during training:

class LossHistory(keras.callbacks.Callback):
    def on_train_begin(self, logs):
        self.per_batch_losses = []

    def on_batch_end(self, batch, logs):
        self.per_batch_losses.append(logs.get("loss"))

Checkpointing models

When you’re training model on relatively large datasets, it’s crucial to save
checkpoints of your model at frequent intervals.

The easiest way to achieve this is with the ModelCheckpoint callback:

model = get_compiled_model()

callbacks = [
    keras.callbacks.ModelCheckpoint(
        # Path where to save the model
        # The two parameters below mean that we will overwrite
        # the current checkpoint if and only if
        # the `val_loss` score has improved.
        # The saved model name will include the current epoch.
        filepath="mymodel_{epoch}",
        save_best_only=True,  # Only save a model if `val_loss` has improved.
        monitor="val_loss",
        verbose=1,
    )
]
model.fit(
    x_train, y_train, epochs=2, batch_size=64, callbacks=callbacks, validation_split=0.2
)

Epoch 1/2
613/625 [============================>.] - ETA: 0s - loss: 0.3693 - sparse_categorical_accuracy: 0.8972
Epoch 00001: val_loss improved from inf to 0.23508, saving model to mymodel_1
2021-11-12 20:11:50.182298: W tensorflow/python/util/util.cc:368] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
INFO:tensorflow:Assets written to: mymodel_1/assets
625/625 [==============================] - 3s 4ms/step - loss: 0.3660 - sparse_categorical_accuracy: 0.8979 - val_loss: 0.2351 - val_sparse_categorical_accuracy: 0.9273
Epoch 2/2
620/625 [============================>.] - ETA: 0s - loss: 0.1659 - sparse_categorical_accuracy: 0.9507
Epoch 00002: val_loss improved from 0.23508 to 0.16898, saving model to mymodel_2
INFO:tensorflow:Assets written to: mymodel_2/assets
625/625 [==============================] - 2s 3ms/step - loss: 0.1657 - sparse_categorical_accuracy: 0.9507 - val_loss: 0.1690 - val_sparse_categorical_accuracy: 0.9482
<keras.callbacks.History at 0x7ff8b577cc90>

The ModelCheckpoint callback can be used to implement fault-tolerance:
the ability to restart training from the last saved state of the model in case training
gets randomly interrupted. Here’s a basic example:

import os

# Prepare a directory to store all the checkpoints.
checkpoint_dir = "./ckpt"
if not os.path.exists(checkpoint_dir):
    os.makedirs(checkpoint_dir)


def make_or_restore_model():
    # Either restore the latest model, or create a fresh one
    # if there is no checkpoint available.
    checkpoints = [checkpoint_dir + "/" + name for name in os.listdir(checkpoint_dir)]
    if checkpoints:
        latest_checkpoint = max(checkpoints, key=os.path.getctime)
        print("Restoring from", latest_checkpoint)
        return keras.models.load_model(latest_checkpoint)
    print("Creating a new model")
    return get_compiled_model()


model = make_or_restore_model()
callbacks = [
    # This callback saves a SavedModel every 100 batches.
    # We include the training loss in the saved model name.
    keras.callbacks.ModelCheckpoint(
        filepath=checkpoint_dir + "/ckpt-loss={loss:.2f}", save_freq=100
    )
]
model.fit(x_train, y_train, epochs=1, callbacks=callbacks)

Creating a new model
  88/1563 [>.............................] - ETA: 3s - loss: 1.1203 - sparse_categorical_accuracy: 0.6911INFO:tensorflow:Assets written to: ./ckpt/ckpt-loss=1.04/assets
 185/1563 [==>...........................] - ETA: 6s - loss: 0.7768 - sparse_categorical_accuracy: 0.7858INFO:tensorflow:Assets written to: ./ckpt/ckpt-loss=0.75/assets
 286/1563 [====>.........................] - ETA: 6s - loss: 0.6382 - sparse_categorical_accuracy: 0.8211INFO:tensorflow:Assets written to: ./ckpt/ckpt-loss=0.63/assets
 383/1563 [======>.......................] - ETA: 6s - loss: 0.5584 - sparse_categorical_accuracy: 0.8433INFO:tensorflow:Assets written to: ./ckpt/ckpt-loss=0.55/assets
 484/1563 [========>.....................] - ETA: 6s - loss: 0.5032 - sparse_categorical_accuracy: 0.8578INFO:tensorflow:Assets written to: ./ckpt/ckpt-loss=0.50/assets
 586/1563 [==========>...................] - ETA: 5s - loss: 0.4644 - sparse_categorical_accuracy: 0.8684INFO:tensorflow:Assets written to: ./ckpt/ckpt-loss=0.46/assets
 685/1563 [============>.................] - ETA: 5s - loss: 0.4356 - sparse_categorical_accuracy: 0.8762INFO:tensorflow:Assets written to: ./ckpt/ckpt-loss=0.43/assets
 783/1563 [==============>...............] - ETA: 5s - loss: 0.4127 - sparse_categorical_accuracy: 0.8825INFO:tensorflow:Assets written to: ./ckpt/ckpt-loss=0.41/assets
 883/1563 [===============>..............] - ETA: 4s - loss: 0.3958 - sparse_categorical_accuracy: 0.8868INFO:tensorflow:Assets written to: ./ckpt/ckpt-loss=0.39/assets
 985/1563 [=================>............] - ETA: 3s - loss: 0.3766 - sparse_categorical_accuracy: 0.8918INFO:tensorflow:Assets written to: ./ckpt/ckpt-loss=0.38/assets
1086/1563 [===================>..........] - ETA: 3s - loss: 0.3624 - sparse_categorical_accuracy: 0.8958INFO:tensorflow:Assets written to: ./ckpt/ckpt-loss=0.36/assets
1184/1563 [=====================>........] - ETA: 2s - loss: 0.3498 - sparse_categorical_accuracy: 0.8994INFO:tensorflow:Assets written to: ./ckpt/ckpt-loss=0.35/assets
1283/1563 [=======================>......] - ETA: 1s - loss: 0.3383 - sparse_categorical_accuracy: 0.9029INFO:tensorflow:Assets written to: ./ckpt/ckpt-loss=0.34/assets
1386/1563 [=========================>....] - ETA: 1s - loss: 0.3265 - sparse_categorical_accuracy: 0.9058INFO:tensorflow:Assets written to: ./ckpt/ckpt-loss=0.33/assets
1485/1563 [===========================>..] - ETA: 0s - loss: 0.3184 - sparse_categorical_accuracy: 0.9081INFO:tensorflow:Assets written to: ./ckpt/ckpt-loss=0.32/assets
1563/1563 [==============================] - 11s 7ms/step - loss: 0.3122 - sparse_categorical_accuracy: 0.9097
<keras.callbacks.History at 0x7ff8b53e1dd0>

You call also write your own callback for saving and restoring models.

For a complete guide on serialization and saving, see the
guide to saving and serializing Models.

Using learning rate schedules

A common pattern when training deep learning models is to gradually reduce the learning
as training progresses. This is generally known as «learning rate decay».

The learning decay schedule could be static (fixed in advance, as a function of the
current epoch or the current batch index), or dynamic (responding to the current
behavior of the model, in particular the validation loss).

Passing a schedule to an optimizer

You can easily use a static learning rate decay schedule by passing a schedule object
as the learning_rate argument in your optimizer:

initial_learning_rate = 0.1
lr_schedule = keras.optimizers.schedules.ExponentialDecay(
    initial_learning_rate, decay_steps=100000, decay_rate=0.96, staircase=True
)

optimizer = keras.optimizers.RMSprop(learning_rate=lr_schedule)

Several built-in schedules are available: ExponentialDecay, PiecewiseConstantDecay,
PolynomialDecay, and InverseTimeDecay.

Using callbacks to implement a dynamic learning rate schedule

A dynamic learning rate schedule (for instance, decreasing the learning rate when the
validation loss is no longer improving) cannot be achieved with these schedule objects,
since the optimizer does not have access to validation metrics.

However, callbacks do have access to all metrics, including validation metrics! You can
thus achieve this pattern by using a callback that modifies the current learning rate
on the optimizer. In fact, this is even built-in as the ReduceLROnPlateau callback.

Visualizing loss and metrics during training

The best way to keep an eye on your model during training is to use
TensorBoard — a browser-based application
that you can run locally that provides you with:

• Live plots of the loss and metrics for training and evaluation
• (optionally) Visualizations of the histograms of your layer activations
• (optionally) 3D visualizations of the embedding spaces learned by your Embedding
  layers

If you have installed TensorFlow with pip, you should be able to launch TensorBoard
from the command line:

tensorboard --logdir=/full_path_to_your_logs

Using the TensorBoard callback

The easiest way to use TensorBoard with a Keras model and the fit() method is the
TensorBoard callback.

In the simplest case, just specify where you want the callback to write logs, and
you’re good to go:

keras.callbacks.TensorBoard(
    log_dir="/full_path_to_your_logs",
    histogram_freq=0,  # How often to log histogram visualizations
    embeddings_freq=0,  # How often to log embedding visualizations
    update_freq="epoch",
)  # How often to write logs (default: once per epoch)

<keras.callbacks.TensorBoard at 0x7ff88c8c04d0>

For more information, see the
documentation for the TensorBoard callback.

Introduction

tf.keras.losses.BinaryCrossentropy(
    from_logits=False, label_smoothing=0, reduction="auto", name="binary_crossentropy"
)

y_true = [[0., 1.], [0., 0.]]
y_pred = [[0.6, 0.4], [0.4, 0.6]]
# Using 'auto'/'sum_over_batch_size' reduction type.  
bce = tf.keras.losses.BinaryCrossentropy()
bce(y_true, y_pred).numpy()

tf.keras.losses.CategoricalCrossentropy(from_logits=False,label_smoothing=0, reduction="auto",name="categorical_crossentropy",)

y_true = [[0, 1, 0], [0, 0, 1]]
y_pred = [[0.05, 0.95, 0], [0.1, 0.8, 0.1]]
# Using 'auto'/'sum_over_batch_size' reduction type.  
cce = tf.keras.losses.CategoricalCrossentropy()
cce(y_true, y_pred).numpy()

tf.keras.losses.KLDivergence(reduction="auto", name="kl_divergence")

y_true = [[0, 1], [0, 0]]
y_pred = [[0.6, 0.4], [0.4, 0.6]]
# Using 'auto'/'sum_over_batch_size' reduction type.  
kl = tf.keras.losses.KLDivergence()
kl(y_true, y_pred).numpy()

tf.keras.losses.Poisson(reduction="auto", name="poisson")

y_true = [[0., 1.], [0., 0.]]
y_pred = [[1., 1.], [0., 0.]]
# Using 'auto'/'sum_over_batch_size' reduction type.  
p = tf.keras.losses.Poisson()
p(y_true, y_pred).numpy()

tf.keras.losses.Hinge(reduction="auto", name="hinge")

y_true = [[0., 1.], [0., 0.]]
y_pred = [[0.6, 0.4], [0.4, 0.6]]
# Using 'auto'/'sum_over_batch_size' reduction type.  
h = tf.keras.losses.Hinge()
h(y_true, y_pred).numpy()

tf.keras.losses.squared_hinge(y_true, y_pred)

import numpy as np

y_true = np.random.choice([-1, 1], size=(2, 3))
y_pred = np.random.random(size=(2, 3))
loss = tf.keras.losses.squared_hinge(y_true, y_pred)
assert loss.shape == (2,)
assert np.array_equal(loss.numpy(),np.mean(np.square(np.maximum(1. - y_true * y_pred, 0.)), axis=-1))

tf.keras.losses.CategoricalHinge(reduction="auto", name="categorical_hinge")

y_true = [[0, 1], [0, 0]]
y_pred = [[0.6, 0.4], [0.4, 0.6]]
# Using 'auto'/'sum_over_batch_size' reduction type.  
h = tf.keras.losses.CategoricalHinge()
h(y_true, y_pred).numpy()

tf.keras.losses.MeanSquaredError(reduction="auto", name="mean_squared_error")

y_true = [[0., 1.], [0., 0.]]
y_pred = [[1., 1.], [1., 0.]]
# Using 'auto'/'sum_over_batch_size' reduction type.  
mse = tf.keras.losses.MeanSquaredError()
mse(y_true, y_pred).numpy()

tf.keras.losses.MeanAbsoluteError(
    reduction="auto", name="mean_absolute_error"
)

y_true = [[0., 1.], [0., 0.]]
y_pred = [[1., 1.], [1., 0.]]
# Using 'auto'/'sum_over_batch_size' reduction type.  
mae = tf.keras.losses.MeanAbsoluteError()
mae(y_true, y_pred).numpy()

tf.keras.losses.CosineSimilarity(
    axis=-1, reduction="auto", name="cosine_similarity"
)