metrics module

metrics.py: Custom Metrics for Model Evaluation and Utility Functions for Model Visualization

This module provides a collection of custom metrics that can be used for evaluating model performance in tasks such as image segmentation. This module includes a wide variety of evaluation metrics listed below. Additionally, it contains utility functions for plotting and visualizing model metrics during training.

`Metrics` ¶

A class containing various metrics functions for model evaluation.

This class provides a collection of static methods, each representing a different evaluation metric. These metrics can be used to evaluate the performance of classification or segmentation models during training or testing.

Methods

precision(): Computes the precision metric. recall(): Computes the recall metric. f1_m(): Computes the F1 score metric. [van1979information, chicco2020advantages] one_hot_io_u(num_classes): Computes the Intersection over Union (IoU) metric for each class. [jaccard1901etude] true_positives(): Computes the true positives metric also known as sensitivity.[yerushalmy1947statistical] false_positives(): Computes the false positives metric. [cohen2013statistical] true_negatives(): Computes the true negatives metric also known as specificity. [yerushalmy1947statistical] false_negatives(): Computes the false negatives metric. [cohen2013statistical] binary_accuracy(): Computes the binary accuracy metric. auc(): Computes the Area Under the Curve (AUC) metric. [provost2001robust] prc(): Computes the Precision-Recall Curve (PRC) metric. dice_coef(): Computes the Dice coefficient metric. [milletari2016v] dice_loss(): Computes the Dice loss metric. [sudre2017generalised] bce_loss(): Computes the Binary Cross-Entropy (BCE) loss metric. [zhu2018negative; yi2004automated] bce_dice_loss(): Computes the BCE and Dice loss metric. cite [taghanaki2019combo] tversky(): Computes the Tversky index metric.[tversky1977features] tversky_loss(): Computes the Tversky loss metric. [salehi2017tversky] focal_tversky_loss(): Computes the focal Tversky loss metric [abraham2019novel]

Source code in aces/metrics.py

class Metrics:
    """
    A class containing various metrics functions for model evaluation.

    This class provides a collection of static methods, each representing a different evaluation metric. These metrics can be used to
    evaluate the performance of classification or segmentation models during training or testing.

    Methods:
        precision():
            Computes the precision metric.
        recall():
            Computes the recall metric.
        f1_m():
            Computes the F1 score metric. [van1979information, chicco2020advantages]
        one_hot_io_u(num_classes):
            Computes the Intersection over Union (IoU) metric for each class. [jaccard1901etude]
        true_positives():
            Computes the true positives metric also known as sensitivity.[yerushalmy1947statistical]
        false_positives():
            Computes the false positives metric. [cohen2013statistical]
        true_negatives():
            Computes the true negatives metric also known as specificity. [yerushalmy1947statistical]
        false_negatives():
            Computes the false negatives metric. [cohen2013statistical]
        binary_accuracy():
            Computes the binary accuracy metric.
        auc():
            Computes the Area Under the Curve (AUC) metric. [provost2001robust]
        prc():
            Computes the Precision-Recall Curve (PRC) metric.
        dice_coef():
            Computes the Dice coefficient metric. [milletari2016v]
        dice_loss():
            Computes the Dice loss metric. [sudre2017generalised]
        bce_loss():
            Computes the Binary Cross-Entropy (BCE) loss metric. [zhu2018negative; yi2004automated]
        bce_dice_loss():
            Computes the BCE and Dice loss metric. cite [taghanaki2019combo]
        tversky():
            Computes the Tversky index metric.[tversky1977features]
        tversky_loss():
            Computes the Tversky loss metric. [salehi2017tversky]
        focal_tversky_loss():
            Computes the focal Tversky loss metric [abraham2019novel]
    """

    @staticmethod
    def recall_m(y_true, y_pred):
        """
        Calculate the recall metric.

        Args:
            y_true: Ground truth labels.
            y_pred: Predicted labels.

        Returns:
            Recall metric value.
        """
        true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
        possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))
        recall = true_positives / (possible_positives + K.epsilon())
        return recall

    @staticmethod
    def precision_m(y_true, y_pred):
        """
        Calculate the precision metric.

        Args:
            y_true: Ground truth labels.
            y_pred: Predicted labels.

        Returns:
            Precision metric value.
        """
        true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
        predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))
        precision = true_positives / (predicted_positives + K.epsilon())
        return precision

    @staticmethod
    def f1_m(y_true, y_pred):
        """
        Calculate the F1-score metric.

        Args:
            y_true: Ground truth labels.
            y_pred: Predicted labels.

        Returns:
            F1-score metric value.
        """
        precision = Metrics.precision_m(y_true, y_pred)
        recall = Metrics.recall_m(y_true, y_pred)
        return 2 * ((precision * recall) / (precision + recall + K.epsilon()))

    @staticmethod
    def dice_coef(y_true, y_pred, smooth=1):
        """
        Calculate the Dice coefficient metric.

        Args:
            y_true: Ground truth labels.
            y_pred: Predicted labels.
            smooth: Smoothing parameter.

        Returns:
            Dice coefficient metric value.
        """
        y_true_f = K.flatten(y_true)
        y_pred_f = K.flatten(y_pred)
        intersection = K.sum(y_true_f * y_pred_f)
        return (2. * intersection + smooth) / (K.sum(y_true_f) + K.sum(y_pred_f) + smooth)

    @staticmethod
    def dice_loss(y_true, y_pred, smooth=1):
        """
        Calculate the Dice loss metric.

        Args:
            y_true: Ground truth labels.
            y_pred: Predicted labels.
            smooth: Smoothing parameter.

        Returns:
            Dice loss metric value.
        """
        intersection = K.sum(K.abs(y_true * y_pred), axis=-1)
        true_sum = K.sum(K.square(y_true), -1)
        pred_sum = K.sum(K.square(y_pred), -1)
        return 1 - ((2. * intersection + smooth) / (true_sum + pred_sum + smooth))

    @staticmethod
    def bce_dice_loss(y_true, y_pred):
        """
        Calculate the BCE-Dice loss metric.

        Args:
            y_true: Ground truth labels.
            y_pred: Predicted labels.

        Returns:
            BCE-Dice loss metric value.
        """
        return Metrics.bce_loss(y_true, y_pred) + Metrics.dice_loss(y_true, y_pred)

    @staticmethod
    def bce_loss(y_true, y_pred):
        """
        Calculate the Binary Cross-Entropy (BCE) loss metric.

        Args:
            y_true: Ground truth labels.
            y_pred: Predicted labels.

        Returns:
            BCE loss metric value.
        """
        return keras.losses.binary_crossentropy(y_true, y_pred, label_smoothing=0.2)

    @staticmethod
    def tversky(y_true, y_pred, smooth=1, alpha=0.7):
        """
        Calculate the Tversky index metric.

        Args:
            y_true: Ground truth labels.
            y_pred: Predicted labels.
            smooth: Smoothing parameter.
            alpha: Weighting factor.

        Returns:
            Tversky index metric value.
        """
        y_true_pos = K.flatten(y_true)
        y_pred_pos = K.flatten(y_pred)
        true_pos = K.sum(y_true_pos * y_pred_pos)
        false_neg = K.sum(y_true_pos * (1 - y_pred_pos))
        false_pos = K.sum((1 - y_true_pos) * y_pred_pos)
        return (true_pos + smooth) / (true_pos + alpha * false_neg + (1 - alpha) * false_pos + smooth)

    @staticmethod
    def tversky_loss(y_true, y_pred):
        """
        Calculate the Tversky loss metric.

        Args:
            y_true: Ground truth labels.
            y_pred: Predicted labels.

        Returns:
            Tversky loss metric value.
        """
        return 1 - Metrics.tversky(y_true, y_pred)

    @staticmethod
    def focal_tversky_loss(y_true, y_pred, gamma=0.75):
        """
        Calculate the focal Tversky loss metric.

        Args:
            y_true: Ground truth labels.
            y_pred: Predicted labels.
            gamma: Focusing parameter.

        Returns:
            Focal Tversky loss metric value.
        """
        return K.pow((1 - Metrics.tversky(y_true, y_pred)), gamma)

    @staticmethod
    def true_positives():
        """
        Create a metric for counting true positives.

        Returns:
            True positives metric.
        """
        return keras.metrics.TruePositives(name="tp")

    @staticmethod
    def false_positives():
        """
        Create a metric for counting false positives.

        Returns:
            False positives metric.
        """
        return keras.metrics.FalsePositives(name="fp")

    @staticmethod
    def true_negatives():
        """
        Create a metric for counting true negatives.

        Returns:
            True negatives metric.
        """
        return keras.metrics.TrueNegatives(name="tn")

    @staticmethod
    def false_negatives():
        """
        Create a metric for counting false negatives.

        Returns:
            False negatives metric.
        """
        return keras.metrics.FalseNegatives(name="fn")

    @staticmethod
    def binary_accuracy():
        """
        Create a metric for calculating binary accuracy.

        Returns:
            Binary accuracy metric.
        """
        return keras.metrics.BinaryAccuracy(name="accuracy")

    # check difference between this and precision_m output
    @staticmethod
    def precision():
        """
        Create a metric for calculating precision.

        Returns:
            Precision metric.
        """
        return keras.metrics.Precision(name="precision")

    @staticmethod
    def recall():
        """
        Create a metric for calculating recall.

        Returns:
            Recall metric.
        """
        return keras.metrics.Recall(name="recall")

    @staticmethod
    def auc():
        """
        Create a metric for calculating Area Under the Curve (AUC).

        Returns:
            AUC metric.
        """
        return keras.metrics.AUC(name="auc")

    @staticmethod
    def prc():
        """
        Create a metric for calculating Precision-Recall Curve (PRC).

        Returns:
            PRC metric.
        """
        return keras.metrics.AUC(name="prc", curve="PR")

    @staticmethod
    def one_hot_io_u(num_classes, name="one_hot_io_u"):
        """
        Create a metric for calculating Intersection over Union (IoU) using one-hot encoding.

        Args:
            num_classes: Number of classes.
            name: Name of the metric.

        Returns:
            One-hot IoU metric.
        """
        return keras.metrics.OneHotIoU(num_classes=num_classes, target_class_ids=list(range(num_classes)), name=name)

`auc()` `staticmethod` ¶

Create a metric for calculating Area Under the Curve (AUC).

Returns:

Type	Description
	AUC metric.

Source code in aces/metrics.py

@staticmethod
def auc():
    """
    Create a metric for calculating Area Under the Curve (AUC).

    Returns:
        AUC metric.
    """
    return keras.metrics.AUC(name="auc")

`bce_dice_loss(y_true, y_pred)` `staticmethod` ¶

Calculate the BCE-Dice loss metric.

Parameters:

Name	Type	Description	Default
`y_true`		Ground truth labels.	required
`y_pred`		Predicted labels.	required

Returns:

Type	Description
	BCE-Dice loss metric value.

Source code in aces/metrics.py

@staticmethod
def bce_dice_loss(y_true, y_pred):
    """
    Calculate the BCE-Dice loss metric.

    Args:
        y_true: Ground truth labels.
        y_pred: Predicted labels.

    Returns:
        BCE-Dice loss metric value.
    """
    return Metrics.bce_loss(y_true, y_pred) + Metrics.dice_loss(y_true, y_pred)

`bce_loss(y_true, y_pred)` `staticmethod` ¶

Calculate the Binary Cross-Entropy (BCE) loss metric.

Parameters:

Name	Type	Description	Default
`y_true`		Ground truth labels.	required
`y_pred`		Predicted labels.	required

Returns:

Type	Description
	BCE loss metric value.

Source code in aces/metrics.py

@staticmethod
def bce_loss(y_true, y_pred):
    """
    Calculate the Binary Cross-Entropy (BCE) loss metric.

    Args:
        y_true: Ground truth labels.
        y_pred: Predicted labels.

    Returns:
        BCE loss metric value.
    """
    return keras.losses.binary_crossentropy(y_true, y_pred, label_smoothing=0.2)

`binary_accuracy()` `staticmethod` ¶

Create a metric for calculating binary accuracy.

Returns:

Type	Description
	Binary accuracy metric.

Source code in aces/metrics.py

@staticmethod
def binary_accuracy():
    """
    Create a metric for calculating binary accuracy.

    Returns:
        Binary accuracy metric.
    """
    return keras.metrics.BinaryAccuracy(name="accuracy")

`dice_coef(y_true, y_pred, smooth=1)` `staticmethod` ¶

Calculate the Dice coefficient metric.

Parameters:

Name	Description	Default
`y_true`	Ground truth labels.	required
`y_pred`	Predicted labels.	required
`smooth`	Smoothing parameter.	`1`

Returns:

Type	Description
	Dice coefficient metric value.

Source code in aces/metrics.py

@staticmethod
def dice_coef(y_true, y_pred, smooth=1):
    """
    Calculate the Dice coefficient metric.

    Args:
        y_true: Ground truth labels.
        y_pred: Predicted labels.
        smooth: Smoothing parameter.

    Returns:
        Dice coefficient metric value.
    """
    y_true_f = K.flatten(y_true)
    y_pred_f = K.flatten(y_pred)
    intersection = K.sum(y_true_f * y_pred_f)
    return (2. * intersection + smooth) / (K.sum(y_true_f) + K.sum(y_pred_f) + smooth)

`dice_loss(y_true, y_pred, smooth=1)` `staticmethod` ¶

Calculate the Dice loss metric.

Parameters:

Name	Description	Default
`y_true`	Ground truth labels.	required
`y_pred`	Predicted labels.	required
`smooth`	Smoothing parameter.	`1`

Returns:

Type	Description
	Dice loss metric value.

Source code in aces/metrics.py

@staticmethod
def dice_loss(y_true, y_pred, smooth=1):
    """
    Calculate the Dice loss metric.

    Args:
        y_true: Ground truth labels.
        y_pred: Predicted labels.
        smooth: Smoothing parameter.

    Returns:
        Dice loss metric value.
    """
    intersection = K.sum(K.abs(y_true * y_pred), axis=-1)
    true_sum = K.sum(K.square(y_true), -1)
    pred_sum = K.sum(K.square(y_pred), -1)
    return 1 - ((2. * intersection + smooth) / (true_sum + pred_sum + smooth))

`f1_m(y_true, y_pred)` `staticmethod` ¶

Calculate the F1-score metric.

Parameters:

Name	Type	Description	Default
`y_true`		Ground truth labels.	required
`y_pred`		Predicted labels.	required

Returns:

Type	Description
	F1-score metric value.

Source code in aces/metrics.py

@staticmethod
def f1_m(y_true, y_pred):
    """
    Calculate the F1-score metric.

    Args:
        y_true: Ground truth labels.
        y_pred: Predicted labels.

    Returns:
        F1-score metric value.
    """
    precision = Metrics.precision_m(y_true, y_pred)
    recall = Metrics.recall_m(y_true, y_pred)
    return 2 * ((precision * recall) / (precision + recall + K.epsilon()))

`false_negatives()` `staticmethod` ¶

Create a metric for counting false negatives.

Returns:

Type	Description
	False negatives metric.

Source code in aces/metrics.py

@staticmethod
def false_negatives():
    """
    Create a metric for counting false negatives.

    Returns:
        False negatives metric.
    """
    return keras.metrics.FalseNegatives(name="fn")

`false_positives()` `staticmethod` ¶

Create a metric for counting false positives.

Returns:

Type	Description
	False positives metric.

Source code in aces/metrics.py

@staticmethod
def false_positives():
    """
    Create a metric for counting false positives.

    Returns:
        False positives metric.
    """
    return keras.metrics.FalsePositives(name="fp")

`focal_tversky_loss(y_true, y_pred, gamma=0.75)` `staticmethod` ¶

Calculate the focal Tversky loss metric.

Parameters:

Name	Description	Default
`y_true`	Ground truth labels.	required
`y_pred`	Predicted labels.	required
`gamma`	Focusing parameter.	`0.75`

Returns:

Type	Description
	Focal Tversky loss metric value.

Source code in aces/metrics.py

@staticmethod
def focal_tversky_loss(y_true, y_pred, gamma=0.75):
    """
    Calculate the focal Tversky loss metric.

    Args:
        y_true: Ground truth labels.
        y_pred: Predicted labels.
        gamma: Focusing parameter.

    Returns:
        Focal Tversky loss metric value.
    """
    return K.pow((1 - Metrics.tversky(y_true, y_pred)), gamma)

`one_hot_io_u(num_classes, name='one_hot_io_u')` `staticmethod` ¶

Create a metric for calculating Intersection over Union (IoU) using one-hot encoding.

Parameters:

Name	Type	Description	Default
`num_classes`		Number of classes.	required
`name`		Name of the metric.	`'one_hot_io_u'`

Returns:

Type	Description
	One-hot IoU metric.

Source code in aces/metrics.py

@staticmethod
def one_hot_io_u(num_classes, name="one_hot_io_u"):
    """
    Create a metric for calculating Intersection over Union (IoU) using one-hot encoding.

    Args:
        num_classes: Number of classes.
        name: Name of the metric.

    Returns:
        One-hot IoU metric.
    """
    return keras.metrics.OneHotIoU(num_classes=num_classes, target_class_ids=list(range(num_classes)), name=name)

`prc()` `staticmethod` ¶

Create a metric for calculating Precision-Recall Curve (PRC).

Returns:

Type	Description
	PRC metric.

Source code in aces/metrics.py

@staticmethod
def prc():
    """
    Create a metric for calculating Precision-Recall Curve (PRC).

    Returns:
        PRC metric.
    """
    return keras.metrics.AUC(name="prc", curve="PR")

`precision()` `staticmethod` ¶

Create a metric for calculating precision.

Returns:

Type	Description
	Precision metric.

Source code in aces/metrics.py

@staticmethod
def precision():
    """
    Create a metric for calculating precision.

    Returns:
        Precision metric.
    """
    return keras.metrics.Precision(name="precision")

`precision_m(y_true, y_pred)` `staticmethod` ¶

Calculate the precision metric.

Parameters:

Name	Type	Description	Default
`y_true`		Ground truth labels.	required
`y_pred`		Predicted labels.	required

Returns:

Type	Description
	Precision metric value.

Source code in aces/metrics.py

@staticmethod
def precision_m(y_true, y_pred):
    """
    Calculate the precision metric.

    Args:
        y_true: Ground truth labels.
        y_pred: Predicted labels.

    Returns:
        Precision metric value.
    """
    true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
    predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))
    precision = true_positives / (predicted_positives + K.epsilon())
    return precision

`recall()` `staticmethod` ¶

Create a metric for calculating recall.

Returns:

Type	Description
	Recall metric.

Source code in aces/metrics.py

@staticmethod
def recall():
    """
    Create a metric for calculating recall.

    Returns:
        Recall metric.
    """
    return keras.metrics.Recall(name="recall")

`recall_m(y_true, y_pred)` `staticmethod` ¶

Calculate the recall metric.

Parameters:

Name	Type	Description	Default
`y_true`		Ground truth labels.	required
`y_pred`		Predicted labels.	required

Returns:

Type	Description
	Recall metric value.

Source code in aces/metrics.py

@staticmethod
def recall_m(y_true, y_pred):
    """
    Calculate the recall metric.

    Args:
        y_true: Ground truth labels.
        y_pred: Predicted labels.

    Returns:
        Recall metric value.
    """
    true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
    possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))
    recall = true_positives / (possible_positives + K.epsilon())
    return recall

`true_negatives()` `staticmethod` ¶

Create a metric for counting true negatives.

Returns:

Type	Description
	True negatives metric.

Source code in aces/metrics.py

@staticmethod
def true_negatives():
    """
    Create a metric for counting true negatives.

    Returns:
        True negatives metric.
    """
    return keras.metrics.TrueNegatives(name="tn")

`true_positives()` `staticmethod` ¶

Create a metric for counting true positives.

Returns:

Type	Description
	True positives metric.

Source code in aces/metrics.py

@staticmethod
def true_positives():
    """
    Create a metric for counting true positives.

    Returns:
        True positives metric.
    """
    return keras.metrics.TruePositives(name="tp")

`tversky(y_true, y_pred, smooth=1, alpha=0.7)` `staticmethod` ¶

Calculate the Tversky index metric.

Parameters:

Name	Description	Default
`y_true`	Ground truth labels.	required
`y_pred`	Predicted labels.	required
`smooth`	Smoothing parameter.	`1`
`alpha`	Weighting factor.	`0.7`

Returns:

Type	Description
	Tversky index metric value.

Source code in aces/metrics.py

@staticmethod
def tversky(y_true, y_pred, smooth=1, alpha=0.7):
    """
    Calculate the Tversky index metric.

    Args:
        y_true: Ground truth labels.
        y_pred: Predicted labels.
        smooth: Smoothing parameter.
        alpha: Weighting factor.

    Returns:
        Tversky index metric value.
    """
    y_true_pos = K.flatten(y_true)
    y_pred_pos = K.flatten(y_pred)
    true_pos = K.sum(y_true_pos * y_pred_pos)
    false_neg = K.sum(y_true_pos * (1 - y_pred_pos))
    false_pos = K.sum((1 - y_true_pos) * y_pred_pos)
    return (true_pos + smooth) / (true_pos + alpha * false_neg + (1 - alpha) * false_pos + smooth)

`tversky_loss(y_true, y_pred)` `staticmethod` ¶

Calculate the Tversky loss metric.

Parameters:

Name	Type	Description	Default
`y_true`		Ground truth labels.	required
`y_pred`		Predicted labels.	required

Returns:

Type	Description
	Tversky loss metric value.

Source code in aces/metrics.py

@staticmethod
def tversky_loss(y_true, y_pred):
    """
    Calculate the Tversky loss metric.

    Args:
        y_true: Ground truth labels.
        y_pred: Predicted labels.

    Returns:
        Tversky loss metric value.
    """
    return 1 - Metrics.tversky(y_true, y_pred)

metrics module

Metrics ¶

auc() staticmethod ¶

bce_dice_loss(y_true, y_pred) staticmethod ¶

bce_loss(y_true, y_pred) staticmethod ¶

binary_accuracy() staticmethod ¶

dice_coef(y_true, y_pred, smooth=1) staticmethod ¶

dice_loss(y_true, y_pred, smooth=1) staticmethod ¶

f1_m(y_true, y_pred) staticmethod ¶

false_negatives() staticmethod ¶

false_positives() staticmethod ¶

focal_tversky_loss(y_true, y_pred, gamma=0.75) staticmethod ¶

one_hot_io_u(num_classes, name='one_hot_io_u') staticmethod ¶

prc() staticmethod ¶

precision() staticmethod ¶

precision_m(y_true, y_pred) staticmethod ¶

recall() staticmethod ¶

recall_m(y_true, y_pred) staticmethod ¶

true_negatives() staticmethod ¶

true_positives() staticmethod ¶

tversky(y_true, y_pred, smooth=1, alpha=0.7) staticmethod ¶

tversky_loss(y_true, y_pred) staticmethod ¶

`Metrics` ¶

`auc()` `staticmethod` ¶

`bce_dice_loss(y_true, y_pred)` `staticmethod` ¶

`bce_loss(y_true, y_pred)` `staticmethod` ¶

`binary_accuracy()` `staticmethod` ¶

`dice_coef(y_true, y_pred, smooth=1)` `staticmethod` ¶

`dice_loss(y_true, y_pred, smooth=1)` `staticmethod` ¶

`f1_m(y_true, y_pred)` `staticmethod` ¶

`false_negatives()` `staticmethod` ¶

`false_positives()` `staticmethod` ¶

`focal_tversky_loss(y_true, y_pred, gamma=0.75)` `staticmethod` ¶

`one_hot_io_u(num_classes, name='one_hot_io_u')` `staticmethod` ¶

`prc()` `staticmethod` ¶

`precision()` `staticmethod` ¶

`precision_m(y_true, y_pred)` `staticmethod` ¶

`recall()` `staticmethod` ¶

`recall_m(y_true, y_pred)` `staticmethod` ¶

`true_negatives()` `staticmethod` ¶

`true_positives()` `staticmethod` ¶

`tversky(y_true, y_pred, smooth=1, alpha=0.7)` `staticmethod` ¶

`tversky_loss(y_true, y_pred)` `staticmethod` ¶