sMPAE

timecave.validation_strategy_metrics.sMPAE(estimated_error, test_error)

Compute the symmetric Mean Predictive Accuracy Error (sMPAE).

This function computes the sMPAE metric. Both the estimated (i.e. validation) error and the test error must be passed as parameters.

Parameters:

Name	Type	Description	Default
estimated_error	float | int	Validation error.	required
test_error	float | int	True (i.e. test) error.	required

Returns:

Type	Description
float	Symmetric Mean Predictive Accuracy Error.

Raises:

Type	Description
ValueError	If test_error is zero.

See also

PAE: Predictive Accuracy Error.

APAE: Absolute Predictive Accuracy Error.

RPAE: Relative Predictive Accuracy Error.

RAPAE: Relative Absolute Predictive Accuracy Error.

Notes

The symmetric Mean Predictive Accuracy Error is obtained by dividing the Predictive Accuracy Error (PAE) by half the sum of the absolute values of both the error estimate and the true error:

\[ sMPAE = 2 \cdot \frac{(\hat{L}_m - L_m)}{|\hat{L}_m| + |L_m|} = 2 \cdot \frac{PAE}{|\hat{L}_m| + |L_m|} \]

Similarly to the Relative Predictive Accuracy Error (RPAE), this metric can be seen as a scaled version of the PAE. Unlike the RPAE, however, the sMPAE is symmetric, as all possible values lie in the interval of \([-2, 2]\). If the error estimate is equal to the true error (perfect estimation), the sMPAE is zero. Since this metric is based on the PAE, the sign retains its significance (negative sign for underestimation, positive sign for overestimation).

Note that, in all likelihood, the true error will not be known. It is usually estimated using an independent test set.

Examples:

>>> from timecave.validation_strategy_metrics import sMPAE
>>> sMPAE(3, 2)
0.4
>>> sMPAE(3, 5)
-0.5
>>> sMPAE(5, 5)
0.0
>>> sMPAE(5, 0)
2.0
>>> sMPAE(0, 5)
-2.0

If both the true error and the estimated error are zero, this metric is undefined:

>>> sMPAE(0, 0)
Traceback (most recent call last):
...
ValueError: sMPAE is undefined.

Source code in timecave/validation_strategy_metrics.py

def sMPAE(estimated_error: float | int, test_error: float | int) -> float:

    """
    Compute the symmetric Mean Predictive Accuracy Error (sMPAE).

    This function computes the sMPAE metric. Both the estimated (i.e. validation) error
    and the test error must be passed as parameters.

    Parameters
    ----------
    estimated_error : float | int
        Validation error.

    test_error : float | int
        True (i.e. test) error.

    Returns
    -------
    float
        Symmetric Mean Predictive Accuracy Error.

    Raises
    ------
    ValueError
        If `test_error` is zero.

    See also
    --------
    [PAE](pae.md):
        Predictive Accuracy Error.

    [APAE](apae.md):
        Absolute Predictive Accuracy Error.

    [RPAE](rpae.md):
        Relative Predictive Accuracy Error.

    [RAPAE](rapae.md):
        Relative Absolute Predictive Accuracy Error.

    Notes
    -----
    The symmetric Mean Predictive Accuracy Error is obtained by dividing the Predictive Accuracy Error (PAE) \
    by half the sum of the absolute values of both the error estimate and the true error:

    $$
    sMPAE = 2 \cdot \\frac{(\hat{L}_m - L_m)}{|\hat{L}_m| + |L_m|} = 2 \cdot \\frac{PAE}{|\hat{L}_m| + |L_m|}
    $$

    Similarly to the Relative Predictive Accuracy Error (RPAE), this metric can be seen as a scaled version of \
    the PAE. Unlike the RPAE, however, the sMPAE is symmetric, as all possible values lie in the interval of $[-2, 2]$. If the \
    error estimate is equal to the true error (perfect estimation), the sMPAE is zero. \
    Since this metric is based on the PAE, the sign retains its significance (negative sign for underestimation, positive sign for overestimation).

    Note that, in all likelihood, the true error will not be known. It is usually estimated using an independent test set.

    Examples
    --------
    >>> from timecave.validation_strategy_metrics import sMPAE
    >>> sMPAE(3, 2)
    0.4
    >>> sMPAE(3, 5)
    -0.5
    >>> sMPAE(5, 5)
    0.0
    >>> sMPAE(5, 0)
    2.0
    >>> sMPAE(0, 5)
    -2.0

    If both the true error and the estimated error are zero, this metric is undefined:

    >>> sMPAE(0, 0)
    Traceback (most recent call last):
    ...
    ValueError: sMPAE is undefined.
    """

    if((abs(estimated_error) + abs(test_error)) == 0):

        raise ValueError("sMPAE is undefined.");

    return 2*(estimated_error - test_error) / (abs(estimated_error) + abs(test_error));