Skip to content

PAE

timecave.validation_strategy_metrics.PAE(estimated_error, test_error)

Compute the Predictive Accuracy Error (PAE).

This function computes the PAE metric. Both the estimated (i.e. validation) error and the test error must be passed as parameters.

Parameters:

Name Type Description Default
estimated_error float | int

Validation error.

 required
test_error float | int

True (i.e. test) error.

 required

Returns:

Type Description
float

Predictive Accuracy Error.
See also

APAE: Absolute Predictive Accuracy Error.

RPAE: Relative Predictive Accuracy Error.

RAPAE: Relative Absolute Predictive Accuracy Error.

sMPAE: Symmetric Mean Predictive Accuracy Error.

Notes

The Predictive Accuracy Error is defined as the difference between the estimate of a model's error given by a validation method and the model's true error:

\[ PAE = \hat{L}_m - L_m \]

The sign allows one to determine whether the validation method is overestimating or underestimating the model's true error: a negative value denotes an underestimation, while a positive value corresponds to an overestimation.
Note that, in all likelihood, the true error will not be known. It is usually estimated using an independent test set. For more details, please refer to [1].

References

1

Cerqueira, V., Torgo, L., Mozetiˇc, I., 2020. Evaluating time series forecasting models: An empirical study on performance estimation methods. Machine Learning 109, 1997–2028.

Examples:

>>> from timecave.validation_strategy_metrics import PAE
>>> PAE(10, 3)
7
>>> PAE(1, 5)
-4
>>> PAE(8, 8)
0
Source code in timecave/validation_strategy_metrics.py 
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
def PAE(estimated_error: float | int, test_error: float | int) -> float:

    """
    Compute the Predictive Accuracy Error (PAE).

    This function computes the PAE metric. Both the estimated (i.e. validation) error
    and the test error must be passed as parameters.

    Parameters
    ----------
    estimated_error : float | int
        Validation error.

    test_error : float | int
        True (i.e. test) error.

    Returns
    -------
    float
        Predictive Accuracy Error.

    See also
    --------
    [APAE](apae.md):
        Absolute Predictive Accuracy Error.

    [RPAE](rpae.md): 
        Relative Predictive Accuracy Error.

    [RAPAE](rapae.md):
        Relative Absolute Predictive Accuracy Error.

    [sMPAE](smpae.md):
        Symmetric Mean Predictive Accuracy Error.

    Notes
    -----
    The Predictive Accuracy Error is defined as the difference between the estimate of a model's error given by a validation method\
    and the model's true error:

    $$
    PAE = \hat{L}_m - L_m
    $$ 

    The sign allows one to determine whether the validation method is overestimating or underestimating the model's true error:\
    a negative value denotes an underestimation, while a positive value corresponds to an overestimation.\

    Note that, in all likelihood, the true error will not be known. It is usually estimated using an independent test set. For more details, please refer to [[1]](#1).

    References
    ----------
    ##1
    Cerqueira, V., Torgo, L., Mozetiˇc, I., 2020. Evaluating time series forecasting
    models: An empirical study on performance estimation methods.
    Machine Learning 109, 1997–2028.

    Examples
    --------
    >>> from timecave.validation_strategy_metrics import PAE
    >>> PAE(10, 3)
    7
    >>> PAE(1, 5)
    -4
    >>> PAE(8, 8)
    0
    """

    return estimated_error - test_error;