neural_trees.combined_5x2cv_f_test#

neural_trees.combined_5x2cv_f_test(clf_A: Any, clf_B: Any, X, y, alpha: float = 0.05, random_state: int = 42) → TestResult[source]#

Alpaydın’s Combined 5×2 Cross-Validation F Test.

Compares two classifiers by repeating 2-fold CV 5 times (giving 10 difference measurements) and computing an F statistic.

H0: The two classifiers have equal expected error rates. H1: The classifiers differ.

This test avoids the high variance of the standard paired t-test and the loss of power in McNemar’s test. It is the recommended method for classifier comparison (Alpaydın, 1999).

Parameters:

clf_Asklearn-compatible classifier
clf_Bsklearn-compatible classifier
Xarray-like of shape (n_samples, n_features)
yarray-like of shape (n_samples,)
alphafloat, default=0.05: Significance level.
random_stateint, default=42: Seed for reproducibility.

Returns:

TestResult

References

Alpaydın, E. (1999). Combined 5x2cv F Test for Comparing Supervised Classification Learning Algorithms. Neural Computation, 11(8), 1885-1892.

Examples

>>> from sklearn.tree import DecisionTreeClassifier
>>> from sklearn.neighbors import KNeighborsClassifier
>>> from sklearn.datasets import load_breast_cancer
>>> X, y = load_breast_cancer(return_X_y=True)
>>> result = combined_5x2cv_f_test(DecisionTreeClassifier(), KNeighborsClassifier(), X, y)
>>> print(result)