.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "auto_examples/03_classifier_comparison.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        :ref:`Go to the end <sphx_glr_download_auto_examples_03_classifier_comparison.py>`
        to download the full example code.

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_auto_examples_03_classifier_comparison.py:


Is the difference significant?
==============================

Compare a Soft Decision Tree against CART with Alpaydin's combined 5x2cv F test.

The test asks whether the accuracy gap between the two models is larger than
what the variance of resampling alone would explain.

Run with:
    python examples/03_classifier_comparison.py

.. GENERATED FROM PYTHON SOURCE LINES 13-36




.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    F statistic: 8.7014
    p-value:     0.0138
    Interpretation: the two classifiers differ significantly (p < 0.05).






|

.. code-block:: Python

    from sklearn.datasets import load_breast_cancer
    from sklearn.pipeline import Pipeline
    from sklearn.preprocessing import StandardScaler
    from sklearn.tree import DecisionTreeClassifier

    from neural_trees import SoftDecisionTree, combined_5x2cv_f_test

    X, y = load_breast_cancer(return_X_y=True)

    soft_tree = Pipeline([
        ("scaler", StandardScaler()),
        ("model", SoftDecisionTree(depth=4, max_epochs=40, random_state=42)),
    ])
    cart = DecisionTreeClassifier(max_depth=4, random_state=42)

    result = combined_5x2cv_f_test(soft_tree, cart, X, y)

    print(f"F statistic: {result.statistic:.4f}")
    print(f"p-value:     {result.p_value:.4f}")
    if result.reject_null:
        print("Interpretation: the two classifiers differ significantly (p < 0.05).")
    else:
        print("Interpretation: no significant difference; the gap is within resampling noise.")


.. rst-class:: sphx-glr-timing

   **Total running time of the script:** (0 minutes 4.321 seconds)


.. _sphx_glr_download_auto_examples_03_classifier_comparison.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: 03_classifier_comparison.ipynb <03_classifier_comparison.ipynb>`

    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: 03_classifier_comparison.py <03_classifier_comparison.py>`

    .. container:: sphx-glr-download sphx-glr-download-zip

      :download:`Download zipped: 03_classifier_comparison.zip <03_classifier_comparison.zip>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_