neural_trees.DistilledTree#

class neural_trees.DistilledTree(teacher=None, student=None, alpha: float = 0.0, threshold: float = 0.01, synthetic: int = 10, random_state: int | None = None)[source]#

Bases: ClassifierMixin, BaseEstimator

One soft tree trained on what a strong model learned: fit trains the teacher on (X, y), then the student tree on the teacher’s class probabilities on the same rows, on synthetic rows near them (synthetic) and on unlabelled rows if given (mixed with the true labels by alpha). Prediction, explain and every export go through the single student tree, so the deployed model is one readable tree; the teacher is kept only to measure agreement.

Why: on large tables a single soft tree trained on the labels trails gradient boosting, and the probabilities of a boosted model are a better training signal than the labels where the tree overfits them. On six OpenML tables at 5 000 to 100 000 rows (XGBoost teacher, depth-8 student; benchmarks/damitma_olcek.py) this raised Higgs by 0.9 to 2.1 points and jannis at 20 000 rows by 2.7, and did nothing where the tree is short of capacity (electricity, covertype); +0.6 points on average over 13 cells, not significant. What moves it is more rows for the teacher to label. Synthetic ones (MUNGE passes, see munge) raised the same tree, with two passes, by 1.5 points over the tree on labels and 0.9 over plain distillation, higher in 12 of 13 cells each (Wilcoxon p=0.0002 and 0.0005; benchmarks/munge.py), and cut the mean gap to XGBoost from 3.3 to 1.8 points. Ten passes (synthetic=10, the default) added 0.5 points over two on the 11 cells up to 20 000 rows (7 higher, none lower, p=0.001; benchmarks/munge.py with MUNGE_PASSES=10), leaving the tree 1.3 points behind XGBoost with 300 trees and level with XGBoost with 30 trees, about ten times its size. With the default teacher ten passes gave +0.3 over two (8 of 12 fits higher, none lower, p=0.009; benchmarks/munge_lib.py). Thirty passes were no better than ten. The cost is fit time: ten passes fit in four to six times the time of two (about 15 minutes at 20 000 rows on one CPU thread); synthetic=2 is the faster setting. With a LightGBM teacher two passes gained about as much as with XGBoost (+0.9 over plain distillation, 9 of 11 cells higher, none lower; benchmarks/munge_lgbm.py). Real unlabelled rows (X_unlabeled in fit) do a little better again. Below about 1 000 rows use SoftDecisionTree(growth="per_leaf") instead: on the 24 datasets of benchmarks/rakipler.py it was 1.2 points better there (7 of 8, p=0.016; benchmarks/damitma_kucuk.py). The documentation page “Large tables” has the full tables.

Parameters:

teacherclassifier with predict_proba, default=None: Cloned and fitted on (X, y). None uses scikit-learn’s HistGradientBoostingClassifier(); XGBoost or LightGBM work the same.
studentSoftDecisionTree, default=None: Cloned and fitted on the teacher’s probabilities. None uses SoftDecisionTree(depth=8, init="cart", learn_temperature=True, max_epochs=60, batch_size=256).
alphafloat, default=0.0: Weight of the true labels against the teacher’s probabilities in the student’s loss; 0 learns the teacher only.
thresholdfloat, default=0.01: A class enters a row’s soft target only with at least this teacher probability.
syntheticint, default=10: MUNGE passes over the labelled rows; each adds len(X) synthetic rows that the teacher labels for the student. 0 turns it off; fewer passes fit faster and score a little lower.
random_stateint, optional: Seeds the synthetic rows and the default student.

Attributes:

teacher_, student_the fitted models
classes_, n_features_in_

explain(X, **kwargs)[source]#: The student tree’s explanation; see SoftDecisionTree.explain.

fidelity(X) → float[source]#: Share of rows of X where the student predicts the teacher’s class.

fit(X, y, sample_weight=None, X_unlabeled=None) → DistilledTree[source]#

Fit the teacher on (X, y), then the student on the teacher’s probabilities on X and, if given, on X_unlabeled: rows without a label, from the same population, that the teacher labels for the student. Labels are often scarcer than rows (every customer is in the table, few have an outcome), and the teacher can pass on what it learned from the labelled rows through the unlabelled ones. With twice as many unlabelled rows as labelled ones the depth-8 tree gained 1.85 points over the same tree on labels, in 13 of 13 cells, and halved its mean gap to the XGBoost teacher (3.3 to 1.45 points); on Higgs and jannis it reached the teacher. The teacher itself sees only (X, y).

Like SoftDecisionTree, this passes scikit-learn’s estimator checks except check_sample_weight_equivalence_on_dense_data, which no stochastic mini-batch learner can satisfy.

set_fit_request(*, X_unlabeled: bool | None | str = '$UNCHANGED$', sample_weight: bool | None | str = '$UNCHANGED$') → DistilledTree#

Configure whether metadata should be requested to be passed to the fit method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config()). Please check the User Guide on how the routing mechanism works.

The options for each parameter are:

True: metadata is requested, and passed to fit if provided. The request is ignored if metadata is not provided.
False: metadata is not requested and the meta-estimator will not pass it to fit.
None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.
str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters:

X_unlabeledstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED: Metadata routing for X_unlabeled parameter in fit.
sample_weightstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED: Metadata routing for sample_weight parameter in fit.

Returns:

selfobject: The updated object.

set_score_request(*, sample_weight: bool | None | str = '$UNCHANGED$') → DistilledTree#

Configure whether metadata should be requested to be passed to the score method.

The options for each parameter are:

True: metadata is requested, and passed to score if provided. The request is ignored if metadata is not provided.
False: metadata is not requested and the meta-estimator will not pass it to score.
None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.
str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters:

sample_weightstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED: Metadata routing for sample_weight parameter in score.

Returns:

selfobject: The updated object.