neural_trees.SoftDecisionTree#

class neural_trees.SoftDecisionTree(depth: int = 5, max_epochs: int = 40, learning_rate: float = 0.01, batch_size: int = 64, penalty_coef: float = 0.001, device: str = 'cpu', verbose: bool = False, random_state: int | None = None, class_weight=None, growth: str = 'none', warm_start: bool = False, learn_temperature: bool = False, early_stopping: bool = False, validation_fraction: float = 0.1, n_iter_no_change: int = 10)[source]#

Bases: ClassifierMixin, BaseEstimator

Soft Decision Tree Classifier (sklearn-compatible).

A fully differentiable decision tree where each internal node applies a soft (sigmoid) split, allowing end-to-end gradient training.

Parameters:
depthint, default=5

Depth of the tree. The tree has 2^depth leaves.

max_epochsint, default=40

Number of training epochs.

learning_ratefloat, default=0.01

Learning rate for Adam optimizer.

batch_sizeint, default=64

Mini-batch size for training.

penalty_coeffloat, default=1e-3

Regularization coefficient for the entropy penalty on internal nodes. Higher values encourage more balanced splits.

devicestr, default=”cpu”

PyTorch device. “auto” picks CUDA if it is available, then Apple silicon’s MPS, then CPU. Anything else is passed to torch as given, so “cuda:1” works. Resolved once in fit and recorded as device_, so prediction always runs where training did.

verbosebool, default=False

Whether to print training progress.

random_stateint or None, default=None

Seed for model initialization and shuffled mini-batches.

class_weightdict, “balanced” or None, default=None

Weights per class, combined multiplicatively with sample_weight. “balanced” uses n_samples / (n_classes * bincount(y)), which is what an imbalanced target usually needs: without it a rare class contributes so little to the loss that the tree can ignore it entirely.

warm_startbool, default=False

When True, a second call to fit continues from the parameters the first one left, instead of reinitializing. Useful for training in stages, or for extending a run that turned out too short.

This is warm_start rather than partial_fit deliberately. sklearn’s partial_fit contract promises that a model updated on batches approaches one trained on the union, and requires handling classes that were absent from the first call. Neither holds here: the architecture is fixed at the first fit, and mini-batch gradient descent over a second dataset drifts toward that dataset rather than the union. warm_start promises only what is actually delivered, which is continuation.

The label set must not change between calls; a new class would need an output layer this model cannot grow.

growth{“none”, “incremental”, “per_leaf”}, default=”none”

How the tree reaches its shape.

  • "none" builds the complete tree of depth depth up front, which is the Frosst & Hinton (2017) formulation.

  • "incremental" starts from a single split and deepens one level at a time, keeping a level only if it improves validation loss (Irsoy, Yildiz & Alpaydin, ICPR 2012). depth becomes an upper bound and tree_depth_ reports what was actually kept. Requires a validation split, so validation_fraction applies whether or not early_stopping is on, and the max_epochs budget is divided across rounds rather than spent per round. That last point matters in practice: a budget that trains a fixed tree adequately can leave an incremental one under-trained, so raise max_epochs when switching.

  • "per_leaf" splits one leaf at a time, the one carrying the most expected error, so the tree can end up unbalanced and spend depth only where the data needs it. This is the growth rule of İrsoy, Yıldız & Alpaydın (ICPR 2012); level-wise growth was the tractable approximation of it.

    It produces by far the sparsest trees, and wins where a fixed depth over-parameterizes. 3 seeds of 5-fold CV, accuracy and splits kept:

    none / incremental / per_leaf Iris 0.958 / 15 0.931 / 15 0.880 / 7.7 Wine 0.977 / 15 0.981 / 15 0.966 / 5.2 Breast Cancer 0.971 / 15 0.971 / 9.9 0.978 / 4.0 synthetic 20d 0.839 / 63 0.881 / 13 0.885 / 3.7

    On the synthetic problem it reaches better accuracy than a fixed depth-6 tree using 3.7 splits against 63. On Iris it loses, which is why the default is still “none”.

learn_temperaturebool, default=False

Learn a per-node inverse temperature on the gate, so a node can sharpen its split instead of saturating in the flat part of the sigmoid (Frosst & Hinton, 2017). Off by default because the effect is mixed: averaged over 5 seeds of 5-fold CV at depth 4 it moved Iris from 0.900 to 0.928, and cost about 0.6 points on Wine and Breast Cancer.

early_stoppingbool, default=False

Hold out validation_fraction of the training data and stop once validation loss has not improved for n_iter_no_change epochs. The parameters of the best epoch are restored.

validation_fractionfloat, default=0.1

Fraction held out when early_stopping=True.

n_iter_no_changeint, default=10

Epochs without validation improvement before stopping.

Attributes:
classes_ndarray of shape (n_classes,)

The class labels.

n_features_in_int

Number of features seen during fit.

training_history_list of dict

Loss and accuracy per epoch, plus validation loss when early stopping is on.

feature_importances_ndarray of shape (n_features,)

Gate weight magnitudes, weighted by how much probability mass reaches each node on the training data, normalized to sum to 1.

n_iter_int

Epochs actually run.

tree_depth_int

Depth of the fitted tree. Equals depth unless growth=”incremental” stopped earlier.

growth_str

The growth mode actually used. Falls back to “none” when the data is too small to hold out a stratified validation split.

References

İrsoy, O., Yıldız, O. T., & Alpaydın, E. (2012). Soft Decision Trees. ICPR 2012.

Examples

>>> from neural_trees import SoftDecisionTree
>>> from sklearn.datasets import load_iris
>>> X, y = load_iris(return_X_y=True)
>>> sdt = SoftDecisionTree(depth=4, max_epochs=30)
>>> sdt.fit(X, y)
>>> sdt.score(X, y)
fit(X, y, sample_weight=None) → SoftDecisionTree[source]#

Fit the Soft Decision Tree.

Parameters:
Xarray-like of shape (n_samples, n_features)
yarray-like of shape (n_samples,)
sample_weightarray-like of shape (n_samples,), default=None

Per-sample weights applied to the loss. Combined multiplicatively with class_weight when both are given. The entropy penalty is left unweighted: it regularizes the shape of the tree, not the fit to any particular sample.

Weighting a sample by k gives the same loss and the same gradient as repeating it k times, but not bit-for-bit the same fit: the repeated dataset is larger, so mini-batches are composed differently and the optimizer follows a different path. This is why check_sample_weight_equivalence_on_dense_data is the one estimator check this class does not pass (62 of 63), and it is not satisfiable by any stochastic mini-batch learner.

Returns:
self
get_leaf_distributions() → ndarray[source]#

Return the class distribution stored in each leaf node.

Returns:
distributionsndarray of shape (n_leaves, n_classes)
get_split_weights() → List[ndarray][source]#

Return the weight vectors for each internal node’s split.

Returns:
weightslist of ndarray, one per internal node
predict(X) → ndarray[source]#

Predict class labels.

Parameters:
Xarray-like of shape (n_samples, n_features)
Returns:
y_predndarray of shape (n_samples,)
predict_proba(X) → ndarray[source]#

Predict class probabilities.

Each sample reaches every leaf with some probability, so the returned distribution is the path-probability weighted average of the leaf distributions, P(y | x) = sum_l mu_l(x) Q_l(y). This is why the output is smooth rather than the piecewise constant output of a hard tree.

Parameters:
Xarray-like of shape (n_samples, n_features)

Samples to score. Cast to float32 internally, so any numeric dtype is accepted. Must have the same number of features seen in fit.

Returns:
probandarray of shape (n_samples, n_classes)

Class probabilities in the order of self.classes_. Each row sums to 1.

set_fit_request(*, sample_weight: bool | None | str = '$UNCHANGED$') → SoftDecisionTree#

Configure whether metadata should be requested to be passed to the fit method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config()). Please check the User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to fit if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to fit.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters:
sample_weightstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for sample_weight parameter in fit.

Returns:
selfobject

The updated object.

set_score_request(*, sample_weight: bool | None | str = '$UNCHANGED$') → SoftDecisionTree#

Configure whether metadata should be requested to be passed to the score method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config()). Please check the User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to score if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to score.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters:
sample_weightstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for sample_weight parameter in score.

Returns:
selfobject

The updated object.

to_hard_tree()[source]#

Export the trained tree with its gates read as hard decisions.

Each internal node’s gate is sigmoid(beta * (w . x + b)); the sign of w . x + b is the decision it has settled on, and beta only sharpens it. Taking that sign and routing each sample down one path gives a plain numpy model with readable rules and no PyTorch in the prediction path.

This is a different model, not a re-encoding: a mixture over leaves is not a single path, and the two disagree on samples that sit near a split. Measure the agreement on held-out data before relying on it.

Returns:
HardDecisionTree

Examples

>>> hard = sdt.to_hard_tree()
>>> (hard.predict(X_test) == sdt.predict(X_test)).mean()
>>> print(hard.export_text(feature_names=feature_names))