neural_trees.HierarchicalMixtureOfExperts#

class neural_trees.HierarchicalMixtureOfExperts(depth: int = 2, branching_factor: int = 2, gate_hidden: int = 32, expert_hidden: int = 64, dropout_rate: float = 0.3, dropout_type: str = 'subtree', max_epochs: int = 50, learning_rate: float = 0.001, batch_size: int = 64, device: str = 'cpu', verbose: bool = False, random_state: int | None = None, class_weight=None, warm_start: bool = False)[source]#

Bases: ClassifierMixin, BaseEstimator

Hierarchical Mixture of Experts with Dropout Regularization.

A tree-structured neural network where gating networks route inputs to expert leaves. Dropout on gating networks prevents co-adaptation and improves generalization.

Parameters:
depthint, default=2

Depth of the expert tree. n_experts = branching_factor^depth.

branching_factorint, default=2

Number of children per gating node.

gate_hiddenint, default=32

Hidden units in each gating network.

expert_hiddenint, default=64

Hidden units in each expert network.

dropout_ratefloat, default=0.3

Dropout probability applied at each gating node during training.

dropout_type{“subtree”, “activation”}, default=”subtree”

Which dropout mechanism to use.

  • "subtree" is the mechanism from Irsoy & Alpaydin (2021): a gating node drops one of its children with probability dropout_rate, so the whole subtree below it receives no probability mass for that sample, and the surviving children are renormalized.

  • "activation" is the earlier behaviour of this class, a plain nn.Dropout on the gating network’s hidden activations. It perturbs a gate but never removes a branch, and measures as close to inert. Kept so results can be compared.

max_epochsint, default=50

Training epochs.

learning_ratefloat, default=1e-3

Adam learning rate.

batch_sizeint, default=64
devicestr, default=”cpu”

PyTorch device. “auto” picks CUDA if it is available, then Apple silicon’s MPS, then CPU. Resolved once in fit and recorded as device_.

verbosebool, default=False
random_stateint or None, default=None

Seed for weight initialization and shuffled mini-batches.

warm_startbool, default=False

When True, a second call to fit continues from the parameters the first one left, instead of reinitializing. Useful for training in stages, or for extending a run that turned out too short.

This is warm_start rather than partial_fit deliberately. sklearn’s partial_fit contract promises that a model updated on batches approaches one trained on the union, and requires handling classes that were absent from the first call. Neither holds here: the architecture is fixed at the first fit, and mini-batch gradient descent over a second dataset drifts toward that dataset rather than the union. warm_start promises only what is actually delivered, which is continuation.

The label set must not change between calls; a new class would need an output layer this model cannot grow.

class_weightdict, “balanced” or None, default=None

Weights per class, combined multiplicatively with sample_weight. “balanced” uses n_samples / (n_classes * bincount(y)). Without it a rare class contributes so little loss that the model can ignore it.

References

İrsoy, O., & Alpaydın, E. (2021). Dropout Regularization in Hierarchical Mixture of Experts. Neurocomputing, 419, 148-156.

Examples

>>> from neural_trees import HierarchicalMixtureOfExperts
>>> from sklearn.datasets import load_digits
>>> X, y = load_digits(return_X_y=True)
>>> moe = HierarchicalMixtureOfExperts(depth=2, branching_factor=4)
>>> moe.fit(X, y)
>>> moe.score(X, y)
predict_proba(X) → ndarray[source]#

Predict class probabilities, shape (n_samples, n_classes).

The forward pass runs in float64 on the CPU, whatever device training used. Rows are independent, but BLAS picks different blocking for different memory layouts, so in float32 the same sample scored inside a reordered batch came out up to 1.2e-07 different and a borderline argmax could flip with it. Doubling the width of the predict-time arithmetic puts that at 2.2e-16, which makes a prediction a property of the sample rather than of its position in the batch. It is pinned to the CPU because MPS has no float64 and CUDA’s is slow. Training stays float32, on whichever device was chosen.

set_fit_request(*, sample_weight: bool | None | str = '$UNCHANGED$') → HierarchicalMixtureOfExperts#

Configure whether metadata should be requested to be passed to the fit method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config()). Please check the User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to fit if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to fit.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters:
sample_weightstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for sample_weight parameter in fit.

Returns:
selfobject

The updated object.

set_score_request(*, sample_weight: bool | None | str = '$UNCHANGED$') → HierarchicalMixtureOfExperts#

Configure whether metadata should be requested to be passed to the score method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config()). Please check the User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to score if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to score.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters:
sample_weightstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for sample_weight parameter in score.

Returns:
selfobject

The updated object.

to_hard_router() → HardRoutedExperts[source]#

Export the trained mixture with its gates read as hard routing.

A trained mixture evaluates every expert for every sample and blends them, so one prediction costs branching_factor^depth expert forward passes. Each gating node has a preferred child for any given input; taking that preference as a decision sends a sample down one path to one expert, in numpy.

This is a different model, not a re-encoding: a blend of experts is not one expert, and the two disagree where the gating was undecided. Measure the agreement on held-out data before relying on it.

Returns:
HardRoutedExperts