neural_trees.GALNetwork#

class neural_trees.GALNetwork(initial_hidden: int = 2, max_hidden: int = 50, grow_threshold: float = 0.1, prune_threshold: float = 0.0001, max_epochs: int = 100, learning_rate: float = 0.01, check_interval: int = 5, growth_policy: str = 'error_threshold', growth_init: str = 'residual', n_candidates: int = 4, candidate_epochs: int = 40, validation_fraction: float = 0.2, tol: float = 0.01, patience: int = 5, error_patience: int = 2, batch_size: int = 32, momentum: float = 0.9, device: str = 'cpu', verbose: bool = False, random_state: int | None = None, class_weight=None, warm_start: bool = False)[source]#

Bases: ClassifierMixin, BaseEstimator

GAL (Grow and Learn) Constructive Neural Network.

Starts with a minimal network and adds hidden units automatically when learning stagnates, prunes them when they become redundant.

Parameters:
initial_hiddenint, default=2

Initial number of hidden units.

max_hiddenint, default=50

Maximum hidden units before stopping growth.

grow_thresholdfloat, default=0.1

Training-error threshold above which a new unit is added. Only used by the "error_threshold" policy.

prune_thresholdfloat, default=1e-4

Activation variance below which a unit is pruned. Only used by the "error_threshold" policy.

max_epochsint, default=100

Maximum training epochs.

learning_ratefloat, default=0.01
check_intervalint, default=5

How often (in epochs) to reconsider the architecture.

growth_policy{“error_threshold”, “validation”}, default=”error_threshold”

How growth and pruning decide.

  • "validation" holds out validation_fraction of the training data and changes the architecture only when validation loss has stopped improving. A unit is pruned when removing it does not hurt validation loss, and one is grown otherwise. Training stops after patience consecutive changes that fail to improve validation loss, and the parameters of the best epoch are restored.

  • "error_threshold" is the earlier behaviour: prune any unit whose activation variance falls below prune_threshold, otherwise grow whenever training error exceeds grow_threshold. That rule grows the network until it fits the training set, with nothing held out to say whether the extra capacity helped.

Falls back to "error_threshold" when the data is too small to hold out a usable validation split; growth_policy_ records what was used.

"validation" is not the default yet. It reaches far smaller networks for the same accuracy where capacity is not the constraint (Breast Cancer: 0.977 with 3.9 units against 0.975 with 2.0), but it under-grows where capacity is the constraint (6 separable blobs: 0.620 with 4.1 units against 0.967 with 15.1). The cause is measured, not guessed: a capacity-starved network keeps improving its validation loss slowly, so “loss is still falling” never signals that more units are what is missing. Deciding this properly needs a signal about what a new unit would buy, which a randomly initialized unit cannot provide.

growth_init{“residual”, “random”}, default=”residual”

How a new hidden unit is initialized.

  • "residual" fits the unit to what the frozen network still gets wrong, maximizing the correlation between its activation and the residual error (Fahlman & Lebiere, 1990), and gives it zero outgoing weights so the network’s function is unchanged at the moment of growth.

  • "random" is the earlier behaviour: random incoming and outgoing weights, which perturbs every logit on arrival.

n_candidatesint, default=4

Candidate units trained in parallel at each growth step; the one that correlates best with the residual is installed. Ignored when growth_init=”random”.

candidate_epochsint, default=40

Gradient steps spent fitting each candidate. Ignored when growth_init=”random”.

validation_fractionfloat, default=0.2

Fraction held out under the "validation" policy.

tolfloat, default=1e-2

Relative improvement in validation loss that counts as progress. A loss still creeping down by a fraction of a percent per check is a network that has stopped learning anything useful with the capacity it has, and treating that as progress is what keeps it from ever growing.

patienceint, default=5

Consecutive architecture changes without a validation improvement before training stops.

error_patienceint, default=2

Checks without any improvement in validation error before the architecture is reconsidered, even while validation loss is still falling. A network that has run out of capacity keeps sharpening the same mistakes, which shows up as a falling loss over a flat error.

batch_sizeint, default=32

Mini-batch size. Training used to take a single full-batch step per epoch, which left the network barely moved from its initialization when the growth criterion was evaluated.

momentumfloat, default=0.9

Momentum for the SGD optimizer.

devicestr, default=”cpu”

PyTorch device. “auto” picks CUDA if it is available, then Apple silicon’s MPS, then CPU. Resolved once in fit and recorded as device_.

verbosebool, default=False
random_stateint or None, default=None

Seed for weight initialization and for the units added during growth. Set it for reproducible architectures.

warm_startbool, default=False

When True, a second call to fit continues from the network the first one left, keeping both its weights and the architecture growth chose, instead of restarting from initial_hidden.

class_weightdict, “balanced” or None, default=None

Weights per class, combined multiplicatively with sample_weight. “balanced” uses n_samples / (n_classes * bincount(y)). Without it a rare class contributes so little loss that the model can ignore it.

References

Alpaydın, E. (1994). GAL: Networks that Grow when they Learn and Shrink when they Forget. IJPRAI, 8, 391-414.

fit(X, y, sample_weight=None) → GALNetwork[source]#

Fit the network, growing and pruning hidden units as it trains.

Parameters:
Xarray-like of shape (n_samples, n_features)
yarray-like of shape (n_samples,)
Returns:
self
predict_proba(X) → ndarray[source]#

Predict class probabilities, shape (n_samples, n_classes).

The forward pass runs in float64 on the CPU so that a prediction is a property of the sample rather than of its position in the batch. Training stays float32 on whichever device was chosen.

set_fit_request(*, sample_weight: bool | None | str = '$UNCHANGED$') → GALNetwork#

Configure whether metadata should be requested to be passed to the fit method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config()). Please check the User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to fit if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to fit.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters:
sample_weightstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for sample_weight parameter in fit.

Returns:
selfobject

The updated object.

set_score_request(*, sample_weight: bool | None | str = '$UNCHANGED$') → GALNetwork#

Configure whether metadata should be requested to be passed to the score method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config()). Please check the User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to score if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to score.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters:
sample_weightstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for sample_weight parameter in score.

Returns:
selfobject

The updated object.