I Built neural-trees. It Is Ready for Other People. Here Is What It Does and What It Does Not.
The library started with a textbook. Ethem Alpaydin's Introduction to Machine Learning has a chapter on decision trees that stops where the interesting part begins: trees whose splits are weighted sums of features rather than one feature at a time, trees whose splits are soft, and models that decide their own size while they train. His group published each of these over the years. None of them had a pip install. If you wanted to put a soft decision tree next to scikit-learn's CART on your own data, you wrote it yourself, and then you had no way of knowing whether it was your version or the paper's that did not work.
So I wrote it. The first version went to PyPI in April; the current one is 0.7.0. This post is an account of what it is for, one step at a time, and of where it does not help, because that second part is the one a reader can actually use.
One boundary first. Every number below comes from a script in the repository and can be re-run. Where a design choice was mine rather than the paper's, the choice was measured, and the measurements are on the documentation site, including the ones that went against me.
Step 1. What a decision tree does, and what a soft one changes
A decision tree asks one question at a time. Is the loan longer than 24 months? Go left or right. Then another question, and another, until a leaf gives the answer. That is why people trust trees: the path from the root to the leaf is the reason for the decision, and you can read it aloud to the person it concerns. It is also why trees are weak. One feature per question means every boundary is a step along an axis, and two classes separated by a diagonal line take a staircase of many steps to fit.
The left panel of Figure 1 is that staircase. The middle panel is a multivariate tree: each question is about a weighted sum of the features, so a cut can be a straight line at any angle, and two cuts do what the staircase needed a dozen for. The right panel is the soft tree, and it changes one more thing. At each gate a point is not sent left or right. A share of it goes each way, the share being a sigmoid of the weighted sum, and the same happens at the next gate, so every point flows down every path in proportion and the prediction is the mixture of all the leaves. Because every piece of that is differentiable, the whole tree is trained by gradient descent, the way a neural network is. That is the "neural" in the name, and it is why the boundary comes out smooth.
The training accuracies are printed above the panels, and the staircase wins them, 0.97 against 0.88. Training accuracy rewards the model that memorises, so this is the first number in the post to read with suspicion. The comparisons that matter come later, on data the models never saw.
Step 2. What is in the box
Five models, one scikit-learn interface (fit, predict, predict_proba, class and sample weights), and a regressor. In the order I would show them to someone:
- SoftDecisionTree. The model in the right panel, from Irsoy, Yildiz and Alpaydin. It can be handed a depth, or told to grow one leaf at a time and stop when a held-out split says so. There is a regression version.
- MultivariateDecisionTree. The middle panel: straight cuts, hard decisions, readable weights.
- OmnivariateDecisionTree, from Yildiz and Alpaydin. At each node it chooses between a single-feature split, a linear one and a nonlinear one by a statistical test, so the tree is only as complicated as the data at that node requires.
- HierarchicalMixtureOfExperts, from Jordan and Jacobs. A soft tree whose leaves are small models rather than constants.
- GALNetwork, from Alpaydin. A network that adds a unit when it makes a mistake and removes units it no longer needs, so it decides its own size.
Alongside them is the combined 5x2cv F test, also Alpaydin's, because the difference between two models on a single train-test split is mostly noise and I wanted the library to say so rather than leave it to the user.
Step 3. The part that took the months
Four of the models did not work when first written, and I mean did not work, not "were slow". The mixture of experts could not take a single gradient step. The omnivariate tree returned the same leaf for every input. GAL scored at chance on data a straight line separates. Each one was diagnosed, fixed, and given a regression test that fails against the old code, so the fault cannot come back quietly.
Then there are the choices the papers leave open, and the library has to make them. The one I would tell first is how a soft tree should grow. The natural way to deepen a tree is to split a leaf into two children and give both the parent's distribution, so that nothing changes at the moment of the split and training carries on. That is what I did, and the new gate never learned anything. The reason is not that its gradient was small. When both children predict exactly the same thing, the gradient of the gate that chooses between them is exactly zero; there is nothing for it to move towards. The fix is one line: start the two children from the residual, the part the parent gets wrong, pushed in opposite directions. The proof and the measurements are in the paper that accompanies the library, and the same defect turned up again in the per-leaf growth and was fixed there.
The others were settled the same way, by running the alternatives on the benchmark datasets and keeping what won: growth stops on held-out error rather than on a fixed threshold; new GAL units start from the residual; a temperature on the soft gates exists but is off by default, because its measured effect was mixed. Every one of these has a table on the documentation site, and the alternatives that did not help are listed as not helping, not deleted.
Step 4. Reading the model back
Figure 2 is what the tree from Figure 1 actually is, and it is small: seven gates, eight leaves, two weights each. This is the drawing the playground shows for any model on any of its datasets. For a single prediction, explain() walks the same structure and reports, in this order:
- the predicted class and its probability;
- the leaf that received most of that row, and how much of it;
- every gate on the way to that leaf, with the direction taken, the probability of that direction, and the two or three features that decided it;
- the smallest single-feature change that flips the predicted class. The library does not guess this. It moves the feature, re-predicts, and reports the change only if the class actually changes; when no single change on that path flips it, it says so.
Two exports go with that. The fitted tree can be read as hard rules, every gate taken as a yes-or-no decision, which is a different model and is reported as one: the library measures how often the rules agree with the tree they came from instead of assuming they do. And the tree can be written to a JSON file that predicts with numpy alone, and to ONNX, where a prediction of one row takes about 11 microseconds on one CPU thread and needs neither torch nor the library. The torch model takes 91 on the same machine.
Step 5. Where it wins and where it loses
Nine models, 24 datasets (the usual scikit-learn ones and twenty from OpenML, capped at 5 000 rows), three seeds of five-fold cross-validation, and nothing tuned: every model in one fixed configuration everywhere, the soft models and XGBoost, LightGBM, GRANDE and NODE alike.
Over all 24, the boosted ensembles, Random Forest and a plain MLP lead, and the soft models sit a point or two behind. Split the datasets by size and the order reverses. On the eight with at most 1 000 rows, GAL beats untuned XGBoost on every one, by 2.9 points on average, and the per-leaf soft tree beats it on seven of the eight, by 2.1. Above 1 000 rows XGBoost wins nearly every comparison.
Three caveats, because the paragraph above is the one people will quote. The split at 1 000 rows was chosen after I had seen the data. The MLP wins on the small datasets too, so the finding is that smooth models trained by gradient descent beat untuned boosting on small tables, and what the soft tree adds is that it can be read. And "untuned" is doing real work in that sentence; a tuned LightGBM would close some of the gap, and I have not measured how much.
So the rule is short. If you have thousands of rows and the only thing you care about is accuracy, use LightGBM.
Step 6. One table where every decision has to be explained
The case I built the library for is the other one, so I ran it end to end on the public German Credit data: 1 000 loan applicants, 20 attributes, 700 repaid and 300 did not. The dataset ships a cost matrix. Accepting an applicant who turns out bad costs 5, refusing one who would have been good costs 1. Follow that through and the decision that minimises expected cost is not "refuse if P(bad) is above one half" but "refuse if P(bad) is above one sixth", because a missed bad applicant costs five times a refused good one. That threshold is where everything below is measured.
Same folds for every model, three seeds of five, nothing tuned, cost per applicant at the one-sixth threshold:
| Model | Accuracy | AUC | Cost per applicant |
|---|---|---|---|
| Soft tree, grown per leaf | 0.747 | 0.786 | 0.548 |
| Logistic regression | 0.747 | 0.787 | 0.536 |
| Random forest | 0.763 | 0.794 | 0.544 |
| XGBoost | 0.749 | 0.775 | 0.625 |
| LightGBM | 0.746 | 0.775 | 0.710 |
The soft tree and logistic regression, which is what a credit scorecard is underneath, cannot be told apart on this data. Random forest is 1.6 points more accurate and no cheaper, and it refuses 71% of applicants to get there against the soft tree's 59%. The two boosted models match everyone on accuracy and lose on cost, and the reason is worth following, because it is the whole argument for calibration. Boosting pushes its probabilities towards 0 and 1. On this data LightGBM put 604 of the 1 000 applicants below 0.1 and 16% of those turned out bad; the soft tree's probabilities track the observed rate bin by bin, with an expected calibration error of 0.03 against LightGBM's 0.16. At a threshold of one half that does not matter much. At one sixth, where the business decision is, it is the difference between the two cost columns.
What the soft tree adds over the scorecard is what steps 3 and 4 were for. The final model has two soft splits and three leaves; read as hard rules they agree with it on 92% of held-out applicants. For any single applicant there is the path, and there is a counterfactual a real person could act on. For the applicant sitting right at the threshold, changing the job category from "skilled" to "unskilled resident" moves the decision across it, and the report says so in the table's own words rather than in standardised units. The report also slices the decisions by age, sex and residency, because a reviewer will: applicants under 25 are refused 77% of the time against 56% for the rest, and the same is true of logistic regression, so it is the data and not the model. Everything, from the fold-level numbers to the model document, regenerates from one script in cases/german-credit.
Who this is for
Three situations, and the answer is different in each.
You have thousands of rows and you are scored on accuracy. Not this. Use LightGBM, and use the step 5 numbers if anyone asks you why.
You have a few hundred to a few thousand rows and every decision has to be explained to someone: a credit refusal, a clinical flag, an audit finding. This is what the library is for, and ten minutes is enough to find out whether it fits. Upload your own CSV or Excel file to the playground at neural-trees.streamlit.app; it runs the step 6 comparison on your table, shows the rules and the explanations, and gives you the JSON and ONNX files. Nothing you upload is stored.
You teach or study these models. Every one of them is one import away, with the paper it came from named in the docstring and the places where the implementation deviates from it written down with numbers.
- Code: github.com/cgrtml/neural-trees
- Install:
pip install neural-trees - Docs: cagritemel.com/neural-trees
- Archived release: doi.org/10.5281/zenodo.22718897
If you have a table of the second kind, I will run the German Credit package on it, free, within two weeks; the report linked above is exactly what you would get back. Issues and pull requests are welcome, and the ones opened so far all came with a measured report, which is how I would like it to stay.