How to Use Decision Trees the Right Way

Decision tree is a sort of regulated learning algorithm (possessing a target variable that is pre-defined) that is generally utilized in characterization issues. It works for both categorical and continuous input and result factors.

Suppose, we have 30 children with following factors: Class(IX/X), Gender(Girl/Boy) and Height (5 – 6 feet). 15 out of these 30, play badminton in recreation time. We need to make a model to foresee who will play badminton in their recreation period? In this issue, we have to isolate children who play badminton in their recreation time in light of very critical input variable among each of the three.

This is when the decision tree enters, it isolates the students in light of values of the three variables and distinguishes the variable which makes the best homogeneous arrangements of students (which are heterogeneous to each other). In the observations, you can see that the variable ‘Gender’ has best recognized homogeneous sets contrasted with the other two factors.

As specified above, decision tree distinguishes the most critical variable and its value that gives best homogeneous arrangements of the populace. Presently the question which emerges is, how can it recognize the variable and the parting? To do this, Decision tree utilizes different algorithms, which we will examine in the below article.

Types of Decision Trees

Types of Decision tree depends on the kind of target variable we have. It can be of two types:

  1. Categorical Variable Decision Tree: Decision Tree with categorical target variable is called as categorical variable Decision tree. In the above situation of student issue, where the objective variable was will play cricket or not” i.e. YES or NO.
  2. Continuous Variable Decision Tree: If the Decision Tree has consistent target variable then it is called as Continuous Variable Decision Tree. Case: Let’s say we have an issue to foresee whether a client will pay his renewal premium with an insurance agency (No/Yes). Here we realize that income of client is a huge variable yet insurance agency does not have income details for all clients. Presently, as we are aware this is a vital variable, we can construct a Decision tree to anticipate client income in light of occupation, product, and other different variables. For this situation, we are foreseeing values for a constant variable.

Relapse Trees versus Classification Trees

The terminal nodes (or leaves) lie at the base of the Decision tree. This implies that decision trees are commonly drawn inverted, that the leaves form the base and roots are at the tops. Both the trees work relatively like each other, let’s take a glance at the essential contrasts and similarity amongst classification and regression trees:

  1. Regression trees are utilized when dependent variable is consistent. Classification trees are utilized when that variable is categorical.
  2. In case of regression tree, the result obtained by terminal nodes in the training info is the mean reaction of perception occurring in that region. Accordingly, if an unseen information perception falls in that area, we’ll make its forecast with mean value.
  3. In case of the classification tree, the result (class) acquired by the terminal node in the training information is the method of perceptions occurring in that locale. Subsequently, if an unseen information perception falls in that region, we’ll influence its forecast with mode value.
  4. Both the trees isolate the predictor space (independent variables) into specific and non-covering districts. To make it simple, you can think about these districts as high dimensional boxes.
  5. Both the trees take after a top-down greedy approach known as recursive binary splitting. We call it ‘top-down’ on the grounds that it starts from the highest point of the tree when every one of the observations is available in a single area and progressively splits the indicator space into new branches down the tree.
  6. This splitting process proceeds until the point when a client-characterized stopping criterion is achieved. For instance, we can specify the calculation to stop once the quantity of observations per node turns out to be under 50.
  7. In both the cases, the split part procedure brings about completely developed trees until the point when the stopping criterion is reached. The completely developed tree is probably going to overfit information, prompting poor accuracy on new, unseen data. This brings ‘pruning’. Pruning is one of the systems utilized to handle overfitting.

How does a tree decide where to split?

The choice of making strategic splits greatly influences a tree’s exactness. The choice criteria are diverse for classification and regression trees.

Decision trees utilize various algorithms to choose to split a node into at least two sub-nodes. The production of sub-nodes expands the homogeneity of the ultimate sub-nodes. As such, we can state that pureness of the node increments concerning to the target variable. Decision tree partitions the nodes on every accessible variable and afterward chooses the split which brings about most homogeneous sub-nodes.

The calculation choice is additionally in view of sort of target variables. How about we take a note at the four most normally utilized algorithms in decision tree:

Gini Index

In this, we select two things from a populace at random then they should be of the same class and likelihood for this is 1 if the populace is entirely pure.

  • It works with unmitigated target variable “Success” or “Failure”.
  • It performs just Binary partitions.
  • Higher the result of Gini higher the homogeneity.
  • CART (Classification and Regression Tree) utilizes Gini technique to make binary partitions.

Method to calculate Gini for a split:

  • Calculate Gini for sub-nodes, utilizing equation total of square of probability for success and failure (p2 + q2).
  • Calculate Gini for split utilizing weighted Gini score of every node of that split.


It is a calculation to discover the measurable importance between the contrasts between sub-nodes and parent node. We measure it by summation of squares of contrasts amongst observed and expected frequencies of target variable.

  • It works with straight out target variable “Success” or “Failure”.
  • It can perform at least two splits.
  • Higher the estimation of Chi-Square higher the measurable importance of contrasts between sub-nodes and Parent node.
  • Chi-Square of every node is figured utilizing the formula: Chi-square = ((Actual – Expected)2/Expected)1/2
  • It creates a tree called CHAID (Chi-square Automatic Interaction Detector)

Method to Calculate Chi-square for a split:

  • Calculate Chi-square for singular nodes by computing the deviation for Success and Failure both
  • Calculated Chi-square of Split Utilizing Sum of all Chi-square of Success and Failure of each and every node of the split

Information Gain

Fewer contaminated nodes require less data to portray it. Also, more unclean nodes require more data. Data hypothesis is a measure to characterize this level of confusion in a framework known as Entropy. On the off chance that the example is totally homogeneous, at that point the entropy is zero and if the example is a similarly partitioned (half – half), it has an entropy of one.

Entropy can be ascertained utilizing equation:
decision trees -magoosh

Here p and q are probabilities of success and failure accordingly in that node. Entropy is additionally utilized with downright target variable. It picks the split which has most minimal entropy contrasted with the parent node and different partitions. The lesser the entropy, the proficient it is.

Methods to figure entropy for a split:

  • Compute the entropy of parent hub
  • Compute entropy of every single node of the split and figure weighted average of all sub-nodes accessible in split.

Reduction in Variance

Till now, we have talked about the calculations for clear cut target variable. Decrement in fluctuations is a calculation utilized for continuous target variables (regression issues). This algorithm utilizes the standard formula of change to pick the best split. The split with lower variance is chosen as the criteria to partition the populace:

Above, X-bar is mean of the results, X is genuine and n is the number of results.

Method to compute Variance:

  • Compute it for every node.
  • Compute it for each split as weighted normal of every node variance.

Avoiding Overfitting in decision trees

Overfitting is one of the key difficulties confronted while displaying decision trees. On the off chance that there is no restriction set of a decision tree, it will give you 100% precision on training set in light of the fact that in the worst case, it will wind up mentioning 1 leaf for every measurement. Subsequently, avoiding overfitting is significant while demonstrating a decision tree and it is possible by below given 2 ways:

  • Setting imperatives on tree structure and size
  • Tree pruning

Let’s examine both of these quickly.

Setting Constraints on Tree Size

This should be possible by utilizing different parameters used to characterize a tree.

The parameters utilized for characterizing a tree are clarified beneath. The parameters depicted beneath are regardless of tools. It is vital to comprehend the part of these parameters utilized as a part of tree modeling.

Least samples for a node split

  • Defines the base number of tests (or samples) which are required in a node to be considered for partition.
  • Keep a check on overfitting. Higher values keep a model from learning relations which may be exceedingly particular to the specific example chosen for a tree.
  • Too high values can prompt under-fitting thus, it ought to be tuned utilizing Cross Validation (CV).

Minimum observations for a terminal node (leaf)

  • Defines the base examples (or samples) required in a terminal node or leaf.
  • Used to control overfitting.
  • Generally lower results ought to be decided for imbalanced class issues in light of the fact that the districts in which the minority class will be in larger part will be little.

Highest depth of tree (vertical depth)

  • The most extreme depth of a tree.
  • Used to control over-fitting as, when the depth is higher, it will enable the model to learn relations certain to a specific example.
  • Should be tuned utilizing CV.

Maximum number of terminal nodes

  • The greatest number of terminal nodes or leaves in a tree.
  • Can be characterized as a set-up of max_depth. Since paired trees are made, a profundity of ‘n’ would deliver a greatest of 2n leaves.

Maximum highlights to consider for split

  • The number of highlights to consider while hunting down the best split. These will be randomly chosen.
  • As a thumb-control, square root computation of the aggregate number of highlights works incredibly; however, we should check up to 30-40% of the aggregate number of highlights.
  • Higher values can prompt over-fitting yet relies upon case to case.

Tree Pruning

As talked about before, the procedure of setting stoppage is a greedy approach. As such, it will check for the best split at that very moment and advance until the point when one of the predefined ceasing conditions is come to.

Case: Let’s consider the accompanying situation when you’re driving:

There are 2 paths:

  • A path with autos moving at 80 km/h
  • A path with trucks moving at 30 km/h

At right now, you are the yellow auto and you have 2 decisions:

  • Using a left and overtake the other 2 autos rapidly
  • Keep moving in current path

Let’s examine these decisions. In the previous decision, you’ll quickly overwhelm the auto ahead and reach behind the truck and begin moving at 30 km/h, searching for a chance to move back right. All autos initially behind you advance in the then. This would be the ideal decision if your goal is to increase the path covered in next say 10 seconds. In the later decision, you deal through at the same speed, cross trucks and after that overtake perhaps relying upon circumstances ahead. Greedy you!

This is precisely the distinction between ordinary decision tree and pruning. A decision tree with limitations won’t spot the truck ahead and embrace a greedy approach by taking a left. Then again on the off chance that we utilize pruning, we, as a result, take a glance at a couple of obstacles ahead and then settle on a decision.

So, we know pruning is better. But how to actualize it in a decision tree? The thought is quite simple.

  • We first settle on the decision tree with a substantial depth.
  • Then we begin at the base and begin evacuating leaves which are giving us negative returns when looked at from the top.
  • Suppose a split is giving us a pick up of say – 10 (loss of 10) and after that the following split on that gives us a pick up of 20. A normal decision tree will stop at stage 1 yet in pruning, we will see that the actual pick up is +10 and keep the two leaves.

Programming languages like Python and R provide great tools and libraries that you can use to efficiently program your own decision tree algorithms on various data-sets.

Comments are closed.

Magoosh blog comment policy: To create the best experience for our readers, we will only approve comments that are relevant to the article, general enough to be helpful to other students, concise, and well-written! 😄 Due to the high volume of comments across all of our blogs, we cannot promise that all comments will receive responses from our instructors.

We highly encourage students to help each other out and respond to other students' comments if you can!

If you are a Premium Magoosh student and would like more personalized service from our instructors, you can use the Help tab on the Magoosh dashboard. Thanks!