Sklearn notes
From raju
Decision Trees
Notes
The sklearn library provides two ways to do decision trees - classification based, regression based.
- http://scikit-learn.org/stable/modules/tree.html - tutorial on running decision trees.
Pros: well written, easy to follow, contains sample code.
Issues: Does not show how to do it with pandas dataframes, does not show how to handle categorical variables. Only shows how to export the decision tree to an image and does not show how to print it in text format.
- http://chrisstrelioff.ws/sandbox/2015/06/08/decision_trees_in_python_with_scikit_learn_and_pandas.html - tells how to construct decision trees in python using sklearn and pandas.
Pros: well written, easy to follow. It helped me to get decision trees up and running in no time.
Cons: In the preprocessing step, the code uses an encode_target() function to handle categorical columns. But this approach has multiple problems.
- The function "encodes" different "states" in the "target" column with integers. But those different "states" need not have a monotonic relationship. For example, if the "target" column consists of RED, BLUE, GREEN states and if we assign 1, 2,3 to them then there could be some artifacts since the mapping inherently assumes that RED < BLUE < GREEN and BLUE = AVERAGE(RED, GREEN) etc.,
- The function cannot handle multiple target variables.
A better way of handling categorical columns is to use pd.get_dummies( df[features] ) and create a dummy variable for each "state" in each "target" column.
API Documentation:
- http://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html
- http://scikit-learn.org/stable/modules/classes.html#module-sklearn.tree
Related Links:
- http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelEncoder.html
- http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html
- http://scikit-learn.org/stable/modules/preprocessing.html
handling categorical variables
- To handle categorical variables, create dummy variables for each state of a categorical variable by using pd.get_dummies( df[features] )
internal structure
The inspect module can be used to get the elements of a tree object
from inspect import getmembers print( getmembers( clf.tree_ ) )
traverse a decision tree
- http://scikit-learn.org/stable/auto_examples/tree/plot_unveil_tree_structure.html - shows how to traverse each node of a decision tree, get depth of each node. The decision_path function did not work for me on sklearn 0.17.1
- http://stackoverflow.com/questions/20224526/how-to-extract-the-decision-rules-from-scikit-learn-decision-tree - shows how to print the decision tree in a human readable manner on screen.
- http://stackoverflow.com/questions/20156951/how-do-i-find-which-attributes-my-tree-splits-on-when-using-scikit-learn - shows the internal variables of the decision tree model.
- https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/tree/export.py - source code of export_graphviz function
- http://nbviewer.jupyter.org/gist/aflaxman/d20c723f75d336865940 - contains code to print a tree
external links
- http://scikit-learn.org/stable/modules/preprocessing.html#encoding-categorical-features - tells how to use the OneHotEncoder. Similar functionality can also be achieved by pd.get_dummies
- https://en.wikipedia.org/wiki/Decision_tree