Compare metrics such as f1 score and confusion matrices for your machine learning models and through voting or stacking them, then predict on test data with your choice of voting or stacking!
This package is helpful when you are deciding whether to use a single Classifier or combine multiple well-performing Classifiers through an ensemble using Voting or Stacking to yield a more stable and trustworthy classification result. Each of the four functions serves a unique purpose:
confusion_matrices: provides confusion matrices side-by-side for all Classifiers to compare their performances.
compare_f1: provides a Pandas data frame, each row listing model fit time, and training and test scores for each Classifier.
ensemble_compare_f1: provides a Pandas data frame containing model fit time, training and test scores for both Voting and Stacking ensembles, with each ensemble in its own row.
ensemble_predict: provides classification predictions via Voting or Stacking multiple Classifiers.
Before using ensemble_predict on test or unseen data, we recommend that you run each of the three other functions on training data to examine how Classifiers perform individually on their own, and the ensemble performances of Voting against Stacking to make a well-informed decision. Sometimes, an individual Classifier could generate a better controlled machine learning environment if its performance rivals that of an ensemble.
Ke Gao: [email protected] Bryan Lee: [email protected] Susannah Sun: [email protected] Wangkai Zhu
$ pip install compare_classifiersWe have attached a downloaded version of the UCI Wine Quality dataset, named 'example_dataset.csv'. The dataset contains physicochemical features as numerical values of wines and a color column containing text 'red' or 'white' for wine color, and a target variable, column quality, representing wine quality scores, with integer values from 3 to 9 (9 = highest quality; 3 = lowest quality).
compare_classifiers can be used to show confusion matrices and f1 scores for individual estimators, as well as f1 score for voting or stacking the estimators. We made a tutorial with detailed documentation and here is a brief overview of how to use the package:
Note: We changed an original dataset column,
colorcontaining text indicating a wine being 'red' or 'white' into a binary column now namedis_redwith 1 indicating red wine, and 0 indicating white wine. This ensures all columns are numeric and facilitates the upcoming training process for our models.
import pandas as pd
example_dataset = pd.read_csv('../example_dataset.csv')
# convert the `color` column to a binary variable: `is_red`: red = 1, white = 0, and drop the original `color` column
example_dataset['is_red'] = example_dataset['color'].apply(lambda x: 1 if x == 'red' else 0)
example_dataset = example_dataset.drop(['color'], axis=1)
# move the `color` column to the beginning
last_col = example_dataset.pop(example_dataset.columns[-1])
example_dataset.insert(0, last_col.name, last_col)from sklearn.model_selection import train_test_split
train_df, test_df = train_test_split(example_dataset, test_size=0.2)
X_train, X_test, y_train, y_test = (
train_df.drop(columns='quality'), test_df.drop(columns='quality'),
train_df['quality'], test_df['quality']
)from compare_classifiers.confusion_matrices import confusion_matrices
from compare_classifiers.compare_f1 import compare_f1
from compare_classifiers.ensemble_compare_f1 import ensemble_compare_f1
from sklearn.svm import SVC
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.ensemble import RandomForestClassifier
logreg = LogisticRegression(multi_class='multinomial', solver='lbfgs', C=92)
gb = GradientBoostingClassifier(learning_rate=0.001, n_estimators=1000)
svm = SVC(kernel='rbf', decision_function_shape='ovr', max_iter=2000)
rf = RandomForestClassifier(n_estimators=10)
knn5 = KNeighborsClassifier(n_neighbors=5)
estimators = [
('logreg', logreg),
('gb', gb),
('svm', svm),
('rf', rf),
('knn5', knn5)
]
# show confusion matrices for estimators:
confusion_matrices(estimators, X_train, X_test, y_train, y_test)
# show cross validation fit time and f1 scores of each estimator:
compare_f1(estimators, X_train, y_train)
# show cross validation fit time and f1 scores by voting and stacking the estimators:
ensemble_compare_f1(estimators, X_train, y_train) from compare_classifiers.ensemble_predict import ensemble_predict
# predict class labels for unseen data through voting results of estimators:
ensemble_predict(estimators, X_train, y_train, 'voting', X_test) We are not aware of similar packages existing. Though there are available functions to present metrics for a single model and a single ensemble, we have not found functions that compare and display metrics and results for multiple models or ensembles all at once. Neither is there a function that predicts based on dynamic input of ensemble method.
Interested in contributing? Check out the contributing guidelines. Please note that this project is released with a Code of Conduct. By contributing to this project, you agree to abide by its terms.
compare_classifiers was created by Ke Gao, Bryan Lee, Susannah Sun, and Wangkai Zhu. It is licensed under the terms of the MIT license.
compare_classifiers was created with cookiecutter and the py-pkgs-cookiecutter template.