Multi-Class Classification Tutorial with the Keras Deep Learning Library

 Keras is a Python library for deep learning that wraps the efficient numerical libraries Theano and TensorFlow.

In this tutorial, you will discover how to use Keras to develop and evaluate neural network models for multi-class classification problems.

After completing this step-by-step tutorial, you will know:

• How to load data from CSV and make it available to Keras
• How to prepare multi-class classification data for modeling with neural networks
• How to evaluate Keras neural network models with scikit-learn

  1. Problem Description

  In this tutorial, you will use the standard machine learning problem called the iris flowers dataset.

  This dataset is well studied and makes a good problem for practicing on neural networks because all four input variables are numeric and have the same scale in centimeters. Each instance describes the properties of an observed flower’s measurements, and the output variable is a specific iris species.

  This is a multi-class classification problem, meaning that there are more than two classes to be predicted. In fact, there are three flower species. This is an important problem for practicing with neural networks because the three class values require specialized handling.

  The iris flower dataset is a well-studied problem, and as such, you can expect to achieve a model accuracy in the range of 95% to 97%. This provides a good target to aim for when developing your models.

  You can download the iris flowers dataset from the UCI Machine Learning repository and place it in your current working directory with the filename “iris.csv“.

  • Iris Flowers Dataset (iris.csv)

  Need help with Deep Learning in Python?

  Take my free 2-week email course and discover MLPs, CNNs and LSTMs (with code).

  Click to sign-up now and also get a free PDF Ebook version of the course.

  2. Import Classes and Functions

  You can begin by importing all the classes and functions you will need in this tutorial.

  This includes both the functionality you require from Keras and the data loading from pandas, as well as data preparation and model evaluation from scikit-learn.

  import pandas from keras.models import Sequential from keras.layers import Dense from keras.wrappers.scikit_learn import KerasClassifier from keras.utils import np_utils from sklearn.model_selection import cross_val_score from sklearn.model_selection import KFold from sklearn.preprocessing import LabelEncoder from sklearn.pipeline import Pipeline ...

  3. Load the Dataset

  The dataset can be loaded directly. Because the output variable contains strings, it is easiest to load the data using pandas. You can then split the attributes (columns) into input variables (X) and output variables (Y).

  ... # load dataset dataframe = pandas.read_csv("iris.csv", header=None) dataset = dataframe.values X = dataset[:,0:4].astype(float) Y = dataset[:,4]

  4. Encode the Output Variable

  The output variable contains three different string values.

  When modeling multi-class classification problems using neural networks, it is good practice to reshape the output attribute from a vector that contains values for each class value to a matrix with a Boolean for each class value and whether a given instance has that class value or not.

  This is called one-hot encoding or creating dummy variables from a categorical variable.

  For example, in this problem, three class values are Iris-setosa, Iris-versicolor, and Iris-virginica. If you had the observations:

  Iris-setosa Iris-versicolor Iris-virginica

  You can turn this into a one-hot encoded binary matrix for each data instance that would look like this:

  Iris-setosa, Iris-versicolor, Iris-virginica 1, 0, 0 0, 1, 0 0, 0, 1

  You can first encode the strings consistently to integers using the scikit-learn class LabelEncoder. Then convert the vector of integers to a one-hot encoding using the Keras function to_categorical().

  ... # encode class values as integers encoder = LabelEncoder() encoder.fit(Y) encoded_Y = encoder.transform(Y) # convert integers to dummy variables (i.e. one hot encoded) dummy_y = np_utils.to_categorical(encoded_Y)

  5. Define the Neural Network Model

  If you are new to Keras or deep learning, see this helpful Keras tutorial.

  The Keras library provides wrapper classes to allow you to use neural network models developed with Keras in scikit-learn.

  There is a KerasClassifier class in Keras that can be used as an Estimator in scikit-learn, the base type of model in the library. The KerasClassifier takes the name of a function as an argument. This function must return the constructed neural network model, ready for training.

  Below is a function that will create a baseline neural network for the iris classification problem. It creates a simple, fully connected network with one hidden layer that contains eight neurons.

  The hidden layer uses a rectifier activation function which is a good practice. Because you used a one-hot encoding for your iris dataset, the output layer must create three output values, one for each class. The output value with the largest value will be taken as the class predicted by the model.

  The network topology of this simple one-layer neural network can be summarized as follows:

  4 inputs -> [8 hidden nodes] -> 3 outputs

  Note that a “softmax” activation function was used in the output layer. This ensures the output values are in the range of 0 and 1 and may be used as predicted probabilities.

  Finally, the network uses the efficient Adam gradient descent optimization algorithm with a logarithmic loss function, which is called “categorical_crossentropy” in Keras.

  ... # define baseline model def baseline_model(): # create model model = Sequential() model.add(Dense(8, input_dim=4, activation='relu')) model.add(Dense(3, activation='softmax')) # Compile model model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy']) return model

  You can now create your KerasClassifier for use in scikit-learn.

  You can also pass arguments in the construction of the KerasClassifier class that will be passed on to the fit() function internally used to train the neural network. Here, you pass the number of epochs as 200 and batch size as 5 to use when training the model. Debugging is also turned off when training by setting verbose to 0.

  ... estimator = KerasClassifier(build_fn=baseline_model, epochs=200, batch_size=5, verbose=0)

  6. Evaluate the Model with k-Fold Cross Validation

  You can now evaluate the neural network model on our training data.

  The scikit-learn has excellent capability to evaluate models using a suite of techniques. The gold standard for evaluating machine learning models is k-fold cross validation.

  First, define the model evaluation procedure. Here, you set the number of folds to 10 (an excellent default) and shuffle the data before partitioning it.

  ... kfold = KFold(n_splits=10, shuffle=True)

  Now, you can evaluate your model (estimator) on your dataset (X and dummy_y) using a 10-fold cross-validation procedure (k-fold).

  Evaluating the model only takes approximately 10 seconds and returns an object that describes the evaluation of the ten constructed models for each of the splits of the dataset.

  ... results = cross_val_score(estimator, X, dummy_y, cv=kfold) print("Baseline: %.2f%% (%.2f%%)" % (results.mean()*100, results.std()*100))

  7. Complete Example

  You can tie all of this together into a single program that you can save and run as a script:

  # multi-class classification with Keras import pandas from keras.models import Sequential from keras.layers import Dense from keras.wrappers.scikit_learn import KerasClassifier from keras.utils import np_utils from sklearn.model_selection import cross_val_score from sklearn.model_selection import KFold from sklearn.preprocessing import LabelEncoder from sklearn.pipeline import Pipeline # load dataset dataframe = pandas.read_csv("iris.data", header=None) dataset = dataframe.values X = dataset[:,0:4].astype(float) Y = dataset[:,4] # encode class values as integers encoder = LabelEncoder() encoder.fit(Y) encoded_Y = encoder.transform(Y) # convert integers to dummy variables (i.e. one hot encoded) dummy_y = np_utils.to_categorical(encoded_Y)   # define baseline model def baseline_model(): # create model model = Sequential() model.add(Dense(8, input_dim=4, activation='relu')) model.add(Dense(3, activation='softmax')) # Compile model model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy']) return model   estimator = KerasClassifier(build_fn=baseline_model, epochs=200, batch_size=5, verbose=0) kfold = KFold(n_splits=10, shuffle=True) results = cross_val_score(estimator, X, dummy_y, cv=kfold) print("Baseline: %.2f%% (%.2f%%)" % (results.mean()*100, results.std()*100))

  The results are summarized as both the mean and standard deviation of the model accuracy on the dataset.

  Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.

  This is a reasonable estimation of the performance of the model on unseen data. It is also within the realm of known top results for this problem.

  Accuracy: 97.33% (4.42%)

  Summary

  In this post, you discovered how to develop and evaluate a neural network using the Keras Python library for deep learning.

  By completing this tutorial, you learned:

  • How to load data and make it available to Keras
  • How to prepare multi-class classification data for modeling using one-hot encoding
  • How to use Keras neural network models with scikit-learn
  • How to define a neural network using Keras for multi-class classification
  • How to evaluate a Keras neural network model using scikit-learn with k-fold cross validation

  Do you have any questions about deep learning with Keras or this post?

  Ask your questions in the comments below, and I will do my best to answer them.