Use Keras Deep Learning Models with Scikit-Learn in Python

 Keras is one of the most popular deep learning libraries in Python for research and development because of its simplicity and ease of use.

The scikit-learn library is the most popular library for general machine learning in Python.

In this post, you will discover how you can use deep learning models from Keras with the scikit-learn library in Python.

This will allow you to leverage the power of the scikit-learn library for tasks like model evaluation and model hyper-parameter optimization.

Overview

Keras is a popular library for deep learning in Python, but the focus of the library is deep learning models. In fact, it strives for minimalism, focusing on only what you need to quickly and simply define and build deep learning models.

The scikit-learn library in Python is built upon the SciPy stack for efficient numerical computation. It is a fully featured library for general machine learning and provides many useful utilities in developing deep learning models. Not least of which are:

• Evaluation of models using resampling methods like k-fold cross validation
• Efficient search and evaluation of model hyper-parameters

There was a wrapper in the TensorFlow/Keras library to make deep learning models used as classification or regression estimators in scikit-learn. But recently, this wrapper was taken out to become a standalone Python module.

In the following sections, you will work through examples of using the KerasClassifier wrapper for a classification neural network created in Keras and used in the scikit-learn library.

The test problem is the Pima Indians onset of diabetes classification dataset. This is a small dataset with all numerical attributes that is easy to work with. Download the dataset and place it in your currently working directly with the name pima-indians-diabetes.csv (update: download from here).

The following examples assume you have successfully installed TensorFlow 2.x, SciKeras, and scikit-learn. If you use the pip system for your Python modules, you may install them with:

pip install tensorflow scikeras scikit-learn

Need help with Deep Learning in Python?

Take my free 2-week email course and discover MLPs, CNNs and LSTMs (with code).

Click to sign-up now and also get a free PDF Ebook version of the course.

Evaluate Deep Learning Models with Cross Validation

The KerasClassifier and KerasRegressor classes in SciKeras take an argument model which is the name of the function to call to get your model.

You must define a function called whatever you like that defines your model, compiles it, and returns it.

In the example below, you will define a function create_model() that creates a simple multi-layer neural network for the problem.

You pass this function name to the KerasClassifier class by the model argument. You also pass in additional arguments of nb_epoch=150 and batch_size=10. These are automatically bundled up and passed on to the fit() function, which is called internally by the KerasClassifier class.

In this example, you will use the scikit-learn StratifiedKFold to perform 10-fold stratified cross-validation. This is a resampling technique that can provide a robust estimate of the performance of a machine learning model on unseen data.

Next, use the scikit-learn function cross_val_score() to evaluate your model using the cross-validation scheme and print the results.

# MLP for Pima Indians Dataset with 10-fold cross validation via sklearn from tensorflow.keras.models import Sequential from tensorflow.keras.layers import Dense from scikeras.wrappers import KerasClassifier from sklearn.model_selection import StratifiedKFold from sklearn.model_selection import cross_val_score import numpy as np   # Function to create model, required for KerasClassifier def create_model(): # create model model = Sequential() model.add(Dense(12, input_dim=8, activation='relu')) model.add(Dense(8, activation='relu')) model.add(Dense(1, activation='sigmoid')) # Compile model model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy']) return model   # fix random seed for reproducibility seed = 7 np.random.seed(seed) # load pima indians dataset dataset = np.loadtxt("pima-indians-diabetes.csv", delimiter=",") # split into input (X) and output (Y) variables X = dataset[:,0:8] Y = dataset[:,8] # create model model = KerasClassifier(model=create_model, epochs=150, batch_size=10, verbose=0) # evaluate using 10-fold cross validation kfold = StratifiedKFold(n_splits=10, shuffle=True, random_state=seed) results = cross_val_score(model, X, Y, cv=kfold) print(results.mean())

Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.

Running the example displays the skill of the model for each epoch. A total of 10 models are created and evaluated, and the final average accuracy is displayed.

0.646838691487

In comparison, the following is an equivalent implementation with a neural network model in scikit-learn:

# MLP for Pima Indians Dataset with 10-fold cross validation via sklearn from sklearn.model_selection import StratifiedKFold from sklearn.model_selection import cross_val_score from sklearn.neural_network import MLPClassifier import numpy as np   # fix random seed for reproducibility seed = 7 np.random.seed(seed) # load pima indians dataset dataset = np.loadtxt("pima-indians-diabetes.csv", delimiter=",") # split into input (X) and output (Y) variables X = dataset[:,0:8] Y = dataset[:,8] # create model model = MLPClassifier(hidden_layer_sizes=(12,8), activation='relu', max_iter=150, batch_size=10, verbose=False) # evaluate using 10-fold cross validation kfold = StratifiedKFold(n_splits=10, shuffle=True, random_state=seed) results = cross_val_score(model, X, Y, cv=kfold) print(results.mean())

The role of the KerasClassifier is to work as an adapter to make the Keras model work like a MLPClassifier object from scikit-learn.

Grid Search Deep Learning Model Parameters

The previous example showed how easy it is to wrap your deep learning model from Keras and use it in functions from the scikit-learn library.

In this example, you will go a step further. The function that you specify to the model argument when creating the KerasClassifier wrapper can take arguments. You can use these arguments to further customize the construction of the model. In addition, you know you can provide arguments to the fit() function.

In this example, you will use a grid search to evaluate different configurations for your neural network model and report on the combination that provides the best-estimated performance.

The create_model() function is defined to take two arguments, optimizer and init, both of which must have default values. This will allow you to evaluate the effect of using different optimization algorithms and weight initialization schemes for your network.

After creating your model, define the arrays of values for the parameter you wish to search, specifically:

• Optimizers for searching different weight values
• Initializers for preparing the network weights using different schemes
• Epochs for training the model for a different number of exposures to the training dataset
• Batches for varying the number of samples before a weight update

The options are specified into a dictionary and passed to the configuration of the GridSearchCV scikit-learn class. This class will evaluate a version of your neural network model for each combination of parameters (2 x 3 x 3 x 3 for the combinations of optimizers, initializations, epochs, and batches). Each combination is then evaluated using the default of 3-fold stratified cross validation.

That is a lot of models and a lot of computation. This is not a scheme you want to use lightly because of the time it will take. It may be useful for you to design small experiments with a smaller subset of your data that will complete in a reasonable time. This is reasonable in this case because of the small network and the small dataset (less than 1000 instances and nine attributes).

Finally, the performance and combination of configurations for the best model are displayed, followed by the performance of all combinations of parameters.

# MLP for Pima Indians Dataset with grid search via sklearn from tensorflow.keras.models import Sequential from tensorflow.keras.layers import Dense from scikeras.wrappers import KerasClassifier from sklearn.model_selection import GridSearchCV import numpy as np   # Function to create model, required for KerasClassifier def create_model(optimizer='rmsprop', init='glorot_uniform'):     # create model     model = Sequential()     model.add(Dense(12, input_dim=8, kernel_initializer=init, activation='relu'))     model.add(Dense(8, kernel_initializer=init, activation='relu'))     model.add(Dense(1, kernel_initializer=init, activation='sigmoid'))     # Compile model     model.compile(loss='binary_crossentropy', optimizer=optimizer, metrics=['accuracy'])     return model   # fix random seed for reproducibility seed = 7 np.random.seed(seed) # load pima indians dataset dataset = np.loadtxt("pima-indians-diabetes.csv", delimiter=",") # split into input (X) and output (Y) variables X = dataset[:,0:8] Y = dataset[:,8] # create model model = KerasClassifier(model=create_model, verbose=0) print(model.get_params().keys()) # grid search epochs, batch size and optimizer optimizers = ['rmsprop', 'adam'] init = ['glorot_uniform', 'normal', 'uniform'] epochs = [50, 100, 150] batches = [5, 10, 20] param_grid = dict(optimizer=optimizers, epochs=epochs, batch_size=batches, model__init=init) grid = GridSearchCV(estimator=model, param_grid=param_grid) grid_result = grid.fit(X, Y) # summarize results print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_)) means = grid_result.cv_results_['mean_test_score'] stds = grid_result.cv_results_['std_test_score'] params = grid_result.cv_results_['params'] for mean, stdev, param in zip(means, stds, params):     print("%f (%f) with: %r" % (mean, stdev, param))

Note that in the dictionary param_grid,  model__init was used as the key for the init argument to our create_model() function. The prefix model__ is required for KerasClassifier models in SciKeras to provide custom arguments.

This might take about 5 minutes to complete on your workstation executed on the CPU (rather than GPU). Running the example shows the results below.

Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.

You can see that the grid search discovered that using a uniform initialization scheme, rmsprop optimizer, 150 epochs, and a batch size of 10 achieved the best cross-validation score of approximately 77% on this problem.

For a fuller example of tuning hyperparameters with Keras, see the tutorial:

• How to Grid Search Hyperparameters for Deep Learning Models in Python With Keras

Best: 0.772167 using {'batch_size': 10, 'epochs': 150, 'model__init': 'normal', 'optimizer': 'rmsprop'} 0.691359 (0.024495) with: {'batch_size': 5, 'epochs': 50, 'model__init': 'glorot_uniform', 'optimizer': 'rmsprop'} 0.690094 (0.026691) with: {'batch_size': 5, 'epochs': 50, 'model__init': 'glorot_uniform', 'optimizer': 'adam'} 0.704482 (0.036013) with: {'batch_size': 5, 'epochs': 50, 'model__init': 'normal', 'optimizer': 'rmsprop'} 0.727884 (0.016890) with: {'batch_size': 5, 'epochs': 50, 'model__init': 'normal', 'optimizer': 'adam'} 0.709600 (0.030281) with: {'batch_size': 5, 'epochs': 50, 'model__init': 'uniform', 'optimizer': 'rmsprop'} 0.694067 (0.019733) with: {'batch_size': 5, 'epochs': 50, 'model__init': 'uniform', 'optimizer': 'adam'} 0.720134 (0.038867) with: {'batch_size': 5, 'epochs': 100, 'model__init': 'glorot_uniform', 'optimizer': 'rmsprop'} 0.743536 (0.022213) with: {'batch_size': 5, 'epochs': 100, 'model__init': 'glorot_uniform', 'optimizer': 'adam'} 0.744809 (0.022928) with: {'batch_size': 5, 'epochs': 100, 'model__init': 'normal', 'optimizer': 'rmsprop'} 0.734360 (0.024717) with: {'batch_size': 5, 'epochs': 100, 'model__init': 'normal', 'optimizer': 'adam'} 0.742127 (0.035137) with: {'batch_size': 5, 'epochs': 100, 'model__init': 'uniform', 'optimizer': 'rmsprop'} 0.729225 (0.035580) with: {'batch_size': 5, 'epochs': 100, 'model__init': 'uniform', 'optimizer': 'adam'} 0.717537 (0.033634) with: {'batch_size': 5, 'epochs': 150, 'model__init': 'glorot_uniform', 'optimizer': 'rmsprop'} 0.724039 (0.033990) with: {'batch_size': 5, 'epochs': 150, 'model__init': 'glorot_uniform', 'optimizer': 'adam'} 0.753926 (0.016855) with: {'batch_size': 5, 'epochs': 150, 'model__init': 'normal', 'optimizer': 'rmsprop'} ...

Summary

In this post, you discovered how to wrap your Keras deep learning models and use them in the scikit-learn general machine learning library.

You can see that using scikit-learn for standard machine learning operations such as model evaluation and model hyperparameter optimization can save a lot of time over implementing these schemes yourself.

Wrapping your model allowed you to leverage powerful tools from scikit-learn to fit your deep learning models into your general machine learning process.

Do you have any questions about using Keras models in scikit-learn or about this post? Ask your question in the comments, and I will do my best to answer