Spot-Check Regression Machine Learning Algorithms in Python with scikit-learn

 Spot-checking is a way of discovering which algorithms perform well on your machine learning problem.

You cannot know which algorithms are best suited to your problem before hand. You must trial a number of methods and focus attention on those that prove themselves the most promising.

In this post you will discover 6 machine learning algorithms that you can use when spot checking your regression problem in Python with scikit-learn.

Algorithms Overview

We are going to take a look at 7 classification algorithms that you can spot check on your dataset.

4 Linear Machine Learning Algorithms:

1. Linear Regression
2. Ridge Regression
3. LASSO Linear Regression
4. Elastic Net Regression

3 Nonlinear Machine Learning Algorithms:

1. K-Nearest Neighbors
2. Classification and Regression Trees
3. Support Vector Machines

Each recipe is demonstrated on a Boston House Price dataset. This is a regression problem where all attributes are numeric (update: download data from here).

Each recipe is complete and standalone. This means that you can copy and paste it into your own project and start using it immediately.

A test harness with 10-fold cross validation is used to demonstrate how to spot check each machine learning algorithm and mean squared error measures are used to indicate algorithm performance. Note that mean squared error values are inverted (negative). This is a quirk of the cross_val_score() function used that requires all algorithm metrics to be sorted in ascending order (larger value is better).

The recipes assume that you know about each machine learning algorithm and how to use them. We will not go into the API or parameterization of each algorithm.

Need help with Machine Learning in Python?

Take my free 2-week email course and discover data prep, algorithms and more (with code).

Click to sign-up now and also get a free PDF Ebook version of the course.

Linear Machine Learning Algorithms

This section provides examples of how to use 4 different linear machine learning algorithms for regression in Python with scikit-learn.

1. Linear Regression

Linear regression assumes that the input variables have a Gaussian distribution. It is also assumed that input variables are relevant to the output variable and that they are not highly correlated with each other (a problem called collinearity).

You can construct a linear regression model using the LinearRegression class.

# Linear Regression import pandas from sklearn import model_selection from sklearn.linear_model import LinearRegression url = "https://raw.githubusercontent.com/jbrownlee/Datasets/master/housing.data" names = ['CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE', 'DIS', 'RAD', 'TAX', 'PTRATIO', 'B', 'LSTAT', 'MEDV'] dataframe = pandas.read_csv(url, delim_whitespace=True, names=names) array = dataframe.values X = array[:,0:13] Y = array[:,13] seed = 7 kfold = model_selection.KFold(n_splits=10, random_state=seed) model = LinearRegression() scoring = 'neg_mean_squared_error' results = model_selection.cross_val_score(model, X, Y, cv=kfold, scoring=scoring) print(results.mean())

Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.

Running the example provides an estimate of mean squared error.

-34.7052559445

2. Ridge Regression

Ridge regression is an extension of linear regression where the loss function is modified to minimize the complexity of the model measured as the sum squared value of the coefficient values (also called the l2-norm).

You can construct a ridge regression model by using the Ridge class.

# Ridge Regression import pandas from sklearn import model_selection from sklearn.linear_model import Ridge url = "https://raw.githubusercontent.com/jbrownlee/Datasets/master/housing.data" names = ['CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE', 'DIS', 'RAD', 'TAX', 'PTRATIO', 'B', 'LSTAT', 'MEDV'] dataframe = pandas.read_csv(url, delim_whitespace=True, names=names) array = dataframe.values X = array[:,0:13] Y = array[:,13] seed = 7 kfold = model_selection.KFold(n_splits=10, random_state=seed) model = Ridge() scoring = 'neg_mean_squared_error' results = model_selection.cross_val_score(model, X, Y, cv=kfold, scoring=scoring) print(results.mean())

Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.

Running the example provides an estimate of the mean squared error.

-34.0782462093

3. LASSO Regression

The Least Absolute Shrinkage and Selection Operator (or LASSO for short) is a modification of linear regression, like ridge regression, where the loss function is modified to minimize the complexity of the model measured as the sum absolute value of the coefficient values (also called the l1-norm).

You can construct a LASSO model by using the Lasso class.

# Lasso Regression import pandas from sklearn import model_selection from sklearn.linear_model import Lasso url = "https://raw.githubusercontent.com/jbrownlee/Datasets/master/housing.data" names = ['CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE', 'DIS', 'RAD', 'TAX', 'PTRATIO', 'B', 'LSTAT', 'MEDV'] dataframe = pandas.read_csv(url, delim_whitespace=True, names=names) array = dataframe.values X = array[:,0:13] Y = array[:,13] seed = 7 kfold = model_selection.KFold(n_splits=10, random_state=seed) model = Lasso() scoring = 'neg_mean_squared_error' results = model_selection.cross_val_score(model, X, Y, cv=kfold, scoring=scoring) print(results.mean())

Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.

Running the example provides an estimate of the mean squared error.

-34.4640845883

4. ElasticNet Regression

ElasticNet is a form of regularization regression that combines the properties of both Ridge Regression and LASSO regression. It seeks to minimize the complexity of the regression model (magnitude and number of regression coefficients) by penalizing the model using both the l2-norm (sum squared coefficient values) and the l1-norm (sum absolute coefficient values).

You can construct an ElasticNet model using the ElasticNet class.

# ElasticNet Regression import pandas from sklearn import model_selection from sklearn.linear_model import ElasticNet url = "https://raw.githubusercontent.com/jbrownlee/Datasets/master/housing.data" names = ['CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE', 'DIS', 'RAD', 'TAX', 'PTRATIO', 'B', 'LSTAT', 'MEDV'] dataframe = pandas.read_csv(url, delim_whitespace=True, names=names) array = dataframe.values X = array[:,0:13] Y = array[:,13] seed = 7 kfold = model_selection.KFold(n_splits=10, random_state=seed) model = ElasticNet() scoring = 'neg_mean_squared_error' results = model_selection.cross_val_score(model, X, Y, cv=kfold, scoring=scoring) print(results.mean())

Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.

Running the example provides an estimate of the mean squared error.

-31.1645737142

Nonlinear Machine Learning Algorithms

This section provides examples of how to use 3 different nonlinear machine learning algorithms for regression in Python with scikit-learn.

1. K-Nearest Neighbors

K-Nearest Neighbors (or KNN) locates the K most similar instances in the training dataset for a new data instance. From the K neighbors, a mean or median output variable is taken as the prediction. Of note is the distance metric used (the metric argument). The Minkowski distance is used by default, which is a generalization of both the Euclidean distance (used when all inputs have the same scale) and Manhattan distance (for when the scales of the input variables differ).

You can construct a KNN model for regression using the KNeighborsRegressor class.

# KNN Regression import pandas from sklearn import model_selection from sklearn.neighbors import KNeighborsRegressor url = "https://raw.githubusercontent.com/jbrownlee/Datasets/master/housing.data" names = ['CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE', 'DIS', 'RAD', 'TAX', 'PTRATIO', 'B', 'LSTAT', 'MEDV'] dataframe = pandas.read_csv(url, delim_whitespace=True, names=names) array = dataframe.values X = array[:,0:13] Y = array[:,13] seed = 7 kfold = model_selection.KFold(n_splits=10, random_state=seed) model = KNeighborsRegressor() scoring = 'neg_mean_squared_error' results = model_selection.cross_val_score(model, X, Y, cv=kfold, scoring=scoring) print(results.mean())

Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.

Running the example provides an estimate of the mean squared error.

-107.28683898

2. Classification and Regression Trees

Decision trees or the Classification and Regression Trees (CART as they are known) use the training data to select the best points to split the data in order to minimize a cost metric. The default cost metric for regression decision trees is the mean squared error, specified in the criterion parameter.

You can create a CART model for regression using the DecisionTreeRegressor class.

# Decision Tree Regression import pandas from sklearn import model_selection from sklearn.tree import DecisionTreeRegressor url = "https://raw.githubusercontent.com/jbrownlee/Datasets/master/housing.data" names = ['CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE', 'DIS', 'RAD', 'TAX', 'PTRATIO', 'B', 'LSTAT', 'MEDV'] dataframe = pandas.read_csv(url, delim_whitespace=True, names=names) array = dataframe.values X = array[:,0:13] Y = array[:,13] seed = 7 kfold = model_selection.KFold(n_splits=10, random_state=seed) model = DecisionTreeRegressor() scoring = 'neg_mean_squared_error' results = model_selection.cross_val_score(model, X, Y, cv=kfold, scoring=scoring) print(results.mean())

Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.

Running the example provides an estimate of the mean squared error.

-35.4906027451

3. Support Vector Machines

Support Vector Machines (SVM) were developed for binary classification. The technique has been extended for the prediction real-valued problems called Support Vector Regression (SVR). Like the classification example, SVR is built upon the LIBSVM library.

You can create an SVM model for regression using the SVR class.

# SVM Regression import pandas from sklearn import model_selection from sklearn.svm import SVR url = "https://raw.githubusercontent.com/jbrownlee/Datasets/master/housing.data" names = ['CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE', 'DIS', 'RAD', 'TAX', 'PTRATIO', 'B', 'LSTAT', 'MEDV'] dataframe = pandas.read_csv(url, delim_whitespace=True, names=names) array = dataframe.values X = array[:,0:13] Y = array[:,13] seed = 7 kfold = model_selection.KFold(n_splits=10, random_state=seed) model = SVR() scoring = 'neg_mean_squared_error' results = model_selection.cross_val_score(model, X, Y, cv=kfold, scoring=scoring) print(results.mean())

Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.

Running the example provides an estimate of the mean squared error.

-91.0478243332

Summary

In this post you discovered machine learning recipes for regression in Python using scikit-learn.

Specifically, you learned about:

4 Linear Machine Learning Algorithms:

• Linear Regression
• Ridge Regression
• LASSO Linear Regression
• Elastic Net Regression

3 Nonlinear Machine Learning Algorithms:

• K-Nearest Neighbors
• Classification and Regression Trees
• Support Vector Machines

Do you have any questions about regression machine learning algorithms or this post? Ask your questions in the comments and I will do my best to answer them.

Deep Learning Courses

 It can be difficult to get started in deep learning.

Thankfully, a number of universities have opened up their deep learning course material for free, which can be a great jump-start when you are looking to better understand the foundations of deep learning.

In this post you will discover the deep learning courses that you can browse and work through to develop and cement your understanding of the field.

This is a long post that deep links into many videos. It is intended for you to bookmark, browse and jump into specific topics across courses rather than pick one course and complete it end-to-end.

Overview

We will take a quick look at the following 6 deep learning courses.

1. Deep Learning at Oxford
2. Deep Learning at Udacity by Google
3. Deep Learning Summer School at Montreal
4. Deep Learning for Natural Language Processing at Stanford
5. Convolutional Neural Networks for Visual Recognition at Stanford
6. Neural networks Class at Université de Sherbrooke

There is also a “Other Courses” section at the end to gather additional video courses that are not free, broken or smaller in scope and don’t neatly fit into this summary review.

Course Tips and How To Use This Post

There are a lot of courses and a lot of great free material out there.

My best advice is:

Do not pick a course and work through it end-to-end.

This is counter to what most people suggest.

Your impulse will be to “get serious” and pick “the best” course and work through all of the material. You will almost certainly fail.

The material is tough and you will need to take your time and get multiple different perspectives on each topic.

The very best way to really get into this material is to work through it topic by topic and draw from across all of the courses until you really understand a topic, before moving onto the next topic.

You do not need to understand all topics and you do not need to use a single source to understand a single topic.

Bookmark this page, then browse, sample and dip into the material you need, when you need it as you learn how to implement actual real deep learning models in code using a platform like Keras.

Need help with Deep Learning in Python?

Take my free 2-week email course and discover MLPs, CNNs and LSTMs (with code).

Click to sign-up now and also get a free PDF Ebook version of the course.

Deep Learning at Oxford

This is a machine learning course that focuses on deep learning taught at Oxford by Nando de Freitas.

I really like this course. I watched all of the videos on double time and took notes. It provides a good foundation in theory and covers modern deep learning topics such as LSTMs. Code examples are shown in Torch.

I noted that the syllabus differed from the actual video lectures available and the YouTube playlist listed the lectures out of order, so below is the list of 2015 video lectures in order.

• Deep Learning Lecture 1: Introduction
• Deep Learning Lecture 2: linear models
• Deep Learning Lecture 3: Maximum likelihood and information
• Deep Learning Lecture 4: Regularization, model complexity and data complexity (part 1)
• Deep Learning Lecture 5: Regularization, model complexity and data complexity (part 2)
• Deep Learning Lecture 6: Optimization
• Deep learning Lecture 7: Logistic regression, a Torch approach
• Deep Learning Lecture 8: Modular back-propagation, logistic regression and Torch
• Deep Learning Lecture 9: Neural networks and modular design in Torch
• Deep Learning Lecture 10: Convolutional Neural Networks
• Deep Learning Lecture 11: Max-margin learning, transfer and memory networks
• Deep Learning Lecture 12: Recurrent Neural Nets and LSTMs
• Deep Learning Lecture 13: Alex Graves on Hallucination with RNNs
• Deep Learning Lecture 14: Karol Gregor on Variational Autoencoders and Image Generation
• Deep Learning Lecture 15: Deep Reinforcement Learning – Policy search
• Deep Learning Lecture 16: Reinforcement learning and neuro-dynamic programming

The highlight for me was Alex Graves‘ talk on RNNs (Lecture 13). A smart guy doing great work. I was reading a lot of Alex’s papers at the time I watch this video so I may be biased.

Resources

• YouTube Playlist
• Course Summary and Materials
• Nando de Freitas homepage

Deep Learning at Udacity by Google

This is a mini course collaboration between Arpan Chakraborty from Udacity and Vincent Vanhoucke, a Principal Scientist at Google.

The course is free, hosted on Udacity and focuses on TensorFlow. It is a small piece of the broader Machine Learning Engineer Nanodegree by Google hosted on Udacity.

You must sign-up to Udacity, but once you sign-in you can access this course for free.

All course videos are on YouTube, but (intentionally) really hard to find with poor naming and linking. If anyone knows of a pirate playlist with all of the videos, please post it in the comments.

The course is divided into 4 lessons:

• Lesson 1: From Machine Learning to Deep Learning
• Lesson 2: Deep Neural Networks
• Lesson 3: Convolutional Neural Networks
• Lesson 4: Deep Models for Text and Sequences

The course is short but is broken up into many short videos and the Udacity interface is nice. Vincent seems to present in all of the videos I looked at (which is great) and videos are shown in the YouTube interface.

There is also a discussion form where you can ask and answer questions, driven by the slick discourse software.

My preference was to dip into videos that interested me rather than completing the whole course or doing any of the course work.

Resources

• Udacity Deep Learning Course
• Course Announcement on the Google Blog

Deep Learning Summer School at Montreal

A deep learning supper school was held in 2015 at the University of Montreal.

According to the website, the summer school was aimed at graduate students and industrial engineers and researchers who already have some basic knowledge of machine learning.

There were at least 30 talks (there are 30 videos) from notable researchers in the field of deep learning on a range of topics from introductory material to state of the art research.

Deep Learning Summer School at Montreal

These videos are are a real treasure trove. Take your time and pick your topics carefully. All videos are hosted on the VideoLectures.net site, which has an good enough interface, but not as clean as YouTube.

Many (all?) talks had PDF slides linked below the video, and more information is available from the schedule page on the official website.

Here’s the full list of lecture topics with links to the videos. I’ve tried to list related videos together (e.g. part 1, part 2).

• Introduction to Machine Learning
• Deep Learning: Theoretical Motivations
• Multilayer Neural Networks
• Training Deep Neural Networks
• Multilayer Neural Networks
• Deep Learning for Distribution Estimation
• Undirected Graphical Models
• Stacks of Restricted Boltzmann Machines
• On manifolds and autoencoders
• Visual features: From Fourier to Gabor
• Visual features II
• Convolutional Networks
• Learning to Compare
• NLP and Deep Learning 1: Human Language & Word Vectors
• NLP and Deep Learning 2: Compositional Deep Learning
• Seeing People with Deep Learning
• Deep Learning
• Deep Learning 2
• Speech Recognition and Deep Learning
• Tutorial on Neural Network Optimization Problems
• Deep Learning (hopefully faster)
• Adversarial Examples
• From Language Modelling to Machine Translation
• Deep NLP Recurrent Neural Networks
• Deep NLP Applications and Dynamic Memory Networks
• Memory, Reading, and Comprehension
• Smooth, Finite, and Convex Optimization Deep Learning Summer School
• Non Smooth, Non Finite, and Non Convex Optimization
• Variational Autoencoder and Extensions
• Deep Generative Models

Pick a topic and dive in. So good!

It looks like there will be a 2016 summer school and hopefully there will be videos.

Resources

• Deep Learning Summer School Official Website
• Deep Learning Summer School Videos

Deep Learning for Natural Language Processing at Stanford

This is a deep learning course focusing on natural language processing (NLP) taught by Richard Socher at Stanford.

An interesting note is that you can access PDF versions of student reports, work that might inspire you or give you ideas.

The YouTube playlist has poorly named files and some missing lectures. The 2016 videos are not all uploaded yet. Below is a list of the 2015 lectures and the links to the videos. Much easier for just jumping into a specific topic.

• Lecture 1: Intro to NLP and Deep Learning
• Lecture 2: Simple Word Vector representations: word2vec, GloVe
• Lecture 3: Advanced word vector representations: language models, softmax, single layer networks
• Lecture 4: Word Window Classification and Neural Networks
• Lecture 5: Project Advice, Neural Networks and Back-Prop (in full gory detail)
• Lecture 6: Practical tips: gradient checks, overfitting, regularization, activation functions, details
• Lecture 7: Recurrent neural networks for language modeling and other tasks
• Lecture 7 (8!?): GRUs and LSTMs for machine translation
• Lecture 9: Recursive neural networks for parsing
• Lecture 10: Recursive neural networks for different tasks (e.g. sentiment analysis)
• Lecture 11: Review Session for Midterm
• Lecture 13: Convolutional neural networks for sentence classification
• Lecture 15: Applications of DL to NLP
• Guest Lecture with Andrew Maas: Speech recognition
• Guest Lecture with Jason Weston: Memory Networks
• Guest Lecture with Elliot English: Efficient Implementations and GPUs

This is great material if you are into deep learning for NLP, an area where it really excels.

Resources

• CS224d: Deep Learning for Natural Language Processing Homepage
• Course Syllabus
• 2015 Course Video Play List
• 2016 Course Video Play List

Convolutional Neural Networks for Visual Recognition at Stanford

This course focuses on the use of deep learning for computer vision applications with convolutional neural networks.

It is another course taught at Stanford, this time by Andrej Karpathy and others.

Unfortunately, the course videos were taken down, but some clever people have found ways to put them back up in other places. See the playlists in the resources section below.

Another great course.

Below are the video lectures for the 2016 course, but I’m not sure how long the links will last. Leave a comment and let me know if you discover the links turned bad and I’ll fix them up.

• Lecture 1 Introduction and Historical Context
• Lecture 2 Data driven approach, kNN, Linear Classification 1
• Lecture 3 Linear Classification 2, Optimization
• Lecture 4 Backpropagation, Neural Networks 1
• Lecture 5 Neural Networks Part 2
• Lecture 6 Neural Networks Part 3 Intro to ConvNets
• Lecture 7 Convolutional Neural Networks
• Lecture 8 Localization and Detection
• Lecture 9 Visualization, Deep Dream, Neural Style, Adversarial Examples
• Lecture 10 Recurrent Neural Networks, Image Captioning, LSTM
• Lecture 11 ConvNets in practice
• Lecture 12 Deep Learning libraries
• Lecture 13 Segmentation, soft attention, spatial transformers
• Lecture 14 Videos and Unsupervised Learning
• Lecture 15 Invited Talk by Jeff Dean

Resources

• Convolutional Neural Networks for Visual Recognition Homepage
• Course Syllabus (and the 2015 Syllabus)
• Course Videos on Archive.org
• Course Videos on YouTube
• Course Sample Code

Neural Networks Class at Université de Sherbrooke

This is a course on neural networks taught by Hugo Larochelle at the University in Sherbrooke in Québec.

There is a ton of material. A ton.

The videos are one-on-one rather than lectures and there are many small videos for each topic rather than large one hour info dumps.

I think this might be a better format than the traditional lectures, but I’m not completely won over yet. A difficulty is there are 92 videos (!!!) to browse and it can be hard to find specific videos to watch.

The material is taught covering 10 main topics:

• Topic 1: Feedforward neural networks
• Topic 2: Training neural networks
• Topic 3: Conditional random fields
• Topic 4: Training Conditional random fields
• Topic 5: Restricted Boltzmann machine
• Topic 6: Autoencoders
• Topic 7: Deep learning
• Topic 8: Sparse coding
• Topic 9: Computer vision
• Topic 10: Natural language processing

My recommendation is to use the main course home page to browse the topics and then use those links into the specific videos. The YouTube playlist has far too many videos to browse and understand. The paradox of choice will kill you.

Resources

• Course Homepage
• YouTube Video Playlist

Other Courses

Below are some additional video courses that are either not free, difficult to access or smaller in scope.

• Deep Learning Course at CILVR Lab @ NYU (broken links?)
• Smaller Deep Learning Courses on Udemy
• Deep Learning at CMU
• Nvidia Self-Paced Courses for Deep Learning
• Neural Networks for Machine Learning at Coursera by the University of Toronto (awesome, but no longer free)
  • Update: You can watch all the videos for free here.
• Graduate Summer School: Deep Learning, Feature Learning, 2012

Summary

In this post you have discover a number of world class video courses on deep learning covering, theory, computer vision, natural language processing and more.

Heed the advice at the top of this post.

Browse and dip into lectures by topic and do not try to take on a whole course. Learn one thing rather than try and learn everything.

Take your time, bookmark this page so you can come back, and have fun.

Do you know of some other video courses on deep learning that I have not listed?
Let me know in the comments and I will update the list.