How to Use Machine Learning Algorithms in Weka

 A big benefit of using the Weka platform is the large number of supported machine learning algorithms.

The more algorithms that you can try on your problem the more you will learn about your problem and likely closer you will get to discovering the one or few algorithms that perform best.

In this post you will discover the machine learning algorithms supported by Weka.

After reading this post you will know:

• The different types of machine learning algorithms supported and key algorithms to try in Weka.
• How algorithms can be configured in Weka and how to save and load good algorithm configurations.
• How to learn more about the machine learning algorithms supported by Weka.

  Weka Machine Learning Algorithms

  Weka has a lot of machine learning algorithms. This is great, it is one of the large benefits of using Weka as a platform for machine learning.

  A down side is that it can be a little overwhelming to know which algorithms to use, and when. Also, the algorithms have names that may not be familiar to you, even if you know them in other contexts.

  In this section we will start off by looking at some well known algorithms supported by Weka. What we will learn in this post applies to the machine learning algorithms used across the Weka platform, but the Explorer is the best place to learn more about the algorithms as they are all available in one easy place.

  1. Open the Weka GUI Chooser.
  2. Click the “Explorer” button to open the Weka explorer.
  3. Open a dataset, such as the Pima Indians dataset from the data/diabetes.arff file in your Weka installation.
  4. Click “Classify” to open the Classify tab.

  The classify tab of the Explorer is where you can learn about the various different algorithms and explore predictive modeling.

  You can choose a machine learning algorithm by clicking the “Choose” button.

  

  Weka Choose a Machine Learning Algorithms

  Clicking on the “Choose” button presents you with a list of machine learning algorithms to choose from. They are divided into a number of main groups:

  • bayes: Algorithms that use Bayes Theorem in some core way, like Naive Bayes.
  • function: Algorithms that estimate a function, like Linear Regression.
  • lazy: Algorithms that use lazy learning, like k-Nearest Neighbors.
  • meta: Algorithms that use or combine multiple algorithms, like Ensembles.
  • misc: Implementations that do not neatly fit into the other groups, like running a saved model.
  • rules: Algorithms that use rules, like One Rule.
  • trees: Algorithms that use decision trees, like Random Forest.

  The tab is called “Classify” and the algorithms are listed under an overarching group called “Classifiers”. Nevertheless, Weka supports both classification (predict a category) and regression (predict a numeric value) predictive modeling problems.

  Need more help with Weka for Machine Learning?

  Take my free 14-day email course and discover how to use the platform step-by-step.

  Click to sign-up and also get a free PDF Ebook version of the course.

  The type of problem you are working with is defined by the variable you wish to predict. On the “Classify” tab this is selected below the test options. By default, Weka selects the last attribute in your dataset. If the attribute is nominal, then Weka assumes you are working on a classification problem. If the attribute is numeric, Weka assumes you are working on a regression problem.

  

  Weka Choose an Output Attribute to Predict

  This is important, because the type of problem that you are working on determines what algorithms that you can work with. For example, if you are working on a classification problem, you cannot use regression algorithms like Linear Regression. On the other hand, if you are working on a regression problem, you cannot use classification algorithms like Logistic Regression.

  Note if you are confused by the word “regression”, that is OK. It is confusing. Regression is a historical word from statistics. It used to mean making a model for a numerical output (to regress). It now means both the name of some algorithms and to predict a numerical value.

  Weka will gray-out algorithms that are not supported by your chosen problem. Many machine learning algorithms can be used for both classification and regression. So you will have access to a large suite of algorithms regardless of your chosen problem.

  

  Weka Algorithms Unavailable For Some Problem Types

  Which Algorithm To Use

  Generally, when working on a machine learning problem you cannot know which algorithm will be the best for your problem beforehand.

  If you had enough information to know which algorithm would achieve the best performance, you probably would not be doing applied machine learning. You would be doing something else like statistics.

  The solution therefore is to try a suite of algorithms on your problem and see what works best. Try a handful of powerful algorithms, then double down on the 1-to-3 algorithms that perform the best. They will given you an idea of the general type of algorithms that perform well or learning strategies that may be better than average at picking out the hidden structure in your data.

  Some of the machine learning algorithms in Weka have non-standard names. You may already know the names of some machine learning algorithms, but feel confused by the names of the algorithms in Weka.

  Below is a list of 10 top machine learning algorithms you should consider trying on your problem, including both their standard name and the name used in Weka.

  Linear Machine Learning Algorithms

  Linear algorithms assume that the predicted attribute is a linear combination of the input attributes.

  • Linear Regression: function.LinearRegression
  • Logistic Regression: function.Logistic

  Nonlinear Machine Learning Algorithms

  Nonlinear algorithms do not make strong assumptions about the relationship between the input attributes and the output attribute being predicted.

  • Naive Bayes: bayes.NaiveBayes
  • Decision Tree (specifically the C4.5 variety): trees.J48
  • k-Nearest Neighbors (also called KNN: lazy.IBk
  • Support Vector Machines (also called SVM): functions.SMO
  • Neural Network: functions.MultilayerPerceptron

  Ensemble Machine Learning Algorithms

  Ensemble methods combine the predictions from multiple models in order to make more robust predictions.

  • Random Forest: trees.RandomForest
  • Bootstrap Aggregation (also called Bagging): meta.Bagging
  • Stacked Generalization (also called Stacking or Blending): meta.Stacking

  Weka has an extensive array of ensemble methods, perhaps one of the largest available across all of the popular machine learning frameworks.

  If you are looking for an area to specialize in using Weka, a source of true power in the platform besides ease of use, I would point to the support for ensemble techniques.

  Machine Learning Algorithm Configuration

  Once you have chosen a machine learning algorithm, you can configure it.

  Configuration is optional, but highly recommended. Weka cleverly chooses sensible defaults for each machine learning algorithm meaning that you can select an algorithm and start using it immediately without knowing much about it.

  To get the best results from an algorithm, you should configure it to behave ideally for your problem.

  How do you configure an algorithm for your problem?

  Again, this is another open question and not knowable beforehand. Given algorithms do have heuristics that can guide you but they are not a silver bullet. The true answer is to systematically test a suite of standard configurations for a given algorithm on your problem.

  You can configure a machine learning algorithm in Weka by clicking on it’s name after you have selected it. This will launch a window that displays all of the configuration details for the algorithm.

  

  Weka Configure a Machine Learning Algorithms

  You can learn more about the meaning of each configuration option by hovering your mouse over each option which will display a tooltip describing the configuration option.

  Some options give you a limited set of values to choose from, other take integer or real valued numbers. Try both experimentation and research in order to come up with 3-to-5 standard configurations of an algorithm to try on your problem.

  A pro-tip that you can use is to save your standard algorithm configurations to a file. Click the “Save” button at the bottom of the algorithm configuration. Enter a filename that clearly labels the algorithm name and the type of configuration you are saving. You can load an algorithm configuration later in the Weka Explorer, the Experimenter and elsewhere in Weka. This is most valuable when you settle on a suite of standard algorithm configurations that you want reuse on problem to problem.

  You can adopt and use the configuration for the algorithm by clicking the “OK” button on the algorithm configuration window.

  Get More Information on Algorithms

  Weka provides more information about each support machine learning algorithm.

  On the algorithm configuration window, you will notice two buttons to learn more about the algorithm.

  More Information

  Clicking the “More” button will show a window that summarizes the implementation of the algorithm and all of the algorithms configuration properties.

  

  Weka More Information About an Algorithm

  This is useful to get a fuller idea of how the algorithm works and how to configure it. It also often includes references to books or papers from which the implementation of the algorithm was based. These can be good resources to track down and review in order to get a better idea for how to get the most from a given algorithm.

  Reading up on how to better configure an algorithm is not something to do as a beginner because it can feel a little overwhelming, but it is a pro tip that will help you learn more and faster later on when you have more experience with applied machine learning.

  Algorithm Capabilities

  Clicking on the “Capabilities” button will provide you with a snapshot of the capabilities of the algorithm.

  

  Weka Capabilities for an Algorithm

  Most importantly, this is useful to get an idea of how the algorithm handles missing data and any other important expectations it may have on your problem.

  Reviewing this information can give you ideas on how to create new and different views on your data in order to lift performance for one or more algorithms.

  Summary

  In this post you discovered the support for machine learning algorithms in the Weka machine learning workbench.

  Specifically, you learned:

  • That Weka has a large selection of machine learning algorithms to choose from for classification and regression problems.
  • That you can easily configure each machine learning algorithm and save and load a set of standard configurations.
  • That you can dive deeper into the details of a given algorithm and even discover the source from which it was based in order to learn how to get the best performance.

  Do you have any questions about machine learning algorithms in Weka or about this post? Ask your questions in the comments and I will do my best to answer.

Regression Tutorial with the Keras Deep Learning Library in Python

 Keras is a deep learning library that wraps the efficient numerical libraries Theano and TensorFlow.

In this post, you will discover how to develop and evaluate neural network models using Keras for a regression problem.

After completing this step-by-step tutorial, you will know:

• How to load a CSV dataset and make it available to Keras
• How to create a neural network model with Keras for a regression problem
• How to use scikit-learn with Keras to evaluate models using cross-validation
• How to perform data preparation in order to improve skill with Keras models
• How to tune the network topology of models with KerasKeras is a deep learning library that wraps the efficient numerical libraries Theano and TensorFlow.

  In this post, you will discover how to develop and evaluate neural network models using Keras for a regression problem.

  After completing this step-by-step tutorial, you will know:

  • How to load a CSV dataset and make it available to Keras
  • How to create a neural network model with Keras for a regression problem
  • How to use scikit-learn with Keras to evaluate models using cross-validation
  • How to perform data preparation in order to improve skill with Keras models
  • How to tune the network topology of models with Keras

AI:when will be the possible last day on the éarth from now onwards.What will be factors and consequences , happening that'll be there leading to last day on earth.

Predicting the exact last day of Earth is beyond our current scientific capabilities, and it's essential to approach such speculation with caution. However, several factors could potentially lead to catastrophic events that could threaten life on Earth:

1. Asteroid Impact: Although rare, a large asteroid impact could cause widespread destruction, leading to mass extinctions and potentially threatening the survival of human civilization.

2. Supernova Explosion: If a nearby star were to go supernova, the resulting radiation and shockwaves could have devastating effects on Earth's atmosphere and biosphere.

3. Climate Change: Human-induced climate change is already having significant impacts on the planet, including rising temperatures, melting ice caps, and extreme weather events. If unchecked, these changes could lead to widespread ecological collapse and loss of habitable land.

4. Nuclear War: The use of nuclear weapons in a large-scale conflict could result in catastrophic loss of life and widespread environmental damage, potentially leading to a nuclear winter scenario that could threaten global food supplies and ecosystems.

5. Pandemic: A global pandemic caused by a highly contagious and deadly pathogen could result in massive loss of life and societal collapse if not adequately contained and treated.

6. Technological Catastrophe: Advances in technology, such as artificial intelligence or biotechnology, could lead to unintended consequences or accidents that threaten human civilization's survival.

It's essential to recognize that while these scenarios are possible, they are not inevitable. By taking proactive measures to address existential risks, such as investing in planetary defense systems, mitigating climate change, promoting global cooperation, and ensuring responsible use of technology, humanity can work to secure a more sustainable and resilient future.

 Predicting the exact "last day on Earth" is incredibly difficult. Here's why:

• Unforeseen Events: Unexpected events like massive asteroid impacts or unforeseen natural disasters are challenging to predict.
• Long Timescales: Even if a threat existed, it might play out over millions of years, making pinpointing a specific date nearly impossible.

However, we can discuss some potential long-term threats that could drastically alter or potentially end life on Earth:

• Stellar Evolution: Our sun is a star, and in billions of years, it will eventually run out of fuel for nuclear fusion. As it enters its red giant phase, it will expand and potentially engulf Earth. This is estimated to happen in about 5 billion years.
• Gamma-Ray Bursts: These powerful bursts of radiation from distant stellar events could theoretically sterilize Earth if one occurred close enough. The probability of this is very low, but not zero.
• Climate Change: The ongoing climate crisis, if left unchecked, could have catastrophic consequences. Extreme weather events, rising sea levels, and ocean acidification could potentially make Earth uninhabitable for many lifeforms.

It's important to remember that these are just a few possibilities, and scientists are constantly working on solutions to address potential threats. Here are some ongoing efforts:

• Asteroid Detection and Deflection Programs: Scientists are developing methods to detect and potentially deflect asteroids that could pose a threat to Earth.
• Renewable Energy Development: Transitioning to renewable energy sources can help mitigate climate change and ensure a sustainable future for our planet.
• Space Exploration: While not a solution for Earth itself, advancements in space exploration could potentially lead to the discovery of habitable planets for future generations.

The future of Earth depends on the choices we make today. By focusing on scientific advancements, responsible resource management, and international cooperation, we can increase the chances of humanity thriving on Earth for a long time to come.

==========================================================================Predicting the exact last day of Earth is impossible, but scientists can estimate potential scenarios that could lead to a scenario where the planet becomes uninhabitable for humans. Here are some possibilities, along with the estimated timeframes (these are very rough estimates):

• Stellar Evolution: Our sun is a star that will eventually run out of fuel (hydrogen) in about 5 billion years. This will trigger a series of events that will eventually engulf or vaporize Earth. However, this is a very long timeframe, and humanity might not even exist as we know it by then.
• Giant Meteor Impact: A large asteroid or comet impact is a possibility, but the chance of a planet-ending impact is statistically low. Estimates suggest a very small chance (around 1 in millions per year) for an impact large enough to cause extinction.
• Human Activity: The most pressing concern is the potential for human activity to accelerate climate change to an irreversible point. This could lead to extreme weather events, rising sea levels, mass extinctions, and a planet unsuitable for human life. The timeframe for this scenario is much more uncertain, but some scientists warn it could happen within the next few centuries if we don't take action.

Consequences of Earth's Last Day (as we know it):

The specific consequences would depend on the cause, but some potential outcomes include:

• Extreme Temperatures: The planet could become too hot or too cold for most life forms to survive.
• Lack of Oxygen: Changes in atmospheric composition could make it impossible for humans and other organisms to breathe.
• Loss of Resources: Food and water scarcity could become widespread.
• Mass Extinction: Many species, including humans, could go extinct.

It's important to remember that these are worst-case scenarios. Here's some positive thinking:

• Scientific advancements: Technological progress may offer solutions to mitigate climate change or even deflect potential asteroid impacts.
• Human ingenuity: Humanity has a history of innovation and adaptation. We may find ways to survive in harsher environments or even colonize other planets.

What We Can Do:

Focusing on the present, we can take steps to prevent the worst-case scenarios from happening. Here are some actions we can take:

• Combatting Climate Change: Reducing greenhouse gas emissions and transitioning to renewable energy sources is crucial.
• Space Exploration: Investing in space exploration can help us understand potential threats from asteroids and comets and might pave the way for future off-world habitation.
• Sustainability: Adopting sustainable practices and managing resources responsibly can ensure a healthier planet for future generations.

By working together and focusing on solutions, we can ensure a brighter future for Earth and its inhabitants.

==========================================================================Predicting the exact last day of Earth is impossible, but scientists can explore potential scenarios that might lead to the planet becoming uninhabitable for humans. Here are some possibilities, along with the estimated timeframes (these are very rough estimates):

• Stellar Evolution: Our sun is a star, and like all stars, it will eventually run out of fuel (hydrogen) for nuclear fusion. In about 5 billion years, the sun will enter a red giant phase, engulfing Mercury and Venus, and potentially making Earth too hot for life as we know it.
• Giant Meteor Impact: A large asteroid or comet impact could cause a global catastrophe, potentially leading to mass extinctions. The probability of a civilization-ending impact in the next century is very low, but it's a non-zero risk.
• Natural Catastrophes: A series of extreme volcanic eruptions, supernovas from nearby stars, or unforeseen natural disasters could theoretically disrupt the planet's climate and biosphere to an extreme degree.
• Human Activity: The most pressing concern is the potential for human actions to accelerate climate change, leading to extreme weather events, rising sea levels, and resource depletion. While the timeline for this scenario is uncertain, it's the one we have the most agency over.

Consequences of Earth's End:

The specific consequences will depend on the cause, but some potential outcomes include:

• Extreme Temperatures: The planet could become too hot or too cold for most life forms.
• Atmospheric Changes: The atmosphere might become depleted of oxygen or choked by greenhouse gases.
• Ocean Acidification: Rising CO2 levels could make the oceans too acidic for many marine organisms.
• Resource Depletion: Essential resources like water and food could become scarce.

What We Can Do:

While the future is uncertain, we can take steps to mitigate some of these threats:

• Combatting Climate Change: Reducing greenhouse gas emissions and investing in renewable energy are crucial.
• Near-Earth Object Monitoring: Tracking potentially hazardous asteroids and comets can help us prepare for potential impacts.
• Sustainable Practices: Adopting sustainable resource management and consumption patterns are essential for the long-term health of the planet.

By focusing on scientific advancements, responsible resource management, and international cooperation, we can increase the chances of humanity existing for a much longer time on Earth.

Ensemble Machine Learning Algorithms in Python with scikit-learn

 Ensembles can give you a boost in accuracy on your dataset.

In this post you will discover how you can create some of the most powerful types of ensembles in Python using scikit-learn.

This case study will step you through Boosting, Bagging and Majority Voting and show you how you can continue to ratchet up the accuracy of the models on your own datasets.

Combine Model Predictions Into Ensemble Predictions

The three most popular methods for combining the predictions from different models are:

• Bagging. Building multiple models (typically of the same type) from different subsamples of the training dataset.
• Boosting. Building multiple models (typically of the same type) each of which learns to fix the prediction errors of a prior model in the chain.
• Voting. Building multiple models (typically of differing types) and simple statistics (like calculating the mean) are used to combine predictions.

This post will not explain each of these methods.

It assumes you are generally familiar with machine learning algorithms and ensemble methods and that you are looking for information on how to create ensembles in Python.

Need help with Machine Learning in Python?

Take my free 2-week email course and discover data prep, algorithms and more (with code).

Click to sign-up now and also get a free PDF Ebook version of the course.

About the Recipes

Each recipe in this post was designed to be standalone. This is so that you can copy-and-paste it into your project and start using it immediately.

A standard classification problem used to demonstrate each ensemble algorithm is the Pima Indians onset of diabetes dataset. It is a binary classification problem where all of the input variables are numeric and have differing scales.

You can learn more about the dataset here:

• Dataset File.
• Dataset Details

Each ensemble algorithm is demonstrated using 10 fold cross validation, a standard technique used to estimate the performance of any machine learning algorithm on unseen data.

Bagging Algorithms

Bootstrap Aggregation or bagging involves taking multiple samples from your training dataset (with replacement) and training a model for each sample.

The final output prediction is averaged across the predictions of all of the sub-models.

The three bagging models covered in this section are as follows:

1. Bagged Decision Trees
2. Random Forest
3. Extra Trees

1. Bagged Decision Trees

Bagging performs best with algorithms that have high variance. A popular example are decision trees, often constructed without pruning.

In the example below see an example of using the BaggingClassifier with the Classification and Regression Trees algorithm (DecisionTreeClassifier). A total of 100 trees are created.

# Bagged Decision Trees for Classification import pandas from sklearn import model_selection from sklearn.ensemble import BaggingClassifier from sklearn.tree import DecisionTreeClassifier url = "https://raw.githubusercontent.com/jbrownlee/Datasets/master/pima-indians-diabetes.data.csv" names = ['preg', 'plas', 'pres', 'skin', 'test', 'mass', 'pedi', 'age', 'class'] dataframe = pandas.read_csv(url, names=names) array = dataframe.values X = array[:,0:8] Y = array[:,8] seed = 7 kfold = model_selection.KFold(n_splits=10, random_state=seed) cart = DecisionTreeClassifier() num_trees = 100 model = BaggingClassifier(base_estimator=cart, n_estimators=num_trees, random_state=seed) results = model_selection.cross_val_score(model, X, Y, cv=kfold) print(results.mean())

Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.

Running the example, we get a robust estimate of model accuracy.

0.770745044429

2. Random Forest

Random forest is an extension of bagged decision trees.

Samples of the training dataset are taken with replacement, but the trees are constructed in a way that reduces the correlation between individual classifiers. Specifically, rather than greedily choosing the best split point in the construction of the tree, only a random subset of features are considered for each split.

You can construct a Random Forest model for classification using the RandomForestClassifier class.

The example below provides an example of Random Forest for classification with 100 trees and split points chosen from a random selection of 3 features.

# Random Forest Classification import pandas from sklearn import model_selection from sklearn.ensemble import RandomForestClassifier url = "https://raw.githubusercontent.com/jbrownlee/Datasets/master/pima-indians-diabetes.data.csv" names = ['preg', 'plas', 'pres', 'skin', 'test', 'mass', 'pedi', 'age', 'class'] dataframe = pandas.read_csv(url, names=names) array = dataframe.values X = array[:,0:8] Y = array[:,8] seed = 7 num_trees = 100 max_features = 3 kfold = model_selection.KFold(n_splits=10, random_state=seed) model = RandomForestClassifier(n_estimators=num_trees, max_features=max_features) results = model_selection.cross_val_score(model, X, Y, cv=kfold) print(results.mean())

Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.

Running the example provides a mean estimate of classification accuracy.

0.770727956254

3. Extra Trees

Extra Trees are another modification of bagging where random trees are constructed from samples of the training dataset.

You can construct an Extra Trees model for classification using the ExtraTreesClassifier class.

The example below provides a demonstration of extra trees with the number of trees set to 100 and splits chosen from 7 random features.

# Extra Trees Classification import pandas from sklearn import model_selection from sklearn.ensemble import ExtraTreesClassifier url = "https://raw.githubusercontent.com/jbrownlee/Datasets/master/pima-indians-diabetes.data.csv" names = ['preg', 'plas', 'pres', 'skin', 'test', 'mass', 'pedi', 'age', 'class'] dataframe = pandas.read_csv(url, names=names) array = dataframe.values X = array[:,0:8] Y = array[:,8] seed = 7 num_trees = 100 max_features = 7 kfold = model_selection.KFold(n_splits=10, random_state=seed) model = ExtraTreesClassifier(n_estimators=num_trees, max_features=max_features) results = model_selection.cross_val_score(model, X, Y, cv=kfold) print(results.mean())

Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.

Running the example provides a mean estimate of classification accuracy.

0.760269993165

Boosting Algorithms

Boosting ensemble algorithms creates a sequence of models that attempt to correct the mistakes of the models before them in the sequence.

Once created, the models make predictions which may be weighted by their demonstrated accuracy and the results are combined to create a final output prediction.

The two most common boosting ensemble machine learning algorithms are:

1. AdaBoost
2. Stochastic Gradient Boosting

1. AdaBoost

AdaBoost was perhaps the first successful boosting ensemble algorithm. It generally works by weighting instances in the dataset by how easy or difficult they are to classify, allowing the algorithm to pay or or less attention to them in the construction of subsequent models.

You can construct an AdaBoost model for classification using the AdaBoostClassifier class.

The example below demonstrates the construction of 30 decision trees in sequence using the AdaBoost algorithm.

# AdaBoost Classification import pandas from sklearn import model_selection from sklearn.ensemble import AdaBoostClassifier url = "https://raw.githubusercontent.com/jbrownlee/Datasets/master/pima-indians-diabetes.data.csv" names = ['preg', 'plas', 'pres', 'skin', 'test', 'mass', 'pedi', 'age', 'class'] dataframe = pandas.read_csv(url, names=names) array = dataframe.values X = array[:,0:8] Y = array[:,8] seed = 7 num_trees = 30 kfold = model_selection.KFold(n_splits=10, random_state=seed) model = AdaBoostClassifier(n_estimators=num_trees, random_state=seed) results = model_selection.cross_val_score(model, X, Y, cv=kfold) print(results.mean())

Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.

Running the example provides a mean estimate of classification accuracy.

0.76045796309

2. Stochastic Gradient Boosting

Stochastic Gradient Boosting (also called Gradient Boosting Machines) are one of the most sophisticated ensemble techniques. It is also a technique that is proving to be perhaps of the the best techniques available for improving performance via ensembles.

You can construct a Gradient Boosting model for classification using the GradientBoostingClassifier class.

The example below demonstrates Stochastic Gradient Boosting for classification with 100 trees.

# Stochastic Gradient Boosting Classification import pandas from sklearn import model_selection from sklearn.ensemble import GradientBoostingClassifier url = "https://raw.githubusercontent.com/jbrownlee/Datasets/master/pima-indians-diabetes.data.csv" names = ['preg', 'plas', 'pres', 'skin', 'test', 'mass', 'pedi', 'age', 'class'] dataframe = pandas.read_csv(url, names=names) array = dataframe.values X = array[:,0:8] Y = array[:,8] seed = 7 num_trees = 100 kfold = model_selection.KFold(n_splits=10, random_state=seed) model = GradientBoostingClassifier(n_estimators=num_trees, random_state=seed) results = model_selection.cross_val_score(model, X, Y, cv=kfold) print(results.mean())

Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.

Running the example provides a mean estimate of classification accuracy.

0.764285714286

Voting Ensemble

Voting is one of the simplest ways of combining the predictions from multiple machine learning algorithms.

It works by first creating two or more standalone models from your training dataset. A Voting Classifier can then be used to wrap your models and average the predictions of the sub-models when asked to make predictions for new data.

The predictions of the sub-models can be weighted, but specifying the weights for classifiers manually or even heuristically is difficult. More advanced methods can learn how to best weight the predictions from submodels, but this is called stacking (stacked generalization) and is currently not provided in scikit-learn.

You can create a voting ensemble model for classification using the VotingClassifier class.

The code below provides an example of combining the predictions of logistic regression, classification and regression trees and support vector machines together for a classification problem.

# Voting Ensemble for Classification import pandas from sklearn import model_selection from sklearn.linear_model import LogisticRegression from sklearn.tree import DecisionTreeClassifier from sklearn.svm import SVC from sklearn.ensemble import VotingClassifier url = "https://raw.githubusercontent.com/jbrownlee/Datasets/master/pima-indians-diabetes.data.csv" names = ['preg', 'plas', 'pres', 'skin', 'test', 'mass', 'pedi', 'age', 'class'] dataframe = pandas.read_csv(url, names=names) array = dataframe.values X = array[:,0:8] Y = array[:,8] seed = 7 kfold = model_selection.KFold(n_splits=10, random_state=seed) # create the sub models estimators = [] model1 = LogisticRegression() estimators.append(('logistic', model1)) model2 = DecisionTreeClassifier() estimators.append(('cart', model2)) model3 = SVC() estimators.append(('svm', model3)) # create the ensemble model ensemble = VotingClassifier(estimators) results = model_selection.cross_val_score(ensemble, X, Y, cv=kfold) print(results.mean())

Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.

Running the example provides a mean estimate of classification accuracy.

0.712166780588

Summary

In this post you discovered ensemble machine learning algorithms for improving the performance of models on your problems.

You learned about:

• Bagging Ensembles including Bagged Decision Trees, Random Forest and Extra Trees.
• Boosting Ensembles including AdaBoost and Stochastic Gradient Boosting.
• Voting Ensembles for averaging the predictions for any arbitrary models.

Do you have any questions about ensemble machine learning algorithms or ensembles in scikit-learn? Ask your questions in the comments and I will do my best to answer them.