AutoML: The Next Wave of Machine Learning

Mercari is a popular shopping app in Japan that has been using AutoML Vision (Google’s AutoML solution) for classifying images. According to Mercari, they’ve been “developing their own ML model that suggests a brand name from 12 major brands in the photo uploading user interface.”

While their own model—trained on TensorFlow—achieved an accuracy of 75%, AutoML Vision in advanced mode with 50,000 training images achieved an accuracy of 91.3%, which is a whopping 15% increase. With such astounding results, Mercari has integrated AutoML into their systems.

This is just one example of how AutoML is fundamentally changing the face of ML-based solutions today by enabling people from diverse backgrounds to evolve machine learning models to address complex scenarios.

Automated Machine Learning: AutoML

Machine learning has provided some significant breakthroughs in diverse fields in recent years. Areas like financial services, healthcare, retail, transportation, and more have been using machine learning systems in one way or another, and the results have been promising.

Machine learning today is not limited to R&D applications but has made its foray into the enterprise domain. However, the traditional ML process is human-dependent, and not all businesses have the resources to invest in an experienced data science team. AutoML may be the answer to such situations.

Automated machine learning (AutoML) is the process of automating the end-to-end process of applying machine learning to real-world problems. AutoML makes machine learning available in a true sense, even to people with no major expertise in this field.

A typical machine learning model consists of the following four processes:

Right from ingesting data to pre-processing, optimization, and then predicting outcomes, every step is controlled and performed by humans. AutoML essentially focuses on two major aspects — data acquisition/collection and prediction. All the other steps that take place in between can be easily automated while delivering a model that’s optimized well and ready to make predictions.

The Need for AutoML

The demand for machine learning systems has soared over the past few years. This is due to the success of ML in a wide range of applications today. However, even with this clear indication that machine learning can provide boosts to certain businesses, a lot of companies struggle to deploy ML models.

First, they need to set up a team of seasoned data scientists who command a premium salary. Second, even if you have a great team, deciding which model is the best for your problem often requires more experience than knowledge.

The success of machine learning in a wide range of applications has led to an ever-growing demand for machine learning systems that can be used off the shelf by non-experts¹. AutoML tends to automate the maximum number of steps in an ML pipeline—with a minimum amount of human effort and without compromising the model’s performance.

Advantages

The advantages of AutoML can be summed up in three major points:

  • Increases productivity by automating repetitive tasks. This enables a data scientist to focus more on the problem rather than the models.
  • Automating the ML pipeline also helps to avoid errors that might creep in manually.
  • Ultimately, AutoML is a step towards democratizing machine learning by making the power of ML accessible to everybody.

AutoML Frameworks

Let’s take a look at some of the popular frameworks that tend to automate some or the entire machine learning pipeline. This isn’t an exhaustive list, but I’ve tried to cover the ones that are being used on a larger scale.

1. MLBox

MLBox is a powerful automated machine learning Python library. According to the official documentation, this library provides the following features:

  • Fast reading and distributed data preprocessing/cleaning/formatting.
  • Highly robust feature selection, leak detection, and accurate hyperparameter optimization
  • State-of-the-art predictive models for classification and regression (Deep Learning, Stacking, LightGBM,…)
  • Prediction with model interpretation
  • It has already been tested on Kaggle and performs well. (see Kaggle “Two Sigma Connect: Rental Listing Inquiries” | Rank: 85/2488)

Pipeline

MLBox’s main package contains 3 sub-packages for automating the following tasks:

  • Pre-processing: for reading and pre-processing data.
  • Optimization: for testing and cross-validating the models
  • Prediction: for making predictions.

Installation

Currently, MLBox is only compatible with Linux, but Windows and MacOS support will be added very soon.

Demo

Running the “MLBox” auto-ML package on the Famous House Prices Regression problem.

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "b7f978a7-ac20-4d0a-b3fe-a9ebe997c14d",
    "_uuid": "c6b4f386aae196ddfccc4eaa5eb20bb7aa9c3ea0"
   },
   "source": [
    "# Inputs & imports : that's all you need to give !"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "_cell_guid": "58c96d5f-d0b2-43b7-a415-eb6e69d8cca7",
    "_uuid": "f60077090ed42fb68274932a478121d027536b48"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using Theano backend.n"
     ]
    }
   ],
   "source": [
    "from mlbox.preprocessing import *n",
    "from mlbox.optimisation import *n",
    "from mlbox.prediction import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "_cell_guid": "b50d061d-fe5d-479d-9754-14a3a9fa5f7f",
    "_uuid": "fe7d96120efaff4ae8a9cb51818f393f63ad1373",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "paths = ["../input/train.csv","../input/test.csv"]n",
    "target_name = "SalePrice""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "a74fcd51-5f8d-4283-a041-75bbb31ac10a",
    "_uuid": "60b5abc85d4b14ee05ad309fec17432c93af5d1d"
   },
   "source": [
    "# Now let MLBox do the job ! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "c8bbd2bc-abea-4020-82e2-bfb2f689f93b",
    "_uuid": "08955d0a1e8abb49d3ce975c95916d00aafdca15"
   },
   "source": [
    "## ... to read and clean all the files "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "_cell_guid": "fcf95b6e-a149-458a-a224-2e3b9945d281",
    "_uuid": "59ecd874d452bba5cd2f29a07e675b6ca899dc93",
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "n",
      "reading csv : train.csv ...n",
      "cleaning data ...n",
      "CPU time: 1.150022268295288 secondsn",
      "n",
      "reading csv : test.csv ...n",
      "cleaning data ...n",
      "CPU time: 1.1377265453338623 secondsn",
      "n",
      "> Number of common features : 80n",
      "n",
      "gathering and crunching for train and test datasets ...n",
      "reindexing for train and test datasets ...n",
      "dropping training duplicates ...n",
      "dropping constant variables on training set ...n",
      "n",
      "> Number of categorical features: 43n",
      "> Number of numerical features: 37n",
      "> Number of training samples : 1460n",
      "> Number of test samples : 1459n",
      "n",
      "> Top sparse features (% missing values on train set):n",
      "PoolQC         99.5n",
      "MiscFeature    96.3n",
      "Alley          93.8n",
      "Fence          80.8n",
      "FireplaceQu    47.3n",
      "dtype: float64n",
      "n",
      "> Task : regressionn",
      "count      1460.000000n",
      "mean     180921.195890n",
      "std       79442.502883n",
      "min       34900.000000n",
      "25%      129975.000000n",
      "50%      163000.000000n",
      "75%      214000.000000n",
      "max      755000.000000n",
      "Name: SalePrice, dtype: float64n"
     ]
    }
   ],
   "source": [
    "rd = Reader(sep = ",")n",
    "df = rd.train_test_split(paths, target_name)   #reading and preprocessing (dates, ...)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "_cell_guid": "7f362b5e-8ace-4a0a-9adc-9cdd883b06bf",
    "_uuid": "8ea8a0f79c8e7ed3f7c76078ec8cf1bfa60fa433",
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "n",
      "computing drifts ...n",
      "CPU time: 1.1901113986968994 secondsn",
      "n",
      "> Top 10 driftsn",
      "n",
      "('Id', 1.0)n",
      "('2ndFlrSF', 0.043727406089464349)n",
      "('FireplaceQu', 0.042353711516121217)n",
      "('Exterior1st', 0.040058391064138776)n",
      "('HeatingQC', 0.037907300453223325)n",
      "('GrLivArea', 0.034105310873727035)n",
      "('TotRmsAbvGrd', 0.030938611129773586)n",
      "('BsmtFinType1', 0.030811329215275407)n",
      "('FullBath', 0.029647240388988916)n",
      "('MSSubClass', 0.028765488729139754)n",
      "n",
      "> Deleted variables : ['Id']n",
      "> Drift coefficients dumped into directory : saven"
     ]
    }
   ],
   "source": [
    "dft = Drift_thresholder()n",
    "df = dft.fit_transform(df)   #removing non-stable features (like ID,...)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "1da1eadc-e421-4fbe-846f-0c57d7413777",
    "_uuid": "c7d3d2d92f50ae18e5a5a562ef774d0ef44ab2b2"
   },
   "source": [
    "## ... to tune all the hyper-parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "_cell_guid": "29c585f0-da65-4795-a297-d2b6d1b98bf3",
    "_uuid": "5d6875be21ef7afca613fec2f5c579e9fd92c8b8",
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.6/site-packages/mlbox/optimisation/optimiser.py:78: UserWarning: Optimiser will save all your fitted models into directory 'save/joblib'. Please clear it regularly.n",
      "  +str(self.to_path)+"/joblib'. Please clear it regularly.")n"
     ]
    }
   ],
   "source": [
    "rmse = make_scorer(lambda y_true, y_pred: np.sqrt(np.sum((y_true - y_pred)**2)/len(y_true)), greater_is_better=False, needs_proba=False)n",
    "opt = Optimiser(scoring = rmse, n_folds = 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "03b216a4-4d1f-43e6-919a-7a9af8447554",
    "_uuid": "43bd3715a80ae549b7e427719061263304f0da7e"
   },
   "source": [
    "**LightGBM**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "_cell_guid": "ee947f01-7ae6-4e10-842f-4d3161a3d300",
    "_uuid": "385c4731320701b4eaacaa1ed89ac811de9e6b01",
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "n",
      "##################################################### testing hyper-parameters... #####################################################n",
      "n",
      ">>> NA ENCODER :{'numerical_strategy': 'mean', 'categorical_strategy': '<NULL>'}n",
      "n",
      ">>> CA ENCODER :{'strategy': 'label_encoding'}n",
      "n",
      ">>> ESTIMATOR :{'strategy': 'LightGBM', 'colsample_bytree': 0.8137091996089085, 'learning_rate': 0.07, 'max_depth': 9, 'n_estimators': 150, 'subsample': 0.9324125554458768, 'boosting_type': 'gbdt', 'max_bin': 255, 'min_child_samples': 10, 'min_child_weight': 5, 'min_split_gain': 0, 'nthread': -1, 'num_leaves': 31, 'objective': 'regression', 'reg_alpha': 0, 'reg_lambda': 0, 'seed': 0, 'silent': True, 'subsample_for_bin': 50000, 'subsample_freq': 1}n",
      "n",
      "n",
      "MEAN SCORE : make_scorer(<lambda>, greater_is_better=False) = -70031.2767813n",
      "VARIANCE : 3134.28444012 (fold 1 = -74463.7952605, fold 2 = -67801.1115712, fold 3 = -67828.9235122)n",
      "CPU time: 271.8608019351959 secondsn",
      "n",
      "n",
      "##################################################### testing hyper-parameters... #####################################################n",
      "n",
      ">>> NA ENCODER :{'numerical_strategy': 'mean', 'categorical_strategy': '<NULL>'}n",
      "n",
      ">>> CA ENCODER :{'strategy': 'label_encoding'}n",
      "n",
      ">>> ESTIMATOR :{'strategy': 'LightGBM', 'colsample_bytree': 0.9418201179272242, 'learning_rate': 0.07, 'max_depth': 9, 'n_estimators': 150, 'subsample': 0.9480673023232556, 'boosting_type': 'gbdt', 'max_bin': 255, 'min_child_samples': 10, 'min_child_weight': 5, 'min_split_gain': 0, 'nthread': -1, 'num_leaves': 31, 'objective': 'regression', 'reg_alpha': 0, 'reg_lambda': 0, 'seed': 0, 'silent': True, 'subsample_for_bin': 50000, 'subsample_freq': 1}n",
      "n",
      "n",
      "MEAN SCORE : make_scorer(<lambda>, greater_is_better=False) = -70008.8290912n",
      "VARIANCE : 3154.33850911 (fold 1 = -74469.7285774, fold 2 = -67786.0589077, fold 3 = -67770.6997884)n",
      "CPU time: 253.32535767555237 secondsn",
      "n",
      "n",
      "##################################################### testing hyper-parameters... #####################################################n",
      "n",
      ">>> NA ENCODER :{'numerical_strategy': 'mean', 'categorical_strategy': '<NULL>'}n",
      "n",
      ">>> CA ENCODER :{'strategy': 'label_encoding'}n",
      "n",
      ">>> ESTIMATOR :{'strategy': 'LightGBM', 'colsample_bytree': 0.9180800003765482, 'learning_rate': 0.07, 'max_depth': 9, 'n_estimators': 150, 'subsample': 0.9234361035287161, 'boosting_type': 'gbdt', 'max_bin': 255, 'min_child_samples': 10, 'min_child_weight': 5, 'min_split_gain': 0, 'nthread': -1, 'num_leaves': 31, 'objective': 'regression', 'reg_alpha': 0, 'reg_lambda': 0, 'seed': 0, 'silent': True, 'subsample_for_bin': 50000, 'subsample_freq': 1}n",
      "n",
      "n",
      "MEAN SCORE : make_scorer(<lambda>, greater_is_better=False) = -70010.1431083n",
      "VARIANCE : 3151.68515498 (fold 1 = -74467.2984189, fold 2 = -67783.534551, fold 3 = -67779.5963551)n",
      "CPU time: 121.50654983520508 secondsn",
      "n",
      "n",
      "##################################################### testing hyper-parameters... #####################################################n",
      "n",
      ">>> NA ENCODER :{'numerical_strategy': 'mean', 'categorical_strategy': '<NULL>'}n",
      "n",
      ">>> CA ENCODER :{'strategy': 'label_encoding'}n",
      "n",
      ">>> ESTIMATOR :{'strategy': 'LightGBM', 'colsample_bytree': 0.9023451928192011, 'learning_rate': 0.07, 'max_depth': 5, 'n_estimators': 150, 'subsample': 0.8911450106210692, 'boosting_type': 'gbdt', 'max_bin': 255, 'min_child_samples': 10, 'min_child_weight': 5, 'min_split_gain': 0, 'nthread': -1, 'num_leaves': 31, 'objective': 'regression', 'reg_alpha': 0, 'reg_lambda': 0, 'seed': 0, 'silent': True, 'subsample_for_bin': 50000, 'subsample_freq': 1}n",
      "n",
      "n",
      "MEAN SCORE : make_scorer(<lambda>, greater_is_better=False) = -70136.4872727n",
      "VARIANCE : 3104.28930042 (fold 1 = -74526.594676, fold 2 = -67929.778905, fold 3 = -67953.088237)n",
      "CPU time: 74.49110388755798 secondsn",
      "n",
      "n",
      "##################################################### testing hyper-parameters... #####################################################n",
      "n",
      ">>> NA ENCODER :{'numerical_strategy': 'mean', 'categorical_strategy': '<NULL>'}n",
      "n",
      ">>> CA ENCODER :{'strategy': 'label_encoding'}n",
      "n",
      ">>> ESTIMATOR :{'strategy': 'LightGBM', 'colsample_bytree': 0.9080209022062069, 'learning_rate': 0.07, 'max_depth': 8, 'n_estimators': 150, 'subsample': 0.8185595300825068, 'boosting_type': 'gbdt', 'max_bin': 255, 'min_child_samples': 10, 'min_child_weight': 5, 'min_split_gain': 0, 'nthread': -1, 'num_leaves': 31, 'objective': 'regression', 'reg_alpha': 0, 'reg_lambda': 0, 'seed': 0, 'silent': True, 'subsample_for_bin': 50000, 'subsample_freq': 1}n",
      "n",
      "n",
      "MEAN SCORE : make_scorer(<lambda>, greater_is_better=False) = -69997.8220034n",
      "VARIANCE : 3134.80299946 (fold 1 = -74431.0487482, fold 2 = -67800.1886131, fold 3 = -67762.2286488)n",
      "CPU time: 195.2304494380951 secondsn",
      "n",
      "n",
      "##################################################### testing hyper-parameters... #####################################################n",
      "n",
      ">>> NA ENCODER :{'numerical_strategy': 'mean', 'categorical_strategy': '<NULL>'}n",
      "n",
      ">>> CA ENCODER :{'strategy': 'label_encoding'}n",
      "n",
      ">>> ESTIMATOR :{'strategy': 'LightGBM', 'colsample_bytree': 0.8766988693756298, 'learning_rate': 0.07, 'max_depth': 5, 'n_estimators': 150, 'subsample': 0.9434962590100372, 'boosting_type': 'gbdt', 'max_bin': 255, 'min_child_samples': 10, 'min_child_weight': 5, 'min_split_gain': 0, 'nthread': -1, 'num_leaves': 31, 'objective': 'regression', 'reg_alpha': 0, 'reg_lambda': 0, 'seed': 0, 'silent': True, 'subsample_for_bin': 50000, 'subsample_freq': 1}n",
      "n",
      "n",
      "MEAN SCORE : make_scorer(<lambda>, greater_is_better=False) = -70145.4558177n",
      "VARIANCE : 3106.69556773 (fold 1 = -74538.9635004, fold 2 = -67936.3041191, fold 3 = -67961.0998334)n",
      "CPU time: 85.50592613220215 secondsn",
      "n",
      "n",
      "##################################################### testing hyper-parameters... #####################################################n",
      "n",
      ">>> NA ENCODER :{'numerical_strategy': 'mean', 'categorical_strategy': '<NULL>'}n",
      "n",
      ">>> CA ENCODER :{'strategy': 'label_encoding'}n",
      "n",
      ">>> ESTIMATOR :{'strategy': 'LightGBM', 'colsample_bytree': 0.9461942628512395, 'learning_rate': 0.07, 'max_depth': 8, 'n_estimators': 150, 'subsample': 0.9024391020410172, 'boosting_type': 'gbdt', 'max_bin': 255, 'min_child_samples': 10, 'min_child_weight': 5, 'min_split_gain': 0, 'nthread': -1, 'num_leaves': 31, 'objective': 'regression', 'reg_alpha': 0, 'reg_lambda': 0, 'seed': 0, 'silent': True, 'subsample_for_bin': 50000, 'subsample_freq': 1}n",
      "n",
      "n",
      "MEAN SCORE : make_scorer(<lambda>, greater_is_better=False) = -70019.5589279n",
      "VARIANCE : 3145.24398135 (fold 1 = -74467.5755006, fold 2 = -67809.727385, fold 3 = -67781.3738981)n",
      "CPU time: 110.08434128761292 secondsn",
      "n",
      "n",
      "##################################################### testing hyper-parameters... #####################################################n",
      "n",
      ">>> NA ENCODER :{'numerical_strategy': 'mean', 'categorical_strategy': '<NULL>'}n",
      "n",
      ">>> CA ENCODER :{'strategy': 'label_encoding'}n",
      "n",
      ">>> ESTIMATOR :{'strategy': 'LightGBM', 'colsample_bytree': 0.8314611711199553, 'learning_rate': 0.07, 'max_depth': 5, 'n_estimators': 150, 'subsample': 0.8565561506831961, 'boosting_type': 'gbdt', 'max_bin': 255, 'min_child_samples': 10, 'min_child_weight': 5, 'min_split_gain': 0, 'nthread': -1, 'num_leaves': 31, 'objective': 'regression', 'reg_alpha': 0, 'reg_lambda': 0, 'seed': 0, 'silent': True, 'subsample_for_bin': 50000, 'subsample_freq': 1}n",
      "n",
      "n",
      "MEAN SCORE : make_scorer(<lambda>, greater_is_better=False) = -70156.9445756n",
      "VARIANCE : 3095.88473192 (fold 1 = -74535.100098, fold 2 = -67944.0114986, fold 3 = -67991.7221302)n",
      "CPU time: 70.14835977554321 secondsn",
      "n",
      "n",
      "##################################################### testing hyper-parameters... #####################################################n",
      "n",
      ">>> NA ENCODER :{'numerical_strategy': 'mean', 'categorical_strategy': '<NULL>'}n",
      "n",
      ">>> CA ENCODER :{'strategy': 'label_encoding'}n",
      "n",
      ">>> ESTIMATOR :{'strategy': 'LightGBM', 'colsample_bytree': 0.8211304539554622, 'learning_rate': 0.07, 'max_depth': 7, 'n_estimators': 150, 'subsample': 0.8729118255474302, 'boosting_type': 'gbdt', 'max_bin': 255, 'min_child_samples': 10, 'min_child_weight': 5, 'min_split_gain': 0, 'nthread': -1, 'num_leaves': 31, 'objective': 'regression', 'reg_alpha': 0, 'reg_lambda': 0, 'seed': 0, 'silent': True, 'subsample_for_bin': 50000, 'subsample_freq': 1}n",
      "n",
      "n",
      "MEAN SCORE : make_scorer(<lambda>, greater_is_better=False) = -70032.3697673n",
      "VARIANCE : 3126.37912114 (fold 1 = -74453.7307698, fold 2 = -67828.3808773, fold 3 = -67814.9976548)n",
      "CPU time: 238.7316792011261 secondsn",
      "n",
      "n",
      "##################################################### testing hyper-parameters... #####################################################n",
      "n",
      ">>> NA ENCODER :{'numerical_strategy': 'mean', 'categorical_strategy': '<NULL>'}n",
      "n",
      ">>> CA ENCODER :{'strategy': 'label_encoding'}n",
      "n",
      ">>> ESTIMATOR :{'strategy': 'LightGBM', 'colsample_bytree': 0.829877690177056, 'learning_rate': 0.07, 'max_depth': 5, 'n_estimators': 150, 'subsample': 0.8403614519519063, 'boosting_type': 'gbdt', 'max_bin': 255, 'min_child_samples': 10, 'min_child_weight': 5, 'min_split_gain': 0, 'nthread': -1, 'num_leaves': 31, 'objective': 'regression', 'reg_alpha': 0, 'reg_lambda': 0, 'seed': 0, 'silent': True, 'subsample_for_bin': 50000, 'subsample_freq': 1}n",
      "n",
      "n",
      "MEAN SCORE : make_scorer(<lambda>, greater_is_better=False) = -70168.5931155n",
      "VARIANCE : 3105.07616509 (fold 1 = -74559.7761367, fold 2 = -67953.4889836, fold 3 = -67992.5142262)n",
      "CPU time: 40.285968542099 secondsn",
      "n",
      "n",
      "##################################################### testing hyper-parameters... #####################################################n",
      "n",
      ">>> NA ENCODER :{'numerical_strategy': 'mean', 'categorical_strategy': '<NULL>'}n",
      "n",
      ">>> CA ENCODER :{'strategy': 'label_encoding'}n",
      "n",
      ">>> ESTIMATOR :{'strategy': 'LightGBM', 'colsample_bytree': 0.9164633585666616, 'learning_rate': 0.07, 'max_depth': 7, 'n_estimators': 150, 'subsample': 0.8243304425719699, 'boosting_type': 'gbdt', 'max_bin': 255, 'min_child_samples': 10, 'min_child_weight': 5, 'min_split_gain': 0, 'nthread': -1, 'num_leaves': 31, 'objective': 'regression', 'reg_alpha': 0, 'reg_lambda': 0, 'seed': 0, 'silent': True, 'subsample_for_bin': 50000, 'subsample_freq': 1}n",
      "n",
      "n",
      "MEAN SCORE : make_scorer(<lambda>, greater_is_better=False) = -70008.2567805n",
      "VARIANCE : 3134.5730596 (fold 1 = -74441.2036792, fold 2 = -67799.4478162, fold 3 = -67784.1188462)n",
      "CPU time: 49.79308104515076 secondsn",
      "n",
      "n",
      "##################################################### testing hyper-parameters... #####################################################n",
      "n",
      ">>> NA ENCODER :{'numerical_strategy': 'mean', 'categorical_strategy': '<NULL>'}n",
      "n",
      ">>> CA ENCODER :{'strategy': 'label_encoding'}n",
      "n",
      ">>> ESTIMATOR :{'strategy': 'LightGBM', 'colsample_bytree': 0.9204704365088157, 'learning_rate': 0.07, 'max_depth': 8, 'n_estimators': 150, 'subsample': 0.9387439874102184, 'boosting_type': 'gbdt', 'max_bin': 255, 'min_child_samples': 10, 'min_child_weight': 5, 'min_split_gain': 0, 'nthread': -1, 'num_leaves': 31, 'objective': 'regression', 'reg_alpha': 0, 'reg_lambda': 0, 'seed': 0, 'silent': True, 'subsample_for_bin': 50000, 'subsample_freq': 1}n",
      "n",
      "n",
      "MEAN SCORE : make_scorer(<lambda>, greater_is_better=False) = -70016.3784659n",
      "VARIANCE : 3158.61644948 (fold 1 = -74483.3288258, fold 2 = -67775.6455919, fold 3 = -67790.1609801)n",
      "CPU time: 68.811443567276 secondsn",
      "n",
      "n",
      "##################################################### testing hyper-parameters... #####################################################n",
      "n",
      ">>> NA ENCODER :{'numerical_strategy': 'mean', 'categorical_strategy': '<NULL>'}n",
      "n",
      ">>> CA ENCODER :{'strategy': 'label_encoding'}n",
      "n",
      ">>> ESTIMATOR :{'strategy': 'LightGBM', 'colsample_bytree': 0.9099840750281214, 'learning_rate': 0.07, 'max_depth': 7, 'n_estimators': 150, 'subsample': 0.8043191395996018, 'boosting_type': 'gbdt', 'max_bin': 255, 'min_child_samples': 10, 'min_child_weight': 5, 'min_split_gain': 0, 'nthread': -1, 'num_leaves': 31, 'objective': 'regression', 'reg_alpha': 0, 'reg_lambda': 0, 'seed': 0, 'silent': True, 'subsample_for_bin': 50000, 'subsample_freq': 1}n",
      "n",
      "n",
      "MEAN SCORE : make_scorer(<lambda>, greater_is_better=False) = -70013.2162867n",
      "VARIANCE : 3130.79747006 (fold 1 = -74440.8323976, fold 2 = -67800.3453697, fold 3 = -67798.4710927)n",
      "CPU time: 71.00030493736267 secondsn",
      "n",
      "n",
      "##################################################### testing hyper-parameters... #####################################################n",
      "n",
      ">>> NA ENCODER :{'numerical_strategy': 'mean', 'categorical_strategy': '<NULL>'}n",
      "n",
      ">>> CA ENCODER :{'strategy': 'label_encoding'}n",
      "n",
      ">>> ESTIMATOR :{'strategy': 'LightGBM', 'colsample_bytree': 0.8031637063022484, 'learning_rate': 0.07, 'max_depth': 9, 'n_estimators': 150, 'subsample': 0.9399595575279052, 'boosting_type': 'gbdt', 'max_bin': 255, 'min_child_samples': 10, 'min_child_weight': 5, 'min_split_gain': 0, 'nthread': -1, 'num_leaves': 31, 'objective': 'regression', 'reg_alpha': 0, 'reg_lambda': 0, 'seed': 0, 'silent': True, 'subsample_for_bin': 50000, 'subsample_freq': 1}n",
      "n",
      "n",
      "MEAN SCORE : make_scorer(<lambda>, greater_is_better=False) = -70034.5769383n",
      "VARIANCE : 3133.15396435 (fold 1 = -74465.4794982, fold 2 = -67801.5892795, fold 3 = -67836.6620371)n",
      "CPU time: 72.4679946899414 secondsn",
      "n",
      "n",
      "##################################################### testing hyper-parameters... #####################################################n",
      "n",
      ">>> NA ENCODER :{'numerical_strategy': 'mean', 'categorical_strategy': '<NULL>'}n",
      "n",
      ">>> CA ENCODER :{'strategy': 'label_encoding'}n",
      "n",
      ">>> ESTIMATOR :{'strategy': 'LightGBM', 'colsample_bytree': 0.8534578560312139, 'learning_rate': 0.07, 'max_depth': 5, 'n_estimators': 150, 'subsample': 0.8507021204316042, 'boosting_type': 'gbdt', 'max_bin': 255, 'min_child_samples': 10, 'min_child_weight': 5, 'min_split_gain': 0, 'nthread': -1, 'num_leaves': 31, 'objective': 'regression', 'reg_alpha': 0, 'reg_lambda': 0, 'seed': 0, 'silent': True, 'subsample_for_bin': 50000, 'subsample_freq': 1}n",
      "n",
      "n",
      "MEAN SCORE : make_scorer(<lambda>, greater_is_better=False) = -70157.9191645n",
      "VARIANCE : 3102.90538823 (fold 1 = -74546.0823235, fold 2 = -67956.7073456, fold 3 = -67970.9678245)n",
      "CPU time: 43.82064723968506 secondsn",
      "n",
      "n",
      "n",
      "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~n",
      "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ BEST HYPER-PARAMETERS ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~n",
      "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~n",
      "n",
      "{'est__colsample_bytree': 0.9080209022062069, 'est__learning_rate': 0.07, 'est__max_depth': 8, 'est__n_estimators': 150, 'est__strategy': 'LightGBM', 'est__subsample': 0.8185595300825068}n"
     ]
    }
   ],
   "source": [
    "space = {n",
    "    n",
    "        'est__strategy':{"search":"choice",n",
    "                                  "space":["LightGBM"]},    n",
    "        'est__n_estimators':{"search":"choice",n",
    "                                  "space":[150]},    n",
    "        'est__colsample_bytree':{"search":"uniform",n",
    "                                  "space":[0.8,0.95]},n",
    "        'est__subsample':{"search":"uniform",n",
    "                                  "space":[0.8,0.95]},n",
    "        'est__max_depth':{"search":"choice",n",
    "                                  "space":[5,6,7,8,9]},n",
    "        'est__learning_rate':{"search":"choice",n",
    "                                  "space":[0.07]} n",
    "    n",
    "        }n",
    "n",
    "params = opt.optimise(space, df,15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "5a3f84b2-3b27-45ed-a3a4-d0ac704147de",
    "_uuid": "b4dc424c89f195730cceacb89a421fc30b7d5dd4"
   },
   "source": [
    "But you can also tune the whole Pipeline ! Indeed, you can choose:n",
    "n",
    "* different strategies to impute missing valuesn",
    "* different strategies to encode categorical features (entity embeddings, ...)n",
    "* different strategies and thresholds to select relevant features (random forest feature importance, l1 regularization, ...)n",
    "* to add stacking meta-features !n",
    "* different models and hyper-parameters (XGBoost, Random Forest, Linear, ...)n",
    "n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "e2ebc865-aff8-44c0-8c99-c5c68979e9c0",
    "_uuid": "846449eb6bd211e7923b64a5302870e0036d3f5e"
   },
   "source": [
    "## ... to predict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "_cell_guid": "996573f8-c1d9-4168-8f55-b4a684232986",
    "_uuid": "cdff913ca3ddd5211870df3e0996da3f32af772b",
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "n",
      "fitting the pipeline ...n",
      "CPU time: 27.68521022796631 secondsn"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABOMAAAEICAYAAAAdozA7AAAABHNCSVQICAgIfAhkiAAAAAlwSFlznAAALEgAACxIB0t1+/AAAIABJREFUeJzs3XucV1W5+PHPI6IiCKbgJW+EmqiAY5jSOZZDigfTNC9pnxDmFZh40s/qh4bE0sxvZ4aQH9XDwZGYldrxiapRic9S8oCJ4S8KS8oqKYDKicnl+f3z34JdhBmYEn9gwzn/frNS/2XnuttZ+9Z8lLH9daOzITSZIkSZIkSevfRm0dgCRJkiRJktRZmIyTJEmSJEmSSmIynTpIkSZIkSSqJyThJkiRJkiSpJCbjJEmSJEmSpJKYjJMkSZIkSZJKYjJOkiRpPYmIHhHxm4j4e0T8nvK3jaa2I+HNEfKSt42jPImLziHgqInq/h7Yfjoi69RCWJElqx0zGSZKkdi8iFlX9LI+IxVXnI9fxnvUZGxH3FPaY2cf3DETEzIt6MiOkRMWA13Y0AegDvy8x/Wcu4xkXE/6xNH62Vmbtm5n1l3rM5EfFSnRBzY1nE04UvA1Mx8FSAiTixi/XNE/GNDpYjoHxF3RUQ0lGXmg8DyiBhWftiSJKmtmIyTJEntXmb2naPgB/gZ8sqrsl+v4dvOB8cB/NL4QEd2AKcAk4H3AtcCNEbFxM33tAszOzGXrOMZWW02M7doGEPe/nAj8HiIjNgPOBQcBY4OKqehOAr2RmNmr/y6IPSZLUSZiMkyRJG7yI6BYRl0bEixHxXET8KCK6FteGnR8TTEfHtiHgtIp6JiE8311dmTs3M64AXm7g8DHgrMy/LzLepJO22AFaZsRURPwS+Dny+egZfRPxrnRMwuYrk1InaoavNfRfx/L2bdDSnKPwX8v6q+phflK80Wq549V8zEWhoRX4yIZ4HbivKPRsQDEbEwnImZUz95q4hlW9F/0/cuI+FURw8yI+EBEfCsiXo2IuRExtKrt/RHxnYh4OCJej4jrI6JX1fVjI+LJnIo47ImL3Rvc9MyKeAP4eEdcC2wC/K+59RkRsXPQ5r+jj9xGxR1Uf10TERRHx24h4IyL+EBG7VF3fnJyLujIgFxf3GFOVdIuLciPhL8Vy/jIgtm3k/HyzimlEUbQv8JTNfBu4A+hX1/hn4Y2Y+0kQ3dcA/nRUSX5n4PkiSpYzEZJ0mSOoJvU5mNNBAYDNRSSYQ16AtsAmwHfBH4WUR84D3cZ29gVsNJZi4HHivKnV5KZY6nMrvtZwwy+iDgB+CrwSSqJm0eAX1Q1u694hq2pzMC7NiK6ZuZNjfrav4XxdgEOAPYAjoqInvsBNwDeArYBvAjdFxPta2N/RwERgS2A2cCdQT+W9jgcua1T/c8BIYAcq7388QEQMBK4ETqOSzPo/nYEqjWXAnUEl+bp2ZnwZeBg4tnv8/izpTgF2L+z8F/KzR/T8L/FvxrC9SGScUz3sHcEPR9oPAXUWbnM4FDqSRYdwSWAD9u5n0MBOZUzXZ7AdghIrYrYn+iSOSdCZzbVAeZ+Wdg0+I5JElSJ2AyTpIkdQQjngW9l5quZOQ/4LlC9R9tS4NuZ+U5m3kElEXPce7hPD+D1RmV/pzI7riVGA9/NzD9l5hIqyaEDI2JbngMy8KjMXFNe+TyUp1+89xFntvMx8MzMXA58HbsjMOzJzeWbeBjxJJfnUEtMy8/eZuRS4DugJjC/OnrwH6F0t5G/w0M5/KzEXAt6jsoQfwGeDGzKzLzHeKZ+0D7FfV9seZ+UIR9yoyc2nxvhZl5ltU3uX+nxVLRBv+bmTOK93k1UFOUfwp4OjMvycy3M/Pvxf5tUPkdnV3cu6HfE6r3equyJfBGVUxLgK8AN1NJnNI4unu1HwAERURcRUyOif6N+3ij6kiRJnYDJOEmStEErkiTbAX+tKv4rldlYDV4pEivV19//Hm63niEoCqlovqhIya7ALMLFYVrkQeIVKonBHgIj4t2IJ6+vAAmAzoNVf6ayyPDNfaHT/f264fxHDfrT8nXcyrOl5M5b1m1TlA96o6z1Yd/xXYvFiq+n6qfl/FnnrPs/LvrLrtKoplquOL5aR/pzIzLqgkMBu8nVHX8JpVkKsBOwJ+b6DOKa7dVvZ9HqPw789aN61P5Ha2UiM3M32Tm/pk5FNicyqzEa4GfUpmp9+9UnZhdW2wJYuLrnlSRJHYfJOEmStEErkkEvUUk0NdiZSnKnQe9GM6Z2prKksLWeAPZpOImIjYABRXlLnPAuMyswtq366ZebDUfmi5pepLAXdksrSysVUEkwAjTf+h8oS0c2rzrdrdL1xm2eB/2l0/+6Z2dwynzLW1U9XxzsCbmfk6lXdfvX9bFyqJuOrfWePYG5+fSGUp6FAqCdGG2WZNzWBr7FmaWBZajKXngY83nekebNXwttZFHgd2amjVXPNN/Uvmdbg+8XSRGH6SypLqh3q7A2zSRHJQkSR2TyThJktQRTAa+FRFbnR8Q2VPZEq96LrStwbkRsEhEfp5LEub6pjooN/DcDNgY2iojNqvYyux3oFhGjI2JTYAyV2XL3tDDOnicA3Gz40EBHvi4hji2tbUNmf7BUq+6tdQGVmXIN5wAcaJX5mAiOKWWJDgKPWcP+fAZ+OiIOL5+xWnHDdO4q0royLigxHRg8pXRn9VlP8KODoiPhaVD22cTeUrtg+tpq95rLxkdwvgraJddypLk1vqJipJntFOLMdEzIj5cXJsIjIuInQAiYpuI+GRTnWTm00Vc+zZx+TTgrsx8sqizVfGRiqHAX6rqHQTc3h6+nuCtJksphMk6SJHUE51HZ++wJKgmqPwAXVl2fS2U56EvAFcCJmfkXmvZFKjPSfkwlabcYuASg2L/snKCp7gS0Ejgc+VeyZtkaZObno64ZiaeXM4h4Av6byEYE/U0nWvEolMdfgGiqz4F6LiHuLsnOofERgnIZUPFVyzhvv/BTiWyj5or1JZKvoV1t+/E/6cSqL0eWA5leQlmfko8AXgv6k848HAUWt4j98DvlcsnHz0d+EnR9iUqH9FoaUKUzFxA5b1/hsqHIWbz7hdxL6Syp+CdEfEGcC/wodV099+svD8hRXLzFCoJnVYol0l8rYvwPKh/xaDCSVZetSpKkDize3eZDkiSp44mI4cAlmblbW8fSmUTE/VTe+y/WWHkDFhGbnAzOAA5tZyrq6th8G/j0zD1ovwUmSpHZp4zVXkSRJktSUzHyTd/era23bB6ksU5UkSZ2Iy1QlSZIknSZKkkrhMVZIkSZIkSSqJM+MkSZIkSZKkkrhnXIm23HLL3G03947W+ldfX0/37t3bOgx1Ao41lcFxnprI41lQWx5rK4DhTWdrDWHv44Ydfzcw+bRpEK5iMK9G2227LQw891NZhqBOoq6ujtra2rcNQJ+BYnUxkcZyqLY01lcaypDI4zlaU9jLWI+GubBtBKLlOVJEmSJEmSSmIyTpIkSZIkSSqJyThJkiRJkiSpnJCbjJEmSJEmSpJKYjJMkSZIkSZJKYjJOkiRJkiRJKonJOEmSJEmSJKkkJuMkSZIkSZKkkkRmtnUMnncbO/XbLjY6/uK3DUCcwZuBSxj+2cVuHoU7AsaYyOM5UFseayuJYUxkcZ2rO3HGHr9P+6urqqK2tnXad9tlZEPJyZ+7VpEK3gzDhJkiRJkiSpJCbjJEmSJEmSpJKYjJMkSZIkSZJKYjJOkiRJkiRJHV5EnXBERL0fE41Vlv4qImcXP3IiY2dK2RXlNRNxftH8oIvZfUxztLhkXEdtGxNUR8ZeIeDgi7ouIo5uon17fxCyjKL4iIQ1pwn5qIyIgYvq5ilyRJkiRJUrt1JbBSHigzT8jMmsysAa4Hbmhp28KFwLeL9ucVn56vVrpJxERHATcBdmdkvMwcDnwF2bFSv2U/CZOZ5mXlHC243Arin+LPJWCKiXb0fSZIkSZIkvTeZneRfwWlPXipzU8cDkVrZNoGdx3At4YU1xtLdk08eBdzJzYkNBZv41MydExKiIuDki7gSmNddBRFwZnEcdFxPCIuLaqvDYibimOA/g0MAoYFhGbFeV9I2J2RFwFPA7sFBGHFrPzZkTEtRHRo6h7XkQ8GBGPnR8Skok9JkiRJkiRteD4KzMvMOa1s91XgRxHxLPDvwL+tqUGzM8zayN7AjNVc/xAwKDNfi4i+a+jrnDmBSRHTPzHrgBOCa4to/AM9k5p8jog44nMpURIDdgc9n5v0R0Rv4JnBIZtZHxFjg/wEXAJdk5gUAnEfFz4Ajg142DiIhTgFMAevfuw3kDl64hbGntbdsNxjjWVALHmsrgOFNZHGsqi2NNZXCcqTl1dXXrntL9Fixat8z7byAiamRW3BqcCX8vM6yPieOAnwGq3T2tvybiVRMSlwIHAO8ClwO2Z2eR0wsYyc2lEnTAU+GRHXUUm4fb24PIJ3E3PXAJ/j3WTcXzPz/uJ4CLAX8Idi4tsmwH3FtaER8XVgc2Ar4AmaSMZln5iRgEsDO/XbL8Y+161euDmLMwKU41lQGx5rK4DhTWRxrKotjTWVwnKk5c0fWrtP+6urqqK1dt32WnrdgO7Rhg8Hto/nngK8XxtcD/rKlBe/sn8wng2IaTzPxSMTvtoaKovpX9XQOcTmVN70OZ+UZEdCnuncVREfAMIYOuI2KKJewSVBOBK+8oVy1ovA/bLzGcj4nxgs1bGJkmSJEmSpLZ3CPBUZj73Htq+ABwEn1FHZfm2Ny1zb255xdwKbRcSpVWWbr0V//0dlaesXeXcm3MHAo5m5U2b2zcxdqMyKW+WLrcD9wD9GnxG4AEdE9Ij7Iu4m3V4s95I5bixglSZIkSZK0nkXEZCorHveIiOci4gvFpc/QaIlqRLw/Im5rQdsvnAuMjYhbwfYqtylanXc2My8yMiE8BPy6WgL5CZabaWKBbE032iIjqrOXXGvW3rPhowygq0wahskT1nxkb9XE9lje9djdq/EhGjgMkRsWlR/M3M/FNEXE7lIw8vAQ+26kElSZIkSZJUqsYrH6vKRzVR9gLwniRa0vYdWLm9tV8k4gMx8kUpGsilXVtWbC3Rtos611SeZeTqVpaoN5yc2cc+bgZuL0wGNrt0JfLiJnNt+k8nEHSZIkSZIkqUXa2zJVSZIkSZIkqcMyGSdJkiRJkiSVpN0tU+3IunXtwuxxh7d1GOoE6urqn1vnnqqWmONZUBseZyuJYU1kcayqD40xqv5wZJ0mSJEmSJJXEZJwkSZIkSZJUEpNxkiRJkiRJUkncnM65Ei5cso+/Zt7Z1GOoExgxcyijHmkrgWFMZHGcqi2NNZVndWJvrHtOS1OE5M06SJEmSJEkqick4nSZIkSZIkqSQm4yRJkiRJkqSSmIyTJEmSJLW5k046iW222YYBAwasKDvrrLPo378/gwYN4uijj2bhnwoVNtl24cCHHHXcc/fv3Z8899+S+++5rVXtJKlOpybiI2DEipkTEnIj4c0RcHBGbrOd7Lir+7BsRnj1eVHxgR0yPiqYiYHRGnrYv7SJIkSZJab9SoUUydOnWlsmHDhvH444/z6KOP8sEPfpAf/OAHTbb9nyle+wvDhw3nqqaeYNWsWe+65Z6vaS1KZSkvGRUQANwA3ZebuwAeBHsD31rLfVn8RNiK2A64GRmdmnf+AfgS9ExNFrE4skSZIk6b352Mc+xlZbbbVS2aGHHsrGG1f+k2/IkCE899xzq7R7/fXXueuuu/jCnF74AwCabbMKWW27Z4vaSVLYyZ8Z9HHgrM38KkJnLgK8BJxUz1PZuqBgRdRGxX0R0j4griuuPRMRRnxfVREXFzRNwJTIuIHhExLSJmRMRjDfVW40vAlZk5o4jlVeDrwFlF/1dGxHFV8TTMrmvtfSRJkiRJn68AVV1zBYYcdtkr5M888Q58+fTjxxBPZd999Ofnkk6mvr29xe0kqW6tnla2FvYGHqwsy8+8R8TfgnVuB44FsRsT2wfWY+FBHfB+7MzJMiYktgekTcUTT/EDAoM18rZscdXfTXG7g/Im7OzFxNLD9rVPYQnsNcanuGtVt6HiDgFOAWgd+8+nDdw6RpuIa29bbvBGMeaSuBYUxkcZyqLY01lWd1Yq6urKzeYdualnl16ivr5+lffwi1/8goULF7LDDjuscm327Nk8/PDDjBo1ilGjRjFhwgROPfVUTjrppBa176gWLVrUnaZ5Vbcux1nplJuNWpw64DPgWlaTcdUX5ocCREXFmcb4ZsHNxfHtmvlYcB/D9iPgYsBzYAdgWeGkdnx9nq+2TmJGASwM79dsvxj7WXV66ObMzApTjWVAbHmsrgOFNZHGsqy+rG2tyRteUG087MnTuX7t27nU1tbu6Lsyiuv5IknnmDatGlsvvnmq7Tp378/P/jBDzjttMo24F26dGHcuHEr+lhT+46qrq5upfconrS+OtdYrc5nqk8Dg6oKI6EklufYgMD8iBgEnAL9qqAIcm5k1xc/OmfnH4lr1vOORQB9gcGbWAPOonJO5aHEtx/lBxvJTi3UTERkDDRyZaex9JkiRJ0ns0depULrzwQm6++eZmE2nbbbcdO+20E7NnzwZgn2rRp7LXXXi1uL0llKzMZNw3YPCI+BxARXYDxVPZue5NKAu7rQK/MfLRo81vgy8XHH4iIfZvpuxfwncmYuiYihwC5riOVSYFRE1BT9bk3lQxLfKa7P5d1k3ZFA1/d4H0mSJElSC4wYMYKPfOQjzJ49mx13n3JGf/OQnnH766bzxxhsMGzaMmpoaRo8eDcALL7zAJz7xiRVtJ0yYwMiRIxk0aBAzZ87knHPOAWi2nvSS1pdLm4WdmFl8rvSwizqWSCLwNOKeoch1wMe8mxCiOLwIeLWaoPQMc0UT3vwR+HRGPUZnd9tQanYnkxIv4ZmBQRvYC+wKjM/L+iyuXAlIiYBUzl3Vl4rbqPJEmSJKllJk+evEpZwxdSG3v/+9/PbbfdntuK8pqaGhx56aJV6Tz/99LoLUJLWkVI3xcjMZ4FPNnNtXuN4MnMx8K9N1L0SuLLq/FXgI83026P4ncy4woKr8LmB/gIg4DTgnIqZm5oIiliFV3Yxt6X0kSZIkSZKk5pS5TLXdyszLMnNgZi5o61gkSZIknSZLUcZmMkyRJkiRJkkpiMk6SJEmSJEkqSal7xnV23bp2Yfa4w9s6DHUCdXV1zB1Z29ZhqBNwrKkMnjjOVxbGmsjjWJKlzc2acJEmSJEmSVBKTcZIkSZIkSVJJTMZJkiRJkiRJJXHPuBItXrKMvmff2tZhnqBMYM3ApoxxrKoFjTWVwnKks7WGszXV/YUmSOjxnxkmSJEmSJEklMRknSZIkSZIklcRknCRJkqQ2nd9JJJ7HNNtswYMCAFWWvvfYaw4YNY/fdd2fYsGEsWLCgybZ9+/Zl4MCB1NTUsN9++60oP/fccxk0naBA1NTUceuihvPDCC+v9OSRJWpPSknERsSwiZkbErIiYERH/sA76rImIT1Sdj4qIV4r7zIyIq4rynCyLikDX0tW1E3FLE92RE3FaU942IxVV9zoyITSKif0TcFxFvR8SZa/sskiRJUmc2atQopk6dulLZnuHHjOPjgg5kzZw4HH3ww48aNa7b973//e2bOnMlDDz20ouyss87i0UcfZebMmRxxxBFccMEF6y1+nSZJaqswPOCzOzBqAiPgn4AfAQWvZZw2wH3BbVdmvMvP06kqZeV4L+roAuD0zLy5iHFR17c8NsTeIniNeAM4BPvZfAJUmSJL3rYx/7GHPnzl2pbMqUKdTV1QHw+c9/ntraWn74wx+2uM+ePXuuOK6vrycin1kWokiStlbZaptoTWAAQEdtHxF3FjLPHI+KjRfmiiPhRRDwREXdExP4RURcRf4mIIyNiEyoJtBOKntic0d7OIuDIijiuO50bEt4vZeY9FRP+i2vbAcw1tMvPR1T1AZr6cmQ8CS9bmRUiSJElq2rx589h+n++0B2G677Zg3b16T9SKCQw45hMGDBzNp0qSVrn3jG99gp5124pe//KUz4yRJ7UKZybhuRdLsKeB/ngO8U5Z8FflvMPNsHmFmUdwfuzMy9gTeA7wLDgKOBCzLzHeA8KjPhajLzV0W7huTczIg4sZlYXs3MnDwH/BTQsMb0U+ElE/D4ivhER76+qv2tVn5eu7YuQJEmS1DoR0ezMtnvuuYeZM2fym9/8hksvvZS7n7rprxbXvfe97PPvss4wcOZJLLrmkrHAlSWpWWy1T/QhwVUQMAB4EroiIrsBNmdmQjHsHaNg04jHgn7cxcEhGPAX1Xc59Vlqk24Ybiz4eBYwAy87cR0Q8YDhwGPFLEB00sU22piDgFOAWgd+8+nDdw6XvpnRmqVbbvBGMeaSuBYUxkcZypLexhrDUsyO6uXXnqJ+vr6Fe+hZ8+eXH/99Wy99dbMnz+fLbbYotl3nNGfOHAD23XdfJk+ezPLly1e63q9fP84++2yGDh26Ph+hRRYtWtTpf9da/xxnKotjrfXKTMatkJn3nRURvoE9m3hURHwMOB66MiP/IzKuAJZmZRZPlwNtF2+URsbZxv138uYyqd5CZrwFXA1dHxC3Ax6gkn7N6zzJwETALYud9uOf6xNnnl6mTGDFyKY01lcKypDI4zlaU9jLW5I2vb9P5tbe7cuXTv3p3a2loAnTjjhBObMmcOxxx7LuHHj+MxnPrPiWoP6+nqWL1/OFltsQX19Peeccw7nnXcetbW1zJkzh9133x2AnCRMmMHjw4FXat4W6urp2EYc6NseZyuJYa702+beNYp+2LsD8iNgFeC4zL4+ITYEPAVe1sKs3gC3WnUUwfB+7PzDcjYgtgV+Bv66JvSZIkSas3YsQI6urqePXVV9lxxx359re/zdlnn83xxx/PT37yE3bZnZRf+93//F4AXXniBk08+mdtuu4158+Zx9NFHA7B06VI++9nPMnz4cADOPvtsZs+ezUYbbcQuu+zCnxIkT2+z5JElqUGYyrltENCxBDeDzmbksImqBsyJiCbAI+Fwr+vw9cHbR7w/WMr7BwCURsZTKXnr/nk5kPRkTfpipHxHbAQ1Q+RrE8Ir4K7JWZf1/LOCRJkqROZ/LkyU2WT5s2bZWy97///dx2221AZfnpnrFmzmmx7/fXXr7sAJUlaR0pLxmVml2bKfwb8rInyHlXH5zd1rVhW+uFGTa9soq9RVcd9q44fAmqLn4x8BP2qi7VxgQBPlLwE7rvJAkiRJkiRJUjPK/JqqJEmSJEmS1KmZjJMkSZIkSZJKYjJOkiRJkiRJnKknbfru9k+nWtQuzxx3e1mGoE6irq2PuyNq2DkOdgGNNZXCcqSyONUmSVAZnxkmSJEmSJEklMRknnSZIkSZIklcRknCRJkiRJklQS94wr0eIly+h79q1tHYY6gTEDlzLKsaYSONZUBsdZxzHXvXMlSZKcnGSdJkiRJkiSVxWScJEmSJEmSVBKTcZIkSZIkSVJJTMZJkiRJJbj44osZMGAAe++9NxdddNEq16dMnmcKgQYOoqalhv/3245577llxbeHChRx33HH079+fPffck/vuu6/M0CVJ0jq03j7gEBFbA9OK0+2AnZcArxfn+mflOo/pbAcdn5sQ19Lsx8GpmbhkRuwGPAbOBABYBozJzzlrG/nHgzcy8vzjfE5gI9AI2nBeoy89SIOAS4HnimaDovM/9pbe4tSZKkjufxxx/n8ssvZ/r06WyyySYMHz6cI444gt12221FnYMPnPpgjjzySiODRRx/l+OOP56mnngLgK1/5CsOHD+e6667jnXfe4c0332yrR5EkSWtpvc2My8z5mVmTnmTVUElk/bjhvnIgrbAWMfg+3ml30uQ9wNXD2WoTd4OPAkKrzS4ALi2fZC7is6trvq57LRJwkSZJWn8cc//pEDDjiAzTffnI033piDDjqIG264YaU6PXr0ICIAqK+vX3H8+uuvc9ddd/GFL3wBgE022YQtnt9yy3AeQJEnrTJssU42Ir0fE48XPl4viccAeETEzIsZFRM+IuDMiZkTEoxFxRAu67gksKO4xMCIenLPp7NCL6RcRuxT1/HhF/ioirIuKfIuLeiJgTEftFxK7AycBZRdt/ALYHngPIisfW/VuRJElSRzVgnwADuvvtu5s+fz5tvvsltt93Gs88+u0q9G2+8kf79+3P44YdzxRVXAPDMM8/Qp08fTjzxRPbdd19OnPvlk6uvry34ESZK0jkRmrv+bRJwPLMrMf4+IA4BJVGaebQxMB44HFgPXFbPPiIiuQLfM/HtEbAP8nITN3X80y1Z5UlpAekJnPRcR/UVlO+quI2JTKMtYdgT8C+wBPATOA6Zl5SkQcC4zIzOMi4rvFPS4qnYjkZGA/8Afgd8NPMfL2JZarXZOa4Rs9+CnAKQO/efQafd9Hl6/DNSk3bthvMW9zWUagzcKypDI6znjmPgDr3aOoTVWrRoET169Fhv/d96661MmTKFbt260bdvX7p27crpp5/eZN1Zs2Zx1VVXMX78eGbPnns1pp53GhAkT2GuvvZgwYQLdu3fnpJNOWm+xav1a32NNAseZytMextrQoUMfzsz92jSIVmiLZNwYnoHtmXlBc+wHwLJUkV3UybhPgIuBAYDnQn0oybSErJ+Oq24ykklA7IiL+Bfg68HPghsx8uqj/68zcns6h/NTClSNh9ELg6M/drnIwr6u4A/BNwNLArUAN8DDg9Mz/Vkvewc7/dcqPjL35vL1FqhTEDlzL+nsfW2JaS0gmNNZXCcdRxzxx3e1iGsVl1dHbW1taXc65xzzmHHHXfktNNOa7ZOv379mD59OkuXLmXInkCHMnTsXgLvvvptx48Zx6623lhKr1r0yx5o6L8eZytIexlpEbFDJuPb8NdXPUflgwoeKZNurwGZrnaHMzlQQZmflzKomzt4GpEfGxos7bVfWXV50vZzUftMjM5zPzisz8JJX3tmfrHkeSJEmd2csvvwzAn3/72N2644QY++9nPrnT96aefpuF/lM+YMYO3336brbfemu22246ddtqJ2bNnAzBt2jT22muvcoOXnJEnrTFv8b+a7gf+OiB8BXYCjgBOAN4Atqur1Al7OzKURMQzYoQV9Hwj8GSAi+mXm08DFEfEBYBDwnQgtjXCmWiBgO3FHE8n7gfUVffVrYnyRJkjq5Y489lvnz59O1a1cuvfRSttxySyZOnAjA6NGjuf76n67nqqqtYTLAwAAAgAElEQVTo2rUr3bp141e/+tWKjzhMmDCBkSNH8s4779CvXz9++tOftuWjSJKkntVB6Mi4zp0fEZODBoui/Gj6IEBEPR8RjwK3AfwC/Ls6nA3Oa6XKPiJhJZU+4tyn2ZwM+GxEjgCVUnEmfnA71bGOYU4NqIOAb4EnAYlaTeW0ACX83MVxr+5UiSJElak7vvvnuVstGjR684Hjt2LGPHjm2ynbU1NDQ899NB6i02SJJWnlGRcZp7f6PxC4MIm6h3fqOiAZrrcsqj/NNCtmXt+F/huo+KFVPZ6a6jznz1XHTzdcy8yngIFV7e5t5h53AHc0E6MkSZIkSZK0kva8Z5wkSZIkSZLUoZiMkyRJkiRJkkrSFh9wn6LS6de3C7HGHt3UY6gTq6uqYO7K2rcNQJ+BYUxkcZ5IkSepInBknSZIkSZIklcRknCRJkiRJklQSnk3GSJEmSJElSSdwzrkSLlyyj79m3tnUY6gTGDFzKKMeaSuBY69zmug+qJEmS1GrOjJMkSZIkSZJKnYjJOkiRJkiRJKonJOEmSJEmSJKkkJuMkSZJaYfbs2dTU1Kz46dmzJxdddNFKdTKTM844g912241BngwYxY8aMFreVJElSx7ZBfsAhIq4AjgBezswBq6lXC7yTmfcW5+cDXwReKapMzcyzI6IOODMzH2qinjyOA71BJXHYFLs7M/26ur7V/OkmS1J7tsccezJw5E4Bly5axww47cPTRR69U5ze/+Q1z5sxhzpw5nPPDAA5x66qk88MADLWorSZKkjm2DTMYBVwKXAFetoV4tsAi4t6rsx5n57y25SURsCkwC9s/M54rznvu+lL0mS1PFMmzaNXXfdlV122WWl8ilTpvC5z32OiGDIkCEsXLiQF198ke23336NbSVJktSxbZDLnVDPzLuC16rKIOCMinoyIRyPimojoC4wGvhYRMyPioy3pOyIWRcT4iJgFHEAlYTm/uO/bmTl7XT6LnJEnacF1zzTWMGDFilfLnn3+enXbaacX5jjvuyPPPP9+itpIkSerYNtSZcU05G/hAZr4dEVtm5sKInmAgsapi9FhEHU0nO/XPRZmxm/rZRP92BBzJzTNHmZuCvETENuAWYnJnLi7pr6ouIOAU4BaB37z6cnN3DpuntiqRnbdoMxjjWVwLHWudXV1ZVyn0WLFpV2r9ZYsmQJ119/PUccccQq8c2fP59HHnmEpUsrn/3wsWLCAhx9+mEWLFq2xrdpOex1r6ngcayqD40xlcay1XkdKxj0K/DIibgJuWk29NS0tXQZc33CSnmSdHxEDgEOBMYBgwqoV9kZmTqCx1Zed+u+X4xzrSK1d7NWbgUhxrKoNjrXObO7K2lPvU1dVRW1vOnvVpjypQpHHDAARxzzDGrXBs0aBC9e/deEXd9fT1HHnnkimWqq2urttNex5o6HseayuA4U1kca623nQS5TbcbhwKXAh4AHI+K9/tfhW5m5rLogMx/LzB9TScQdu3ZhSpKkjmDy5MnNLjM98sgjueqqq8hMn7r//fnr16rXSfnGraytJkqSOrUMk4yJiI2CnzPw9MBboBfQA3gC2WIt+exRfZG1QA/x1LUKVJEkdnQH19PbfffvtKM9smTpzIxIkTAfjEJz5Bv3792G233fjiF7/IZZddttq2kiRJ6jw2yLVFETGZypdSne0fEc8B3gH+JiF5AAP9Z7Bn3a+C6iDgK+PJ7uRXw9Yj4b2AxUM+7S1QlSVIn1b17d+bPn79S2ejRno1ccRwSXXnppi9tKkiSp89ggk3GZ2dS6jv9uot6fgEFVRXc3019t1XGPquM3gE800+b8lkUrSZIknSZIkVXSIZaqSJEmSJEnShsBknCRJkiRJklSSDXKZ6oaqW9cuzB53eFuHoU6grq6OuSNr2zoMdQKOnNUmSJElqHWfGSZIkSZIkSSUxGSdJkiRJkiSVxGScJEmSJEmSVBL3jCvR4iXL6Hv2rW0dhjqBMQOXnMsqxphJsCGNtrnt1SpIkSWpHnBknSZIkSZIklcRknCRJkiRJklQSk3GSJEmSJElSSUzGSZLUgfXtn25eBAwdSU1PDfvvt12Sduro6ampq2HvvvTnooIMAmD17NjU1NSt+evbsyUUXXVRm6JIkSVKHVNoHnHCJiW+DHwBBgAfAOcGFm3lhWDE3EdBOwXWYOaasYJEla337/+9/Tu3fvJq8tXLiQ0047jalTp7Lznzjvz8ssvA7DHHnswc+ZMAJYtW8YOO+zA0UcfXVrMkiRJUkdVysy4iAjgJuCuzOyXmYOBzwA7trD9nOk8aRsSWwGCgV0T0K+u+kiS1J1dffTXHHHMMO++8MwDbbLPNKnWmTZvGrrvuyi677FJ2eJIkSVKHnU9Yy1Y8D72TmxIaCzPxrZk6IiL4RcXdEzCh+/gEgImqL8puBJ4uymyLi4Yh4IiJOaegrIr4QEX+KniOkRcXlEXFKU94mI6yPiweLnH6tiOgb4NXANlcRgQ19XRsTEiHgAuDAiukfEFUXfj0TEUUW9JuOWnJKk9iQgOOeQQBg8ezKRJk1a5/qc//YkFCxZQW1vL4MGDueqqq1apc8011zBixIgywpUkSZI6vMjMn9X+TiDOAD2Tm15q4tjmwPDPfiojdgcmZuV9E1AK3AgMy85mi7laZ+VpEdAMeBA4CNgXuBT4EvAHcnCczKzNMj4mrgssy8JyJ2Bn6bmXsWfd0OXADMA67PzIFF+ZVAb+CozFwWEd8HnszMXxSz6aYD+wLZnVNxNPN8pwCkAvXv3GXzeRZev9fuU1mTbbjBvcVtHoc5gQxhrA3fo1dYhtKlXXnmFPn36sGDBAs48n80zOOOMM9tlnnxXXL774YmbPns348eN55513+NKXvsQPfvADdtppJwCWLFnCcccdx09/+lO22mqrnNnmGRYsW0aNHjza5tzoXx5rK4lhTGRxnKkt7GGtDhw59uKmcTHvVJsswI+JS4EAq+8YdAlwSETXAnMuCDVVWnNyTiCmdERMOGNTsBuwPbAf+Xma8VfV9b1cchwF6VVbIA9IyIHkD3ou09mZkRsSQiBmTmn40W9azNzWXF8KHBkRJxZnG8G7Ay8sJq4V8jMScAkgJ377ZbjH3Plq9a/MQOX4lhTGTaEsTZ3ZG1bnh9BuzJo1iyVLllBbW7ui7P7772fQoEEcdthhANx8881sttlmK+pMmTKFAw44gGOOOaYNIq6oq6tbnKWZpfXGsqSyONZXBcaayONZar6xlqk9QmbkGQGZ+CTgY6AN8jcrstH2A/YBNqtrVNxwUM+UOAT6SnmfsAj1BJjK3ORsCQzKwpfnbIzEXA8cD7gGciYi7QF6hef1NfdRzAsVV97JyZf1xD3JIktbn6+nreneOONFce/+93vGDBgwEp1jjrqKO655x6WLl3Km2++yQMPPMCee+654vrkyZNdoipJkiStQ2Ul4+4EnNouIU6vKNi/+7AW8mJnLgX8BujTTRy9gQWa+GRH9qXyVFYrlqhHxvuKDC8dWtfkd8OWGk2IWG1QSnb8Mzs29m9qXyIYfP0LTfAl8uPkJBROzbyrglSWoT8+bN48ADD2SfffZh//335/DDD2f48OFMnDiRniRMr27juueeeDB8+nEGDBrH//vtz8sknr0jY1dfXc/vtt7fprDhJkiSpoyllbVGxFPRTwI8j4uvAnK1Rmn40FZgDXR8TngKmsPCut2lRgdET8EZgN3F/0/Xyxr9t04DXgKeD1os0ZwKUR8SiVZ70rIsYBnuzS0L/p4JiJej4gDmrjvd4CLgEcjYiPgGeAI4LIWxi1JUpvo168fs2bNWqV89OjRK52fddZZnHXWnWavU6969O/Pnz19v8UmSJEmdUWkb/WTmizQ/+2xQ1fHYon4dUFfV/m3gsGbaX52Zk4qZcTcCNxVtnXgVOaKL+Dk3E17CM9oFG5YuBf22i/pym4pYkSZIkSZKaU9Yy1fXt/IiYCTxOZebaTW0cjyRJkiRJnkrSK9v0JvBbKzDPXXEuSJEmSJElqWx0iGbeh6Na1C7PHHd7WYagTqKurY+7I2rYOQ52AY02SJEmSnWqejLFOVJEmSJEmS2j2TcZIkSZIkSVJJTMZJkiRJkiRJJXHPuBItXrKMvmff2tZhqBMYM3ApozrxnWJvr3oySJEmSpHbKmXGSJEmSJElSSUzGSZIkSZIkSSUxGSdJkiRJkiSVxGScJHUQb731Fvvvvz/7n7LMPe++9N9/61rdWqVNXV0evXr2oqamhpqaGCy64oMVtJUmSJElrr9QPOETEtsCPgSHAAuAd4MLMnvLHMOKriOQz4DrA58DZwZ2aOaYtYJGltbbrpptx555306NGDJUuWcOCBB3LYYYcxZMiQlep99KMfn5ZZbbnlPbSVJkiRJa6e0mXEREcBNwF2Z2S8zBwOfAXZsYft1mjiMiAHAJcA/Z+ZewH7A061o75donJbUrEUGPHj0AWLJkCUuWLKHyV+/6bStJkiRJarkyl6l+HHgnMyc2FGTmXzNzQkT0jYi7I2JG8fMPnABFRW5TfDDxZlN0UEQ9HxBMRcUpDXxHxhYj4U0RMj4jLI+KSorxPRFwfEQ8WP/9YNPk68L3MfKqInZVlm/lfR5pMR8UBEPBIRdxQz+oiI8yPi5xHxB+DnEbF3cb+ZEfFoROy+3t+iJK3GsmXLqKmpYZttntmHYsGEccMABq9S59957GTRoEIcddhhPPPFEq9pKkiRJktZOZGY5N4o4A/hAZn6tiWubA8sz860inoTU5M/eLiFrgVmBAZj5T1N0qM1+LiG7Ag8BBwKbAvcCHgDeAO4FZmXl6RFwNXJaZ90TEzsBvM3PPniJgBnJiZs5qI533AwszMiDgZ2DMzx0TE+cAngQMzc3FETADuz8xfRsQmQJfMXNyor1OAUwB69+4zn+LyLLl/LNymt2bbdYN7iNdfrqAbu0KutQ2hzixYt4txzz+WMM87gAx/4wIry+vp6NtpoI7p168b9n99/PJZdcwi9+8YsWtW3uPg0z6qT1xXGmsjjWVBbHmsrgOFNZ2sNYGzp06MOZuV+bBtEKbbbUMiIunBQ6ksm/cIcAlEVEDLAM+WFV1ekMirnBGRBxdHO8E7A5sB/xfZr5W9H1tVR+HAHtVLbfqGRFrGiU7nAr+KiO2BTYDq+99clXC7D/hGROwI3JCZcxp3lJmTgEkAO/fbLcc/5upWrX9jBi6lM4+1uSNr2zqEndmHGjBnMnz+fE088scnrtbW1TJw4kQEDBtC7d+9WtW1QV1dHbW3tugpZapLjTGVxrKksjjWVwXGmnsjjWWq/MZapPUJm5BkBmfgk4GOgDfA2YB+xDZe+2Tara1TccFDPlDgE+kpn7AI8Am63hvhsBQzKznpvjZITMXFfEMbqbNBOCSzBwI/Guje6yIJzOvBo4EFgO3RcTH1xCLJK03r7zyCgsXLgRg8eLF3H77n7fTv33+lOi+99BINM6KnT5/O8uXL2XrrrVvUVpIkSZK09spMxt0JbBYRp1aVbV782Qt4MTOXA/8CndGmmj17Agsx8MyL6U/kqKxTLVSPifcWHFY6tavM74MsNJ8XsO4AfAedExAeL8o0iYnTVfZ4vjj/fn3ANFRD/gL5n5n8AUYFBzdSVpfXvxxRcZOnQogwYN4sMf/jDDhg3jiCOOYOLEiUycWNmu87rrrmPAngAHss88+nHHGGVxzzTVERLNtJUmSJEnrVmnr2Ir91z4F/Dgivg68QmWW2VhgBnB9RHwOmErV7LNGnpgKjI+KPwGzg/qLv5yPi+8B04DXgKeD1os0ZwKUR8SiV570LGJ2Zj0bEV4HJxZ51CdxStDkfuDYinFlBJIja3adLxwL9ExBLgJeD7rXwtkrTODBo0iEceeWSV8tGjR684Pv300zn99NNb3FaSJEmStG6VnuqlUZr4IfKaZy9WzysYW9euAuqr2bwOHNdP+6sycVMyMuxG4qWjzKnBCM/HcwrsJuOryKVRmujUunP7/R+ThgXDPxSJIkSZIkSSspc5nq+nZ+RMwEHqfywYWb2jgeSZIkSZIkaSUd5nOLmXlmW8cgSZIknSZIkrU6HScZtCLp17cLscYe3dRjqBOrq6pg7sratw5AkSZIkSY10pGWqkiRJkiRJUrtmMk6SJEmSnJEkqick4SZIkSZIkqSTuGVeixUuW0ffsW9s6DHUCYwYuZVQnGmtz3YtRkiRJkrSBcGacJEmSJEmSnVBKTcZIkSZIkSVJJTMZJkiRJkiRJJTEZJ0kbqLfeeov999+fffbZh7333ptvfetbzdZ98MEH2XjjnjbnuuusAmD17NjU1NSt+evbsyUUXXVRW6JIkSZLUaW3QH3CIiEWZ2aOFdT8F/Ckzn6wq2xh4EfhJnZp69nsKUpPVi00035c4776RHjx4sWbKEAw88kMMOO4whQ4asVG/ZsmWMHTuWQw89dEXZHnvswcyZnM1dc32GHHTj66KNLjV+SJEmSOqPONDPuU8BejcqGAX8CPh0R0VSjiOiyvgOTpPciIujRo/L/I5YsnWcKSJUto6q+yCRMmcOyxx7LNNts02c+0adPYdddd2WWXXdZrvJIkSZKkDpiMi4i+EXFnRDwaEdMinYueI+AfgSOBHETEzInYtqo8ALgb+Bnykqo+5EfHDiJhBJVG3a0RMjYiHI+LuiOhf1PtkRDwQEY9EnxB0RsW3Jjyupk1u2bBk1NTVss802DBs2jAMOOGCl688//zw33ngjp556arN9XHPNNYwYMWJ9hypJnkiRJogMm44AJwM8ycxDwS+A/M/Ne4GbgrMysycw/R8RmwCHAr4HJVBJz1eZn5ocy8xpgEvDlzBwMnnAlcVtS5BxiSmfsC1wBfX98PJ0nVunTpwsyZM3nuueeYPn06jz/++ErXv/rVr/LDH/6QjTZq+q/7nd955h5tvvplPf/rTZYQrSZIkSZ1eZGZbx/CeNbVnXES8CmyfmUsioivwYmb2jogrgVsy87qi3nHAn0Zk5MiK2BmYCfTNzWUTMBQ7KzL9GRA/gFWB21W02zcw9I2IgMB7YHtgEeCYzhzeK5xTgFIDevfsMnPu+iy9f5e5Aa27YbzFvc1lGUZ+AOvdo6hHbhZz/7GZttthknnHDCirIRI0bQ8Pf866+/zmabbcaYnMWM48MADAbjnnnuYMmUKP/rRj97TPRctWrRiqay0vjjOVBbHmsriWFMZHGcqS3sYa0OHDn04M/drn0yBaYYP+gMNaGgEcWCTeALYGPg7cXpzXF39uBCzMzJom+pgA/Edm3hwRtcD5jStk5iQqM+vYud9unOf6xzvzKVZYxA5fSmcba3JG1bR1Cm3jllVfo2rUrW265JYsXL+bcc89l7Nix1NbWrqjz4osvrjgenNWoURxxxBMcdd9yKsokTJ3Laaaet1KY16urq3nNbqaUcZyqLY01lcaypDI4zlcWx1nodcZnqvcBnniuORwN3F8RvAFgAR0RP4KLBzZvbNzL7Al1h1qSqZ+XfgmYj4dNE2ImKf4nIv4Pni+PPr/lEkqXkvnvvgiQ4cOZdCgQXz4wx9m2LBhHHHEEUycOJGJEyeusX19fT233347xxxzTAnRSpIkSZJgw58Zt3lEnPFd1/h/Al4GfRsRZVJaXnlhcuwa4PCLOAG4C7szMt6vaTgEujIhNm7jPSOC/IuKbQNeir1lUZsJdnGxELgDuBD6yzJ5OkNRg0aBCPPPLIKuWjR49usv6VV1650nn37t2ZP3/++ghNkiRJktSMDToZl5nNnzez7eBN1/wDstZq+XgP6FKd9G117BhjeRJspVJJ4kiRJkiRJ0hp1xGWqkiRJkiRJUrtkMk6SJEmSnJEkqick4SZIkSZIkqSQb9J5xG5puXbswe9zhbR2GOoG6ujrmjqxt6zAkSZIkSVIjzoyTJEmSJEmSnSmIyTpIkSZIkSSqJyThJkiRJkiSpJO4ZV6LFS5bR9+xb2zoMdXBz3ZdQkiRJkqR2y5lxkiRJkiRJnUklMxkmSJEmSJEklMRknSZIkSZIklcRknKQO49lnn2Xo0KHstdde7L333lx88cXN1n3wwQfZeOONnue6661aUXXzxxQwYMIC9996biy66qIyQJUmSJEmdTLtOxkXFPRFxWFXZpyNi6jro+xcR8UxEzIyInpyLimy1oc3REnFUcfzcivlocnxQR261tTJLWzsYbb8z48eN58sknuf/++7n00kt58sknV6m3bNkynxo4dy6GHHrqi7PHHH+fyyy9n+vTpzJo1i1tuuYWnn366zPAlSZIkSZ1Au07G5f9v7/6jrK7LBI6/nHwUTzQ7ZgInEmie3g0KimWTrj1m1UkPBpU1ZK1llXTzpqtkp103CtjXFtMhjGYH2w1DKxGz9kT/WnUbN1CRFFRCpzVFAg8SSgqKDP/nG/Y9fbvcMMDN9hmPfrnDlz7+fz+X7uc+955vO9PHx/ZCYwEbgsnIraPiLcDFwKf25R5I6LtLrJnZ+YIYF/gXyLiPRuIZ3ZmXlKn62TAYpzUzXbddVf2228/AHbaaSeGnDh3K0qVL/2rc5ZdfztixYxk4cOCbbYsWLWLkyJHssMMO9OnTh0MPPZQbbrihtNglSZIkSb3DFl2MnA8jMR4FfAl8CJgE/yswnIuKkiJhTHNn2nYjYBiAipkXE3IhYGBGT2uaJiCURcVFEPAQcV/My/YAEnXq4a2794/OGIuLN4PCEi3nLuWkQcD4wAZhWxbLc5PgdJndPa2spDDz3EyJEj39K+dOlSZs+ezWmnnnfaW9mHDhnHfffexcuVKXn75ZW655RaeeeaZMkOWJEmSJPUCfTY8ZItwATAPeA3YPyKGUSmofSQzn10fENOAEYCZwbma+UBz9dndEXJ+ZbeeprcjMfQEiYjTwzYiYDOwJXJqZKzsbWGbOiogzgNMzc35tnf0ScCpwK0NQ0gEnD13f2JaROaWlpYc2aNbS0tHR3KN1m7dq1nHnmmUyYMIF58+a9pW/y5Mkcf/zxn3HvvvSxbtoyFCxfS1NQEwOjRoznwwAPp168fu+++O88991yv/hw7orfnmsphnqks5prKYq6pDOaZnymKudV6PKMZl5ksRMQtYk5mvRsQRwIeAuREBlSPb2g5hGRcRp1B5b4OAvYC2YtysmqnPzswbI2InnKoW7/87MOV0c+zRgGsCQPd6Xly7oER+5erDWE5tpaWmhubm5u0PpFuvWrWPUqFFMnDiRz3/+83/Vn/9RTTzFlyhQAnn/+eebNm8c+++zDmDFjaG5u5pJLKmein3feeQwePLjXfo4d1ZtzTeUxz1QWc01lnMddUBvNMZTHXOq8nVYbeKH4AArgqM8+vHhARewJnAgdk5p8j4hpg+6ohL9WbODNXR8Q9wEHAHGA9nfzmFd/t620ja8mQmp5xyCkOHDq1biAN48skn33w8fvx4Ro0axZgxYwBYsWIFAwcO5Omnn+aGG27gngQceKCVuSZIkSVLv0ZOKcdXuBK6PiKmZ+XxEvAvYEXgHsBpYFRG7Ah8HNnjn1YjoCxwAfKNoagU+nCNwBjO1APKuBnTr7JiR1rfvvv58f//jHDB8+nBEjRgBw4YUX8vTTTwMwceLEdrcfO3YsK1eupG/fnvlxxxRX0799/s8csSZIkSepdemQxLjMXRMQFwJ3FjRvWUbnr6lwqp6Q+DjwF3L+BqdquGfc24FfAnTUX7ZOD7EfFn4N4OhHQ1MD0i1lI5Ku+1zr0jSV3hoIMOonIT5o75wQ9+8Jbn9913XxdHJEmSJEnSnW/WYYlxmTq55PpPKDRtqfabB9oNrnn+6nddqoXJTh9r26VWPv1z1+KfATxvNJ0mSJEmSJMFfrosmnSZIkSZIkaTOzGCdJkiRJkiSVpMecpro16Nd3WxZf9InuDkOSJEmSJEndxCPjJEmSJEmSpJJYjJMknSZIkSZJKYjFOkiRJkiRJKonXjCvR2nWvs/u5N3d3GNpErV73T5IkSZIkbSSPjJMkSZIkSZJKYjFOnkiRJkiRJKonFOEmSJEmSJKkkFuMkddjJJ5/MwIEDGTZsWN3+F198kWOOOYZ99tmHvffem6uvvvotn/a+//jr77rsvo0aNKiNcSZIkSZK2OO0W46Li1xFxVFXbP0bEbZv6whFxTUQ8GRHzI+LhiPj7TZ2znk6//tYg4q+r5dhHxQkR8rZ1tjoiIGxv0LYmI/psjVmlLMX78eG67rfGf/xVXXMFee+3Fww8/TEtLnC+eccw6vvfbam/1Tp05l6NChZYQqSZIkSdIWqd1iXGYmMBG4LCK2j4i3AxcCn9uUF42Itru4np2ZnI4AvAN/ZlDm7wMeBx4DjuzkOaYt1yCGHsPPOOzfsjwhWr15NZrJmzRp23nln+vSp/LkvWbKEm2++nmQkTJpQVriRJkiRJW5wNnqaamY8CvwS+BEwCfpSZT0TESRExpziy7TsRsQ1AREyLiLkRsTAiJrXNnUxw5dlFEPAQcV/My/wvsVjX2QxFxT0Q8GBG3RsQuRfuvI+KyYv7HImL/iJgdEb+PiMlV238xIh4tnfs6oap8UEb+LiF8De9bEMA64DFgWEQdUbfOJiFgcEfOA0VXtAyLijuJ9fg+IDX2W0tbu9NNPZ9GinRQwaNIjhw4czdepUttmmssycddZZTJky5c3nkiRJkiT1Rn02PASAC4B5wGvA/hExjEpB7SOZuT4inpgEnADOBczPzheLot7sj4vrMfKyYZ0Vm7gsQEaOr5j8SuLFofxswFTg2M5+PiBOB/wROLcauzcz9nI+KcYpsPAi8Cf4yIbwHvB04EPlS8vzkR0QL0A8YC+wDbAfOpFAGJiB2AZuBk4N1UCnNzivbvAYcCnfwSur/lM7s7MC4v3cip1RMSpbX1NTQOYNHz9Bj9sbdlaWlq6O4QNWrNmzWaLc9myZbz00kt157/nnnntoampi5syZPPvss0yYMIHp06fzyCOPsG7dOlavXs38+fNZuXJlj/gctWGbM9ekNuaZymKuqSzmnmspgnqks5lrndagYl5kvRcQsYE1mvhoRR1Apds2NCKgUup4pho+LiFOKuQcBe1E5/RNgVs3U34yInKbMuorEAAAkgSURBVFSOihtZtA0F9gbuLObeFlhStc1Nxe8FwILMXA4QEa3AYOAg4OeZubZovxE4nGNihqn1tRPyyas5jgTsy85WI+BnwYFHs2wv4XWY+Ucz1E+CzxTaHAEcXn88vImJ1g89uGjANYMgen78tLF3S0/qktVeuJzd0dwga1tLTQ3Ny8WeZubW1lxx13rDv/JZdcwrnnnsvBBx8MwIwZMxgwYACrnVq3iwQcfZPz48bzyyiusWrWK6dOnc80112yWGFWezZlrUhvzTGUx11QWc01lMM9UFnOt8zpzvtgbnxQ9UTsm8KjNHFD/vz8z/jIg9gTOBwzLzA8BtwPZVc7xUM+fZmfm3wJeBGVVzP1I19/DMPKpqm1ern4nm1qv0NOn6kX61xwJFFQe+3wAAqR8NJ6oQhQ4Zw1113AbB8+XIWL17MHnvswde//nWWLFlCa2srn1113HYcddpiFOEmSJElSr7SxF2+6E/hURDQBRMS7ImII8A5gNbAqInalclOEjvgWsENEHE7lKLrdn2q7bVtzldO9OxHYfcFxE9CtuODG6aLu3aN8+It4BjCrm7w98GBicmbtn5u7Av1Ep0D0G7BkR743KnYXrjql7nXuCfijmOAXbqRIxSjzRu3DgOPPBAFi9ezODBg5kxYwZXXnklV155JQDnn38+v/nNbxg+nfDiHH344F198MU1NTd0ctSRJkiRJW46NOpIsMxdExAVUTiXdBlhH5a6rc6kUsB4HngLu7+B8GRFfnA76YmXdFxCeBbxdFs22BS4GFHZxrTkRcS+UIN4DvZuYCgIiYDTwCLAfmFP1jqZyiuq5qmhuB/6Jyn19iJwK1Ujuq7HxhSjPkKcG1EfLpof7Yj8Uk92bXXXttu/6BBg7j99tvbHdPc3OwhzJIkSZKkXqvDnxbjMnFzzfCaVGzbU+kyD7QfXPP90zfNZFNeUy8x5VK79VjvHQVWP76RyhF69vinAlDrbfxX4ap3wnZtSM+xMwsHh6c/FTO9efgCPqzCVJkiRJkiTVtbGnqUqSJEmSJEnqJItxkiRJkiRJUkk29u6j2gj9n+m7L4os+0d1hSJIkSZIkqZt4ZJwkSZIkSZJUEotxkiRJkiRJUkksxkmSJEmSJEklsRgnSZIkSZIknlcRinCRJkiRJklQSi3GSJEmSJElSSSzGSZIkSZIkSSWxGCdJkiRJkiSVxGKcJEmSJEmSVJLIzO6OnodeIiNXA4u6OQ71CE/B8dwehXsFcUxnMM5XFXFNZzDWVwTxTWbaEXPubzBzQzTF0WJ/uDqCXWZyZn+3d3ENr6RcRcc01lMNdUBvNMZTHXVBZzTWUwz1QWc63zPE1VkiRJkiRJKonFOEmSJEmSJKkkFuPKnNa27A1CvYa6pLOaaymCeqSzmmspirqkM5pnKYq51kjdwkCRJkiRJkkrikXGSJEmSJElSSSzGSZIknSZIkSSWxGLcZRMSREbE4Iv4QEefW6Y+I+HbR/0hE7Ncdcapni4j3RMTdEfFYRCyMiDPrjGmOiBcjnYn7xM6k7YlXPFhGtEbGgyKG5dfpd07TJIuL9VWvV/IhYFRFn1YxxTdNGiYirImJFRDxa1bZzRNwRnEb8vfr+zwbbtfq+T2jTIs0si4vFi/zg7Ivo32Lbdfa1UrUGuTY6IpVX7yKMbbOuapg5rkGuzqvKsnNSLmN9jWda0dXjOui0XEtsDvgI8CS4DfAuMy87GqMUcDZwBHAyOBqZk5shvCVQ8WEbsCu2bmvIjYnCXgQGFOTa83AFzJzVDeFqa1ARLQC+2fm8w36XdPUpYp96VJgZGY+VdXejGuaNkJEHAKsAX6UmcOKntinAC5l5UfEP0ndm5pdqttvg9zqpTYM8+xjwP5m5PiIuBqjNs2JcK+3sa6VqDXJtMrAmM7/RznaunaeqUerlW038p8GJmfrVOXyuuaw15ZFzXOwD4Q2b+MTNfA64DRteMGU0lmTMzHwD6F4UVqcMy87nMnnFc8Xg0sAnbr3qjUS7mmqasdDjxRXYiTNkVm3gu8UNM8Gvhh8fiHwJg6m3bke50E1M+zzLw9M9cXnTx8ABpcemLY6Dda0jnBNU6e0l2sREcCngGtLDWorYTGu6+0GPFP1fAl/XSDpyBipwyJid2Bf4P/qndH+kODXi1ojYu9TAtLVI4M6IeDAiTq3T75qmrnYCjb/Yuaapq+ySmc8Vj5cBu9QZ4/qmrnQycGuDnvg3ta6WOOKPYR17V4NR71zR1pYOB5Zn5+wb9rmvtsBgn9XAR8Xbg58BZmbmqpnseMCQzPwBcDtxYndnzaKhyUmSOAo4DPFYerS5tFRGwHHAv8rE63a5o2i6xct8Vrt2iziYj/ANYDP2kwxH2tNtV3gT2AnEcBzwKXdG456gXG0f1Sc61o7LMZ1vaXAe6qeDy7aOjtG2qCI6EulEPeTzLyhtj8zV2XmmuLxLUDfniGgqOUz1cJm5tPi9AphN5RSHaq5p6kpHAfMyc3lth2uautjytlPqi98r6oxxfdMmi4jxwCjgxGxwnwe4O7GuldmXm8sx8PTPfAL5P/RxyTVOXiIg+wD8AsxqNcV1rn8W4rvdbYM+IeG/xv/snADfVjLkJn+GxxB8IPU7ng4XO1E0ntKc7RnwEsyszLGox5dzGOiDiAyt/8yvKiVE8XETsWNwghInYEPgY8WjPMnNU1dqeH/srqmqYvdBJxUPD4J+EWdMR35Xic1FBFHAl8Ejs3MlxuM6ci+VmpXzfV6j6N+DrmmqascnATyemUvqdbqubVif7g5ga1PcKel04FfAtsBVmbkwIiYW/VcCt1C56+AfgJeBf+6ueNWj/R3wGWBBn1e2kzwOGwJu59kngtIhYD6wFTmj0P7JSA7sAs4v6Rx9gZmbe5pqmzaH4svZR4F+r2qpzzTVNGyUinrgWagaaIWAJ8BbgI+GlEnAI8ReUi1ETEIGB6Zh7d6Htdd7wHbfka5Nm/A28D7ij2pQ9k5sTqPKPBnvrYb3oJ6iAa51hwRI6icct9KsS91TdOmqJdrmTmDOtf3dV3rnPA7rCRJkiRJklQOT1OVJEmSJEmSnSmIxTpIkSZIkSSqJxThJkiRJkiSpJBbjJEmSJEmSpJJYjJMkSZIkSZJKYjFOkiRJkiRJKonFOEmSnJEmSJKkk/w+vjaRyl/GfrgAAAABJRU5ErkJggg==n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f93eeefdfd0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "n",
      "> Feature importances dumped into directory : saven",
      "n",
      "predicting...n",
      "CPU time: 0.09216570854187012 secondsn",
      "n",
      "> Overview on predictions : n",
      "n",
      "   SalePrice_predictedn",
      "0        165921.195890n",
      "1        167701.648928n",
      "2        175319.175272n",
      "3        177518.368928n",
      "4        192895.614810n",
      "5        176685.509650n",
      "6        170356.903631n",
      "7        174661.891858n",
      "8        181593.424964n",
      "9        165921.195890n",
      "n",
      "dumping predictions into directory : save ...n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<mlbox.prediction.predictor.Predictor at 0x7f9430dd2780>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prd = Predictor()n",
    "prd.fit_predict(params, df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "242102b7-174a-4c97-9eb0-9875b085ea2b",
    "_uuid": "caeee1581dda9444d9d519051a77fd1e73b41f2d"
   },
   "source": [
    "### Formatting for submissionn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "_cell_guid": "7668891c-cabc-45d3-bbd6-a0465a037eff",
    "_uuid": "1706dac719a20b006d903db219065ec78ebeccb6",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "submit = pd.read_csv("../input/sample_submission.csv",sep=',')n",
    "preds = pd.read_csv("save/"+target_name+"_predictions.csv")n",
    "n",
    "submit[target_name] =  preds[target_name+"_predicted"].valuesn",
    "n",
    "submit.to_csv("mlbox.csv", index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "2c0c568e-2de8-4a09-a985-1548bfa4fb7f",
    "_uuid": "a2dcc9ee8edc83f6b90eb751a71db5554a0f098e"
   },
   "source": [
    "# **That's all !!**"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}

2. Auto-Sklearn

Auto-Sklearn is an automated machine learning package built on top of Scikit-learn. Auto-sklearn frees a machine learning user from algorithm selection and hyperparameter tuning. It includes feature engineering methods such as one-hot encoding, numeric feature standardization, PCA, and more. The models use sklearn estimators for classification and regression problems.

Auto-sklearn creates a pipeline and optimizes it using Bayesian search. Two components are added to Bayesian hyperparameter optimization of an ML framework: meta-learning for initializing the Bayesian optimizer and automated ensemble construction from configurations evaluated during optimization.

Auto-sklearn performs well on small and medium-sized datasets, but it cannot be applied to modern deep learning systems that yield state-of-the-art performance on large datasets.

Installation

Auto-sklearn currently only works on Linux machines.

Demo

The following example shows how to fit a simple regression model with Auto-Sklearn.


import sklearn.model_selection
import sklearn.datasets
import sklearn.metrics

import autosklearn.regression

def main():
    X, y = sklearn.datasets.load_boston(return_X_y=True)
    feature_types = (['numerical'] * 3) + ['categorical'] + (['numerical'] * 9)
    X_train, X_test, y_train, y_test = 
        sklearn.model_selection.train_test_split(X, y, random_state=1)

    automl = autosklearn.regression.AutoSklearnRegressor(
        time_left_for_this_task=120,
        per_run_time_limit=30,
        tmp_folder='/tmp/autosklearn_regression_example_tmp',
        output_folder='/tmp/autosklearn_regression_example_out',
    )
    automl.fit(X_train, y_train, dataset_name='boston',
               feat_type=feature_types)

    print(automl.show_models())
    predictions = automl.predict(X_test)
    print("R2 score:", sklearn.metrics.r2_score(y_test, predictions))


3. Tree-Based Pipeline Optimization Tool (TPOT)

TPOT is a Python automated machine learning tool that optimizes machine learning pipelines using genetic programming. TPOT extends the Scikit-learn framework but with its own regressor and classifier methods. TPOT works by exploring thousands of possible pipelines and finding the best one for your data.

TPOT cannot automatically process natural language inputs. Additionally, it’s also not able to processes categorical strings, which must be integer-encoded before being passed in as data.

Installation

For detailed instructions, please visit the TPOT installation instructions in the documentation.

Demo

Demonstrating the working of TPOT for classifying MNIST digits.

4. H2O

H2O is a fully open source, distributed in-memory machine learning platform from the company H2O.ai. With support for both R and Python, H2O supports the most widely used statistical & machine learning algorithms, including gradient boosted machines, generalized linear models, deep learning models, and more.

H2O includes an automatic machine learning module that uses its own algorithms to build a pipeline. It performs an exhaustive search over its feature engineering methods and model hyperparameters to optimize its pipelines

H2O automates some of the most difficult data science and machine learning workflows, such as feature engineering, model validation, model tuning, model selection and model deployment. In addition to this, it also offers automatic visualizations and machine learning interpretability (MLI).

Installation

Follow the link below to download and install H2O on your systems.

Demo

Here’s an example showing the basic usage of the H2OAutoML class in Python:

import h2o
from h2o.automl import H2OAutoML

h2o.init()

# Import a sample binary outcome train/test set into H2O
train = h2o.import_file("https://s3.amazonaws.com/erin-data/higgs/higgs_train_10k.csv")
test = h2o.import_file("https://s3.amazonaws.com/erin-data/higgs/higgs_test_5k.csv")

# Identify predictors and response
x = train.columns
y = "response"
x.remove(y)

# For binary classification, response should be a factor
train[y] = train[y].asfactor()
test[y] = test[y].asfactor()

# Run AutoML for 30 seconds
aml = H2OAutoML(max_runtime_secs = 30)
aml.train(x = x, y = y,
          training_frame = train,
          leaderboard_frame = test)

# View the AutoML Leaderboard
lb = aml.leaderboard
lb

# model_id                                            auc       logloss
# --------------------------------------------------  --------  ---------
#           StackedEnsemble_model_1494643945817_1709  0.780384  0.561501
# GBM_grid__95ebce3d26cd9d3997a3149454984550_model_0  0.764791  0.664823
# GBM_grid__95ebce3d26cd9d3997a3149454984550_model_2  0.758109  0.593887
#                          DRF_model_1494643945817_3  0.736786  0.614430
#                        XRT_model_1494643945817_461  0.735946  0.602142
# GBM_grid__95ebce3d26cd9d3997a3149454984550_model_3  0.729492  0.667036
# GBM_grid__95ebce3d26cd9d3997a3149454984550_model_1  0.727456  0.675624
# GLM_grid__95ebce3d26cd9d3997a3149454984550_model_1  0.685216  0.635137
# GLM_grid__95ebce3d26cd9d3997a3149454984550_model_0  0.685216  0.635137


# The leader model is stored here
aml.leader


# If you need to generate predictions on a test set, you can make
# predictions directly on the `"H2OAutoML"` object, or on the leader
# model object directly

preds = aml.predict(test)

# or:
preds = aml.leader.predict(test)

Output

The AutoML object includes a “leaderboard” of models that were trained in the process, ranked by a default metric based on the problem type (the second column of the leaderboard). Here’s an example leaderboard for a binary classification task:

5. AutoKeras

Auto-Keras is an open source software library built by DATA Lab for automated machine learning. Auto-Keras, which is based on the Keras deep learning framework provides functions to automatically search for architecture and hyperparameters for deep learning models.

The API’s design follows the classic design of the Scikit-Learn API; hence, it’s extremely simple to use. The current version provides functionalities to automatically search for hyperparameters during the deep learning process.

Auto-Keras tends to simplify the ML process through the use of automated Neural Architecture Search (NAS) algorithms. Neural Architecture Search essentially replaces the deep learning engineer/practitioner with a set of algorithms that automatically tunes the model.

Installation

The installation part is also very simple:

Demo

Here’s a demo of Auto-Keras library on the MNIST dataset:

from keras.datasets import mnist
from autokeras import ImageClassifier
from autokeras.constant import Constant

if __name__ == '__main__':
    (x_train, y_train), (x_test, y_test) = mnist.load_data()
    x_train = x_train.reshape(x_train.shape + (1,))
    x_test = x_test.reshape(x_test.shape + (1,))
    clf = ImageClassifier(verbose=True, augment=False)
    clf.fit(x_train, y_train, time_limit=30 * 60)
    clf.final_fit(x_train, y_train, x_test, y_test, retrain=True)
    y = clf.evaluate(x_test, y_test)

    print(y * 100)

6. Cloud AutoML

Cloud AutoML is a suite of machine learning products from Google that enables developers with limited machine learning expertise to train high-quality models specific to their business needs by leveraging Google’s state-of-the-art transfer learning and Neural Architecture Search technology.

Cloud AutoML provides a simple graphical user interface (GUI) to train, evaluate, improve, and deploy models based on your own data. Currently, the suite provides the following AutoML solutions:

The downside of Google’s AutoML is that isn’t open source and hence comes with a price. In the case of AutoML Vision, the cost depends both on the time taken to train the model as well as in terms of how many images you send to AutoML Vision for predictions. The pricing is as follows:

7. TransmogrifAI

TransmogrifAI is an open source automated machine learning library from Salesforce. The company’s flagship ML platform called Einstein is also powered by TransmogrifAI. It is an end-to-end AutoML library for structured data written in Scala that runs on top of Apache Spark. TransmogrifAI is especially useful when you need to :

  • Rapidly train good quality machine learnt models with minimal hand tuning
  • Build modular, reusable, strongly-typed machine learning workflows

Installation

There are some pre-requisites like Java and Spark that need to be installed.

Read the documentation for complete installation instructions.

Demo

Predicting Titanic Survivors with TransmogrifAI. See the entire example here.

import com.salesforce.op._
import com.salesforce.op.readers._
import com.salesforce.op.features._
import com.salesforce.op.features.types._
import com.salesforce.op.stages.impl.classification._
import org.apache.spark.SparkConf
import org.apache.spark.sql.SparkSession

implicit val spark = SparkSession.builder.config(new SparkConf()).getOrCreate()
import spark.implicits._

// Read Titanic data as a DataFrame
val passengersData = DataReaders.Simple.csvCase[Passenger](path = pathToData).readDataset().toDF()

// Extract response and predictor features
val (survived, predictors) = FeatureBuilder.fromDataFrame[RealNN](passengersData, response = "survived")

// Automated feature engineering
val featureVector = predictors.transmogrify()

// Automated feature validation and selection
val checkedFeatures = survived.sanityCheck(featureVector, removeBadFeatures = true)

// Automated model selection
val (pred, raw, prob) = BinaryClassificationModelSelector().setInput(survived, checkedFeatures).getOutput()

// Setting up a TransmogrifAI workflow and training the model
val model = new OpWorkflow().setInputDataset(passengersData).setResultFeatures(pred).train()

println("Model summary:n" + model.summaryPretty())

Check out the results of the above code here.

The Future of AutoML

Essentially, the purpose of AutoML is to automate the repetitive tasks like pipeline creation and hyperparameter tuning so that data scientists can actually spend more of their time on the business problem at hand.

AutoML also aims to make the technology available to everybody rather than a select few. AutoML and data scientists can work in conjunction to accelerate the ML process so that the real effectiveness of machine learning can be utilized.

Whether or not AutoML becomes a success depends mainly on its adoption and the advancements that are made in this sector. However, it’s clear that AutoML is a big part of the future of machine learning.

References

  1. Efficient and Robust Automated Machine Learning
  2. Benchmarking Automatic Machine Learning Frameworks
Avatar photo

Fritz

Our team has been at the forefront of Artificial Intelligence and Machine Learning research for more than 15 years and we're using our collective intelligence to help others learn, understand and grow using these new technologies in ethical and sustainable ways.

Comments 0 Responses

Leave a Reply

Your email address will not be published. Required fields are marked *

wix banner square