{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Anomaly Detection\n",
    "\n",
    "Anomaly detection describes the process of finding outliers, for some definition of what an outler is. Since the definition of outlier is quite context dependent, there are many anomaly deteciton methods. Anomaly deteciton is generally used in an unsupervised fashion, although we use labeled data for evaluatoin. \n",
    "\n",
    "This lecture does not focus on specific mehtods of anomaly detection. Instead, we focus on the general principles and pragmatic coding examples uisng `scikit-learn`. Topics covered:\n",
    "\n",
    "- Use of PCA for anomaly detection\n",
    "- Use of validation and test data sets\n",
    "- Model evaluation and scoring\n",
    "- ROC curve\n",
    "- Precision recall curve\n",
    "- Overfitting\n",
    "- Example of embarassingly parallel code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.io import loadmat\n",
    "from scipy.io.arff import loadarff"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import OrderedDict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler, MinMaxScaler\n",
    "from sklearn.decomposition import PCA\n",
    "from sklearn.model_selection import StratifiedKFold, train_test_split, GridSearchCV\n",
    "from sklearn.metrics import precision_recall_curve, roc_curve, average_precision_score, auc, make_scorer\n",
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.externals.joblib import Parallel, delayed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "import seaborn as sns\n",
    "sns.set(font_scale=1.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Information about data\n",
    "\n",
    "The [Cardiotocogrpahy dataset](http://odds.cs.stonybrook.edu/cardiotocogrpahy-dataset/) has the following description (from the linked URL):\n",
    "\n",
    "```\n",
    "The original Cardiotocography (Cardio) dataset from UCI machine learning repository consists of measurements of fetal heart rate (FHR) and uterine contraction (UC) features on cardiotocograms classified by expert obstetricians. This is a classification dataset, where the classes are normal, suspect, and pathologic. For outlier detection, The normal class formed the inliers, while the pathologic (outlier) class is downsampled to 176 points. The suspect class is discarded.\n",
    "```\n",
    "\n",
    "We will try to detect outliers wihtout using any lables."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load data and inspect"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = loadmat('data/cardio.mat')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['__header__', '__version__', '__globals__', 'X', 'y'])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = data['X']\n",
    "y = data['y']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((1831, 21), (1831, 1))"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.shape, y.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = y.squeeze()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Scale data to have zero mean and unit standard deviation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "scaler = StandardScaler()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = scaler.fit_transform(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Split into test and trainign data sets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, random_state=20198)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define score function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def score(a, b):\n",
    "    \"\"\"Normalized differnece of squares between a and b.\"\"\"\n",
    "    \n",
    "    loss = np.sum((a-b)**2, axis=1)\n",
    "    loss = (loss-loss.min())/(loss.max() - loss.min())\n",
    "    return loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Concept of using PCA for outlier detection\n",
    "\n",
    "We find the robust features of the data set using PCA, expecting these to be mostly contributed by the more abundant inliers. In other words, we expect the inliers to be well summarized by the reduced dimensionality PCA data set, and the outliers to be less well summarized. Now we can reconstruct the original $n$-dimensional space from the $k$-dimensional reduced space, and compare the reconstructed data vectors with the original ones. If the reconstruction is good, we expect the distance between original and reconstructed vectors to be small, and big otherwise. Hence the normalized distance between original and reconstructed vectors might give some clue as to which data points are poorly captured by the reduced dimension PCA representation, and hence possible outliers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fit(n, X):\n",
    "    pca = PCA(n_components=n)\n",
    "    X_pca = pca.fit_transform(X)\n",
    "    X_pca_inverse = pca.inverse_transform(X_pca)\n",
    "    y_score = score(X, X_pca_inverse)\n",
    "    return (n, y_score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Find opitmal parameter (n) by grid search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "search = Parallel(n_jobs=-1)(delayed(fit)(n, X_train) for n in range(2,X.shape[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "scores =  np.array([(n, average_precision_score(y_train, y_score)) for (n, y_score) in search])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([2.        , 0.43654539])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best = max(scores, key=lambda x: x[1])\n",
    "best"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualize scores "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAD9CAYAAABTJWtQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XtwU1UCP/Bv+k43D1ooCpQCobYL\n9EVtKU4L1UphsCps3R2ruLSDE6i7BrdlfDBWnWUVxBG1EloQWKp1ARGYnUUL7uKjXRyVYda6v5+u\nxTZgW3/VFmiTBtI0Jvn9gb0YE3r7SJvX9zPjjLn3Jvfc05Dvveece67EbrfbQUREBCDI0wUgIiLv\nwVAgIiIBQ4GIiAQMBSIiEjAUiIhIEOLpAoxWV1evp4vgFaKiItHdfcXTxfAbrE/3Y52612jqMyZG\nft11vFLwEyEhwZ4ugl9hfbof69S9xqo+GQpERCRgKBARkYChQEREAoYCEREJGApERCQYUig0Nzej\nuLgYqampyMnJQWVlJaxW65B3YrPZUFhYiMTERHz44YcO65544gkkJiY6/dfS0jK8IyEiolETvU9B\nr9ejpKQE8fHxqKqqQmtrK7Zu3QqbzYaysrIh7eTtt9/GDz/8cN31KpUKW7ZscVgWGxs7pM8mIiL3\nEQ2FgwcPwmw2Q6vVQiaTITs7G0ajEVqtFmq1GjKZbND36/V6vPzyy9iwYQMqKipcbiOVSpGWljay\nIyAiIrcRbT5qaGhATk6Ow49/QUEB+vr6cPr0adEdVFZWIj09HbfccsvoSkpERGNONBR0Oh1UKpXD\nsqlTp0IqlUKn0w363q+//hpHjhzB448/Puh2LS0tSE9PR1JSEu67774hhQ0REbmfaPORwWCAXO48\nT4ZCoYDBYBj0vc8++yxWrVqFGTNmoL293eU2c+bMQUpKCuLj43Hp0iXs27cPa9aswf79+5GSkjLE\nwyAiIncYswnx3n33XZw7dw47d+4cdLvi4mKH17m5uSgoKMDOnTtRVVUlup+oqEjOqfKTwSa5ouFj\nfbof69S9xqI+RUNBoVDAaDQ6LTcYDFAoFC7fY7FY8MILL0CtVsNms8FgMAifYTKZYDQar9tBLZVK\nkZub6zR09Xo46+JVMTFyzhjrRqxP92Odutdo6nOwMBENBZVK5dR30NHRAZPJ5NTXMMBkMuH777/H\nli1bnIaalpWVIS4uDv/617+uu0+JRAKJRCJWNCIicjPRUFi8eDH27t3rcHZfV1eHiIgILFiwwOV7\nIiMj8cYbbzgsu3DhAsrLy1FeXo6FCxded399fX346KOPMG/evOEcBxERuYFoKBQVFaG2thYajQZq\ntRptbW3QarUoKSlxaALKz89HZmYmNm/ejJCQEGRlZTl8zkBHc0JCAlJTUwEAvb29WLduHe6++27M\nmDED3d3dqKmpQWdnJyorK915nERENASioaBUKlFTU4NNmzahtLQUCoUCxcXF0Gg0DttZrVbYbLZh\n7TwsLAzR0dGorq7GxYsXER4ejrS0NLz55ptITk4e3pEQEdGoSex2u93ThRgNdlxdxU4892J9uh/r\n1L3GqqOZs6QSEZEgYEPBbLGis/sKzJahz/ZKROTvxuzmNW9ltdnw1gfN+PxsFy4ZzIhWhGN+Qgzu\nzYtHcFDAZiQREYAADIW3PmjGyTPXpty4aDALr+9fkuCpYhEReYWAOjU2W6z4/GyXy3Wfn73ApiQi\nCngBFQp6oxmXDGaX67p7+6A3ul5HRBQoAioUlLJwRCvCXa6LkkdAKXO9jogoUARUKISHBmN+QozL\ndfMTJiE8lLOtElFgC7iO5nvz4gFc7UPo7u1DlDwC8xMmCcuJiAJZwIVCcFAQ7l+SgHtyZ0NvNEMp\nC+cVAhHRTwIuFAaEhwZjclSkp4tBRORVAqpPgYiIBsdQICIiAUOBiIgEDAUiIhIwFIiISMBQICIi\nAUOBiIgEDAUiIhIwFIiISMBQICIiAUOBiIgEDAUiIhIwFIiISMBQICIiAUOBiIgEDAUiIhIwFIiI\nSMBQICIiAUOBiIgEDAUiIhIwFIiISDCkUGhubkZxcTFSU1ORk5ODyspKWK3WIe/EZrOhsLAQiYmJ\n+PDDD53Wnzx5EnfddReSk5Nxxx13oK6ubuhHQEREbiMaCnq9HiUlJZBIJKiqqsIf//hH7Nu3D6++\n+uqQd/L222/jhx9+cLnuzJkzWL9+PbKysrB7927k5uaivLwcp06dGvpREBGRW4SIbXDw4EGYzWZo\ntVrIZDJkZ2fDaDRCq9VCrVZDJpMN+n69Xo+XX34ZGzZsQEVFhdP66upqZGRkCOsWLlyI5uZm7Nix\nAzk5OSM8LCIiGgnRK4WGhgbk5OQ4/PgXFBSgr68Pp0+fFt1BZWUl0tPTccsttzit6+/vx2effYbl\ny5c7LC8oKEBjYyN6e3uHcgxEROQmoqGg0+mgUqkclk2dOhVSqRQ6nW7Q93799dc4cuQIHn/8cZfr\nW1tbYbFYnD5fpVLBZrPh3LlzYsUjIiI3Em0+MhgMkMvlTssVCgUMBsOg73322WexatUqzJgxA+3t\n7U7r9Xq98Fk/p1QqhX2LiYqKREhIsOh2gSAmxvnvRCPH+nQ/1ql7jUV9iobCSL377rs4d+4cdu7c\nOVa7AAB0d18Z08/3FTExcnR1sbnNXVif7sc6da/R1OdgYSLafKRQKGA0Gp2WGwwGpzP8ARaLBS+8\n8ALUajVsNhsMBoPwGSaTSfj/gSuCX/YdXO8KgoiIxpZoKKhUKqe+g46ODphMJqe+gAEmkwnff/89\ntmzZgszMTGRmZmLFihUAgLKyMvzmN78BAMTFxSE0NNTp83U6HYKCgjBr1qwRHRQREY2MaPPR4sWL\nsXfvXhiNRmEEUl1dHSIiIrBgwQKX74mMjMQbb7zhsOzChQsoLy9HeXk5Fi5cCAAICwtDVlYWTpw4\ngaKiImHb48ePIy0tzWVfBhERjR3RUCgqKkJtbS00Gg3UajXa2tqg1WpRUlLiMEw1Pz8fmZmZ2Lx5\nM0JCQpCVleXwOQMdzQkJCUhNTRWWP/TQQ1i9ejWee+45LFmyBPX19aivr8eePXvcdYxERDREos1H\nSqUSNTU1sFqtKC0txfbt21FcXIz169c7bGe1WmGz2YZdgIyMDLz66qv45JNP8OCDD+KDDz7Atm3b\neOMaEZEHSOx2u93ThRgNjma4iiM73Iv16X6sU/fy2OgjIiIKHAwFIiISMBSIiEjAUCAiIgFDgYiI\nBAwFIiISMBSIiEjAUCAiIgFDgYiIBAwFIiISMBSIiEjAUCAiIgFDgYiIBAwFIiISMBSIiEjAUBgF\ns8WKzu4rMFusni4KEZFbiD6Ok5xZbTa89UEzPj/bhUsGM6IV4ZifEIN78+IRHMScJSLfxVAYgbc+\naMbJM+3C64sGs/D6/iUJnioWEdGo8bR2mMwWKz4/2+Vy3ednL7ApiYh8GkNhmPRGMy4ZzC7Xdff2\nQW90vY6IyBcwFIZJKQtHtCLc5booeQSUMtfriIh8AUNhmMJDgzE/IcbluvkJkxAeGjzOJSIich92\nNI/AvXnxAK72IXT39iFKHoH5CZOE5UREvoqhMALBQUG4f0kC7smdDb3RDKUsnFcIROQXGAqjEB4a\njMlRkZ4uBhGR27BPgYiIBAwFIiISMBSIiEjAUCAiIgFDgYiIBAwFIiISMBSIiEgwpFBobm5GcXEx\nUlNTkZOTg8rKSlitg88G+s033+DBBx9ETk4OkpKScOutt+LJJ59EZ2enw3ZPPPEEEhMTnf5raWkZ\n+VEREdGIiN68ptfrUVJSgvj4eFRVVaG1tRVbt26FzWZDWVnZdd/X29uL2NhYrFy5EpMnT0Z7ezt2\n7NiBL7/8EocPH0ZIyLVdq1QqbNmyxeH9sbGxozgsIiIaCdFQOHjwIMxmM7RaLWQyGbKzs2E0GqHV\naqFWqyGTyVy+Lz09Henp6cLrrKws3HjjjVizZg2ampowb948YZ1UKkVaWpobDoc8xWyxcsoPIj8g\nGgoNDQ3Iyclx+PEvKCjAiy++iNOnTyMvL2/IO5swYQIAwGKxjKCo5I34aFIi/yL6r1an00GlUjks\nmzp1KqRSKXQ6negObDYb+vv7odPpsG3bNiQnJyMlJcVhm5aWFqSnpyMpKQn33XcfTp8+PczDIE8Z\neDTpRYMZdlx7NOlbHzR7umhENAKiVwoGgwFyudxpuUKhgMFgEN2BWq3GqVOnAADz5s3D7t27EfSz\nM8g5c+YgJSUF8fHxuHTpEvbt24c1a9Zg//79TuHhSlRUJEJC2FwBADExzn+nsdTX/yP+23LR5br/\ntlzEunukiAjz3TkXx7s+AwHr1L3Goj7H/F/sU089Bb1ej/Pnz6O6uhpqtRoHDhxAePjVJ5QVFxc7\nbJ+bm4uCggLs3LkTVVVVop/f3X1lTMrta2Ji5Ojq6h3XfXZ2X0FXt8nlugs9JrScv+izs8h6oj79\nHevUvUZTn4OFiWjzkUKhgNFodFpuMBigUChEdz5z5kykpqZixYoV2Lt3L7766iscO3bsuttLpVLk\n5ubiq6++Ev1suspssaLjwmWYLYMPE3Y3PpqUyP+IXimoVCqnvoOOjg6YTCanvgYx06ZNg1KpRFtb\n26DbSSQSSCSSYX12IHLo5O01I1o+vp28A48mPXmm3WkdH01K5JtEfzkWL16MU6dOOVwt1NXVISIi\nAgsWLBjWznQ6HXp6ega9B6Gvrw8fffSRw5BVcs2hk9fumU7ee/PisSQjFhMVEQiSABMVEViSEctH\nkxL5KNErhaKiItTW1kKj0UCtVqOtrQ1arRYlJSUOw1Tz8/ORmZmJzZs3AwC2bt2K4OBgpKamQi6X\no6WlBXv37kVcXBwKCgoAXL3Bbd26dbj77rsxY8YMdHd3o6amBp2dnaisrByjQ/YuIx3fb7ZY8fnZ\nLpfrPj97Affkzh6XM3U+mpTIv4iGglKpRE1NDTZt2oTS0lIoFAoUFxdDo9E4bGe1WmGz2YTXSUlJ\nqK2txaFDh2A2mzFlyhQsXboUa9euRWTk1c7HsLAwREdHo7q6GhcvXkR4eDjS0tLw5ptvIjk52c2H\n6l1GO75fbzTjksHscl13bx/0RvO4dvLy0aRE/kFit9vtni7EaPjqaIb9J8+6bItfkhGL+5ckiL7f\nbLGiYvenuOgiGCYqIvCsOotn7KPAkTLuxzp1L4+NPiL3E2v6GcooooFOXlfYyUtEI+W7dxb5MHc1\n/Qx05n5+9gK6e/sQJY/A/IRJ7OQlohFjKHjAwPh+V00/wxnf//NO3uCwUFj7LbxCIKJRYfORB7i7\n6Sc8NBhTJv2KgUBEo8YrBQ9h049/41Ti5KsYCh7ibeP7+SPmHpxKnHwdQ8HDPD2+nz9i7jVwl/mA\ngbvMAQxpqDGRp/FffYDj8xDcxx1DjYk8jaEQwPgj5l5DGWpM5O0YCgHMH3/EzBYrOruveCTQOJU4\n+QP2KQQwd90v4Q28oW+EU4mTP+CVQgDzp6kyvKVvhFOJk6/jlUKA84f7JbxlGnHA+4YaEw0XQyHA\n+cOPmLdNIw54fqgx0Uix+YgAXPsR87VAANjBS+RODAXyef7UN0LkaWw+Ir/gD30jRN6AoUB+wR/6\nRoi8AUOB/Ao7eIlGh30KREQkYCgQEZGAoUBERAKGAhERCRgKROT1PDn7baDh6CMi8lreMPttoGEo\nEJHX4uNNxx+jloi8Ep8M6BkMBSLySv74ZEBfwFAgIq/E2W89g6FARF6Js996Bjuaichrcfbb8Tek\nUGhubsZf/vIXNDY2Qi6X43e/+x0efvhhBAdfP6m/+eYbPP/882hqakJPTw8mTZqE7OxsPPLII5g8\nebLDtidPnkRlZSXOnz+P6dOn4+GHH8Ydd9wxuiMjIp/H2W/Hn2go6PV6lJSUID4+HlVVVWhtbcXW\nrVths9lQVlZ23ff19vYiNjYWK1euxOTJk9He3o4dO3bgyy+/xOHDhxEScnXXZ86cwfr163H//ffj\nySefRH19PcrLy6FQKJCTk+O+IyUin8XZb8ePaCgcPHgQZrMZWq0WMpkM2dnZMBqN0Gq1UKvVkMlk\nLt+Xnp6O9PR04XVWVhZuvPFGrFmzBk1NTZg3bx4AoLq6GhkZGaioqAAALFy4EM3NzdixYwdDgYho\nnIl2NDc0NCAnJ8fhx7+goAB9fX04ffr0sHY2YcIEAIDFYgEA9Pf347PPPsPy5csdtisoKEBjYyN6\ne3uH9flERDQ6oqGg0+mgUqkclk2dOhVSqRQ6nU50BzabDf39/dDpdNi2bRuSk5ORkpICAGhtbYXF\nYnH6fJVKBZvNhnPnzg3nWIiIaJREm48MBgPkcrnTcoVCAYPBILoDtVqNU6dOAQDmzZuH3bt3I+in\nOUv0er3wWT+nVCqFfRMR0fgZ8yGpTz31FPR6Pc6fP4/q6mqo1WocOHAA4eHuufEkKioSISEcjQAA\nMTHO4U0jx/p0n77+H9Fx4TKilFJEhHEkvLuMxXdU9K+jUChgNBqdlhsMBqczfFdmzpwJAEhNTUVG\nRgZuv/12HDt2DL/97W+FK4Jf9h1c7wrCle7uK6LbBIKYGDm6utgH4y6sT/dwmOW014xoOWc5dZfR\nfEcHCxPRv4pKpXLqO+jo6IDJZHLqCxAzbdo0KJVKtLW1AQDi4uIQGhrq9Pk6nQ5BQUGYNWvWsD6f\nfBvnzPc/A7OcXjSYYbdfm+X0rQ+aPV00ug7RUFi8eDFOnTrlcLVQV1eHiIgILFiwYFg70+l06Onp\nQWxsLAAgLCwMWVlZOHHihMN2x48fR1pamsu+DPI/VpsN+0+eRcXuT7Fx16eo2P0p9p88C6vN5umi\n0ShwllPfJNp8VFRUhNraWmg0GqjVarS1tUGr1aKkpMRhmGp+fj4yMzOxefNmAMDWrVsRHByM1NRU\nyOVytLS0YO/evYiLi0NBQYHwvoceegirV6/Gc889hyVLlqC+vh719fXYs2fPGBwueSPOme+fhjLL\nKW9I8z6ioaBUKlFTU4NNmzahtLQUCoUCxcXF0Gg0DttZrVbYfnZml5SUhNraWhw6dAhmsxlTpkzB\n0qVLsXbtWkRGXvsiZGRk4NVXX8Urr7yCAwcOIDY2Ftu2beONawFC7GzyntzZnNbARw3McnrRRTBw\nllPvJbHb7XZPF2I02Bl4la92jHZ2X8HGXZ/C1ZcwSAJsXrvQI2eTvlqf3mb/ybMOV4EDlmTE8ipw\nlMaqo5ljw8ijeDbp3zjLqe9hKJBHDcyZ7+psknPm+76fz3IaHBYKa7+Ff1Mvx1Agj+PZpP8LDw1G\nzKRfsUnOBzAUyOM4Zz6R92AokNfgnPlEnsf7zIl+wWyxouPCZd5cRQGJVwpEP+E8PUQMBSIB76wm\nXzFwNWu1WN3e/8ZQIALvrCbfMB5Xs7wmJsLQ5ukh8rTxmHWWoUCEa3dWu8I7q8kbjNesswwFIly7\ns9oV3llN3mC8rmbZp0D0E95ZTd5svOYJYygQ/YTz9JAYs8Xqsbvux2ueMIYC0S9wnh76JYdRPwYz\nohWeuYdlPK5mGQpERCK85R6W8biaZUczkR8zW6zo7L7CKTtGwRufNR0eGowpk341Jk1YvFIg8kPe\n0tzhDwLtWdP8dhD5IYebnDA2NzkFikC7h4WhQORnvLG5w5cF2j0sbD4i8jPubu7w5DBMbxFI97Aw\nFIj8jLtucmK/xDWB9HTAwPrLEgUAdzV3sF/C2cDTAf01EACGApFfujcvHksyYjFREYEgCTBREYEl\nGbFDbu5gv0TgYvMRkR8abXNHoA3DpGt4pUDkx0ba3BFowzDpGoYCETkJtGGYdA2bj4jIpUAahknX\nMBSIyKVAGoZJ1zAUiGhQA/0SFBjYp0BERAKGAhERCYYUCs3NzSguLkZqaipycnJQWVkJq3Xwm1f+\n+9//YuPGjcjPz0dqaiqWLVsGrVYLs9lx7PP27duRmJjo9F9DQ8PIj4qIiEZEtE9Br9ejpKQE8fHx\nqKqqQmtrK7Zu3QqbzYaysrLrvu/48eNobW2FWq3GjBkz0NTUhMrKSjQ1NWH79u0O28rlcuzZs8dh\n2ezZs0d4SERE3svbJxgUDYWDBw/CbDZDq9VCJpMhOzsbRqMRWq0WarUaMpnM5fvUajWio6OF11lZ\nWQgPD8fTTz+N7777DtOmTRPWBQcHIy0tzQ2HQ0TknXxlgkHRkjQ0NCAnJ8fhx7+goAB9fX04ffr0\ndd/380AYMGfOHABAZ2fnSMpKFFD4KE3/4isTDIpeKeh0OixcuNBh2dSpUyGVSqHT6ZCXlzfknTU2\nNiIoKAhxcXEOy3t7e5GVlQWj0YibbroJf/jDH7B06dIhfy6RP/GVM0pf48lmG7EJBu/Jne01TUmi\noWAwGCCXy52WKxQKGAyGIe+oq6sL1dXVWLFiBSZOnCgsj4uLw6OPPoo5c+bg8uXLeOutt6DRaLB9\n+/YhBUNUVCRCQryjMj0tJsb570Qj56n63P33/4OTZ9qF1wNnlJHSMKhXJnukTO7iiTq1Wm3467Ev\n8en/7UBXjwkxE6RYmDQFa+6ah+Dg8QnZjguXcan3+hMMBoeFImbSr4b9uWNRn+Ny81p/fz/+9Kc/\nITIyEhs3bnRYt2LFCofXeXl5KCoqwo4dO4YUCt3dV9xaVl8VEyNHV1evp4vhNzxVn2aLFR9/8Z3L\ndR9/8f+wfMF0rzmjHC5P1en+k2cdQraz24R//FuHK6Z+3L8kYVzKYLVYES2//oOPrP2WYdfNaOpz\nsDARjUmFQgGj0ei03GAwQKFQiO7cbrfj8ccfR3NzM1577TUolcpBt5dIJMjPz0dTU5PosFcifzOU\nKatp6LzluRC+NMGg6JWCSqWCTqdzWNbR0QGTyQSVSiW6g+eeew7vv/8+/vrXvw55mKlEIoFEIhnS\ntkT+xF2P0qSrvOm5EL4ywaBoKCxevBh79+6F0WgURiDV1dUhIiICCxYsGPS9u3btwt/+9je88sor\nyMjIGFKB7HY7/vnPf+LXv/41goO9Jz2JxsPAGeXPmzsGeNsZpS/wppD1lQkGRUOhqKgItbW10Gg0\nUKvVaGtrg1arRUlJicMw1fz8fGRmZmLz5s0AgGPHjuGll15CYWEhbrjhBjQ2NgrbxsXFCUNWH3jg\nASxduhQqlQomkwmHDh3CF198gR07drj7WIl8gq+cUfoCbwxZb59gUDQUlEolampqsGnTJpSWlkKh\nUKC4uBgajcZhO6vVCpvNJrz++OOPAQBHjx7F0aNHHbbdsmULCgsLAVwNiNdffx1dXV0ICgrC3Llz\nsWvXLuTm5o764Ih8ka+cUfoKhuzwSOx2u93ThRgNjri5iqOP3Iv16X6erlNvn15iuMZq9BGfp0BE\nAcHbm228BW+PJCIiAUOBiIgEDAUiIhIwFIiISMBQICIiAUOBiIgEDAUiIhIwFIiISMBQICIiAUOB\niIgEDAUiIhIwFIiISMBQICIiAUOBiIgEDAUiIhIwFIiISMBQICIiAUOBiIgEPv+MZiIich9eKRAR\nkYChQEREAoYCEREJGApERCRgKBARkYChQEREAoYCEREJGAo+6ujRo0hMTHT678CBA54umk/49ttv\n8fTTT+Ouu+7CnDlz8Pvf/95pG7vdjp07dyI3NxcpKSlYtWoV/ve//3mgtL5hKHWal5fn9J3Nzs72\nQGm93/Hjx1FaWopFixZh/vz5KCwsxDvvvOO03aFDh7B06VIkJyejsLAQn3zyyaj2GzKqd5PHvf76\n64iIiBBeT58+3YOl8R3ffPMN6uvrkZqaih9//NHlNq+99hqqqqrw2GOPQaVSYd++fSgpKcE777yD\nmJiYcS6x9xtKnQLAnXfe6RAYoaGh41E8n1NTU4PY2Fhs3LgRUVFRaGhowIYNG9Dd3S3U3zvvvINn\nnnkGDz/8MG6++WYcPXoU69atw+HDh5GQkDCyHdvJJx05csSekJBgNxqNni6KT7JarcL/azQa+wMP\nPOCwvq+vz56enm7fvn27sOzy5cv2rKws+0svvTRu5fQlYnVqt9vtt912m/35558fz2L5rIsXLzot\nKy8vt992223C66VLl9qfeOIJ4bXVarXfeeed9g0bNox4v2w+ooAUFDT4V/8///kPjEYjli9fLiyL\njIzEbbfdhn//+99jXTyfJFanNDzR0dFOy+bMmYPOzk4AQFtbG86fP+/wHQ0KCsKyZctG9R3lX9HH\n5efnY+7cuVi2bBkOHjzo6eL4DZ1Oh+DgYMycOdNh+ezZs6HT6TxTKD9x+PBhJCUl4eabb8b69evx\n3XffebpIPqOxsRGzZs0CAOF7qFKpHLaZPXs2enp6cOnSpRHtg30KPiomJgaPPPIIUlJSYLVaUVdX\nh2eeeQZ9fX0oKSnxdPF8nsFgQGRkJIKDgx2WK5VKmEwm9Pf3IywszEOl8115eXlIS0vDjTfeiJaW\nFmi1WqxatQrHjh2DXC73dPG82ieffIKTJ09i8+bNAAC9Xg8AUCgUDtsplUphvaurDTEMBR+1aNEi\nLFq0SHidm5sLs9mM6upqrF69mpfy5JUqKiqE/8/IyMD8+fOxcuVKHDlyhCczg2hvb8eGDRtw++23\no7CwcEz3xV8OP7Js2TL09PTwctwNFAoFrly5AqvV6rBcr9dDKpXyKsFNEhISMGvWLHz11VeeLorX\n6unpgVqtxtSpU/Hiiy8KyweuCHp7ex22H7iCGFg/XAwFPyKRSDxdBL+hUqlgtVrx7bffOizX6XRO\nbbg0OhKJhN/d6zCZTCgtLYXFYsGuXbsglUqFdQPfw1/2cel0OkyYMGFETUcAQ8GvvPfee4iKisK0\nadM8XRSfl56eDplMhhMnTgjLTCYTPvzwQ4dmOxqds2fPQqfTYd68eZ4uitf58ccf8cgjj+D8+fPY\ns2cPJk6c6LB++vTpmDlzpsN31Gaz4cSJE6P6jrJPwUdpNBokJycjMTERNpsNdXV1qKurQ0VFBfsT\nhsBkMqG+vh4A8MMPP8BoNAr/uHJzcyGVSrF27VpUVVVBqVQKN6/ZbDaXd+qSeJ1+9tln+Mc//oFb\nb70VkydPhk6nQ3V1NaZMmTLm7eS+6M9//jPq6+vx5JNPoqenB42NjcK6uXPnIiwsDBqNBo8++iim\nTZuG9PR0/P3vf8e3336LbduIg+jYAAAAzElEQVS2jXi/fBynj3rppZfw3nvv4fvvv4fdbkd8fDxW\nr16NlStXerpoPqG9vR233367y3Xvv/8+YmNjhWkuDhw4gJ6eHiQlJaGiogJz584d59L6BrE6NRqN\n2LJlC5qamtDb24sJEyZg0aJFKCsrww033DDOpfV+eXl51+0fHPiOAlenudi9ezc6Ojpw00034bHH\nHsMtt9wy4v0yFIiISMB2BiIiEjAUiIhIwFAgIiIBQ4GIiAQMBSIiEjAUiIhIwFAgIiIBQ4GIiAT/\nHx5uI1iLJl5vAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.scatter(scores[:, 0], scores[:, 1])\n",
    "pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Precision-recall and ROC curves\n",
    "\n",
    "### Confusion matrix\n",
    "\n",
    "| | | Truth=P | Truth=N |\n",
    "| - | - | - | \n",
    "| **Predict=P** | | TP | FP  |\n",
    "| **Predict=N** | | FN | TN |\n",
    "\n",
    "\n",
    "We will use these definitnios in the construciton of the preciison-recall and ROC curves.\n",
    "\n",
    "### PRC (Precision-recall Curve)\n",
    "\n",
    "- X axis is the recall \n",
    "$$\n",
    "\\frac{\\text{TP}}{\\text{TP + FN}}\n",
    "$$\n",
    "- Y axis is the preciison\n",
    "$$\n",
    "\\frac{\\text{TP}}{\\text{TP + FP}}\n",
    "$$\n",
    "\n",
    "### ROC (Receiver Operating Characteristics)\n",
    "\n",
    "- X axis is the False Positive Rate (FPR)\n",
    "$$\n",
    "\\frac{\\text{FP}}{\\text{FP + TN}}\n",
    "$$\n",
    "- Y axis is the True Positive Rate (TPR)\n",
    "$$\n",
    "\\frac{\\text{TP}}{\\text{TP + FN}}\n",
    "$$\n",
    "\n",
    "To plot the PRC and ROC, we calcualte the necessary statistics for a range of cutoff values or thresholds - for example, the cutoff may be whether the score is lower or higher than 0.2, or 0.3 etc.\n",
    "\n",
    "### Imbalanced data sets\n",
    "\n",
    "In real world settlings, imbalanced data sets, where members of one class are abundant while the other are infrequent, are common. Imbalanced data sets are nearly always due to the preponderance of true negatives (TN), for example, when screening natural compounds, nearly all will be expected to have no or minimal anti-cancer activity. \n",
    "\n",
    "Notice that TN is NOT used in the PRC, which is hence robust to data sets imbalanced becasue most of the data come from the TN class. In contrast, the use of the FPR makes the ROC sensitive to such data sets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test on hold-out data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "n = int(best[0])\n",
    "n, y_score = fit(n, X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.4636398109078834"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "average_precision_score(y_test, y_score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualize scores for inliers and outliers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame(dict(score=y_score, y=y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAD9CAYAAACWV/HBAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3Xd8m9XZ8PGfhm3JtuS9V+KRvYcz\nMRkEAmEVGmYfEkZo2KOFFNry9CktYbwdISmUstKS0rAplAQIkMRkmmwncYYtx3sPyZJl7fcPJUoU\nO8O1E2Vc38+HP3SkW7okh/u6z7nOObfC4/F4EEIIIf5LykAHIIQQ4vwmiUQIIUSPSCIRQgjRI5JI\nhBBC9IgkEiGEED2iDnQAZ1tDQ1ugQxBCiPNOXJzuhM9Jj0QIIUSPSCIRQgjRI5JIhBBC9IgkEiGE\nED0iiUQIIUSPBDSRlJWV8cwzz3DNNdcwcOBA/ud//ue0jmtra+Opp55i7NixjB49mp/97Ge0tLSc\n4WiFEEJ0JaCJ5ODBg6xdu5a+ffvSp0+f0z7u0UcfZfPmzfzud7/j+eefZ/fu3TzwwANnLlAhhBAn\nFNB1JNOmTeOyyy4D4OGHHz6tXsX27dtZt24dy5YtY+zYsQAkJCQwe/ZsNmzYwMSJE89ozEKIc5vb\n42Zf80HMDgtDYgYSGqQNdEgXvIAmEqWy+x2i/Px8YmNjfUkEYNiwYaSmppKfny+JRIiL3Gu7lrK7\naR8A4UFh/Hz0g8SFxgQ4qgvbeVdsNxgMZGZmdmrPysrCYDAEICIhxLnCYCzzJREAs8PC6sp1AYzo\n4nDebZFiMpnQ6Tov1dfr9VRWVp7y+KioUNRq1ZkITQgRYDWuzv9vK4M8J93eQ/TceZdIeqqlpT3Q\nIQghzpB4RRKJYQnUWuoAUClUjIoeKXvs9YKTJePzLpHo9Xqam5s7tZtMJvR6fQAiEkKcK1RKFY+P\nuo/1VZtpc5gZmziSdF1qoMO64J13iSQzM5OtW7d2ajcYDL4ZYEKIi1dYUCiX95ka6DAuKuddsT0v\nL4+Ghga2bNniayssLKSiooK8vLwARiaEEBengPZIrFYra9euBaCurg6z2cyXX34JwKWXXopWq2XG\njBmMHTuW5557DoCRI0cyefJkFixYwIIFC1Aqlbz00kuMHj1apv4KIUQAKDwejydQH15ZWcn06dO7\nfO7bb78lNTWVadOmkZuby/PPP+97zmQysXDhQlatWoXb7Wbq1Kn88pe/JDo6+pSfKUU3IYTovpMV\n2wOaSAJBEokQQnTfBTVrSwghTqbJ2sw35fmYHWbGJ41hcMyAQId0wZNEIoS4YDhcDv647VVabUYA\nttcXcv/wuxgU0z/AkV3YzrtZW0IIcSIHWkt8SQTAg4cf6rYHMKKLgyQSIcQFQx/cxfZJXbSJ3iWJ\nRAhxwUjTpTAxKdf3OE4bw7S0SwIY0cVBZm0JIS441eZazA4LWRF9UCllk9beILO2hBAXleTwxECH\ncFGRoS0hhBA9IolECCFEj0giEUII0SOSSIQQQvSIJBIhhBA9IolECCFEj0giEUII0SOSSIQQQvSI\nJBIhhBA9IolECCFEj0giEUII0SOSSIQQQvSIJBIhhBA9IolECCFEj0giEUII0SOSSIQQQvSIJBIh\nhBA9IolECCFEj0giEUII0SMBTSTFxcXMmTOH4cOHM3nyZBYtWoTL5TrlcYWFhdx1113k5uaSm5vL\n3Llz2blz51mIWAghxPEClkiMRiNz585FoVDwyiuv8MADD/D222/z8ssvn/S4mpoa7rzzTpxOJy++\n+CIvvvgiLpeLO++8k6qqqrMUvRBCiCPUgfrg5cuXY7PZWLJkCeHh4UyaNAmz2cySJUuYN28e4eHh\nXR63Zs0aLBYLf/nLX9DpdACMHDmS8ePHs3btWm677baz+TWEEOKiF7AeSX5+PpMnT/ZLGLNmzaKj\no4OCgoITHud0OlGpVGi1Wl9baGgoKpUKj8dzRmMWQgjRWcASicFgIDMz068tOTkZrVaLwWA44XGX\nX345Wq2W559/nqamJpqamli4cCERERFceeWVZzpsIYQQxwnY0JbJZPINTR1Lr9djMplOeFxCQgL/\n+Mc/+OlPf8o777wDQFxcHG+++SbR0dGn/NyoqFDUatV/H7gQQgg/AUsk/636+noeeeQRBg8ezO9+\n9zsA3n33Xe69916WL19OcnLySY9vaWk/G2EKIcQFJS6u84X/EQEb2tLr9ZjN5k7tJpMJvV5/wuPe\nfPNNnE4nL7/8Mnl5eeTl5fHyyy+jUql46623zmTIQgghuhCwRJKZmdmpFlJTU4PVau1UOzmWwWAg\nOzuboKAgX1twcDDZ2dmUl5efsXiFEEJ0LWCJJC8vj3Xr1vn1SlasWIFGoyE3N/eExyUnJ3Pw4EHs\ndruvzW63c/DgQVJSUs5ozEIIIToLWCK55ZZbCA4O5qGHHmLDhg289957LFmyhLlz5/pNCZ4xYwZP\nP/207/Hs2bOpr6/nwQcfZM2aNaxevZr777+fhoYGbr755kB8FSGEuKgFLJFERESwdOlSXC4X8+fP\nZ/HixcyZM4eHH37Y73Uulwu32+17PGTIEN544w0sFgtPPvkkCxYsoKOjg7feeosBAwac7a8hhBAX\nPYXnIlvF19DQFugQhBDivHNOztoSQghxYZBEIoQQokckkQghhOgRSSRCCCF6RBKJEEKIHpFEIoQQ\nokckkQghhOgRSSRCCCF6RBKJEEKIHpFEIoQQokckkQghhOgRSSRCCCF6RBKJEEKIHpFEIoQQokck\nkQghhOgRSSRCCCF6RBKJEEKIHpFEIoQQokckkQghhOgRSSRCCCF6RBKJEEKIHpFEIoQQokckkQgh\nhOgRSSRCCCF6RBKJEEKIHpFEIoQQokcCmkiKi4uZM2cOw4cPZ/LkySxatAiXy3Vax3799dfceOON\nDBs2jHHjxnH33XfT3t5+hiMWQghxPHWgPthoNDJ37lyys7N55ZVXKC8v54UXXsDtdvPYY4+d9NgP\nPviA3/72t9xzzz08+eSTmEwmNm3adNpJSAghRO9ReDweTyA++LXXXuONN95g9erVhIeHA/D666+z\nZMkS1q9f72s7XnNzM9OnT+epp57ipptu6vbnNjS09ShuIYS4GMXF6U74XMCGtvLz85k8ebJfwpg1\naxYdHR0UFBSc8LiVK1cCcP3115/xGIUQQpxawBKJwWAgMzPTry05ORmtVovBYDjhcbt27aJv3758\n+OGH5OXlMXjwYGbPns22bdvOdMhCCCG6ELAaiclkQqfr3FXS6/WYTKYTHtfY2EhpaSmvvvoqTzzx\nBJGRkbzxxhvcc889fP3118TGxp70c6OiQlGrVT2OXwghhFe3E4ndbueTTz6hoKCA5uZmnnjiCQYN\nGoTRaGTVqlVMmjSJpKSkMxErAB6Ph/b2dhYtWkReXh4Ao0aNYurUqSxbtoxHH330pMe3tMjMLiGE\n6K6T1Ui6lUiampqYM2cOJSUlxMXF0dDQgNFoBLw9iVdffZWSkhIWLFhwyvfS6/WYzeZO7SaTCb1e\nf9LjFAoF48aN87WFh4czePBgSkpKuvN1hBBC9IJu1Uheeukl6urqeO+99/j00085dsKXQqHg8ssv\nZ926daf1XpmZmZ1qITU1NVit1k61k2NlZWXh8Xg4frKZx+NBoVB049sIIYToDd1KJGvWrGHOnDkM\nGzasy5N2eno61dXVp/VeeXl5rFu3zq9XsmLFCjQaDbm5uSc8bsqUKQBs3rzZ19bW1saePXsYMGDA\naX4TIYQQvaVbicRqtRIXF3fS591u92m91y233EJwcDAPPfQQGzZs4L333mPJkiXMnTvXb0rwjBkz\nePrpp32Phw4dyvTp0/nlL3/JJ598wpo1a7jvvvtQq9Xcfvvt3fk6QgghekG3EklWVtZJp9muXr36\ntHsFERERLF26FJfLxfz581m8eDFz5szh4Ycf9nudy+XqlJxeeuklpk+fzvPPP8/DDz+MWq3m73//\nOxEREd35OkIIIXpBt1a2f/TRR/z617/mySefZMaMGUyfPp23336bjIwM/vKXv/Dxxx/zpz/9iZkz\nZ57JmHtEVrYLIUT3nWzWVre3SFmyZAmvvvoq4O0tqNVqXC4XCoWChx9+mPnz5/cs2jNMEokQQnRf\nryYSgKqqKlatWkVZWRlut5v09HRmzJhBenp6jwI9GySRCCFE9/VKIrHZbKxcuZLMzEyGDRvWa8Gd\nbZJIhBCi+3pl08aQkBB+/etfU1RU1CtBCSGEuDB0a9ZWTk4OtbW1ZyoWIYQQ56FuJZJHH32Uf/3r\nXyfd5l0IIcTFpVt7bb3//vtEREQwZ84cMjIySE1NRaPR+L1GoVCwePHiXg1SCCHEuatbiWTv3r0A\nJCUlYbfbu7xviOx3JYQQF5eA3Wo3UGTWlhBCdN85eatdIYQQF4b/6g6Ja9asYc2aNb6dfpOTk5k6\ndSqXXnpprwYnhBDi3Netoa2Ojg4eeOABNmzYgEqlIiEhAYC6ujpcLheTJk1iyZIlnQrw5xIZ2hJC\niO7rtaGtRYsWsWHDBh577DEKCgr49ttv+fbbb/nhhx/42c9+xoYNG1i0aFGPAxZCCHH+6FYiWbFi\nBTfffDP33nsvoaGhvnatVss999zD7NmzWbFiRa8HKYQQ4tzVrUTS3NxMTk7OCZ/v168fzc3NPQ5K\nCCHE+aNbiSQtLY21a9ee8Pm1a9eSlpbW46CEEEKcP7qVSG677Tby8/OZP38+GzdupLa2ltraWjZu\n3Mh9993H999/L7e7FUKIi0y3FyQuXryYv/3tbzidTr92tVrNT3/6Ux588MFeDbC3yawtIYTovl6/\nsVVzczMbN26kqqoKgJSUFCZMmEB0dPR/H+VZIolECCG6r9cTyflMEokQQnRfr60j+eabb3j22WdP\n+Pyzzz7L6tWru/OWQgghznPdSiRvvPEGVqv1hM/bbDZef/31HgclhBDi/NGtRFJcXMzgwYNP+Pyg\nQYMoLi7ucVBCCCHOH91KJC6XC4vFcsLnzWYzDoejx0EJIYQ4f3QrkQwePJiVK1d2mSzsdjsrVqyg\nf//+vRacEEKIc1+3Esk999xDUVERc+fOZe3atVRXV1NdXc3atWuZM2cO+/fvZ968eWcqViGEEOeg\nbk///eCDD1i4cKFf0d3j8RAaGsovfvELbrrpptN+r+LiYp599ll27NiBTqdj9uzZPPjgg6hUqtM6\n3u128+Mf/5g9e/bw17/+lalTp57yGJn+K4QQ3Xey6b/dvrHV7NmzufLKK1m3bh0VFRUApKenM2nS\nJMLDw0/7fYxGI3PnziU7O5tXXnmF8vJyXnjhBdxuN4899thpvccHH3xAXV1dd7+CEEKIXtStoa2i\noiI+//xzwsPDmTlzJvPmzWPgwIG888473Hnnnfz9738/7fdavnw5NpuNJUuWMGnSJG699VYeeOAB\nli5ditlsPuXxRqORP/3pTzz66KPd+QpCCCF6WbcSyUsvvcQXX3zhe1xdXc1DDz1EVVUVKpWK559/\nnvfee++03is/P5/Jkyf79WJmzZpFR0cHBQUFpzx+0aJFjBo1igkTJnTnKwghhOhl3Uok+/btY8yY\nMb7Hn332GQqFgk8++YTly5dz5ZVXsnz58tN6L4PBQGZmpl9bcnIyWq0Wg8Fwyjg++ugjFixY0J3w\nhRBCnAHdqpG0tbX5bcy4du1aJk2aRGRkJAATJkxgzZo1p/VeJpMJna5z8Uav12MymU567O9+9ztu\nv/12MjIyqKysPP0vAERFhaJWn14xXwhx/mmxGll5cDWmjjby+oxnUPyJb8Yneke3EklcXJyvwN7a\n2kphYSG/+tWvfM9bLBaUym51crrtiy++oLS0lL/+9a//1fEtLe29HJEQ4lzhdDt5dtP/o7HDe6fW\n1aUbeXjkvfSLygpwZOe/Xpu1NX36dJYtW4ZOp6OgoAC1Ws1ll13me37fvn2nfYdEvV7fZVHdZDKh\n1+u7PMbhcPDiiy8yb9483G43JpPJ9x5WqxWz2dytmWNCiAvLgZYSXxIB8OBhU80WSSRnWLcSyaOP\nPkpzczOvvvoqOp2OhQsXEhsbC3i3R/nqq69O+w6JmZmZnWohNTU1WK3WTrWTI6xWK7W1tSxcuJCF\nCxf6PffYY4+Rnp7OqlWruvOVhBAXkLCg0NNqE72rW4kkLCyMP/zhD10+FxoaSn5+PhqN5rTeKy8v\njzfffNOvF7FixQo0Gg25ubkn/Ix//OMffm2NjY08/vjjPP7444wfP74b30YIcaHJ0KcxOn44W+t3\nAhAZEsHUtMkBjurCF7AbWxmNRmbNmkVOTg7z5s2joqKC559/njvuuMNvQeKMGTMYO3Yszz33XJfv\nU1lZyfTp02VluxDCp9RYjtlhpn9UDsGqoECHc0Ho1ZXtvSUiIoKlS5fy29/+lvnz56PX65kzZw4P\nPfSQ3+tcLhdutztAUQohzkd9I9IDHcJFRW61K4QQ4pR67Va7QgghxPEkkQghhOgRSSRCCCF6RBKJ\nEEKIHgnYrC1x/vJ4PKzcXM6mPbVE6kK4MS+LjMQTF+KEEBc26ZGIbluzo5oP15RQ2WBht6GZP7y3\nA7vDFeiwhBABIolEdNvO4ka/x2arA0P1yXdsFkJcuCSRiG5Ljg3ze6xSKkiIlv2MhLhYSSIRp6XR\naMVqcwJw1fgM+qV570ETEqTiluk5ROlCAhmeECKAZGW7OCmTxc7LH+3CUG0iOEjJzVOzmToqFYBm\nUwehGjWaYJmzIcSF7pzca0ucHz7fcMhX/7A73Lz7zUFG9Y+nuLKVTXvqiNSFcOW4dKL1p7frsxDi\nwiOJRJxUbZPF77HL7SF/RzWffH/0XjKFhiZ+P28cqjN8d0whxLlJ/s8XJzU8O9bvcUR4MIYao19b\nfYuVkiqZtSXExUp6JOKkpo9Oxe50U7C3jmi9hhsuzWT1tiq/1yiAyPDgwAQoxCm4PW4arE1EhUQQ\nrJJ/p2eCFNtFtzUarbzwz+00mToAuHxsGrdMzwlwVEJ0Vmep59Vdb9NgbUKr1vCTAbMZET800GGd\nl05WbJdEIv4rTpebg5VGonQhJMoaEhEgdpeDFaWr2NdykLTwZK7NuhJdcLjv+Vd3vs3upiLf4/Cg\nMH4/6ZeolTIY010ya0v0OrVKycCMqECHIS5ynxT/h/yqjQBUtFXRaG3mkVE/9T1f397g93qzw4LF\n0U5EiP6sxnmhk0QihDhv7WzY4/f4QGsJFkc75aZK2hxmBkb3o77q6JY+Gbo0SSJngCQSIcR5KyE0\nDqP96IzByJAI/r5nOXua9wEQpg7lkuQJHDKVkRSeyHVZVwYq1AuaTP8VQpy3bsy5hqgQ73Y9oWot\nU9Mm+5IIgMXZjlKp5Be5jzJn0C1EhkQEKtQLmvRIhBDnrVRdMr+d+Avq2xuJ1kRhMB7q9Bqby3b2\nA7vISI9ECHFeUyqUJIbFE6wKIicyk8TQeN9zKoWKScnjAhjdxUGm/wohLihmh4X1VZsxOyyMTRhJ\nuj410CFdEGQdyTEkkQghRPedLJHI0JYQQogekUQihBCiRwKaSIqLi5kzZw7Dhw9n8uTJLFq0CJfL\nddJjdu3axVNPPcWMGTMYPnw4V1xxBUuWLMFmk5kZQggRCAGb/ms0Gpk7dy7Z2dm88sorlJeX88IL\nL+B2u3nsscdOeNzKlSspLy9n3rx5ZGRksH//fhYtWsT+/ftZvHjxWfwGQgghIICJZPny5dhsNpYs\nWUJ4eDiTJk3CbDazZMkS5s2bR3h4eJfHzZs3j+joaN/jcePGERISwjPPPENVVRUpKSln6ysIIYQg\ngENb+fn5TJ482S9hzJo1i46ODgoKCk543LFJ5IiBAwcCUF9f3/uBCiGEOKmAJRKDwUBmZqZfW3Jy\nMlqtFoPBcIKjurZjxw6USiXp6em9GaLoQrOpg92lTVhtzkCHIkS3uD1u8is38ubuZXxTvhaHW/4N\n95aADW2ZTCZ0us7zkvV6PSbT6d+2taGhgVdffZXrrruOmJiYU74+KioUtVrVrViF1xfrDPzt37tx\nuz2EaYP4zbzxDMjo3EMU4ly0bOcnfHbgawC21e+i2dnEA+PmBDiqC8N5vdeW3W7n0UcfJTQ0lKee\neuq0jmlpaT/DUV2YbA4Xb3+xF7fbu37VYnXw1r9388StIwMcmRCnZ3XJBr/H35cVcGOf61Ap5cLy\ndJyTN7bS6/WYzeZO7SaTCb3+1PcL8Hg8LFiwgOLiYt59910iImRXzzPJZndhs/tPzW41y5Rrce6q\nb2/gi9JVNHe0MjphOOHBYbQ5jp5zwtShKBWylK43BCyRZGZmdqqF1NTUYLVaO9VOuvL73/+eb7/9\nlrfeeousrKwzFaY4TB8WzOC+0ewpbfa1TRySGMCIhDhqU80WVpR+g8Pt4NLUicxIn8LL21+nxdYK\ngMF4iLyUCTRam3C4nSgVSn6UPQuFQhHgyC8MAUskeXl5vPnmm5jNZt/MrRUrVqDRaMjNzT3psa+9\n9hr//Oc/+fOf/8yYMWPORrgCuP/6IXy5uZyqRjMKhYLNe+s4WGnkR5dkkpF44m6vEGdStbmWZUUf\n4ME77Pq54SuUCqUviRzRYG3i2YlPc8hUTmp4MlGayECEe0EKWL/ulltuITg4mIceeogNGzbw3nvv\nsWTJEubOnes3JXjGjBk8/fTTvseff/45f/zjH7n++utJSEhgx44dvv+am5u7+ijRS7Qhan6Ul8mQ\nvjFs3d9AZYOFXSVN/PH9HdgdJ9+RQIgzpcR4yJdEjmiwNnUatorVxqALDmdo7CBJIr0sYD2SiIgI\nli5dym9/+1vmz5+PXq9nzpw5PPTQQ36vc7lcuN1u3+P169cD8PHHH/Pxxx/7vXbhwoXccMMNZz74\ni9yO4ka/x23tDgzVJgZkRAUoInEx66NP69TWPyqbxNB4PitZidPjIjkskZl9pgUguouDbCMvuu39\n1cV8ubnc91ilVPDifROJ0oUEMCpxMVtdsY6Vpd9gdzsYHDOAxvZGzM52RscPZ3zSGJLCEqQe0kPn\n5Kwtcf66anwGJVVGDlYaCQ5SMntKtiQREVBT0yYzJXUSJruZ/934PA63A4BvK/JJCI0jOVwmhpxJ\nkkhEt4Vrg3jqJ6NpNnUQqlGjCZZ/RiLwFAoFZaZyXxI5Yn9LMZNS5Ha7Z5KcAcR/LVqvCXQIQvhJ\nCU9GgcKv+J6mk41czzRZjSOEuGDEaKO4dcANvsWGYxJGMCV1UqDDuuBJsV0IcV5rtRn56ODnVLRV\n0T8qm+uzZxGsDMLlcROsCgp0eBcMKbYLIS5Yb+7+JwbjIcC7fsTpdvE/g25CheyhdbbI0JYQ4rzV\n4ezwJZEj9jTvC0wwFzFJJEKI81aIKoQYjf9C2OSwRA62lLCtfhcdTtlY9GyQGonwsdldvLe6mMKS\nJlLjwrj1shzio0IDHZYQJ3WwpYSle5fTajOSEBpPZIie/S3FAOiDdfxs9APEauW+OT11shqJJBLh\n885X+1m9vcr3ODUunN/enYvJYudARSspcWEkxYSd8PgDFa18v6uaME0Ql41JJTZCS1FZC5+vL8Vq\nczF1VAp5w5PPxlcRFxm3x43J3karzchLW5b4PTc1bTI/zrm20zEtHa0EKYMIDz7xv2lxlBTbLyBu\njwdDlYmQYBVp8eGnPqAbjt0iHqCywcwPRXW8+UURdqd3v7Obp2VzRW46zaYOdhQ3EhUewrDsGEpr\n2njx3e24D1+XFBTVseC2Ufz5g504Dh+7dOU+IsKCGZ4d26txi4tXk7WZ76s24XQ7mZici9XR0ek1\nx7fZXQ7e3L2M3U1FKBVKpqVdwo+yZ52tkC9IkkjOI+0dTl7613bK6ry9qjED4rnvusEoFAqaTR14\nPBAT8d8vEkyND6e+1ep7HBEezNc/VPiSCMCn35eSlRzBH97bge3wjr8jsmOJ1IX4kghAq9nOt1sr\nfEnkiF0lTb2WSFrNNj5aU0JFg5nBfaO5fnImQWop+10szHYLL25ZjNlhAWB9TQFPjn6QeG0s9Vbv\nxqJKhRKFQsn/bnwBXVA412bNpNpcy+6mIsDbk/mmfC0j4obSNyI9YN/lfCeJ5DyydmeVL4kAbNlX\nz97hSWzaU8eG3bV48CaXn147iLZ2B19uLqfJ1EHuwATGDog/5fvfMi2bJmMHZXVtROlCuGvWQN5d\ndcDvNXaHi1VbKnxJBLy7AV86ovOQVUpc5x5TUsypay7Npg7W7aoBYPKwJKL1Gsrr2vhwbQktbTbG\nDUzgqgkZ/OWTQkqqTACU15lxOj3celnOKd9fXBh2Nuz2JREAu8vOtvpdPD76fvIrN2B2WNCoQ/i6\nbA0AjdYmXt31NmPih3d6r7r2ekkkPSCJ5DzS2mbv1LarpIn1u2t9j7fsq2dkTiz/2XCImibv/em3\n7m/Ade0gxg1MYM+hZmqa2hmWGUNCtP9JPTZSy//eORZTu51wTRBKpYJLhiXz/upi32tG9Y+jq01U\nR/WLo6ishfoWb49mdL84Lh2RQpPJxpeby3C6PIzqF9cp4ThdbvaVtxASpCInNRKj2cb/Lf2BtvbD\nm+5tq+TXc8byx/d2YDrc9nGDAfD4ksgR2w82SCK5iGiDtJ3agpRBbK/fBcClqRP5T+kqv+ftLnun\ne5EEKYMYGN3vzAV6EZBEch7JHRTPt1srfUNIYRo1oZrOK3f3l7f6ksgRG3fXcaDCyJrDxfT3lcU8\nMnsYKbHhvLWiiH1lLfRJ1DH3qoGkxIaxq6SJjXtqiQgL5vYZORyqaSMlLpzxgxMoq23zJie3N47s\n1AiGZsbwu3vGsa+shVBNEB48fLu1krED4pmZm4a5w8kXGw7xs79sIDE6lNtm5BAXqWXhsm1UN3qv\nKof0jWZw32hfEgHvvU6+Kij3JZFjv6M+NMivPfE0ejviwjEsdhB99RmUmsoASAiNY2v9TirN1QB8\nXb6GCYn+d1BVoGBc4hhiNNF8X7WREFUIM/tMJyJEf9bjv5DIrK3T5PZ4+N3ft1BW24ZSqeCGvL5c\nOb5P7wZ3GorKWli7o4qQIBWX56ajVMAzbxb4TuoKBTx84zBe/nCX3z3jxg1KoKCojmP/2gPSI9EE\nq/1uVJUWH86Nl2by5w92+doFwEnYAAAgAElEQVRiIzT8ft443vuumLU7qvF4YGRODEkxYbTbnOwu\nbcZotjNuUAK3z+jHN1sr+GB1iTce4M6rBlLVaOarggrfe0bpQpg2KoWP1hr8vt+0USl8t63Kr232\nlCw+zjf4viPA5WPT6J8WyZtfFNFucxIXqeHhHw8nJfbszsBxutz8/p2tVNS1EaxWMefK/owbJFuW\n94TD5UChUKBWnvo61+1xs6/5IE63k2BlEIt3vuH3/NjEkdhdDnY17EGjDuHazCvJS51wWnF4PJ5O\n9zCxuexYHBaiNRffTdxk1lYvePHdbRyq9SYhl9vDB2sMjO4ff9bXWQzMiGLgcXcifHT2cL7cXIbb\nAzPGpjE8O5aZ49NZucl786kjJ+3Ne+v8jnO5PRRXGf3aKurNfH+4PnFEo7GDrwoq/E7wWw80cves\nWN75+gB2h7egnr+zmhh9CF8WHL3plQf4fEMpoSH+PaeWNht1zVaOlxIbRmJ0KLXN3h5VYnQoU0am\noNWoef+7YjrsLvqlRXL1xD6Ea4P4Y99omkwdJESHogzAjYsWLttK2eF/Fx0OF699tpfh2TFogmWP\np+5ye9x8ePBz1ldtQqlUcXn6VK7sOx2A8rZK3i36kCpLLQOic7hj4M3ogsNRKpQMiukPgMFY1uk9\ngxRBJITH025vp09EOrmJo/ye39u0n1Xla3F7XExLu4ThcUOwONp5p+g9djfuIyE0jlsH3Eh2ZF82\nVBfw4cHPsLns9NGn89Nhc9AHn/jkejFR/eY3v/lNoIM4m9rbO9cZTsfSlfs4vutW3WQmVBOEud3h\nt6W6yWLHbHUQqjk7eTo+SsvEIUlMGppE4uG6x+A+0YwflMConFhumuYdRqppslB1eBhJAdw8LQdL\nh9N30gZvjyQ1LhxDjX/9IS0hvFNNQhui9iXXI4LUShpaO/x6DyFBKvqnR/pOuOC9p8kt07PZsLuW\nIy8N1wYx58oBTBuVSkpcGKP7x3HL9Bw0wWr6JOq5bEwa00alMmNMGsFB3n2UVColutDgM3b3u/YO\nJ1a7k5Cgo/s2tbTZKK9rIyIsmGXHTUYAUCuVF8Rth9vsZn6o3UZjRzPx2thO90DvbVvrd/JpyQrc\neHB5XBxoLWFgdA6RIRH8edtfqWmvx4OHBmsTrTYjI+OHYrSZ2NO0D7VSTaoumZLWUpo6vNPYNSoN\n0ZpI1latp9nWgsF4iEZrE6PihwFQZ6nnj9tepdHaRHNHK9vqdzE4ZgBfl61ha/1OAMwOC0VN+xmb\nMJLFO9/w3euk1WbE4XIwJHbgGf1Nqsw1vL//U/KrNgKQqgvcOqywsBPfvE56JKfJ3cUA4IHyVvYe\nagW8xeUHbhjKv7456KtjDMuK4f7rh1BW18bfv9xPbVM7w7NjuPOqgWiCVazYVMbe0mbSE3RcO7kv\n4dogqhotbN5bS7g2mMlDkwjVqHG53ewpbcbucDM0K8Z3UjNbHbS02UiJC/Ndjbs9HlpMNqJ0ISRE\nh/oK6marg7kzBzAsK4aapnZGZMeSlRJBVkoEDpebfWUtZCTqvLEFqdh+sJEmk3f+/ZQRyUweksSq\nHyp8Q2MKYMKQRLbsq/ebHpydEkFKbLhfr+SyMalcMiyZJlMHewzNxERouGNmfzKTI/jF7aPJ31lF\ncJCKy0anEhqiRqFQMP7w8JDb46GlzUZEeDAhQSq/7+52e9CHBfs+p7LBTJvFTk5aJGqV96TX0Gpl\nf3krGYk637qbncWN7C5tJi0+nElDE1EqFHy2/hDrdlUTpg3ixkuzGJoZw0drS/iqoByXy8PYgfHc\nc/UgVm+r4r3vinF7vJ/d1cDwhTBa3NDexEtbF2NxeC8ysiP78sjIn1LYWMSXh77F6XYyJXUSk1LG\n4fa42dmwh0ZrE0NiB5IUlgB4T8JlpgpSw5NPqwZR0VbVqa28rYo4bSwN1ia/doOxjN2NRbxe+A+c\nHhcKFNzU7zoeGH43uxr30mZvY3jcEH6/+Y9+x+1o2I3L7UKlVLG7aR9uz9F/ux48FDYWcchU7neM\n0d6GwViO0+30a6+11HeKt8Npo81uJi40xtd2/O/g9rj56tB3bKnbQWRIBNdlX0m6LpXNNVtZVb4G\nD3BZ+qWMjBvKou2v+f4GB1sNhKhCGBk/9JS/5dkmiaQHjl0isfVAA2u2V7Fqy9E6wK6SJtbuqGbl\n5jJazd6e0PaDjejWlKAJVvH1D97XHqg0UtPczo2XZvLcO9twurxvvG5XDb+8YxR/WL7TNwQVG6Hh\nV3PGULC3jvdXF+N0eUiMDuXxm4fT3uFkyceFNBo7iNKFMP+6waTGhfPXf++h0NCENkTNzdOyufHS\nLFZtqWDxx4W43R4uH5vGz24ewcHKVt78z14aWq2M6hfHqH6xRISHsNvQzPurixnTP45Goze5DM2M\nYdv+Bkb1j+NgpRGj2c7AjChqmtrRhKi47bIcmtts6EODKak2UlJl4qpxGTx0w1BKqoz869tiXv10\nD2MHxHP7jH5U1FtY/HEh1Q0WBvWN5p6rB9Fs6uDVT3fTaOwgNkLD/T8aQp9EPe9+c4DV26pwuz3k\nDkrgnqsH8s5XB8jf6S2yxkdqWXD7KAzVJl79dLdvcsLN07JRq5T885hexIGKVgakR/HvdaUANJls\nLP6okAd+NIQvNh4dKikoqicrWc9Haw2+9zNZuu7dljdYumw/n6ytWu87gQEUt5ZSULudf+77wHfy\nfXf/R8Roo9lQXeC7gv/M8CX3D78Lt8fD64X/wOF2oFKo+MnA2b5hpZaOVsKDwwk6rgbSPyqbb8rX\n+h4rUNAvMovwoDC/tSEAWRF9+MzwJU6Pdxq6Bw+fGb5iYlIuLo+L+vZGDMYyojSRWMxHv0dEsB6V\n0nsxkhAa1+l7x4fGkhnZh2rL0ZmQkSERDI7pT2RIBK22o0PBx/dGNlb/wAcH/43NZSclPIn7ht1J\ntaWW1wvf8fsdLI52/lP6NQC17fVU7qjm3qF38E7R+74bci0reh+Lw+L3NwDY0VB4TiYSGdo6TZ+v\nL+00tHW82EgNpTX+Qz26sGCKK/3rEE63m/K6Nqy2o2sxGlqtOJxuv3UipnY7apWSjXuO/qNutzlR\nqxT8e10pTpc3IrPVgdXmYn1hjW/oqsPuoqTKiKXD4VuT4XS5KTQ0kR6v480virA5XNidborKWkhP\nCOe1z/ZSfziO8jozcZGhHKw0snJzOfWtVqob2xmQEcXlY9L4+5f7Ka1to7LBQlyEhgduGMIn+QYq\n6s0cqm2jtKaN2VOy+Nvne6lqsFDT1M6mvXUMz47h5Q+9yc7p8n5fBd6FjkdqJg2tVlrNNtbtqqHm\n8LBbu83JoZo2EqK0LPv6gK8nUNVgQaVU+BXyLR1OFAr4flc1xmNO9oZqIzVN7ZitR2d6VTVYUKkU\nvplj4O0FhYaoOg3bRYQHd2rryoyxqfRNOr9nARU27qWsrdKvLTIkotPVepAiiM11W32PPXhos5vZ\n2bAbo93kazMYDzEidih/3v5X/l2yku+rNhIfGktiWDxlpgq+OvQdbXYzA6NzaLa1ogsOZ0jMQCwO\nC1GaSIbEDqKirQqLw8KA6H7cOuAG1lSsp8N1dNW62+PG5rTxSckXHDKVs61+FwOj+9Hc0eIrxt8+\n8MckhnnXVMVpY2mxtVJlrkGBgtHxw5mWnofL7aLDacNkbyMlPIlrMmeyuW4ruuBwwtShaNQaJiSN\npcNpY31NAeBNUC/v+Bv2w0NfbXYzVmcHm2u2+v0OpcYyOlw2Go/pYdndDpQKFeXH/d4J2jjK2ir8\n2obHDaZfVHb3/6C94GRDW7IM+DR1NbR1rBh9CDPGpBN83MrqCYMTiNL5/wFyUiOJjfCfA68PDfIb\nhz/iSO/kWK1muy+JHFHfavUlkSNqm9qpqDP7tbncHnYeM0vriB0HG/1OsAD7y1s6Feh/KKpnzY4q\nv6RaXm/m64IKv7qI2ergy4IKv/hdbg9rd1Rj6fAfIigqa/H1dI4wVBs7fZ+qRjPVx01rBu+Q1vGM\nFjsddpdfm93hJljt/xurVQoyEv0Lpgpg7MAEVEr/usu4QQlkpZw6QRwobz3la851k5LHEaQ8OmEg\nKSyBEfFDOr0uIazrXQra7P5/E4ujnU9LvqDKXON7/M+iDzG0HuIPW18hv2ojX5V9x/rqAhaMeQil\nQsmm2i18ZviS5wr+RLBSzdWZlxMWFMbe5v28tuvvvlrHEWMSRrCuZrNf256mffx63M+5JHk8I+KG\nolJ4//5N1mb+tO1VNtZsISksgQdH3MPVmVfw7Kb/x9K9/2Jfy0FGJQxn/rC7+Ne+j/i2PJ+C2m2U\nt1Vy75A72Nu0n28q1rK9fhdv7l5GfuWGTveKr2uvx2T3v/AwOywkhSb4takVKvpFZXX6DQfG9GNm\nxjRfbaqPPp2paZd0+XsHmvRITtORoY/j5Q1PYlDfaGaOy+BQrYnhWbE4XW4iwoO5dlJflAoFmUkR\nNLd1YLU5GdUvjsvHppEYE8r+8lZsDhchQSpun9GP/ulRbDvQ4NtWZGBGFLdOz2HjnlrfSTE4SMnd\nswaxp7TZ74R8eW4awWql38l3SGY0w7JiKTQcvfoJDVFz/SV92XDMIkaAayb2YXdpk1+CGtM/HnO7\nw29dR7QuhOTYcMqPS1CD+kZjqPYvxo/KieNAhf9JNW9EMgcqWv0+J3dQPG3tDr/vMzI7liidxm8i\nwPDsWKaOTGHN9iq/2sRtl/VjX3mLXw9v9tRsYvQa9h5q8bVNGprE+EHx7Cxu8h1/5fh0+qVGYne6\nqW1qRxuiZvbULHShwWSnRmDpcBCuDeLqiX2wO9zERWpIiApFFxrM5WPT2FXiP3YPUNVo4brJfTu1\nn0/0ITpGxg0lNEjLiLih3NTvOhLDEnC7XRw6fJU8Lmk012TOpLa9gVqL94JDqVByU7/rCFGHUHrM\nLKoxCSOpsdTR5jj678bhduDyuHzrPgA6XB0oFLCzYY+vzelxoVAoWXnoG9/xrTYj0ZpIZqRPwYOH\n7MhMbsi+mnVVm3y9AoCwoDAOthrY3rCLKksNW+p2EKuJ5quy1Rxo9U5Rb3OYqTLX0NLRysHWo9PR\nK83VBCnVFLUc8IvF7fFQ2LS302+mQEG78+hMxLzUicRrY33rXMA7HfnarCs5ZCqnuaMFjUrDj/td\ny/ikMdhddsrbqlCiYEraZC5NnUT/6Bwmp4xjYtJYrugzLaB3fDxZj0TWkZymu57/rsv2J24dSWub\njTf+s9d3lX7bZTkMzYxh4bKtvgVzk4Ykcuesgbzx+V42Hb7KH5YVw9UTMzBUmfh0XSkddhf90yIY\n1S8eXVgQu0ua2FnSRGykhowEPdoQFWGaIIqrjISGqHG43JgsdjKT9bS123G5PTicHqobLaTGhaEP\nD8bS7sADHKptIzI8hNH94zCa7bSabewqacLt9jB1VAoJUaE0Gq2sK6yhpc3GiOxYLh+bRlldG598\nX4rN7iJYreTOqwYQERbMK5/u8fVgxg6IY+qoVJZ9fcA3RDS4TzQThySwfnet72Q+ID2S8YMTMVps\nfL+zhmaTjWFZ0WQk6miz2CmpNlHbbGVgRhR9k/U0Ga00m2zUt1rJTokgNS6cfeUtBKmUtJptuD0e\nYvQaapraCQtVE3n4H3qoJogt++pxuNwM7hNFRFgILreHLfvqaLe56JukY9ygBNRKJR/lG7DavLOy\n7po1kPT4MP7f8p2+iQaXj03jusl9+b+lP/hW7cfoQ/jfO3NZunIf2w40dPnv4q1fTPuv/p2dD2wu\nO26PC63a26t2uV3saCikwdrEsNjBJIcn4va4ya/cyMHWEjJ0aUxNm8wXpatYVb7G9z6JYQkMjx3M\nV2X+/29dk3kFnxu+8mublDyO9dX+vY3EsAQGx/Tn2/J8AHRB4UxKHseXZd8C3hP7dVkz+bRkpd9x\nWRHeGojV6d8LHhE3hB0Nu/3aZmZM973fEddmXsHnhq999Qzw9oau6juDz0u+pNHaxPC4oVzRZyrA\n0d9Bn8bUtEt8tSGjrQ2tWuOXHOwu74VusCqYc42sIzmDvioop6bJ4jfU8+91pVQ2mP1WXa/fXUtG\nks6XRMBbjB/UJ5oP1pT4hoX2VxjJTo2kvtXKhj3e11pqzZjbHVw7uS9vrzh697eE6FAeuXEov3n7\nB9/MqeAgJf93Vy5LPi5ky/6jJ7n7rx+Cqd3Osq+PXl1NH53KZaNT+f07W31JYUhmNM/dO56/frqH\nF97dDhxdpFjT1M7SlfuxOVz0SdJx46WZVDZYWLujih/2NZAQpeXeawfRZOzg0+9L2XOoGYUCbsjL\nJDk2lLe+2Me+cm/8uQPj+dUd/fjftwvYUey9qg/XBvHcveN57bM9fJLvvTJUAA/eOJTapna/rVr6\npUYwbkAC735z0NemDw3ioRuH8vt3tvnath1o5K6rBrJs1X7fepfSmjYykyLYX9GC1ebtBdkcLj5Y\nXczQzBhfEgH4+ocKwrVBviQC3oL8VwXlJ0wiF7qQ405yKqWK0Qkj/NqUCiVT0iYxJW2Sr21W5uW+\nmVFJYfFcnzWLYFUQP9Rtp7nDe7ExKLo/l6Vdys6G3ZQfnsUVpg5lRvqlHGwtob796LBsX30a35V/\n73vc5jDTbGvhsVH3sbZyPVq1lr4RfVAqlH6zs7RqLdmRfSlsLPK1ZejTmJg8jp0Ne3wJIjE0nssz\nplDUfMBXq0gOSyQvdRJ2l4MvDydAXXA4V/aZTkJoHPcM/Z9Ov9fxv8MRESGdT8znYgI5HZJIesjj\nwXeCOsLhdHeqAwBdLsArqzX51RbAWwBuNPq/tslk8xXNj75fO99srfSbfmt3uPlmSyVVx80cWl9Y\n4zdMBLB2RxUul8evNrLb0Mx32yr9VrsfKaB/vv6QL9ZDNW1kxLexrrDG11bXYmVvaQsHKlp9bR4P\nfLO1kqGZ0bTbjv4mBUX1JERpMZqPDjWarQ6+/qHcbzjMA6zZXk2r2f9OdwcqjSiOq2GY2h38sK/z\nyX3voeZOf6OK+ja/hAHQbLJ1+hwAy3G1I6DT30ycWpBSzY+yZ3Xasv3X435OUfN+tGotOZGZKBQK\nHht1P9vrd2F1dTAybhgRITp+OnQOHx78nFpLPUNiB5KbMJKNNVv83qvNbuaDA//2DZdtrt3KmPgR\nFNR5Ly5CVMHM7DOdaE0kHs+HHGg1kKFL5bYBPyY+NJaHRszjh7rtRITomZI6iRB1CD8bfT/7Wg7i\n9rgZFN0flVLFNVkzyU0aTbO1hazIvgEdcjoXBDSRFBcX8+yzz7Jjxw50Oh2zZ8/mwQcfRKXqXHQ+\nVltbG8899xzffPMNbrebKVOm8Ktf/YqoqLO/CGzysESqGnR8tv6Qry1vRDKD+0SzbX+Dr6eSEhvG\n1JEprN1R5asPKBUKpo5MZVdJk1/iGZoZ7ZsRdYQ+NIj4SC0Hj5kBplB4FyMeLzZCgwL8ekm60OBO\nBW2lUoHd6V+Qhq43h6xtau908qxoMHdqazRasXT4n3jbO5ydCt/Q9QSGriYchAQpidKFUFFv9ntd\nalwY+48pbKuUCoZnxfimVR8xIieWovIWv6Q1JDOGxJgw35RhgDED4hjTP57tB48m0fhILZeMSKZg\nXz0tbd4kow8LZsLgRGoaLezsokYiuidYFcTwuCGd2sYljfZrSzxcFD/C4/GQGp7sV2PJ0KX6egoA\nTrcTtVLNgrEP09DeSP/oHMKDvNvo3Df8rk6x9I/Opn+0/6wolVLF4JgBnV6bEBrX5RTii1HAiu1G\no5GbbrqJmJgYnnnmGfr378+SJUuw2WxMmHDyvXDuv/9+du/ezTPPPMOMGTP49NNPWb16NTfeeOMp\nP7e3i+1b9jVwqK6NSUMSyUmNJFofwg/76inYV8+ofnFkp+hJjg2jrqWdFZvKyE6OIDk2jNgIDSlx\nYRQU1ZMWH44uLJjgICX90iKpqGtDqVQSplHTbLIRo9cwIjsWnTaIJlMHVrsLhQImDE4kJkKDud1B\ny+Er6b5JOrKSI9Bo1L5EpA8LJndAAnFRWr8T7/TRqd71IAcafCf1lNgwbj684vzIjCuVUsEdV/Rn\nV0mT3/bxV+SmUd9q9UuCV+SmEx+l9dt65dIRyYwdEE9B0dFhvYwEHVNGJmOoNvl6RHGRGqaMTMHl\n8vhqLSFBKsYPSiA2QkNFnRm7041KqWDCkASidRo67E5azXY0wSpGZMVSWmsiJS4Mq82FWqVkWHYM\nFfVm4iO1hGrUqJVKslIi2F/RSpPRSk5qJFG6EPqlRVLb3E7BvnoGpEWSEhdGeoKO1rYOPt9Qhj40\nmMnDkhjSN4b2Dgcf5xswWjrPnjvifC+2nw8UCgUj4oagUqiI0kRwdeYVJIcnsem4XkrfiAwmJeeS\nHJ543g4dnQvOyWL7a6+9xhtvvMHq1asJD/euOH799ddZsmQJ69ev97Udb/v27dxyyy0sW7aMsWPH\nArBr1y5mz57N22+/zcSJE0/6ub1dbD9CqVBw3/WD+csn/sW6e68ZxD9XHfA72V42JhWrzcn6wqMz\np4ZlxTBhcCKvfXZ0tkqULoSnfzKKhf/cRrPJmyjCtWrmXjmQgqI6Coq8K2uD1Apum9GP1jYbn60/\n5JuRdPWEDGIiNXy81uCbeTUiJ4YB6dHsLm1ij6EZDzAoI4rU+HAsVge7DzVjMtvplxZBtF6Dw+nG\n5nBT19JOQrQWt8uD0eIgWh+CyWInTBtEsFpJe4eTcG0QBypacHtgQEYUmiAVapWC7QcbMbU7yErR\nkxYXjgf4ochb+NYEq5g6MoWIsGC+2VZJY2sHCmD84ARyUiNZtaXCt5NxYrSWm6flsHp7lW+2VJBa\nyfxrB1NoaGLNjqNXplNGJNM/Pcrv90yI0nL31YN47p2j6x4Anrx1JH/5pNDvb3RDXibbDzb4rQvK\nSNTRPy2yU4+nKxdysf1c5vF4WLzjdd8927VqLU+MfoCEsFPfj0ec3MmK7QFbR5Kfn8/kyZP9Esas\nWbPo6OigoKDgpMfFxsb6kgjAsGHDSE1NJT8//4zGfDJuj6fTrWrBOz5/fL2ktNrE9gP+azkKS5r8\nrtjBu6fTqi0VviQCYLY6Kak28kPR0e0ZHE4PhSXNbNztv7vvmh3VlFab/Kbv7jjYhEqlYPfhJAKw\nt6yFpJhQthxowGi248Fb9A/Xeq/eCg1N1LdYKSxpRqFUcNkY73Dcodo29pQ2Y6g2cd3kPmw90ECb\n1Ymlw8nW/Q2MzIlj095636SDkioTwUEqSqqMtB+eqtthd7H1QAN1LVYaW71Dbx5g0546rDan33b4\ntc1WDtWY/KbcOpxuNuyu9ZvEAN7JDccu5ARvDWdDoX+dCWDT3tou17YcO5QGUFFn7rS2RZxbFAoF\nDwy/m3lD7+CW/jfwzPifSxI5CwKWSAwGA5mZmX5tycnJaLVaDAbDCY7q+jiArKyskx53pmlD1EwY\nksjxWweO7h+PLtS/ENcvLbLTvTNiIzXERfrXOxR4axudeOi0yt7mcNFhd3Zq66pQXNvFydBQY8J2\nXB2jrNZEocE/Oe4xNLNtv/8eQ0aL3dc7OtbOkka/oTDve7bRcFytpsnYQXObf5sHfMN1x+pwdK61\nuNwe9Mf9TvrQoE4LQRVAZnLnBYUDMqI6LSTNSNQxqE+0X9vgvtEMzYzhVP7y2ORTvkacOSqlihFx\nQ7gkZbzsznuWBKzYbjKZ0Ok6/5H1ej0mk6mLI059XGVlZRdH+IuKCkWtPnkxvyuf/+E6rvnZv32P\nR/aLYWh2PN9tqSBar+F/rhzIgD7RPOL08N43B3C53Fybl8VlE/qSlKDnb58WUtNoYeKwJO66fijl\ntW38/u0Cmk0d6MOCeeTmUaQn6thX0Up5bRtKBcye3o8bp+WwaW+dbxZWbKSW268aRHVzOzsPF4UV\nCrh+SjaGKiP/+nq/L8aZE/owekA82w42+noqaQk6brysP2t3Vh9T9IdrLsliV0mTX0F69KBEFMp6\n9pcfXdSXlqgjMy2KXcckGKVSwYThKX5DSwCXjk5nl8H/PccMTiQ1Uce3xwwPTRyWzLjBiewsPtrT\nSIgO5SdXDqKgqN63p1WYNohbrhhIg9HmS2YqpYIbp+dgtTl5adlWHE43apWCe64fSk5aFEXlrdQ3\nt6NQwI1Tc/jR9P60tjv5fJ33omPWpL5cnZdNdGQYr32yi1azjdxBidx57RDsDjd/+7SQotImBvSJ\n5t4fDSUiLASlSsna7VXERWn5ycwB/Pzlo1NQr53ch/TUUycbIS4kAauRDB48mCeeeIK5c+f6tefl\n5XH99dfz+OOPd3ncnXfeiVar5ZVXXvFr//nPf05lZSXLly8/6ef+tzWSM8HldlPXbCUuUkvQ4Sti\nj8dDeZ0ZXWiQb2v6DruTLfsacLndjB0QT6gmCJvDxbpdNb4NFvuleW8funlvHQcqWumbpGfi4Z1t\niw41s2lvnfe+JKNT0YcGs+dQM19uKsPl9nD52HRG5MRSUmVk+bcHaWi1MmZAPDdPy6GmycIrn+ym\nvtVKjD6E+dcNIS5Ky6IPdlFaY0KtUnJDXiYzx6Xz5eZyVmwqw+PxcHluOtdM7ENJlZH3viumwWhl\n7IB4bpqajcvt4T8bDnGwopXMlAiundQHTbCaTXu8Q1TRuhCumpBBbISWRqOV/J3em2ldMjyZ+Egt\nDqebTXtraTJ2MKpfHOkJ3gsLo8VOaY2Jvok6IsK9vRGny01JlZEoXYjfvWPsjiM7BRy9qHB7PDic\n7i5njglxsTtZjSRgiWTChAncfvvtPPjgg37tI0aM4MEHH+See+7p8rhHHnmE5uZm3nnnHb/2e++9\nF4C//e1vJ/3ccymRnC88Hg+tZjsRYcEoj1m7UdfSjk4b1OXtfoUQF5ZzstiemZnZqaZRU1OD1Wrt\nsgZy7HGlpZ2n4p6odiJ6TqFQEKUL8UsiAAlRoZJEhBCBSyR5eXmsW7cOs/nozJgVK1ag0WjIzc09\n6XENDQ1s2XJ0rnhhYb543gEAAAc2SURBVCEVFRXk5eWd0ZiFEEJ0FrChLaPRyKxZs8jJyWHevHlU\nVFTw/PPPc8cdd/DYY4/5XjdjxgzGjh3Lc88952u7++67OXToEAsWLECpVPLSSy8RExPDu+++e8rP\nlaEtIYTovnNyaCsiIoKlS5ficrmYP38+ixcvZs6cOTz88MN+r3O5XLjd/vsk/elPfyI3N5enn36a\nJ598kiFDhrBkyZKzGb4QQojDZBt5IYQQp3RO9kiEEEJcGCSRCCGE6BFJJEIIIXpEEokQQogeueiK\n7UIIIXqX9EiEEEL0iCQSIYQQPSKJRAghRI9IIhFCCNEjkkiEEEL0iCQSIYQQPSKJRAghRI9IIrlI\nLV68mHHjxnXrmF/84hfccMMNvscff/wx/fv3x2Kx9HZ4/7+9+wtpqg8DOP7dm1P7Ky2XomZGwpI5\n/5V/J1pqEEjlTWqhZEKgRkII7SZrKIF6kZVbWVBmGiqSEEiSVBr9w+jCpIsgI4kwIWQiu8i18L2Q\nxju10ncrNZ8PCJ6d5zznOfLDZ7/fOTrxl5qcnKSjo4MDBw4QHR1NTEwMeXl5PHjwYN653r9/T11d\nHePj406vTx+XHz9+RKPR0NPT45ZrEDNJIxH/286dO2lra2PlypULXYpYIoxGI6dOnSIyMhKTyURt\nbS2BgYGUlJT88mOypxsaGsJkMs1oJNNt3LiRtrY2tm/f7krp4ic8FroAsXSpVCpUKpXLeb58+YK3\nt7cbKhKL2f3792ltbcVoNHLw4EHH66mpqfj6+lJbW4ter0er1br1vJ6enkRFRbk1588sx/EsMxJB\nX18fGo2Gvr4+SktLiY6OJj09nVu3bv30uNmWtiYmJqipqSE1NZXw8HD27dvHo0ePnI5LS0ujqqoK\ns9lMSkqKvFNcJhobG9m8eTPZ2dkz9hUVFbF69Wqam5uBqTFSXV3tFPPf8dbX10dRUREA6enpaDQa\n0tLSZj3vbEtb3/PfuHGDlJQUYmNjOXHixIzZzdjYGOXl5SQlJaHT6cjNzeXVq1dOMRqNhoaGBs6e\nPUtCQgJ79+6d/w9niZMZiXAoLy8nKyuLnJwcOjs7qaioQKfTERERMeccpaWlDAwMcPz4cYKDg+nq\n6qK4uJjbt28TFhbmiOvs7CQ0NJQzZ87w7du333E5YhGx2+309/dz6NAhVqxYMWP/2rVriY+P5+XL\nl3PKp9VqMRgMVFdXYzKZUKvVeHp6zqumrq4uNBoNlZWVjIyMUFVVxblz5zAajQDYbDaOHDnC+Pg4\nJ0+eRKVS0dLSQkFBAd3d3ajVakeua9eusWPHDmpqaliO/75QGolwyMzMpKSkBIC4uDh6enro7u6e\ncyN5/vw5vb29NDU1ERcXB0BycjJDQ0NcvnyZixcvOsVfuXIFLy8v916EWJQsFgs2m42AgIAfxgQE\nBPD48eM55VuzZg1btmwBICwsjKCgoHnX5OHhgdlsxsNj6tfg4OAgd+/edTSSO3fu8PbtWzo7OwkJ\nCQEgKSmJPXv2cP36dQwGgyOXWq3m/Pnz867hbyFLW8JBr9c7vlcqlYSEhDAyMjLn4589e4ZarSYm\nJga73e74SkxM5PXr106xCQkJ0kTEgoqPj3c0EYDQ0FBGR0f5+vUrMPXGSKvVEhQU5BjLALGxsTPG\nc0pKyp8rfBGSGYlwWLdundO2UqnEZrPN+XiLxcLnz59nvVk6fTnD19f3/xUplqT169fj6enJ8PDw\nD2OGh4fx8/P7YzXNNt4nJyex2WwolUosFgv9/f2zjufg4GCn7eU+nqWRCLfx8fHBz88Ps9n8y1iF\nQvEHKhKLhYeHB1FRUfT29mIwGPjnH+fFEKvVyosXL8jIyACmnrT6PjP47leP+bqbj48P4eHhjqWu\n/5p+P2a5j2dpJMJtEhMTaWhoYNWqVWzdunWhyxGLzOHDhzl27Bjt7e3k5OQ47bt69SpWq5W8vDwA\n/P39effunVPMkydPnLaVSiUw9aTg75CYmMjTp08JCAhgw4YNv+UcfwtpJMJt9Ho9ycnJFBYWcvTo\nUUJDQ7Farbx584aJiQnKysoWukSxgDIyMsjNzaWiooLBwUF27dqF3W6nq6uLjo4OysrKHMtIu3fv\nprKykvr6enQ6Hffu3WNwcNAp3/eb7W1tbWRmZuLt7Y1Go3FbvVlZWbS2tpKfn09hYSGbNm1ibGyM\ngYEB1Go1BQUFbjvXUieNRLiNQqHAZDJRX19PY2Mjnz59wsfHh23btpGfn7/Q5YlFwGg0EhkZSUtL\nC+3t7SgUCrRaLZcuXSI9Pd0Rl52dzYcPH2hqasJms7F//36Ki4s5ffq0IyYwMBCDwUBTUxPNzc34\n+/vz8OFDt9Xq5eXFzZs3uXDhAnV1dYyOjqJSqYiIiPjh36wsV/KZ7UIIIVwij/8KIYRwiTQSIYQQ\nLpFGIoQQwiXSSIQQQrhEGokQQgiXSCMRQgjhEmkkQgghXCKNRAghhEv+BZ9+3yRiIoMzAAAAAElF\nTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "g = sns.swarmplot(data=df, x='y', y='score')\n",
    "g.set_xticklabels(['Inlier', 'Outliner'])\n",
    "g.set_xlabel('')\n",
    "pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluate model "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "precision, recall, thresholds = precision_recall_curve(y_test, y_score)\n",
    "fpr, tpr, thresholds = roc_curve(y_test, y_score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plot ROC curve\n",
    "\n",
    "While this looks reasoably good, it is misleading because the data set is unbalanced. Why is ROC misleading when there is unbalanced data?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEnCAYAAACDhcU8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XtYVOW+B/DvOIhAMIhKbJG0ABFv\nsBUBLzSmbrQyRWt7qwws6aKRYCVHPbWLTEw7GjqiXfCuG7Q62+NWy9KSYxy1jpnmVgPhETVUBGQY\n5SIz6/zRwxzHgQFmzZrFMN/P8/g88c56Z35v2Hx717vWuxSCIAggIiKyUge5CyAiIsfGICEiIlEY\nJEREJAqDhIiIRGGQEBGRKAwSIiIShUFCRESiuMhdAJGtHTt2DM8995xJm4eHBx588EHExcXh2Wef\nhYtL43/1f/zxR2zduhUnTpzAzZs34eXlhQEDBmDatGn4y1/+0uRnFhUVYfPmzTh69CiuXr0Kg8GA\n7t27IyoqClOmTEFYWJhNx0jUlih4QyK1Nw1B8sQTT0CtVkMQBNy4cQO7d+/Gb7/9hqlTp+K9994z\n67dy5Up8/PHH6NGjByZOnIiAgADcuHED//znP5Gfn4+4uDikp6dDqVSa9Nu1axfeffdduLq64okn\nnkBoaChcXFxQVFSEAwcO4PLly9i7dy+Cg4Pt9a+AyL4Eonbm6NGjQkhIiPDZZ5+ZtN+6dUtQq9VC\nnz59hLKyMpPXdu7cKYSEhAgJCQnC7du3TV67c+eOsGDBAiEkJET46KOPTF774YcfhNDQUOGJJ54Q\nrl69albLnTt3hI0bNwr5+fk2Gp04BoNB0Ol0cpdB7QzXSMhpeHh4IDw8HIIgoLi42NheV1eHjIwM\neHh44MMPP4S7u7tJPxcXF6SlpcHf3x8bNmxAeXm58bUPP/wQgiBg1apV8PPzM/tMFxcXJCQktGg2\notPpsGrVKjz22GMYOHAgoqOjMWPGDOzdu9d4zMyZMzF69GizvpcvX0afPn2wZs0aY9uxY8fQp08f\nfPnll9i+fTsef/xxDBw4EBs2bEBycjIGDBiAiooKs/cqLCxEnz598P7775u079u3DzNmzMCgQYMQ\nHh6OKVOm4Kuvvmp2XNT+MUjIqVy6dAkA4O3tbWw7ceIESktLMWbMGHTt2rXRfp06dcLEiRNRU1OD\nw4cPG9/rzJkziIiIEH3aSqvVYvr06Vi/fj169+6NN998E6+88goeeOABfPfdd6Lee/Pmzfjkk0/w\n+OOP46233kJYWBgmT56MO3fumIRUg927dwMAJk+ebGxbtWoVUlJScN9992HevHl4/fXX4ebmhnnz\n5mH79u2i6iPHx8V2areqq6uNs4fS0lJkZ2fjX//6F8LCwvDQQw8Zj8vPzwcA9O/f3+L7Nbz+22+/\nmfTr27ev6FpXrlyJ/Px8pKWlYdq0aSavGQwGUe9dUlKC/fv3m4SkXq+Hr68v/vGPf+DZZ581tguC\ngP/6r/9CSEgI+vXrBwA4c+YM1q9fj5deegnz5883Hvvcc89hzpw5+I//+A/ExcXB09NTVJ3kuDgj\noXZrzZo1GDZsGIYNG4aJEydix44dGDt2LDIzM02O0+l0ANDsF2HD61VVVSb97rvvPlF1GgwG7Nu3\nD0FBQWYhAgAdOoj7zzQuLs5spqVUKjFhwgScPn0aFy5cMLYfO3YMv//+u8lsZM+ePVAoFJg0aRLK\ny8tN/owePRq3bt3CyZMnRdVIjo0zEmq3pk2bhkcffRR37tzBb7/9hs8++wxXr15Fp06dTI5rCIiG\nYGhKw+teXl4m/W7duiWqzoqKClRWVuLhhx8W9T5NefDBBxttnzRpEjZs2IDdu3cbZxq7d+82hkyD\nCxcuQBAEPPbYY01+xo0bN2xaMzkWBgm1W7169cLw4cMBACNHjkRERASefvpp/O1vf8OqVauMx/Xu\n3RvAH6dwLGl4PSQkxKTf2bNnbV57a+n1+iZfu/figQZ9+vRB3759sWfPHqSkpKCmpgZff/01RowY\nAV9fX+NxgiBAoVDg008/Nbv0uQEvbXZuDBJyGoMHD0ZcXBz+8Y9/YObMmRg8eLCxvVu3bjh48CDK\ny8vRpUsXs761tbXYs2cPOnXqBLVaDQB44IEH0K9fP5w4cQIXLlxAUFCQVXX5+PjA29sb586da/bY\nzp07Nxp4DRcRtNakSZOQnp6Oo0ePorS0FLdu3TI5rQX8MaP57//+b/j7+1s9RmrfuEZCTmXOnDlQ\nKpVYvXq1sc3V1RWvvfYabt++jTfffBM1NTUmffR6Pd555x1cuXIFL7zwgsl6wxtvvAEAmD9/PkpL\nS80+T6/XY9OmTSgoKGiypg4dOmD8+PEoKCjArl27zF4X7rpn+MEHH8StW7dw6tQpY5vBYMCmTZua\nH3wjJkyYABcXF+zevRu7d++Gl5cXxowZY3LMxIkTAfxxQUBjMx+e1iLOSMip9OrVC48//jj27NmD\nn376CUOGDAHwx3rKxYsXkZWVhccffxyTJk1Cjx49jHe2//bbb5g4cSJeffVVk/cbMWIE0tLS8O67\n7+LRRx/F+PHj0bdvX7i4uODixYs4cOAAiouL8c9//tNiXcnJyTh69Cj+/d//HT/88AMiIiIgCALO\nnj2L+vp6rFixAgAwdepUbNy4EXPnzsVzzz2Hjh074uuvv7Z4asuSrl274uGHH8bXX3+N2tpa/PWv\nfzVbQwoLC0NSUhLWrFmDSZMmYdy4cfDz88P169dx5swZ5Obm4tdff7Xq86l9YJCQ03nllVewd+9e\nZGRkYOvWrcb2BQsWYOTIkdi2bRt27tyJmzdvwtPTEwMGDMBrr72G2NjYRt9vypQpiIiIMO61tXv3\nbhgMBvj7+2Po0KH46KOPml1D8Pb2Rk5ODtavX49vvvkG3377Le677z4EBQWZXJ77wAMPYO3atVi5\nciUyMjLQuXNnxMXF4amnnrK4GG7J5MmTjfeqxMXFNXrMq6++igEDBmDr1q3YsmULbt++ja5du6J3\n795YvHixVZ9L7Qf32iIiIlG4RkJERKIwSIiISBQGCRERicIgISIiUZzuqq3S0iqr+/r4eKCi4rYN\nq2n7OGbnwDE7BzFj9vX1avI1zkhawcWl8e0h2jOO2TlwzM5BqjEzSIiISBQGCRERicIgISIiUWQN\nkosXL+Ltt9/GhAkT0LdvX8ycObNF/aqqqrBw4UJERkYiIiICr7/+eqPPniYiIunJGiT5+fk4fPgw\nHnrooSYfvtOY5ORkHDt2DEuWLMGyZcvw66+/Yu7cudIVSkRETZL18t/Ro0fjL3/5CwDgtddea9Gs\n4ueff8aRI0ewbds2REZGAgD8/PwwZcoU5OXlGR9kRERE9iHrjMSaZ1Hn5uaiW7duxhAB/tjmOiAg\nALm5ubYsj4iIWsDhbkgsLCxEYGCgWXtQUBAKCwtlqIjIee08VIAfz12XuwyrKJUK6PXOtfm5enAA\nJgztafP3dbgg0Wq18PIyv8NSpVLh8uXLzfb38fEQdVOOpbs72yuO2TlYM+YT+aWo0NWim7ebBBVJ\nT6lUyF2C3Unxd9vhgkQsMVsi+Pp6idpixRFxzM7B2jHr9QJ8PDth2UvDJKhKWvw9t75vUxzuPhKV\nSoWqKvN/EVqtFiqVSoaKiIicm8MFSWBgIIqKiszam1o7ISIiaTncqS21Wo3MzEz89NNPGDJkCADg\n9OnTuHTpEtRqtczVkSO6d8HYGRdhrR1zRVUtfLw6SVARORJZg6S6uhqHDx8GAFy7dg06nQ5fffUV\nAGDkyJFwd3dHbGwsIiMjsXTpUgDAoEGDEBMTg9TUVKSmpqJDhw5YsWIFIiIieA8JWeXHc9f5hWgl\nH69OiAy9X+4ySGayBklZWRnmzZtn0tbw88GDBxEQEAC9Xg+DwWByzKpVq5Ceno5FixbBYDBg1KhR\nWLx4sd3qpvbHx6sTVsz5439EuAhL1DqyBklAQADOnz9v8ZhDhw6ZtalUKqSnpyM9PV2q0oiIqIUc\nbo2E2j973+TG01pE4jjcVVvU/jWsWdgLz/MTicMZCbVJd69ZEFHbxhkJERGJwiAhIiJReGqL7K65\nxXQufhM5Fs5IyO6aW0zn4jeRY+GMhGTBxXSi9oMzEiIiEoVBQkREojBIiIhIFAYJERGJwiAhIiJR\nGCRERCQKL/8lm2nprr284ZCofeGMhGympbv28oZDovaFMxKyKd5oSOR8OCMhIiJRGCRERCQKT22R\naA2L7FxEJ3JOnJGQaHeHCBfRiZwPZyRkE1xkJ3JenJEQEZEonJGQ1bg2QkQAZyQkAtdGiAjgjIRE\n4toIEXFGQkREojBIiIhIFFmDpKCgAPHx8QgPD0dMTAwyMjKg1+ub7Xf69Gk8//zziIqKQlRUFBIS\nEvDLL7/YoWIiIrqXbEFSWVmJhIQEKBQKZGZmYu7cudi4cSNWr15tsV9JSQlmzZqF+vp6LF++HMuX\nL4der8esWbNw5coVO1VPREQNZFtsz87ORm1tLTQaDTw9PTFixAjodDpoNBokJibC09Oz0X7ff/89\nbt26hbVr18LLywsAMGjQIAwdOhSHDx/G008/bc9hEBE5PdlmJLm5uYiJiTEJjPHjx6OmpgbHjx9v\nsl99fT2USiXc3d2NbR4eHlAqlRAEQdKaiYjInGxBUlhYiMDAQJM2f39/uLu7o7CwsMl+Y8eOhbu7\nO5YtW4aysjKUlZUhPT0d3t7eeOyxx6Qum4iI7iHbqS2tVms8NXU3lUoFrVbbZD8/Pz9s2bIFL730\nErZu3QoA8PX1RVZWFrp06dLs5/r4eMDFRWl13b6+5jW3d02NWalUWHzdkbXHMTWHY3YOUozZ4W5I\nvH79OubNm4f+/ftjyZIlAIAdO3bgxRdfRHZ2Nvz9/S32r6i4bfVn+/p6obS0yur+jsjSmPX6P04l\ntrd/J/w9OweOufV9myLbqS2VSgWdTmfWrtVqoVKpmuyXlZWF+vp6rF69Gmq1Gmq1GqtXr4ZSqcSG\nDRukLJmIiBohW5AEBgaarYWUlJSgurrabO3kboWFhQgODkbHjh2Nba6urggODkZxcbFk9RIRUeNk\nO7WlVquRlZUFnU5nvHJr3759cHNzQ1RUVJP9/P39kZubi7q6Ori6ugIA6urqkJ+fj1GjRtml9vas\nYUffBkqlwngK617c9ZeIABlnJNOnT4erqyuSkpKQl5eHnJwcaDQaJCQkmFwSHBsbi0WLFhl/njJl\nCq5fv45XX30V33//Pb777jvMmTMHpaWlmDZtmhxDaVcadvRtCe76S0SAjDMSb29vbNq0CWlpaXj5\n5ZehUqkQHx+PpKQkk+P0ej0MBoPx5wEDBuCzzz6DRqPBggULAAAhISHYsGEDQkND7TqG9uruHX2d\ncUGSiFpH1qu2goODsWXLFovHHDp0yKxt2LBhGDZsmFRlERFRK3D3XyIiEsXh7iMhce5dTL8XF9CJ\nqLU4I3EyzS2mcwGdiFqLMxInxMfjEpEtcUZCRESicEbSjjS3/gFwDYSIbI8zknakJTcTcg2EiGyN\nM5J2husfRGRvnJEQEZEoDBIiIhKFp7bagYZFdi6kE5EcOCNpB+4OES6kE5G9cUbSTnCRnYjkwhkJ\nERGJwiAhIiJRGCRERCQKg4SIiERhkBARkSgMEiIiEoVB4uB2HipAmbZG7jKIyIkxSBxcw7bxvBGR\niOTCIGkHuqrcMHV0sNxlEJGTYpAQEZEoDBIiIhKFe205KO74S0RtBWckDoo7/hJRW8EZiQPjjr9E\n1BbIOiMpKChAfHw8wsPDERMTg4yMDOj1+hb1PXDgAJ566imEhYUhOjoaL7zwAm7fvi1xxUREdC/Z\nZiSVlZVISEhAcHAwMjMzUVxcjA8++AAGgwEpKSkW++7atQtpaWmYPXs2FixYAK1Wi6NHj7Y4hIiI\nyHZkC5Ls7GzU1tZCo9HA09MTI0aMgE6ng0ajQWJiIjw9PRvtV15ejqVLl+Ktt97C1KlTje2xsbH2\nKp2IiO4i26mt3NxcxMTEmATG+PHjUVNTg+PHjzfZb//+/QCASZMmSV4jERE1T7YgKSwsRGBgoEmb\nv78/3N3dUVhY2GS/U6dO4aGHHsLnn38OtVqN/v37Y8qUKThx4oTUJRMRUSNkO7Wl1Wrh5eVl1q5S\nqaDVapvsd+PGDRQVFWHdunV488030blzZ3z22WeYPXs2Dhw4gG7duln8XB8fD7i4KK2u29fXvGY5\nKJUKAPapp62M2Z44ZufAMduGw13+KwgCbt++jYyMDKjVagDA4MGDMWrUKGzbtg3JyckW+1dUWH9l\nl6+vF0pLq6zub0t6vQAAktfTlsZsLxyzc+CYW9+3KbKd2lKpVNDpdGbtWq0WKpXKYj+FQoHo6Ghj\nm6enJ/r3748LFy5IUisRETVNtiAJDAw0WwspKSlBdXW12drJ3YKCgiAIAgRBMGkXBAEKhUKSWomI\nqGk2DZJ7v9wtUavVOHLkiMmsZN++fXBzc0NUVFST/R555BEAwLFjx4xtVVVVOHPmDEJDQ1tfNBER\niWKTIKmrq0NOTg4effTRFveZPn06XF1dkZSUhLy8POTk5ECj0SAhIcHkkuDY2FgsWrTI+PPAgQMx\nZswYLF68GP/5n/+J77//Hq+88gpcXFzwzDPP2GI4RETUCs0uttfV1eHQoUMoLi6Gt7c3HnnkEfj5\n+QEAqqursW3bNmzevBk3btxAr169WvzB3t7e2LRpE9LS0vDyyy9DpVIhPj4eSUlJJsfp9XoYDAaT\nthUrVmD58uVYtmwZqqurMXjwYGzevBne3t4t/nxH0rDT79246y8RtRUKwcL5qKtXr2LmzJm4fPmy\n8bSVm5sbMjMz0bFjR7zxxhu4du0aIiIiMGvWLIwZM6bNr1OIuUpDrqs83szMazQ4IkPvl/zJiLyy\nxTlwzM5Bqqu2LM5IVq5ciZKSEiQmJiIiIgKXL1/GunXr8NZbb+HmzZsYOHAgVq9ejfDwcKsKo5bj\nTr9E1FZZDJK8vDw89dRTmD9/vrHNz88Pr776KsaNG4eMjAzJCyQiorbN4mJ7eXk5wsLCTNoGDhwI\nAJgwYYJ0VZHRzkMFKNPWyF0GEVGTLAaJwWCAq6urSVvDz/fdd590VZFRwyI7n4JIRG1Vs1dt/etf\n/4KHh4fx51u3bkGhUODkyZONPkhqzJgxtq2Q0FXlJvmiOhGRtZoNko0bN2Ljxo1m7Y2tjygUCpw9\ne9Y2lRERkUOwGCRbtmyxVx1EROSgLAaJpa1KSDp334DIGw+JqK1r9tRWbW0tvv32W1y5cgWdO3fG\nyJEjjXe2kzR+PHfdGCA+Xp240E5EbZrFICktLcWMGTNw5coVkzvb16xZg4cfftguBTor3oBIRI7C\n4uW/a9euxe+//45Zs2bh448/xuLFi+Hu7o6//e1v9qqPiIjaOIszkv/5n//Bk08+iQULFhjbunXr\nhvnz5+PChQsICgqSvEBncO+mjFwXISJHYnFGcuXKFfz5z382aRs8eDAEQUBZWZmkhTmThjWRBlwX\nISJHYnFGUl9fj06dTP/PuOHOdr1eL11VTohrIkTkqHhnOxERicI724mISBTe2U5ERKJYDJKAgAB0\n6dIFbm5u9qqHiIgcjMWrtsaMGYNvvvnGXrUQEZEDshgkFh7nTkREBKCZICEiImpOs0GiUCjsUQcR\nETmoZi//Xbp0KVatWtWiN1MoFPj2229FF0VERI6j2SDp3r07/vSnP9mjFiIickDNBklCQgImTJhg\nj1qIiMgBcbGdiIhEYZAQEZEosgZJQUEB4uPjER4ejpiYGGRkZLRqV2GDwYAnn3wSffr0wXfffSdh\npURE1BSLayTnzp2T7IMrKyuRkJCA4OBgZGZmori4GB988AEMBgNSUlJa9B67du3CtWvXJKuRiIia\nJ9uMJDs7G7W1tdBoNBgxYgRmzJiBuXPnYtOmTdDpdM32r6ysxKpVq5CcnGyHaomIqCmyBUlubi5i\nYmLg6elpbBs/fjxqampw/PjxZvtnZGRg8ODBGDZsmJRlEhFRM2QLksLCQgQGBpq0+fv7w93dHYWF\nhRb7njt3Dl988QVSU1OlLJGIiFpAtiDRarXw8vIya1epVNBqtRb7LlmyBM888wx69eolVXlERNRC\nzd6Q2Nbs3bsXRUVFWL9+vVX9fXw84OKitPrzfX3Nw08spVIh2XvbQlutS0ocs3PgmG1DtiBRqVSN\nLqprtVqoVKpG+9y5cwfLly9HYmIiDAYDtFqt8T2qq6uh0+lM1lwaU1Fh/pz5lvL19UJpaZXV/Zui\n1/+xXb8U7y2WVGNuyzhm58Axt75vU2QLksDAQLO1kJKSElRXV5utnTSorq7G1atXkZ6ejvT0dJPX\nUlJS0LNnTz6Ii4jIzmQLErVajaysLJNZxL59++Dm5oaoqKhG+3h4eJg9R/7GjRuYP38+5s+fj6FD\nh0peNxERmZItSKZPn46tW7ciKSkJiYmJuHTpEjQaDRISEkxOT8XGxiIyMhJLly6Fi4sLoqOjTd7n\n8uXLAICQkBCEh4fbdQxERCTjVVve3t7YtGkT9Ho9Xn75ZaxZswbx8fF47bXXTI7T6/UwGAwyVUlE\nRM2R9aqt4OBgs1NV9zp06JDF1wMCAnD+/HlblmUTOw8V4Mdz11t0bEVVLXy8OklcERGRNLj7r0R+\nPHcdFVW1LTrWx6sTIkPvl7giIiJpONx9JI7Ex6sTVswZLncZRESS4oyEiIhEYZAQEZEoPLVlI/cu\nrnMBnYicBWckNnLv4joX0InIWXBGYkNcXCciZ8QZCRERicIgsYGdhwpQpq2RuwwiIlkwSGygYZGd\nayJE5IwYJDbSVeWGqaOD5S6DiMjuGCRERCQKg4SIiERhkBARkSgMEiIiEoVBQkREojBIiIhIFG6R\nYqW7N2nkBo1E5Mw4I7HS3Zs0coNGInJmnJGIwE0aiYg4IyEiIpEYJEREJAqDhIiIRGGQEBGRKAwS\nIiIShUFCRESiMEiIiEgUBgkREYkia5AUFBQgPj4e4eHhiImJQUZGBvR6vcU+p06dwsKFCxEbG4vw\n8HCMGzcOGo0GtbW1dqqaiIjuJtud7ZWVlUhISEBwcDAyMzNRXFyMDz74AAaDASkpKU32279/P4qL\ni5GYmIhevXrh/PnzyMjIwPnz57FmzRo7joCIiAAZgyQ7Oxu1tbXQaDTw9PTEiBEjoNPpoNFokJiY\nCE9Pz0b7JSYmokuXLsafo6Oj0alTJ7z99tu4cuUKevToYa8hEBERZDy1lZubi5iYGJPAGD9+PGpq\nanD8+PEm+90dIg369u0LALh+/brtC23EzkMFKNPW2OWziIjaOtmCpLCwEIGBgSZt/v7+cHd3R2Fh\nYave6+TJk+jQoQN69uxpyxKb1LB9PHf8JSKS8dSWVquFl5eXWbtKpYJWq23x+5SWlmLdunWIi4tD\n165dmz3ex8cDLi7KVtV6N19fLyiVCtzv44650wZZ/T6OxNfX/PfU3nHMzoFjtg2H3ka+rq4OycnJ\n8PDwwMKFC1vUp6LittWf5+vrhdLSKuj1AgCgtLTK6vdyFA1jdiYcs3PgmFvftymyBYlKpYJOpzNr\n12q1UKlUzfYXBAGpqakoKCjAjh074O3tLUWZRETUDNmCJDAw0GwtpKSkBNXV1WZrJ415//33cfDg\nQWzYsAFBQUFSlUlERM2QbbFdrVbjyJEjJrOSffv2wc3NDVFRURb7fvzxx9i+fTtWrFiBIUOGSF0q\nERFZIFuQTJ8+Ha6urkhKSkJeXh5ycnKg0WiQkJBgcklwbGwsFi1aZPx5z549WLlyJSZNmgQ/Pz+c\nPHnS+Ke8vFyOoRAROTXZTm15e3tj06ZNSEtLw8svvwyVSoX4+HgkJSWZHKfX62EwGIw///DDDwCA\nL7/8El9++aXJsenp6XjyySelL56IiIwUgiAIchdhT2Ku0vD19cLanJ/x1fFidFW5YcWc4TasrG3i\nlS3OgWN2DlJdtcXdf1uJNyMSEZlikFihq8oNU0cHy10GEVGbwCAhIiJRGCRERCQKg4SIiERhkBAR\nkSgMEiIiEoVBQkREojBIiIhIFAYJERGJwiAhIiJRGCRERCQKg4SIiERhkBARkSgMEiIiEoVBQkRE\nojBIiIhIFAYJERGJItsz2x3NzkMFOJFfioqqWvh4dZK7HCKiNoNB0kI/nruOCt0fIcLH7BIR/T8G\nSSt083bDspeGyV0GEVGbwjUSIiIShUFCRESiMEiIiEgUBgkREYnCICEiIlEYJEREJIqsQVJQUID4\n+HiEh4cjJiYGGRkZ0Ov1zfarqqrCwoULERkZiYiICLz++uuoqKiwQ8VERHQv2e4jqaysREJCAoKD\ng5GZmYni4mJ88MEHMBgMSElJsdg3OTkZRUVFWLJkCTp06IAPP/wQc+fOxY4dOySrNzL0frh7uEr2\n/kREjkq2IMnOzkZtbS00Gg08PT0xYsQI6HQ6aDQaJCYmwtPTs9F+P//8M44cOYJt27YhMjISAODn\n54cpU6YgLy8Pw4cPl6TeqaOD4evrhdLSKknen4jIUcl2ais3NxcxMTEmgTF+/HjU1NTg+PHjFvt1\n69bNGCIAEBYWhoCAAOTm5kpaMxERmZMtSAoLCxEYGGjS5u/vD3d3dxQWFraqHwAEBQVZ7EdERNKQ\nLUi0Wi28vLzM2lUqFbRarc37ERGRNJxu00YfHw+4uCit7u/rax5i7R3H7Bw4ZucgxZhlCxKVSgWd\nTmfWrtVqoVKpLPYrLy9vdb8GFRW3W1foXZxxsZ1jdg4cs3MQM2ZLASTbqa3AwECzNY2SkhJUV1c3\nugZyd7+ioiKz9qbWToiISFqyBYlarcaRI0dMZiX79u2Dm5sboqKiLPYrLS3FTz/9ZGw7ffo0Ll26\nBLVaLWnNRERkTrYgmT59OlxdXZGUlIS8vDzk5ORAo9EgISHB5JLg2NhYLFq0yPjzoEGDEBMTg9TU\nVBw4cADffvst3njjDUREREh2DwkRETVNIQiCINeHFxQUIC0tDSdPnoRKpcJf//pXJCUlQan8/8Xw\n0aNHIyoqCsuWLTO2abVapKen45tvvoHBYMCoUaOwePFidOnSRY5hEBE5NVmDhIiIHB93/yUiIlEY\nJEREJAqDhIiIRGGQEBGRKAzUW5oHAAAJJElEQVQSIiIShUFCRESiMEjgnI/8tWbMp06dwsKFCxEb\nG4vw8HCMGzcOGo0GtbW1dqpaHGt/zw0MBgOefPJJ9OnTB999952EldqOmDEfOHAATz31FMLCwhAd\nHY0XXngBt29bv1edvVg75tOnT+P5559HVFQUoqKikJCQgF9++cUOFYtz8eJFvP3225gwYQL69u2L\nmTNntqifLb+/nG7333s52iN/bcHaMe/fvx/FxcVITExEr169cP78eWRkZOD8+fNYs2aNHUfQemJ+\nzw127dqFa9euSVyp7YgZ865du5CWlobZs2djwYIF0Gq1OHr0aKuCVw7WjrmkpASzZs1Cv379sHz5\ncgBAVlYWZs2ahT179qBHjx72GkKr5efn4/DhwwgPD0d9fX2L+9n0+0twcuvXrxeGDBkiVFVVGds+\n+eQTISwszKTtXidOnBBCQkKE48ePG9t++eUXISQkRPjhhx8krVksa8dcVlZm1padnS2EhIQIly9f\nlqRWW7F2zA1u3rwpREdHCzt37hRCQkKEQ4cOSVmuTYj5Pf/5z38WcnJy7FGmTVk75h07dgihoaGC\nVqs1tt28eVMIDQ0Vtm/fLmnNYun1euM/JyUlCc8++2yzfWz9/eX0p7ac8ZG/1o65sS1o+vbtCwC4\nfv267Qu1IWvH3CAjIwODBw/GsGHDpCzTpqwd8/79+wEAkyZNkrxGW7N2zPX19VAqlXB3dze2eXh4\nQKlUQmjjm3906ND6r3Fbf385fZA44yN/rR1zY06ePIkOHTqgZ8+etizR5sSM+dy5c/jiiy+Qmpoq\nZYk2Z+2YT506hYceegiff/451Go1+vfvjylTpuDEiRNSlyyatWMeO3Ys3N3dsWzZMpSVlaGsrAzp\n6enw9vbGY489JnXZdmfr7y+nDxJnfOSvrWovLS3FunXrEBcXh65du9qyRJsTM+YlS5bgmWeeQa9e\nvaQqTxLWjvnGjRsoKirCunXr8MYbb2DdunVwd3fH7NmzcePGDSlLFs3aMfv5+WHLli04cOAAhg8f\njuHDh+PAgQPIyspql5vB2vr7y+mDhKxTV1eH5ORkeHh4YOHChXKXI5m9e/eiqKgIc+bMkbsUuxEE\nAbdv38b777+PiRMnQq1WIzMzE0qlEtu2bZO7PElcv34d8+bNQ//+/fHpp5/i008/xYABA/Diiy/i\n999/l7u8Ns/pg0TMI3+rqswfWdnSR/7KydoxNxAEAampqSgoKMAnn3wCb29vKcq0KWvGfOfOHSxf\nvhyJiYkwGAzQarXG96iurm70/doSMX+3FQoFoqOjjW2enp7o378/Lly4IEmttmLtmLOyslBfX4/V\nq1dDrVZDrVZj9erVUCqV2LBhg5Qly8LW319OHyTO+Mhfa8fc4P3338fBgwexdu1aBAUFSVWmTVkz\n5urqaly9ehXp6emIjIxEZGQk4uLiAAApKSmYPHmy5HWLYe3vOSgoCIIgmC0yC4IAhUIhSa22Yu2Y\nCwsLERwcjI4dOxrbXF1dERwcjOLiYsnqlYutv7+cPkic8ZG/1o4ZAD7++GNs374dK1aswJAhQ6Qu\n1WasGbOHhwe2bNli8mflypUAgPnz5+PDDz+0S+3Wsvb3/MgjjwAAjh07ZmyrqqrCmTNnEBoaKlm9\ntmDtmP39/ZGfn4+6ujpjW11dHfLz89v0PSTWsvX3l/Kdd955x4b1OZzevXsjJycHx44dw/3334+8\nvDysXLkS8fHxGDlypPG42NhYnDt3DmPGjAEAdO/eHSdPnsTnn3+O7t27o6ioCO+88w6CgoKQnJws\n13BaxNox79mzB++++y4mT56M6OhoXL161fjH1dXV5NLJtsaaMXfo0AEBAQEmfxrCJT4+HkOHDpVx\nRM2z9vfs5+eHs2fP4u9//zt8fHxw7do1vPfee7h58yaWL18ONzc3uYbULGvH7Ovri82bN+PXX3+F\nl5cXioqKsGzZMpw/fx5paWno1q2bXENqVnV1NQ4ePIiCggIcOXIElZWV6Nq1KwoKCtCjRw907NhR\n+u+vVt950g7l5+cLM2fOFAYOHCiMGDFCWLVqlVBfX29yzKhRo4TU1FSTtsrKSuHf/u3fhIiICGHQ\noEHC/PnzG71pry2yZsypqalCSEhIo3+++OILew+h1az9Pd/t0qVLDnNDoiBYP2adTie8/fbbQlRU\nlDBw4EAhPj5eOHfunD1Lt5q1Y87LyxOefvppITIyUoiMjBSeeeYZ4ejRo/Ys3SoNfycb+3Pp0iVB\nEKT//uKjdomISBSnXyMhIiJxGCRERCQKg4SIiERhkBARkSgMEiIiEoVBQkREojBIiIhIFKd/1C6R\nFL788ssmd0V+/fXX8eKLL6JPnz7GNoVCgfvvvx/9+vVDUlIS+vfvb3zt7uOAPzbcCwsLwyuvvOJQ\n29RQ+8UgIZJQSkoKunfvbtLWr18/4z/HxMRg4sSJEAQBRUVF2L59O6ZNm4adO3c2edzFixexfft2\nzJo1C7t27Wrz+19R+8cgIZLQyJEjjY8jbkxgYKBxR2EAGDRoEF566SX8/e9/x3vvvdfkcYMHD8bs\n2bORnZ0NJ98uj9oArpEQtSENO9ReuXLF4nEREREAgEuXLkleE1FzOCMhkpBWq0V5ebnxZ4VCAR8f\nnyaPbwiGzp07W3zfhqf2tfWHqJFzYJAQSei5554z+dnDwwM///yz8efa2lqUl5cb10iWLl0KABg7\ndqxJv7uPu3jxIpYtWwYAGDdunMQjIGoeg4RIQu+++y569uxp/FmpVJq8npOTg5ycHOPPHh4eSElJ\nwaOPPtrscampqWbHEcmBQUIkofDwcIuL7WPHjsWMGTOgUCjg6emJ3r17N/rgqIbj7ty5g//93/9F\nVlYW9Hq9lKUTtRiDhEhGf/rTnzB8+PBWHTdy5Ei4urpi5cqVGDp0KAYOHCh1mUQW8aotIgf0/PPP\nQ6VS4aOPPpK7FCIGCZEj8vDwwIwZM3DkyBGcPXtW7nLIyTFIiBzUs88+C1dXV3zyySdyl0JOjkFC\n5KC6deuGCRMm4Ouvv0ZxcbHc5ZATUwiCIMhdBBEROS7OSIiISBQGCRERicIgISIiURgkREQkCoOE\niIhEYZAQEZEoDBIiIhKFQUJERKIwSIiISJT/AzqHvVAoUFveAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.step(fpr, tpr, where='post')\n",
    "plt.title('ROC curve')\n",
    "plt.xlabel('FPR')\n",
    "plt.ylabel('TPR')\n",
    "pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plot precision-recall curve\n",
    "\n",
    "Precision-recall is more useful for evaluating models for unbalanced data sets. Why is precision-recall robust to unbalanced data?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEnCAYAAACDhcU8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XlUU9fePvAnBJEgBEGprQMKImJF\nqQMOyMWRWsv1qrUqTgUt9NoiKnZAra+tI1atyuDQwbFWsU5tXbXWOlKkyO31dbhOF4QKWhQEIVAQ\nhJzfH/2R1xgIJCchAZ7PWqxl9jk757sDK49n3BJBEAQQERHpycLUBRARUcPGICEiIlEYJEREJAqD\nhIiIRGGQEBGRKAwSIiIShUFCZm/YsGGYPn26zv0OHz6Mrl274sKFC0aoqnGbPn06hg0bpta2YMEC\ndO3a1UQVkTmzNHUBZHoXLlzAG2+8odZmY2MDFxcXjBkzBtOmTYNUKjVRdQ3f4cOHsXDhQtVriUSC\nFi1aoEuXLpg4cSJee+01E1ZHJB6DhFT+/ve/w8/PD4IgICcnB0eOHMGqVauQlpaG5cuXm6yu48eP\n69VvzJgxCAgIQLNmzQxckX6mT5+OHj16QBAEZGdn48CBA1i4cCFycnIwa9YsU5dHpDcGCam8+OKL\nGDNmjOr1lClTMGrUKBw4cABz585F69atq+335MkTKJVKNG/e3Ch1WVlZ6dVPKpWa1Z5U37598cor\nr6hev/baa3jllVfw5ZdfIjQ01KxqbSgeP34MS0tLWFryq8yUeI6EamRra4tevXpBEARkZWUBAGJj\nY9G1a1ekpqYiKioKfn5+6NmzJy5duqTql5SUhJkzZ6Jv377o0aMHRo8ejX379lW7jevXr2POnDnw\n8fGBp6cnBg8ejPnz5yMzM1O1TnXnSC5evIiQkBAMGjQIPXr0wN/+9jeEhoaq1VHTOZL8/HwsXboU\ngwcPVm1z6dKlePTokdp6Vf1//fVXbNu2DSNGjICnpydGjhyJI0eO6PehPqVNmzZwdXVFUVER8vPz\nNZYb+nM8duwYZs2ahSFDhsDT0xP9+/fHO++8g5s3b4oeiz413b17F127dkVsbKxG36q/s7t376ra\nqs7R5OfnY+HChfDx8cFLL72E1NRU9OjRA7Nnz662jk8//RRdu3bFjRs3VG1FRUVYu3Yt/P394enp\niQEDBmD+/Pmqv3PSDWOcaiQIAu7cuQMAcHBwUFv23nvvwdraGjNnzgQAODk5AQD279+Pjz76CC+9\n9BJmzZoFmUyGpKQkfPzxx8jMzERkZKTqPc6cOYPw8HDY2Njg9ddfR8eOHZGbm4vExET897//hbOz\nc7V1paenY+bMmWjdujXeeOMNtGrVCnl5efj3v/+Nmzdv4qWXXqpxTEVFRZg8eTLu3LmD8ePH48UX\nX8SNGzewb98+JCcn48CBA7C1tVXrs2HDBjx+/BiTJk2ClZUV9u3bhwULFsDZ2Rl9+vTR/YP9/548\neYLs7GxYWFhALperLTPG57hnzx60bNkSEydOhJOTEzIzM/HNN99g8uTJOHLkCDp16qT3WJ6l7++2\nLmbMmIHWrVvjnXfeQUlJCdq0aYNhw4bh1KlTKCgoQMuWLVXrKpVKHD16FF27dkW3bt0A/PU3EBgY\niD/++APjx49Hly5dkJubi71792LChAk4dOgQ2rVrJ/ozaFIEavKSk5MFd3d3ITY2VsjLyxPy8vKE\nGzduCB9++KHg7u4uTJw4UbVuTEyM4O7uLkybNk148uSJ2vs8ePBA8PT0FObPn6+xjeXLlwseHh5C\nZmamIAiCUFJSIvTv318YMGCAcP/+fY31KysrVf8eOnSoMG3aNNXrXbt2Ce7u7sLly5e1juvQoUOC\nu7u7kJycrGpbv3694O7uLuzZs0dt3T179gju7u7Chg0bNPqPGTNGKCsrU7Xfv39f6N69uxAREaF1\n+8++z8GDB4W8vDzh4cOHwtWrV4Xw8HDB3d1dmDNnjtr6xvoc//zzT43laWlpQvfu3YWPPvpIrX3a\ntGnC0KFD1doiIyMFd3f3Wsdb15qysrIEd3d3ISYmRmOdqr+zrKwsje2/++67GuufOXOm2t9rUlKS\n4O7uLmzfvl3Vtnz5cqFHjx7CjRs31Na9e/eu0KtXLyEyMrLWMZI6HtoildjYWAwcOBADBw7EmDFj\ncOjQIQwbNgybNm3SWDcoKEjjuPRPP/2E8vJyvP7668jPz1f7GTZsGJRKJZKSkgAAiYmJePToEWbM\nmIE2bdpovL+FRc1/mnZ2dgCAU6dOoaysTKcx/vzzz3B0dMSkSZPU2idNmgRHR0ecPHlSo8+UKVPU\nztO0adMGLi4u+P3333Xa9qJFizBw4ED4+Phg/PjxOHHiBCZOnIhVq1aprWesz9HGxgbAX3uaxcXF\nyM/Ph4ODA1xcXHDlyhWdxqKNmN9tXbz55psabb6+vmjdujW+++47tfZvv/0WlpaWGD16NIC/xn70\n6FF4e3vjueeeU/tsZTIZXnrpJSQmJoqqrynioS1SmTRpEl555RVIJBLIZDJ06tRJ7TDB06o7DHL7\n9m0AQHBwcI3bePjwIQCovoRffPFFnesMCAjA999/j61bt2Lnzp3w8vKCr68vAgICaj0kcffuXXh6\nemqEoKWlJTp16oTr169r9OnQoYNGW8uWLXHv3j3V64KCAjx58kRtnarDfVXCwsLQt29flJeX48qV\nK/jyyy+Rm5urcVWZsT7H69evIzo6GikpKSgpKVFb1r59+1r715WY321dVPe3VxUWO3bsQEZGBlxc\nXFBSUoITJ05g0KBBqgtF8vPzUVBQgMTERAwcOLDa9xcbdE0Rg4RUOnbsCB8fnzqta21trdEm/P+p\nbT755BM899xz1far7ktZV1ZWVtixYweuXLmCX375Bb/99htiYmIQFxeHTz/9FP7+/qK38bS6fLGE\nh4cjJSVFre3WrVtqr93d3VWf75AhQ9C5c2fMnz8fMTExeO+991TrGeNz/OOPPzB16lTY2tri7bff\nhqurK2QyGSQSCVatWqURLPVBIpHUuKyioqLGZTKZrNr2sWPHYseOHfj2228RERGBEydOoKSkBGPH\njlWtU/XZ+vj4IDQ0VM/K6VkMEjKYqv8pOjg41BpILi4uAIAbN27A19dXr+317NkTPXv2BABkZ2dj\n7Nix2Lhxo9Yg6dChAzIyMlBRUaG2V1JRUYHff/9d76CLjIyEQqHQqU9AQADi4+Oxc+dOBAYGqvYK\njPE5/vzzzygpKcGWLVswYMAAtWUFBQV6X2ItpiZ7e3sAQGFhocayp6/WqisPDw94eHjg6NGjmDdv\nHr777jvI5XIMHz5ctY6joyPkcjmKi4vr/J8mqh334chgRo0aBSsrK8TGxuLx48cay4uKilBeXg4A\nGDRoEBwcHLBjxw7k5ORorCtombizuktln3/+eTg6Olb7pfS0ESNGID8/HwcOHFBr/+abb5Cfn48R\nI0Zo7V8TT09P+Pj4qP3URVhYGJ48eYItW7ao2ozxOVbdo/Ls5/rNN98gNze3boOso7rWZGtrCycn\nJyQnJ6vVlZWVVe25qroYN24c7t27h6NHjyI5ORmjRo1Su7/JwsICo0ePxpUrV2q80TUvL0+vbTdl\n3CMhg3n++efx8ccfY/HixXj11Vfxj3/8A+3atUN+fj7++9//4uTJk/jhhx/Qvn17yGQyrFy5EnPn\nzsXo0aNVl4jm5+cjMTERwcHBNX6pb9myBefPn8eQIUPQvn17CIKAM2fOID09HSEhIVprDAkJwfHj\nx7Fs2TJcv34d3bp1w40bN3Dw4EG4uLjU2t/QBgwYgN69e+Pbb7/FrFmz0KFDB6N8jn5+fpDJZPjg\ngw8wbdo0yOVyXLx4EQkJCXB2dkZlZaXBxqTL73bq1KnYuHEjQkJCMGLECOTk5CA+Ph5dunTB1atX\ndd726NGjsXbtWixduhRKpRLjxo3TWCciIgIXL17EvHnzMGrUKHh5eaFZs2b4448/kJCQgO7du2P1\n6tWiP4emhEFCBjV+/Hh06tQJ27dvx/79+1FUVISWLVvCxcUFc+fOVTsBPXz4cOzduxdbt27FwYMH\n8eeff6J169bo06eP1ocDjhgxArm5uTh+/DgePnwIa2trdOzYEStWrMDrr7+utT47Ozvs27cPMTEx\nOH36NA4fPoxWrVohMDAQ4eHhGveQ1Id33nkHISEh2Lx5M6KiogAY/nN0dnbGF198gfXr12Pr1q2Q\nSqXo3bs3vvrqKyxfvlztwgFDqOvvNjQ0FEVFRfj++++RkpICNzc3rFy5EteuXdMrSFq1aoW//e1v\nOHPmDDp16oRevXpprFP1N7B9+3YcP34cp06dglQqxfPPP48+ffpgwoQJosbeFEkEbccQiIiIasFz\nJEREJAqDhIiIRGGQEBGRKAwSIiISpcldtZWbW6R3XwcHGzx6VP93AJsSx9w0cMxNg5gxOznZ1biM\neyQ6sLRsehMPccxNA8fcNBhrzAwSIiIShUFCRESiMEiIiEgUkwbJnTt3sGTJEowePRrdunXTmJe7\nJkVFRVi4cCG8vb3Rp08fvPvuuxrzbRMRUf0waZCkpqbi3LlzcHFx0Wm+6Hnz5uHChQtYsWIFVq9e\njf/85z8ICwszXqFERFQjk17+O2zYMNVTQOfMmVOnvYr//d//RWJiIvbs2QNvb28Af019OmHCBCQl\nJXGOASKiembSPRJ9prRMSEhA69atVSEC/DXBUfv27ZGQkGDI8oiIqA4a3A2J6enpcHV11Wjv3Lkz\n0tPTjbbdb06n4WJqLiorDfuwZG+P5zBxmJtB35OIqD41uCBRKBSws9O8w1Iul9dpek4HBxu9bsqR\n2fw1FalUWvM807p6WPgYF1NzETZJc84Ec6LtjtbGimNuGjhmw2hwQSKWvo8HGD3AGTNHdxf1iJVn\nvb85CZWVgkHf09CcnOzMuj5j4JibBo5Z9741aXD3kcjlchQVaX4QCoUCcrncBBURETVtDS5IXF1d\nkZGRodFe07kTIiIyrgYXJH5+fsjNzcVvv/2mart69SqysrLg5+dnwsqIiJomk54jKS0txblz5wAA\nDx48QHFxMY4fPw4AGDx4MGQyGfz9/eHt7Y1Vq1YBAHr16gVfX19ERkYiMjISFhYWWLt2Lfr06cN7\nSIiITMCkQZKXl4e5c+eqtVW9PnXqFNq3b4/KykoolUq1dTZs2ICoqCgsWrQISqUSQ4cOxYcfflhv\ndRMR0f8xaZC0b98et27d0rrO6dOnNdrkcjmioqIQFRVlrNKIiKiOGtw5EiIiMi8MEiIiEoVBQkRE\nojBIiIhIFAYJERGJwiAhIiJRGCRERCQKg4SIiERhkBARkSgMEiIiEoVBQkREojBIiIhIFAYJERGJ\nwiAhIiJRGCRERCQKg4SIiERhkBARkSgMEiIiEoVBQkREojBIiIhIFAYJERGJwiAhIiJRGCRERCQK\ng4SIiERhkBARkSgMEiIiEoVBQkREojBIiIhIFAYJERGJwiAhIiJRGCRERCQKg4SIiERhkBARkSgm\nDZK0tDQEBQXBy8sLvr6+iI6ORmVlZa39rl69ipkzZ6Jfv37o168fgoODcfny5XqomIiInmWyICks\nLERwcDAkEgk2b96MsLAw7NixAzExMVr7ZWdnY8aMGaioqMCaNWuwZs0aVFZWYsaMGbh37149VU9E\nRFUsTbXh+Ph4lJWVIS4uDra2thg0aBCKi4sRFxeH0NBQ2NraVtvv7Nmz+PPPP7Fp0ybY2dkBAHr1\n6oUBAwbg3LlzmDJlSn0Og4ioyTPZHklCQgJ8fX3VAiMgIACPHz9GSkpKjf0qKioglUohk8lUbTY2\nNpBKpRAEwag1ExGRJpMFSXp6OlxdXdXa2rZtC5lMhvT09Br7vfzyy5DJZFi9ejXy8vKQl5eHqKgo\n2NvbY9SoUcYum4iInmGyQ1sKhUJ1aOppcrkcCoWixn5t2rTB7t278c9//hNfffUVAMDJyQnbtm2D\no6Njrdt1cLCBpaVU77qdnDRr1pdUKjH4exqDuddnDBxz08AxG4bJgkRfOTk5mDt3Lrp3744VK1YA\nAPbu3Yu33noL8fHxaNu2rdb+jx6V6L1tJyc75OYW6d3/WZWVfx2KM+R7Gpqhx9wQcMxNA8ese9+a\nmOzQllwuR3FxsUa7QqGAXC6vsd+2bdtQUVGBmJgY+Pn5wc/PDzExMZBKpdi+fbsxSyYiomqYLEhc\nXV01zoVkZ2ejtLRU49zJ09LT0+Hm5oZmzZqp2qysrODm5obMzEyj1UtERNUzWZD4+fkhMTFRba/k\n2LFjsLa2Rr9+/Wrs17ZtW6SmpqK8vFzVVl5ejtTUVLRr186oNRMRkSaTBUlgYCCsrKwQHh6OpKQk\n7N+/H3FxcQgODla7JNjf3x+LFi1SvZ4wYQJycnIwe/ZsnD17FmfOnME777yD3NxcTJo0yRRDISJq\n0kwWJPb29ti5cycqKysxa9YsxMbGIigoCHPmzFFbr7KyEkqlUvXa09MTX375Jf7880988MEHiIyM\nxOPHj7F9+3Z4eHjU9zCIiJo8k1615ebmht27d2td5/Tp0xptAwcOxMCBA41VFhER6YBP/yUiIlEY\nJEREJAqDhIiIRGGQEBGRKAwSIiIShUFCRESiMEiIiEgUBgkREYnCICEiIlEYJEREJAqDhIiIRGGQ\nEBGRKAwSIiISRa+n/96+fRuHDh3C3bt3UVhYCEEQ1JZLJBLs2rXLIAVSw/TN6TT862ZOjcu9PZ7D\nxGFu9VgRERmLzkHy7bffYtGiRbC0tISLi0u186s/GyxUf8zlC/xfN3PwqKgMDnbNNZY9KirDv27m\nMEiIGgmdgyQuLg4eHh748ssv4ejoaIyaSARz+gJ3sGuOte/4aLS/vzmpXrZPRPVD5yDJycnBzJkz\nGSJmTJ8v8Jr2ZKRSCSorBR6KIqIa6Rwk7u7uyM3NNUYtZEL67MloO4xW03sRUeOjc5BERkYiIiIC\nQ4YMgZeXlzFqIhOpbk/GyckOwUt/qnZ9beHjYNcc3h7PGaVOIjIvOgfJrl27YG9vj8DAQHTt2hUv\nvPACpFKp2joSiQSxsbEGK5LMV02H0Yio6dA5SK5fvw4AeOGFF6BQKKBQKDTWkUgk4isjIqIGQecg\nOX36tDHqICKiBkqvGxLJtPQ9yZ2neAyg+qu39OnHE+pEBIgIkrNnz+Ls2bP4448/AABt27bF0KFD\nMXjwYIMVR9UzxkluffrxhDoRAXoEyePHjxEWFoakpCRIpVK0adMGAJCUlIT9+/dj0KBBiIuLg7W1\ntcGLpf8j5iR3ffcjosZN54c2RkdHIykpCREREUhJScGpU6dw6tQp/Otf/8K7776LpKQkREdHG6NW\nIiIyQzoHybFjxzBp0iS89dZbsLGxUbXLZDKEhIRgwoQJOHbsmEGLJCIi86VzkOTn56NLly41Lnd3\nd0d+fr6oooiIqOHQOUg6dOiAc+fO1bj83Llz6NChg6iiiIio4dD5ZPuUKVOwYsUKzJo1C0FBQXBx\ncQEAZGRkYPfu3fjll1+wePFigxdKprN9wTBTl0BEZkznIJk2bRoePXqEzz//XGPPxNLSEmFhYZg6\ndarBCiQiIvOm130k4eHhmDp1Kn799Vfcu3cPANCuXTsMHDiQj5c3EGM8WZd7FkRkDHrfkOjo6IiA\ngABD1kJP4ZN1iaihqDVInr5z/enXtalaX5u0tDQsX74cly5dgp2dHSZMmIDZs2drPE24OidOnMBn\nn32G1NRUyGQyeHp6IjY2Vu2S5IaOT9Ylooag1iAZNmwYJBIJLl++DCsrK9Xr2ty4cUPr8sLCQgQH\nB8PNzQ2bN29GZmYmPvnkEyiVSkRERGjte+DAASxbtgwhISH44IMPoFAokJycjMrKylrrIiIiw6o1\nSFatWgWJRIJmzZqpvRYrPj4eZWVliIuLg62tLQYNGoTi4mLExcUhNDQUtra21fbLz8/HqlWr8D//\n8z+YOHGiqt3f3190TWR62s4NAeCUv0RmqNYgee2117S+1ldCQgJ8fX3VAiMgIADr1q1DSkoKhg2r\n/sTwjz/+CAAYO3asQeog86LPlL9EZFoGeYy8IAi4cOECysrK0KdPnxr3Jp6Wnp6OAQMGqLW1bdsW\nMpkM6enpNQbJlStX4OLigoMHD2Lr1q3Iy8vDiy++iIULF6J3796GGA6ZWE3nhqp7/D0RmZ7OQbJh\nwwZcvHgRX331laotJCQESUlJEAQBbdu2xc6dO+Hs7Kz1fRQKBezs7DTa5XJ5tbMuVnn48CEyMjKw\nZcsWvP/++2jZsiW+/PJLhISE4MSJE2jdurXW7To42MDSsvaT+TVxctKsWV9SqaTG99S2rL4ZugZ9\nx12fn4k5fO71jWNuGowxZp2D5KeffsLQoUNVr0+ePInz58/jvffeg4eHBxYvXozY2FisXbvWoIVW\nEQQBJSUliI6Ohp+fHwCgd+/eGDp0KPbs2YN58+Zp7f/oUYne23ZyskNubpHe/Z9VWSkAQLXvqW1Z\nfTL0mAEg51EpACB46U8ay6oOa5nyMzHGmM0dx9w0iBmztgDSOUgePHiATp06qV7//PPP6NKlC0JC\nQgAAkydPxt69e2t9H7lcjuLiYo12hUIBuVyutZ9EIkH//v1Vbba2tujevTtu376tw0jIHPEeGaKG\nR+cgsbS0VLvMNikpCX//+99Vrx0dHfHo0aNa38fV1RXp6elqbdnZ2SgtLYWrq2uN/Tp37gxBECAI\nglq7IAgGuZqM6g/vkSFqHHQOki5duuDnn3/GlClTcO7cOTx8+FBtet179+7BwcGh1vfx8/PDtm3b\nUFxcrDo5f+zYMVhbW6Nfv3419hsyZAji4uJw4cIF1XaLiopw7do1zJw5U9fhUAOibc55gJcGE5mK\nzo+RDwsLw2+//QZvb2+EhYWhd+/ealdfJSQkoEePHrW+T2BgIKysrBAeHq6apjcuLg7BwcFqV335\n+/tj0aJFqtc9evTA8OHD8eGHH+LIkSM4e/Ys3n77bVhaWvJhkU1Y1aXBRFT/dN4jGTRoEI4cOYLz\n589DLpfj1VdfVS0rKChA3759MXz48Frfx97eHjt37sSyZcswa9YsyOVyBAUFITw8XG29yspKKJVK\ntba1a9dizZo1WL16NUpLS9G7d2/s2rUL9vb2ug7HpLT9D1vfBzM2Bbw0mMi86HUfiZubG9zcNA8h\ntGzZUm3voS7vs3v3bq3rnD59WqOtRYsWWLp0KZYuXVrnbTU0POlMRA2FQW5IJHGa2klnPs6eqHGp\nNUg8PDxgYWGBS5cuwcrKCh4eHrVeHSWRSHD9+nWDFUlEROar1iAJCwuDRCKBpaWl2mui+sY9GSLz\nVGuQPHvy+9nXRETUtOl8+S8REdHTdA6SlStXYuTIkTUuHzlyJD755BNRRRERUcOhc5CcO3cOo0aN\nqnH5q6++Wu0lu0RE1DjpHCT3799Hu3btalzetm1b3L9/X1RRRETUcOgcJHK5HBkZGTUuT09PR4sW\nLUQVRUREDYfOQfK3v/0N8fHxuHXrlsaymzdvIj4+Hr6+vgYpjoiIzJ/Od7bPnTsXv/zyC8aPH48R\nI0aoHpWSmpqKU6dOoWXLlrVOLkVERI2HzkHy/PPP49ChQ1i3bh1Onz6N48ePA/jr+Vevvvoq5s+f\nj+eff97ghRIRkXnS61lbbdq0wdq1ayEIAvLz8wH8NaEV73gnImp6RN2QWF5ejoqKCtjZ2TFEiIia\nKL32SJKTk7Fx40ZcvXoVSqUS27dvx8CBA5Gfn4+5c+ciJCREbdZEImPj7IlEpqPzHklSUhLefPNN\nlJSU4I033lCbO93R0RFWVlY4ePCgQYskEoOzJxIZl857JNHR0ejRowf27t2LgoIC7NixQ215nz59\nGCR1xKfZGh5nTySqfzrvkdy8eROjR4+GhYVFtedFnJyc8PDhQ4MUR0RE5k/nILG2tkZZWVmNy+/e\nvQu5XC6qKCIiajh0DhJvb2989913UCqVGsvy8/Nx8OBB+Pg0raljiYiaMp3PkcybNw+BgYGYOHEi\nRo4cCYlEgl9//RX//ve/ER8fj4qKCk5+RfWO55uITEfnPRI3Nzd8/fXXaNGiBdavXw9BEPD5558j\nLi4OLi4u2LNnDzp06GCMWomIyAzpdR9J165dsWvXLhQWFuLOnTsQBAEdOnSAo6OjoesjIiIzp9Me\nSWlpKYKCgnDo0CEAgL29PXr27AkvLy+GCBFRE6VTkMhkMly7dg0VFRXGqoeIiBoYnc+R+Pr6IimJ\nN3gREdFfdA6SuXPn4vbt21i4cCEuXbqEhw8fori4WOOHiIiaBp1Pto8aNQoAkJaWhm+//bbG9W7c\nuKF/VURE1GDoHCSzZ882Rh1ERNRA6RQkly5dgouLC1q2bIm+ffuiefPmxqqLiIgaiDoFSXFxMUJC\nQnD58mVVW6tWrbB161Z4enoarTgiIjJ/dTrZ/tlnn+HSpUt4+eWXsXjxYgQFBaGoqAgLFiwwdn1E\nRGTm6rRHcurUKbzyyivYuHGjqq1z585YsmQJ7ty5g44dOxqtQCIiMm91CpJ79+4hODhYrc3X1xeC\nICAnJ0fvIElLS8Py5ctx6dIl2NnZYcKECZg9ezakUmmd+iuVSrz++uu4du0atm7diqFDh+pVB1F9\n+eZ0Wo2zNXI6YGqo6nRoq6ysDDKZTK3N2toaAPS+y72wsBDBwcGQSCTYvHkzwsLCsGPHDsTExNT5\nPQ4cOIAHDx7otX0iU/jXzRw8KtKcz4fTAVNDVuertrKzs3Hz5k3V66KiIgB/TWT1dHsVDw8Pre8X\nHx+PsrIyxMXFwdbWFoMGDUJxcTHi4uIQGhoKW1tbrf0LCwuxYcMGvPvuu1i8eHFdh0Fkcg52zTWm\nBOZ0wNSQ1TlINmzYgA0bNmi0L1myRO21IAiQSCS13pCYkJAAX19ftcAICAjAunXrkJKSgmHDtM8v\nER0djd69e2PgwIF1HQJRvdB2+OpRURkc7HjZPDUudQqSqKgog284PT0dAwYMUGtr27YtZDIZ0tPT\ntQbJzZs3cejQIXz//fcGr4tIrKrDV9UFhoNdc3h7PGeCqoiMp05BMm7cOINvWKFQwM7OTqNdLpdD\noVBo7btixQpMnToVHTt2xN1iKclaAAATTUlEQVS7d3XaroODDSwt63YyvzpOTpo1N3YNfcxSqQSA\nbuMQM2apVILWLa2xbfHLOvURu12xGvrvWR8cs2HoNbGVKf3www/IyMjA1q1b9er/6FGJ3tt2crJD\nbm6R3v0bosYw5spKAQDqPI66jLkuh690+dx0rdHQGsPvWVccs+59a6Lz038NRS6XV/uUYIVCAblc\nXm2fJ0+eYM2aNQgNDYVSqYRCoVC9R2lpKZ86TPWmpquvAB6+oqbHZHskrq6uSE9PV2vLzs5GaWkp\nXF1dq+1TWlqK+/fvIyoqSuO8TUREBJydnfHzzz8brWZqWuqy1/Hs1Vf6ylM8BlDz1Vu8x4TMmcmC\nxM/PD9u2bUNxcbHqyq1jx47B2toa/fr1q7aPjY0Ndu/erdb28OFDzJ8/H/Pnz9c4eU8khrmcNK+6\nx4RBQubKZEESGBiIr776CuHh4QgNDUVWVhbi4uIQHBysdkmwv78/vL29sWrVKlhaWqJ///5q71N1\nst3d3R1eXl71OgZq/Ay511EX1W1L2z0m2vaaAO7JUP0wWZDY29tj586dWLZsGWbNmgW5XI6goCCE\nh4errVdZWQmlUmmiKqmxq+6LWCqVoLJSMJt7PrQd9qpa1kpurbFM3z0ZY4STtvfUNgZ9t0f1y6RX\nbbm5uWkcqnrW6dOntS5v3749bt26ZciyqAkxl8NX2xdovwG3NrruyWij7TPJUzzG8ZRMnZ8Xpu09\nteFhvYahwV3+S2Rozx6+MtfLQqsLi5mrtf9HqybP7iFU7YUB2i8k0LY9bSFTl/c0ZBhS/WKQUKOn\n7dCQuRy+0pe+ezJi98R0DRleEt24MUioSWsIX3D6hkVdArQqEAy5F6brxQliD+uR6TFIqMmoz6uv\nzJ2+AartS5+B0HQxSIgaOQYoGZvJHpFCRESNA/dIqNFrqodcmuq4qf4xSIjIbPEZZA0DD20RUYPE\nee7NB/dIiMjs8WZF88YgISKzxfM8DQMPbRERkSgMEiIiEoVBQkREovAcCRE1SNouDeZlwfWLeyRE\n1KjwsuD6xz0SImrQnr00mJcF1z8GCRE1SPV5abAxph9uTBgkRETQf155facfbkwYJETUZDwdFk9P\nLwxoD4squs4M2VTmnGeQEFGjou1qLm1h0UpuXePeg7awqNKUH+PCICGiJuPpsNBlemFjzAzZmM67\nMEiIqFEyh5kh9d07amiHxBgkRNSoNLQHPTaGQ2IMEiIiI9P1JH1DwyAhIjISfc+tNLSZIfmIFCKi\nBsQcHwHDPRIiIjPVUM6fMEiIiMxMQ7tggIe2iIhIFAYJERGJwiAhIiJRGCRERCSKSYMkLS0NQUFB\n8PLygq+vL6Kjo1FZWam1z5UrV7Bw4UL4+/vDy8sLI0eORFxcHMrKyuqpaiIieprJrtoqLCxEcHAw\n3NzcsHnzZmRmZuKTTz6BUqlEREREjf1+/PFHZGZmIjQ0FB07dsStW7cQHR2NW7duITY2th5HQERU\n/8zxZkWTBUl8fDzKysoQFxcHW1tbDBo0CMXFxYiLi0NoaChsbW2r7RcaGgpHR0fV6/79+6N58+ZY\nsmQJ7t27h3bt2tXXEIiIzIqpHvZosiBJSEiAr6+vWmAEBARg3bp1SElJwbBh1V9H/XSIVOnWrRsA\nICcnh0FCRE2COd2saLIgSU9Px4ABA9Ta2rZtC5lMhvT09BqDpDqXLl2ChYUFnJ2dDV0mEZFZMceb\nFU0WJAqFAnZ2dhrtcrkcCoWizu+Tm5uLLVu2YMyYMWjVqlWt6zs42MDSUqpTrU9zctKsubHjmJsG\njrnhk0olALSPyxhjbtCPSCkvL8e8efNgY2ODhQsX1qnPo0clem9PlxnVGguOuWngmBuHqjnoaxqX\nmDFrCyCTBYlcLkdxcbFGu0KhgFwur7W/IAiIjIxEWloa9u7dC3t7e2OUSUREtTBZkLi6uiI9PV2t\nLTs7G6WlpXB1da21/8qVK3Hq1Cls374dnTt3NlaZRERUC5PdkOjn54fExES1vZJjx47B2toa/fr1\n09r3s88+w9dff421a9eib9++xi6ViIi0MFmQBAYGwsrKCuHh4UhKSsL+/fsRFxeH4OBgtUuC/f39\nsWjRItXro0ePYv369Rg7dizatGmDS5cuqX7y8/NNMRQioibNZIe27O3tsXPnTixbtgyzZs2CXC5H\nUFAQwsPD1darrKyEUqlUvT5//jwA4PDhwzh8+LDaulFRUXjttdeMXzwREamY9KotNzc37N69W+s6\np0+fVnu9evVqrF692phlERGRDvj0XyIiEoVBQkREojBIiIhIFAYJERGJ0qAfkUJERP+ntrlK/Hq3\nx+gBhn+4LfdIiIhIFO6REBE1MtXNVQIY70GVDBIiokbCVHOV8NAWERGJwiAhIiJRGCRERCQKg4SI\niERhkBARkSgMEiIiEoVBQkREojBIiIhIFIkgCIKpiyAiooaLeyRERCQKg4SIiERhkBARkSgMEiIi\nEoVBQkREojBIiIhIFAYJERGJwiABkJaWhqCgIHh5ecHX1xfR0dGorKystV9RUREWLlwIb29v9OnT\nB++++y4ePXpUDxWLp8+Yr1y5goULF8Lf3x9eXl4YOXIk4uLiUFZWVk9Vi6Pv77mKUqnEa6+9hq5d\nu+LMmTNGrNRwxIz5xIkTGD9+PHr27In+/fvjzTffRElJiZErFk/fMV+9ehUzZ85Ev3790K9fPwQH\nB+Py5cv1ULE4d+7cwZIlSzB69Gh069YN06dPr1M/Q35/NfkZEgsLCxEcHAw3Nzds3rwZmZmZ+OST\nT6BUKhEREaG177x585CRkYEVK1bAwsIC69atQ1hYGPbu3VtP1etH3zH/+OOPyMzMRGhoKDp27Ihb\nt24hOjoat27dQmxsbD2OQHdifs9VDhw4gAcPHhi5UsMRM+YDBw5g2bJlCAkJwQcffACFQoHk5GSd\ngtcU9B1zdnY2ZsyYgRdffBFr1qwBAGzbtg0zZszA0aNH0a5du/oags5SU1Nx7tw5eHl5oaKios79\nDPr9JTRxW7duFfr27SsUFRWp2j7//HOhZ8+eam3PunjxouDu7i6kpKSo2i5fviy4u7sL58+fN2rN\nYuk75ry8PI22+Ph4wd3dXbh7965RajUUfcdcpaCgQOjfv7/wzTffCO7u7sLp06eNWa5BiPk9v/TS\nS8L+/fvro0yD0nfMe/fuFTw8PASFQqFqKygoEDw8PISvv/7aqDWLVVlZqfp3eHi4MG3atFr7GPr7\nq8kf2kpISICvry9sbW1VbQEBAXj8+DFSUlK09mvdujW8vb1VbT179kT79u2RkJBg1JrF0nfMjo6O\nGm3dunUDAOTk5Bi+UAPSd8xVoqOj0bt3bwwcONCYZRqUvmP+8ccfAQBjx441eo2Gpu+YKyoqIJVK\nIZPJVG02NjaQSqUQzPwpUhYWun+NG/r7q8kHSXp6OlxdXdXa2rZtC5lMhvT0dJ36AUDnzp219jMH\n+o65OpcuXYKFhQWcnZ0NWaLBiRnzzZs3cejQIURGRhqzRIPTd8xXrlyBi4sLDh48CD8/P3Tv3h0T\nJkzAxYsXjV2yaPqO+eWXX4ZMJsPq1auRl5eHvLw8REVFwd7eHqNGjTJ22fXO0N9fTT5IFAoF7Ozs\nNNrlcjkUCoXB+5kDQ9Wem5uLLVu2YMyYMWjVqpUhSzQ4MWNesWIFpk6dio4dOxqrPKPQd8wPHz5E\nRkYGtmzZgvfeew9btmyBTCZDSEgIHj58aMySRdN3zG3atMHu3btx4sQJ+Pj4wMfHBydOnMC2bduq\n3RNv6Az9/dXkg4T0U15ejnnz5sHGxgYLFy40dTlG88MPPyAjIwPvvPOOqUupN4IgoKSkBCtXrsQ/\n/vEP+Pn5YfPmzZBKpdizZ4+pyzOKnJwczJ07F927d8cXX3yBL774Ap6ennjrrbfwxx9/mLo8s9fk\ng0Qul6O4uFijXaFQQC6Xa+1XVFSkcz9zoO+YqwiCgMjISKSlpeHzzz+Hvb29Mco0KH3G/OTJE6xZ\nswahoaFQKpVQKBSq9ygtLa32/cyJmL9tiUSC/v37q9psbW3RvXt33L592yi1Goq+Y962bRsqKioQ\nExMDPz8/+Pn5ISYmBlKpFNu3bzdmySZh6O+vJh8krq6uGscEs7OzUVpaWu0xxKf7ZWRkaLTXdOzR\nnOg75iorV67EqVOnsGnTJnTu3NlYZRqUPmMuLS3F/fv3ERUVBW9vb3h7e2PMmDEAgIiICIwbN87o\ndYuh7++5c+fOEARB4ySzIAiQSCRGqdVQ9B1zeno63Nzc0KxZM1WblZUV3NzckJmZabR6TcXQ319N\nPkj8/PyQmJio9r+YY8eOwdraGv369dPaLzc3F7/99puq7erVq8jKyoKfn59RaxZL3zEDwGeffYav\nv/4aa9euRd++fY1dqsHoM2YbGxvs3r1b7Wf9+vUAgPnz52PdunX1Uru+9P09DxkyBABw4cIFVVtR\nURGuXbsGDw8Po9VrCPqOuW3btkhNTUV5ebmqrby8HKmpqWZ9D4m+DP39Jf34448/NmB9DU6XLl2w\nf/9+XLhwAc899xySkpKwfv16BAUFYfDgwar1/P39cfPmTQwfPhwA8MILL+DSpUs4ePAgXnjhBWRk\nZODjjz9G586dMW/ePFMNp070HfPRo0exdOlSjBs3Dv3798f9+/dVP1ZWVmqXTpobfcZsYWGB9u3b\nq/1UhUtQUBAGDBhgwhHVTt/fc5s2bXDjxg3s27cPDg4OePDgAZYvX46CggKsWbMG1tbWphpSrfQd\ns5OTE3bt2oX//Oc/sLOzQ0ZGBlavXo1bt25h2bJlaN26tamGVKvS0lKcOnUKaWlpSExMRGFhIVq1\naoW0tDS0a9cOzZo1M/73l853njRCqampwvTp04UePXoIgwYNEjZs2CBUVFSorTN06FAhMjJSra2w\nsFBYsGCB0KdPH6FXr17C/Pnzq71pzxzpM+bIyEjB3d292p9Dhw7V9xB0pu/v+WlZWVkN5oZEQdB/\nzMXFxcKSJUuEfv36CT169BCCgoKEmzdv1mfpetN3zElJScKUKVMEb29vwdvbW5g6daqQnJxcn6Xr\npepvsrqfrKwsQRCM//3FOduJiEiUJn+OhIiIxGGQEBGRKAwSIiIShUFCRESiMEiIiEgUBgkREYnC\nICFqRKZPn6421eqFCxfQtWtXtbvUiQytyU+1S6SLw4cPqz3t2NLSEk5OThg6dCjmzZvXIB5gSWRo\nDBIiPUREROCFF17A48ePceHCBezdu1f1WBFzf7AhkaExSIj0MHjwYNU0w5MmTQLw19wlV69eRc+e\nPU1ZGlG94zkSIgPo06cPAGg8cvzQoUMYN24cevbsif79+yMyMrLaWQbPnj2LqVOnolevXujTpw8C\nAwNx8uRJ1fKTJ0/irbfegq+vLzw9PTFixAhs2rQJlZWVxh0YUR1wj4TIAO7duwcAapMCxcXFYdOm\nTQgICMDEiRORm5uL3bt34+rVqzh8+LDqKboHDhzA4sWL4eHhgVmzZqFFixa4du0azp8/jxEjRgAA\njhw5AhsbG8yYMQM2NjZITk5GTEwMiouLG9xc8tT4MEiI9KBQKJCfn4/Hjx8jJSUF+/btg6OjI7y9\nvQEAd+/exebNm/H+++9j5syZqn5+fn4IDAzEkSNHMHnyZBQVFWHVqlXo1asXdu/eDSsrK9W6Tz9P\n9dNPP1V7fPvkyZOxZMkS7Nu3DxEREWr9iOobg4RID2+88Ybaa3d3d6xatUo1J8vJkychCAL8/f2R\nn5+vWs/Z2RlOTk5ISUnB5MmTkZiYiJKSEvzzn//UCIOnT9o/HSLFxcUoLy9H3759sX//fqSnp5v9\nhFPUuDFIiPSwdOlSODs7o6CgAPHx8UhLS1Ob2Ov333+HUqlUHZp6VlW4ZGVlAfhrQiZtUlNTsXHj\nRiQnJ2vMSV7d3NtE9YlBQqQHLy8v1VVbw4cPx9ixY/Hee+/h8OHDsLCwgFKphFQqxRdffFHt5cBP\nn0upjUKhwLRp02Bra4s5c+bA2dkZzZs3x7Vr17Bu3ToolUqDjYtIHwwSIpGaN2+O2bNnY/78+fjx\nxx8REBAAZ2dnVFZWomPHjmjfvn2NfZ2dnQH8tcdR03opKSkoKChAXFyc6hwM8Nd5GCJzwMt/iQzg\nlVdeQbt27fDFF18A+GtOcAsLC2zatEljXaVSiYKCAgDAoEGDYGNjg88++wzl5eVq61WdbLewsFB7\nDQDl5eXYu3evUcZCpCvukRAZgFQqxRtvvIGoqCgkJCTAz88Pc+bMwcaNG5GVlYWhQ4dCJpMhKysL\nP/30E95++21MmDABdnZ2WLBgAZYsWYIJEyYgICAALVq0wPXr12FlZYWPPvoIvXr1gr29PRYsWIDp\n06dDIpHgu+++A2fJJnPBPRIiA3n99ddhZ2en2it5++23sWHDBlRUVCA2NhZr165FQkICRowYAR8f\nH1W/SZMmYdOmTbC2tsamTZuwYcMG3L59G76+vgAABwcHbN26FU5OTti4cSO2bdsGHx8fvP/++yYZ\nJ9GzJAL/W0NERCJwj4SIiERhkBARkSgMEiIiEoVBQkREojBIiIhIFAYJERGJwiAhIiJRGCRERCQK\ng4SIiET5fwXMnOOo8aAEAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.step(recall, precision, where='post')\n",
    "plt.title('Precision-Recall curve')\n",
    "plt.xlabel('Recall')\n",
    "plt.ylabel('Precision')\n",
    "pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Case study\n",
    "\n",
    "This illustrates danger of over-fitting."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dataset information\n",
    "\n",
    "Please download the [Credit Card Fraud Detection data set](https://www.kaggle.com/samkirkiles/credit-card-fraud/data) and put it in the `data` folder. You will need to have a Kaggle account to do so."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('data/creditcard.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df.loc[:, df.columns.drop('Class')]\n",
    "y = df.Class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    284315\n",
       "1       492\n",
       "Name: Class, dtype: int64"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = scaler.fit_transform(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, random_state=2018)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((213605, 30), (213605,))"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.shape, y_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "search = Parallel(n_jobs=-1)(delayed(fit)(n, X_train) for n in range(2,X.shape[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "scores =  np.array([(n, average_precision_score(y_train, y_score)) for (n, y_score) in search])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### This is an amazingly good average precision score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([27.        ,  0.69112164])"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best = max(scores, key=lambda x: x[1])\n",
    "best"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "n = int(best[0])\n",
    "n, y_score = fit(n, X_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "precision, recall, thresholds = precision_recall_curve(y_train, y_score)\n",
    "fpr, tpr, thresholds = roc_curve(y_train, y_score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The precision-recall curve is also amazingly good"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEnCAYAAACDhcU8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XlcVPX+P/DXACKDMCyCKCLIIqCi\nqIgbhEuSGdfccksNLOhShlvdUOtrlqaWpqG4VG6ZueTa9Xetay5JiMgtr0sqBoICCoKyDMgmM+f3\nhznXadhmY8B5PR8Pb5zPOZ85788Md16cXSQIggAiIiINmRi6ACIiatkYJEREpBUGCRERaYVBQkRE\nWmGQEBGRVhgkRESkFQYJNXvDhg3D9OnT1e538OBB+Pj44Ny5c3qo6uk2ffp0DBs2TKlt/vz58PHx\nMVBF1JyZGboAMrxz587hlVdeUWqztLSEu7s7Ro8ejWnTpsHU1NRA1bV8Bw8exIIFCxTTIpEIbdq0\nQZcuXTBx4kSMGzfOgNURaY9BQgp/+9vfEBISAkEQkJ+fj0OHDmHZsmVIT0/HkiVLDFbXjz/+qFG/\n0aNHIywsDK1atdJxRZqZPn06evToAUEQkJubi3379mHBggXIz89HdHS0ocsj0hiDhBS6deuG0aNH\nK6ZffvlljBw5Evv27cPs2bPh4OBQa7+HDx9CLpejdevWeqnL3Nxco36mpqbNakuqb9++eP755xXT\n48aNw/PPP4/NmzcjKiqqWdXaUlRWVsLMzAxmZvwqMyQeI6E6WVlZoXfv3hAEAdnZ2QCAdevWwcfH\nB2lpaVi+fDlCQkLQs2dPXLhwQdEvKSkJr776Kvr27YsePXpg1KhR2L17d63ruHr1KmbNmoVBgwbB\nz88PgwcPxrx585CVlaVYprZjJOfPn0dkZCSCgoLQo0cPPPPMM4iKilKqo65jJIWFhfjwww8xePBg\nxTo//PBDFBUVKS33uP/Zs2exZcsWDB8+HH5+fhgxYgQOHTqk2Zv6BCcnJ3h4eKC0tBSFhYUq83X9\nPh49ehTR0dEYMmQI/Pz80L9/f7z55ptITU3Veiya1JSTkwMfHx+sW7dOpe/j37OcnBxF2+NjNIWF\nhViwYAEGDRqEXr16IS0tDT169MBbb71Vax2fffYZfHx8cO3aNUVbaWkpVq5cidDQUPj5+WHAgAGY\nN2+e4vec1MMYpzoJgoBbt24BAOzs7JTmvfPOO7CwsMCrr74KAHB0dAQA7N27Fx988AF69eqF6Oho\niMViJCUlYfHixcjKykJsbKziNU6dOoWYmBhYWlripZdegpubGwoKCpCYmIg//vgDrq6utdaVkZGB\nV199FQ4ODnjllVfQtm1b3L9/H7/99htSU1PRq1evOsdUWlqKKVOm4NatWxg/fjy6deuGa9euYffu\n3UhOTsa+fftgZWWl1GfNmjWorKzEpEmTYG5ujt27d2P+/PlwdXVFQECA+m/snx4+fIjc3FyYmJhA\nIpEozdPH+7hz507Y2tpi4sSJcHR0RFZWFr777jtMmTIFhw4dQufOnTUey19p+tk2xowZM+Dg4IA3\n33wT5eXlcHJywrBhw3DixAkUFxfD1tZWsaxcLseRI0fg4+ODrl27Anj0OzB58mTcuXMH48ePR5cu\nXVBQUIBdu3ZhwoQJOHDgADp27Kj1e2BUBDJ6ycnJgre3t7Bu3Trh/v37wv3794Vr164J7733nuDt\n7S1MnDhRsezatWsFb29vYdq0acLDhw+VXufu3buCn5+fMG/ePJV1LFmyRPD19RWysrIEQRCE8vJy\noX///sKAAQOEvLw8leVlMpni56FDhwrTpk1TTH/99deCt7e3cPHixXrHdeDAAcHb21tITk5WtK1e\nvVrw9vYWdu7cqbTszp07BW9vb2HNmjUq/UePHi1UVVUp2vPy8oTu3bsLc+fOrXf9f32d/fv3C/fv\n3xfu3bsnXL58WYiJiRG8vb2FWbNmKS2vr/fxwYMHKvPT09OF7t27Cx988IFS+7Rp04ShQ4cqtcXG\nxgre3t4NjrexNWVnZwve3t7C2rVrVZZ5/HuWnZ2tsv63335bZflTp07V+rkmJSUJ3t7ewtatWxVt\nS5YsEXr06CFcu3ZNadmcnByhd+/eQmxsbINjJGXctUUK69atw8CBAzFw4ECMHj0aBw4cwLBhw7B+\n/XqVZcPDw1X2S//73/9GdXU1XnrpJRQWFir9GzZsGORyOZKSkgAAiYmJKCoqwowZM+Dk5KTy+iYm\ndf9qWltbAwBOnDiBqqoqtcb4008/wd7eHpMmTVJqnzRpEuzt7XH8+HGVPi+//LLScRonJye4u7vj\n5s2baq174cKFGDhwIAYNGoTx48fj2LFjmDhxIpYtW6a0nL7eR0tLSwCPtjTLyspQWFgIOzs7uLu7\n49KlS2qNpT7afLaN8dprr6m0BQcHw8HBAd9//71S++HDh2FmZoZRo0YBeDT2I0eOIDAwEO3atVN6\nb8ViMXr16oXExESt6jNG3LVFCpMmTcLzzz8PkUgEsViMzp07K+0meFJtu0Fu3LgBAIiIiKhzHffu\n3QMAxZdwt27d1K4zLCwM//znP7Fp0yZs374d/v7+CA4ORlhYWIO7JHJycuDn56cSgmZmZujcuTOu\nXr2q0qdTp04qbba2trh9+7Ziuri4GA8fPlRa5vHuvsdmzpyJvn37orq6GpcuXcLmzZtRUFCgclaZ\nvt7Hq1evIi4uDikpKSgvL1ea5+Li0mD/xtLms22M2n73HofFtm3bkJmZCXd3d5SXl+PYsWMICgpS\nnChSWFiI4uJiJCYmYuDAgbW+vrZBZ4wYJKTg5uaGQYMGNWpZCwsLlTbhz0fbfPLJJ2jXrl2t/Wr7\nUlaXubk5tm3bhkuXLuGXX37Br7/+irVr1yI+Ph6fffYZQkNDtV7HkxrzxRITE4OUlBSltuvXrytN\ne3t7K97fIUOGwNPTE/PmzcPatWvxzjvvKJbTx/t4584dTJ06FVZWVnjjjTfg4eEBsVgMkUiEZcuW\nqQRLUxCJRHXOq6mpqXOeWCyutX3MmDHYtm0bDh8+jLlz5+LYsWMoLy/HmDFjFMs8fm8HDRqEqKgo\nDSunv2KQkM48/kvRzs6uwUByd3cHAFy7dg3BwcEara9nz57o2bMnACA3NxdjxozB559/Xm+QdOrU\nCZmZmaipqVHaKqmpqcHNmzc1DrrY2FhIpVK1+oSFhWHPnj3Yvn07Jk+erNgq0Mf7+NNPP6G8vBwb\nN27EgAEDlOYVFxdrfIq1NjXZ2NgAAEpKSlTmPXm2VmP5+vrC19cXR44cwZw5c/D9999DIpHg2Wef\nVSxjb28PiUSCsrKyRv/RRA3jNhzpzMiRI2Fubo5169ahsrJSZX5paSmqq6sBAEFBQbCzs8O2bduQ\nn5+vsqxQz4M7aztVtn379rC3t6/1S+lJw4cPR2FhIfbt26fU/t1336GwsBDDhw+vt39d/Pz8MGjQ\nIKV/jTFz5kw8fPgQGzduVLTp4318fI3KX9/X7777DgUFBY0bZCM1tiYrKys4OjoiOTlZqa7s7Oxa\nj1U1xtixY3H79m0cOXIEycnJGDlypNL1TSYmJhg1ahQuXbpU54Wu9+/f12jdxoxbJKQz7du3x+LF\ni/H+++/jhRdewIsvvoiOHTuisLAQf/zxB44fP45//etfcHFxgVgsxscff4zZs2dj1KhRilNECwsL\nkZiYiIiIiDq/1Ddu3IgzZ85gyJAhcHFxgSAIOHXqFDIyMhAZGVlvjZGRkfjxxx/x0Ucf4erVq+ja\ntSuuXbuG/fv3w93dvcH+ujZgwAD06dMHhw8fRnR0NDp16qSX9zEkJARisRjvvvsupk2bBolEgvPn\nzyMhIQGurq6QyWQ6G5M6n+3UqVPx+eefIzIyEsOHD0d+fj727NmDLl264PLly2qve9SoUVi5ciU+\n/PBDyOVyjB07VmWZuXPn4vz585gzZw5GjhwJf39/tGrVCnfu3EFCQgK6d++OFStWaP0+GBMGCenU\n+PHj0blzZ2zduhV79+5FaWkpbG1t4e7ujtmzZysdgH722Wexa9cubNq0Cfv378eDBw/g4OCAgICA\nem8OOHz4cBQUFODHH3/EvXv3YGFhATc3NyxduhQvvfRSvfVZW1tj9+7dWLt2LU6ePImDBw+ibdu2\nmDx5MmJiYlSuIWkKb775JiIjI7FhwwYsX74cgO7fR1dXV3z11VdYvXo1Nm3aBFNTU/Tp0wfffPMN\nlixZonTigC409rONiopCaWkp/vnPfyIlJQVeXl74+OOPceXKFY2CpG3btnjmmWdw6tQpdO7cGb17\n91ZZ5vHvwNatW/Hjjz/ixIkTMDU1Rfv27REQEIAJEyZoNXZjJBLq24dARETUAB4jISIirTBIiIhI\nKwwSIiLSCoOEiIi0YnRnbRUUlGrc187OEkVFTX8FsCFxzMaBYzYO2ozZ0dG6znncIlGDmZnxPXiI\nYzYOHLNx0NeYGSRERKQVBgkREWmFQUJERFoxaJDcunULixYtwqhRo9C1a1eV53LXpbS0FAsWLEBg\nYCACAgLw9ttvqzxvm4iImoZBgyQtLQ2nT5+Gu7u7Ws+LnjNnDs6dO4elS5dixYoV+P333zFz5kz9\nFUpERHUy6Om/w4YNU9wFdNasWY3aqvjvf/+LxMRE7Ny5E4GBgQAePfp0woQJSEpK4jMGiIiamEG3\nSDR5pGVCQgIcHBwUIQI8esCRi4sLEhISdFkeERE1Qou7IDEjIwMeHh4q7Z6ensjIyNDbenPyy7Dr\nRDoelFfrbR36ZGttjvEhnjAxqfvxpkREmmhxQSKVSmFtrXqFpUQiadTjOe3sLDW6KOfstXwc/0+W\n2v2ak7BgT7g51X11al3qu6L1acUxGweOWTdaXJBoS9PbAwzs2g5B/s+hoKBMxxXp3+HEDJy5nId7\n98tgaabeFomjo7VWt5VpiThm48Axq9+3Li0uSCQSSa3P7JZKpZBIJHpdd1sbMeTVNXpdhz6IzVvc\nx0xELUiLuyDRw8MDmZmZKu11HTshIiL9anFBEhISgoKCAvz666+KtsuXLyM7OxshISEGrIyIyDgZ\ndJ9HRUUFTp8+DQC4e/cuysrK8OOPPwIABg8eDLFYjNDQUAQGBmLZsmUAgN69eyM4OBixsbGIjY2F\niYkJVq5ciYCAAF5DQkRkAAYNkvv372P27NlKbY+nT5w4ARcXF8hkMsjlcqVl1qxZg+XLl2PhwoWQ\ny+UYOnQo3nvvvSarm4iI/segQeLi4oLr16/Xu8zJkydV2iQSCZYvX47ly5frqzQiImqkFneMhIiI\nmhcGCRERaYVBQkREWmGQEBGRVhgkRESkFQYJERFphUFCRERaYZAQEZFWGCRERKQVBgkREWmFQUJE\nRFphkBARkVYYJEREpBUGCRERaYVBQkREWmGQEBGRVhgkRESkFQYJUQsgfVCN61lFKC6rUrRVPZSh\nRiavpxdR0zDoo3aJ6H+qH8pw444UV28W4mZeKfLuP8B9aVWD/UL8ndG/mxM8nCVo3cq0CSolUsYg\nIWpiJQ+q8Ud2MVKzinD1ZhHuFpZr9XoJF+8g4eIdAECgbzv8/cXuMDER6aJUokZhkFCzIZPLcTOv\nFLZtWqOtjYWhy9FaRVUNrt4swvXsIqTeKkZOQVmDfRxsLODs0AbenWzh62oHVycrmJk+2gN9r6QC\nV28Woa+PIywtWuFWXim+/ekPpN8uUfT/T2o+QgM7waujjd7GRfRXDBIyqPsllfjp12wc+0+2Uvva\n2c/AStzKQFWpRxAE5BdX4NfUfPyRXYLrWUWorqn/2EWndlZw72ANz4428HNvC1src4hE9W9FONiI\nEeIvVky7tbfGwukBAICHNTL8fdVpAMDZ3/MYJNSkGCTUpARBQNbdMuw+/gf+yCmpc7nF21Kw6s2g\nJqzskaLSKqTllkJsJoKLoxWqHspwJbMQv6bmo7W5Kfw9HeBga4HkK3eRmlWEjDvSel/PStwKvbwc\n4N3JFt3d7WFn3VovdbcyM0U7WzHyiytw6r+3MaJfJ7Szs9TLuhpDLgioqpbBwtwUIpEIgiCgrOIh\n8grLUVBcgdz75ZDLBbg6WaN/NyeD1Um6wSChJpF6qwgHTt/AjVq+eF3bWSE0sBMGdHfCxsNXcP6P\nAtTIBL3XJJcLuJZVhOTf83Ah/R4eVNY02Of0hTt1znNtZ4U+Po7o4dEWbu2tYdLAFoauzZ/WB/Pi\nzzz6+YtkAIBNG3PMmeAPt/bWOl9fZXUNMnNLkXGnBHn3y1H1UIY798tx594DpeUkbcwhfVBd62uY\ntzJhkDwFGCSkN7fvPcCun/7AtVtFKvN8XW3x0hAveDhLlNrfGtcDC744i8qHMrXXV1FVgzv3lb/E\n7hQ8wK/XC2BnbY7hfTsh624pUq7l49KN+41+3UdbIm2Rci1f0ebpLEGgbzv08naEo41Fg7ulmoKt\nVWtMGuaFvSfTFW0lD6rx4fb/AAA+juoPa0tzpN4qQlpOCW7fK4O9tQWGBXSEo2PtQVNRVYObeaW4\nnlWEG3ekuFtYjnsllWrVJX1QDTNTE4hbm6KnZ1uYmZrA2aENdh9PQ/VDOT7fdxEd2lrC2aENJJbm\ncLK3RE5+GSr/3KIJ8HFsFu8v1Y1BQjpVVvEQ351KR+KlXJV53Trb4ZXnfdHOVlxLT/XJ5HJcu1mE\n3/4oQDtbMS6m36t3d1nCRdWaAKCPtyOC/Nqjx59fcqXVchw6lYZxIR5Kx2leHyUAIjT5loY6RvRz\nRVCPDiireIj/pObjUEKGYt57X52rtY9MLodDWyskXbiN7LuluHqrCEWlDZ92DDza4unc/tGxHs+O\nNujo+CgMAOBWXilam5vCwcZCccLAk3YfTwMAXLpxv95gNzMVwaODBH/klGBgdyeM7O+G4gdVqKqW\nw72DNSqrZcgvrkBVtQxeHW2eihM1WhoGCWntXnEF/pOajyNJN1FZrbwl4dPJFi8N9YSns24O/haX\nVeFYSjZOX7yDiirVXVFmpiYI7eui1HbtVhFu5pUCAExNRBjapyMGdGsP9w7Wtf6l69HRBq+M8FFp\nbymn1FqJW8FK3AqjBnXG0N4d8XvGfXx55KpivoONBbw62sBeYoGjybdw9spdnL1yt87X69DWEu3t\nLdG5vTV8XO3g1t66UderNLQ7bVFEX1zJLMSB0xkQtzZFRZVMsb7une1x/LccAECNTFD8gdBQrX4e\n9pg3sVeDtZFuGTRI0tPTsWTJEly4cAHW1taYMGEC3nrrLZia1v9LevnyZaxZswa///47AKBbt26Y\nO3cu/P39m6JsAlAjk6O4rAqHf8lE0u95SvNMTUSIGtUNgb7tdLJLIjNXih/OZeHX1PwGlx3s74wJ\nQ71U2kseVMOmjbnWtbQ0VuJWGNC9veI4xJOfR3llDY4m3wLw6K9+T2cbeHeyRecO1vDuZIs2Fvo9\na65zewk6t5cgbGDnWuf/Lagz/t+ZmxjcyxkOtmK8ufo0BAFwtLVAe/s2uJzxaCvGy8UGtm3M8dv1\nAlRWqb9LlLRnsCApKSlBREQEvLy8sGHDBmRlZeGTTz6BXC7H3Llz6+yXm5uLGTNmoFu3bvj0008B\nAFu2bMGMGTNw5MgRdOzYsamGYLQKiisQu+msSvus8T3RpZON1l9AAoCSsmos3pqCrHzVay8klq3w\nYrA7gnp0QOtWpqh6KMPZ3/MQ3LNDrbtQABhliDyptkC3tDDD1vnDAACOjtYoKCht6rLqJbE0x8uh\n3orpLbHD6l3+1RUnkX67BFl3S+HqpPuTC6huBguSPXv2oKqqCvHx8bCyskJQUBDKysoQHx+PqKgo\nWFlZ1drv559/xoMHD7B+/XpYWz/6ZenduzcGDBiA06dP4+WXX27KYRiVa7eKsHL3f5Xaenk5YEB3\nJ51tfQBAflEFACiFSFc3O/xtoBt83exU1tO6lSmG9OYfEPTI4m3/gZOdGPOnBRj9HxBNxWBBkpCQ\ngODgYKXACAsLw6pVq5CSkoJhw2r/66OmpgampqYQi/93wNbS0hKmpqYQBP2fMmqM/sguxopvzyu1\ndWhriTkT/OGoowPntfH3bIuxIR7865IaZclr/fB/W1IAAHeLKhB/8BLem97XwFUZB4MFSUZGBgYM\nGKDU5uzsDLFYjIyMjDqD5LnnnsPatWuxYsUKvPHGGwCA9evXw8bGBiNHjtR73cbkSmYhPltxUqmt\ni4sNZo7tAYke/9J7vLuFSB0dHa2wJXYoln3zG27ckeLGbSkeVD7U+7EeMmCQSKVSxa6pJ0kkEkil\ndV8t7OTkhB07duDvf/87vvnmGwCAo6MjtmzZAnt7+wbXa2dnCTMzze+QWtf59s3Z+bQCAMCafRfx\nzeLnG9wFlZUnxcyVp5Tagno6462JvVrMbUu01RI/Z209LWNeM28I3vjkJG4XlKFtWytYW9b9R8/T\nMmZ16GPMLe703/z8fMyePRvdu3fH0qVLAQC7du3C66+/jj179sDZ2bne/kVFmt9ptTkekGyMwj9v\nRV5SVo38gtI6r4OQlldj7f5LSrf98HGzQ8zYHrC0MENFWSUqytS7GK0laqmfszaetjE72Ylxu6AM\n9+6VobKOP36etjE3hjZjri+ADBYkEokEZWWqZ+RIpVJIJJJaejyyZcsW1NTUYO3atWjV6tEvyIAB\nAzBixAhs3boV77//vt5qbqlWvjEI/9iYVOd8mVyOvSfTcfzXHEWbk50YsVP7oIu7g9H9n42I1GOw\nIPHw8EBGRoZSW25uLioqKuDh4VFnv4yMDHh5eSlCBADMzc3h5eWFrKwsvdXbkrW1sYCvqy1Ss4pV\n5t24U4KPd/ym1LZ4RiAPcFOL9vjJkRVVNUazO9aQDPao3ZCQECQmJiptlRw9ehQWFhbo169fnf2c\nnZ2RlpaG6ur/3QSuuroaaWlpvIZEDZXVNVjz3UWlEBkX4oEtsUMZItTiPb7lSuyms8ip5Vok0i2D\nBcnkyZNhbm6OmJgYJCUlYe/evYiPj0dERITSKcGhoaFYuHChYnrChAnIz8/HW2+9hZ9//hmnTp3C\nm2++iYKCAkyaNMkQQ2lxrtwsxJurExRXBrs4WmHt7Gfwt0GdeXM8eiqMfcZd8fOirSnI/cvNPEm3\nDBYkNjY22L59O2QyGaKjo7Fu3TqEh4dj1qxZSsvJZDLI5f97SJCfnx82b96MBw8e4N1330VsbCwq\nKyuxdetW+Pr6NvUwWpwt/+8aPttzQTE9d6I/PnqtHzf/6akyKsgdMeN6KKbf++ocUmu5CzXphkHP\n2vLy8sKOHTvqXebkyZMqbQMHDsTAgQP1VdZT6fGtv89eeXRfLCd7SyyeEdiom+8RtUS9vR0xsLuT\n4iaP+cUV8Ha1bdZ3b26pDLZFQk3ryWdIvPK8D5a/PoAhQk+9qFHdMaJfJwDA9h9S8cHWFN4BQw9a\n3HUkpJ0V0QN19jwQopbAyf5/jxy+XfAAMrkAM1NuleiSSDCyeNbmmghewGQcOOan06tP3O5n4lAv\nTP9b96d+zH+lrwsSuWuLiIxOxp26n6RJ6mOQEJFR2Dp/GD6PCTZ0GU8lBgkRGQ8eGtELBgkREWmF\nQUJERFphkBARkVYYJEREpBUGCRERaYVBQkREWmGQEBGRVhgkRESkFQYJERFphUFCRERaYZAQEZFW\nGCRERKQVBgkREWlFoyck3rhxAwcOHEBOTg5KSkpUHl0pEonw9ddf66RAIiJq3tQOksOHD2PhwoUw\nMzODu7s7JBKJyjJG9tBFIiKjpnaQxMfHw9fXF5s3b4a9vb0+aiIiohZE7WMk+fn5eOmllxgiREQE\nQIMg8fb2RkFBgT5qISKiFkjtIImNjcW+fftw8eJFfdRDREQtjNrHSL7++mvY2Nhg8uTJ8PHxQYcO\nHWBqaqq0jEgkwrp163RWJBERNV9qB8nVq1cBAB06dIBUKoVUKlVZRiQSNeq10tPTsWTJEly4cAHW\n1taYMGEC3nrrLZVgqs2xY8fwxRdfIC0tDWKxGH5+fli3bh0sLS3VGxAREWlF7SA5efKkTlZcUlKC\niIgIeHl5YcOGDcjKysInn3wCuVyOuXPn1tt33759+OijjxAZGYl3330XUqkUycnJkMlkOqmNiIga\nT6MLEnVhz549qKqqQnx8PKysrBAUFISysjLEx8cjKioKVlZWtfYrLCzEsmXL8H//93+YOHGioj00\nNLSpSicioidoHCQ///wzfv75Z9y5cwcA4OzsjKFDh2Lw4MGN6p+QkIDg4GClwAgLC8OqVauQkpKC\nYcOG1drvhx9+AACMGTNG09KJiEiH1D5rq7KyEq+99hreeOMN7N+/Hzdu3MCNGzewf/9+REdHIzIy\nEpWVlQ2+TkZGBjw8PJTanJ2dIRaLkZGRUWe/S5cuwd3dHfv370dISAi6d++OCRMm4Pz58+oOhYiI\ndEDtLZK4uDgkJSVh7ty5mDZtmuLgdkVFBb799lusXr0acXFxiI2Nrfd1pFIprK2tVdolEkmtB/Af\nu3fvHjIzM7Fx40b84x//gK2tLTZv3ozIyEgcO3YMDg4O9a7Xzs4SZmYNH8yvi6Ojas1PO47ZOBjD\nmM3LqgAArVu3AmAcY/4rfYxZ7SA5evQoJk2ahNdff12pXSwWIzIyEtnZ2Th69GiDQaIpQRBQXl6O\nuLg4hISEAAD69OmDoUOHYufOnZgzZ069/YuKyjVet6OjNQoKSjXu3xJxzMbBWMYsLa8GAFRVPQQA\noxjzk7T5nOsLILV3bRUWFqJLly51zvf29kZhYWGDryORSFBWVqbSLpVKa70R5JP9RCIR+vfvr2iz\nsrJC9+7dcePGjQbXS0REuqV2kHTq1AmnT5+uc/7p06fRqVOnBl/Hw8ND5VhIbm4uKioqVI6dPMnT\n0xOCIKjcYVgQhEZfv0JERLqjdpC8/PLLSEhIQHR0NM6ePYu8vDzk5eXh7NmzeOONN/DLL79g6tSp\nDb5OSEgIEhMTlbZKjh49CgsLC/Tr16/OfkOGDAEAnDt3TtFWWlqKK1euwNfXV93hEBGRltQ+RjJt\n2jQUFRXhyy+/VNkyMTMzw8yZMxsVJJMnT8Y333yDmJgYREVFITs7G/Hx8YiIiFA6JTg0NBSBgYFY\ntmwZAKBHjx549tln8d577+FYC3tcAAAdb0lEQVTtt9+GnZ0dNm/eDDMzs0atl4iIdEuj60hiYmIw\ndepUnD17Frdv3wYAdOzYEQMHDmz07eVtbGywfft2fPTRR4iOjoZEIkF4eDhiYmKUlpPJZJDL5Upt\nK1euxKeffooVK1agoqICffr0UdwDjIiImpZIMLLHGWpzloaxnNnyJI7ZOBjLmKXl1ZizNhEAsGn+\nszCHUX396e2srQa3SJ68cv3J6YY8Xp6IqLkweeKEnMQLtzGsF7+ndKHBIBk2bBhEIhEuXrwIc3Nz\nxXRDrl27ppMCiYh0xUrcCi8GdcY/z9yE3Lg2RvSqwSBZtmwZRCIRWrVqpTRNRNQS+XSyNXQJT50G\ng2TcuHH1ThMRkXHTyW3kBUHAuXPnUFVVhYCAgDpvAU9ERE8ftYNkzZo1OH/+PL755htFW2RkJJKS\nkiAIApydnbF9+3a4urrqtFAiImqe1L6y/d///jf8/PwU08ePH8eZM2fw9ttvY/PmzZDL5XxeOxGR\nEVF7i+Tu3bvo3LmzYvqnn35Cly5dEBkZCQCYMmUKdu3apbMCiYioeVN7i8TMzEzp2ehJSUkIDg5W\nTNvb26OoqEg31RERUbOndpB06dIFP/30E4BHd/q9d++e0uN1b9++DTs7O91VSEREzZrau7ZmzpyJ\n6OhoBAYGKu5zNWDAAMX8hIQE9OjRQ6dFEhFR86V2kAQFBeHQoUM4c+YMJBIJXnjhBcW84uJi9O3b\nF88++6xOiyQiouZLo+tIvLy84OXlpdJua2uLhQsXal0UERG1HGofIyEiInpSg1skvr6+MDExwYUL\nF2Bubg5fX98G77UlEolw9epVnRVJRETNV4NBMnPmTIhEIpiZmSlNExERAY0Ikr8+sfCv00REZNx4\njISIiLSidpB8/PHHGDFiRJ3zR4wYgU8++USrooiIqOVQO0hOnz6NkSNH1jn/hRdewMmTJ7UqioiI\nWg61gyQvLw8dO3asc76zszPy8vK0KoqIiFoOtYNEIpEgMzOzzvkZGRlo06aNVkUREVHLoXaQPPPM\nM9izZw+uX7+uMi81NRV79uxRuhswERE93dS+Rcrs2bPxyy+/YPz48Rg+fLjiVilpaWk4ceIEbG1t\nMWfOHJ0XSkREzZPaQdK+fXscOHAAq1atwsmTJ/Hjjz8CANq0aYMXXngB8+bNQ/v27XVeKBERNU8a\n3bTRyckJK1euhCAIKCwsBPDogVa84p2IyPhodUFidXU1ampqYG1tzRAhIjJSGgVJcnIyJk+ejD59\n+mDIkCH47bffAACFhYWYPn06Tp8+3ajXSU9PR3h4OPz9/REcHIy4uDilx/g2RC6XY9y4cfDx8cGp\nU6c0GQoREWlJ7SBJSkrCa6+9hvLycrzyyisQBEExz97eHubm5ti/f3+Dr1NSUoKIiAiIRCJs2LAB\nM2fOxLZt27B27dpG17Jv3z7cvXtX3SEQEZEOqR0kcXFx6NGjBw4fPoyoqCiV+QEBAbhy5UqDr7Nn\nzx5UVVUhPj4eQUFBmDJlCmbOnInt27ejrKyswf4lJSVYs2YNzxAjIjIwtYMkNTUVo0aNgomJSa3H\nRRwdHXHv3r0GXychIQHBwcGwsrJStIWFhaGyshIpKSkN9o+Li0OfPn0wcOBA9QZAREQ6pXaQWFhY\noKqqqs75OTk5kEgkDb5ORkYGPDw8lNqcnZ0hFouRkZFRb9/U1FQcOHAAsbGxjSuaiIj0Ru3TfwMD\nA/H9998jIiJCZV5hYSH279+PoKCgBl9HKpXC2tpapV0ikUAqldbbd+nSpZg6dSrc3NyQk5PT6NoB\nwM7OEmZmpmr1eZKjo2rNTzuO2TgYy5jvFFcqfjaWMT9JH2NWO0jmzJmDyZMnY+LEiRgxYgREIhHO\nnj2L3377DXv27EFNTY1eH371r3/9C5mZmdi0aZNG/YuKyjVet6OjNQoKSjXu3xJxzMbBmMZcUvy/\n7wBjGfNj2nzO9QWQ2ru2vLy88O2336JNmzZYvXo1BEHAl19+ifj4eLi7u2Pnzp3o1KlTg68jkUhq\nPagulUrr3DX28OFDfPrpp4iKioJcLodUKlW8RkVFRaMO0hMRkW5pdGW7j48Pvv76a5SUlODWrVsQ\nBAGdOnWCvb19o1/Dw8ND5VhIbm4uKioqVI6dPFZRUYG8vDwsX74cy5cvV5o3d+5cuLq64qefflJ/\nQEREpDG1gqSiogLR0dF48cUXMX78eNjY2KBnz54arTgkJARbtmxBWVmZ4syto0ePwsLCAv369au1\nj6WlJXbs2KHUdu/ePcybNw/z5s3DgAEDNKqFiIg0p9auLbFYjCtXrqCmpkbrFU+ePBnm5uaIiYlB\nUlIS9u7di/j4eERERCidEhwaGoqFCxcCAMzMzNC/f3+lf/7+/gAAb29vxc9ERNR01D5GEhwcjKSk\nJK1XbGNjg+3bt0MmkyE6Ohrr1q1DeHg4Zs2apbScTCaDXC7Xen1ERKQfGj2PJCYmBgsWLMCkSZPg\n4uICCwsLleWe3Kqoi5eXl8quqr9q6PnvLi4utT5ki4iImobaQTJy5EgAj264ePjw4TqXu3btmuZV\nERFRi6F2kLz11lv6qIOIiFootYLkwoULcHd3h62tLfr27YvWrVvrqy4iImohGhUkZWVliIyMxMWL\nFxVtbdu2xaZNm+Dn56e34oiIqPlr1FlbX3zxBS5cuIDnnnsO77//PsLDw1FaWor58+fruz4iImrm\nGrVFcuLECTz//PP4/PPPFW2enp5YtGgRbt26BTc3N70VSEREzVujtkhu376NQYMGKbUFBwdDEATk\n5+frpTAiImoZGhUkVVVVEIvFSm2Prx3RxVXuRETUcjX6rK3c3FykpqYqpktLH92KOCcnR6n9MV9f\nXx2UR0REzV2jg2TNmjVYs2aNSvuiRYuUpgVBgEgk4gWJRERGolFB8tdbthMRET3WqCAZO3asvusg\nIqIWSu27/xIRET2JQUJERFphkBARkVYYJEREpBUGCRERaYVBQkREWmGQEBGRVhgkRESkFQYJERFp\nhUFCRERaYZAQEZFWGCRERKQVBgkREWmFQUJERFoxaJCkp6cjPDwc/v7+CA4ORlxcHGQyWb19Ll26\nhAULFiA0NBT+/v4YMWIE4uPjUVVV1URVExHRkxr9hERdKykpQUREBLy8vLBhwwZkZWXhk08+gVwu\nx9y5c+vs98MPPyArKwtRUVFwc3PD9evXERcXh+vXr2PdunVNOAIiIgIMGCR79uxBVVUV4uPjYWVl\nhaCgIJSVlSE+Ph5RUVGwsrKqtV9UVBTs7e0V0/3790fr1q2xaNEi3L59Gx07dmyqIRAREQy4aysh\nIQHBwcFKgREWFobKykqkpKTU2e/JEHmsa9euAID8/HzdF0pERPUyWJBkZGTAw8NDqc3Z2RlisRgZ\nGRlqvdaFCxdgYmICV1dXXZZIRESNYLAgkUqlsLa2VmmXSCSQSqWNfp2CggJs3LgRo0ePRtu2bXVZ\nIhERNYLBjpHoQnV1NebMmQNLS0ssWLCgUX3s7CxhZmaq8TodHVXD72nHMRsHYxnzneJKxc/GMuYn\n6WPMBgsSiUSCsrIylXapVAqJRNJgf0EQEBsbi/T0dOzatQs2NjaNWm9RUbnatT7m6GiNgoJSjfu3\nRByzcTCmMZcU/+87wFjG/Jg2n3N9AWSwIPHw8FA5FpKbm4uKigqVYye1+fjjj3HixAls3boVnp6e\n+iqTiIgaYLBjJCEhIUhMTFTaKjl69CgsLCzQr1+/evt+8cUX+Pbbb7Fy5Ur07dtX36USEVE9DBYk\nkydPhrm5OWJiYpCUlIS9e/ciPj4eERERSqcEh4aGYuHChYrpI0eOYPXq1RgzZgycnJxw4cIFxb/C\nwkJDDIWIyKgZbNeWjY0Ntm/fjo8++gjR0dGQSCQIDw9HTEyM0nIymQxyuVwxfebMGQDAwYMHcfDg\nQaVlly9fjnHjxum/eCIiUjDoWVteXl7YsWNHvcucPHlSaXrFihVYsWKFPssiIiI18O6/RGSUvjv+\nB6oe1n+TWGocBgkRGRWJVWsAQI1Mjms3iwxczdOBQUJERqWjQxs8F9gJACB74vgraY5BQkRGp62N\nBQBg/aHfUc3dW1pjkBCR0enmZqf4uaiUD8XTFoOEiIxOR0crPNffzdBlPDUYJEREpBUGCRERaYVB\nQkREWmGQEBGRVhgkRESkFQYJERFphUFCRERaYZAQEZFWGCRERKQVBgkREWmFQUJERFphkBCRUePD\nrbTHICEioyQSPfrv4m3/QVpOsWGLaeEYJERklEYO7Kz4ed+pG4Yr5CnAICEio+TpYouNbw8G8Oix\nu6Q5BgkRGa3WrUxh3opfg9riO0hExk0AbuaVouRBtaErabEYJEREAP7fmZuGLqHFYpAQkVGLHuMH\nADhxPgf7f74BuSAYuKKWh0FCREbNz90ebSzMAABHk28h847UwBW1PAwSIjJqZqYmiJv1DNw7SAAA\nZRUPDVxRy2PQIElPT0d4eDj8/f0RHByMuLg4yGQNX2VaWlqKBQsWIDAwEAEBAXj77bdRVFTUBBUT\n0dPIxESE3l0cAABx+y9h57HryCkog8DdXI1isCApKSlBREQERCIRNmzYgJkzZ2Lbtm1Yu3Ztg33n\nzJmDc+fOYenSpVixYgV+//13zJw5swmqJqKnVaBvO8XPJ8/fxqItKTiafMuAFbUcZoZa8Z49e1BV\nVYX4+HhYWVkhKCgIZWVliI+PR1RUFKysrGrt99///heJiYnYuXMnAgMDAQBOTk6YMGECkpKSMGjQ\noKYcBhE9JZzsLbEldijOXbuLxEu5uHqzCAdOZ+Cn/2Rj4fQAWFuao3UrU5iYiAxdarNjsCBJSEhA\ncHCwUmCEhYVh1apVSElJwbBhw+rs5+DgoAgRAOjZsydcXFyQkJDAICEijYlEIgzo1h59fdph6de/\nIiu/DNLyh5j/RfITywBd3exQVFoFT2cbmJqKYGIiQnFpFVwcrWBqKoKpiQimJiaofihD+7aWkMkF\nWIlbwUQkgono0XpMTEQwEYkgMgFM/rzxVyszk0fzRI/aRIr//vmziUgx38zUBKLHr/Xnfx9Pi/4c\nS1MxWJBkZGRgwIABSm3Ozs4Qi8XIyMioM0gyMjLg4eGh0u7p6YmMjAy91EpExsXM1ASLX+2H3PsP\n8PWP1yGxbIWi0ircuCMFBODqzUfHZHPvlyv1+2/aPUOUWyvRn//zOJBamZlg7pQAeLWvfW+PNgwW\nJFKpFNbW1irtEokEUmndp9/V1y8nJ6fB9drZWcLMzFS9Yp/g6Ki67qcdx2wcOOba53/m216pTRAE\nPKyRo7i0CjK5AJlcjqpqGSqqaiAAkMnkqJEJKCgqR41MQH5ROWytWkMuCJDLBcgF/PnfP6flAm4X\nlMHRTgzhiXnKPwuQyx+t+760EgBgaWEGQXjUJghQLPe/nx/1Bx5Nm5qIYGfdWi+fs8GCxFCKisob\nXqgOjo7WKCgo1WE1zR/HbBw4ZvWJ8OgL1EwEtG5tCklr5T9Q3RwstStQD7QZc30BZLCztiQSCcrK\nylTapVIpJBJJvf1KS1XfiIb6ERGRfhgsSDw8PFSOaeTm5qKioqLWYyBP9svMzFRpr+vYCRER6ZfB\ngiQkJASJiYlKWyVHjx6FhYUF+vXrV2+/goIC/Prrr4q2y5cvIzs7GyEhIXqtmYiIVBksSCZPngxz\nc3PExMQgKSkJe/fuRXx8PCIiIpROCQ4NDcXChQsV071790ZwcDBiY2Nx7NgxHD9+HO+88w4CAgJ4\n6i8RkQEYLEhsbGywfft2yGQyREdHY926dQgPD8esWbOUlpPJZJDLlZ9etmbNGvTr1w8LFy7Eu+++\nCz8/P8THxzdl+URE9CeRYGQ3k9HmLA2e2WIcOGbjwDGr37cuvPsvERFphUFCRERaMbpdW0REpFvc\nIiEiIq0wSIiISCsMEiIi0gqDhIiItMIgISIirTBIiIhIKwwSIiLSCoMEQHp6OsLDw+Hv74/g4GDE\nxcVBJpM12K+0tBQLFixAYGAgAgIC8Pbbb6OoqKgJKtaeJmO+dOkSFixYgNDQUPj7+2PEiBGIj49H\nVVVVE1WtHU0/58fkcjnGjRsHHx8fnDp1So+V6o42Yz527BjGjx+Pnj17on///njttddQXq75g+Ga\niqZjvnz5Ml599VX069cP/fr1Q0REBC5evNgEFWvn1q1bWLRoEUaNGoWuXbti+vTpjeqny+8vo3tC\n4l+VlJQgIiICXl5e2LBhA7KysvDJJ59ALpdj7ty59fadM2cOMjMzsXTpUpiYmGDVqlWYOXMmdu3a\n1UTVa0bTMf/www/IyspCVFQU3NzccP36dcTFxeH69etYt25dE45Afdp8zo/t27cPd+/e1XOluqPN\nmPft24ePPvoIkZGRePfddyGVSpGcnKxW8BqCpmPOzc3FjBkz0K1bN3z66acAgC1btmDGjBk4cuQI\nOnbs2FRDUFtaWhpOnz4Nf39/1NTUNLqfTr+/BCO3adMmoW/fvkJpaami7csvvxR69uyp1PZX58+f\nF7y9vYWUlBRF28WLFwVvb2/hzJkzeq1ZW5qO+f79+ypte/bsEby9vYWcnBy91Kormo75seLiYqF/\n//7Cd999J3h7ewsnT57UZ7k6oc3n3KtXL2Hv3r1NUaZOaTrmXbt2Cb6+voJUKlW0FRcXC76+vsK3\n336r15q1JZPJFD/HxMQI06ZNa7CPrr+/jH7XVkJCAoKDg5WegRIWFobKykqkpKTU28/BwQGBgYGK\ntp49e8LFxQUJCQl6rVlbmo7Z3t5epa1r164AgPz8fN0XqkOajvmxuLg49OnTBwMHDtRnmTql6Zh/\n+OEHAMCYMWP0XqOuaTrmmpoamJqaQiwWK9osLS1hamoKoZnfRcrERP2vcV1/fxl9kNT2iF5nZ2eI\nxWKVRwE31A8APD096+3XHGg65tpcuHABJiYmcHV11WWJOqfNmFNTU3HgwAHExsbqs0Sd03TMly5d\ngru7O/bv34+QkBB0794dEyZMwPnz5/VdstY0HfNzzz0HsViMFStW4P79+7h//z6WL18OGxsbjBw5\nUt9lNzldf38ZfZBIpVJYW6veZ18ikUAqleq8X3Ogq9oLCgqwceNGjB49Gm3bttVliTqnzZiXLl2K\nqVOnws3NTV/l6YWmY7537x4yMzOxceNGvPPOO9i4cSPEYjEiIyNx7949fZasNU3H7OTkhB07duDY\nsWMYNGgQBg0ahGPHjmHLli21bom3dLr+/jL6ICHNVFdXY86cObC0tMSCBQsMXY7e/Otf/0JmZibe\nfPNNQ5fSZARBQHl5OT7++GO8+OKLCAkJwYYNG2BqaoqdO3caujy9yM/Px+zZs9G9e3d89dVX+Oqr\nr+Dn54fXX38dd+7cMXR5zZ7RB4lEIkFZWZlKu1QqhUQiqbdfaanqk8Ya6tccaDrmxwRBQGxsLNLT\n0/Hll1/CxsZGH2XqlCZjfvjwIT799FNERUVBLpdDKpUqXqOioqLW12tOtPndFolE6N+/v6LNysoK\n3bt3x40bN/RSq65oOuYtW7agpqYGa9euRUhICEJCQrB27VqYmppi69at+izZIHT9/WX0QeLh4aGy\nTzA3NxcVFRW17kN8sl9mZqZKe137HpsTTcf82Mcff4wTJ05g/fr18PT01FeZOqXJmCsqKpCXl4fl\ny5cjMDAQgYGBGD16NABg7ty5GDt2rN7r1oamn7OnpycEQVA5yCwIAkQikV5q1RVNx5yRkQEvLy+0\natVK0WZubg4vLy9kZWXprV5D0fX3l9EHSUhICBITE5X+ijl69CgsLCzQr1+/evsVFBTg119/VbRd\nvnwZ2dnZCAkJ0WvN2tJ0zADwxRdf4Ntvv8XKlSvRt29ffZeqM5qM2dLSEjt27FD6t3r1agDAvHnz\nsGrVqiapXVOafs5DhgwBAJw7d07RVlpaiitXrsDX11dv9eqCpmN2dnZGWloaqqurFW3V1dVIS0tr\n1teQaErX31+mixcvXqzD+lqcLl26YO/evTh37hzatWuHpKQkrF69GuHh4Rg8eLBiudDQUKSmpuLZ\nZ58FAHTo0AEXLlzA/v370aFDB2RmZmLx4sXw9PTEnDlzDDWcRtF0zEeOHMGHH36IsWPHon///sjL\ny1P8Mzc3Vzp1srnRZMwmJiZwcXFR+vc4XMLDwzFgwAADjqhhmn7OTk5OuHbtGnbv3g07OzvcvXsX\nS5YsQXFxMT799FNYWFgYakgN0nTMjo6O+Prrr/H777/D2toamZmZWLFiBa5fv46PPvoIDg4OhhpS\ngyoqKnDixAmkp6cjMTERJSUlaNu2LdLT09GxY0e0atVK/99fal958hRKS0sTpk+fLvTo0UMICgoS\n1qxZI9TU1CgtM3ToUCE2NlapraSkRJg/f74QEBAg9O7dW5g3b16tF+01R5qMOTY2VvD29q7134ED\nB5p6CGrT9HN+UnZ2dou5IFEQNB9zWVmZsGjRIqFfv35Cjx49hPDwcCE1NbUpS9eYpmNOSkoSXn75\nZSEwMFAIDAwUpk6dKiQnJzdl6Rp5/DtZ27/s7GxBEPT//cVnthMRkVaM/hgJERFph0FCRERaYZAQ\nEZFWGCRERKQVBgkREWmFQUJERFphkBA9RaZPn670qNVz587Bx8dH6Sp1Il0z+kftEqnj4MGDSnc7\nNjMzg6OjI4YOHYo5c+a0iBtYEukag4RIA3PnzkWHDh1QWVmJc+fOYdeuXYrbijT3GxsS6RqDhEgD\ngwcPVjxmeNKkSQAePbvk8uXL6NmzpyFLI2pyPEZCpAMBAQEAoHLL8QMHDmDs2LHo2bMn+vfvj9jY\n2FqfMvjzzz9j6tSp6N27NwICAjB58mQcP35cMf/48eN4/fXXERwcDD8/PwwfPhzr16+HTCbT78CI\nGoFbJEQ6cPv2bQBQeihQfHw81q9fj7CwMEycOBEFBQXYsWMHLl++jIMHDyruortv3z68//778PX1\nRXR0NNq0aYMrV67gzJkzGD58OADg0KFDsLS0xIwZM2BpaYnk5GSsXbsWZWVlLe5Z8vT0YZAQaUAq\nlaKwsBCVlZVISUnB7t27YW9vj8DAQABATk4ONmzYgH/84x949dVXFf1CQkIwefJkHDp0CFOmTEFp\naSmWLVuG3r17Y8eOHTA3N1cs++T9VD/77DOl27dPmTIFixYtwu7duzF37lylfkRNjUFCpIFXXnlF\nadrb2xvLli1TPJPl+PHjEAQBoaGhKCwsVCzn6uoKR0dHpKSkYMqUKUhMTER5eTn+/ve/q4TBkwft\nnwyRsrIyVFdXo2/fvti7dy8yMjKa/QOn6OnGICHSwIcffghXV1cUFxdjz549SE9PV3qw182bNyGX\nyxW7pv7qcbhkZ2cDePRApvqkpaXh888/R3JyssozyWt79jZRU2KQEGnA399fcdbWs88+izFjxuCd\nd97BwYMHYWJiArlcDlNTU3z11Ve1ng785LGUhkilUkybNg1WVlaYNWsWXF1d0bp1a1y5cgWrVq2C\nXC7X2biINMEgIdJS69at8dZbb2HevHn44YcfEBYWBldXV8hkMri5ucHFxaXOvq6urgAebXHUtVxK\nSgqKi4sRHx+vOAYDPDoOQ9Qc8PRfIh14/vnn0bFjR3z11VcAHj0T3MTEBOvXr1dZVi6Xo7i4GAAQ\nFBQES0tLfPHFF6iurlZa7vHBdhMTE6VpAKiursauXbv0MhYidXGLhEgHTE1N8corr2D58uVISEhA\nSEgIZs2ahc8//xzZ2dkYOnQoxGIxsrOz8e9//xtvvPEGJkyYAGtra8yfPx+LFi3ChAkTEBYWhjZt\n2uDq1aswNzfHBx98gN69e8PGxgbz58/H9OnTIRKJ8P3334NPyabmglskRDry0ksvwdraWrFV8sYb\nb2DNmjWoqanBunXrsHLlSiQkJGD48OEYNGiQot+kSZOwfv16WFhYYP369VizZg1u3LiB4OBgAICd\nnR02bdoER0dHfP7559iyZQsGDRqEf/zjHwYZJ9FfiQT+WUNERFrgFgkREWmFQUJERFphkBARkVYY\nJEREpBUGCRERaYVBQkREWmGQEBGRVhgkRESkFQYJERFp5f8DU4NOhELtu98AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.step(recall, precision, where='post')\n",
    "plt.title('Precision-Recall curve')\n",
    "plt.xlabel('Recall')\n",
    "plt.ylabel('Precision')\n",
    "pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### We remember that we should test on the hold-out set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "n = int(best[0])\n",
    "n, y_score = fit(n, X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The results are markedly less impressive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.006965112161324013"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "average_precision_score(y_test, y_score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "precision, recall, thresholds = precision_recall_curve(y_test, y_score)\n",
    "fpr, tpr, thresholds = roc_curve(y_test, y_score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEnCAYAAACDhcU8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XlYVGX/P/D3sMkgDIKSioiCiJgo\nKYKKPLiSGV9TMxVTAwt6KMOthbS+Vu5bKoJL5ZaZYi7Z4y8zc0lCUn7l4/K4PSAkYBgIwoDsM+f7\nBzIxDtvMmWFQ3q/r8pJzn3Of87lnuObNWeYciSAIAoiIiHRkYuwCiIjo8cYgISIiURgkREQkCoOE\niIhEYZAQEZEoDBIiIhKFQULN3vDhwzF9+nSt+x06dAg9evTA+fPnDVDVk2369OkYPny4Wtv777+P\nHj16GKkias7MjF0AGd/58+fxyiuvqLVZWVnBxcUFY8eOxbRp02Bqamqk6h5/hw4dwvz581XTEokE\nrVu3Rvfu3TFp0iS8+OKLRqyOSDwGCan8z//8DwICAiAIArKzs/Htt99i2bJlSElJweLFi41W17Fj\nx3TqN3bsWAQFBcHc3FzPFelm+vTp6N27NwRBQFZWFvbv34/58+cjOzsbERERxi6PSGcMElJ5+umn\nMXbsWNX0yy+/jNGjR2P//v2YPXs22rVrV2u/iooKKJVKtGrVyiB1WVhY6NTP1NS0We1J9e/fH889\n95xq+sUXX8Rzzz2HrVu3Ijw8vFnV+rgoLS2FmZkZzMz4UWZMPEdCdbK2tkbfvn0hCAIyMjIAADEx\nMejRoweSk5OxfPlyBAQEoE+fPrh48aKqX2JiIl599VX0798fvXv3xpgxY7B3795at3Ht2jXMmjUL\nfn5+8PT0xJAhQzBv3jykp6erlqntHMmFCxcQFhaGwYMHo3fv3vjHP/6B8PBwtTrqOkeSl5eHTz75\nBEOGDFFt85NPPsH9+/fVlqvu/+uvv2Lbtm0YOXIkPD09MWrUKHz77be6vag1tG/fHq6urigsLERe\nXp7GfH2/jkePHkVERASGDh0KT09PDBgwAG+++SZu3Lgheiy61JSZmYkePXogJiZGo2/171lmZqaq\nrfocTV5eHubPnw8/Pz8888wzSE5ORu/evfHWW2/VWsenn36KHj164Pr166q2wsJCrF69GoGBgfD0\n9MTAgQMxb9481e85aYcxTnUSBAG3b98GANjZ2anNe+edd2BpaYlXX30VAODg4AAA2LdvHz766CM8\n88wziIiIgFQqRWJiIj7++GOkp6cjKipKtY7Tp08jMjISVlZWeOmll9ClSxfk5OQgISEB//3vf+Hs\n7FxrXampqXj11VfRrl07vPLKK2jbti1yc3Px+++/48aNG3jmmWfqHFNhYSGmTJmC27dvY8KECXj6\n6adx/fp17N27F+fOncP+/fthbW2t1mfdunUoLS3F5MmTYWFhgb179+L999+Hs7MzvL29tX9hH6qo\nqEBWVhZMTEwgk8nU5hniddy9ezfatGmDSZMmwcHBAenp6fjmm28wZcoUfPvtt+jatavOY3mUru9t\nY8yYMQPt2rXDm2++ieLiYrRv3x7Dhw/HyZMnkZ+fjzZt2qiWVSqVOHLkCHr06IGePXsCqPodCA4O\nxp9//okJEyage/fuyMnJwZ49ezBx4kQcPHgQnTp1Ev0atCgCtXjnzp0T3N3dhZiYGCE3N1fIzc0V\nrl+/LnzwwQeCu7u7MGnSJNWyGzZsENzd3YVp06YJFRUVauv566+/BE9PT2HevHka21i8eLHg4eEh\npKenC4IgCMXFxcKAAQOEgQMHCnfv3tVYXqFQqH4eNmyYMG3aNNX0l19+Kbi7uwuXLl2qd1wHDx4U\n3N3dhXPnzqna1q5dK7i7uwu7d+9WW3b37t2Cu7u7sG7dOo3+Y8eOFcrKylTtd+/eFXr16iXMnTu3\n3u0/up4DBw4Iubm5wr1794QrV64IkZGRgru7uzBr1iy15Q31Oj548EBjfkpKitCrVy/ho48+Umuf\nNm2aMGzYMLW2qKgowd3dvcHxNramjIwMwd3dXdiwYYPGMtW/ZxkZGRrbf/vttzWWP336dK3va2Ji\nouDu7i5s375d1bZ48WKhd+/ewvXr19WWzczMFPr27StERUU1OEZSx0NbpBITE4NBgwZh0KBBGDt2\nLA4ePIjhw4dj48aNGsuGhIRoHJf+8ccfUV5ejpdeegl5eXlq/4YPHw6lUonExEQAQEJCAu7fv48Z\nM2agffv2Gus3Man7V9PGxgYAcPLkSZSVlWk1xp9++gn29vaYPHmyWvvkyZNhb2+PEydOaPR5+eWX\n1c7TtG/fHi4uLvjjjz+02vaCBQswaNAg+Pn5YcKECTh+/DgmTZqEZcuWqS1nqNfRysoKQNWeZlFR\nEfLy8mBnZwcXFxdcvnxZq7HUR8x72xivvfaaRpu/vz/atWuH7777Tq398OHDMDMzw5gxYwBUjf3I\nkSPw8fHBU089pfbaSqVSPPPMM0hISBBVX0vEQ1ukMnnyZDz33HOQSCSQSqXo2rWr2mGCmmo7DHLr\n1i0AQGhoaJ3buHfvHgCoPoSffvppresMCgrCv/71L2zZsgU7d+6El5cX/P39ERQU1OAhiczMTHh6\nemqEoJmZGbp27Ypr165p9OncubNGW5s2bXDnzh3VdH5+PioqKtSWqT7cV23mzJno378/ysvLcfny\nZWzduhU5OTkaV5UZ6nW8du0aoqOjkZSUhOLiYrV5Tk5ODfZvLDHvbWPU9rtXHRY7duxAWloaXFxc\nUFxcjOPHj2Pw4MGqC0Xy8vKQn5+PhIQEDBo0qNb1iw26lohBQipdunSBn59fo5a1tLTUaBMePtpm\n5cqVeOqpp2rtV9uHsrYsLCywY8cOXL58Gb/88gt+++03bNiwAbGxsfj0008RGBgoehs1NeaDJTIy\nEklJSWptN2/eVJt2d3dXvb5Dhw5Ft27dMG/ePGzYsAHvvPOOajlDvI5//vknpk6dCmtra7zxxhtw\ndXWFVCqFRCLBsmXLNIKlKUgkkjrnVVZW1jlPKpXW2j5u3Djs2LEDhw8fxty5c3H8+HEUFxdj3Lhx\nqmWqX1s/Pz+Eh4frWDk9ikFCelP9l6KdnV2DgeTi4gIAuH79Ovz9/XXaXp8+fdCnTx8AQFZWFsaN\nG4f169fXGySdO3dGWloaKisr1fZKKisr8ccff+gcdFFRUZDL5Vr1CQoKQlxcHHbu3Ing4GDVXoEh\nXseffvoJxcXF2Lx5MwYOHKg2Lz8/X+dLrMXUZGtrCwAoKCjQmFfzaq3G8vDwgIeHB44cOYI5c+bg\nu+++g0wmw4gRI1TL2NvbQyaToaioqNF/NFHDuA9HejN69GhYWFggJiYGpaWlGvMLCwtRXl4OABg8\neDDs7OywY8cOZGdnaywr1PPgztoule3QoQPs7e1r/VCqaeTIkcjLy8P+/fvV2r/55hvk5eVh5MiR\n9favi6enJ/z8/NT+NcbMmTNRUVGBzZs3q9oM8TpWf0fl0df1m2++QU5OTuMG2UiNrcna2hoODg44\nd+6cWl0ZGRm1nqtqjPHjx+POnTs4cuQIzp07h9GjR6t9v8nExARjxozB5cuX6/yia25urk7bbsm4\nR0J606FDB3z88cf48MMP8fzzz+OFF15Ap06dkJeXh//+9784ceIEvv/+ezg5OUEqlWLp0qWYPXs2\nxowZo7pENC8vDwkJCQgNDa3zQ33z5s04e/Yshg4dCicnJwiCgNOnTyM1NRVhYWH11hgWFoZjx45h\n0aJFuHbtGnr27Inr16/jwIEDcHFxabC/vg0cOBD9+vXD4cOHERERgc6dOxvkdQwICIBUKsV7772H\nadOmQSaT4cKFC4iPj4ezszMUCoXexqTNezt16lSsX78eYWFhGDlyJLKzsxEXF4fu3bvjypUrWm97\nzJgxWL16NT755BMolUqMHz9eY5m5c+fiwoULmDNnDkaPHg0vLy+Ym5vjzz//RHx8PHr16oUVK1aI\nfh1aEgYJ6dWECRPQtWtXbN++Hfv27UNhYSHatGkDFxcXzJ49W+0E9IgRI7Bnzx5s2bIFBw4cwIMH\nD9CuXTt4e3vXe3PAkSNHIicnB8eOHcO9e/dgaWmJLl26YMmSJXjppZfqrc/GxgZ79+7Fhg0bcOrU\nKRw6dAht27ZFcHAwIiMjNb5D0hTefPNNhIWFYdOmTVi+fDkA/b+Ozs7O+OKLL7B27Vps2bIFpqam\n6NevH7766issXrxY7cIBfWjsexseHo7CwkL861//QlJSEtzc3LB06VJcvXpVpyBp27Yt/vGPf+D0\n6dPo2rUr+vbtq7FM9e/A9u3bcezYMZw8eRKmpqbo0KEDvL29MXHiRFFjb4kkQn3HEIiIiBrAcyRE\nRCQKg4SIiERhkBARkSgMEiIiEqXFXbWVk1Ooc187Oyvcv9/03wA2Jo65ZeCYWwYxY3ZwsKlzHvdI\ntGBm1vIePMQxtwwcc8tgqDEzSIiISBQGCRERicIgISIiUYwaJLdv38bChQsxZswY9OzZU+O53HUp\nLCzE/Pnz4ePjA29vb7z99tsaz9smIqKmYdQgSU5OxpkzZ+Di4qLV86LnzJmD8+fPY8mSJVixYgX+\n85//YObMmYYrlIiI6mTUy3+HDx+uugvorFmzGrVX8e9//xsJCQnYvXs3fHx8AFQ9+nTixIlITEzk\nMwaIiJqYUfdIdHmkZXx8PNq1a6cKEaDqAUdOTk6Ij4/XZ3lERNQIj93J9tTUVLi6umq0d+vWDamp\nqQbbbmZ2Eb4+dgNK3iyZiEjNY/fNdrlcDhsbzW9YymSyRj2e087OSqcv5RxKSMP/S0hDQN9O6PxU\n3d/wfBLV943WJxXH3DJwzPrx2AWJWLreHuBBcdWjTXNzi2D52O3H6c7BwUbUbWUeRxxzy8Axa9+3\nLo/dR6JMJkNhoeYLIZfLIZPJjFAREVHL9tgFiaurK9LS0jTa6zp3QkREhvXYBUlAQABycnLw22+/\nqdquXLmCjIwMBAQEGLEyIqKWyajnSEpKSnDmzBkAwF9//YWioiIcO3YMADBkyBBIpVIEBgbCx8cH\ny5YtAwD07dsX/v7+iIqKQlRUFExMTLB69Wp4e3vzOyREREZg1CDJzc3F7Nmz1dqqp0+ePAknJyco\nFAoolUq1ZdatW4fly5djwYIFUCqVGDZsGD744IMmq5uIiP5m1CBxcnLCzZs3613m1KlTGm0ymQzL\nly/H8uXLDVUaERE10mN3joSIiJoXBgkREYnCICEiIlEYJEREJAqDhIiIRGGQEBGRKAwSIiIShUFC\nRESiMEiIiEgUBgkREYnCICEiIlEYJEREJAqDhIiIRGGQEBGRKAwSIiIShUFCRESiMEiIiEgUBgkR\nEYnCICEiIlEYJEREJAqDhIiIRGGQEBGRKAwSIiIShUFCRESiMEiIiEgUBgkREYnCICEiIlGMGiQp\nKSkICQmBl5cX/P39ER0dDYVC0WC/K1eu4NVXX4Wvry98fX0RGhqKS5cuNUHFRET0KKMFSUFBAUJD\nQyGRSLBp0ybMnDkTO3bswIYNG+rtl5WVhRkzZqCyshKrVq3CqlWroFAoMGPGDNy5c6eJqiciompm\nxtpwXFwcysrKEBsbC2trawwePBhFRUWIjY1FeHg4rK2ta+33888/48GDB9i4cSNsbGwAAH379sXA\ngQNx5swZvPzyy005DCKiFs9oeyTx8fHw9/dXC4ygoCCUlpYiKSmpzn6VlZUwNTWFVCpVtVlZWcHU\n1BSCIBi0ZiIi0mS0IElNTYWrq6tam6OjI6RSKVJTU+vs9+yzz0IqlWLFihXIzc1Fbm4uli9fDltb\nW4wePdrQZRMR0SOMdmhLLperDk3VJJPJIJfL6+zXvn177Nq1C//85z/x1VdfAQAcHBywbds22Nvb\nN7hdOzsrmJmZal2v1NK8qr99azg4aNb9JGtp4wU45paCY9YPowWJrrKzszF79mz06tULS5YsAQDs\n2bMHr7/+OuLi4uDo6Fhv//v3i3XabklpRVX/vAewMpXotI7HkYODDXJyCo1dRpPimFsGjln7vnUx\n2qEtmUyGoqIijXa5XA6ZTFZnv23btqGyshIbNmxAQEAAAgICsGHDBpiammL79u2GLJmIiGphtCBx\ndXXVOBeSlZWFkpISjXMnNaWmpsLNzQ3m5uaqNgsLC7i5uSE9Pd1g9RIRUe2MFiQBAQFISEhQ2ys5\nevQoLC0t4evrW2c/R0dHJCcno7y8XNVWXl6O5ORkdOrUyaA1ExGRJqMFSXBwMCwsLBAZGYnExETs\n27cPsbGxCA0NVbskODAwEAsWLFBNT5w4EdnZ2Xjrrbfw888/4/Tp03jzzTeRk5ODyZMnG2MoREQt\nmtGCxNbWFjt37oRCoUBERARiYmIQEhKCWbNmqS2nUCigVCpV056enti6dSsePHiA9957D1FRUSgt\nLcX27dvh4eHR1MMgImrxjHrVlpubG3bt2lXvMqdOndJoGzRoEAYNGmSosoiISAu8+y8REYnCICEi\nIlEYJEREJAqDhIiIRGGQEBGRKAwSIiIShUFCRESiMEiIiEgUBgkREYnCICEiIlEYJEREJAqDhIiI\nRGGQEBGRKAwSIiIShUFCRESiMEiIiEgUBgkREYnCICEiIlEYJEREJIpOz2y/desWDh48iMzMTBQU\nFEAQBLX5EokEX375pV4KJCKi5k3rIDl8+DAWLFgAMzMzuLi4QCaTaSzzaLAQEdGTS+sgiY2NhYeH\nB7Zu3Qp7e3tD1ERERI8Rrc+RZGdn46WXXmKIEBERAB2CxN3dHTk5OYaohYiIHkNaB0lUVBT279+P\nS5cuGaIeIiJ6zGh9juTLL7+Era0tgoOD0aNHD3Ts2BGmpqZqy0gkEsTExOitSCIiar60DpJr164B\nADp27Ai5XA65XK6xjEQiadS6UlJSsHjxYly8eBE2NjaYOHEi3nrrLY1gqs3x48fx2WefITk5GVKp\nFJ6enoiJiYGVlZV2AyIiIlG0DpJTp07pZcMFBQUIDQ2Fm5sbNm3ahPT0dKxcuRJKpRJz586tt+/+\n/fuxaNEihIWF4b333oNcLse5c+egUCj0UhsRETWeTl9I1Ie4uDiUlZUhNjYW1tbWGDx4MIqKihAb\nG4vw8HBYW1vX2i8vLw/Lli3D//7v/2LSpEmq9sDAwKYqnYiIatA5SH7++Wf8/PPP+PPPPwEAjo6O\nGDZsGIYMGdKo/vHx8fD391cLjKCgIKxZswZJSUkYPnx4rf1++OEHAMC4ceN0LZ2IiPRI66u2SktL\n8dprr+GNN97AgQMHcOvWLdy6dQsHDhxAREQEwsLCUFpa2uB6UlNT4erqqtbm6OgIqVSK1NTUOvtd\nvnwZLi4uOHDgAAICAtCrVy9MnDgRFy5c0HYoRESkB1rvkURHRyMxMRFz587FtGnTVCe3S0pK8PXX\nX2Pt2rWIjo5GVFRUveuRy+WwsbHRaJfJZLWewK927949pKWlYfPmzXj33XfRpk0bbN26FWFhYTh+\n/DjatWtX73bt7KxgZtbwyfxHSS3Nq/rbt4aDg2bdT7KWNl6AY24pOGb90DpIjh49ismTJ+P1119X\na5dKpQgLC0NGRgaOHj3aYJDoShAEFBcXIzo6GgEBAQCAfv36YdiwYdi9ezfmzJlTb//794t12m5J\naUVV/7wHsDJt3FVpTwIHBxvk5BQau4wmxTG3DByz9n3rovWhrby8PHTv3r3O+e7u7sjLy2twPTKZ\nDEVFRRrtcrm81htB1uwnkUgwYMAAVZu1tTV69eqFW7duNbhdIiLSL62DpHPnzjhz5kyd88+cOYPO\nnTs3uB5XV1eNcyFZWVkoKSnROHdSU7du3SAIgsYdhgVBaPT3V4iISH+0DpKXX34Z8fHxiIiIwK+/\n/oq7d+/i7t27+PXXX/HGG2/gl19+wdSpUxtcT0BAABISEtT2So4ePQpLS0v4+vrW2W/o0KEAgPPn\nz6vaCgsLcfXqVXh4eGg7HCIiEknrcyTTpk3D/fv38fnnn2vsmZiZmWHmzJmNCpLg4GB89dVXiIyM\nRHh4ODIyMhAbG4vQ0FC1S4IDAwPh4+ODZcuWAQB69+6NESNG4IMPPsDbb78NOzs7bN26FWZmZo3a\nLhER6ZdO3yOJjIzE1KlT8euvv+LOnTsAgE6dOmHQoEGNvr28ra0tdu7ciUWLFiEiIgIymQwhISGI\njIxUW06hUECpVKq1rV69GqtWrcKKFStQUlKCfv36qe4BRkRETUsitLDHGep6xcJXx2/i9IU7WPya\nLzo51P6t+ycRr2xpGTjmlsFQV201uEdS85vrNacbUr08ERE92RoMkuHDh0MikeDSpUuwsLBQTTfk\n+vXreimQiIiatwaDZNmyZZBIJDA3N1ebJiIiAhoRJC+++GK900RE1LLp5TbygiDg/PnzKCsrg7e3\nd523gCcioieP1kGybt06XLhwAV999ZWqLSwsDImJiRAEAY6Ojti5cyecnZ31WigRETVPWn+z/ccf\nf4Snp6dq+sSJEzh79izefvttbN26FUqlks9rJyJqQbTeI/nrr7/QtWtX1fRPP/2E7t27IywsDAAw\nZcoU7NmzR28FEhFR86b1HomZmZnas9ETExPh7++vmra3t8f9+/f1Ux0RETV7WgdJ9+7d8dNPPwGo\nutPvvXv31B6ve+fOHdjZ2emvQiIiata0PrQ1c+ZMREREwMfHR3Wfq4EDB6rmx8fHo3fv3notkoiI\nmi+tg2Tw4MH49ttvcfbsWchkMjz//POqefn5+ejfvz9GjBih1yKJiKj50ul7JG5ubnBzc9Nob9Om\nDRYsWCC6KCIienxofY6EiIiopgb3SDw8PGBiYoKLFy/CwsICHh4eDd5rSyKR4Nq1a3orkoiImq8G\ng2TmzJmQSCQwMzNTmyYiIgIaESSPPrHw0WkiImrZeI6EiIhE0TpIli5dilGjRtU5f9SoUVi5cqWo\nooiI6PGhdZCcOXMGo0ePrnP+888/j1OnTokqioiIHh9aB8ndu3fRqVOnOuc7Ojri7t27oooiIqLH\nh9ZBIpPJkJaWVuf81NRUtG7dWlRRRET0+NA6SP7xj38gLi4ON2/e1Jh348YNxMXFqd0NmIiInmxa\n3yJl9uzZ+OWXXzBhwgSMHDlSdauU5ORknDx5Em3atMGcOXP0XigRETVPWgdJhw4dcPDgQaxZswan\nTp3CsWPHAACtW7fG888/j3nz5qFDhw56L5SIiJonnW7a2L59e6xevRqCICAvLw9A1QOt+I13IqKW\nR9QXEsvLy1FZWQkbGxuGCBFRC6VTkJw7dw7BwcHo168fhg4dit9//x0AkJeXh+nTp+PMmTONWk9K\nSgpCQkLg5eUFf39/REdHqz3GtyFKpRIvvvgievTogdOnT+syFCIiEknrIElMTMRrr72G4uJivPLK\nKxAEQTXP3t4eFhYWOHDgQIPrKSgoQGhoKCQSCTZt2oSZM2dix44d2LBhQ6Nr2b9/P/766y9th0BE\nRHqkdZBER0ejd+/eOHz4MMLDwzXme3t74+rVqw2uJy4uDmVlZYiNjcXgwYMxZcoUzJw5Ezt37kRR\nUVGD/QsKCrBu3TpeIUZEZGRaB8mNGzcwZswYmJiY1HpexMHBAffu3WtwPfHx8fD394e1tbWqLSgo\nCKWlpUhKSmqwf3R0NPr164dBgwZpNwAiItIrrYPE0tISZWVldc7PzMyETCZrcD2pqalwdXVVa3N0\ndIRUKkVqamq9fW/cuIGDBw8iKiqqcUUTEZHBaH35r4+PD7777juEhoZqzMvLy8OBAwcwePDgBtcj\nl8thY2Oj0S6TySCXy+vtu2TJEkydOhVdunRBZmZmo2sHADs7K5iZmWrVBwCkluZV/e1bw8FBs+4n\nWUsbL8AxtxQcs35oHSRz5sxBcHAwJk2ahFGjRkEikeDXX3/F77//jri4OFRWVhr04Vfff/890tLS\nsGXLFp36379frFO/ktKKqv55D2Bl2nIudXZwsEFOTqGxy2hSHHPLwDFr37cuWh/acnNzw9dff43W\nrVtj7dq1EAQBn3/+OWJjY+Hi4oLdu3ejc+fODa5HJpPVelJdLpfXeWisoqICq1atQnh4OJRKJeRy\nuWodJSUljTpJT0RE+qXTN9t79OiBL7/8EgUFBbh9+zYEQUDnzp1hb2/f6HW4urpqnAvJyspCSUmJ\nxrmTaiUlJbh79y6WL1+O5cuXq82bO3cunJ2d8dNPP2k/ICIi0plWQVJSUoKIiAi88MILmDBhAmxt\nbdGnTx+dNhwQEIBt27ahqKhIdeXW0aNHYWlpCV9f31r7WFlZYdeuXWpt9+7dw7x58zBv3jwMHDhQ\np1qIiEh3Wh3akkqluHr1KiorK0VvODg4GBYWFoiMjERiYiL27duH2NhYhIaGql0SHBgYiAULFgAA\nzMzMMGDAALV/Xl5eAAB3d3fVz0RE1HS0Pkfi7++PxMRE0Ru2tbXFzp07oVAoEBERgZiYGISEhGDW\nrFlqyykUCiiVStHbIyIiw9DpeSSRkZGYP38+Jk+eDCcnJ1haWmosV3Ovoi5ubm4ah6oe1dDz352c\nnGp9yBYRETUNrYNk9OjRAKpuuHj48OE6l7t+/bruVRER0WND6yB56623DFEHERE9prQKkosXL8LF\nxQVt2rRB//790apVK0PVRUREj4lGBUlRURHCwsJw6dIlVVvbtm2xZcsWeHp6Gqw4IiJq/hp11dZn\nn32Gixcv4tlnn8WHH36IkJAQFBYW4v333zd0fURE1Mw1ao/k5MmTeO6557B+/XpVW7du3bBw4ULc\nvn0bXbp0MViBRETUvDVqj+TOnTvw8/NTa/P394cgCMjOzjZIYURE9HhoVJCUlZVBKpWqtVV/d0Qf\n33InIqLHV6Ov2srKysKNGzdU04WFVbcizszMVGuv5uHhoYfyiIiouWt0kKxbtw7r1q3TaF+4cKHa\ntCAIkEgk/EIiEVEL0aggefSW7URERNUaFSTjx483dB1ERPSY0vruv0RERDUxSIiISBQGCRERicIg\nISIiURgkREQkCoOEiIhEYZAQEZEoDBIiIhKFQUJERKIwSIiISBQGCRERicIgISIiURgkREQkCoOE\niIhEMWqQpKSkICQkBF5eXvDlkftCAAAW8UlEQVT390d0dDQUCkW9fS5fvoz58+cjMDAQXl5eGDVq\nFGJjY1FWVtZEVRMRUU2NfkKivhUUFCA0NBRubm7YtGkT0tPTsXLlSiiVSsydO7fOfj/88APS09MR\nHh6OLl264ObNm4iOjsbNmzcRExPThCMgIiLAiEESFxeHsrIyxMbGwtraGoMHD0ZRURFiY2MRHh4O\na2vrWvuFh4fD3t5eNT1gwAC0atUKCxcuxJ07d9CpU6emGgIREcGIh7bi4+Ph7++vFhhBQUEoLS1F\nUlJSnf1qhki1nj17AgCys7P1XygREdXLaEGSmpoKV1dXtTZHR0dIpVKkpqZqta6LFy/CxMQEzs7O\n+iyRiIgawWiHtuRyOWxsbDTaZTIZ5HJ5o9eTk5ODzZs3Y+zYsWjbtm2Dy9vZWcHMzFSrWgFAamle\n1d++NRwcNOt+krW08QIcc0vBMeuH0YJEH8rLyzFnzhxYWVlh/vz5jepz/36xTtsqKa2o6p/3AFam\nEp3W8ThycLBBTk6hsctoUhxzy8Axa9+3LkYLEplMhqKiIo12uVwOmUzWYH9BEBAVFYWUlBTs2bMH\ntra2hiiTiIgaYLQgcXV11TgXkpWVhZKSEo1zJ7VZunQpTp48ie3bt6Nbt26GKpOIiBpgtJPtAQEB\nSEhIUNsrOXr0KCwtLeHr61tv388++wxff/01Vq9ejf79+xu6VCIiqofRgiQ4OBgWFhaIjIxEYmIi\n9u3bh9jYWISGhqpdEhwYGIgFCxaopo8cOYK1a9di3LhxaN++PS5evKj6l5eXZ4yhEBG1aEY7tGVr\na4udO3di0aJFiIiIgEwmQ0hICCIjI9WWUygUUCqVqumzZ88CAA4dOoRDhw6pLbt8+XK8+OKLhi+e\niIhUjHrVlpubG3bt2lXvMqdOnVKbXrFiBVasWGHIsoiISAu8+y8REYnCICEiIlEYJEREJAqDhIiI\nRGGQEBGRKAwSIiIShUFCRESiMEiIiEgUBgkREYnCICEiIlEYJEREJAqDhIiIRGGQEBGRKAwSIiIS\nhUFCRESiMEiIiEgUBgkREYnCICEiIlEYJEREJAqDhIiIRGGQEBGRKAwSIiIShUFCRESiMEiIiEgU\nBgkREYnCICEiIlEYJEREJIpRgyQlJQUhISHw8vKCv78/oqOjoVAoGuxXWFiI+fPnw8fHB97e3nj7\n7bdx//79JqiYiIgeZWasDRcUFCA0NBRubm7YtGkT0tPTsXLlSiiVSsydO7fevnPmzEFaWhqWLFkC\nExMTrFmzBjNnzsSePXsMVm9uQSkA4MDPt3DpVi6cHFqjg70VHOykeGlIN0gkEoNtu9qD0gpUVCpR\nVq5ARaUSSkFARnYRWlua40paLsxMTGBtZa5a/vofeejYtjU6tLWCVSsz9O3ertb1lpQpUFahgEIp\nQKkUUKlUQqkU0N7OChbSMsiLywEBMDWVoLWluVpfpVKAQql82BdQCgJMTSSQttLuV0sQBCiFqu1X\nr0ehrGqDAJiYVL2+1dOCIEAAIFT/LADWUnO0sjDV7kUlItGMFiRxcXEoKytDbGwsrK2tMXjwYBQV\nFSE2Nhbh4eGwtrautd+///1vJCQkYPfu3fDx8QEAtG/fHhMnTkRiYiL8/PwMUu/lW7kAgEsP/8/M\neYDMnAcAgH7uDrC0MMPDzzpYmJnCspUpHpRUoFIhoKikArnyUliYmeK3m9lwcmgNU5OqncHyCgV+\n+i0DXm7t8NuNbJRXKuHQxhI5+aUwNZFAoRRE1X0jPV9U/9q0sjAFBKCsov69x1YWpjAzkeBBaaWq\nzVpq/jA0qkJBUAoor1Tqrban2khhbt64He3SMgUqKhVwaCMFJIBEIoEEgIWFGSorFJBUt0kACQCo\nfv67TSKR4H5RGbq0twaq2x+uBzWW+XtagvyiMnTtaKOxHkgAk4d/kJhIHm6v6j+1dZhIJCivVELa\nyhStzE1V8yQPF6havmpbj9ZQ/feOpMZYIAFyiyuQn18ME4kEHeytYG5m8vf8R8ZC9CijBUl8fDz8\n/f3VAiMoKAhr1qxBUlIShg8fXme/du3aqUIEAPr06QMnJyfEx8cbLEiq9e/hgN9u5qi1Ld31u1br\nOF9LW+J/7qp+zsmv2vupLUSqQwYAunWS4dYdOYIGdcGx8+no7doWI7ydVMseOZuGTg7WaC01wx9Z\nhTA30/yAVSgFXL6VC1MTCYY+0wmmplUfFNdv34dDGylatTJDeXklSssq8cfdQrSVWao+dLPvF6Ok\nrCpM+nRrCxOJBEUlFZAXl6O1pblqT6VmkMhaW0Dy8MPQ5OGH2R93CwEAPbvYwcSkqt3UpGrev5Pv\nqeZZtTJT+8Cv+UF/KaVquQelFUBpw+9BdU1mphLV9oUaezrauv1wHY31/29k67CV5sfGyvyRkP07\nbGqflmi8h8Cj01XrKn74HtnZtEKNPPt7ezX6Q2M7f8+sGeKSGuuxaGWGO9lF6NjWSvW7WLOWmn1q\nTleF+d911qwnV16Kp6r/MKkt1B+to44/FlTh/3C5gqJy2Mss//4Do7Y/Omq0qc+vWqG5qQmGyaSN\nf3O1YLQgSU1NxcCBA9XaHB0dIZVKkZqaWmeQpKamwtXVVaO9W7duSE1NNUitNfX3eAptbFphUK8O\n2Hz4P7hXUApXRxlS/5Sr1/PwQ76mMX5dcSTxDwT274yeXewAVH2Q//7fbIzo54RKhRJtbFrBzroV\nJBJJrR/8dZkwpJtGWy8Xex1GqM7BwQY5Odp9SD7uHBxskJ0tf3joTHgYMADw98/Cw5/LK5UoLatU\nLQs8EkhVR+JU6ymvVKCwuKLqUF2N9VT3AR4evkONw3iP1KEUBOTklzzcu8PDbQqNruHhEmpjsZJa\n4EFxOS6l3IOdTSvVevHIOqvrunVHjvb20r+39fCHmq9NzTqUqmnl3zXVODxZcztKASgpqwoRE4kE\n9wvLqitWjUWf/rz3QM9rbL4qAAzyeErv6zVakMjlctjY2Gi0y2QyyOXyWno03C8zM7PB7drZWcHM\nTPvj6JNHumPfif9ixICuCHp4/N+3TycIgqDV7v7rE7w02p7z1wzG5sTBQfP1ftI99ZTM2CVQA6qD\nEagRZPg79B82/R2wNX9+qKJSqRa66kFcM+hrrq9qRcqa2xcEVFQqYWIi0Qhxparvw20o1df7d9gL\n6vMe9imvUEKhVMLc1KTWPyr+rkV4ZH5VgFf/bGoiwaDeHWFtZaH398JoQWIs9+8X69RvVH8nTBvd\nEzk5hWhJf5+31D0SjvnJ15gxqw6lNThTAqlp87/Qw9rKQuf3ub4/KI12+a9MJkNRUZFGu1wuh0xW\n91+DMpkMhYWaL0RD/YiIyDCMFiSurq4a5zSysrJQUlJS6zmQmv3S0tI02us6d0JERIZltCAJCAhA\nQkKC2l7J0aNHYWlpCV9f33r75eTk4LffflO1XblyBRkZGQgICDBozUREpMloQRIcHAwLCwtERkYi\nMTER+/btQ2xsLEJDQ9UuCQ4MDMSCBQtU03379oW/vz+ioqJw/PhxnDhxAu+88w68vb0NfukvERFp\nMlqQ2NraYufOnVAoFIiIiEBMTAxCQkIwa9YsteUUCgWUSvUvrK1btw6+vr5YsGAB3nvvPXh6eiI2\nNrYpyycioockQs1r4VoAMVem8MqWloFjbhk4Zu371oV3/yUiIlEYJEREJEqLO7RFRET6xT0SIiIS\nhUFCRESiMEiIiEgUBgkREYnCICEiIlEYJEREJAqDhIiIRGGQAEhJSUFISAi8vLzg7++P6OhoKBSK\nBvsVFhZi/vz58PHxgbe3N95++23cv3+/CSoWT5cxX758GfPnz0dgYCC8vLwwatQoxMbGoqysrImq\nFkfX97maUqnEiy++iB49euD06dMGrFR/xIz5+PHjmDBhAvr06YMBAwbgtddeQ3Gxbg+Ga0q6jvnK\nlSt49dVX4evrC19fX4SGhuLSpUtNULE4t2/fxsKFCzFmzBj07NkT06dPb1Q/fX5+tbgnJD6qoKAA\noaGhcHNzw6ZNm5Ceno6VK1dCqVRi7ty59fadM2cO0tLSsGTJEpiYmGDNmjWYOXMm9uzZ00TV60bX\nMf/www9IT09HeHg4unTpgps3byI6Oho3b95ETExME45Ae2Le52r79+/HX3/9ZeBK9UfMmPfv349F\nixYhLCwM7733HuRyOc6dO6dV8BqDrmPOysrCjBkz8PTTT2PVqlUAgG3btmHGjBk4cuQIOnXq1FRD\n0FpycjLOnDkDLy8vVFZWNrqfXj+/hBZuy5YtQv/+/YXCwkJV2+effy706dNHre1RFy5cENzd3YWk\npCRV26VLlwR3d3fh7NmzBq1ZLF3HnJubq9EWFxcnuLu7C5mZmQapVV90HXO1/Px8YcCAAcI333wj\nuLu7C6dOnTJkuXoh5n1+5plnhH379jVFmXql65j37NkjeHh4CHK5XNWWn58veHh4CF9//bVBaxZL\noVCofo6MjBSmTZvWYB99f361+ENb8fHx8Pf3V3sGSlBQEEpLS5GUlFRvv3bt2sHHx0fV1qdPHzg5\nOSE+Pt6gNYul65jt7e012nr27AkAyM7O1n+heqTrmKtFR0ejX79+GDRokCHL1Ctdx/zDDz8AAMaN\nG2fwGvVN1zFXVlbC1NQUUqlU1WZlZQVTU1MIzfwuUiYm2n+M6/vzq8UHSW2P6HV0dIRUKtV4FHBD\n/QCgW7du9fZrDnQdc20uXrwIExMTODs767NEvRMz5hs3buDgwYOIiooyZIl6p+uYL1++DBcXFxw4\ncAABAQHo1asXJk6ciAsXLhi6ZNF0HfOzzz4LqVSKFStWIDc3F7m5uVi+fDlsbW0xevRoQ5fd5PT9\n+dXig0Qul8PGRvM++zKZDHK5XO/9mgN91Z6Tk4PNmzdj7NixaNu2rT5L1DsxY16yZAmmTp2KLl26\nGKo8g9B1zPfu3UNaWho2b96Md955B5s3b4ZUKkVYWBju3btnyJJF03XM7du3x65du3D8+HH4+fnB\nz88Px48fx7Zt22rdE3/c6fvzq8UHCemmvLwcc+bMgZWVFebPn2/scgzm+++/R1paGt58801jl9Jk\nBEFAcXExli5dihdeeAEBAQHYtGkTTE1NsXv3bmOXZxDZ2dmYPXs2evXqhS+++AJffPEFPD098frr\nr+PPP/80dnnNXosPEplMhqKiIo12uVwOmUxWb7/CQs0njTXUrznQdczVBEFAVFQUUlJS8Pnnn8PW\n1tYQZeqVLmOuqKjAqlWrEB4eDqVSCblcrlpHSUlJretrTsT8bkskEgwYMEDVZm1tjV69euHWrVsG\nqVVfdB3ztm3bUFlZiQ0bNiAgIAABAQHYsGEDTE1NsX37dkOWbBT6/vxq8UHi6uqqcUwwKysLJSUl\ntR5DrNkvLS1No72uY4/Nia5jrrZ06VKcPHkSGzduRLdu3QxVpl7pMuaSkhLcvXsXy5cvh4+PD3x8\nfDB27FgAwNy5czF+/HiD1y2Gru9zt27dIAiCxklmQRAgkUgMUqu+6Drm1NRUuLm5wdzcXNVmYWEB\nNzc3pKenG6xeY9H351eLD5KAgAAkJCSo/RVz9OhRWFpawtfXt95+OTk5+O2331RtV65cQUZGBgIC\nAgxas1i6jhkAPvvsM3z99ddYvXo1+vfvb+hS9UaXMVtZWWHXrl1q/9auXQsAmDdvHtasWdMktetK\n1/d56NChAIDz58+r2goLC3H16lV4eHgYrF590HXMjo6OSE5ORnl5uaqtvLwcycnJzfo7JLrS9+eX\n6ccff/yxHut77HTv3h379u3D+fPn8dRTTyExMRFr165FSEgIhgwZolouMDAQN27cwIgRIwAAHTt2\nxMWLF3HgwAF07NgRaWlp+Pjjj9GtWzfMmTPHWMNpFF3HfOTIEXzyyScYP348BgwYgLt376r+WVhY\nqF062dzoMmYTExM4OTmp/asOl5CQEAwcONCII2qYru9z+/btcf36dezduxd2dnb466+/sHjxYuTn\n52PVqlWwtLQ01pAapOuYHRwc8OWXX+I///kPbGxskJaWhhUrVuDmzZtYtGgR2rVrZ6whNaikpAQn\nT55ESkoKEhISUFBQgLZt2yIlJQWdOnWCubm54T+/tP7myRMoOTlZmD59utC7d29h8ODBwrp164TK\nykq1ZYYNGyZERUWptRUUFAjvv/++4O3tLfTt21eYN29erV/aa450GXNUVJTg7u5e67+DBw829RC0\npuv7XFNGRsZj84VEQdB9zEVFRcLChQsFX19foXfv3kJISIhw48aNpixdZ7qOOTExUXj55ZcFHx8f\nwcfHR5g6dapw7ty5pixdJ9W/k7X9y8jIEATB8J9ffGY7ERGJ0uLPkRARkTgMEiIiEoVBQkREojBI\niIhIFAYJERGJwiAhIiJRGCRET5Dp06erPWr1/Pnz6NGjh9q31In0rcU/apdIG4cOHVK727GZmRkc\nHBwwbNgwzJkz57G4gSWRvjFIiHQwd+5cdOzYEaWlpTh//jz27Nmjuq1Ic7+xIZG+MUiIdDBkyBDV\nY4YnT54MoOrZJVeuXEGfPn2MWRpRk+M5EiI98Pb2BgCNW44fPHgQ48ePR58+fTBgwABERUXV+pTB\nn3/+GVOnTkXfvn3h7e2N4OBgnDhxQjX/xIkTeP311+Hv7w9PT0+MHDkSGzduhEKhMOzAiBqBeyRE\nenDnzh0AUHsoUGxsLDZu3IigoCBMmjQJOTk52LVrF65cuYJDhw6p7qK7f/9+fPjhh/Dw8EBERARa\nt26Nq1ev4uzZsxg5ciQA4Ntvv4WVlRVmzJgBKysrnDt3Dhs2bEBRUdFj9yx5evIwSIh0IJfLkZeX\nh9LSUiQlJWHv3r2wt7eHj48PACAzMxObNm3Cu+++i1dffVXVLyAgAMHBwfj2228xZcoUFBYWYtmy\nZejbty927doFCwsL1bI176f66aefqt2+fcqUKVi4cCH27t2LuXPnqvUjamoMEiIdvPLKK2rT7u7u\nWLZsmeqZLCdOnIAgCAgMDEReXp5qOWdnZzg4OCApKQlTpkxBQkICiouL8c9//lMjDGqetK8ZIkVF\nRSgvL0f//v2xb98+pKamNvsHTtGTjUFCpINPPvkEzs7OyM/PR1xcHFJSUtQe7PXHH39AqVSqDk09\nqjpcMjIyAFQ9kKk+ycnJWL9+Pc6dO6fxTPLanr1N1JQYJEQ68PLyUl21NWLECIwbNw7vvPMODh06\nBBMTEyiVSpiamuKLL76o9XLgmudSGiKXyzFt2jRYW1tj1qxZcHZ2RqtWrXD16lWsWbMGSqVSb+Mi\n0gWDhEikVq1a4a233sK8efPwww8/ICgoCM7OzlAoFOjSpQucnJzq7Ovs7Aygao+jruWSkpKQn5+P\n2NhY1TkYoOo8DFFzwMt/ifTgueeeQ6dOnfDFF18AqHomuImJCTZu3KixrFKpRH5+PgBg8ODBsLKy\nwmeffYby8nK15apPtpuYmKhNA0B5eTn27NljkLEQaYt7JER6YGpqildeeQXLly9HfHw8AgICMGvW\nLKxfvx4ZGRkYNmwYpFIpMjIy8OOPP+KNN97AxIkTYWNjg/fffx8LFy7ExIkTERQUhNatW+PatWuw\nsLDARx99hL59+8LW1hbvv/8+pk+fDolEgu+++w58SjY1F9wjIdKTl156CTY2Nqq9kjfeeAPr1q1D\nZWUlYmJisHr1asTHx2PkyJHw8/NT9Zs8eTI2btwIS0tLbNy4EevWrcOtW7fg7+8PALCzs8OWLVvg\n4OCA9evXY9u2bfDz88O7775rlHESPUoi8M8aIiISgXskREQkCoOEiIhEYZAQEZEoDBIiIhKFQUJE\nRKIwSIiISBQGCRERicIgISIiURgkREQkyv8BE+uPZUfixlUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.step(recall, precision, where='post')\n",
    "plt.title('Precision-Recall curve')\n",
    "plt.xlabel('Recall')\n",
    "plt.ylabel('Precision')\n",
    "pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Why?\n",
    "\n",
    "There are problay two reasons for the surprisingly bad perfromance.\n",
    "\n",
    "- The first is taht the easy fraud cases, just by chance, ended up mostly in the trainign set\n",
    "- Then grid-search found the a fortuitously effective value of $n$\n",
    "- When we evaluate on the test set, this fails miserably because\n",
    "    - The hard fraud cases are over-represented in the test cases\n",
    "    - The value of $n$ is overfitted on the training data and does not generalize"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's check what happeens with a differnet random number seed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, random_state=2019)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((213605, 30), (213605,))"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.shape, y_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "search = Parallel(n_jobs=-1)(delayed(fit)(n, X_train) for n in range(2,X.shape[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "scores =  np.array([(n, average_precision_score(y_train, y_score)) for (n, y_score) in search])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The average precison score is much worse\n",
    "\n",
    "This strongly suggests that the previous result was a fluke of the distribution of the easy and hard fraud cases."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([10.        ,  0.19941431])"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best = max(scores, key=lambda x: x[1])\n",
    "best"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "n = int(best[0])\n",
    "n, y_score = fit(n, X_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "precision, recall, thresholds = precision_recall_curve(y_train, y_score)\n",
    "fpr, tpr, thresholds = roc_curve(y_train, y_score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEnCAYAAACDhcU8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XlcVNX/P/DXLCyDMMgmCgiyCJgI\nKoIIhCup8TVNUzE10KAww71Q62PlhmmpIC7lmplirn38pX7MlRCRylxyCwQFFGRn2GFm7u8PYnIc\nttkYYN7Px4NH3HPvufM+MzRv7z33nMNiGIYBIYQQoiC2pgMghBDSsVEiIYQQohRKJIQQQpRCiYQQ\nQohSKJEQQghRCiUSQgghSqFEQtq9ESNGYObMmXLXO378OFxcXHD9+nU1RNW5zZw5EyNGjJAqW7p0\nKVxcXDQUEWnPuJoOgGje9evX8c4770iVGRgYwN7eHuPHj8eMGTPA4XA0FF3Hd/z4cSxbtkyyzWKx\n0KVLF/Tu3RtTpkzBxIkTNRgdIcqjREIk/u///g8BAQFgGAZ5eXk4ceIE1q5di7S0NKxatUpjcZ09\ne1aheuPHj0dQUBB0dHRUHJFiZs6ciX79+oFhGOTk5ODIkSNYtmwZ8vLyEBERoenwCFEYJRIi8cor\nr2D8+PGS7bfffhtjx47FkSNHMH/+fJibmzdar66uDmKxGHp6emqJS1dXV6F6HA6nXV1JDRo0CGPG\njJFsT5w4EWPGjMGuXbsQHh7ermLtKKqrq8HlcsHl0leZJlEfCWmSoaEhBgwYAIZhkJWVBQDYsmUL\nXFxckJqaiujoaAQEBMDd3R03b96U1EtKSsLs2bMxaNAg9OvXD+PGjcOhQ4cafY179+5h3rx58PX1\nhZubG4YOHYpFixYhMzNTckxjfSQ3btxAWFgY/Pz80K9fP7z66qsIDw+XiqOpPpKioiJ88cUXGDp0\nqOQ1v/jiCxQXF0sd11D/2rVr2L17N0aNGgU3NzeMHj0aJ06cUOxNfYGlpSUcHBxQVlaGoqIimf2q\nfh9Pnz6NiIgIDBs2DG5ubhg8eDA++OADPHjwQOm2KBJTdnY2XFxcsGXLFpm6DX9n2dnZkrKGPpqi\noiIsW7YMvr6+6N+/P1JTU9GvXz98+OGHjcbx9ddfw8XFBffv35eUlZWVYcOGDQgMDISbmxt8fHyw\naNEiyd85kQ+lcdIkhmHw5MkTAICJiYnUviVLlkBfXx+zZ88GAFhYWAAADh8+jM8++wz9+/dHREQE\neDwekpKS8PnnnyMzMxNRUVGSc1y6dAmRkZEwMDDAW2+9BTs7O+Tn5yMxMRF///03bG1tG40rPT0d\ns2fPhrm5Od555x2YmZmhsLAQf/zxBx48eID+/fs32aaysjJMmzYNT548waRJk/DKK6/g/v37OHTo\nEJKTk3HkyBEYGhpK1dm0aROqq6sxdepU6Orq4tChQ1i6dClsbW3h6ekp/xv7j7q6OuTk5IDNZoPP\n50vtU8f7eODAAXTt2hVTpkyBhYUFMjMz8eOPP2LatGk4ceIEevXqpXBbXqboZ9sas2bNgrm5OT74\n4ANUVlbC0tISI0aMwIULF1BSUoKuXbtKjhWLxTh16hRcXFzQp08fAPV/A8HBwXj27BkmTZqE3r17\nIz8/HwcPHsTkyZNx7NgxWFtbK/0eaBWGaL3k5GTG2dmZ2bJlC1NYWMgUFhYy9+/fZz755BPG2dmZ\nmTJliuTY2NhYxtnZmZkxYwZTV1cndZ7nz58zbm5uzKJFi2ReY9WqVYyrqyuTmZnJMAzDVFZWMoMH\nD2Z8fHyY3NxcmeNFIpHk9+HDhzMzZsyQbH/33XeMs7Mzc+vWrWbbdezYMcbZ2ZlJTk6WlG3cuJFx\ndnZmDhw4IHXsgQMHGGdnZ2bTpk0y9cePH8/U1NRIynNzc5m+ffsyCxcubPb1Xz7P0aNHmcLCQqag\noIC5c+cOExkZyTg7OzPz5s2TOl5d72NFRYXM/rS0NKZv377MZ599JlU+Y8YMZvjw4VJlUVFRjLOz\nc4vtbW1MWVlZjLOzMxMbGytzTMPfWVZWlszrL168WOb4S5cuNfq5JiUlMc7OzsyePXskZatWrWL6\n9evH3L9/X+rY7OxsZsCAAUxUVFSLbSTS6NYWkdiyZQuGDBmCIUOGYPz48Th27BhGjBiBrVu3yhwb\nEhIic1/6f//7H2pra/HWW2+hqKhI6mfEiBEQi8VISkoCACQmJqK4uBizZs2CpaWlzPnZ7Kb/NI2M\njAAAFy5cQE1NjVxt/OWXX2BqaoqpU6dKlU+dOhWmpqY4f/68TJ23335bqp/G0tIS9vb2ePz4sVyv\nvXz5cgwZMgS+vr6YNGkSzp07hylTpmDt2rVSx6nrfTQwMABQf6VZXl6OoqIimJiYwN7eHrdv35ar\nLc1R5rNtjXfffVemzN/fH+bm5vjpp5+kyk+ePAkul4tx48YBqG/7qVOn4OXlhW7dukm9tzweD/37\n90diYqJS8WkjurVFJKZOnYoxY8aAxWKBx+OhV69eUrcJXtTYbZBHjx4BAEJDQ5t8jYKCAgCQfAm/\n8sorcscZFBSE//73v9ixYwf27dsHDw8P+Pv7IygoqMVbEtnZ2XBzc5NJglwuF7169cK9e/dk6vTs\n2VOmrGvXrnj69Klku6SkBHV1dVLHNNzuazB37lwMGjQItbW1uH37Nnbt2oX8/HyZp8rU9T7eu3cP\nMTExSElJQWVlpdQ+GxubFuu3ljKfbWs09rfXkCz27t2LjIwM2Nvbo7KyEufOnYOfn5/kQZGioiKU\nlJQgMTERQ4YMafT8yiY6bUSJhEjY2dnB19e3Vcfq6+vLlDH/LG3z5Zdfolu3bo3Wa+xLWV66urrY\nu3cvbt++jV9//RW///47YmNjERcXh6+//hqBgYFKv8aLWvPFEhkZiZSUFKmyhw8fSm07OztL3t9h\nw4bB0dERixYtQmxsLJYsWSI5Th3v47NnzzB9+nQYGhpizpw5cHBwAI/HA4vFwtq1a2USS1tgsVhN\n7hMKhU3u4/F4jZZPmDABe/fuxcmTJ7Fw4UKcO3cOlZWVmDBhguSYhvfW19cX4eHhCkZOXkaJhKhM\nw78UTUxMWkxI9vb2AID79+/D399foddzd3eHu7s7ACAnJwcTJkzA5s2bm00kPXv2REZGBoRCodRV\niVAoxOPHjxVOdFFRURAIBHLVCQoKQnx8PPbt24fg4GDJVYE63sdffvkFlZWV2L59O3x8fKT2lZSU\nKPyItTIxGRsbAwBKS0tl9r34tFZrubq6wtXVFadOncKCBQvw008/gc/nY+TIkZJjTE1NwefzUV5e\n3up/NJGW0TUcUZmxY8dCV1cXW7ZsQXV1tcz+srIy1NbWAgD8/PxgYmKCvXv3Ii8vT+ZYppmFOxt7\nVLZ79+4wNTVt9EvpRaNGjUJRURGOHDkiVf7jjz+iqKgIo0aNarZ+U9zc3ODr6yv10xpz585FXV0d\ntm/fLilTx/vYMEbl5ff1xx9/RH5+fusa2UqtjcnQ0BAWFhZITk6WiisrK6vRvqrWePPNN/H06VOc\nOnUKycnJGDt2rNT4JjabjXHjxuH27dtNDnQtLCxU6LW1GV2REJXp3r07Pv/8c3z66ad4/fXX8cYb\nb8Da2hpFRUX4+++/cf78efz888+wsbEBj8fDmjVrMH/+fIwbN07yiGhRURESExMRGhra5Jf69u3b\ncfXqVQwbNgw2NjZgGAaXLl1Ceno6wsLCmo0xLCwMZ8+excqVK3Hv3j306dMH9+/fx9GjR2Fvb99i\nfVXz8fHBwIEDcfLkSURERKBnz55qeR8DAgLA4/Hw8ccfY8aMGeDz+bhx4wYSEhJga2sLkUiksjbJ\n89lOnz4dmzdvRlhYGEaNGoW8vDzEx8ejd+/euHPnjtyvPW7cOGzYsAFffPEFxGIx3nzzTZljFi5c\niBs3bmDBggUYO3YsPDw8oKOjg2fPniEhIQF9+/bFunXrlH4ftAklEqJSkyZNQq9evbBnzx4cPnwY\nZWVl6Nq1K+zt7TF//nypDuiRI0fi4MGD2LFjB44ePYqKigqYm5vD09Oz2ckBR40ahfz8fJw9exYF\nBQXQ19eHnZ0dVq9ejbfeeqvZ+IyMjHDo0CHExsbi4sWLOH78OMzMzBAcHIzIyEiZMSRt4YMPPkBY\nWBi2bduG6OhoAKp/H21tbbFz505s3LgRO3bsAIfDwcCBA/H9999j1apVUg8OqEJrP9vw8HCUlZXh\nv//9L1JSUuDk5IQ1a9bg7t27CiUSMzMzvPrqq7h06RJ69eqFAQMGyBzT8DewZ88enD17FhcuXACH\nw0H37t3h6emJyZMnK9V2bcRimruHQAghhLSA+kgIIYQohRIJIYQQpVAiIYQQohRKJIQQQpSidU9t\n5eeXKVzXxMQAxcVtPwJYk6jN2oHarB2UabOFhVGT++iKRA5crvYtPERt1g7UZu2grjZTIiGEEKIU\nSiSEEEKUQomEEEKIUjSaSJ48eYIVK1Zg3Lhx6NOnj8y63E0pKyvDsmXL4OXlBU9PTyxevFhmvW1C\nCCFtQ6OJJDU1FVeuXIG9vb1c60UvWLAA169fx+rVq7Fu3Tr89ddfmDt3rvoCJYQQ0iSNPv47YsQI\nySyg8+bNa9VVxZ9//onExEQcOHAAXl5eAOqXPp08eTKSkpJojQFCCGljGr0iUWRJy4SEBJibm0uS\nCFC/wJGNjQ0SEhJUGR4hhJBW6HADEtPT0+Hg4CBT7ujoiPT0dLW9bnZeOQ5eSENFZS0GuVhggLNF\ny5UIIUQLdLhEIhAIYGQkO8KSz+e3anlOExMDhQblXLufh/O/ZQIAnpdU4TU/2WTWWTU3orWzojZr\nB2qzanS4RKIsRacHGNKnG/w8XsOcLy+gtlak1FQrHYmFhZHWtLUBtVk7UJvlr9uUDjeOhM/no6xM\n9o0QCATg8/lqfW0zYx5YYKn1NQghpKPpcInEwcEBGRkZMuVN9Z0QQghRrw6XSAICApCfn4/ff/9d\nUnbnzh1kZWUhICBAg5ERQoh20mgfSVVVFa5cuQIAeP78OcrLy3H27FkAwNChQ8Hj8RAYGAgvLy+s\nXbsWADBgwAD4+/sjKioKUVFRYLPZ2LBhAzw9PWkMCSGEaIBGE0lhYSHmz58vVdawfeHCBdjY2EAk\nEkEsFksds2nTJkRHR2P58uUQi8UYPnw4PvnkkzaLmxBCyL80mkhsbGzw8OHDZo+5ePGiTBmfz0d0\ndDSio6PVFRohhJBW6nB9JIQQQtoXSiSEEEKUQomEEEKIUiiREEIIUQolEkIIIUqhREIIIUQplEgI\nIYQohRIJIYQQpVAiIYQQohRKJIQQQpRCiYQQQohSKJEQQghRCiUSQgghSqFEQgghRCmUSAghhCiF\nEgkhhBClUCIhhBCiFEokhBBClEKJhBBCiFIokRBCCFEKJRJCCCFKoURCCCFEKZRICCGEKIUSCSGE\nEKVQIiGEEKIUSiSEEEKUQomEEEKIUjSaSNLS0hASEgIPDw/4+/sjJiYGIpGoxXp37tzB7Nmz4e3t\nDW9vb4SGhuLWrVttEDEhhJCXaSyRlJaWIjQ0FCwWC9u2bcPcuXOxd+9exMbGNlsvJycHs2bNglAo\nxPr167F+/XqIRCLMmjULT58+baPoCSGENOBq6oXj4+NRU1ODuLg4GBoaws/PD+Xl5YiLi0N4eDgM\nDQ0brXf58mVUVFRg69atMDIyAgAMGDAAPj4+uHLlCt5+++22bAYhhGg9jV2RJCQkwN/fXyphBAUF\nobq6GikpKU3WEwqF4HA44PF4kjIDAwNwOBwwDKPWmAkhhMjSWCJJT0+Hg4ODVJmVlRV4PB7S09Ob\nrPfaa6+Bx+Nh3bp1KCwsRGFhIaKjo2FsbIyxY8eqO2xCCCEv0ditLYFAILk19SI+nw+BQNBkPUtL\nS+zfvx/vv/8+vv/+ewCAhYUFdu/eDVNT0xZf18TEAFwuR+G4WWwWuFw2LCxkY++stKmtDajN2oHa\nrBoaSySKysvLw/z589G3b1+sXr0aAHDw4EG89957iI+Ph5WVVbP1i4srFX5tCwsjMGIGQqEY+fll\nCp+nI7GwMNKatjagNmsHarP8dZuisVtbfD4f5eXlMuUCgQB8Pr/Jert374ZQKERsbCwCAgIQEBCA\n2NhYcDgc7NmzR50hE0IIaYTGEomDg4NMX0hOTg6qqqpk+k5elJ6eDicnJ+jo6EjKdHV14eTkhMzM\nTLXFSwghpHEaSyQBAQFITEyUuio5ffo09PX14e3t3WQ9KysrpKamora2VlJWW1uL1NRUWFtbqzVm\nQgghsjSWSIKDg6Grq4vIyEgkJSXh8OHDiIuLQ2hoqNQjwYGBgVi+fLlke/LkycjLy8OHH36Iy5cv\n49KlS/jggw+Qn5+PqVOnaqIphBCi1TSWSIyNjbFv3z6IRCJERERgy5YtCAkJwbx586SOE4lEEIvF\nkm03Nzfs2rULFRUV+PjjjxEVFYXq6mrs2bMHrq6ubd0MQgjRehp9asvJyQn79+9v9piLFy/KlA0Z\nMgRDhgxRV1iEEELkQLP/ykkoEqO4rJpG0RNCyD8okcipVihGRbUQJ3/N0HQohBDSLlAiUVB+SZWm\nQyCEkHaBEgkhhBClUCIhhBCiFEokhBBClEKJhBBCiFIokRBCCFEKJRJCCCFKoURCCCFEKZRICCGE\nKIUSCSGEEKVQIiGEEKIUSiSEEEKUQomEEEKIUiiREEIIUQolEkIIIUqhREIIIUQplEgIIYQoRaE1\n2x89eoRjx44hOzsbpaWlMsvOslgsfPfddyoJsL1ZFTYY/9l1XdNhEEJIuyF3Ijl58iSWL18OLpcL\ne3t78Pl8mWM683rmBnoK5V5CCOm05P5WjIuLg6urK3bt2gVTU1N1xEQIIaQDkbuPJC8vD2+99RYl\nEUIIIQAUSCTOzs7Iz89XRyyEEEI6ILkTSVRUFI4cOYJbt26pIx5CCCEdjNx9JN999x2MjY0RHBwM\nFxcX9OjRAxwOR+oYFouFLVu2qCxIQggh7ZfcieTevXsAgB49ekAgEEAgEMgcw2KxWnWutLQ0rFq1\nCjdv3oSRkREmT56MDz/8UCYxNebcuXP45ptvkJqaCh6PBzc3N2zZsgUGBgbyNYgQQohS5E4kFy9e\nVMkLl5aWIjQ0FE5OTti2bRsyMzPx5ZdfQiwWY+HChc3WPXLkCFauXImwsDB8/PHHEAgESE5Ohkgk\nUklshBBCWk9jgyLi4+NRU1ODuLg4GBoaws/PD+Xl5YiLi0N4eDgMDQ0brVdUVIS1a9fiP//5D6ZM\nmSIpDwwMbKvQCSGEvEDhRHL58mVcvnwZz549AwBYWVlh+PDhGDp0aKvqJyQkwN/fXyphBAUF4auv\nvkJKSgpGjBjRaL0zZ84AACZMmKBo6IQQQlRI7qe2qqur8e6772LOnDk4evQoHj16hEePHuHo0aOI\niIhAWFgYqqurWzxPeno6HBwcpMqsrKzA4/GQnp7eZL3bt2/D3t4eR48eRUBAAPr27YvJkyfjxo0b\n8jaFEEKICsh9RRITE4OkpCQsXLgQM2bMkHRuV1VV4YcffsDGjRsRExODqKioZs8jEAhgZGQkU87n\n8xvtwG9QUFCAjIwMbN++HR999BG6du2KXbt2ISwsDOfOnYO5uXmzr2tiYgAut+XO/KaYmXUBAOjp\n6cDCQjb+zkhb2vkiarN2oDarhtyJ5PTp05g6dSree+89qXIej4ewsDBkZWXh9OnTLSYSRTEMg8rK\nSsTExCAgIAAAMHDgQAwfPhwHDhzAggULmq1fXFyp8GtbWBihsLACAFBTU4f8/DKFz9VRWFgYaUU7\nX0Rt1g7UZvnrNkXuW1tFRUXo3bt3k/udnZ1RVFTU4nn4fD7Ky8tlygUCQaMTQb5Yj8ViYfDgwZIy\nQ0ND9O3bF48ePWrxdVWlSFAt+RGKxG32uoQQ0t7InUh69uyJK1euNLn/ypUr6NmzZ4vncXBwkOkL\nycnJQVVVlUzfyYscHR3BMIzMDMMMw7R6/IoyauvqHzH+O7sUS7YlYcm2JKze/7vaX5cQQtoruRPJ\n22+/jYSEBERERODatWvIzc1Fbm4url27hjlz5uDXX3/F9OnTWzxPQEAAEhMTpa5KTp8+DX19fXh7\nezdZb9iwYQCA69f/XROkrKwMd+/ehaurq7zNkVtFtVDy+5C+ljDQ4yK/pErtr0sIIe2V3H0kM2bM\nQHFxMb799luZKxMul4u5c+e2KpEEBwfj+++/R2RkJMLDw5GVlYW4uDiEhoZKPRIcGBgILy8vrF27\nFgDQr18/jBw5Ep988gkWL14MExMT7Nq1C1wut1Wvq0rh4/ri8z0pyC+lREII0V4KjSOJjIzE9OnT\nce3aNTx9+hQAYG1tjSFDhrR6enljY2Ps27cPK1euREREBPh8PkJCQhAZGSl1nEgkglgs3QexYcMG\nrF+/HuvWrUNVVRUGDhwomQOMEEJI22IxnXk5w0Yo85SGhYURrt96KukT2bN0BGavq58yhsthYeZo\nF7zqbqWSONsLerJFO1CbtYO6ntpq8YrkxZHrL263pOF4bSEUMcjIKcOr7pqOhBBC2laLiWTEiBFg\nsVi4desWdHV1JdstuX//vkoCJIQQ0r61mEjWrl0LFosFHR0dqW1CCCEEaEUimThxYrPbhBBCtJtK\nppFnGAbXr19HTU0NPD09m5wCnrSdtKel+OW3LNhaGkJXh4PjV9LhYMXHkuD+dEVJCFEpuRPJpk2b\ncOPGDXz//feSsrCwMCQlJYFhGFhZWWHfvn2wtbVVaaCkdZ4VVGDriTvIKayfU+y3B3mSffefFIMB\nQGmEEKJKco9s/9///gc3NzfJ9vnz53H16lUsXrwYu3btglgspvXaNURQUYtPd12XJBFCCGkLcieS\n58+fo1evXpLtX375Bb1790ZYWBj8/f0xbdo0pKSkqDJG0goisRjLvr0mVeZs8+8AzV7d658B17Jh\nQ4SQNiB3IuFyuVJroyclJcHf31+ybWpqiuLiYtVER1ptx093UVVT/7ksCe6PzfP8ETV9IPz79cC6\n933wOLd+ENK2E39pMkxCSCckdyLp3bs3fvnlFwD1M/0WFBRILa/79OlTmJiYqC5C0qKrd3Lwx8N8\nAEDIGBe80ssUfANdsFgszA7qg24mBpJj6bYXIUTV5O5snzt3LiIiIuDl5SWZ58rHx0eyPyEhAf36\n9VNpkATIyivHZ3v+vWXoZG0M7z7dcDr5CUrKawEAIwfaYGh/60brvxvUB7t/vg82m7raCSGqJXci\n8fPzw4kTJ3D16lXw+Xy8/vrrkn0lJSUYNGgQRo4cqdIgtZ1QJMb6g9Jr0qc9LUXa01LJNofNwvTX\nnJs8h1+/Htj98308K6hARo4A9j2aXjyMEELkodA4EicnJzg5OcmUd+3aFcuXL1c6KPKv/JIqpNx/\nLrUOSmPWz/Ft9TlPX3uC98f3RWWNELW1Ipga64NNY0sIIQpSyYBEbfbNkqF4kluOtQf+UMv5Y4/e\nxtOC+nXiAwf1xC+/Z0ntnzTUAf79esDYUK/V5/zj73y8t+GyZNunryXeG9dXJfESQrRPi4nE1dUV\nbDYbN2/ehK6uLlxdXVscGc1isXDv3j2VBdme6XA5MNBXTz4+ez1TkkTeGeMCb9du8OvXHXfSC3Hs\nSv0yxWN97Fp9NRH9ng+WfZssU5589zlGDLSBkzWt50IIkV+L34Bz584Fi8UCl8uV2ibqVVhajR8v\npUm2h/3TiW6rrwNbSyMIRQwGuVjIdUvK0tQANhZdkJ1fIbMv8XYOJRJCiEJaTCQvr1j48jZRj9PJ\nT5rdP97fXqHzrnx3MMQMg4dPitGnl6lkYS6ABioSQhQj9zgSon5/Z5Xg0p/1Sxh/OLEfvv1omErP\nz2ax0KdX/ZLI0e/5tHA0IYQ0T+5EsmbNGowePbrJ/aNHj8aXX36pVFDaTCxm8M1/7wIAuuhz4e5o\nBi6H8j0hpP2S+xvqypUrGDt2bJP7X3/9dVy8eLHJ/aR5GbkCFJfVAAA+mOBGSYQQ0u7J/S2Vm5sL\na+vGR08D9Wu15+bmKhWUtmIYBmv21z9G7OfWXXL7iRBC2jO5n1vl8/nIyMhocn96ejq6dOmiVFDa\nqKZOhJ8S/31fx/jYaTAaQghpPbkTyauvvor4+HiMHz8eLi4uUvsePHiA+Ph4BAYGqizAzi7zeRk+\n3/ubVNmQvpawNqdkTAjpGOROJPPnz8evv/6KSZMmYdSoUZKpUlJTU3HhwgV07doVCxYsUHmgnVF5\nVZ1MEgGAWa/30UA0hBCiGLkTSffu3XHs2DF89dVXuHjxIs6ePQsA6NKlC15//XUsWrQI3bt3V3mg\nnU1uUSVS7j1vdB91sBNCOhKF5vawtLTEhg0bwDAMioqKANQvaEUj3ltv/9kHeJBZIlW2fs4QmBvz\nNBQRIYQoRqlJomprayEUCmFiYkJJRE41dWJw2Cy8/0ZfvNLLVG3zdRFCiLopdA8lOTkZwcHBGDhw\nIIYNG4Y//qh/ZLWoqAgzZ87ElStXWnWetLQ0hISEwMPDA/7+/oiJiZFaxrclYrEYEydOhIuLCy5d\nuqRIU9pcTZ0I6364gYwcAURiBoNcu1ESIYR0aHInkqSkJLz77ruorKzEO++8A4b5d44mU1NT6Orq\n4ujRoy2ep7S0FKGhoWCxWNi2bRvmzp2LvXv3IjY2ttWxHDlyBM+fN97P0B5VVNdhztdX8HdWScsH\nE0JIByF3IomJiUG/fv1w8uRJhIeHy+z39PTE3bt3WzxPfHw8ampqEBcXBz8/P0ybNg1z587Fvn37\nUF5e3mL90tJSbNq0qUM9IfYkt0zTIRBCiMrJnUgePHiAcePGgc1mN9ovYmFhgYKCghbPk5CQAH9/\nfxgaGkrKgoKCUF1djZSUlGZq1ouJicHAgQMxZMgQ+RqgQZduPJXaXjnbW0OREEKI6sidSPT19VFT\nU9Pk/uzsbPD5La8Hnp6eDgcHB6kyKysr8Hg8pKenN1v3wYMHOHbsGKKioloXdDtQXlWHP/7Ol2xv\nWxQAm26GzdQghJCOQe5eXi8vL/z0008IDQ2V2VdUVISjR4/Cz8+vxfMIBAIYGRnJlPP5fAgEgmbr\nrl69GtOnT4ednR2ys7NbHTtuVxo3AAAgAElEQVQAmJgYgMvlyFXn5foNLCzq468W12/z9HUkZS+r\nyKlvU68efMQuHtZunnKrQ30c+vq6TcbeVHlnRm3WDtRm1ZA7kSxYsADBwcGYMmUKRo8eDRaLhWvX\nruGPP/5AfHw8hEKhWhe/+vnnn5GRkYEdO3YoVL+4uFLh17awMJKqn59f3+dRVFS/4mBVdZ2k7GWf\nfZsEAOhhykNBQct9QG2lqKi+PdXVtY3GbmFh1GSbOitqs3agNstftyly39pycnLCDz/8gC5dumDj\nxo1gGAbffvst4uLiYG9vjwMHDqBnz54tnofP5zfaqS4QCJq8NVZXV4f169cjPDwcYrEYAoFAco6q\nqqpWddJrSp2w/rLlDT/FVjYkhJD2SqEBDC4uLvjuu+9QWlqKJ0+egGEY9OzZE6amrZ/23MHBQaYv\nJCcnB1VVVTJ9Jw2qqqqQm5uL6OhoREdHS+1buHAhbG1t8csvv8jfIDW79OdTlFXWoYeZASxNDVqu\nQAghHYhciaSqqgoRERF44403MGnSJBgbG8Pd3V2hFw4ICMDu3btRXl4ueXLr9OnT0NfXh7d3408z\nGRgYYP/+/VJlBQUFWLRoERYtWgQfn/a5bOzvD/IAAP0czDQcCSGEqJ5ct7Z4PB7u3r0LoVCo9AsH\nBwdDV1cXkZGRSEpKwuHDhxEXF4fQ0FCpR4IDAwOxfPlyAACXy8XgwYOlfjw8PAAAzs7Okt/VqWe3\n+und+/YyadXx1bVC3H9SDACYPNxRbXERQoimyN1H4u/vj6SkJKVf2NjYGPv27YNIJEJERAS2bNmC\nkJAQzJs3T+o4kUgEsVis9Oupig6Xgz1LR2Bx8IBWHX/2eqbkdxbax5NahBCiSgqtRxIZGYlly5Zh\n6tSpsLGxgb6+vsxxL15VNMXJyUnmVtXLWlr/3cbGBg8fPmzxtTSlqqZ+7rBpo3qDzaZEQgjpfORO\nJGPHjgVQP+HiyZMnmzzu/v37ikfVSRQJqvHL71kAAGebrhqOhhBC1EPuRPLhhx+qI45O6e7jIsnv\nZsayV20dnVAkhljMQFdH8QGehJCOT65EcvPmTdjb26Nr164YNGgQ9PT01BVXp9AwMfLs1/vAkKej\n2WBU7PcHedh28i8AwPy33OHhZK7hiAghmtKqRFJeXo6wsDDcunVLUmZmZoYdO3bAzc1NbcF1ZILK\nWuw78wAA0E5mQ2lUwyIACbdy8OiZAB9McEMPsy5NHl9aUYuV+35Dcdm/863lFFbCw0nNgRJC2q1W\nPbX1zTff4ObNm3jttdfw6aefIiQkBGVlZVi6dKm64+uwCkqqJb+/0qv1AzXbWkFpleT3p/kV+GTn\nddTUyS4uxjAMzv2WhYVbEiVJhG/Qua6yCCGKadUVyYULFzBmzBhs3rxZUubo6IgVK1bgyZMnsLOz\nU1uAHVVlTR0AYMxgW5gYtd9bgI09BFBdI4TeP/0eQpEYDzOLsenHW6gV/vsY9uezvFBUVoPYo7fb\nLFZCSPvUqkTy9OlTmdl+/f39wTAM8vLyKJG8RCQWY+Ph+tuAnHb+yK+uTv24GACYva7+UesN8Tfx\n3rhXkJEjwHdnpR+t9nLthvBxr4DLYaOorOnlBAgh2qNViaSmpgY8Hk+qrGHsiCpGuXc2ItG/yw+/\n6mGlwUgU86ygAp/v/U2m/IvZ3uhJa6gQQl7S6qe2cnJy8ODBA8l2WVn9VMTZ2dlS5Q1cXV1VEF7H\nVF1b38fg5mCKbl15LRzdfnz1gS+WbJOdteDruX4w5OlAh9t0l1pVjRDPCirAZrNgZ2lEgy8J0SKt\nTiSbNm3Cpk2bZMpXrFghtc0wDFgsllYPSIw5Wn9bi8uWewYajTLl6yP8/17Bzv93D379usPJ2hgT\nRjij+J/1Vpry46U0/HgpTbI94VV7mi6fEC3SqkTy8pTtRFrDjazLfz7FBH97VFTV3+4b79/xvkyH\nuHXHELfukm0up+lkWFha3Wi5oKJW5XERQtqvViWSN998U91xdGh5Rf+umvjH3/korahFV0Nd2HXv\n3Mt4Vtf+2z/m7mgGFoBbjwpx8cZT3EkvBE+Xi0nDHGGgx0UPsy54XlyJe4+L0NfeFL26N754GSGk\n41FoYSsiTcz827l++GIqauvE4Ol1/mlDxvrYQVeHgwB3K+jpcnA3owi3HhUCAPL/GUez6cdbMvWO\nXUmHk40x3vDtBUtTAxjydFBTJ4IulwMOmwU93c7/3hHSmVAiUbHauvqxFmN9Ov8j0WwWC4GD/l1W\nuYdZ61d/TMsuxcZGkkwD6mchpOOgRKIm3q7dNB1CmzPl60vGpABA4u0cMGDg3ccSz4sqYcrXR2pW\nCbYcv9Piuf5f0mNKJIR0EJRIVKCLvuxUITXC9rMYl6b4u/eQ/G5rWd9fNMDZQpJsqmuFuJtRjOfF\nlRjW3wr6ulykPS3Fuh9uQChi8PO1x/B169GuZwYghFAiUQlXOxOwWP/O9gsAZnz68muJvi4Xni4W\nUmXOPf+dsuXYlXRU14owaSgtUUxIe9axBjq0Y2+Pcpb8HvX2AHA62BiS9qqOruwIaffo205F/s4q\nkfxu0YFGs7dHe5aOwMfTBgAAzv2WhdnrLuLI5bQWahFCNIUSiYo03NWyNDWAKb/zrYbY1l6eYuVM\nciZmr7uI248KNBQRIaQplEhUjKaYUg3nnl0R9fYA+LxiKVX+6+0cDUVECGkKJRIV4f6TQahvRHVc\nbE3w3ht9sWiKh2R9lD8e5qNI0PjULIQQzaBvPRV5M8ABADDrde2d9Vhd3BzMEP2+j2R7ybYkFJRU\nNVODENKW6PFfFbHoypMajEdUq6uhHnS5bMkqjY9zy2BODzUQ0i7QFQnpMHYsGQYbiy4AgG0n/8Kc\nr69g98/36FYXIRpGiYR0KB5O5pLfa+pEuHonF0u2JaGiuk6DURGi3SiRkA5l0lBHzH69j0x58t3n\nGoiGEAJoOJGkpaUhJCQEHh4e8Pf3R0xMDEQiUbN1bt++jWXLliEwMBAeHh4YPXo04uLiUFNT00ZR\nE03zd++BPUtHSE8QeScHYjHTTC1CiLporLO9tLQUoaGhcHJywrZt25CZmYkvv/wSYrEYCxcubLLe\nmTNnkJmZifDwcNjZ2eHhw4eIiYnBw4cPsWXLljZsAWkPxvrY4kxyJp7kluHRs1L0tunaciVCiEpp\nLJHEx8ejpqYGcXFxMDQ0hJ+fH8rLyxEXF4fw8HAYGho2Wi88PBympqaS7cGDB0NPTw8rVqzA06dP\nYW1t3VZNIO3AaO/6RAIAOYWVlEgI0QCN3dpKSEiAv7+/VMIICgpCdXU1UlJSmqz3YhJp0KdP/T3z\nvLw81QdK2jW+gS7MjeunpNl35gFSs0tQUV2HknK61UlIW9FYIklPT4eDg4NUmZWVFXg8HtLT0+U6\n182bN8Fms2Fra6vKEEkHMeHVfxfAij5wA5Gbf8WiuKv4K71Qg1ERoj00dmtLIBDAyMhIppzP50Mg\nELT6PPn5+di+fTvGjx8PMzOzFo83MTEAl6v4muAWFrIxd3btvc3jhxtBV18X245KL93b3FK+C4IH\nYKRX0//waO9tVgdqs3ZQR5s79Mj22tpaLFiwAAYGBli2bFmr6hQXVyr8ehYWRsjPL1O4fkfUUdo8\nyMkMq971hqCyDg8zi/Hfq4+bPX5z/J/YHP9nk/tf8+oJrz7d4GhlrOJI26eO8jmrErVZ/rpN0Vgi\n4fP5KC8vlykXCATg8/kt1mcYBlFRUUhLS8PBgwdhbKwd/8OTpllbGMIaQB87Ewxy7QYOm4WMHAF2\n/b/7GNK3O+x7GMHF1gSf7Wm6D67Bud+ycO63LHDYLIj+eay4j50Jpo5wkiwbTAipp7FE4uDgINMX\nkpOTg6qqKpm+k8asWbMGFy5cwJ49e+DoSEuxEmk2FvUPcfQw6wJftx5S+7YvGor80ioY8nQgFIn/\n+S+DOqEYqdkl2PHTXcmxohfGptx/UozP9/4GABgz2BYTAxzA5dCYXkI09n9BQEAAEhMTpa5KTp8+\nDX19fXh7ezdb95tvvsEPP/yADRs2YNCgQeoOlXQyeroc2FgYoquhHsyNedDX5cKQpwMTIz1497HE\nqa/H461hzf/j5Oz1TLy34TIeZha3UdSEtF8auyIJDg7G999/j8jISISHhyMrKwtxcXEIDQ2VeiQ4\nMDAQXl5eWLt2LQDg1KlT2LhxIyZOnAhLS0vcvHlTcqytrW2jjwcTIq/Xfezwuo+dVFna01I8zhHg\n4PlUSdmXB//EkL7dETzSCXVCMbrwdKDLZaOqRoScwgqUVdbBvocRjLrogs2iVc9I56SxRGJsbIx9\n+/Zh5cqViIiIAJ/PR0hICCIjI6WOE4lEEIvFku2rV68CAI4fP47jx49LHRsdHY2JEyeqP3iilZys\njeFkbYzBr1ji4x3XUFNbP53Ptbu5uHY3V65zdTc1gK9bdxSX16C/kzksTQ3wJLcMWXllyC2qwiAX\nC/S1N4WBHhcsSkCknWMxDKNVExQp85QGPeWhHVrbZpFYjPD1l9Uez5rwwehh1kWtr0Gfs3bodE9t\nEdLRcdhsmcXMGIYBwwBstvRVxNnrmeDpceDuaI7dP9/Dvcet71v5ZOd1mTJLEx7G+tjBvgcfVuYG\ntMQz0ShKJISoEIvFQmN3osYM/nfw45LgAa0615nkJzhy+VGj+54XV2HfmQeS7fVzhsDcmFaMJJpB\niYSQdmqsjx3G+tiBYRhJP4lQJMZne1KQUyg9sPbj7dcAAJOGOsC7jyVM+Xp0lULaDCUSQtq5Fzvb\nuRw21oT7SLZPXc3AiV8zJNvHrqTj2BXp8VmzXnfFq+5W6g+UaC36JwshHdg4P3vsWToCliZN39ba\ne/oBnhcpPjUQIS2hKxJCOoHo94fIlNUJRVgUdxUV1UJU1gg1EBXRFnRFQkgnpcPlSG5p5RbSFQlR\nH7oiIaQT0+HW/1tx5/+7hzqRGL1tjNU+JoVoH0okhHRiIwfZ4FTSYwCQelz4k5me0OGy0c2kfq4x\nQpRBf0GEdGJ8A11s+tAPdx8X4XRyJp4VVAAA1nz/h9RxFiY82Jh3AZvFwqhBNrDoyoNIzMCQpwOe\nHn1NkObRXwghnZyxoR583XrA160HbqUVIObobfS2MUZBaTWKy+rXts8vrkJ+cRUA4I+/82XO4Wrb\nFX79eqC3jTG6mRi0afyk/aNEQogW8XAyl5nWBQC4+jq4n5aPU1cf46+MIgD1E0vm/vPY8IPMEjzI\nLJEczzfQAQC4O5lDX4cDoZhBnVCE7qYGcHc0h41FfT9MayacbBhw+eLAS9KxUCIhhMDESB+9bbpi\n0dT+UuUMw6Csqg4LYhOlygWVdQCAxNs5Mud6eUBkA76BDrhcNkRiBqXltS3G1M/BDGKxGGIGKBJU\n450xruhjZ9LaJpE2RImEENIkFosFvoGu5CqGYRjkFFaiuKwGdzOK4OFkBp4eFxwOG39nFuNkYgbK\n/kkyLyurrIOZsT64Ov+OOrA04eH5P7fUXnYnvVBqe8OhPyW/+/XrDidrY1TWCGFiqAcjA13ocNmw\nszSCni5H2WYTOVEiIYS0GovFgpV5F1iZd0Ffe+lF5KzNu2D4QBuFzy1mGIhEDGrqRGCx6qeDYbNY\nEDMMvv3vXfyZWiA59uqdXFy90/gaMP2dzOHr1h09uxnC0pT6c9oCJRJCSLvAZrHA5rIkY19eFDnJ\nHQBQXStE/IU0ZOeXY/gAazwvrgRPjwuhUCyZc+xmWgFuphVI1dflsmHK10efXiaoqhFixECbZtfX\nIPKhREII6TD0dbkIHeva6L4xg21x/0kJHmYW43FuGYQiMcqr6pBTWIlaoRi5RZWShweS7z7HQqEY\n/ajPRSUokRBCOgUdLgfujmZwdzSTKhczDIRCMZ4WVKC0vBZ7z9xHWWUdNh36Exw2C5OHOYKnzwUL\nLPTuaQxLerxZbpRICCGdGpvFgq4OB/Y9+ACA6J4++PbUPdx+VAiRmEH8xTSZOh7/JKO3A51h0ZUW\nDGsJJRJCiFYx0NfBgskeMOLz8OuNLNTWiVBRVYfTyU9QKKgfoHnrUeE//72GycMc8Zp3T1oorBmU\nSAghWklfj4v+TuaS7YYnzuqEYjwtKMfKfb8DAI5cfoQjlx9h1bvesLYw1Eis7R2lWEIIeYEOl41e\n3fnYsXgohg2wlpT/Z3cKsvPKNRhZ+0WJhBBCGqGrw8E7o13w9Vw/SdmKPSn4+dpjCEVizQXWDtGt\nLUIIaYaJkR5i57+KeTG/AqifAubYlXRYW3RBF30dDO7TDV59LGHI09FwpJpDVySEENICQ54OdkUN\nx6ShDgDqnwR7ml+Bv7NK8P25vzEv5lfkFFZoOErNoSsSQghpBTaLhaAhvRA0pBcAoLS8Bn9nl2L7\nyb8AAJ/svI5lMwait01XDUapGRq9IklLS0NISAg8PDzg7++PmJgYiESiFuuVlZVh2bJl8PLygqen\nJxYvXozi4uI2iJgQQuoZG+rBy7UbVs72lpRFH7iBz/em4NtTd7XqCkVjiaS0tBShoaFgsVjYtm0b\n5s6di7179yI2NrbFugsWLMD169exevVqrFu3Dn/99Rfmzp3bBlETQog0m26GiJ3/KnqY1Y+Iz3xe\njuS7z/HJzutak0w0dmsrPj4eNTU1iIuLg6GhIfz8/FBeXo64uDiEh4fD0LDx57X//PNPJCYm4sCB\nA/Dy8gIAWFpaYvLkyUhKSoKvr29bNoMQQmDI08GacB8AwB8P87H1xB0A9be79HQ4mDTUAd1MeOBy\n2DDQ58K4ix66Gup2moW8NJZIEhIS4O/vL5UwgoKC8NVXXyElJQUjRsiu4tZQz9zcXJJEAMDd3R02\nNjZISEigREII0ShPFwv8J2QQDl9Ixd/ZpaipE+Hg+dQmj2cBMOHrgf3PKpHOPU1QUV2H3jbGqKkT\nwcKYBxaLBRYL9T9gwdhQF1wOGxw2CxwOC3o6HHA4bHBYLLDZLOjrciT72CyW2hOWxhJJeno6fHx8\npMqsrKzA4/GQnp7eZCJJT0+Hg4ODTLmjoyPS0xtfmY0QQtqSfQ8+ls7wREFJFa7dew49HQ6EIjEy\nngmgo8PG/cfF4HBYYIEFXR02KquFKKqon57l2t36dVZuPyps7iXkUj/fGBuLp3vC0VL1o/M1lkgE\nAgGMjGTXA+Dz+RAIBArVy87ObvF1TUwMwOUqvoKaNq5hQG3WDtRm9Zy/T+9urTpWKBKjoqoOpeU1\nyCuuQkFJFfR0OWAYBmJx/SzGz/LLYWigC6FIDKFQjCe5Apjy9SES1y8KJhSLkfG0FN1MDSASMRCJ\nxZJ9LBbQ1VBPLW3Wusd/i4srFa5rYWGE/PwyFUbT/lGbtQO1uf3gcViwMzeAnbnsdPYDHEwbqdF6\nyrS5uQSksae2+Hw+ystl560RCATg8/nN1isrk30jWqpHCCFEPTSWSBwcHGT6NHJyclBVVdVoH8iL\n9TIyMmTKm+o7IYQQol4aSyQBAQFITEyUuio5ffo09PX14e3t3Wy9/Px8/P7775KyO3fuICsrCwEB\nAWqNmRBCiCyNJZLg4GDo6uoiMjISSUlJOHz4MOLi4hAaGir1SHBgYCCWL18u2R4wYAD8/f0RFRWF\nc+fO4fz581iyZAk8PT3p0V9CCNEAjSUSY2Nj7Nu3DyKRCBEREdiyZQtCQkIwb948qeNEIhHEYukp\nmzdt2gRvb28sX74cH3/8Mdzc3BAXF9eW4RNCCPkHi2EYRtNBtCVlntJor095qBO1WTtQm7VDp3tq\nixBCSOdAiYQQQohStO7WFiGEENWiKxJCCCFKoURCCCFEKZRICCGEKIUSCSGEEKVQIiGEEKIUSiSE\nEEKUQomEEEKIUiiRAEhLS0NISAg8PDzg7++PmJgYiESiFuuVlZVh2bJl8PLygqenJxYvXozi4uI2\niFh5irT59u3bWLZsGQIDA+Hh4YHRo0cjLi4ONTU1bRS1chT9nBuIxWJMnDgRLi4uuHTpkhojVR1l\n2nzu3DlMmjQJ7u7uGDx4MN59911UViq+MFxbUbTNd+7cwezZs+Ht7Q1vb2+Ehobi1q1bbRCxcp48\neYIVK1Zg3Lhx6NOnD2bOnNmqeqr8/tK6FRJfVlpaitDQUDg5OWHbtm3IzMzEl19+CbFYjIULFzZb\nd8GCBcjIyMDq1avBZrPx1VdfYe7cuTh48GAbRa8YRdt85swZZGZmIjw8HHZ2dnj48CFiYmLw8OFD\nbNmypQ1bID9lPucGR44cwfPnz9Ucqeoo0+YjR45g5cqVCAsLw8cffwyBQIDk5GS5Eq8mKNrmnJwc\nzJo1C6+88grWr18PANi9ezdmzZqFU6dOwdrauq2aILfU1FRcuXIFHh4eEAqFra6n0u8vRsvt2LGD\nGTRoEFNWViYp+/bbbxl3d3epspfduHGDcXZ2ZlJSUiRlt27dYpydnZmrV6+qNWZlKdrmwsJCmbL4\n+HjG2dmZyc7OVkusqqJomxuUlJQwgwcPZn788UfG2dmZuXjxojrDVQllPuf+/fszhw8fboswVUrR\nNh88eJBxdXVlBAKBpKykpIRxdXVlfvjhB7XGrCyRSCT5PTIykpkxY0aLdVT9/aX1t7YSEhLg7+8v\ntQZKUFAQqqurkZKS0mw9c3NzeHl5Scrc3d1hY2ODhIQEtcasLEXbbGoqu150nz59AAB5eXmqD1SF\nFG1zg5iYGAwcOBBDhgxRZ5gqpWibz5w5AwCYMGGC2mNUNUXbLBQKweFwwOPxJGUGBgbgcDhg2vks\nUmy2/F/jqv7+0vpE0tgSvVZWVuDxeDJLAbdUDwAcHR2brdceKNrmxty8eRNsNhu2traqDFHllGnz\ngwcPcOzYMURFRakzRJVTtM23b9+Gvb09jh49ioCAAPTt2xeTJ0/GjRs31B2y0hRt82uvvQYej4d1\n69ahsLAQhYWFiI6OhrGxMcaOHavusNucqr+/tD6RCAQCGBnJzrPP5/MhEAhUXq89UFXs+fn52L59\nO8aPHw8zMzNVhqhyyrR59erVmD59Ouzs7NQVnloo2uaCggJkZGRg+/btWLJkCbZv3w4ej4ewsDAU\nFBSoM2SlKdpmS0tL7N+/H+fOnYOvry98fX1x7tw57N69u9Er8Y5O1d9fWp9IiGJqa2uxYMECGBgY\nYNmyZZoOR21+/vlnZGRk4IMPPtB0KG2GYRhUVlZizZo1eOONNxAQEIBt27aBw+HgwIEDmg5PLfLy\n8jB//nz07dsXO3fuxM6dO+Hm5ob33nsPz54903R47Z7WJxI+n4/y8nKZcoFAAD6f32y9sjLZlcZa\nqtceKNrmBgzDICoqCmlpafj2229hbGysjjBVSpE219XVYf369QgPD4dYLIZAIJCco6qqqtHztSfK\n/G2zWCwMHjxYUmZoaIi+ffvi0aNHaolVVRRt8+7duyEUChEbG4uAgAAEBAQgNjYWHA4He/bsUWfI\nGqHq7y+tTyQODg4y9wRzcnJQVVXV6D3EF+tlZGTIlDd177E9UbTNDdasWYMLFy5g69atcHR0VFeY\nKqVIm6uqqpCbm4vo6Gh4eXnBy8sL48ePBwAsXLgQb775ptrjVoain7OjoyMYhpHpZGYYBiwWSy2x\nqoqibU5PT4eTkxN0dHQkZbq6unByckJmZqba4tUUVX9/aX0iCQgIQGJiotS/Yk6fPg19fX14e3s3\nWy8/Px+///67pOzOnTvIyspCQECAWmNWlqJtBoBvvvkGP/zwAzZs2IBBgwapO1SVUaTNBgYG2L9/\nv9TPxo0bAQCLFi3CV1991SaxK0rRz3nYsGEAgOvXr0vKysrKcPfuXbi6uqotXlVQtM1WVlZITU1F\nbW2tpKy2thapqantegyJolT9/cX5/PPPP1dhfB1O7969cfjwYVy/fh3dunVDUlISNm7ciJCQEAwd\nOlRyXGBgIB48eICRI0cCAHr06IGbN2/i6NGj6NGjBzIyMvD555/D0dERCxYs0FRzWkXRNp86dQpf\nfPEF3nzzTQwePBi5ubmSH11dXalHJ9sbRdrMZrNhY2Mj9dOQXEJCQuDj46PBFrVM0c/Z0tIS9+/f\nx6FDh2BiYoLnz59j1apVKCkpwfr166Gvr6+pJrVI0TZbWFjgu+++w19//QUjIyNkZGRg3bp1ePjw\nIVauXAlzc3NNNalFVVVVuHDhAtLS0pCYmIjS0lKYmZkhLS0N1tbW0NHRUf/3l9wjTzqh1NRUZubM\nmUy/fv0YPz8/ZtOmTYxQKJQ6Zvjw4UxUVJRUWWlpKbN06VLG09OTGTBgALNo0aJGB+21R4q0OSoq\ninF2dm7059ixY23dBLkp+jm/KCsrq8MMSGQYxdtcXl7OrFixgvH29mb69evHhISEMA8ePGjL0BWm\naJuTkpKYt99+m/Hy8mK8vLyY6dOnM8nJyW0ZukIa/iYb+8nKymIYRv3fX7RmOyGEEKVofR8JIYQQ\n5VAiIYQQohRKJIQQQpRCiYQQQohSKJEQQghRCiUSQgghSqFEQkgnMnPmTKmlVq9fvw4XFxepUeqE\nqJrWL7VLiDyOHz8uNdsxl8uFhYUFhg8fjgULFnSICSwJUTVKJIQoYOHChejRoweqq6tx/fp1HDx4\nUDKtSHuf2JAQVaNEQogChg4dKllmeOrUqQDq1y65c+cO3N3dNRkaIW2O+kgIUQFPT08AkJly/Nix\nY3jzzTfh7u6OwYMHIyoqqtFVBi9fvozp06djwIAB8PT0RHBwMM6fPy/Zf/78ebz33nvw9/eHm5sb\nRo0aha1bt0IkEqm3YYS0Al2REKICT58+BQCpRYHi4uKwdetWBAUFYcqUKcjPz8f+/ftx584dHD9+\nXDKL7pEjR/Dpp5/C1dUVERER6NKlC+7evYurV69i1KhRAIATJ07AwMAAs2bNgoGBAZKTkxEbG4vy\n8vIOt5Y86XwokRCiAIFAgKKiIlRXVyMlJQWHDh2CqakpvLy8AADZ2dnYtm0bPvroI8yePVtSLyAg\nAMHBwThx4gSmTZuGshime3AAAALVSURBVLIyrF27FgMGDMD+/fuhq6srOfbF+VS//vprqenbp02b\nhhUrVuDQoUNYuHChVD1C2holEkIU8M4770htOzs7Y+3atZI1Wc6fPw+GYRAYGIiioiLJcba2trCw\nsEBKSgqmTZuGxMREVFZW4v3335dJBi922r+YRMrLy1FbW4tBgwbh8OHDSE9Pb/cLTpHOjRIJIQr4\n4osvYGtri5KSEsTHxyMtLU1qYa/Hjx9DLBZLbk29rCG5ZGVlAahfkKk5qamp2Lx5M5KTk2XWJG9s\n7W1C2hIlEkIU4OHhIXlqa+TIkZgwYQKWLFmC48ePg81mQywWg8PhYOfOnY0+DvxiX0pLBAIBZsyY\nAUNDQ8ybNw+2trbQ09PD3bt38dVXX0EsFqusXYQoghIJIUrS09PDhx9+iEWLFuHMmTMICgqCra0t\nRCIR7OzsYGNj02RdW1tbAPVXHE0dl5KSgpKSEsTFxUn6YID6fhhC2gN6/JcQFRgzZgysra2xc+dO\nAPVrgrPZbGzdulXmWLFYjJKSEgCAn58fDAwM8M0336C2tlbquIbOdjabLbUNALW1tTh48KBa2kKI\nvOiKhBAV4HA4eOeddxAdHY2EhAQEBARg3rx52Lx5M7KysjB8+HDweDxkZWXhf//7H+bMmYPJkyfD\nyMgIS5cuxYoVKzB58mQEBQWhS5cuuHfvHnR1dfHZZ59hwIABMDY2xtKlSzFz5kywWCz89NNPoFWy\nSXtBVySEqMhbb70FIyMjyVXJnDlzsGnTJgiFQmzZsgUbNmxAQkICRo0aBV9fX0m9qVOnYuvWrdDX\n18fWrVuxadMmPHr0CP7+/gAAExMT7NixAxYWFti8eTN2794NX19ffPTRRxppJyEvYzH0zxpCCCFK\noCsSQgghSqFEQgghRCmUSAghhCiFEgkhhBClUCIhhBCiFEokhBBClEKJhBBCiFIokRBCCFEKJRJC\nCCFK+f+4dOro97aGDgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.step(recall, precision, where='post')\n",
    "plt.title('Precision-Recall curve')\n",
    "plt.xlabel('Recall')\n",
    "plt.ylabel('Precision')\n",
    "pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}