Supervised regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1bba0128",
   "metadata": {},
   "source": [
    "# Supervised regression\n",
    "\n",
    "This is an example of how to build and optimize neural networks with PyTorch with the objective of predicting a known feature in the training dataset.\n",
    "\n",
    "The logic here is very similar to the classification problem and the code is also very close, but the final objective is to minimize the error in the prediction of the missing feature. This is achieved by minimizing the mean-squared-error between the prediction and the target features. As noticed during the presentation, by minimizing the mean-squared-error, we assume that the underlying error distribution is Gaussian. One could use the mean absolute error instead when assuming the distribution is Laplacian.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "d0681795",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: torchvision in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (0.10.0)\n",
      "Requirement already satisfied: torch in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (1.9.0)\n",
      "Requirement already satisfied: pandas in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (1.3.0)\n",
      "Requirement already satisfied: numpy in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (1.19.2)\n",
      "Requirement already satisfied: matplotlib in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (3.4.2)\n",
      "Requirement already satisfied: ipympl in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (0.8.2)\n",
      "Collecting torchbnn\n",
      "  Downloading torchbnn-1.2-py3-none-any.whl (12 kB)\n",
      "Requirement already satisfied: pillow>=5.3.0 in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from torchvision) (8.3.1)\n",
      "Requirement already satisfied: typing_extensions in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from torch) (3.10.0.0)\n",
      "Requirement already satisfied: python-dateutil>=2.7.3 in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from pandas) (2.8.2)\n",
      "Requirement already satisfied: pytz>=2017.3 in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from pandas) (2021.1)\n",
      "Requirement already satisfied: six>=1.5 in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from python-dateutil>=2.7.3->pandas) (1.16.0)\n",
      "Requirement already satisfied: pyparsing>=2.2.1 in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from matplotlib) (2.4.7)\n",
      "Requirement already satisfied: kiwisolver>=1.0.1 in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from matplotlib) (1.3.1)\n",
      "Requirement already satisfied: cycler>=0.10 in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from matplotlib) (0.10.0)\n",
      "Requirement already satisfied: ipywidgets>=7.6.0 in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from ipympl) (7.6.3)\n",
      "Requirement already satisfied: ipykernel>=4.7 in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from ipympl) (5.5.5)\n",
      "Requirement already satisfied: tornado>=4.2 in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from ipykernel>=4.7->ipympl) (6.1)\n",
      "Requirement already satisfied: jupyter-client in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from ipykernel>=4.7->ipympl) (6.1.12)\n",
      "Requirement already satisfied: ipython>=5.0.0 in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from ipykernel>=4.7->ipympl) (7.25.0)\n",
      "Requirement already satisfied: traitlets>=4.1.0 in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from ipykernel>=4.7->ipympl) (5.0.5)\n",
      "Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from ipython>=5.0.0->ipykernel>=4.7->ipympl) (3.0.19)\n",
      "Requirement already satisfied: pickleshare in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from ipython>=5.0.0->ipykernel>=4.7->ipympl) (0.7.5)\n",
      "Requirement already satisfied: decorator in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from ipython>=5.0.0->ipykernel>=4.7->ipympl) (5.0.9)\n",
      "Requirement already satisfied: jedi>=0.16 in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from ipython>=5.0.0->ipykernel>=4.7->ipympl) (0.18.0)\n",
      "Requirement already satisfied: pygments in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from ipython>=5.0.0->ipykernel>=4.7->ipympl) (2.9.0)\n",
      "Requirement already satisfied: backcall in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from ipython>=5.0.0->ipykernel>=4.7->ipympl) (0.2.0)\n",
      "Requirement already satisfied: setuptools>=18.5 in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from ipython>=5.0.0->ipykernel>=4.7->ipympl) (49.6.0.post20210108)\n",
      "Requirement already satisfied: matplotlib-inline in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from ipython>=5.0.0->ipykernel>=4.7->ipympl) (0.1.2)\n",
      "Requirement already satisfied: pexpect>4.3 in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from ipython>=5.0.0->ipykernel>=4.7->ipympl) (4.8.0)\n",
      "Requirement already satisfied: widgetsnbextension~=3.5.0 in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from ipywidgets>=7.6.0->ipympl) (3.5.1)\n",
      "Requirement already satisfied: jupyterlab-widgets>=1.0.0 in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from ipywidgets>=7.6.0->ipympl) (1.0.0)\n",
      "Requirement already satisfied: nbformat>=4.2.0 in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from ipywidgets>=7.6.0->ipympl) (5.1.3)\n",
      "Requirement already satisfied: parso<0.9.0,>=0.8.0 in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from jedi>=0.16->ipython>=5.0.0->ipykernel>=4.7->ipympl) (0.8.2)\n",
      "Requirement already satisfied: jupyter-core in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from nbformat>=4.2.0->ipywidgets>=7.6.0->ipympl) (4.7.1)\n",
      "Requirement already satisfied: jsonschema!=2.5.0,>=2.4 in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from nbformat>=4.2.0->ipywidgets>=7.6.0->ipympl) (3.2.0)\n",
      "Requirement already satisfied: ipython-genutils in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from nbformat>=4.2.0->ipywidgets>=7.6.0->ipympl) (0.2.0)\n",
      "Requirement already satisfied: pyrsistent>=0.14.0 in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=4.2.0->ipywidgets>=7.6.0->ipympl) (0.17.3)\n",
      "Requirement already satisfied: importlib-metadata in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=4.2.0->ipywidgets>=7.6.0->ipympl) (1.7.0)\n",
      "Requirement already satisfied: attrs>=17.4.0 in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=4.2.0->ipywidgets>=7.6.0->ipympl) (21.2.0)\n",
      "Requirement already satisfied: ptyprocess>=0.5 in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from pexpect>4.3->ipython>=5.0.0->ipykernel>=4.7->ipympl) (0.7.0)\n",
      "Requirement already satisfied: wcwidth in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython>=5.0.0->ipykernel>=4.7->ipympl) (0.2.5)\n",
      "Requirement already satisfied: notebook>=4.4.1 in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from widgetsnbextension~=3.5.0->ipywidgets>=7.6.0->ipympl) (6.4.0)\n",
      "Requirement already satisfied: nbconvert in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.6.0->ipympl) (6.1.0)\n",
      "Requirement already satisfied: prometheus-client in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.6.0->ipympl) (0.11.0)\n",
      "Requirement already satisfied: jinja2 in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.6.0->ipympl) (3.0.1)\n",
      "Requirement already satisfied: Send2Trash>=1.5.0 in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.6.0->ipympl) (1.7.1)\n",
      "Requirement already satisfied: argon2-cffi in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.6.0->ipympl) (20.1.0)\n",
      "Requirement already satisfied: pyzmq>=17 in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.6.0->ipympl) (22.1.0)\n",
      "Requirement already satisfied: terminado>=0.8.3 in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.6.0->ipympl) (0.10.1)\n",
      "Requirement already satisfied: cffi>=1.0.0 in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from argon2-cffi->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.6.0->ipympl) (1.14.6)\n",
      "Requirement already satisfied: pycparser in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from cffi>=1.0.0->argon2-cffi->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.6.0->ipympl) (2.20)\n",
      "Requirement already satisfied: zipp>=0.5 in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from importlib-metadata->jsonschema!=2.5.0,>=2.4->nbformat>=4.2.0->ipywidgets>=7.6.0->ipympl) (3.5.0)\n",
      "Requirement already satisfied: MarkupSafe>=2.0 in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from jinja2->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.6.0->ipympl) (2.0.1)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: entrypoints>=0.2.2 in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.6.0->ipympl) (0.3)\n",
      "Requirement already satisfied: pandocfilters>=1.4.1 in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.6.0->ipympl) (1.4.2)\n",
      "Requirement already satisfied: mistune<2,>=0.8.1 in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.6.0->ipympl) (0.8.4)\n",
      "Requirement already satisfied: jupyterlab-pygments in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.6.0->ipympl) (0.1.2)\n",
      "Requirement already satisfied: bleach in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.6.0->ipympl) (3.3.1)\n",
      "Requirement already satisfied: defusedxml in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.6.0->ipympl) (0.7.1)\n",
      "Requirement already satisfied: nbclient<0.6.0,>=0.5.0 in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.6.0->ipympl) (0.5.3)\n",
      "Requirement already satisfied: testpath in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.6.0->ipympl) (0.5.0)\n",
      "Requirement already satisfied: async-generator in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from nbclient<0.6.0,>=0.5.0->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.6.0->ipympl) (1.10)\n",
      "Requirement already satisfied: nest-asyncio in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from nbclient<0.6.0,>=0.5.0->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.6.0->ipympl) (1.5.1)\n",
      "Requirement already satisfied: packaging in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from bleach->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.6.0->ipympl) (21.0)\n",
      "Requirement already satisfied: webencodings in /home/danilo/miniconda3/envs/mlmkl/lib/python3.7/site-packages (from bleach->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.6.0->ipympl) (0.5.1)\n",
      "Installing collected packages: torchbnn\n",
      "Successfully installed torchbnn-1.2\n"
     ]
    }
   ],
   "source": [
    "!pip install torchvision torch pandas numpy matplotlib ipympl torchbnn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "23feddde",
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib notebook\n",
    "\n",
    "from typing import Tuple\n",
    "\n",
    "# import standard PyTorch modules\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "\n",
    "import torchbnn as bnn\n",
    "\n",
    "# import torchvision module to handle image manipulation\n",
    "import torchvision\n",
    "import torchvision.transforms as transforms\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from mpl_toolkits.mplot3d import axes3d\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb1286f0",
   "metadata": {},
   "source": [
    "We start by generating some fake dataset, which is simple enough that we can visualize the results easily. For this reason, the dataset will contain only two independent variables and a third feature variable which we want to determine in test data.\n",
    "\n",
    "The simulated example data will be $f(x) = (3 + \\kappa) x^2 + \\epsilon$, where $\\epsilon \\sim \\mathcal{N}(\\mu=0, \\sigma=10)$ and $\\kappa \\sim \\mathcal{N}(\\mu=0, \\sigma=0.03)$.\n",
    "\n",
    "In this case we do know the true model, so it is interesting to take some time to pinpoint the role of $\\kappa$ and $\\epsilon$. These variables add fluctuation to the results. $\\epsilon$ adds Gaussian noise in a way that is completely independent from $x$ and cannot be traced down to a particular functional dependence. $\\kappa$ changes a specific parameter of the model, in this case the coefficient 3, by around 1%.\n",
    "\n",
    "When fitting a model, the nomenclature *epistemic uncertainty* is often used to refer to uncertainties coming to effects related to different functional models. That is, one can imagine that there are different functions that may fit the data due to the effect of $\\kappa$, such as: $g(x) = 3x^2$ or $h(x) = 2.95x^2$.\n",
    "\n",
    "The nomenclature *aleatoric uncertainty* is used to refer to whichever uncertainty cannot be tracked down to a given model dependence. In this example, different constant factors could be added to the model $g$ to account for the fluctuations in $\\epsilon$.\n",
    "\n",
    "We will see these two effects later on, when we discuss Bayesian Neural Networks, so that we can predict the effect of those uncertainties."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "5d457cd8",
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_data(N: int) -> np.ndarray:\n",
    "    x = 2*np.random.randn(N, 1)\n",
    "    epsilon = 10*np.random.randn(N, 1)\n",
    "    kappa = 0.03*np.random.randn(N, 1)\n",
    "    z = (3 + kappa)*x**2 + epsilon\n",
    "    return np.concatenate((x, z), axis=1).astype(np.float32)\n",
    "\n",
    "train_data = generate_data(N=1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48433f6f",
   "metadata": {},
   "source": [
    "PyTorch allows you to create a class that outputs a single data entry and use that to feed input to your neural network. We will use the following class to feed the data to the neural network. This looks useless if all your data fits in a Numpy array, but notice that if you have a lot of data and cannot load it all in memory, this allows you to read data on demand, as you need it and only the needed samples are stored at a single time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "30205402",
   "metadata": {},
   "outputs": [],
   "source": [
    "class MyDataset(torch.utils.data.Dataset):\n",
    "    def __init__(self, data: np.ndarray):\n",
    "        self.data = data\n",
    "    def __len__(self) -> int:\n",
    "        \"\"\"How many samples do I have?\"\"\"\n",
    "        return len(self.data)\n",
    "    def __getitem__(self, idx):\n",
    "        # give me item with index idx\n",
    "        return {\"data\": self.data[idx, 0:1], \"target\": self.data[idx, 1:]}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "cc0b0774",
   "metadata": {},