Supervised classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1bba0128",
   "metadata": {},
   "source": [
    "# Supervised learning with PyTorch\n",
    "\n",
    "This is an example of how to build and optimize neural networks with PyTorch. PyTorch and Tensorflow offer a handy mechanism to provide automatic differentiation, using the chain rule in Calculus to calculate the derivative of a function very fast and with GPU support.\n",
    "\n",
    "Our dataset will consist of images of handwritten digits and the task shall be to classify those handwritten digits in the classes {0, 1, 2, 3, 4, 5, 6, 7, 8, 9}.\n",
    "\n",
    "If this was a regression problem, we would often try to minimise the mean-squared-error between the output of the neural network and the correct prediction. As we saw in the presentation, this assumes that the underlying probability distribution of the prediction is a Gaussian, which is certainly not true for the distribution of digit classes: for one, the digit classes are discrete and Gaussians are only defined for continuous outputs. The most general probability distribution for a choice of 10 classes is a Categorical distribution (https://en.wikipedia.org/wiki/Categorical_distribution), which is simply a discrete distribution with a given probability value for each class. How can we then sculpt a function that maps the input image to a given class?\n",
    "\n",
    "Suppose the neural network provides an output $f_k(x)$ in the form of a list of probabilities, informing us of the probability that a given image belongs to a certain class $k$. If we know that a given input image x belongs to class C, then the true probability t for this image x to belong to each class is zero for classes that differ from C and 1 for the class C. The network's objective will be to output such probabilities, so that only the i-th component of the output is 1 if the input belongs to class i. The presentation shows how the Bayes' rule leads us naturally to minimize the cross entropy between the target probabilities and the predicted probabilities: $- \\sum_k t_k \\log f_k(x)$. One can gain intuition on this by reading more on the Information Theory concept of cross-entropy and how it relates to Mutual Information: minimizing the mutual information between the labels distribution and the predicted one moves them closer together:  https://en.wikipedia.org/wiki/Cross_entropy\n",
    "\n",
    "The neural network will therefore model a parametrized function that maps the input image pixels into a vector with 10 components, which refer to the probability that the image correspond to that digit.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "d0681795",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: torchvision in /home/daniloefl/miniconda3/envs/ml2/lib/python3.6/site-packages (0.11.2)\r\n",
      "Requirement already satisfied: torch in /home/daniloefl/miniconda3/envs/ml2/lib/python3.6/site-packages (1.10.1)\r\n",
      "Requirement already satisfied: pandas in /home/daniloefl/miniconda3/envs/ml2/lib/python3.6/site-packages (1.1.5)\r\n",
      "Requirement already satisfied: numpy in /home/daniloefl/miniconda3/envs/ml2/lib/python3.6/site-packages (1.19.2)\r\n",
      "Requirement already satisfied: matplotlib in /home/daniloefl/miniconda3/envs/ml2/lib/python3.6/site-packages (3.3.4)\r\n",
      "Requirement already satisfied: pillow!=8.3.0,>=5.3.0 in /home/daniloefl/miniconda3/envs/ml2/lib/python3.6/site-packages (from torchvision) (8.3.1)\r\n",
      "Requirement already satisfied: typing_extensions in /home/daniloefl/miniconda3/envs/ml2/lib/python3.6/site-packages (from torch) (3.10.0.2)\r\n",
      "Requirement already satisfied: dataclasses in /home/daniloefl/miniconda3/envs/ml2/lib/python3.6/site-packages (from torch) (0.8)\r\n",
      "Requirement already satisfied: python-dateutil>=2.7.3 in /home/daniloefl/miniconda3/envs/ml2/lib/python3.6/site-packages (from pandas) (2.8.2)\r\n",
      "Requirement already satisfied: pytz>=2017.2 in /home/daniloefl/miniconda3/envs/ml2/lib/python3.6/site-packages (from pandas) (2021.3)\r\n",
      "Requirement already satisfied: cycler>=0.10 in /home/daniloefl/miniconda3/envs/ml2/lib/python3.6/site-packages (from matplotlib) (0.11.0)\r\n",
      "Requirement already satisfied: kiwisolver>=1.0.1 in /home/daniloefl/miniconda3/envs/ml2/lib/python3.6/site-packages (from matplotlib) (1.3.1)\r\n",
      "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.3 in /home/daniloefl/miniconda3/envs/ml2/lib/python3.6/site-packages (from matplotlib) (3.0.4)\r\n",
      "Requirement already satisfied: six>=1.5 in /home/daniloefl/miniconda3/envs/ml2/lib/python3.6/site-packages (from python-dateutil>=2.7.3->pandas) (1.16.0)\r\n"
     ]
    }
   ],
   "source": [
    "!pip install torchvision torch pandas numpy matplotlib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "23feddde",
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib notebook\n",
    "\n",
    "# import standard PyTorch modules\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "\n",
    "# import torchvision module to handle image manipulation\n",
    "import torchvision\n",
    "import torchvision.transforms as transforms\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48433f6f",
   "metadata": {},
   "source": [
    "PyTorch allows you to create a class that outputs a single data entry and use that to feed input to your neural network. An example of how you would write such a class is given below, but for this exercise we shall use something ready-made which loads the standard MNIST handwritten digits dataset, just to simplify things.\n",
    "\n",
    "If you want to load a different dataset (for example your own data!), feel free to copy and modify the example Dataset class below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "30205402",
   "metadata": {},
   "outputs": [],
   "source": [
    "class MyDataset(object):\n",
    "    def __init__(self):\n",
    "        pass\n",
    "    def __len__(self):\n",
    "        return 10 # these are how many samples I have\n",
    "    def __getitem__(self, idx):\n",
    "        # give me item with index idx\n",
    "        # read this from some file, but for the purposes of this example, generate a random image and label\n",
    "        my_image = np.random.randn(10,10, 1)\n",
    "        my_label = np.array(np.random.randint(10))\n",
    "        my_image = torch.from_numpy(my_image)\n",
    "        my_label = torch.from_numpy(my_label)\n",
    "        return {\"data\": my_image, \"label\": my_label}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "cc0b0774",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10\n"
     ]
    }
   ],
   "source": [
    "my_dataset = MyDataset()\n",
    "print(len(my_dataset))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "6dccfac6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'data': tensor([[[-1.5903],\n",
      "         [ 0.8837],\n",
      "         [ 2.1141],\n",
      "         [-1.8746],\n",
      "         [ 2.3327],\n",
      "         [ 2.1160],\n",
      "         [ 0.0449],\n",
      "         [ 1.2784],\n",
      "         [-0.8988],\n",
      "         [-0.1769]],\n",
      "\n",
      "        [[-0.0614],\n",
      "         [-0.9508],\n",
      "         [ 0.1634],\n",
      "         [ 1.4063],\n",
      "         [-0.9053],\n",
      "         [-0.3190],\n",
      "         [-0.5812],\n",
      "         [-0.2460],\n",
      "         [-0.7800],\n",
      "         [-0.0228]],\n",
      "\n",
      "        [[ 1.3833],\n",
      "         [-2.2977],\n",
      "         [-0.0199],\n",
      "         [ 1.7388],\n",
      "         [-0.0649],\n",
      "         [ 0.2746],\n",
      "         [ 1.3149],\n",
      "         [-0.8462],\n",
      "         [ 0.6525],\n",
      "         [ 2.5329]],\n",
      "\n",
      "        [[-0.3927],\n",
      "         [ 0.3490],\n",
      "         [-1.0040],\n",
      "         [-1.3723],\n",
      "         [-1.3874],\n",
      "         [ 0.4323],\n",
      "         [-0.7121],\n",
      "         [ 0.5934],\n",
      "         [-0.3115],\n",
      "         [-1.2571]],\n",
      "\n",
      "        [[-0.2320],\n",
      "         [ 1.3630],\n",
      "         [-0.1194],\n",
      "         [-0.5032],\n",
      "         [-0.5004],\n",
      "         [-1.2361],\n",
      "         [-0.6109],\n",
      "         [ 0.0380],\n",
      "         [ 0.5508],\n",
      "         [ 0.4312]],\n",
      "\n",
      "        [[-0.2478],\n",
      "         [-0.5053],\n",
      "         [-0.3946],\n",
      "         [ 0.8356],\n",
      "         [ 1.3766],\n",
      "         [-1.7651],\n",
      "         [-0.4945],\n",
      "         [ 0.5495],\n",
      "         [-0.2553],\n",
      "         [-1.1360]],\n",