10-pytorch-01.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "---\n",
        "title: PyTorch 01 - Tensors\n",
        "jupyter: python3\n",
        "---\n",
        "\n",
        "## ![](assets/images/pytorch/pytorch.svg){height=\"48px\"} PyTorch Introduction\n",
        "\n",
        "[![](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/trgardos/ml-549-fa24/blob/main/10-pytorch-01.ipynb)\n",
        "\n",
        "[PyTorch](https://pytorch.org/) is an open-source machine learning library developed by Facebook's AI Research lab. \n",
        "\n",
        "It is widely used for deep learning tasks, particularly in the field of natural language processing (NLP) and computer vision. \n",
        "\n",
        "Started at Facebook AI Research (FAIR), now _Fundamntal AI Research_, to build a consolidated tool for deep learning tasks.\n",
        "\n",
        "* 2015 there was Theano (Montreal) Caffe (Berkeley) and Lua Torch (FAIR).\n",
        "* 2015 December, Google released TensorFlow\n",
        "* ~2016 Released Caffe2 for targeting mobile and edge devices (Facebook Production)\n",
        "* 2015-2016 FAIR refactored Torch to separate computation backend from frontend and create a new frontend in Pythyon\n",
        "    * the core library is still called `torch`\n",
        "* Sep 2016 PyTorch v0.1.1 released (`torch.nn` and `torch.autograd` modules)\n",
        "* Dec 2018 PyTorch v1.0 released (Replace Caffe2 for FB production)\n",
        "* Mar 15 2023 PyTorch v2.0 released (dynamic shapes, distributed training, Transformers support, etc.)\n",
        "* (Latest Release)Aug 2024 PyTorch 2.4.1 \n",
        "\n",
        "## Key Features of PyTorch\n",
        "\n",
        "1. **Dynamic Computation Graph**: PyTorch's dynamic computation graph allows for more intuitive and flexible model \n",
        "   building. This means that the graph is constructed at runtime, allowing for more dynamic and interactive model \n",
        "   development.\n",
        "\n",
        "2. **GPU and TPU Acceleration**: PyTorch can leverage GPUs and TPUs for accelerated training, making it faster than many \n",
        "   other deep learning frameworks.\n",
        "\n",
        "3. **Autograd**: PyTorch's autograd module provides automatic differentiation, which allows for easy computation of \n",
        "   gradients and updates to model parameters during training.\n",
        "\n",
        "4. **High-Level API**: PyTorch has a high-level API that allows for easy model building and training. It also has a low-level \n",
        "   API that provides more control over the model building process.\n",
        "\n",
        "## PyTorch vs TensorFlow\n",
        "\n",
        "\n",
        "::: {layout-ncol=\"2\"}\n",
        "\n",
        "![](assets/images/pytorch/num_hf_models_2023.png)\n",
        "\n",
        "![](assets/images/pytorch/Fraction-of-Papers-Using-PyTorch-vs.-TensorFlow.png)\n",
        "\n",
        "![](assets/images/pytorch/percentage_repo_2023.png)\n",
        "\n",
        ":::\n",
        "\n",
        "From [https://www.assemblyai.com/blog/pytorch-vs-tensorflow-in-2023/](https://www.assemblyai.com/blog/pytorch-vs-tensorflow-in-2023/).\n",
        "\n",
        "## Key Differences\n",
        "\n",
        ":::: {.columns}\n",
        "::: {.column width=\"50%\"}\n",
        "\n",
        "### PyTorch\n",
        "\n",
        "* Dynamic Computation Graph\n",
        "* Autograd\n",
        "* High-Level API\n",
        "* Arguably more \"pythonic\"\n",
        "\n",
        ":::\n",
        "::: {.column width=\"50%\"}\n",
        "\n",
        "### TensorFlow\n",
        "\n",
        "* Static Computation Graph\n",
        "* Eager Execution\n",
        "* Low-Level API (high-level API is Keras)\n",
        "\n",
        ":::\n",
        "::::\n",
        "\n",
        "## Tutorial\n",
        "\n",
        "We'll borrow heavily from [https://pytorch.org/tutorials/](https://pytorch.org/tutorials/)\n",
        "and other sources that we'll cite.\n",
        "\n",
        "## Tensors\n",
        "\n",
        "Fundamentally, PyTorch [tensor](https://pytorch.org/docs/stable/generated/torch.tensor.html) is a data structure for\n",
        "storing matrices and multi-dimensional arrays.\n",
        "\n",
        "Similar to NumPy's ndarrays.\n",
        "\n",
        "But it does much more.\n",
        "\n",
        "* Manages translation to accelerator data formats and hardware memory\n",
        "* Stores information needed for _automatic gradient_ (autograd) calculation for parameter updates\n",
        "* etc..."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "import torch\n",
        "import numpy as np"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Tensor Creation\n",
        "\n",
        "We have to get data into PyTorch tensors before we can operate on them.\n",
        "\n",
        "There are multiple ways to do that.\n",
        "\n",
        "**Create directly from a Python list such as this one:**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "data = [[1, 2], [2, 3]]\n",
        "print(f\"data: {data}\")\n",
        "print(f\"type(data): {type(data)}\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "**Create the tensor and explore some of its attributes.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "x_data = torch.tensor(data)\n",
        "print(f\"x_data: {x_data}\")\n",
        "print(f\"type(x_data): {type(x_data)}\")\n",
        "print(f\"x_data.dtype: {x_data.dtype}\")\n",
        "print(f\"x_data.shape: {x_data.shape}\")\n",
        "print(f\"x_data.device: {x_data.device}\")\n",
        "print(f\"x_data.requires_grad: {x_data.requires_grad}\")\n",
        "print(f\"x_data.is_leaf: {x_data.is_leaf}\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "As mentioned, some of the interesting attributes are:\n",
        "\n",
        "* `x_data.device`: The device where the tensor is stored.\n",
        "* `x_data.requires_grad`: Whether the tensor requires gradient computation.\n",
        "* `x_data.is_leaf`: Whether the tensor is a leaf node in the computation graph.\n",
        "\n",
        "We'll get more into these later.\n",
        "\n",
        "If we use decimal numbers, then a floating point number type is chosen."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "y_data = torch.tensor([[0.1, 1.2], [2.2, 3.1]])\n",
        "print(f\"y_data: {y_data}\")\n",
        "print(f\"y_data.dtype: {y_data.dtype}\")\n",
        "print(f\"y_data.device: {y_data.device}\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "**From a NumPy array**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "np_array = np.array(data)\n",
        "x_np = torch.from_numpy(np_array)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "**As the same shape of another tensor**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "x_ones = torch.ones_like(x_data) # retains the properties of x_data\n",
        "print(f\"Ones Tensor: \\n {x_ones} \\n\")\n",
        "\n",
        "x_rand = torch.rand_like(x_data, dtype=torch.float) # overrides the datatype of x_data\n",
        "print(f\"Random Tensor: \\n {x_rand} \\n\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "**With random or constant values**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "shape = (2,3)\n",
        "rand_tensor = torch.rand(shape)\n",
        "ones_tensor = torch.ones(shape)\n",
        "zeros_tensor = torch.zeros(shape)\n",
        "\n",
        "print(f\"Random Tensor: \\n {rand_tensor} \\n\")\n",
        "print(f\"Ones Tensor: \\n {ones_tensor} \\n\")\n",
        "print(f\"Zeros Tensor: \\n {zeros_tensor}\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "::: {.callout-tip}\n",
        "There are many more creation functions listed in [Creation Ops](https://pytorch.org/docs/stable/torch.html#creation-ops)\n",
        "for things like sparse tensors, other random number generators, etc.\n",
        ":::\n",
        "\n",
        "### Operations on Tensors\n",
        "\n",
        "Let's look at some interesting operations we can perform on tensors.\n",
        "\n",
        "#### Standard numpy-like indexing and slicing\n",
        "\n",
        "Python itself has some flexible indexing and slicing support. See this\n",
        "[tutorial](https://railsware.com/blog/indexing-and-slicing-for-lists-tuples-strings-sequential-types/)\n",
        "for a nice overview.\n",
        "\n",
        "PyTorch follows the NumPy [indexing and slicing](https://numpy.org/doc/stable/user/basics.indexing.html)\n",
        "conventions."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Define a 4x4 tensor\n",
        "tensor = torch.tensor([[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12], [13, 14, 15, 16]], dtype=torch.float32)\n",
        "#tensor = torch.ones(4, 4)\n",
        "print(f\"tensor.dtype: {tensor.dtype}\")\n",
        "\n",
        "# Matrices are stored in row-major order, and like NumPy, it is a list of lists.\n",
        "print(f\"First row: {tensor[0]}\")\n",
        "print(f\"Second row: {tensor[1]}\")\n",
        "\n",
        "# Like python we use `:` to index and slice\n",
        "print(f\"First column: {tensor[:, 0]}\")\n",
        "print(f\"Second column: {tensor[:, 1]}\")\n",
        "\n",
        "# Slicing with `...` is a shortcut for \"all remaining dimensions\"\n",
        "print(f\"Last column: {tensor[..., -1]}\")\n",
        "print(f\"Last column: {tensor[:, -1]}\")\n",
        "\n",
        "# There is also broadcasting support\n",
        "tensor[:,1] = 0\n",
        "print(tensor)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Joining tensors\n",
        "\n",
        "You can use `torch.cat` to concatenate a sequence of\n",
        "tensors along a given dimension. See also\n",
        "[torch.stack](https://pytorch.org/docs/stable/generated/torch.stack.html),\n",
        "another tensor joining operator that is subtly different from\n",
        "`torch.cat`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "t1 = torch.cat([tensor, tensor, tensor], dim=1)\n",
        "print(t1)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "t2 = torch.cat([tensor, tensor, tensor], dim=0)\n",
        "print(t2)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Matrix Multiplication\n",
        "\n",
        "Matrix multiplication is a fundamental operation in deep learning and PyTorch provides a number of ways to perform it.\n",
        "\n",
        "#### Equivalent matrix multiplication operations\n",
        "\n",
        "We'll use the following helper function to format the tensors as LaTeX matrices."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "#| code-fold: true\n",
        "import torch\n",
        "from IPython.display import display, Math\n",
        "\n",
        "def tensor_to_latex(tensor):\n",
        "    \"\"\"Convert a PyTorch tensor to a LaTeX bmatrix string.\"\"\"\n",
        "    if tensor.dim() == 1:\n",
        "        # Convert 1D tensor to a 2D column vector for display\n",
        "        tensor = tensor.unsqueeze(1)\n",
        "    \n",
        "    latex_str = \"\\\\begin{bmatrix}\\n\"\n",
        "    for row in tensor:\n",
        "        latex_str += \" & \".join([f\"{val.item():.2f}\" for val in row]) + \" \\\\\\\\\\n\"\n",
        "    latex_str += \"\\\\end{bmatrix}\"\n",
        "    return latex_str"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Define matrix A and vector x\n",
        "A = torch.tensor([[1.0, 2.0], [3.0, 4.0]])\n",
        "x = torch.tensor([5.0, 6.0])\n",
        "\n",
        "# Compute the product A * x\n",
        "Ax = torch.matmul(A, x)\n",
        "\n",
        "# Convert tensors to LaTeX\n",
        "latex_A = tensor_to_latex(A)\n",
        "latex_x = tensor_to_latex(x)\n",
        "latex_Ax = tensor_to_latex(Ax)\n",
        "\n",
        "# Display the equation A x = Ax on one line\n",
        "display(Math(f\"A \\\\times x = {latex_A} \\\\times {latex_x} = {latex_Ax}\"))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We'll use the tensor from before."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "#| code-fold: true\n",
        "latex_tensor = tensor_to_latex(tensor)\n",
        "display(Math(f\"\\\\mathrm{{tensor}} = {latex_tensor}\"))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "There are three different ways shown below that produce the same result.\n",
        "\n",
        "##### **Method 1:** Use the `@` matrix multiplication operator"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# ``tensor.T`` returns the transpose of a tensor\n",
        "y1 = tensor @ tensor.T\n",
        "# print(f\"y1: {y1}\")\n",
        "\n",
        "latex_tensorT = tensor_to_latex(tensor.T)\n",
        "latex_y1 = tensor_to_latex(y1)\n",
        "display(Math(f\"y1 = \\\\mathrm{{tensor}} \\\\times \\\\mathrm{{tensor.T}} = {latex_tensor} \\\\times {latex_tensorT} = {latex_y1}\"))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "<br>\n",
        "\n",
        "##### **Method 2:** call `.matmul` method on the tensor"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "y2 = tensor.matmul(tensor.T)\n",
        "# print(f\"y2: {y2}\")\n",
        "\n",
        "latex_y2 = tensor_to_latex(y2)\n",
        "display(Math(f\"y2 = \\\\mathrm{{tensor}} \\\\times \\\\mathrm{{tensor.T}} = {latex_tensor} \\\\times {latex_tensorT} = {latex_y2}\"))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "<br>\n",
        "\n",
        "##### **Method 3:** Call the `torch.matmul` function with the two input operands and the output tensor."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# This creates a new tensor with the same shape and type as y1, filled with random numbers from a uniform distribution over [0, 1)\n",
        "y3 = torch.rand_like(y1)\n",
        "#print(f\"y3: {y3}\")\n",
        "\n",
        "torch.matmul(tensor, tensor.T, out=y3)\n",
        "# print(f\"y3: {y3}\")\n",
        "\n",
        "latex_y3 = tensor_to_latex(y3)\n",
        "display(Math(f\"y3 = \\\\mathrm{{torch.matmul(tensor, tensor.T)}} = {latex_tensor} \\\\times {latex_tensorT} = {latex_y3}\"))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "<br>\n",
        "\n",
        "There is additional behavior depending on the dimensionality of the tensors.\n",
        "\n",
        "#### Vector Dot Product\n",
        "\n",
        "If both tensors are 1-D, then the **vector dot product** is performed."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "torch.manual_seed(0)\n",
        "\n",
        "# vector x vector\n",
        "tensor1 = torch.randint(low=0, high=10, size=(3,))\n",
        "tensor2 = torch.randint(low=0, high=10, size=(3,))\n",
        "# print(f\"tensor1: {tensor1} and tensor2: {tensor2}\")\n",
        "\n",
        "latex_tensor1 = tensor_to_latex(tensor1)\n",
        "latex_tensor2 = tensor_to_latex(tensor2)\n",
        "\n",
        "display(Math(f\"\\\\mathrm{{tensor1}} = {latex_tensor1}, \\\\mathrm{{tensor2}} = {latex_tensor2}\"))\n",
        "\n",
        "\n",
        "tensor3 = torch.matmul(tensor1, tensor2)\n",
        "print(f\"tensor3 = tensor1 @ tensor2: {tensor3}\")\n",
        "print(f\"tensor3.size(): {tensor3.size()}\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "So with 1-D tensors, `torch.matmul` and `torch.dot` are equivalent."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "tensor3 = torch.dot(tensor1, tensor2)\n",
        "print(f\"tensor3 = torch.dot(tensor1, tensor2): {tensor3}\")\n",
        "print(f\"tensor3.size(): {tensor3.size()}\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Matrix-Vector Multiplication\n",
        "\n",
        "Now let's look at the case where one operand is 1D and the other is 2D.\n",
        "\n",
        "If the first argument is 2-dimensional and the second argument is 1-dimensional,\n",
        "the matrix-vector product is returned.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "tensor1 = torch.tensor([[1, 2], [3, 4]])\n",
        "print(f\"Operand 1 --tensor1.size(): {tensor1.size()}\")\n",
        "# print(f\"Operand 1 -- tensor1: {tensor1}\")\n",
        "latex_tensor1 = tensor_to_latex(tensor1)\n",
        "\n",
        "tensor2 = torch.tensor([1, 2])\n",
        "print(f\"Operand 2 --tensor2.size(): {tensor2.size()}\")\n",
        "#print(f\"Operand 2 --tensor2: {tensor2}\")\n",
        "\n",
        "latex_tensor2 = tensor_to_latex(tensor2)\n",
        "\n",
        "tensor3 = torch.matmul(tensor1, tensor2)\n",
        "#print(f\"Output -- tensor3: {tensor3}\")\n",
        "print(f\"Output -- tensor3.size(): {tensor3.size()}\")\n",
        "\n",
        "latex_tensor3 = tensor_to_latex(tensor3)\n",
        "display(Math(f\"\\\\mathrm{{tensor3}} = \\\\mathrm{{tensor1}} \\\\times \\\\mathrm{{tensor2}} = {latex_tensor1} \\\\times {latex_tensor2} = {latex_tensor3}\"))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "So this would be the similar to"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Change tensor2 to be shape (2,1)\n",
        "tensor2 = torch.tensor([[1], [2]])\n",
        "print(f\"Operand 2 -- tensor2.size(): {tensor2.size()}\")\n",
        "print(f\"Operand 2 -- tensor2: {tensor2}\")\n",
        "\n",
        "tensor3 = torch.matmul(tensor1, tensor2)\n",
        "print(f\"Output -- tensor3: {tensor3}\")\n",
        "print(f\"Output -- tensor3.size(): {tensor3.size()}\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "But note the difference in the size of the output.\n",
        "\n",
        "If the first argument is 1-dimensional and the second argument is 2-dimensional,\n",
        "a 1 is prepended to its dimension for the purpose of the matrix multiply.\n",
        "After the matrix multiply, the prepended dimension is removed."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "tensor2 = torch.tensor([1, 2])\n",
        "print(f\"Operand 1 -- tensor2.size(): {tensor2.size()}\")\n",
        "print(f\"Operand 1 -- tensor2: {tensor2}\")\n",
        "\n",
        "tensor3 = torch.matmul(tensor2, tensor1)\n",
        "print(f\"Output -- tensor3: {tensor3}\")\n",
        "print(f\"Output -- tensor3.size(): {tensor3.size()}\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "which is similar to"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "tensor2 = torch.tensor([[1, 2]])\n",
        "print(f\"Operand 1 -- tensor2.size(): {tensor2.size()}\")\n",
        "print(f\"Operand 1 -- tensor2: {tensor2}\")\n",
        "\n",
        "tensor3 = torch.matmul(tensor2, tensor1)\n",
        "print(f\"Output -- tensor3: {tensor3}\")\n",
        "print(f\"Output -- tensor3.size(): {tensor3.size()}\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Batched Matrix Multiplication\n",
        "\n",
        "It is very common to have a \"batch\" of inputs to a neural network, where maybe\n",
        "the input to the network is a 3x4 matrix, but PyTorch optimizes the computation\n",
        "to process an entire batch of matrices at once.\n",
        "\n",
        "![Batches of size 5](assets/images/pytorch/batches.png){fig-align=\"center\"}\n",
        "\n",
        "PyTorch follows the following broadcast conventions.\n",
        "\n",
        "##### Batched Matrix x Broadcasted Vector\n",
        "In the next example, we have batches of 5 matrices, each of size 3x4, and we\n",
        "multiply each by a vector of size 4.\n",
        "\n",
        "So $[5\\times3\\times4] @ [4\\times1] = [5\\times3\\times1]$."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# batched matrix x broadcasted vector\n",
        "tensor1 = torch.randint(low=0, high=10, size=(5, 3, 4))\n",
        "print(f\"tensor1.size(): {tensor1.size()}\")\n",
        "print(f\"tensor1: {tensor1}\")\n",
        "\n",
        "tensor2 = torch.randint(low=0, high=10, size=(4,))\n",
        "print(f\"tensor2.size(): {tensor2.size()}\")\n",
        "print(f\"tensor2: {tensor2}\")\n",
        "\n",
        "tensor3 = torch.matmul(tensor1, tensor2)\n",
        "print(f\"tensor3 = torch.matmul(tensor1, tensor2): {tensor3}\")\n",
        "print(f\"tensor3.size(): {tensor3.size()}\")\n",
        "\n",
        "latex_tensor3 = tensor_to_latex(tensor3)\n",
        "display(Math(f\"\\\\mathrm{{tensor3}} = {latex_tensor3}\"))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "##### Batched Matrix x Batched Matrix\n",
        "What about if we multiply two batched matrices?\n",
        "\n",
        "For example $[5\\times3\\times4] @ [5\\times4\\times5] = [5\\times3\\times5]$."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# batched matrix x batched matrix\n",
        "tensor1 = torch.randint(low=0, high=10, size=(5, 3, 4))\n",
        "print(f\"tensor1.size(): {tensor1.size()}\")\n",
        "print(f\"tensor1: {tensor1}\")\n",
        "\n",
        "tensor2 = torch.randint(low=0, high=10, size=(5, 4, 5))\n",
        "print(f\"tensor2.size(): {tensor2.size()}\")\n",
        "print(f\"tensor2: {tensor2}\")\n",
        "\n",
        "tensor3 = torch.matmul(tensor1, tensor2)\n",
        "print(f\"tensor3 = torch.matmul(tensor1, tensor2): {tensor3}\")\n",
        "print(f\"tensor3.size(): {tensor3.size()}\")\n",
        "# torch.Size([10, 3, 5])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "So we multiply the first matrix in the first batch by the first matrix in the second batch.\n",
        "\n",
        "\n",
        "##### Batched Matrix x Broadcasted Matrix\n",
        "\n",
        "And here we have $[5\\times3\\times4] @ [4\\times5] = [5\\times3\\times5]$."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# batched matrix x broadcasted matrix\n",
        "tensor1 = torch.randint(low=0, high=10, size=(5, 3, 4))\n",
        "tensor2 = torch.randint(low=0, high=10, size=(4, 5))\n",
        "tensor3 = torch.matmul(tensor1, tensor2)\n",
        "print(f\"tensor3: {tensor3}\")\n",
        "print(f\"tensor3.size(): {tensor3.size()}\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Element-wise Multiplication"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# This computes the element-wise product. z1, z2, z3 will have the same value\n",
        "z1 = tensor * tensor\n",
        "z2 = tensor.mul(tensor)\n",
        "\n",
        "z3 = torch.rand_like(tensor)\n",
        "torch.mul(tensor, tensor, out=z3)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "**Single-element tensors** If you have a one-element tensor, for example\n",
        "by aggregating all values of a tensor into one value, you can convert it\n",
        "to a Python numerical value using `item()`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "agg = tensor.sum()\n",
        "agg_item = agg.item()\n",
        "print(agg_item, type(agg_item))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "**In-place operations** Operations that store the result into the\n",
        "operand are called in-place. They are denoted by a `_` suffix. For\n",
        "example: `x.copy_(y)`, `x.t_()`, will change `x`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "print(f\"{tensor} \\n\")\n",
        "tensor.add_(5)\n",
        "print(tensor)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "::: {.callout-note}\n",
        "In-place operations save some memory, but can be problematic when computing derivatives because of an immediate loss of history. Hence, their use is discouraged.\n",
        ":::\n",
        "\n",
        "## Fully Connected Layer as Matrix Multiplication\n",
        "\n",
        "As a reminder, here's a fully connected network (also known as  dense network)\n",
        "with 3 inputs, 4 hidden nodes and 2 outputs.\n",
        "\n",
        "![Deep Fully Connected Network](assets/images/pytorch/shallow_net.png)\n",
        "\n",
        "For example, we can express the hidden layer of this network as a matrix multiplication.\n",
        "\n",
        "$$\n",
        "\\begin{bmatrix}\n",
        "w_{00} & w_{01} & w_{02} \\\\\n",
        "w_{10} & w_{11} & w_{12} \\\\\n",
        "w_{20} & w_{21} & w_{22} \\\\\n",
        "w_{30} & w_{31} & w_{32} \\\\\n",
        "\\end{bmatrix}\n",
        "\\begin{bmatrix}\n",
        "x_{0} \\\\\n",
        "x_{1} \\\\\n",
        "x_{2} \\\\\n",
        "\\end{bmatrix} +\n",
        "\\begin{bmatrix}\n",
        "b_{0} \\\\\n",
        "b_{1} \\\\\n",
        "b_{2} \\\\\n",
        "b_{3} \\\\\n",
        "\\end{bmatrix}\n",
        "=\n",
        "\\begin{bmatrix}\n",
        "h_{0} \\\\\n",
        "h_{1} \\\\\n",
        "h_{2} \\\\\n",
        "h_{3} \\\\\n",
        "\\end{bmatrix}\n",
        "$$\n",
        "\n",
        "Which we can compute directly on tensors."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "import torch\n",
        "\n",
        "torch.manual_seed(42)\n",
        "\n",
        "# Define the input tensor (batch size of 1 for simplicity)\n",
        "input_tensor = torch.tensor([[1.0, 2.0, 3.0]])\n",
        "\n",
        "# Initialize weights and biases for the hidden layer\n",
        "weights_hidden = torch.randn(3, 4)  # 3 inputs to 4 hidden nodes\n",
        "bias_hidden = torch.randn(4)        # 4 hidden nodes\n",
        "\n",
        "# Initialize weights and biases for the output layer\n",
        "weights_output = torch.randn(4, 2)  # 4 hidden nodes to 2 outputs\n",
        "bias_output = torch.randn(2)        # 2 outputs\n",
        "\n",
        "# Perform matrix multiplication and add biases for the hidden layer\n",
        "hidden_layer = torch.matmul(input_tensor, weights_hidden) + bias_hidden\n",
        "\n",
        "# Apply ReLU activation function\n",
        "hidden_layer_activated = torch.relu(hidden_layer)\n",
        "\n",
        "# Perform matrix multiplication and add biases for the output layer\n",
        "output_layer = torch.matmul(hidden_layer_activated, weights_output) + bias_output\n",
        "\n",
        "print(\"Output:\", output_layer)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Let's look at the matrix equations.\n",
        "\n",
        "Here's the calculation of the hidden layer."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "#| code-fold: true\n",
        "latex_input_tensor = tensor_to_latex(input_tensor.T)\n",
        "latex_weights_hidden = tensor_to_latex(weights_hidden.T)\n",
        "latex_bias_hidden = tensor_to_latex(bias_hidden)\n",
        "latex_weights_output = tensor_to_latex(weights_output.T)\n",
        "latex_bias_output = tensor_to_latex(bias_output)\n",
        "latex_hidden_layer = tensor_to_latex(hidden_layer.T)\n",
        "latex_hidden_layer_activated = tensor_to_latex(hidden_layer_activated.T)\n",
        "latex_output_layer = tensor_to_latex(output_layer.T)\n",
        "\n",
        "# display(Math(f\"\\\\text{{input tensor}} = {latex_input_tensor}\"))\n",
        "# display(Math(f\"\\\\text{{weights hidden}} = {latex_weights_hidden}\"))\n",
        "# display(Math(f\"\\\\text{{bias hidden}} = {latex_bias_hidden}\"))\n",
        "# display(Math(f\"\\\\text{{weights output}} = {latex_weights_output}\"))\n",
        "# display(Math(f\"\\\\text{{bias output}} = {latex_bias_output}\"))\n",
        "# display(Math(f\"\\\\text{{hidden layer}} = {latex_hidden_layer}\"))\n",
        "# display(Math(f\"\\\\text{{hidden layer activated}} = {latex_hidden_layer_activated}\"))\n",
        "# display(Math(f\"\\\\text{{output layer}} = {latex_output_layer}\"))\n",
        "\n",
        "display(Math(f\"{latex_weights_hidden} \\\\times {latex_input_tensor} + {latex_bias_hidden} = {latex_hidden_layer}\"))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Then we apply the ReLU activation function."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "#| code-fold: true\n",
        "display(Math(f\"\\\\mathrm{{ReLU}}({latex_hidden_layer}) = {latex_hidden_layer_activated}\"))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "And finally we calculate the output linear layer."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "#| code-fold: true\n",
        "display(Math(f\"{latex_weights_output} \\\\times {latex_hidden_layer_activated} + {latex_bias_output} = {latex_output_layer}\"))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Fully Connected Layer with Batched Input\n",
        "\n",
        "With no change to the previous code, we can now process a batch of inputs."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "import torch\n",
        "\n",
        "torch.manual_seed(42)\n",
        "\n",
        "# Define the input tensor (batch size of 5)\n",
        "input_tensor = torch.tensor([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0], [7.0, 8.0, 9.0], [10.0, 11.0, 12.0], [13.0, 14.0, 15.0]])\n",
        "\n",
        "#vvvv The code below is the same as above ^^^^^\n",
        "\n",
        "# Initialize weights and biases for the hidden layer\n",
        "weights_hidden = torch.randn(3, 4)  # 3 inputs to 4 hidden nodes\n",
        "bias_hidden = torch.randn(4)        # 4 hidden nodes\n",
        "\n",
        "# Initialize weights and biases for the output layer\n",
        "weights_output = torch.randn(4, 2)  # 4 hidden nodes to 2 outputs\n",
        "bias_output = torch.randn(2)        # 2 outputs\n",
        "\n",
        "# Perform matrix multiplication and add biases for the hidden layer\n",
        "hidden_layer = torch.matmul(input_tensor, weights_hidden) + bias_hidden\n",
        "\n",
        "# Apply ReLU activation function\n",
        "hidden_layer_activated = torch.relu(hidden_layer)\n",
        "\n",
        "# Perform matrix multiplication and add biases for the output layer\n",
        "output_layer = torch.matmul(hidden_layer_activated, weights_output) + bias_output\n",
        "\n",
        "print(\"Output:\", output_layer)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "------------------------------------------------------------------------\n",
        "\n",
        "### Further Reading\n",
        "\n",
        "-   [torch.utils.data API](https://pytorch.org/docs/stable/data.html)\n",
        "\n",
        "\n",
        "## References\n",
        "\n",
        "- [PyTorch](https://pytorch.org/)\n",
        "- [PyTorch Tutorials](https://pytorch.org/tutorials/)\n",
        "- [PyTorch Documentation](https://pytorch.org/docs/)\n",
        "- [PyTorch GitHub](https://github.com/pytorch/pytorch)\n",
        "- [PyTorch Brand Guidelines](https://pytorch.org/assets/brand-guidelines/PyTorch-Brand-Guidelines.pdf)"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "name": "python3",
      "language": "python",
      "display_name": "Python 3 (ipykernel)",
      "path": "/Users/tomg/Source/courses/ds549/ml-549-fa24/.venv/share/jupyter/kernels/python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 4
}