05_pytorch_sequence_model.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Feed Forward Network and RNN for NLP Tasks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feedforward networks for NLP: Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The input element $x_i$ could be scalar features like those in Fig. 5.2, e.g., \n",
    "\n",
    "- $x_1 = \\text{count(words 2 doc)}$, \n",
    "- $x_2 = \\text{count(positive lexicon words 2 doc)}$, \n",
    "- $x_3 = \\text{1 if “no” 2 doc}$, and \n",
    "- ...  \n",
    "\n",
    "And the output layer $\\hat{y}$ could have 3 nodes (positive, negative, neutral), in which case $\\hat{y}_1$ would be the estimated probability of positive sentiment, $\\hat{y}_2$ the probability of negative and $\\hat{y}_3$ the probability of neutral"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The resulting equations\n",
    "would be just what we saw above for a 2-layer network (as always, we’ll continue\n",
    "to use the $\\sigma$ to stand for any non-linearity, whether sigmoid, ReLU or other)\n",
    "\n",
    "- $x = [x_1, x_2, ... ,x_N]$ (each xi is a hand-designed feature)\n",
    "- $h = \\sigma(Wx+b)$\n",
    "- $z = Uh$\n",
    "- $\\hat{y} = \\text{softmax}(z)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instead of using hand-built human-engineered features as the input to our classifier, we draw on deep learning’s ability to learn features from the data by representing words as embeddings, like the *word2vec* or *GloVe* embeddings "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, for a text with $n$ input words/tokens $w_1, ..., w_n$, we can turn the $n$ embeddings $e(w_1), ..., e(w_n)$ (each of dimensionality $d$) then apply some poolling\n",
    "\n",
    "- $x = \\text{mean}(e(w_1), e(w_2), ... , e(w_n)) $ \n",
    "- $h = \\sigma(Wx+b)$\n",
    "- $z = Uh$\n",
    "- $\\hat{y} = \\text{softmax}(z)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The efficience in the computation we need to convert to vector and matrix\n",
    "\n",
    "- $H = \\sigma(XW^T +b)$\n",
    "- $Z = HU^T$\n",
    "- $\\hat{Y} = \\text{softmax}(Z)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The idea of using word2vec or GloVe embeddings as our input representation— and more generally the idea of relying on another algorithm to have already learned an embedding representation for our input words—is called pretrainin"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feedforward Neural Language Modeling "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Prediction upcoming word from prior word context.\n",
    "\n",
    "Neural Language Modeling \n",
    "\n",
    "Tasks:\n",
    "- Machine Tranlation\n",
    "- Text Summarization\n",
    "- Speech Recognicion\n",
    "- Grammar Correnction\n",
    "- Chat "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compared to *n-gram* models, neural language models can handle much longer histories, can generalize better over contexts of similar words, and are more accurate at word-prediction\n",
    "\n",
    "On the other hand, neural net language models are much more complex, are slower and need more energy to train, and are less interpretable than n-gram models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The feedforward neural LM approximates the probability of a word given the entire prior context $P(w_t|w_{1:{t−1}})$ by approximating based on the $N − 1$ previous words\n",
    "\n",
    "$$ P(w_t|w_1, ..., w_{t−1}) ≈ P(w_t|w_{t−N+1}, ..., w_{t−1})$$\n",
    "\n",
    "In the following examples we’ll use a 4-gram example, so we’ll show a neural net to estimate the probability $P(w_t = i|w_{t−3};w_{t−2};w_{t−1})$.\n",
    "\n",
    "Neural language models represent words in this prior context by their embeddings, rather than just by their word identity as used in n-gram language models. Using embeddings allows neural language models to generalize better to unseen data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Forward inference (decoding) in the neural language model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Forward inference is the task, given an input, of running a forward pass on the\n",
    "network to produce a probability distribution over possible outputs, in this case next\n",
    "word"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Represent $N$ ($N=3$ for our example) previous words in one-hot representation of length $|V|$. \n",
    "2. Let $x_{t}$ of the dimention $|V|$ x $1$  is one-hot representation of $w_{t}$\n",
    "3. Given the $E$ matrix of embedding with dimention $\\text{d}$ x $\\text{|V|}$ (where $d$ is the length of vector that represent the word).\n",
    "4. Multiply $Ex_{t}=e_t$ of the dimention $\\text{d}$ x $1$.\n",
    "5. For $N=3$ we stacked $e = [e_{t-1}; e_{t-2}; e_{t-3}]$ of the dimention $\\text{3d}$ x $1$.\n",
    "6. Then multiply the weights of the dimention $d_h$ x $\\text{3d}$ with stacked embeding, and then apply some non-linear function $h=\\sigma(We)$\n",
    "7. Then $h$ is multiplied by $U$ of the dimention $|V|$ x $d_h$\n",
    "8. Apply softmax: After the softmax, each node i in the output layer estimates\n",
    "the probability $P(w_t = i|w_{t−1};w_{t−2};w_{t−3})$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Recurrent Neural Network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Certain data types such as time-series, text, and biological data contain sequential dependencies. In such cases, a recurrent neural network (RNN) is a type of neural network that can be used to solve such problems. RNN is specifically designed for processing sequential data.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> A Bit of Vector Semantic and Embedding \n",
    "> \n",
    "> **Embedding**\n",
    "> \n",
    "> It is a way to represent of the meaning of a word in a *vector*. All relate to this word is represented in this vector (semantic, syntax, etc).\n",
    "> \n",
    "> Exists a static embedding and dynamic contextualized embedding (like BERT).\n",
    "> \n",
    "> This avoids the need to feature engineering. \n",
    "> \n",
    "> This conducts to *self-supervised* ways to learn representations of the input.\n",
    "> \n",
    "> **Vector Semantic**\n",
    "> \n",
    "> It is vector that represents the meaning of a word. Recall, \"semantics\" is the process of understanding the meaning of a word and this an standar in NLP.\n",
    "> \n",
    "> The idea of vector semantics is to represent a word as a point in a multidimensional space \n",
    "> <!-- that is derived from the distributions of embeddings word neighbors. -->\n",
    "> \n",
    "> Vectors for representing words are called *embeddings* (although the term is sometimes more strictly applied only to dense vectors like *word2vec*)\n",
    "> \n",
    "> **Cosine for measuring similarity**\n",
    "> \n",
    "> To measure similarity between two target words v and w, we need a metric that\n",
    "> takes two vectors gives a measure of their similarity.\n",
    "> \n",
    "> The *dot product* (inner product) acts as a similarity metric because it will tend to be high just when the two vectors have large values in > the same dimensions. Alternatively, vectors that have zeros in different dimensions—orthogonal vectors—will have a dot product of 0, > representing their strong dissimilarity.\n",
    "> \n",
    "> If we normalize this dot product by its length of the vector the result is :\n",
    "> \n",
    "> $$\\text{consine} = \\frac{\\sum_i^N v_iw_i}{\\sqrt{\\sum_i^N v_i^2}  \\sqrt{\\sum_i^N w_i^2} } $$\n",
    "> \n",
    "> **Word2vec**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sequence Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Motivating Example \n",
    "\n",
    "Task: *Entity Recognition*. It is used to find companies's name, times location, currency names, \n",
    "\n",
    "$\\text{input}$: Harry Potter and Hermione Granger invented a new spell\n",
    "\n",
    "$$\\textbf{x}: [x^{<1>}, x^{<2>} ..., x^{<9>}]$$\n",
    "\n",
    "$$\\textbf{y}: [1, 1, 0, 1, 1, 0, 0, 0, 0]$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Representating Words\n",
    "\n",
    "$\\text{input}$: Harry Potter and Hermione Granger invented a new spell.\n",
    "\n",
    "We can represent this input sequence in a vector based on this *vocabulary* \n",
    "\n",
    "$$\\text{vocabulary} = \\{\\text{a}, \\text{aaron}, ...,\\text{Harry,} ..., \\text{Potter}, ...,\\text{zulu}\\}$$\n",
    "\n",
    "We split the sentence in tokens, each token is a word.\n",
    "\n",
    "$$\\text{Input} \\: \\text{tokens} = [\\text{Harry}, \\text{and}, \\text{Potter}, \\text{Hermione}, \\text{Granger}, \\text{invented}, \\text{a}, \\text{new}, \\text{spell}]$$\n",
    "\n",
    "$$\\text{Input} \\: \\text{tokens} = [x^{<1>}, x^{<2>}, x^{<3>}, x^{<4>}, x^{<5>}, x^{<6>}, x^{<7>}, x^{<8>}]$$\n",
    "\n",
    "So $ x^{<1>}=\\text{Harry}$ it converted to vector\n",
    "$$x^{<1>} = \\begin{bmatrix} \n",
    "0\\\\ \n",
    "0\\\\\n",
    "0\\\\\n",
    "...\\\\\n",
    "1\\\\\n",
    "...\\\\\n",
    "0\\\\\n",
    "...\\\\\n",
    "0\n",
    "\\end{bmatrix}$$\n",
    "\n",
    "This is called one-hot representation. What if we encounter a word that is not in the *vocabulary*? we usually create a token $\\text{<UNK>}$.\n",
    "\n",
    "Other tokes that we use are $\\text{<PAD>}$,$\\text{<EOS>}$, $\\text{<SOS>}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Problemns with Feed Forward Neural Network\n",
    "> * The *inputs* and the *outputs* can have several lenghts. \"This is my iPhone\" (inputs tokenized by word) (T = 4), \"This laptop don't have enough storage\" (inputs tokenized by word) (T = 7).\n",
    "> * The inputs tokenized can be related to earlier inputs tokenized."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Recurrent Neural Networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are focusing on a class of recurrent networks referred to as *Elman Networks* (Elman, 1990) or *simple recurrent networks*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These networks are useful in their own right and serve as the basis for more\n",
    "complex approaches like the Long Short-Term Memory (LSTM). In this chapter when we use the term RNN we’ll be referring to\n",
    "these simpler more constrained networks (although you will often see the term RNN to mean any net with recurrent properties including LSTMs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "A recurrent neural network (RNN) is any network that contains a cycle within its\n",
    "network connections, meaning that the value of some unit is directly, or indirectly,\n",
    "dependent on its own earlier outputs as an input.\n",
    "\n",
    "Critically, this approach does not impose a fixed-length limit on this prior context; the context embodied in the previous hidden layer can include information extending back to the beginning of the sequence."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inference in RNNs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Mapping a sequence of inputs to a sequence of outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The inference used in this example is for *language modeling task*\n",
    "\n",
    "Language modeling, or LM, is the use of various statistical and probabilistic techniques to determine the probability of a given sequence of words occurring in a sentence. Language models analyze bodies of text data to provide a basis for their word predictions.\n",
    "\n",
    "It one of the most basic and important tasks in natural language processing and RNN do very well. \n",
    "\n",
    "RNN language models (Mikolov et al., 2010) process the input sequence one\n",
    "word at a time, attempting to predict the next word from the current word and the\n",
    "previous hidden state. RNNs thus don’t have the limited context problem that n-gram\n",
    "models have, or the fixed context that feedforward language models have, since the\n",
    "hidden state can in principle represent information about all of the preceding words\n",
    "all the way back to the beginning of the sequence."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$P(x^{<t+1>} | x^{<1>},...,x^{<t>})$$\n",
    "\n",
    "In order to diference the input and the output, the next word ($x^{<t+1>}$) is represented by $y^{<t>}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "0. $e^{<t>} = E\\textbf{x}^{<t>}$\n",
    "1. $a^{<t>} = \\Phi(W_{aa}a^{<t-1>} + W_{ae}e^{<t>} + b_a)$\n",
    "2. $y^{<t>} = \\Theta(W_{ya}a^{<t>} + b_y)$\n",
    "\n",
    "> Where $W_{ya}$ are the weights used to compute $y$ multiplied the vector $a$ \n",
    ">\n",
    "> $a^{<0>}$ is a vector of zeros"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Let $\\textbf{x}^{<{t}>}$, of the dimention $|V|$ x $1$, the one-hot representation of $x^{<{t}>}$ (word).\n",
    "2. Given the $E$ matrix of embedding with dimention ${d}_e$ x $\\text{|V|}$ (where $d$ is the length of vector that represent the word).\n",
    "3. Multiply $E$ with $\\textbf{x}^{<t>}$ resulting $e^{<t>}$ of the dimention $d_e$ x $1$.\n",
    "4. $e^{<t>}$ is multiplied by $W_{ae}$ (of the dimention $d_h$ x $d_e$) and $a^{<t-1>}$ (of the dimention $d_a$ x $1$ with initial values set to zeros) is multiplied by $W_{aa}$ (of the dimention $d_h$ x $d_a$). This last component summatize all preceding relationship.\n",
    "4. The previous results are sum and added the bias $b_a$ (of dimention $d_h$ x $1$ ), then is passed to no-linear function (ReLu for example) resulting $a^{<t>}$ of dimention $d_h$ x 1.\n",
    "5. $a^{<t>}$ is multiply by $W_{ya}$ (of dimention $|V|$ x $d_h$) and added a bias $b_y$ (of teh dimention $|V|$ x $1$).\n",
    "6. At the previously result (called *logits*) apply softmax ($\\Theta$) getting the probability distribution over the possible output classes\n",
    "7. The class that maximize the probability is the class chosen."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- <div><span style=\"font-size:25px;font-weight:300\">RNN Architecture (One Hidden Layer)</span> </div> -->\n",
    "\n",
    "Graphical representation\n",
    "\n",
    "<div style = \"\">\n",
    "    <div style = \"display:flex;align-items:center; justify-content:center; flex-direction:column\" >\n",
    "    <img src = \"./assets/architecture-rnn.png\" style=\"height:150px\">\n",
    "    <span>By: Andrew Ng</span>\n",
    "    </div>\n",
    "    \n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Note (*Weight tying*)\n",
    "> \n",
    "> This set $d_h=d_e$ in order to use $E$ and $W_{ya}^T$\n",
    "> \n",
    "> *This approach significantly reduces the number of parameters required for the model*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> *Note*\n",
    "> \n",
    "> In actually this is an specific structure called *Many-to-Many* in which the inputs and the outputs has the same length ( $T_x = T_y$) and, also Its is only in one direction (left to right) with one hidden layer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> In *teacher forcing*\n",
    "> \n",
    "> Instead of using the model's predicted output at each time step as input for the next step, the correct target output (the ground truth) is fed into the model. This means the model gets to see the actual sequence it is supposed to generate during training, which speeds up learning and makes the model converge faster."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Examples of RRN architectures\n",
    "\n",
    "- *Many-to-Many*: Both inputs and outputs are sequences. Can be\n",
    "direct or delayed.\n",
    "  - Ex.: Video-captioning, i.e., describing a sequence of images via text (direct).\n",
    "  - Translating one language into another (delayed)\n",
    "\n",
    "- *Many-to-One* (Sentimential Classification ) The input data is a sequence, but the output is a fixedsize vector, not a sequence.\n",
    "  - Ex.: sentiment analysis, the input is some text, and the output is a class label.\n",
    "- *One-to-Many*:  Input data is in a standard format (not a sequence), theoutput is a sequence.\n",
    "  - Ex.: Image captioning, where the input is an image, the output is a textdescription of that image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training RNN\n",
    "\n",
    "Back Propagation "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the Back Propagation we need to compute the loss function.\n",
    "\n",
    "Let be $\\textbf{y}^{<t>}$ (of the dimention $|V|$ x $1$) is the one-hot representation of $y^{<t>}$ (word) and the *logits*  (results before apply softmax of the dimention $|V|$ x $1$).\n",
    "\n",
    "1. $L^{<t>} = - {\\textbf{y}_i^{<t>}}^{T} \\log(\\text{logits}^{<t>})$ = -$\\log(\\text{logits}^{<t>})[<t>]$ (from the vector logits extract the index $[<t>]$)\n",
    "2. $L = \\sum_{t=1}^{T_y} L^{<t>}$\n",
    "\n",
    "$$\\frac{\\partial L^{<t>}}{\\partial W_{aa}} = \\frac{\\partial L^{<t>}}{\\partial y^{<t>}} . \\Theta\\text{'}. W_{ya}. \\Phi \\text{'}  a^{<t-1>} + \\frac{\\partial L^{<t>}}{\\partial y^{<t>}} . \\Theta\\text{'}. W_{ya}. \\Phi \\text{'}  a^{<t-1>}[\\Phi \\text{'}  a^{<t-2>}] + ... + \\frac{\\partial L^{<t>}}{\\partial y^{<t>}} . \\Theta\\text{'}. W_{ya}. \\Phi \\text{'}  a^{<t-1>}[\\Phi \\text{'}  a^{<t-2>}]..[\\Phi \\text{'}  a^{<0>}]$$\n",
    "\n",
    "This is very problematic:\n",
    "Vanishing/Exploding gradient problem!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Note**\n",
    "> \n",
    "> *Training set*: large corpus (NLP terminology for text with large body) of any language text\n",
    "> \n",
    "> *Tokenize*: Divide the sentences into piece of string (words, n-gram) \n",
    "> \n",
    "> *Word Tokenize*: divide the sentences into words. Punctuation if is useful can be considere like a word.\n",
    "> \n",
    "> $$\\text{cats average 15 hours of sleep a day}.$$\n",
    "> \n",
    "> If considere `.` like a token the tokens would be $y^{<1>}, ..., y^{<8>}, y^{<9>}$, but if it not the tokens would be $y^{<1>}, ..., y^{<8>}$. \n",
    "> \n",
    "> If we want to tokenize the end of sentence we can add `<EOS>` to tokens.\n",
    "> \n",
    "> If a word is not in a vocabulary (10,000 words) it can be tokenize like `<UNK>` and add to tokens."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Image generation* and *code generation*, constitute a new area of AI that is often called generative AI\n",
    "\n",
    "*Autoregressive generation* or *causal LM generation* sample words conditioned on our previous choices. Predicts a value at time $t$ based\n",
    "on a linear function of the previous values at times $t−1$, $t−2$, and so on. \n",
    "\n",
    "Recall, the predicction start with appropriate *context*. It is state-of-the-art for taks like machine translation, summarization, and question answering.\n",
    "\n",
    "For translation the context is the sentence in the source language; for summarization it’s the long text we want to summarize."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stacked and Bidirectional RNN architectures\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Stacked RNNs consist of multiple networks where the output of one layer serves as\n",
    "the input to a subsequent layer\n",
    "\n",
    "- A bidirectional RNN (Schuster and Paliwal, 1997) combines two independent RNNs, one where the input is processed from the start to the end, and the other from the end to the start.\n",
    "Bidirectional RNNs have also proven to be quite effective for sequence classification. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style = \"\">\n",
    "    <div style = \"display:flex;align-items:center; justify-content:center; flex-direction:column\" >\n",
    "    <img src = \"./assets/Bidirectional RNN.png\" style=\"height:180px; border: 0.5px solid;\">\n",
    "    </div>\n",
    "    \n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- ### In the practice\n",
    "\n",
    "Let the sequence `\"The reinforcement learning is the key of machine learning\"` and we want to build a *Language Model* to predict the next word. \n",
    "\n",
    "So first we need to convert the sequence into *tokens* (in this case the token is a word). \n",
    "\n",
    "So the sequence is convert to a vector: $[\\text{The}, \\: \\text{reinforcement}, \\: \\text{learning},\\: \\text{is}, \\:\\text{the}, \\: \\text{key}, \\: \\text{of},\\: \\text{machine},\\: \\text{learning}]$. \n",
    "\n",
    "Above sequence is splitted in *Source Text* and *Target Text* of equal length. $T_x = T_y$\n",
    "\n",
    "Source Text : $[\\text{The}, \\: \\text{reinforcement}, \\: \\text{learning},\\: \\text{is}, \\:\\text{the}, \\: \\text{key}, \\: \\text{of},\\: \\text{machine}]$. \n",
    "\n",
    "Target Text : $[\\text{reinforcement}, \\: \\text{learning},\\: \\text{is}, \\:\\text{the}, \\: \\text{key}, \\: \\text{of},\\: \\text{machine},\\: \\text{learning}]$. \n",
    "\n",
    "We can fomalize into variables\n",
    "\n",
    "Let be $x^{<t>} = [x^{<1>}, .., x^{<8>}]$ and $y^{<t>} = [y^{<1>}, .., y^{<8>}]$ -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- When $x^{<t>}$ and $y^{<t>}$ is fed to RNN (1 hidden layer) to train, it apply the following steps:\n",
    "\n",
    "- $x^{<1>} = \\text{The}$ is transformed using *embedding transformation* (we define the size of vocabulary and the dimention of embedding) or *one-hot* (based in to the vocabulary). So if $\\text{vocabulary}\\_\\text{size} = 20$ and the dimention of  $\\text{embedding}\\_\\text{size} = 5$ the vector would be $x^{<1>} = [0.79632116, 0.09251072, 0.20748794, 0.20226105, 0.533899253]$ and each time that the word *the* needs to encode this vector will be used.\n",
    "\n",
    "- $a^{<0>}$ init with zero values $[0, 0, 0, 0, 0]$, $W_{aa}$ and $W_{ax}$ are matrices  of shape = $(5, 5)$ inits with random values. Where the first $5$ to left are the weights for the first unit and $b_a$ is a vector of shape $(1, 5)$ where each single value is the bias for each unit.\n",
    "- With those parameters and data we can compute $a^{<1>} =\\Phi(a^{<0>}\\text{.}W_{aa} + x^{<1>}\\text{.}W_{ax} + b_a)$.\n",
    "\n",
    "- Since we need to predict the probability of the next word, we use $\\text{softmax}$ function and output should be equal to $20$ the size of the vocabulary. So we can compute the probabilities $P(y^{<1>}), P(y^{<2>}), ..., P(y^{<20>})$.\n",
    "\n",
    "- With those requeriments the weights $W_{ya}$ is a matrix of shape $(5,20)$ and $b_y$ is vecor of shape $(1, 20)$ with random values at init.\n",
    "\n",
    "- So we can compute the $y^{<1>} = \\Theta(a^{<1>}\\text{.}W_{ya} + b_y)$.\n",
    "- In the next iterartion we will apply the same steps but with the data and parameters of the previous iteration. -->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "from torch.utils.data import Dataset, DataLoader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define a simple dataset for word prediction\n",
    "class WordDataset(Dataset):\n",
    "    def __init__(self, sequences, sequence_length):\n",
    "        \n",
    "        self.sequences = sequences\n",
    "        self.sequence_length = sequence_length\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.sequences)\n",
    "\n",
    "    def __getitem__(self, idx):\n",
    "        sequence = self.sequences[idx]\n",
    "        input_sequence = torch.tensor(sequence[:-1], dtype=torch.long)\n",
    "        target = torch.tensor(sequence[1:], dtype=torch.long)\n",
    "        return input_sequence, target\n",
    "\n",
    "# Define the RNN model\n",
    "class RNN(nn.Module):\n",
    "    def __init__(self, vocabulary_size, embedding_size, hidden_size, output_size):\n",
    "        super(RNN, self).__init__()\n",
    "        self.hidden_size = hidden_size\n",
    "        self.embedding = nn.Embedding(vocabulary_size, embedding_size)\n",
    "        self.rnn = nn.RNN(embedding_size, hidden_size, batch_first=True)\n",
    "        self.fc = nn.Linear(hidden_size, output_size)\n",
    "\n",
    "    def forward(self, x, hidden):\n",
    "        embedded = self.embedding(x)\n",
    "        out, hidden = self.rnn(embedded, hidden)\n",
    "        print(\"Forward: \",out)\n",
    "        print(\"Hidden: \",hidden)\n",
    "        out = self.fc(out)\n",
    "        print(\"out: \",out)\n",
    "        return out, hidden"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function to generate a simple dataset\n",
    "def generate_dataset():\n",
    "    # Simple dataset of word sequences\n",
    "    sequences = [\n",
    "        [0, 1, 2, 3, 4],  # \"hello world\"\n",
    "        [5, 6, 7, 8, 4],  # \"how are you\"\n",
    "        [9, 1, 10, 3, 4], # \"goodbye world\"\n",
    "    ]\n",
    "    return sequences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define hyperparameters\n",
    "\n",
    "# input_size = 11\n",
    "vocabulary_size = 11\n",
    "# Input size = Vocabulary size\n",
    "# This implies that the range of posibles values \n",
    "# is between 0-10. \n",
    "hidden_size = 8 # Size of the hidden state\n",
    "output_size = 11  # Output size (same as input size)\n",
    "sequence_length = 4  # Sequence length for training\n",
    "batch_size = 1  # Batch size for training\n",
    "num_epochs = 100  # Number of training epochs\n",
    "embedding_size = 10\n",
    "# Create the dataset and dataloader\n",
    "sequences = generate_dataset()\n",
    "dataset = WordDataset(sequences, sequence_length)\n",
    "dataloader = DataLoader(dataset, batch_size=batch_size, shuffle=True)\n",
    "\n",
    "# Create the RNN model instance\n",
    "model = RNN(vocabulary_size, embedding_size, hidden_size, output_size)\n",
    "\n",
    "# Define the loss function and optimizer\n",
    "# For multinomial model\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "optimizer = optim.SGD(model.parameters(), lr=0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Training loop\n",
    "# for epoch in range(num_epochs):\n",
    "#     total_loss = 0\n",
    "for idx, (batch_inputs, batch_targets) in enumerate(dataloader):\n",
    "    optimizer.zero_grad()\n",
    "    hidden = torch.zeros(1, batch_size, hidden_size) # Initialize hidden state\n",
    "    outputs, _ = model(batch_inputs, hidden)\n",
    "    print(\"outputs\", outputs)\n",
    "    print('-', _)\n",
    "    loss = criterion(outputs.view(-1, output_size), batch_targets.view(-1))\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "    # total_loss += loss.item()    \n",
    "    # print(f'Epoch {epoch+1}/{num_epochs}, Loss: {total_loss/len(dataloader):.4f}')\n",
    "    if idx == 0:\n",
    "        break\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prediction\n",
    "test_input = torch.tensor([[0, 1, 2, 3]], dtype=torch.long)  # Input sequence: \"hell\"\n",
    "hidden = torch.zeros(1, 1, hidden_size)  # Initialize hidden state\n",
    "with torch.no_grad():\n",
    "    output, _ = model(test_input, hidden)\n",
    "predicted_idx = torch.argmax(output, dim=2)\n",
    "print(\"Predicted sequence:\", [idx.item() for idx in predicted_idx.squeeze()])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The LSTM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The main reason to \n",
    "\n",
    "Downsides of using RNN:\n",
    "\n",
    "- *Retrieval Distant Information* is difficult since the information encoded in hidden states tends to be fairly local, more relevant to the most recent parts of the input sequence and recent decisions. This happen since the weights perform two task simultaneously *provide information useful for the current decision, and updating and carrying forward information required for future decisions*.\n",
    "- A second difficulty with training RNN is *vanishing gradients problem*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**LSTM** divide the context management problem into two subproblems: removing information no longer\n",
    "needed from the context, and adding information likely to be needed for later decision making.\n",
    "\n",
    "LSTMs accomplish this by first adding an explicit context layer to the architecture (in addition to the\n",
    "usual recurrent hidden layer), and through the use of specialized neural units that\n",
    "make use of gates to control the flow of information into and out of the units that\n",
    "comprise the network layers. \n",
    "\n",
    "The *gates* in an LSTM share a common design pattern; each consists of a feedforward layer, followed by a sigmoid activation function, followed by a pointwise multiplication with the layer being gated.\n",
    "\n",
    "\n",
    "Values in the layer being *gated* that align with values near $1$ in the mask are passed through\n",
    "nearly unchanged; values corresponding to lower values are essentially erased"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first gate we’ll consider is the **forget gate**. The purpose of this gate is to delete information from the context that is no longer needed.\n",
    "- $f_t = \\sigma(U_f h_{t-1} + W_f x_t)$\n",
    "- $k_t = c_{t-1} \\odot f_{t}$\n",
    "\n",
    "The next task is to compute the actual information we need to extract from the previous hidden state and current inputs—the same basic computation we’ve been using for all our recurrent networks\n",
    "\n",
    "- $g_t=\\tanh(U_g h_{t-1} + W_g x_{t})$\n",
    "\n",
    "Next, we generate the mask for the add gate to select the information to add to the *current context*\n",
    "\n",
    "- $i_t = \\sigma(U_i h_{t-1} + W_i x_{t})$\n",
    "- $j_t = g_t \\odot i_t$\n",
    "\n",
    "Next, we add this to the modified context vector to get our new context vector\n",
    "\n",
    "- $c_t = k_t + j_t $\n",
    "\n",
    "The final gate we’ll use is the output gate which is used to decide what information is required for the current hidden state (as opposed to what information needs to be preserved for future decisions\n",
    "- $o_t=\\sigma(U_o h_{t-1} + W_o x_{t})$\n",
    "- $h_t = o_t \\odot \\tanh(c_t) $"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Encoder-Decoder Model with RNN\n",
    "\n",
    "The *encoder-decoder* (sequence-to-sequence) networks model is used in *machine translation task*, in which the input and output sequence has different lenght.\n",
    "\n",
    "The key idea is train an *encoder network* that take an input sequence and create a context that is passed to *decoder network* that generate a sequence."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. An *encoder* that accepts an input sequence, $x$, and generates a corresponding\n",
    "sequence of contextualized representations, $h^e$ . LSTMs, convolutional networks, and Transformers can all be employed as encoders.\n",
    "2. A *context vector*, $c$, which is a function of $h^e$, and conveys (carry) the essence (of information) of the input to the decoder.\n",
    "3. A *decoder*, which accepts $c$ as input and generates an arbitrary length sequence of hidden states $h^d$ , from which a corresponding sequence of output states $y$ , can be obtained. Just as with encoders, decoders can be realized by\n",
    "any kind of sequence architecture"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style = \"\">\n",
    "    <div style = \"display:flex;align-items:center; justify-content:center; flex-direction:column\" >\n",
    "    <img src = \"./assets/encoder-decoder-model.png\" style=\"height:220px;\">\n",
    "    </div>\n",
    "    \n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> *Note*\n",
    "> \n",
    "> The entire purpose of the encoder is to generate a contextualized representation of the input"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "0. $e^{<t>}_x = E\\textbf{x}^{<t>}$\n",
    "1. $a^{<t>}_e = \\Phi(W_{aa}a^{<t-1>}_e + W_{ae}e^{<t>}_x + b_a)$\n",
    "2. $c = a^{T_x}_e=a^{0}_d$, $y^{0} = \\text{<s>}$\n",
    "3. $e^{<\\tau>}_y = E\\textbf{y}^{<\\tau>}$\n",
    "4. $a^{<\\tau>}_d = \\Phi(W_{aa}a^{<\\tau-1>}_d + W_{ae}e^{<\\tau>}_y + W_{ac}c + b_a)$\n",
    "<!-- 5. $c = a^{T_x}_e$ -->\n",
    "5. $y^{<\\tau>} = \\Theta(W_{ya}a^{<\\tau>}_d + b_y)$\n",
    "\n",
    "> Where $W_{ya}$ are the weights used to compute $y$ multiplied the vector $a$ \n",
    ">\n",
    "> $a^{<0>}$ is a vector of zeros"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Enconder \n",
    "1. Let $\\textbf{x}^{<{t}>}$, of the dimention $|V|$ x $1$, the one-hot representation of $x^{<{t}>}$ (word).\n",
    "2. Given the $E$ matrix of embedding with dimention ${d}_e$ x $\\text{|V|}$ (where $d$ is the length of vector that represent the word).\n",
    "3. Multiply $E$ with $\\textbf{x}^{<t>}$ resulting $e^{<t>}_x$ of the dimention $d_e$ x $1$.\n",
    "4. $e^{<t>}_x$ is multiplied by $W_{ae}$ (of the dimention $d_h$ x $d_e$) and $a^{<t-1>}_x$ (of the dimention $d_a$ x $1$ with initial values set to zeros) is multiplied by $W_{aa}$ (of the dimention $d_h$ x $d_a$). This last component summarize all preceding relationship.\n",
    "4. The previous results are sum and added the bias $b_a$ (of dimention $d_h$ x $1$ ), then is passed to no-linear function (ReLu for example) resulting $a^{<t>}_e$ of dimention $d_h$ x 1.\n",
    "5. At time $t=T_x$ (the length of the input sequence) we obtain $a^{T_x}_e$, this is the context vector ($c$) used in decoder net, the vector that summarize all input information.\n",
    "\n",
    "Decoder\n",
    "\n",
    "1. Start with initial values $a^{<0>}_d=c$, $y^{<1>}=\\text{<s>}$\n",
    "1. Let $\\textbf{y}^{<\\tau>}$, of the dimention $|V|$ x $1$, the one-hot representation of $y^{<\\tau>}$ (word).\n",
    "2. Given the $E$ matrix of embedding with dimention ${d}_e$ x $\\text{|V|}$ (where $d$ is the length of vector that represent the word).\n",
    "3. Multiply $E$ with $\\textbf{y}^{<\\tau>}$ resulting $e^{<\\tau>}_y$ of the dimention $d_e$ x $1$.\n",
    "4. $e^{<\\tau>}_y$ is multiplied by $W_{ae}$ (of the dimention $d_h$ x $d_e$), $c$ (of the dimention $d_a$ x $1$) is multiplied by $W_{ac}$ (of the dimention $d_h$ x $d_a$) and $a^{<\\tau-1>}_d$ (of the dimention $d_a$ x $1$ with initial values set to zeros) is multiplied by $W_{aa}$ (of the dimention $d_h$ x $d_a$). \n",
    "4. The previous results are sum and added the bias $b_a$ (of dimention $d_h$ x $1$ ), then is passed to no-linear function (ReLu for example) resulting $a^{<\\tau>}_d$ of dimention $d_h$ x 1.\n",
    "5. $a^{<\\tau>}_d$ is multiply by $W_{ya}$ (of dimention $|V|$ x $d_h$) and added a bias $b_y$ (of the dimention $|V|$ x $1$).\n",
    "6. At the previously result (called *logits*) apply softmax ($\\Theta$) getting the probability distribution over the possible output classes\n",
    "7. The class that maximize the probability is the class chosen."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In training, therefore, it is more common to use teacher forcing in the\n",
    "decoder. *Teacher forcing* means that we force the system to use the gold target token\n",
    "from training as the next input, rather than allowing it to rely on the (possibly\n",
    "erroneous) decoder output."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Attention \n",
    "\n",
    "Loung Attention "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Downsides of using a simple encoder-decoder:\n",
    "- The final hidden state is thus acting as a **bottleneck**. It must represent absolutely\n",
    "everything about the meaning of the source text, since the only thing the decoder\n",
    "knows about the source text is what’s in this context vector\n",
    "\n",
    "The attention mechanism is a solution to the bottleneck problem, a way of\n",
    "allowing the decoder to get information from all the hidden states of the encoder,\n",
    "not just the last hidden state\n",
    "\n",
    "The idea of attention is instead to create the single fixed-length vector c by taking\n",
    "a weighted sum of all the encoder hidden states\n",
    "\n",
    "The weights focus on (‘attend to’) a particular part of the source text that is relevant for the token the decoder is\n",
    "currently producing\n",
    "\n",
    "Attention thus replaces the static context vector with one that\n",
    "is dynamically derived from the encoder hidden states"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the weighted average over all the *encoder hidden states*.\n",
    "\n",
    "So $c \\rightarrow c^{<\\tau>} = \\sum_{<t>=1}^{T_x} \\alpha^{<t,\\tau>} a^{<t>}_e$\n",
    "\n",
    "Where the alpha tells us the proportional relevance of each encoder hidden\n",
    "state $<t>$ to the prior hidden decoder state $a^{<\\tau>}_d$\n",
    "\n",
    "$\\alpha^{<t,\\tau>} = \\text{softmax}(\\text{score}(a^{<t>}_e, a^{<\\tau>}_d))$\n",
    "\n",
    "\n",
    "\n",
    "Dot Product Attenction\n",
    "\n",
    "$\\text{score}(a^{<t>}_e, a^{<\\tau>}_d) = a^{<\\tau>}_d.a^{<t>}_e$\n",
    "\n",
    "Implements relevance as attention similarity: measuring how similar the decoder hidden state is to an encoder hidden state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Application\n",
    "\n",
    "The most common way for checking if an email is valid or not is with authentication. But if the company where you work not have an application that can check if an email is valid or not, then you can build a model that can correct the email (correct only domain part). \n",
    "\n",
    "$$\\text{wings@gmial.com} \\rightarrow \\text{wings@gmail.com}$$\n",
    "\n",
    "We will be applying the following concepts:\n",
    "- Embedding\n",
    "- RNN (GRU or LSTM)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from __future__ import unicode_literals, print_function, division\n",
    "from io import open\n",
    "import re\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "from torch import optim\n",
    "import torch.nn.functional as F\n",
    "import numpy as np\n",
    "from torch.utils.data import TensorDataset, DataLoader, RandomSampler\n",
    "\n",
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading data files\n",
    "\n",
    "The data for this project is a set of hundred of domains + extension, correct and incorrect domains + extensions.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We\\'ll need a unique index per letter to use as the inputs and targets of\n",
    "the networks later. To keep track of all this we will use a helper class\n",
    "called `Domain` which has letter → index (`letter2index`) and index → letter\n",
    "(`index2letter`) dictionaries, as well as a count of each letter\n",
    "`letter2count` which will be used to replace rare letters later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 273,
   "metadata": {},
   "outputs": [],
   "source": [
    "SOS_token = 0\n",
    "EOS_token = 1\n",
    "\n",
    "class Domain:\n",
    "    def __init__(self, name):\n",
    "        self.name = name\n",
    "        self.letter2index = {}\n",
    "        self.letter2count = {}\n",
    "        self.index2letter = {0: \"SOS\", 1: \"EOS\"}\n",
    "        self.n_letter = 2  # Count SOS and EOS\n",
    "\n",
    "    def addWord(self, word):\n",
    "        for letter in re.split(\"\",  word):\n",
    "            if letter != '':\n",
    "                self.addLetter(letter)\n",
    "\n",
    "    def addLetter(self, letter):\n",
    "        if letter not in self.letter2index:\n",
    "            self.letter2index[letter] = self.n_letter\n",
    "            self.letter2count[letter] = 1\n",
    "            self.index2letter[self.n_letter] = letter\n",
    "            self.n_letter += 1\n",
    "        else:\n",
    "            self.letter2count[letter] += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 274,
   "metadata": {},
   "outputs": [],
   "source": [
    "def readFile():\n",
    "    print(\"Reading lines...\")\n",
    "    # Read the file and split into lines\n",
    "    lines = open('%s.csv' % ('data'), encoding='utf-8').\\\n",
    "        read().strip().split('\\n')\n",
    "\n",
    "    # Split every line into pairs and normalize\n",
    "    pairs = [[s for s in l.split('\\t')] for l in lines]\n",
    "    \n",
    "    input_domain = Domain('incorrect')\n",
    "    output_domain = Domain('correct')\n",
    "\n",
    "    return input_domain, output_domain, pairs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def prepareData():\n",
    "    input_domain, output_domain, pairs = readFile()\n",
    "\n",
    "    print(\"Read %s sentence pairs\" % len(pairs))\n",
    "\n",
    "    print(\"Counting letters...\")\n",
    "\n",
    "    for pair in pairs:\n",
    "        input_domain.addWord(pair[0])\n",
    "        output_domain.addWord(pair[1])\n",
    "\n",
    "    print(\"Counted letters:\")\n",
    "    print(input_domain.name, input_domain.n_letter)\n",
    "    print(output_domain.name, output_domain.n_letter)\n",
    "    return input_domain, output_domain, pairs\n",
    "\n",
    "input_domain, output_domain, pairs = prepareData()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 277,
   "metadata": {},
   "outputs": [],
   "source": [
    "MAX_LENGTH = 35"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Encoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 278,
   "metadata": {},
   "outputs": [],
   "source": [
    "class EncoderRNN(nn.Module):\n",
    "    def __init__(self,  input_size, hidden_size, p_dropout=0.1):\n",
    "        super(EncoderRNN, self).__init__()\n",
    "        # If it setted \n",
    "        # input size is the vocabulary size\n",
    "        # in this example is = 35\n",
    "        #\n",
    "        self.embedding = nn.Embedding(input_size, \n",
    "                                      hidden_size)\n",
    "        \n",
    "        self.gru = nn.GRU(hidden_size,\n",
    "                           hidden_size, \n",
    "                           batch_first=True)\n",
    "        \n",
    "        self.dropout = nn.Dropout(p=p_dropout)\n",
    "\n",
    "    def forward(self, input):\n",
    "        embedded = self.dropout(self.embedding(input))\n",
    "        output, hidden = self.gru(embedded)\n",
    "        return output, hidden"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Decoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 279,
   "metadata": {},
   "outputs": [],
   "source": [
    "class BahdanauAttention(nn.Module):\n",
    "    def __init__(self, hidden_size) -> None:\n",
    "        super(BahdanauAttention, self).__init__()\n",
    "        self.Wa = nn.Linear(hidden_size, hidden_size)\n",
    "        self.Ua = nn.Linear(hidden_size, hidden_size)\n",
    "        self.Va = nn.Linear(hidden_size, 1)\n",
    "\n",
    "    def forward(self, query, keys):\n",
    "        scores = self.Va(torch.tanh(self.Wa(query) + self.Ua(keys)))\n",
    "        scores = scores.squeeze(2).unsqueeze(1)\n",
    "        # Batch Matrix-Matrix Product\n",
    "        # Both input tensors must be 3-dimensional with the same batch size.\n",
    "        weights = F.softmax(scores, dim=-1)\n",
    "        context = torch.bmm(weights, keys)\n",
    "\n",
    "        return context, weights\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 280,
   "metadata": {},
   "outputs": [],
   "source": [
    "class AttnDecoderRNN(nn.Module):\n",
    "    def __init__(self, hidden_size, output_size, dropout_p=0.1):\n",
    "        super(AttnDecoderRNN, self).__init__()\n",
    "        self.embedding = nn.Embedding(output_size,hidden_size)\n",
    "        self.attention = BahdanauAttention(hidden_size)\n",
    "        self.gru = nn.GRU(2*hidden_size, hidden_size, batch_first=True)\n",
    "        self.out = nn.Linear(hidden_size, output_size)\n",
    "        self.dropout = nn.Dropout(dropout_p)\n",
    "\n",
    "    def forward(self, encoder_outputs, encoder_hidden, target_tensor=None):\n",
    "        batch_size = encoder_outputs.size(0)\n",
    "        decoder_input = torch.empty(batch_size, 1, \n",
    "                                    dtype=torch.long, device=device).fill_(SOS_token)\n",
    "        decoder_hidden = encoder_hidden\n",
    "        decoder_outputs = []\n",
    "        attentions = []\n",
    "\n",
    "        for i in range(MAX_LENGTH):\n",
    "            decoder_output, decoder_hidden, attn_weights = self.forward_step(\n",
    "                decoder_input, decoder_hidden, encoder_outputs\n",
    "            )\n",
    "            decoder_outputs.append(decoder_output)\n",
    "            attentions.append(attn_weights)\n",
    "\n",
    "            if target_tensor is not None:\n",
    "                # Teacher forcing: Feed the target as the next input\n",
    "                decoder_input = target_tensor[:, i].unsqueeze(1) # Teacher forcing\n",
    "            else:\n",
    "                # Without teacher forcing: use its own predictions as the next input\n",
    "                _, topi = decoder_output.topk(1)\n",
    "                decoder_input = topi.squeeze(-1).detach()  # detach from history as input\n",
    "\n",
    "        decoder_outputs = torch.cat(decoder_outputs, dim=1)\n",
    "        decoder_outputs = F.log_softmax(decoder_outputs, dim=-1)\n",
    "        attentions = torch.cat(attentions, dim=1)\n",
    "\n",
    "        return decoder_outputs, decoder_hidden, attentions\n",
    "\n",
    "\n",
    "    def forward_step(self, input, hidden, encoder_outputs):\n",
    "        # batch_size = 32\n",
    "        # hidden_size = 20\n",
    "        # input shape = (32, 1)\n",
    "        embedded =  self.dropout(self.embedding(input))\n",
    "        # for each value the embedding will be of a shape of (1, 20)\n",
    "        # for all values the embedding will be of shape of (32, 20)\n",
    "\n",
    "        query = hidden.permute(1, 0, 2)\n",
    "        context, attn_weights = self.attention(query, encoder_outputs)\n",
    "        input_gru = torch.cat((embedded, context), dim=2)\n",
    "\n",
    "        output, hidden = self.gru(input_gru, hidden)\n",
    "        output = self.out(output)\n",
    "\n",
    "        return output, hidden, attn_weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 281,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = torch.rand(5, 3) \n",
    "#.permute(1, 0, 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x.size(1) #.permute(2, 1, 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 284,
   "metadata": {},
   "outputs": [],
   "source": [
    "def indexesFromWord(domain, word):\n",
    "    return [domain.letter2index[letter]\n",
    "            for letter in re.split(\"\",word) if letter != \"\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 285,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tensorFromWord(domain, word):\n",
    "    indexes = indexesFromWord(domain, word)\n",
    "    indexes.append(EOS_token)\n",
    "    out = torch.tensor(indexes, dtype=torch.long, device=device).view(1, -1)\n",
    "    return  out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 286,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tensorsFromPair(pair):\n",
    "    input_tensor = tensorFromWord(input_domain, pair[0])\n",
    "    target_tensor = tensorFromWord(output_domain, pair[1])\n",
    "    return (input_tensor, target_tensor)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 287,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_dataloader(batch_size):\n",
    "    input_domain, output_domain, pairs = prepareData()\n",
    "\n",
    "    n = len(pairs)\n",
    "    input_ids = np.zeros((n, MAX_LENGTH), dtype=np.int32)\n",
    "    target_ids = np.zeros((n, MAX_LENGTH), dtype=np.int32)\n",
    "\n",
    "    for idx, (inp, tgt) in enumerate(pairs):\n",
    "        inp_ids = indexesFromWord(input_domain, inp)\n",
    "        tgt_ids = indexesFromWord(output_domain, tgt)\n",
    "        inp_ids.append(EOS_token)\n",
    "        tgt_ids.append(EOS_token)\n",
    "        input_ids[idx, :len(inp_ids)] = inp_ids\n",
    "        target_ids[idx, :len(tgt_ids)] = tgt_ids\n",
    "\n",
    "    train_data = TensorDataset(torch.LongTensor(input_ids).to(device),\n",
    "                               torch.LongTensor(target_ids).to(device))\n",
    "\n",
    "    train_sampler = RandomSampler(train_data)\n",
    "    train_dataloader = DataLoader(train_data, sampler=train_sampler, batch_size=batch_size)\n",
    "    return input_domain, output_domain, train_dataloader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_domain, output_domain, train_dataloader = get_dataloader(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 289,
   "metadata": {},
   "outputs": [],
   "source": [
    "encoder = EncoderRNN(35, 20)\n",
    "encoder_optimizer = optim.Adam(encoder.parameters(), lr=0.001)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 290,
   "metadata": {},
   "outputs": [],
   "source": [
    "decoder = AttnDecoderRNN(20,35)\n",
    "decoder_optimizer = optim.Adam(decoder.parameters(), lr=0.001)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 291,
   "metadata": {},
   "outputs": [],
   "source": [
    "attn = BahdanauAttention(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for idx, data in enumerate(train_dataloader):\n",
    "    input_tensor, target_tensor = data\n",
    "    encoder_optimizer.zero_grad()\n",
    "    encoder_outputs, encoder_hidden = encoder(input_tensor)\n",
    "    # decoder_outputs, _, _ = decoder(encoder_outputs, encoder_hidden, target_tensor)\n",
    "    # if idx == 0:\n",
    "    #     break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "decoder_outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 252,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = encoder_outputs.size(0)\n",
    "decoder_input = torch.empty(batch_size, 1, \n",
    "                                    dtype=torch.long, device=device).fill_(SOS_token)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "encoder_hidden.permute(1, 0, 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "encoder_hidden"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "encoder_outputs.permute(1, 0, 2)[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "decoder_input"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "qq.forward_step(decoder_input, encoder_hidden, encoder_outputs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 205,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "decoder_hidden = encoder_hidden\n",
    "decoder_outputs = []\n",
    "attentions = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "encoder_hidden[0][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_epoch(dataloader, encoder, decoder, encoder_optimizer,\n",
    "          decoder_optimizer, criterion):\n",
    "\n",
    "    total_loss = 0\n",
    "    for data in dataloader:\n",
    "        input_tensor, target_tensor = data\n",
    "\n",
    "        encoder_optimizer.zero_grad()\n",
    "        decoder_optimizer.zero_grad()\n",
    "\n",
    "        encoder_outputs, encoder_hidden = encoder(input_tensor)\n",
    "        decoder_outputs, _, _ = decoder(encoder_outputs, encoder_hidden, target_tensor)\n",
    "\n",
    "        loss = criterion(\n",
    "            decoder_outputs.view(-1, decoder_outputs.size(-1)),\n",
    "            target_tensor.view(-1)\n",
    "        )\n",
    "        loss.backward()\n",
    "\n",
    "        encoder_optimizer.step()\n",
    "        decoder_optimizer.step()\n",
    "\n",
    "        total_loss += loss.item()\n",
    "\n",
    "    return total_loss / len(dataloader)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "import math\n",
    "\n",
    "def asMinutes(s):\n",
    "    m = math.floor(s / 60)\n",
    "    s -= m * 60\n",
    "    return '%dm %ds' % (m, s)\n",
    "\n",
    "def timeSince(since, percent):\n",
    "    now = time.time()\n",
    "    s = now - since\n",
    "    es = s / (percent)\n",
    "    rs = es - s\n",
    "    return '%s (- %s)' % (asMinutes(s), asMinutes(rs))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train(train_dataloader, encoder, decoder, n_epochs, learning_rate=0.001,\n",
    "               print_every=100, plot_every=100):\n",
    "    start = time.time()\n",
    "    plot_losses = []\n",
    "    print_loss_total = 0  # Reset every print_every\n",
    "    plot_loss_total = 0  # Reset every plot_every\n",
    "\n",
    "    encoder_optimizer = optim.Adam(encoder.parameters(), lr=learning_rate)\n",
    "    decoder_optimizer = optim.Adam(decoder.parameters(), lr=learning_rate)\n",
    "    criterion = nn.NLLLoss()\n",
    "\n",
    "    for epoch in range(1, n_epochs + 1):\n",
    "        loss = train_epoch(train_dataloader, encoder, decoder, encoder_optimizer, decoder_optimizer, criterion)\n",
    "        print_loss_total += loss\n",
    "        plot_loss_total += loss\n",
    "\n",
    "        if epoch % print_every == 0:\n",
    "            print_loss_avg = print_loss_total / print_every\n",
    "            print_loss_total = 0\n",
    "            print('%s (%d %d%%) %.4f' % (timeSince(start, epoch / n_epochs),\n",
    "                                        epoch, epoch / n_epochs * 100, print_loss_avg))\n",
    "\n",
    "        if epoch % plot_every == 0:\n",
    "            plot_loss_avg = plot_loss_total / plot_every\n",
    "            plot_losses.append(plot_loss_avg)\n",
    "            plot_loss_total = 0\n",
    "\n",
    "    # showPlot(plot_losses)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hidden_size = 128\n",
    "batch_size = 32\n",
    "\n",
    "input_lang, output_lang, train_dataloader = get_dataloader(batch_size)\n",
    "\n",
    "encoder = EncoderRNN(input_lang.n_words, hidden_size).to(device)\n",
    "# decoder = AttnDecoderRNN(hidden_size, output_lang.n_words).to(device)\n",
    "decoder = AttnDecoderRNN(hidden_size, output_lang.n_words).to(device)\n",
    "\n",
    "train(train_dataloader, encoder, decoder, 80, print_every=5, plot_every=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "len_max = max(data_model.DOMAIN_EXTENSION_FEATURES.str.len())\n",
    "domain_extension_feat = data_model.DOMAIN_EXTENSION_FEATURES.str.ljust(len_max, '|') \n",
    "set_of_chars = set(domain_extension_feat.sum())\n",
    "vocabulary = {char:idx for idx, char in enumerate(set_of_chars)}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def replace_chars(word):\n",
    "    replaced_word = [vocabulary.get(char, char) for char in word]\n",
    "    return replaced_word"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = domain_extension_feat.apply(replace_chars)\n",
    "data_idx = list(data.values)\n",
    "target_idx = data_model[[\"IDX_DOMAIN\"]].values.tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "from torch.utils.data import Dataset, DataLoader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define a simple dataset for word prediction\n",
    "class WordDataset(Dataset):\n",
    "    def __init__(self, sequences, target, sequence_length):\n",
    "        self.sequences = sequences\n",
    "        self.target = target\n",
    "        self.sequence_length = sequence_length\n",
    "        \n",
    "    def __len__(self):\n",
    "        return len(self.sequences)\n",
    "\n",
    "    def __getitem__(self, idx):\n",
    "        sequence = self.sequences[idx]\n",
    "        input_sequence = torch.tensor(sequence, dtype=torch.long)\n",
    "        target = torch.tensor(self.target[idx], dtype=torch.long)\n",
    "        return input_sequence, target\n",
    "\n",
    "# Define the RNN model\n",
    "class RNN(nn.Module):\n",
    "    def __init__(self, input_size, hidden_size, output_size, num_layers):\n",
    "        super(RNN, self).__init__()\n",
    "        self.hidden_size = hidden_size\n",
    "        self.embedding = nn.Embedding(input_size, hidden_size)\n",
    "        self.rnn = nn.RNN(hidden_size, hidden_size,num_layers=num_layers, batch_first=True)\n",
    "        self.fc = nn.Linear(hidden_size, output_size)\n",
    "\n",
    "    def forward(self, x, hidden):\n",
    "        embedded = self.embedding(x)\n",
    "        out, hidden = self.rnn(embedded, hidden)\n",
    "        out = self.fc(out)\n",
    "        return out, hidden"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define hyperparameters\n",
    "input_size = len(vocabulary)  # Vocabulary size\n",
    "hidden_size = 20  # Size of the hidden state\n",
    "output_size = 1  # Output size\n",
    "sequence_length = len_max  # Sequence length for training\n",
    "batch_size = 1  # Batch size for training\n",
    "num_epochs = 10  # Number of training epochs\n",
    "num_layers = 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the dataset and dataloader\n",
    "# sequences = generate_dataset()\n",
    "dataset = WordDataset(data_idx, target_idx, sequence_length)\n",
    "dataloader = DataLoader(dataset, batch_size=batch_size, shuffle=True)\n",
    "\n",
    "# Create the RNN model instance\n",
    "model = RNN(input_size, hidden_size, output_size, num_layers)\n",
    "\n",
    "# Define the loss function and optimizer\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "optimizer = optim.SGD(model.parameters(), lr=0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Training loop\n",
    "for epoch in range(num_epochs):\n",
    "    total_loss = 0\n",
    "    for batch_inputs, batch_targets in dataloader:\n",
    "        optimizer.zero_grad()\n",
    "        hidden = torch.zeros(num_layers, batch_size, hidden_size)  # Initialize hidden state\n",
    "        outputs, _ = model(batch_inputs, hidden)\n",
    "        b_target = (torch.nn.functional.one_hot(batch_targets.view(-1).to(torch.int64), len_max).T).to(torch.float64)\n",
    "        loss = criterion(outputs.view(-1, output_size), b_target)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        total_loss += loss.item()\n",
    "    print(f'Epoch {epoch+1}/{num_epochs}, Loss: {total_loss/len(dataloader):.4f}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "outputs.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "s = dataset[11][0].view(1, -1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset[11][1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "(torch.nn.functional.one_hot(dataset[11][1].to(torch.int64), len_max).T)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prediction\n",
    "# test_input = torch.tensor([[0, 1, 2, 3]], dtype=torch.long)  # Input sequence: \"hell\"\n",
    "hidden = torch.zeros(num_layers, 1, hidden_size)  # Initialize hidden state\n",
    "with torch.no_grad():\n",
    "    output, _ = model(s, hidden)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "base",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}