experiment_C11.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 深度学习处理文本\r\n",
    "---\r\n",
    "* 为机器学习应用预处理文本数据\r\n",
    "* 用于文本处理的词袋方法和序列模型方法\r\n",
    "* `Transformer`架构\r\n",
    "* 序列到序列学习"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 11.1 自然语言处理概述 `natural language processing, NLP`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 11.2 准备文本数据\r\n",
    "---\r\n",
    "* 1、将本吧标准化。\r\n",
    "* 2、将文本拆分为单元(词元`token`)。\r\n",
    "* 3、将每个词元转换为一个数值向量。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 11.2.1 文本标准化\r\n",
    "---\r\n",
    "* 简单的特征工程:将所有字母转换为小写并删除标点符号。\r\n",
    "* 高级的标准化方法:词干提取(`stemming`):将一个词的变体转换为相同的表示。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 11.2.2 文本拆分(词元化)\r\n",
    "---\r\n",
    "* **单词级词元化(`word-level tokenization`)**\r\n",
    "* **N元语法词元化(`N-gram tokenization`)**\r\n",
    "* **字符级词元化(`character-level tokenization`)**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 11.2.3 建立词表索引"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 11.2.4 使用`TextVectorization`层"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import string\r\n",
    "\r\n",
    "class Vectorizer:\r\n",
    "\r\n",
    "    def standardize(self, text):\r\n",
    "        text = text.lower()\r\n",
    "        \r\n",
    "        return \"\".join(char for char in text if char not in string.punctuation)\r\n",
    "\r\n",
    "    def tokenizer(self, text):\r\n",
    "        text = self.standardize(text)\r\n",
    "\r\n",
    "        return text.split()\r\n",
    "    \r\n",
    "    def make_vocabulary(self, dataset):\r\n",
    "        self.vocabulary = {\"\":0, \"[UNK]\":1}\r\n",
    "\r\n",
    "        for text in dataset:\r\n",
    "            text   = self.standardize(text)\r\n",
    "            tokens = self.tokenizer(text)\r\n",
    "\r\n",
    "            for token in tokens:\r\n",
    "                if token not in self.vocabulary:\r\n",
    "                    self.vocabulary[token] = len(self.vocabulary)\r\n",
    "        \r\n",
    "        self.inverse_vocabulary = dict((v, k) for k, v in self.vocabulary.items())\r\n",
    "    \r\n",
    "    def encode(self, text):\r\n",
    "        text   = self.standardize(text)\r\n",
    "        tokens = self.tokenizer(text)\r\n",
    "        \r\n",
    "        return [self.vocabulary.get(token, 1) for token in tokens]\r\n",
    "\r\n",
    "    def decode(self, int_sequence):\r\n",
    "        \r\n",
    "        return \" \".join(self.inverse_vocabulary.get(i, \"[UNK]\") for i in int_sequence)\r\n",
    "\r\n",
    "vectorizer = Vectorizer()\r\n",
    "\r\n",
    "dataset = [\r\n",
    "    'I write, erase, rewrite',\r\n",
    "    'Erase again, and then',\r\n",
    "    'A poppy blooms.',\r\n",
    "]\r\n",
    "\r\n",
    "vectorizer.make_vocabulary(dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2, 3, 5, 7, 1, 5, 6]\n"
     ]
    }
   ],
   "source": [
    "test_sentence    = 'I write, rewrite, and still rewrite again'\r\n",
    "encoded_sentence = vectorizer.encode(test_sentence)\r\n",
    "print(encoded_sentence)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "i write rewrite and [UNK] rewrite again\n"
     ]
    }
   ],
   "source": [
    "decoded_sentence = vectorizer.decode(encoded_sentence)\r\n",
    "print(decoded_sentence)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.layers import TextVectorization\r\n",
    "\r\n",
    "text_vectorization = TextVectorization(output_mode='int',)\r\n",
    "\r\n",
    "# TextVectorization层 --> 文本标准化方法是:转换为小写字母并删除标点符号.词元化方法是:利用空格进行拆分."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\r\n",
    "import string\r\n",
    "import tensorflow as tf\r\n",
    "\r\n",
    "def custom_standardization_fn(string_tensor):\r\n",
    "    # 将字符串转换为小写字母\r\n",
    "    lowercase_string = tf.strings.lower(string_tensor)\r\n",
    "    \r\n",
    "    # 将标点符号替换为空字符串\r\n",
    "    return tf.strings.regex_replace(lowercase_string, f\"[{re.escape(string.punctuation)}]\", \"\")\r\n",
    "\r\n",
    "def custom_split_fn(string_tensor):\r\n",
    "    # 利用空格对字符串进行拆分\r\n",
    "    return tf.strings.split(string_tensor)\r\n",
    "\r\n",
    "text_vectorization = TextVectorization(\r\n",
    "    output_mode=\"int\",\r\n",
    "    standardize=custom_standardization_fn,\r\n",
    "    split=custom_split_fn,\r\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = [\r\n",
    "    'I write, erase, rewrite',\r\n",
    "    'Erase again, and then',\r\n",
    "    'A poppy blooms.',\r\n",
    "]\r\n",
    "\r\n",
    "text_vectorization.adapt(dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### [C] 11.1 显示词表"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "['',\n '[UNK]',\n 'erase',\n 'write',\n 'then',\n 'rewrite',\n 'poppy',\n 'i',\n 'blooms',\n 'and',\n 'again',\n 'a']"
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text_vectorization.get_vocabulary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor([ 7  3  5  9  1  5 10], shape=(7,), dtype=int64)\n"
     ]
    }
   ],
   "source": [
    "vocabulary       = text_vectorization.get_vocabulary()\r\n",
    "test_sentence    = 'I write, rewrite, and still rewrite again'\r\n",
    "encoded_sentence = text_vectorization(test_sentence)\r\n",
    "print(encoded_sentence)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inverse_vocab    = dict(enumerate(vocabulary))\r\n",
    "decoded_sentence = \" \".join(inverse_vocab[int(i)] for i in encoded_sentence)\r\n",
    "print(decoded_sentence)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TextVectorization 层有两种用法：\r\n",
    "# 1、将其放在tf.data管道中\r\n",
    "int_sequence_dataset = string_dataset.map(text_vectorization, num_parallel_calls=4)  # 参数num_parallel_calls的作用是在多个CPU内核中并行调用map()\r\n",
    "\r\n",
    "# 2、将其作为模型的一部分\r\n",
    "text_input      = keras.Input(shape=(), dtype='string')\r\n",
    "vectorized_text = text_vectorization(text_input)\r\n",
    "embedded_input  = keras.Embedding(...)(vectorized_text)\r\n",
    "output          = ...\r\n",
    "model           = keras.Model(text_input， output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 11.3 表示单词组的两种方法：集合和序列"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 11.3.1 准备`IMDB`影评数据\r\n",
    "---\r\n",
    " `!curl -O https://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz`\r\n",
    "\r\n",
    " `!tar -xf aclImdb_v1.tar.gz`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 将20%的训练文本文件放入一个新目录中\r\n",
    "import os, pathlib, shutil, random\r\n",
    "\r\n",
    "base_dir  = pathlib.Path('aclImdb')\r\n",
    "val_dir   = base_dir / 'val'\r\n",
    "train_dir = base_dir / 'train'\r\n",
    "\r\n",
    "for category in ('neg', 'pos'):\r\n",
    "    if not os.path.exists(val_dir / category):\r\n",
    "        # 创建目录\r\n",
    "        os.makedirs(val_dir / category)\r\n",
    "    else:\r\n",
    "        # 清空目录下文件\r\n",
    "        exist_files = os.listdir(val_dir / category)\r\n",
    "        for exist_file in exist_files:\r\n",
    "            os.remove(os.path.join(val_dir / category, exist_file))\r\n",
    "\r\n",
    "    files = os.listdir(train_dir / category)\r\n",
    "    \r\n",
    "    # 使用种子随机打乱训练文件列表，以确保每次运行代码都会得到相同的验证集\r\n",
    "    random.Random(1337).shuffle(files)\r\n",
    "\r\n",
    "    # 将20%的训练文件用于验证\r\n",
    "    num_val_samples = int(0.2 * len(files))\r\n",
    "    val_files = files[-num_val_samples:]\r\n",
    "\r\n",
    "    for fname in val_files:\r\n",
    "        # 将文件移动到 aclImdb/val/{category}目录中\r\n",
    "        shutil.move(train_dir / category / fname, val_dir / category / fname)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 4196 files belonging to 2 classes.\n",
      "Found 1048 files belonging to 2 classes.\n",
      "Found 25000 files belonging to 2 classes.\n"
     ]
    }
   ],
   "source": [
    "from tensorflow import keras\r\n",
    "\r\n",
    "batch_size = 32\r\n",
    "\r\n",
    "train_ds = keras.utils.text_dataset_from_directory('aclImdb/train', batch_size=batch_size)\r\n",
    "val_ds   = keras.utils.text_dataset_from_directory('aclImdb/val'  , batch_size=batch_size)\r\n",
    "test_ds  = keras.utils.text_dataset_from_directory('aclImdb/test' , batch_size=batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [C] 11.2 显示第一个批量的形状和数据类型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for inputs, targets in train_ds:\r\n",
    "    print('inputs.shape:' , inputs.shape)\r\n",
    "    print('inputs.dtype:' , inputs.dtype)\r\n",
    "    print('targets.shape:', targets.shape)\r\n",
    "    print('targets.dtype:', targets.dtype)\r\n",
    "    print('inputs[0]:'    , inputs[0])\r\n",
    "    print('targets[0]:'   , targets[0])\r\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 11.3.2 将单词作为集合处理:词袋方法"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1 单个单词(一元语法)的二进制编码"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### [C] 11.3 用`TextVectorization`层预处理数据集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "text_vectorization = TextVectorization(\r\n",
    "    max_tokens=20000,         # 前20000个最常出现的单词\r\n",
    "    output_mode='multi_hot',  # 将输出词元编码为 multi_hot 二进制向量\r\n",
    ")\r\n",
    "\r\n",
    "# 准备一个数据集，只包含原始文本输入(不包含标签)\r\n",
    "text_only_train_ds = train_ds.map(lambda x, y:x)\r\n",
    "\r\n",
    "# 利用adapt()方法对数据集词表建立索引\r\n",
    "text_vectorization.adapt(text_only_train_ds)\r\n",
    "\r\n",
    "# 分别对训练、验证、测试数据集进行处理\r\n",
    "binary_1gram_train_ds = train_ds.map(lambda x, y:(text_vectorization(x), y), num_parallel_calls=4)\r\n",
    "binary_1gram_val_ds   = val_ds.map  (lambda x, y:(text_vectorization(x), y), num_parallel_calls=4)\r\n",
    "binary_1gram_test_ds  = test_ds.map (lambda x, y:(text_vectorization(x), y), num_parallel_calls=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### [C] 11.4 查看一元语法二进制数据集的输出"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for inputs, targets in binary_1gram_train_ds:\r\n",
    "    print('inputs.shape:', inputs.shape)\r\n",
    "    print('inputs.dtype:', inputs.dtype)\r\n",
    "    print('targets.shape:', targets.shape)\r\n",
    "    print('targets.dtype:', targets.dtype)\r\n",
    "    print('inputs[0]:'    , inputs[0])\r\n",
    "    print('targets[0]:'   , targets[0])\r\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### [C] **▲** 11.5 模型构建函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow import keras\r\n",
    "from tensorflow.keras import layers\r\n",
    "\r\n",
    "def get_model(max_tokens=20000, hidden_dim=16):\r\n",
    "    inputs  = keras.Input(shape=(max_tokens,))\r\n",
    "    x       = layers.Dense(hidden_dim, activation='relu')(inputs)\r\n",
    "    x       = layers.Dropout(0.5)(x)\r\n",
    "    outputs = layers.Dense(1, activation='sigmoid')(x)\r\n",
    "    \r\n",
    "    model   = keras.Model(inputs, outputs)\r\n",
    "    model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['accuracy'])\r\n",
    "\r\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### [C] 11.6 对一元语法二进制模型进行训练的测试"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " input_1 (InputLayer)        [(None, 20000)]           0         \n",
      "                                                                 \n",
      " dense (Dense)               (None, 16)                320016    \n",
      "                                                                 \n",
      " dropout (Dropout)           (None, 16)                0         \n",
      "                                                                 \n",
      " dense_1 (Dense)             (None, 1)                 17        \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 320,033\n",
      "Trainable params: 320,033\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Epoch 1/10\n",
      "164/164 [==============================] - 3s 16ms/step - loss: 0.5161 - accuracy: 0.7632 - val_loss: 0.3797 - val_accuracy: 0.8519\n",
      "Epoch 2/10\n",
      "164/164 [==============================] - 1s 7ms/step - loss: 0.3083 - accuracy: 0.8883 - val_loss: 0.3044 - val_accuracy: 0.8794\n",
      "Epoch 3/10\n",
      "164/164 [==============================] - 1s 6ms/step - loss: 0.2172 - accuracy: 0.9247 - val_loss: 0.2786 - val_accuracy: 0.8847\n",
      "Epoch 4/10\n",
      "164/164 [==============================] - 1s 7ms/step - loss: 0.1792 - accuracy: 0.9424 - val_loss: 0.2719 - val_accuracy: 0.8924\n",
      "Epoch 5/10\n",
      "164/164 [==============================] - 1s 6ms/step - loss: 0.1369 - accuracy: 0.9539 - val_loss: 0.2763 - val_accuracy: 0.8924\n",
      "Epoch 6/10\n",
      "164/164 [==============================] - 1s 7ms/step - loss: 0.1153 - accuracy: 0.9630 - val_loss: 0.2968 - val_accuracy: 0.8847\n",
      "Epoch 7/10\n",
      "164/164 [==============================] - 1s 6ms/step - loss: 0.1062 - accuracy: 0.9668 - val_loss: 0.3082 - val_accuracy: 0.8840\n",
      "Epoch 8/10\n",
      "164/164 [==============================] - 1s 6ms/step - loss: 0.0857 - accuracy: 0.9739 - val_loss: 0.3289 - val_accuracy: 0.8847\n",
      "Epoch 9/10\n",
      "164/164 [==============================] - 1s 6ms/step - loss: 0.0847 - accuracy: 0.9737 - val_loss: 0.3388 - val_accuracy: 0.8824\n",
      "Epoch 10/10\n",
      "164/164 [==============================] - 1s 6ms/step - loss: 0.0720 - accuracy: 0.9792 - val_loss: 0.3498 - val_accuracy: 0.8824\n",
      "782/782 [==============================] - 6s 8ms/step - loss: 0.3246 - accuracy: 0.8664\n",
      "Test acc: 0.866\n"
     ]
    }
   ],
   "source": [
    "model = get_model()\r\n",
    "model.summary()\r\n",
    "\r\n",
    "callbacks = [\r\n",
    "    keras.callbacks.ModelCheckpoint(\"binary_1gram.keras\",\r\n",
    "                                    save_best_only=True)\r\n",
    "]\r\n",
    "\r\n",
    "model.fit(binary_1gram_train_ds.cache(),\r\n",
    "          validation_data=binary_1gram_val_ds.cache(),\r\n",
    "          epochs=10,\r\n",
    "          callbacks=callbacks)\r\n",
    "\r\n",
    "model = keras.models.load_model(\"binary_1gram.keras\")\r\n",
    "\r\n",
    "print(f\"Test acc: {model.evaluate(binary_1gram_test_ds)[1]:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2 二元语法的二进制编码"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### [C] 11.7 设置`TextVectorization`层返回二元语法"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text_vectorization = TextVectorization(\r\n",
    "    ngrams=2,\r\n",
    "    max_tokens=20000,\r\n",
    "    output_mode='multi_hot',\r\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### [C] 11.8 对二元语法二进制模型进行训练和测试"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text_vectorization.adapt(text_only_train_ds)\r\n",
    "\r\n",
    "binary_2gram_train_ds = train_ds.map(lambda x, y:(text_vectorization(x), y), num_parallel_calls=4)\r\n",
    "binary_2gram_val_ds   = val_ds.map(lambda x, y:(text_vectorization(x), y), num_parallel_calls=4)\r\n",
    "binary_2gram_test_ds  = test_ds.map(lambda x, y:(text_vectorization(x), y), num_parallel_calls=4)\r\n",
    "\r\n",
    "model = get_model()\r\n",
    "model.summary()\r\n",
    "\r\n",
    "callbacks = [\r\n",
    "    keras.callbacks.ModelCheckpoint(\"binary_2gram.keras\",\r\n",
    "                                    save_best_only=True)\r\n",
    "]\r\n",
    "\r\n",
    "model.fit(binary_2gram_train_ds.cache(),\r\n",
    "          validation_data=binary_2gram_val_ds.cache(),\r\n",
    "          epochs=10,\r\n",
    "          callbacks=callbacks)\r\n",
    "\r\n",
    "model = keras.models.load_model(\"binary_2gram.keras\")\r\n",
    "\r\n",
    "print(f\"Test acc: {model.evaluate(binary_2gram_test_ds)[1]:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3 二元语法的`TF-IDF`编码"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### [C] 11.9 设置 `TextVectorization` 层返回词元出现的次数\r\n",
    "---\r\n",
    "将单词计数减去均值并除以方差，对其进行规范化。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "text_vectorization = TextVectorization(\r\n",
    "    ngrams=2,\r\n",
    "    max_tokens=20000,\r\n",
    "    output_mode='count',\r\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### [C] 11.10 设置 `TextVectorization` 层返回 `TF-IDF` 加权输出\r\n",
    "---\r\n",
    "理解 `TF-IDF` 规范化\r\n",
    "* 某个词在一个文档中出现的次数越多，它对理解文档的内容就越重要。\r\n",
    "* 同时，某个词在数据集所有文档中的出现频次也很重要：如果一个词几乎出现在每个文档中，如：the， a，那么这个词就不是特别有信息量，而仅在一小部分文本中出现的词则是非常独特的，因此也非常重要。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "text_vectorization = TextVectorization(\r\n",
    "    ngrams=2,\r\n",
    "    max_tokens=20000,\r\n",
    "    output_mode='tf_idf',\r\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TF-IDF 的计算方法如下：\r\n",
    "def tfidf(term, document, dataset):\r\n",
    "    term_freq = document.count(term)\r\n",
    "    doc_freq  = math.log(sum(doc.count(term) for doc in dataset) + 1)\r\n",
    "\r\n",
    "    return term_freq / doc_freq"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### [C] 11.11 对 `TF-IDF` 二元语法模型进行训练和测试"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_1\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " input_2 (InputLayer)        [(None, 20000)]           0         \n",
      "                                                                 \n",
      " dense_2 (Dense)             (None, 16)                320016    \n",
      "                                                                 \n",
      " dropout_1 (Dropout)         (None, 16)                0         \n",
      "                                                                 \n",
      " dense_3 (Dense)             (None, 1)                 17        \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 320,033\n",
      "Trainable params: 320,033\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Epoch 1/10\n",
      "164/164 [==============================] - 3s 15ms/step - loss: 0.6579 - accuracy: 0.6154 - val_loss: 0.5000 - val_accuracy: 0.7977\n",
      "Epoch 2/10\n",
      "164/164 [==============================] - 1s 6ms/step - loss: 0.4550 - accuracy: 0.7531 - val_loss: 0.3449 - val_accuracy: 0.8611\n",
      "Epoch 3/10\n",
      "164/164 [==============================] - 1s 7ms/step - loss: 0.3754 - accuracy: 0.8129 - val_loss: 0.3429 - val_accuracy: 0.8511\n",
      "Epoch 4/10\n",
      "164/164 [==============================] - 1s 7ms/step - loss: 0.3111 - accuracy: 0.8375 - val_loss: 0.3241 - val_accuracy: 0.8710\n",
      "Epoch 5/10\n",
      "164/164 [==============================] - 1s 7ms/step - loss: 0.2792 - accuracy: 0.8659 - val_loss: 0.3150 - val_accuracy: 0.8672\n",
      "Epoch 6/10\n",
      "164/164 [==============================] - 1s 7ms/step - loss: 0.2455 - accuracy: 0.8741 - val_loss: 0.3559 - val_accuracy: 0.8344\n",
      "Epoch 7/10\n",
      "164/164 [==============================] - 1s 7ms/step - loss: 0.2214 - accuracy: 0.8802 - val_loss: 0.4580 - val_accuracy: 0.7954\n",
      "Epoch 8/10\n",
      "164/164 [==============================] - 1s 7ms/step - loss: 0.2114 - accuracy: 0.8785 - val_loss: 0.3776 - val_accuracy: 0.8702\n",
      "Epoch 9/10\n",
      "164/164 [==============================] - 1s 7ms/step - loss: 0.2001 - accuracy: 0.9006 - val_loss: 0.3574 - val_accuracy: 0.8473\n",
      "Epoch 10/10\n",
      "164/164 [==============================] - 1s 5ms/step - loss: 0.1879 - accuracy: 0.8984 - val_loss: 0.4129 - val_accuracy: 0.8550\n",
      "782/782 [==============================] - 7s 8ms/step - loss: 0.3374 - accuracy: 0.8443\n",
      "Test acc: 0.844\n"
     ]
    }
   ],
   "source": [
    "text_vectorization.adapt(text_only_train_ds)\r\n",
    "\r\n",
    "tfidf_2gram_train_ds = train_ds.map(lambda x, y: (text_vectorization(x), y), num_parallel_calls=4)\r\n",
    "tfidf_2gram_val_ds   = val_ds.map(lambda x, y: (text_vectorization(x), y)  , num_parallel_calls=4)\r\n",
    "tfidf_2gram_test_ds  = test_ds.map(lambda x, y: (text_vectorization(x), y) , num_parallel_calls=4)\r\n",
    "\r\n",
    "model = get_model()\r\n",
    "model.summary()\r\n",
    "\r\n",
    "callbacks = [\r\n",
    "    keras.callbacks.ModelCheckpoint(\"tfidf_2gram.keras\",\r\n",
    "                                    save_best_only=True)\r\n",
    "]\r\n",
    "\r\n",
    "model.fit(tfidf_2gram_train_ds.cache(),\r\n",
    "          validation_data=tfidf_2gram_val_ds.cache(),\r\n",
    "          epochs=10,\r\n",
    "          callbacks=callbacks)\r\n",
    "\r\n",
    "model = keras.models.load_model(\"tfidf_2gram.keras\")\r\n",
    "\r\n",
    "print(f\"Test acc: {model.evaluate(tfidf_2gram_test_ds)[1]:.3f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs           = keras.Input(shape=(1,), dtype=\"string\")  # 每个输入样本都是一个字符串\r\n",
    "processed_inputs = text_vectorization(inputs)               # 应用文本预处理\r\n",
    "outputs          = model(processed_inputs)                  # 应用前面训练好的模型\r\n",
    "inference_model  = keras.Model(inputs, outputs)             # 将端到端的模型实例化"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "91.82 percent positive\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\r\n",
    "\r\n",
    "raw_text_data = tf.convert_to_tensor([\r\n",
    "    [\"That was an excellent movie, I loved it.\"],\r\n",
    "])\r\n",
    "\r\n",
    "predictions = inference_model(raw_text_data)\r\n",
    "\r\n",
    "print(f\"{float(predictions[0] * 100):.2f} percent positive\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 11.3.3 将单词作为序列处理:序列模型方法 `sequence model`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1 第一个实例"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### [C] 11.12 准备整数序列数据集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras import layers\r\n",
    "\r\n",
    "max_length = 600\r\n",
    "max_tokens = 20000\r\n",
    "\r\n",
    "text_vectorization = layers.TextVectorization(\r\n",
    "    max_tokens=max_tokens,\r\n",
    "    output_mode=\"int\",\r\n",
    "    output_sequence_length=max_length,  # 评论的平均长度是233个单词，只有5%的评论超过600个单词\r\n",
    ")\r\n",
    "\r\n",
    "text_vectorization.adapt(text_only_train_ds)\r\n",
    "\r\n",
    "int_train_ds = train_ds.map(lambda x, y: (text_vectorization(x), y), num_parallel_calls=4)\r\n",
    "int_val_ds   = val_ds.map  (lambda x, y: (text_vectorization(x), y), num_parallel_calls=4)\r\n",
    "int_test_ds  = test_ds.map (lambda x, y: (text_vectorization(x), y), num_parallel_calls=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 11.13 构建于 `one-hot` 编码的向量序列之上的序列模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\r\n",
    "\r\n",
    "inputs   = keras.Input(shape=(None,), dtype=\"int64\")\r\n",
    "embedded = tf.one_hot(inputs, depth=max_tokens)\r\n",
    "x        = layers.Bidirectional(layers.LSTM(32))(embedded)\r\n",
    "x        = layers.Dropout(0.5)(x)\r\n",
    "outputs  = layers.Dense(1, activation=\"sigmoid\")(x)\r\n",
    "model    = keras.Model(inputs, outputs, name='OneHot')\r\n",
    "\r\n",
    "model.compile(optimizer=\"rmsprop\",\r\n",
    "              loss=\"binary_crossentropy\",\r\n",
    "              metrics=[\"accuracy\"])\r\n",
    "\r\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### [C] 11.14 训练第一个简单的序列模型\r\n",
    "---\r\n",
    "观察结果：\r\n",
    "* 训练速度非常慢\r\n",
    "* 测试精度不高"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "callbacks = [\r\n",
    "    keras.callbacks.ModelCheckpoint(\"one_hot_bidir_lstm.keras\",\r\n",
    "                                    save_best_only=True)\r\n",
    "]\r\n",
    "\r\n",
    "model.fit(int_train_ds, validation_data=int_val_ds, epochs=10, callbacks=callbacks)\r\n",
    "\r\n",
    "model = keras.models.load_model(\"one_hot_bidir_lstm.keras\")\r\n",
    "\r\n",
    "print(f\"Test acc: {model.evaluate(int_test_ds)[1]:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2 理解词嵌入 `word embedding`\r\n",
    "---\r\n",
    "两个词向量之间的**几何关系**应该反映这两个单词之间的**语义关系**。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3 利用 `Embedding` 层学习词嵌入"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### [C] 11.15 将 `Embedding` 层实例化"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding_layer = layers.Embedding(input_dim=max_tokens, output_dim=256)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### [C] 11.16 从头开始训练一个使用 `Embedding` 层的模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_4\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " input_5 (InputLayer)        [(None, None)]            0         \n",
      "                                                                 \n",
      " embedding_2 (Embedding)     (None, None, 256)         5120000   \n",
      "                                                                 \n",
      " bidirectional_1 (Bidirectio  (None, 64)               73984     \n",
      " nal)                                                            \n",
      "                                                                 \n",
      " dropout_3 (Dropout)         (None, 64)                0         \n",
      "                                                                 \n",
      " dense_5 (Dense)             (None, 1)                 65        \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 5,194,049\n",
      "Trainable params: 5,194,049\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Epoch 1/10\n",
      "164/164 [==============================] - 77s 449ms/step - loss: 0.6779 - accuracy: 0.5601 - val_loss: 0.5987 - val_accuracy: 0.6840\n",
      "Epoch 2/10\n",
      "164/164 [==============================] - 108s 656ms/step - loss: 0.5064 - accuracy: 0.7630 - val_loss: 0.4411 - val_accuracy: 0.8061\n",
      "Epoch 3/10\n",
      "164/164 [==============================] - 134s 814ms/step - loss: 0.3535 - accuracy: 0.8644 - val_loss: 0.4380 - val_accuracy: 0.7885\n",
      "Epoch 4/10\n",
      "164/164 [==============================] - 135s 820ms/step - loss: 0.2647 - accuracy: 0.9085 - val_loss: 0.4002 - val_accuracy: 0.8351\n",
      "Epoch 5/10\n",
      "164/164 [==============================] - 155s 944ms/step - loss: 0.1912 - accuracy: 0.9367 - val_loss: 0.4333 - val_accuracy: 0.8252\n",
      "Epoch 6/10\n",
      "164/164 [==============================] - 162s 990ms/step - loss: 0.1427 - accuracy: 0.9580 - val_loss: 0.5182 - val_accuracy: 0.8328\n",
      "Epoch 7/10\n",
      "164/164 [==============================] - 153s 927ms/step - loss: 0.1053 - accuracy: 0.9695 - val_loss: 0.6439 - val_accuracy: 0.7718\n",
      "Epoch 8/10\n",
      "164/164 [==============================] - 185s 1s/step - loss: 0.0942 - accuracy: 0.9764 - val_loss: 0.5708 - val_accuracy: 0.8374\n",
      "Epoch 9/10\n",
      "164/164 [==============================] - 152s 926ms/step - loss: 0.0734 - accuracy: 0.9811 - val_loss: 1.1898 - val_accuracy: 0.7412\n",
      "Epoch 10/10\n",
      "164/164 [==============================] - 174s 1s/step - loss: 0.0659 - accuracy: 0.9830 - val_loss: 0.7843 - val_accuracy: 0.8321\n",
      "782/782 [==============================] - 206s 262ms/step - loss: 0.4301 - accuracy: 0.8158\n",
      "Test acc: 0.816\n"
     ]
    }
   ],
   "source": [
    "inputs   = keras.Input(shape=(None,), dtype=\"int64\")\r\n",
    "embedded = layers.Embedding(input_dim=max_tokens, output_dim=256)(inputs)\r\n",
    "x        = layers.Bidirectional(layers.LSTM(32))(embedded)\r\n",
    "x        = layers.Dropout(0.5)(x)\r\n",
    "outputs  = layers.Dense(1, activation=\"sigmoid\")(x)\r\n",
    "model    = keras.Model(inputs, outputs)\r\n",
    "\r\n",
    "model.compile(optimizer=\"rmsprop\",\r\n",
    "              loss=\"binary_crossentropy\",\r\n",
    "              metrics=[\"accuracy\"])\r\n",
    "\r\n",
    "model.summary()\r\n",
    "\r\n",
    "callbacks = [\r\n",
    "    keras.callbacks.ModelCheckpoint(\"embeddings_bidir_gru.keras\",\r\n",
    "                                    save_best_only=True)\r\n",
    "]\r\n",
    "\r\n",
    "model.fit(int_train_ds, validation_data=int_val_ds, epochs=10, callbacks=callbacks)\r\n",
    "\r\n",
    "model = keras.models.load_model(\"embeddings_bidir_gru.keras\")\r\n",
    "\r\n",
    "print(f\"Test acc: {model.evaluate(int_test_ds)[1]:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4理解填充和掩码"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### [C] 11.17 使用带有掩码的 `Embedding` 层"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs   = keras.Input(shape=(None,), dtype=\"int64\")\r\n",
    "embedded = layers.Embedding(input_dim=max_tokens, output_dim=256, mask_zero=True)(inputs)\r\n",
    "x        = layers.Bidirectional(layers.LSTM(32))(embedded)\r\n",
    "x        = layers.Dropout(0.5)(x)\r\n",
    "outputs  = layers.Dense(1, activation=\"sigmoid\")(x)\r\n",
    "model    = keras.Model(inputs, outputs)\r\n",
    "\r\n",
    "model.compile(optimizer=\"rmsprop\",\r\n",
    "              loss=\"binary_crossentropy\",\r\n",
    "              metrics=[\"accuracy\"])\r\n",
    "\r\n",
    "model.summary()\r\n",
    "\r\n",
    "callbacks = [\r\n",
    "    keras.callbacks.ModelCheckpoint(\"embeddings_bidir_gru_with_masking.keras\",\r\n",
    "                                    save_best_only=True)\r\n",
    "]\r\n",
    "\r\n",
    "model.fit(int_train_ds, validation_data=int_val_ds, epochs=10, callbacks=callbacks)\r\n",
    "\r\n",
    "model = keras.models.load_model(\"embeddings_bidir_gru_with_masking.keras\")\r\n",
    "\r\n",
    "print(f\"Test acc: {model.evaluate(int_test_ds)[1]:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5 使用预训练词嵌入"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### [C] 11.18 解析 `GloVe` 词嵌入文件\r\n",
    "---\r\n",
    "`!wget http://nlp.stanford.edu/data/glove.6B.zip`\r\n",
    "\r\n",
    "`!unzip -q glove.6B.zip`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 400000 word vectors.\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\r\n",
    "\r\n",
    "glove_dir          = pathlib.Path('glove')\r\n",
    "path_to_glove_file = os.path.join(glove_dir, \"glove.6B.100d.txt\")\r\n",
    "\r\n",
    "embeddings_index = {}\r\n",
    "\r\n",
    "with open(path_to_glove_file, encoding='utf-8') as f:\r\n",
    "    for line in f:\r\n",
    "        word, coefs = line.split(maxsplit=1)\r\n",
    "        coefs = np.fromstring(coefs, \"f\", sep=\" \")\r\n",
    "        embeddings_index[word] = coefs\r\n",
    "\r\n",
    "print(f\"Found {len(embeddings_index)} word vectors.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### [C] 11.19 准备 `GloVe` 词嵌入矩阵"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding_dim = 100\r\n",
    "\r\n",
    "vocabulary = text_vectorization.get_vocabulary()            # 获取前面 TextVectorization 层索引的词表\r\n",
    "word_index = dict(zip(vocabulary, range(len(vocabulary))))  # 利用这个词表创建一个从单词到其词表索引的映射\r\n",
    "\r\n",
    "embedding_matrix = np.zeros((max_tokens, embedding_dim))    # 准备一个矩阵，后续将用 GloVe 向量填充\r\n",
    "\r\n",
    "for word, i in word_index.items():\r\n",
    "    if i < max_tokens:\r\n",
    "        embedding_vector = embeddings_index.get(word)\r\n",
    "\r\n",
    "    if embedding_vector is not None:\r\n",
    "        embedding_matrix[i] = embedding_vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding_layer = layers.Embedding(\r\n",
    "    max_tokens,\r\n",
    "    embedding_dim,\r\n",
    "    embeddings_initializer=keras.initializers.Constant(embedding_matrix),\r\n",
    "    trainable=False,  # 冻结\r\n",
    "    mask_zero=True,\r\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### [C] 11.20 使用预训练 `Embedding` 层的模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_6\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " input_7 (InputLayer)        [(None, None)]            0         \n",
      "                                                                 \n",
      " embedding_3 (Embedding)     (None, None, 100)         2000000   \n",
      "                                                                 \n",
      " bidirectional_3 (Bidirectio  (None, 64)               34048     \n",
      " nal)                                                            \n",
      "                                                                 \n",
      " dropout_5 (Dropout)         (None, 64)                0         \n",
      "                                                                 \n",
      " dense_7 (Dense)             (None, 1)                 65        \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 2,034,113\n",
      "Trainable params: 34,113\n",
      "Non-trainable params: 2,000,000\n",
      "_________________________________________________________________\n",
      "Epoch 1/10\n",
      "164/164 [==============================] - 57s 309ms/step - loss: 0.6622 - accuracy: 0.5952 - val_loss: 0.6802 - val_accuracy: 0.5611\n",
      "Epoch 2/10\n",
      "164/164 [==============================] - 68s 415ms/step - loss: 0.5792 - accuracy: 0.7052 - val_loss: 0.5055 - val_accuracy: 0.7641\n",
      "Epoch 3/10\n",
      "164/164 [==============================] - 77s 471ms/step - loss: 0.5335 - accuracy: 0.7353 - val_loss: 0.5076 - val_accuracy: 0.7542\n",
      "Epoch 4/10\n",
      "164/164 [==============================] - 77s 471ms/step - loss: 0.4960 - accuracy: 0.7674 - val_loss: 0.5753 - val_accuracy: 0.7244\n",
      "Epoch 5/10\n",
      "164/164 [==============================] - 72s 437ms/step - loss: 0.4567 - accuracy: 0.7942 - val_loss: 0.4457 - val_accuracy: 0.7901\n",
      "Epoch 6/10\n",
      "164/164 [==============================] - 63s 386ms/step - loss: 0.4297 - accuracy: 0.8095 - val_loss: 0.4223 - val_accuracy: 0.8023\n",
      "Epoch 7/10\n",
      "164/164 [==============================] - 73s 446ms/step - loss: 0.4045 - accuracy: 0.8213 - val_loss: 0.4218 - val_accuracy: 0.8008\n",
      "Epoch 8/10\n",
      "164/164 [==============================] - 71s 434ms/step - loss: 0.3846 - accuracy: 0.8322 - val_loss: 0.4023 - val_accuracy: 0.8084\n",
      "Epoch 9/10\n",
      "164/164 [==============================] - 73s 448ms/step - loss: 0.3619 - accuracy: 0.8497 - val_loss: 0.4027 - val_accuracy: 0.8130\n",
      "Epoch 10/10\n",
      "164/164 [==============================] - 72s 441ms/step - loss: 0.3459 - accuracy: 0.8486 - val_loss: 0.3916 - val_accuracy: 0.8176\n",
      "782/782 [==============================] - 85s 105ms/step - loss: 0.3851 - accuracy: 0.8265\n",
      "Test acc: 0.827\n"
     ]
    }
   ],
   "source": [
    "inputs   = keras.Input(shape=(None,), dtype=\"int64\")\r\n",
    "embedded = embedding_layer(inputs)\r\n",
    "x        = layers.Bidirectional(layers.LSTM(32))(embedded)\r\n",
    "x        = layers.Dropout(0.5)(x)\r\n",
    "outputs  = layers.Dense(1, activation=\"sigmoid\")(x)\r\n",
    "model    = keras.Model(inputs, outputs)\r\n",
    "\r\n",
    "model.compile(optimizer=\"rmsprop\",\r\n",
    "              loss=\"binary_crossentropy\",\r\n",
    "              metrics=[\"accuracy\"])\r\n",
    "\r\n",
    "model.summary()\r\n",
    "\r\n",
    "callbacks = [\r\n",
    "    keras.callbacks.ModelCheckpoint(\"glove_embeddings_sequence_model.keras\",\r\n",
    "                                    save_best_only=True)\r\n",
    "]\r\n",
    "\r\n",
    "model.fit(int_train_ds, validation_data=int_val_ds, epochs=10, callbacks=callbacks)\r\n",
    "\r\n",
    "model = keras.models.load_model(\"glove_embeddings_sequence_model.keras\")\r\n",
    "\r\n",
    "print(f\"Test acc: {model.evaluate(int_test_ds)[1]:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 11.4 **`Transformer` 架构**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 11.4.1 理解自注意力\r\n",
    "---\r\n",
    "首先要对一组特征计算重要性分数。特征相关性越大，分数越高，反之。\r\n",
    "\r\n",
    "---\r\n",
    "你有一个参考序列，用于描述你要查找的内容：`查询`。\r\n",
    "\r\n",
    "你有一个知识体系，并试图从中提取信息：`值`。\r\n",
    "\r\n",
    "每个值都有一个`键`，用于描述这个值，并可以很容易于查询进行对比。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 11.4.2 多头注意力 `Attention Is All You Need`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 11.4.3 `Transformer` 编码器"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### [C] 11.21 将 `Transformer` 编码器实现为 `Layer` 子类"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\r\n",
    "from tensorflow import keras\r\n",
    "from tensorflow.keras import layers\r\n",
    "\r\n",
    "class TransformerEncoder(layers.Layer):\r\n",
    "\r\n",
    "    def __init__(self, embed_dim, dense_dim, num_heads, **kwargs):\r\n",
    "        \r\n",
    "        super().__init__(**kwargs)\r\n",
    "\r\n",
    "        self.embed_dim = embed_dim  # 输入词元向量的尺寸\r\n",
    "        self.dense_dim = dense_dim  # 内部密集层的尺寸\r\n",
    "        self.num_heads = num_heads  # 注意力头的个数\r\n",
    "\r\n",
    "        self.attention  = layers.MultiHeadAttention(num_heads=num_heads, key_dim=embed_dim)\r\n",
    "        self.dense_proj = keras.Sequential([layers.Dense(dense_dim, activation='relu'), layers.Dense(embed_dim), ])\r\n",
    "\r\n",
    "        self.layernorm_1 = layers.LayerNormalization()\r\n",
    "        self.layernorm_2 = layers.LayerNormalization()\r\n",
    "\r\n",
    "    def call(self, inputs, mask=None):\r\n",
    "        # Embedding 层生成的掩码是二维的，但注意力层的输入应该是三维或四维的，所以我们需要增加它的维数\r\n",
    "        if mask is not None:\r\n",
    "            mask = mask[:, tf.newaxis, :]\r\n",
    "        \r\n",
    "        attention_output = self.attention(inputs, inputs, attention_mask=mask)\r\n",
    "        proj_input       = self.layernorm_1(inputs + attention_output)\r\n",
    "        proj_output      = self.dense_proj(proj_input)\r\n",
    "\r\n",
    "        return self.layernorm_2(proj_input + proj_output)\r\n",
    "    \r\n",
    "    # 实现序列化，以便保存模型\r\n",
    "    def get_config(self):\r\n",
    "        config = super().get_config()\r\n",
    "\r\n",
    "        config.update({\r\n",
    "            'embed_dim':self.embed_dim,\r\n",
    "            'num_heads':self.num_heads,\r\n",
    "            'dense_dim':self.dense_dim,\r\n",
    "        })\r\n",
    "\r\n",
    "        return config"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### [C] 11.22 将 `Transformer` 编码器用于文本分类"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"transformerEncode\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " input_1 (InputLayer)        [(None, None)]            0         \n",
      "                                                                 \n",
      " embedding (Embedding)       (None, None, 256)         5120000   \n",
      "                                                                 \n",
      " transformer_encoder (Transf  (None, None, 256)        543776    \n",
      " ormerEncoder)                                                   \n",
      "                                                                 \n",
      " global_max_pooling1d (Globa  (None, 256)              0         \n",
      " lMaxPooling1D)                                                  \n",
      "                                                                 \n",
      " dropout (Dropout)           (None, 256)               0         \n",
      "                                                                 \n",
      " dense_2 (Dense)             (None, 1)                 257       \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 5,664,033\n",
      "Trainable params: 5,664,033\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "vocab_size = 20000\r\n",
    "embed_dim  = 256\r\n",
    "num_heads  = 2\r\n",
    "dense_dim  = 32\r\n",
    "\r\n",
    "inputs  = keras.Input(shape=(None,), dtype=\"int64\")\r\n",
    "x       = layers.Embedding(vocab_size, embed_dim)(inputs)\r\n",
    "x       = TransformerEncoder(embed_dim, dense_dim, num_heads)(x)\r\n",
    "x       = layers.GlobalMaxPooling1D()(x)\r\n",
    "x       = layers.Dropout(0.5)(x)\r\n",
    "outputs = layers.Dense(1, activation=\"sigmoid\")(x)\r\n",
    "model   = keras.Model(inputs, outputs, name='transformerEncode')\r\n",
    "\r\n",
    "model.compile(optimizer=\"rmsprop\",\r\n",
    "              loss=\"binary_crossentropy\",\r\n",
    "              metrics=[\"accuracy\"])\r\n",
    "\r\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### [C] 11.23 训练并评估基于 `Transformer` 编码器的模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "164/164 [==============================] - 318s 2s/step - loss: 0.7069 - accuracy: 0.6566 - val_loss: 0.5090 - val_accuracy: 0.7519\n",
      "Epoch 2/20\n",
      "164/164 [==============================] - 342s 2s/step - loss: 0.4082 - accuracy: 0.8160 - val_loss: 0.3504 - val_accuracy: 0.8359\n",
      "Epoch 3/20\n",
      "164/164 [==============================] - 348s 2s/step - loss: 0.3309 - accuracy: 0.8596 - val_loss: 0.3380 - val_accuracy: 0.8389\n",
      "Epoch 4/20\n",
      "164/164 [==============================] - 382s 2s/step - loss: 0.2722 - accuracy: 0.8883 - val_loss: 0.3300 - val_accuracy: 0.8557\n",
      "Epoch 5/20\n",
      "164/164 [==============================] - 330s 2s/step - loss: 0.2383 - accuracy: 0.9045 - val_loss: 0.4107 - val_accuracy: 0.8221\n",
      "Epoch 6/20\n",
      "164/164 [==============================] - 329s 2s/step - loss: 0.1862 - accuracy: 0.9275 - val_loss: 0.3514 - val_accuracy: 0.8496\n",
      "Epoch 7/20\n",
      "164/164 [==============================] - 324s 2s/step - loss: 0.1566 - accuracy: 0.9413 - val_loss: 0.3375 - val_accuracy: 0.8611\n",
      "Epoch 8/20\n",
      "164/164 [==============================] - 328s 2s/step - loss: 0.1320 - accuracy: 0.9512 - val_loss: 0.3614 - val_accuracy: 0.8550\n",
      "Epoch 9/20\n",
      "164/164 [==============================] - 337s 2s/step - loss: 0.1010 - accuracy: 0.9626 - val_loss: 0.3757 - val_accuracy: 0.8618\n",
      "Epoch 10/20\n",
      "164/164 [==============================] - 309s 2s/step - loss: 0.0788 - accuracy: 0.9703 - val_loss: 0.3831 - val_accuracy: 0.8641\n",
      "Epoch 11/20\n",
      "164/164 [==============================] - 300s 2s/step - loss: 0.0695 - accuracy: 0.9767 - val_loss: 0.4313 - val_accuracy: 0.8496\n",
      "Epoch 12/20\n",
      "164/164 [==============================] - 305s 2s/step - loss: 0.0524 - accuracy: 0.9842 - val_loss: 0.4400 - val_accuracy: 0.8420\n",
      "Epoch 13/20\n",
      "164/164 [==============================] - 301s 2s/step - loss: 0.0405 - accuracy: 0.9868 - val_loss: 0.4429 - val_accuracy: 0.8542\n",
      "Epoch 14/20\n",
      "164/164 [==============================] - 308s 2s/step - loss: 0.0313 - accuracy: 0.9907 - val_loss: 0.4577 - val_accuracy: 0.8511\n",
      "Epoch 15/20\n",
      "164/164 [==============================] - 303s 2s/step - loss: 0.0276 - accuracy: 0.9905 - val_loss: 0.4845 - val_accuracy: 0.8542\n",
      "Epoch 16/20\n",
      "164/164 [==============================] - 310s 2s/step - loss: 0.0240 - accuracy: 0.9910 - val_loss: 0.4998 - val_accuracy: 0.8473\n",
      "Epoch 17/20\n",
      "164/164 [==============================] - 307s 2s/step - loss: 0.0211 - accuracy: 0.9926 - val_loss: 0.5273 - val_accuracy: 0.8466\n",
      "Epoch 18/20\n",
      "164/164 [==============================] - 304s 2s/step - loss: 0.0175 - accuracy: 0.9949 - val_loss: 0.5411 - val_accuracy: 0.8435\n",
      "Epoch 19/20\n",
      "164/164 [==============================] - 333s 2s/step - loss: 0.0124 - accuracy: 0.9968 - val_loss: 0.5824 - val_accuracy: 0.8519\n",
      "Epoch 20/20\n",
      "164/164 [==============================] - 345s 2s/step - loss: 0.0093 - accuracy: 0.9977 - val_loss: 0.6220 - val_accuracy: 0.8427\n",
      "782/782 [==============================] - 494s 632ms/step - loss: 0.3622 - accuracy: 0.8441\n",
      "Test acc: 0.844\n"
     ]
    }
   ],
   "source": [
    "callbacks = [\r\n",
    "    keras.callbacks.ModelCheckpoint(\"transformer_encoder.keras\",\r\n",
    "                                    save_best_only=True)\r\n",
    "]\r\n",
    "\r\n",
    "model.fit(int_train_ds, validation_data=int_val_ds, epochs=20, callbacks=callbacks)\r\n",
    "\r\n",
    "model = keras.models.load_model(\r\n",
    "    \"transformer_encoder.keras\",\r\n",
    "    custom_objects={\"TransformerEncoder\": TransformerEncoder})\r\n",
    "\r\n",
    "print(f\"Test acc: {model.evaluate(int_test_ds)[1]:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1 使用位置编码重新注入顺序信息"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### [C] 11.24 将位置嵌入实现为 `Layer` 子类"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "class PositionalEmbedding(layers.Layer):\r\n",
    "    def __init__(self, sequence_length, input_dim, output_dim, **kwargs):\r\n",
    "        super().__init__(**kwargs)\r\n",
    "        \r\n",
    "        self.token_embeddings    = layers.Embedding(input_dim=input_dim      , output_dim=output_dim)  # 用于保存词元索引\r\n",
    "        self.position_embeddings = layers.Embedding(input_dim=sequence_length, output_dim=output_dim)  # 用于保存词元位置\r\n",
    "        self.sequence_length     = sequence_length\r\n",
    "        self.input_dim           = input_dim\r\n",
    "        self.output_dim          = output_dim\r\n",
    "\r\n",
    "    def call(self, inputs):\r\n",
    "        length             = tf.shape(inputs)[-1]\r\n",
    "        positions          = tf.range(start=0, limit=length, delta=1)\r\n",
    "        embedded_tokens    = self.token_embeddings(inputs)\r\n",
    "        embedded_positions = self.position_embeddings(positions)\r\n",
    "        \r\n",
    "        return embedded_tokens + embedded_positions\r\n",
    "\r\n",
    "    def compute_mask(self, inputs, mask=None):\r\n",
    "        \r\n",
    "        return tf.math.not_equal(inputs, 0)\r\n",
    "\r\n",
    "    # 实现序列化，以便保存模型\r\n",
    "    def get_config(self):\r\n",
    "        config = super().get_config()\r\n",
    "        \r\n",
    "        config.update({\r\n",
    "            \"output_dim\"     : self.output_dim,\r\n",
    "            \"sequence_length\": self.sequence_length,\r\n",
    "            \"input_dim\"      : self.input_dim,\r\n",
    "        })\r\n",
    "\r\n",
    "        return config"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2 综合示例:文本分类 `Transformer`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### [C] 11.25 将 `Transformer` 编码器与位置嵌入相结合"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_7\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " input_9 (InputLayer)        [(None, None)]            0         \n",
      "                                                                 \n",
      " positional_embedding (Posit  (None, None, 256)        5273600   \n",
      " ionalEmbedding)                                                 \n",
      "                                                                 \n",
      " transformer_encoder_1 (Tran  (None, None, 256)        543776    \n",
      " sformerEncoder)                                                 \n",
      "                                                                 \n",
      " global_max_pooling1d_1 (Glo  (None, 256)              0         \n",
      " balMaxPooling1D)                                                \n",
      "                                                                 \n",
      " dropout_7 (Dropout)         (None, 256)               0         \n",
      "                                                                 \n",
      " dense_15 (Dense)            (None, 1)                 257       \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 5,817,633\n",
      "Trainable params: 5,817,633\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "164/164 [==============================] - 307s 2s/step - loss: 0.8475 - accuracy: 0.5711 - val_loss: 0.5021 - val_accuracy: 0.7565\n",
      "782/782 [==============================] - 541s 691ms/step - loss: 0.4904 - accuracy: 0.7766\n",
      "Test acc: 0.777\n"
     ]
    }
   ],
   "source": [
    "vocab_size      = 20000\r\n",
    "sequence_length = 600\r\n",
    "embed_dim       = 256\r\n",
    "num_heads       = 2\r\n",
    "dense_dim       = 32\r\n",
    "\r\n",
    "inputs  = keras.Input(shape=(None,), dtype=\"int64\")\r\n",
    "x       = PositionalEmbedding(sequence_length, vocab_size, embed_dim)(inputs)\r\n",
    "x       = TransformerEncoder(embed_dim, dense_dim, num_heads)(x)\r\n",
    "x       = layers.GlobalMaxPooling1D()(x)\r\n",
    "x       = layers.Dropout(0.5)(x)\r\n",
    "outputs = layers.Dense(1, activation=\"sigmoid\")(x)\r\n",
    "model   = keras.Model(inputs, outputs)\r\n",
    "\r\n",
    "model.compile(optimizer=\"rmsprop\",\r\n",
    "              loss=\"binary_crossentropy\",\r\n",
    "              metrics=[\"accuracy\"])\r\n",
    "\r\n",
    "model.summary()\r\n",
    "\r\n",
    "callbacks = [\r\n",
    "    keras.callbacks.ModelCheckpoint(\"full_transformer_encoder.keras\",\r\n",
    "                                    save_best_only=True)\r\n",
    "]\r\n",
    "\r\n",
    "model.fit(int_train_ds, validation_data=int_val_ds, epochs=1, callbacks=callbacks)\r\n",
    "\r\n",
    "model = keras.models.load_model(\r\n",
    "    \"full_transformer_encoder.keras\",\r\n",
    "    custom_objects={\"TransformerEncoder\": TransformerEncoder,\r\n",
    "                    \"PositionalEmbedding\": PositionalEmbedding})\r\n",
    "\r\n",
    "print(f\"Test acc: {model.evaluate(int_test_ds)[1]:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 11.4.4 何时使用序列模型而不是词袋模型\r\n",
    "### **When to use sequence models over bag-of-words models?**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 11.5 超越文本分类:序列到序列学习\r\n",
    "---\r\n",
    "* 机器翻译(`machine translation`)\r\n",
    "* 文本摘要(`text summarization`)\r\n",
    "* 问题答(`question answering`)\r\n",
    "* 聊天机器人(`chatbot`)\r\n",
    "* 文本生成(`text generation`)\r\n",
    "---\r\n",
    "训练过程：\r\n",
    "- **编码器**模型将源序列转换为中间表示。\r\n",
    "- 对**解码器**进行训练，使其可以通过查看前面的词元(从 `0` 到 `i-1` )和编码后的源序列，预测目标序列的下一个词元 `i`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 11.5.1 机器翻译示例"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 英语到西班牙语的翻译数据集\r\n",
    "`!wget http://storage.googleapis.com/download.tensorflow.org/data/spa-eng.zip`\r\n",
    "\r\n",
    "`!unzip -q spa-eng.zip`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "text_file = 'spa-eng/spa.txt'\r\n",
    "\r\n",
    "with open(text_file, encoding='utf-8') as f:\r\n",
    "    lines = f.read().split('\\n')[:-1]\r\n",
    "\r\n",
    "text_pairs = []\r\n",
    "# 对文件中每一行进行遍历\r\n",
    "for line in lines:\r\n",
    "    # 每一行都包含一个英语句子和它的西班牙译文，二者以制表符分隔\r\n",
    "    english, spanish = line.split('\\t')\r\n",
    "    \r\n",
    "    # 将[start]和[end]分别添加到西班牙语句子的开头和结尾\r\n",
    "    spanish = '[start]' + spanish + '[end]'\r\n",
    "\r\n",
    "    text_pairs.append((english, spanish))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('My new class starts today.', '[start]Mi nuevo curso comienza hoy.[end]')\n"
     ]
    }
   ],
   "source": [
    "import random\r\n",
    "\r\n",
    "# 显示text_pairs示例\r\n",
    "print(random.choice(text_pairs))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 将text_pairs打乱, 并将其划分为常见的训练集、验证集和测试集\r\n",
    "import random\r\n",
    "\r\n",
    "random.shuffle(text_pairs)\r\n",
    "\r\n",
    "num_val_samples   = int(0.15 * len(text_pairs))\r\n",
    "num_train_samples = len(text_pairs) - 2 * num_val_samples\r\n",
    "\r\n",
    "train_pairs = text_pairs[:num_train_samples]\r\n",
    "val_pairs   = text_pairs[num_train_samples:num_train_samples + num_val_samples]\r\n",
    "test_pairs  = text_pairs[num_train_samples + num_val_samples:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 准备两个单独的 `TextVectorization` 层:一个用于英语,一个用于西班牙语.\r\n",
    "---\r\n",
    "* 需要保留插入的词元`[start]`和`[end]`\r\n",
    "* 不同语言的标点符号是不同的"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### [C] 11.26 将英语和西班牙语的文本对向量化"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\r\n",
    "import string\r\n",
    "import re\r\n",
    "\r\n",
    "strip_chars = string.punctuation + '¿'\r\n",
    "strip_chars = strip_chars.replace('[', '')\r\n",
    "strip_chars = strip_chars.replace(']', '')\r\n",
    "\r\n",
    "def custom_standardization(input_string):\r\n",
    "\r\n",
    "    lowercase = tf.strings.lower(input_string)\r\n",
    "\r\n",
    "    return tf.strings.regex_replace(lowercase, f'[{re.escape(strip_chars)}]', '')\r\n",
    "\r\n",
    "vocab_size      = 15000\r\n",
    "sequence_length = 20\r\n",
    "\r\n",
    "# 英语层\r\n",
    "source_vectorization = layers.TextVectorization(\r\n",
    "    max_tokens=vocab_size,\r\n",
    "    output_mode=\"int\",\r\n",
    "    output_sequence_length=sequence_length,\r\n",
    ")\r\n",
    "\r\n",
    "# 西班牙语层\r\n",
    "target_vectorization = layers.TextVectorization(\r\n",
    "    max_tokens=vocab_size,\r\n",
    "    output_mode=\"int\",\r\n",
    "    output_sequence_length=sequence_length + 1,  # 在训练过程中需要将句子偏移一个时间步\r\n",
    "    standardize=custom_standardization,\r\n",
    ")\r\n",
    "\r\n",
    "train_english_texts = [pair[0] for pair in train_pairs]\r\n",
    "train_spanish_texts = [pair[1] for pair in train_pairs]\r\n",
    "\r\n",
    "# 学习每种语言的词表\r\n",
    "source_vectorization.adapt(train_english_texts)\r\n",
    "target_vectorization.adapt(train_spanish_texts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### [C] 11.27 准备翻译任务的数据集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 64\r\n",
    "\r\n",
    "def format_dataset(eng, spa):\r\n",
    "    \r\n",
    "    eng = source_vectorization(eng)\r\n",
    "    spa = target_vectorization(spa)\r\n",
    "\r\n",
    "    return ({\r\n",
    "        'english':eng,\r\n",
    "        'spanish':spa[:, :-1],  # 输入西班牙语句子不包含最后一个词元，以保证输入和目标具有相同的场地\r\n",
    "    }, \r\n",
    "    spa[:, 1:]  # 目标西班牙语句子后偏移一个时间步。二者长度相同，都是20个单词\r\n",
    "    )\r\n",
    "\r\n",
    "def make_dataset(pairs):\r\n",
    "    \r\n",
    "    eng_texts, spa_texts = zip(*pairs)\r\n",
    "    \r\n",
    "    eng_texts = list(eng_texts)\r\n",
    "    spa_texts = list(spa_texts)\r\n",
    "\r\n",
    "    dataset = tf.data.Dataset.from_tensor_slices((eng_texts, spa_texts))\r\n",
    "    dataset = dataset.batch(batch_size)\r\n",
    "    dataset = dataset.map(format_dataset, num_parallel_calls=4)\r\n",
    "\r\n",
    "    return dataset.shuffle(2048).prefetch(16).cache()  # 利用内存缓存来加快预处理速度\r\n",
    "\r\n",
    "train_ds = make_dataset(train_pairs)\r\n",
    "val_ds   = make_dataset(val_pairs)\r\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "inputs['english'].shape: (64, 20)\n",
      "inputs['spanish'].shape: (64, 20)\n",
      "targets.shape: (64, 20)\n"
     ]
    }
   ],
   "source": [
    "for inputs, targets in train_ds.take(1):\r\n",
    "    print(f\"inputs['english'].shape: {inputs['english'].shape}\")\r\n",
    "    print(f\"inputs['spanish'].shape: {inputs['spanish'].shape}\")\r\n",
    "    print(f\"targets.shape: {targets.shape}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 11.5.2 `RNN` 的序列到序列学习"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 使用`RNN`将一个序列转换到另一个序列，最简单的方法是在每个时间步都保存`RNN`的输出\r\n",
    "---\r\n",
    "有两个主要问题：\r\n",
    "* 目标序列必须始终与源序列的长度相同。\r\n",
    "* 由于`RNN`逐步处理的性质，模型将仅通过查看源序列第`0~N`个词元来预测目标序列的第`N`个词元。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs  = keras.Input(shape=(sequence_length,), dtype='int64')\r\n",
    "x       = layers.Embedding(input_dim=vocab_size, output_dim=128)(inputs)\r\n",
    "x       = layers.LSTM(32, return_sequences=True)(x)\r\n",
    "outputs = layers.Dense(vocab_size, activation='softmax')(x)\r\n",
    "model   = keras.Model(inputs, outputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### [C] 11.28 基于`GRU`的编码器"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow import keras\r\n",
    "from tensorflow.keras import layers\r\n",
    "\r\n",
    "embed_dim  = 256\r\n",
    "latent_dim = 1024\r\n",
    "\r\n",
    "# 掩码\r\n",
    "source = keras.Input(shape=(None,), dtype='int64', name='english')\r\n",
    "# 英语源句子。指定输入名称，我们就可以用输入组成的字典来拟合模型\r\n",
    "x      = layers.Embedding(vocab_size, embed_dim, mask_zero=True)(source)\r\n",
    "# 编码后的源句子即为双向GRU的最后一个食醋\r\n",
    "encoded_source = layers.Bidirectional(layers.GRU(latent_dim), merge_mode='sum')(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### [C] 11.29 基于`GRU`的解码器与端到端模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "past_target      = keras.Input(shape=(None,), dtype=\"int64\", name=\"spanish\")             # 西班牙语目标的句子\r\n",
    "x                = layers.Embedding(vocab_size, embed_dim, mask_zero=True)(past_target)  # 掩码\r\n",
    "decoder_gru      = layers.GRU(latent_dim, return_sequences=True)\r\n",
    "x                = decoder_gru(x, initial_state=encoded_source)                          # 编码后的源句子作为解码器GRU的初始状态\r\n",
    "x                = layers.Dropout(0.5)(x)\r\n",
    "target_next_step = layers.Dense(vocab_size, activation=\"softmax\")(x)                     # 预测下一个词元\r\n",
    "seq2seq_rnn      = keras.Model([source, past_target], target_next_step)                  # 端到端模型：将源句子和目标句子映射为偏移一个时间步的目标句子"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### [C] 11.30 训练序列到序列循环模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1302/1302 [==============================] - 3236s 2s/step - loss: 5.4389 - accuracy: 0.2327 - val_loss: 4.5894 - val_accuracy: 0.3054\n"
     ]
    },
    {
     "data": {
      "text/plain": "<keras.callbacks.History at 0x20001bf58d0>"
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "seq2seq_rnn.compile(optimizer='rmsprop', loss='sparse_categorical_crossentropy', metrics=['accuracy'])\r\n",
    "\r\n",
    "seq2seq_rnn.fit(train_ds, epochs=1, validation_data=val_ds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### [C] 11.31 利用 `RNN` 编码器和 `RNN` 解码器来翻译句子"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-\n",
      "She's hot.\n",
      "1/1 [==============================] - 3s 3s/step\n",
      "1/1 [==============================] - 0s 111ms/step\n",
      "1/1 [==============================] - 0s 107ms/step\n",
      "1/1 [==============================] - 0s 104ms/step\n",
      "1/1 [==============================] - 0s 110ms/step\n",
      "1/1 [==============================] - 0s 121ms/step\n",
      "1/1 [==============================] - 0s 110ms/step\n",
      "1/1 [==============================] - 0s 107ms/step\n",
      "1/1 [==============================] - 0s 102ms/step\n",
      "1/1 [==============================] - 0s 108ms/step\n",
      "1/1 [==============================] - 0s 109ms/step\n",
      "1/1 [==============================] - 0s 97ms/step\n",
      "1/1 [==============================] - 0s 103ms/step\n",
      "1/1 [==============================] - 0s 89ms/step\n",
      "1/1 [==============================] - 0s 100ms/step\n",
      "1/1 [==============================] - 0s 92ms/step\n",
      "1/1 [==============================] - 0s 96ms/step\n",
      "1/1 [==============================] - 0s 97ms/step\n",
      "1/1 [==============================] - 0s 99ms/step\n",
      "1/1 [==============================] - 0s 92ms/step\n",
      "[start] [start]mañana volverás mantenido [start]actualmente demostrar elevadora[end] conducir tarde [start]voy aburres[end] madrid empresas empresas siendo momentos[end] morado[end] permito limpias ganarse velocidad\n",
      "-\n",
      "This milk has a peculiar smell.\n",
      "1/1 [==============================] - 0s 113ms/step\n",
      "1/1 [==============================] - 0s 93ms/step\n",
      "1/1 [==============================] - 0s 90ms/step\n",
      "1/1 [==============================] - 0s 91ms/step\n",
      "1/1 [==============================] - 0s 90ms/step\n",
      "1/1 [==============================] - 0s 91ms/step\n",
      "1/1 [==============================] - 0s 91ms/step\n",
      "1/1 [==============================] - 0s 88ms/step\n",
      "1/1 [==============================] - 0s 89ms/step\n",
      "1/1 [==============================] - 0s 92ms/step\n",
      "1/1 [==============================] - 0s 91ms/step\n",
      "1/1 [==============================] - 0s 88ms/step\n",
      "1/1 [==============================] - 0s 97ms/step\n",
      "1/1 [==============================] - 0s 90ms/step\n",
      "1/1 [==============================] - 0s 97ms/step\n",
      "1/1 [==============================] - 0s 94ms/step\n",
      "1/1 [==============================] - 0s 91ms/step\n",
      "1/1 [==============================] - 0s 104ms/step\n",
      "1/1 [==============================] - 0s 80ms/step\n",
      "1/1 [==============================] - 0s 82ms/step\n",
      "[start] [start]roma pide resfría [start]deseas presencia[end] cuándo[end] esperas corriendo descubrieron cambios[end] bella[end] valla[end] [start]comimos lentamente repararlo[end] optimista[end] tenedor[end] pude fbi[end] [start]terminamos\n",
      "-\n",
      "They are going to meet at the hotel tomorrow.\n",
      "1/1 [==============================] - 0s 87ms/step\n",
      "1/1 [==============================] - 0s 82ms/step\n",
      "1/1 [==============================] - 0s 81ms/step\n",
      "1/1 [==============================] - 0s 80ms/step\n",
      "1/1 [==============================] - 0s 81ms/step\n",
      "1/1 [==============================] - 0s 76ms/step\n",
      "1/1 [==============================] - 0s 79ms/step\n",
      "1/1 [==============================] - 0s 77ms/step\n",
      "1/1 [==============================] - 0s 77ms/step\n",
      "1/1 [==============================] - 0s 77ms/step\n",
      "1/1 [==============================] - 0s 75ms/step\n",
      "1/1 [==============================] - 0s 74ms/step\n",
      "1/1 [==============================] - 0s 76ms/step\n",
      "1/1 [==============================] - 0s 75ms/step\n",
      "1/1 [==============================] - 0s 75ms/step\n",
      "1/1 [==============================] - 0s 76ms/step\n",
      "1/1 [==============================] - 0s 75ms/step\n",
      "1/1 [==============================] - 0s 78ms/step\n",
      "1/1 [==============================] - 0s 78ms/step\n",
      "1/1 [==============================] - 0s 78ms/step\n",
      "[start] vendiendo base velozmente[end] twitter[end] relativamente [start]pensé ¡qué único[end] violento[end] primeros tranquilo[end] llegamos quedó[end] usted[end] satélites limpiaste predecir intenté[end] pulgar camino[end]\n",
      "-\n",
      "What's your answer?\n",
      "1/1 [==============================] - 0s 90ms/step\n",
      "1/1 [==============================] - 0s 94ms/step\n",
      "1/1 [==============================] - 0s 75ms/step\n",
      "1/1 [==============================] - 0s 78ms/step\n",
      "1/1 [==============================] - 0s 75ms/step\n",
      "1/1 [==============================] - 0s 123ms/step\n",
      "1/1 [==============================] - 0s 76ms/step\n",
      "1/1 [==============================] - 0s 80ms/step\n",
      "1/1 [==============================] - 0s 81ms/step\n",
      "1/1 [==============================] - 0s 133ms/step\n",
      "1/1 [==============================] - 0s 87ms/step\n",
      "1/1 [==============================] - 0s 77ms/step\n",
      "1/1 [==============================] - 0s 72ms/step\n",
      "1/1 [==============================] - 0s 76ms/step\n",
      "1/1 [==============================] - 0s 64ms/step\n",
      "1/1 [==============================] - 0s 100ms/step\n",
      "1/1 [==============================] - 0s 117ms/step\n",
      "1/1 [==============================] - 0s 82ms/step\n",
      "1/1 [==============================] - 0s 76ms/step\n",
      "1/1 [==============================] - 0s 77ms/step\n",
      "[start] vehículo resistir[end] recibieron marzo [start]ambos preguntando disney[end] pierdas [start]pienso tornó excavó derecha[end] quejando sí[end] viene[end] teatro[end] goteara apetece alemán querrías\n",
      "-\n",
      "Tom doesn't remember doing what everybody says he did.\n",
      "1/1 [==============================] - 0s 80ms/step\n",
      "1/1 [==============================] - 0s 66ms/step\n",
      "1/1 [==============================] - 0s 81ms/step\n",
      "1/1 [==============================] - 0s 88ms/step\n",
      "1/1 [==============================] - 0s 76ms/step\n",
      "1/1 [==============================] - 0s 126ms/step\n",
      "1/1 [==============================] - 0s 80ms/step\n",
      "1/1 [==============================] - 0s 116ms/step\n",
      "1/1 [==============================] - 0s 90ms/step\n",
      "1/1 [==============================] - 0s 99ms/step\n",
      "1/1 [==============================] - 0s 91ms/step\n",
      "1/1 [==============================] - 0s 113ms/step\n",
      "1/1 [==============================] - 0s 99ms/step\n",
      "1/1 [==============================] - 0s 84ms/step\n",
      "1/1 [==============================] - 0s 84ms/step\n",
      "1/1 [==============================] - 0s 81ms/step\n",
      "1/1 [==============================] - 0s 93ms/step\n",
      "1/1 [==============================] - 0s 95ms/step\n",
      "1/1 [==============================] - 0s 97ms/step\n",
      "1/1 [==============================] - 0s 83ms/step\n",
      "[start] construcción[end] minuto difundió[end] esclavitud levantarte[end] levantarte[end] amigo[end] joyas inteligentes insultó reescribir escribía pierde[end] aristócrata[end] mucha dado[end] torre vendrías eléctrico[end] anticuado[end]\n",
      "-\n",
      "She depends on her husband for everything.\n",
      "1/1 [==============================] - 0s 158ms/step\n",
      "1/1 [==============================] - 0s 70ms/step\n",
      "1/1 [==============================] - 0s 62ms/step\n",
      "1/1 [==============================] - 0s 77ms/step\n",
      "1/1 [==============================] - 0s 76ms/step\n",
      "1/1 [==============================] - 0s 78ms/step\n",
      "1/1 [==============================] - 0s 79ms/step\n",
      "1/1 [==============================] - 0s 77ms/step\n",
      "1/1 [==============================] - 0s 77ms/step\n",
      "1/1 [==============================] - 0s 110ms/step\n",
      "1/1 [==============================] - 0s 68ms/step\n",
      "1/1 [==============================] - 0s 76ms/step\n",
      "1/1 [==============================] - 0s 92ms/step\n",
      "1/1 [==============================] - 0s 95ms/step\n",
      "1/1 [==============================] - 0s 93ms/step\n",
      "1/1 [==============================] - 0s 100ms/step\n",
      "1/1 [==============================] - 0s 77ms/step\n",
      "1/1 [==============================] - 0s 97ms/step\n",
      "1/1 [==============================] - 0s 82ms/step\n",
      "1/1 [==============================] - 0s 100ms/step\n",
      "[start] hundió[end] moría[end] culparme descanso[end] plástico[end] árboles[end] ocupo sociedad cruza geniales[end] toalla[end] pelotas propio muriera[end] evitando rosado[end] estado[end] sangrar[end] sí[end] golpeando\n",
      "-\n",
      "Tom was afraid that they'd lock him up and throw away the key.\n",
      "1/1 [==============================] - 0s 115ms/step\n",
      "1/1 [==============================] - 0s 82ms/step\n",
      "1/1 [==============================] - 0s 86ms/step\n",
      "1/1 [==============================] - 0s 98ms/step\n",
      "1/1 [==============================] - 0s 101ms/step\n",
      "1/1 [==============================] - 0s 98ms/step\n",
      "1/1 [==============================] - 0s 92ms/step\n",
      "1/1 [==============================] - 0s 81ms/step\n",
      "1/1 [==============================] - 0s 77ms/step\n",
      "1/1 [==============================] - 0s 75ms/step\n",
      "1/1 [==============================] - 0s 76ms/step\n",
      "1/1 [==============================] - 0s 76ms/step\n",
      "1/1 [==============================] - 0s 76ms/step\n",
      "1/1 [==============================] - 0s 76ms/step\n",
      "1/1 [==============================] - 0s 77ms/step\n",
      "1/1 [==============================] - 0s 76ms/step\n",
      "1/1 [==============================] - 0s 78ms/step\n",
      "1/1 [==============================] - 0s 77ms/step\n",
      "1/1 [==============================] - 0s 77ms/step\n",
      "1/1 [==============================] - 0s 79ms/step\n",
      "[start] construcción[end] escondieron vendrá[end] importar respirando[end] firmemente pregunté secundario[end] caminata ventana washington[end] [start]ellos mensualmente[end] poetas[end] bebí descubrimiento almacén ruidoso[end] ellas[end] mal[end]\n",
      "-\n",
      "They are collecting contributions for the church.\n",
      "1/1 [==============================] - 0s 79ms/step\n",
      "1/1 [==============================] - 0s 93ms/step\n",
      "1/1 [==============================] - 0s 84ms/step\n",
      "1/1 [==============================] - 0s 93ms/step\n",
      "1/1 [==============================] - 0s 75ms/step\n",
      "1/1 [==============================] - 0s 79ms/step\n",
      "1/1 [==============================] - 0s 81ms/step\n",
      "1/1 [==============================] - 0s 81ms/step\n",
      "1/1 [==============================] - 0s 116ms/step\n",
      "1/1 [==============================] - 0s 77ms/step\n",
      "1/1 [==============================] - 0s 78ms/step\n",
      "1/1 [==============================] - 0s 77ms/step\n",
      "1/1 [==============================] - 0s 76ms/step\n",
      "1/1 [==============================] - 0s 77ms/step\n",
      "1/1 [==============================] - 0s 84ms/step\n",
      "1/1 [==============================] - 0s 78ms/step\n",
      "1/1 [==============================] - 0s 74ms/step\n",
      "1/1 [==============================] - 0s 156ms/step\n",
      "1/1 [==============================] - 0s 72ms/step\n",
      "1/1 [==============================] - 0s 75ms/step\n",
      "[start] vendiendo retrase accidentes imaginación[end] líneas entregarse [start]o tal[end] terminé despegado[end] vendría quedar[end] [start]salgo cansada año[end] atentamente[end] millas[end] pensado montañas malentendido[end]\n",
      "-\n",
      "I bribed them.\n",
      "1/1 [==============================] - 0s 78ms/step\n",
      "1/1 [==============================] - 0s 78ms/step\n",
      "1/1 [==============================] - 0s 76ms/step\n",
      "1/1 [==============================] - 0s 78ms/step\n",
      "1/1 [==============================] - 0s 106ms/step\n",
      "1/1 [==============================] - 0s 67ms/step\n",
      "1/1 [==============================] - 0s 77ms/step\n",
      "1/1 [==============================] - 0s 84ms/step\n",
      "1/1 [==============================] - 0s 83ms/step\n",
      "1/1 [==============================] - 0s 76ms/step\n",
      "1/1 [==============================] - 0s 77ms/step\n",
      "1/1 [==============================] - 0s 74ms/step\n",
      "1/1 [==============================] - 0s 74ms/step\n",
      "1/1 [==============================] - 0s 75ms/step\n",
      "1/1 [==============================] - 0s 74ms/step\n",
      "1/1 [==============================] - 0s 75ms/step\n",
      "1/1 [==============================] - 0s 80ms/step\n",
      "1/1 [==============================] - 0s 77ms/step\n",
      "1/1 [==============================] - 0s 79ms/step\n",
      "1/1 [==============================] - 0s 76ms/step\n",
      "[start] metros sumo[end] pony[end] alice pásate musulmanes quebrar [start]recibiste hicisteis[end] [start]quizá verdad actitud[end] galletas[end] conduces[end] 10 oficinas[end] lindo[end] 5[end] asustan hambrientos\n",
      "-\n",
      "My sister is a nurse.\n",
      "1/1 [==============================] - 0s 87ms/step\n",
      "1/1 [==============================] - 0s 89ms/step\n",
      "1/1 [==============================] - 0s 76ms/step\n",
      "1/1 [==============================] - 0s 77ms/step\n",
      "1/1 [==============================] - 0s 76ms/step\n",
      "1/1 [==============================] - 0s 75ms/step\n",
      "1/1 [==============================] - 0s 77ms/step\n",
      "1/1 [==============================] - 0s 77ms/step\n",
      "1/1 [==============================] - 0s 91ms/step\n",
      "1/1 [==============================] - 0s 89ms/step\n",
      "1/1 [==============================] - 0s 68ms/step\n",
      "1/1 [==============================] - 0s 79ms/step\n",
      "1/1 [==============================] - 0s 77ms/step\n",
      "1/1 [==============================] - 0s 77ms/step\n",
      "1/1 [==============================] - 0s 74ms/step\n",
      "1/1 [==============================] - 0s 78ms/step\n",
      "1/1 [==============================] - 0s 76ms/step\n",
      "1/1 [==============================] - 0s 76ms/step\n",
      "1/1 [==============================] - 0s 77ms/step\n",
      "1/1 [==============================] - 0s 78ms/step\n",
      "[start] doy drásticamente cuidarte lucha[end] se crió dólar[end] música Él devolver discreto[end] propuesta llanto[end] llanto[end] verde habló contar[end] nevando[end] esperando gato[end]\n",
      "-\n",
      "How do you assess your students?\n",
      "1/1 [==============================] - 0s 88ms/step\n",
      "1/1 [==============================] - 0s 90ms/step\n",
      "1/1 [==============================] - 0s 72ms/step\n",
      "1/1 [==============================] - 0s 76ms/step\n",
      "1/1 [==============================] - 0s 76ms/step\n",
      "1/1 [==============================] - 0s 75ms/step\n",
      "1/1 [==============================] - 0s 128ms/step\n",
      "1/1 [==============================] - 0s 71ms/step\n",
      "1/1 [==============================] - 0s 71ms/step\n",
      "1/1 [==============================] - 0s 79ms/step\n",
      "1/1 [==============================] - 0s 79ms/step\n",
      "1/1 [==============================] - 0s 76ms/step\n",
      "1/1 [==============================] - 0s 77ms/step\n",
      "1/1 [==============================] - 0s 78ms/step\n",
      "1/1 [==============================] - 0s 74ms/step\n",
      "1/1 [==============================] - 0s 77ms/step\n",
      "1/1 [==============================] - 0s 78ms/step\n",
      "1/1 [==============================] - 0s 78ms/step\n",
      "1/1 [==============================] - 0s 77ms/step\n",
      "1/1 [==============================] - 0s 97ms/step\n",
      "[start] doy destino [start]italia japonesa[end] gafas regla[end] fuerzas vasos alas rehusaron enviado matar[end] ello estropeó[end] perdida[end] programa[end] [start]estáis calle[end] afuera[end] almorzar\n",
      "-\n",
      "Canada is larger than Japan.\n",
      "1/1 [==============================] - 0s 111ms/step\n",
      "1/1 [==============================] - 0s 106ms/step\n",
      "1/1 [==============================] - 0s 107ms/step\n",
      "1/1 [==============================] - 0s 102ms/step\n",
      "1/1 [==============================] - 0s 107ms/step\n",
      "1/1 [==============================] - 0s 91ms/step\n",
      "1/1 [==============================] - 0s 109ms/step\n",
      "1/1 [==============================] - 0s 105ms/step\n",
      "1/1 [==============================] - 0s 102ms/step\n",
      "1/1 [==============================] - 0s 110ms/step\n",
      "1/1 [==============================] - 0s 127ms/step\n",
      "1/1 [==============================] - 0s 95ms/step\n",
      "1/1 [==============================] - 0s 96ms/step\n",
      "1/1 [==============================] - 0s 106ms/step\n",
      "1/1 [==============================] - 0s 130ms/step\n",
      "1/1 [==============================] - 0s 101ms/step\n",
      "1/1 [==============================] - 0s 79ms/step\n",
      "1/1 [==============================] - 0s 159ms/step\n",
      "1/1 [==============================] - 0s 145ms/step\n",
      "1/1 [==============================] - 0s 101ms/step\n",
      "[start] importantes[end] leía [start]tienes objeto completar despejar enigma[end] traicionaré conejos reglas[end] soplaba codo[end] requiere extraterrestres[end] fastidioso[end] sacado honestos[end] chicos inmediatamente carretilla[end]\n",
      "-\n",
      "I just need some sleep.\n",
      "1/1 [==============================] - 0s 110ms/step\n",
      "1/1 [==============================] - 0s 81ms/step\n",
      "1/1 [==============================] - 0s 88ms/step\n",
      "1/1 [==============================] - 0s 85ms/step\n",
      "1/1 [==============================] - 0s 94ms/step\n",
      "1/1 [==============================] - 0s 84ms/step\n",
      "1/1 [==============================] - 0s 65ms/step\n",
      "1/1 [==============================] - 0s 82ms/step\n",
      "1/1 [==============================] - 0s 82ms/step\n",
      "1/1 [==============================] - 0s 85ms/step\n",
      "1/1 [==============================] - 0s 87ms/step\n",
      "1/1 [==============================] - 0s 81ms/step\n",
      "1/1 [==============================] - 0s 81ms/step\n",
      "1/1 [==============================] - 0s 85ms/step\n",
      "1/1 [==============================] - 0s 85ms/step\n",
      "1/1 [==============================] - 0s 89ms/step\n",
      "1/1 [==============================] - 0s 80ms/step\n",
      "1/1 [==============================] - 0s 83ms/step\n",
      "1/1 [==============================] - 0s 82ms/step\n",
      "1/1 [==============================] - 0s 83ms/step\n",
      "[start] difícil[end] diligentes[end] archivo [start]estaría dominó medias[end] suelen pesa conocías [start]iremos pasados[end] cansó tuviéramos mensualmente[end] poetas[end] bebí descubrimiento almacén ruidoso[end] ellas[end]\n",
      "-\n",
      "We're credible.\n",
      "1/1 [==============================] - 0s 92ms/step\n",
      "1/1 [==============================] - 0s 75ms/step\n",
      "1/1 [==============================] - 0s 100ms/step\n",
      "1/1 [==============================] - 0s 89ms/step\n",
      "1/1 [==============================] - 0s 78ms/step\n",
      "1/1 [==============================] - 0s 65ms/step\n",
      "1/1 [==============================] - 0s 78ms/step\n",
      "1/1 [==============================] - 0s 67ms/step\n",
      "1/1 [==============================] - 0s 94ms/step\n",
      "1/1 [==============================] - 0s 94ms/step\n",
      "1/1 [==============================] - 0s 88ms/step\n",
      "1/1 [==============================] - 0s 90ms/step\n",
      "1/1 [==============================] - 0s 95ms/step\n",
      "1/1 [==============================] - 0s 88ms/step\n",
      "1/1 [==============================] - 0s 87ms/step\n",
      "1/1 [==============================] - 0s 84ms/step\n",
      "1/1 [==============================] - 0s 86ms/step\n",
      "1/1 [==============================] - 0s 97ms/step\n",
      "1/1 [==============================] - 0s 86ms/step\n",
      "1/1 [==============================] - 0s 78ms/step\n",
      "[start] [start]dejó únicamente sentía[end] tratamiento refieres[end] identidad[end] Él miraron[end] evitar fumador[end] pues pianista[end] espía permitir villa[end] casadas[end] muñeca[end] secretario querían[end] dudoso[end]\n",
      "-\n",
      "In February it snows at least every three days.\n",
      "1/1 [==============================] - 0s 91ms/step\n",
      "1/1 [==============================] - 0s 99ms/step\n",
      "1/1 [==============================] - 0s 94ms/step\n",
      "1/1 [==============================] - 0s 81ms/step\n",
      "1/1 [==============================] - 0s 98ms/step\n",
      "1/1 [==============================] - 0s 89ms/step\n",
      "1/1 [==============================] - 0s 90ms/step\n",
      "1/1 [==============================] - 0s 90ms/step\n",
      "1/1 [==============================] - 0s 91ms/step\n",
      "1/1 [==============================] - 0s 90ms/step\n",
      "1/1 [==============================] - 0s 95ms/step\n",
      "1/1 [==============================] - 0s 96ms/step\n",
      "1/1 [==============================] - 0s 137ms/step\n",
      "1/1 [==============================] - 0s 95ms/step\n",
      "1/1 [==============================] - 0s 103ms/step\n",
      "1/1 [==============================] - 0s 98ms/step\n",
      "1/1 [==============================] - 0s 82ms/step\n",
      "1/1 [==============================] - 0s 93ms/step\n",
      "1/1 [==============================] - 0s 93ms/step\n",
      "1/1 [==============================] - 0s 90ms/step\n",
      "[start] doy pide morder alcohol alas nuestro[end] documento[end] muriera libros despertarte[end] mentes[end] derribar sequía[end] relojes[end] tareas [start]pidió temprano económico[end] planchó igual[end]\n",
      "-\n",
      "Who'll cook?\n",
      "1/1 [==============================] - 0s 138ms/step\n",
      "1/1 [==============================] - 0s 90ms/step\n",
      "1/1 [==============================] - 0s 82ms/step\n",
      "1/1 [==============================] - 0s 74ms/step\n",
      "1/1 [==============================] - 0s 79ms/step\n",
      "1/1 [==============================] - 0s 91ms/step\n",
      "1/1 [==============================] - 0s 141ms/step\n",
      "1/1 [==============================] - 0s 129ms/step\n",
      "1/1 [==============================] - 0s 125ms/step\n",
      "1/1 [==============================] - 0s 110ms/step\n",
      "1/1 [==============================] - 0s 94ms/step\n",
      "1/1 [==============================] - 0s 118ms/step\n",
      "1/1 [==============================] - 0s 94ms/step\n",
      "1/1 [==============================] - 0s 79ms/step\n",
      "1/1 [==============================] - 0s 79ms/step\n",
      "1/1 [==============================] - 0s 85ms/step\n",
      "1/1 [==============================] - 0s 79ms/step\n",
      "1/1 [==============================] - 0s 110ms/step\n",
      "1/1 [==============================] - 0s 110ms/step\n",
      "1/1 [==============================] - 0s 110ms/step\n",
      "[start] doy pasajeros profecía cometa[end] mudo 10 pocos indicar traerte [start]intentemos entregaron adoptar iban alcalde[end] cuesta pasan [start]trabajaré amaré parado comenzó\n",
      "-\n",
      "Let's start!\n",
      "1/1 [==============================] - 0s 125ms/step\n",
      "1/1 [==============================] - 0s 100ms/step\n",
      "1/1 [==============================] - 0s 79ms/step\n",
      "1/1 [==============================] - 0s 82ms/step\n",
      "1/1 [==============================] - 0s 126ms/step\n",
      "1/1 [==============================] - 0s 110ms/step\n",
      "1/1 [==============================] - 0s 111ms/step\n",
      "1/1 [==============================] - 0s 110ms/step\n",
      "1/1 [==============================] - 0s 94ms/step\n",
      "1/1 [==============================] - 0s 94ms/step\n",
      "1/1 [==============================] - 0s 79ms/step\n",
      "1/1 [==============================] - 0s 107ms/step\n",
      "1/1 [==============================] - 0s 109ms/step\n",
      "1/1 [==============================] - 0s 104ms/step\n",
      "1/1 [==============================] - 0s 90ms/step\n",
      "1/1 [==============================] - 0s 95ms/step\n",
      "1/1 [==============================] - 0s 100ms/step\n",
      "1/1 [==============================] - 0s 94ms/step\n",
      "1/1 [==============================] - 0s 95ms/step\n",
      "1/1 [==============================] - 0s 95ms/step\n",
      "[start] doy destino [start]italia [start]caminar bomba hornea esperarme[end] casarse botón inteligentes modo[end] [start]saluda esfuerzos[end] solicitud[end] salida[end] quejar»[end] tenedor[end] estás estabais tendré\n",
      "-\n",
      "I'm going straight home.\n",
      "1/1 [==============================] - 0s 110ms/step\n",
      "1/1 [==============================] - 0s 97ms/step\n",
      "1/1 [==============================] - 0s 93ms/step\n",
      "1/1 [==============================] - 0s 108ms/step\n",
      "1/1 [==============================] - 0s 112ms/step\n",
      "1/1 [==============================] - 0s 112ms/step\n",
      "1/1 [==============================] - 0s 106ms/step\n",
      "1/1 [==============================] - 0s 104ms/step\n",
      "1/1 [==============================] - 0s 103ms/step\n",
      "1/1 [==============================] - 0s 100ms/step\n",
      "1/1 [==============================] - 0s 89ms/step\n",
      "1/1 [==============================] - 0s 101ms/step\n",
      "1/1 [==============================] - 0s 94ms/step\n",
      "1/1 [==============================] - 0s 93ms/step\n",
      "1/1 [==============================] - 0s 93ms/step\n",
      "1/1 [==============================] - 0s 101ms/step\n",
      "1/1 [==============================] - 0s 108ms/step\n",
      "1/1 [==============================] - 0s 108ms/step\n",
      "1/1 [==============================] - 0s 93ms/step\n",
      "1/1 [==============================] - 0s 98ms/step\n",
      "[start] modo[end] toser[end] operarse[end] mostrarnos[end] sueles racha iniciar esposo cuerda proyector doctores fáciles[end] fina amarillo compraba homicida[end] vació soldados conductores entregara\n",
      "-\n",
      "All the roads leading into the city are full of cars.\n",
      "1/1 [==============================] - 0s 114ms/step\n",
      "1/1 [==============================] - 0s 101ms/step\n",
      "1/1 [==============================] - 0s 112ms/step\n",
      "1/1 [==============================] - 0s 98ms/step\n",
      "1/1 [==============================] - 0s 91ms/step\n",
      "1/1 [==============================] - 0s 118ms/step\n",
      "1/1 [==============================] - 0s 96ms/step\n",
      "1/1 [==============================] - 0s 86ms/step\n",
      "1/1 [==============================] - 0s 133ms/step\n",
      "1/1 [==============================] - 0s 86ms/step\n",
      "1/1 [==============================] - 0s 131ms/step\n",
      "1/1 [==============================] - 0s 85ms/step\n",
      "1/1 [==============================] - 0s 118ms/step\n",
      "1/1 [==============================] - 0s 79ms/step\n",
      "1/1 [==============================] - 0s 129ms/step\n",
      "1/1 [==============================] - 0s 126ms/step\n",
      "1/1 [==============================] - 0s 95ms/step\n",
      "1/1 [==============================] - 0s 134ms/step\n",
      "1/1 [==============================] - 0s 105ms/step\n",
      "1/1 [==============================] - 0s 86ms/step\n",
      "[start] doy pasajeros profecía cometa[end] tal ocurriera egipto[end] [start]cometiste [start]léelo ríen [start]pide carne[end] adónde desastre[end] amamos[end] supieras cebollas curar gata rana[end]\n",
      "-\n",
      "I want a book to read.\n",
      "1/1 [==============================] - 0s 108ms/step\n",
      "1/1 [==============================] - 0s 105ms/step\n",
      "1/1 [==============================] - 0s 81ms/step\n",
      "1/1 [==============================] - 0s 117ms/step\n",
      "1/1 [==============================] - 0s 99ms/step\n",
      "1/1 [==============================] - 0s 108ms/step\n",
      "1/1 [==============================] - 0s 89ms/step\n",
      "1/1 [==============================] - 0s 107ms/step\n",
      "1/1 [==============================] - 0s 108ms/step\n",
      "1/1 [==============================] - 0s 120ms/step\n",
      "1/1 [==============================] - 0s 87ms/step\n",
      "1/1 [==============================] - 0s 67ms/step\n",
      "1/1 [==============================] - 0s 113ms/step\n",
      "1/1 [==============================] - 0s 55ms/step\n",
      "1/1 [==============================] - 0s 55ms/step\n",
      "1/1 [==============================] - 0s 56ms/step\n",
      "1/1 [==============================] - 0s 57ms/step\n",
      "1/1 [==============================] - 0s 57ms/step\n",
      "1/1 [==============================] - 0s 55ms/step\n",
      "1/1 [==============================] - 0s 57ms/step\n",
      "[start] ir presos cliente alcohol[end] kong[end] compañía[end] cobarde[end] brazos divertida siento[end] siento[end] canario murieron[end] humano ipad quejas[end] villa llevaría trabajara generalizaciones\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\r\n",
    "\r\n",
    "# 准备一个字典，将词元索引预测值映射为字符串词元\r\n",
    "spa_vocab        = target_vectorization.get_vocabulary()\r\n",
    "spa_index_lookup = dict(zip(range(len(spa_vocab)), spa_vocab))\r\n",
    "max_decoded_sentence_length = 20\r\n",
    "\r\n",
    "def decode_sequence(input_sentence):\r\n",
    "    \r\n",
    "    tokenized_input_sentence = source_vectorization([input_sentence])\r\n",
    "    decoded_sentence         = '[start]'  # 种子词元\r\n",
    "\r\n",
    "    for i in range(max_decoded_sentence_length):\r\n",
    "        tokenized_target_sentence = target_vectorization([decoded_sentence])\r\n",
    "        \r\n",
    "        # 对下一个词元进行采样\r\n",
    "        next_token_predictions = seq2seq_rnn.predict([tokenized_input_sentence, tokenized_target_sentence])\r\n",
    "        sampled_token_index    = np.argmax(next_token_predictions[0, i, :])\r\n",
    "        # 将下一个词元预测值转换为字符串，并添加到生成的句子中\r\n",
    "        sampled_token          = spa_index_lookup[sampled_token_index]\r\n",
    "        decoded_sentence      += ' ' + sampled_token\r\n",
    "\r\n",
    "        # 退出条件:达到最大长度或遇到停止词元\r\n",
    "        if sampled_token == '[end]':\r\n",
    "            break\r\n",
    "    \r\n",
    "    return decoded_sentence\r\n",
    "\r\n",
    "test_eng_texts = [pair[0] for pair in test_pairs]\r\n",
    "\r\n",
    "for _ in range(20):\r\n",
    "    input_sentence = random.choice(test_eng_texts)\r\n",
    "    print('-')\r\n",
    "    print(input_sentence)\r\n",
    "    print(decode_sequence(input_sentence))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 11.5.3 使用 `Transformer` 进行序列到序列学习"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1 `Transformer` 解码器"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### [C] 11.33 `TransformerDecoder`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "class TransformerDecoder(layers.Layer):\r\n",
    "    \r\n",
    "    def __init__(self, embed_dim, dense_dim, num_heads, **kwargs):\r\n",
    "        super().__init__()\r\n",
    "\r\n",
    "        self.embed_dim   = embed_dim\r\n",
    "        self.dense_dim   = dense_dim\r\n",
    "        self.num_heads   = num_heads\r\n",
    "        self.attention_1 = layers.MultiHeadAttention(num_heads=num_heads, key_dim=embed_dim)\r\n",
    "        self.attention_2 = layers.MultiHeadAttention(num_heads=num_heads, key_dim=embed_dim)\r\n",
    "        self.dense_proj  = keras.Sequential([layers.Dense(dense_dim, activation='relu'),  layers.Dense(embed_dim), ])\r\n",
    "        self.layernorm_1 = layers.LayerNormalization()\r\n",
    "        self.layernorm_2 = layers.LayerNormalization()\r\n",
    "        self.layernorm_3 = layers.LayerNormalization()\r\n",
    "        self.supports_masking = True\r\n",
    "    \r\n",
    "    def get_config(self):\r\n",
    "        \r\n",
    "        config = super().get_config()\r\n",
    "        config.update({\r\n",
    "            \"embed_dim\": self.embed_dim,\r\n",
    "            \"num_heads\": self.num_heads,\r\n",
    "            \"dense_dim\": self.dense_dim,\r\n",
    "        })\r\n",
    "\r\n",
    "        return config\r\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### [C] 11.34 `TransformerDecoder` 中可以生成因果掩码的方法"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_causal_attention_mask(self, inputs):\r\n",
    "\r\n",
    "    input_shape = tf.shape(inputs)\r\n",
    "    batch_size, sequence_length = input_shape[0], input_shape[1]\r\n",
    "    i    = tf.range(sequence_length)[:, tf.newaxis]\r\n",
    "    j    = tf.range(sequence_length)\r\n",
    "    mask = tf.cast(i >= j, dtype=\"int32\")\r\n",
    "    mask = tf.reshape(mask, (1, input_shape[1], input_shape[1]))\r\n",
    "    mult = tf.concat([tf.expand_dims(batch_size, -1),\r\n",
    "                      tf.constant([1, 1], dtype=tf.int32)], axis=0)\r\n",
    "    \r\n",
    "    return tf.tile(mask, mult)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### [C] 11.35 `TransformerDecoder` 的前向传播"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "def call(self, inputs, encoder_outputs, mask=None):\r\n",
    "    \r\n",
    "    causal_mask = self.get_causal_attention_mask(inputs)\r\n",
    "    \r\n",
    "    if mask is not None:\r\n",
    "        padding_mask = tf.cast(\r\n",
    "                mask[:, tf.newaxis, :], dtype=\"int32\")\r\n",
    "        padding_mask = tf.minimum(padding_mask, causal_mask)\r\n",
    "    \r\n",
    "    attention_output_1 = self.attention_1(\r\n",
    "        query=inputs,\r\n",
    "        value=inputs,\r\n",
    "        key=inputs,\r\n",
    "        attention_mask=causal_mask)\r\n",
    "    \r\n",
    "    attention_output_1 = self.layernorm_1(inputs + attention_output_1)\r\n",
    "    \r\n",
    "    attention_output_2 = self.attention_2(\r\n",
    "        query=attention_output_1,\r\n",
    "        value=encoder_outputs,\r\n",
    "        key=encoder_outputs,\r\n",
    "        attention_mask=padding_mask,\r\n",
    "    )\r\n",
    "    \r\n",
    "    attention_output_2 = self.layernorm_2(\r\n",
    "        attention_output_1 + attention_output_2)\r\n",
    "    proj_output = self.dense_proj(attention_output_2)\r\n",
    "    \r\n",
    "    return self.layernorm_3(attention_output_2 + proj_output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "TransformerDecoder.get_causal_attention_mask = get_causal_attention_mask\r\n",
    "TransformerDecoder.call = call"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2 综合示例:用于机器翻译的 `Transformer`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### [C] 11.36 端到端 `Transformer`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "embed_dim = 256\r\n",
    "dense_dim = 2048\r\n",
    "num_heads = 8\r\n",
    "\r\n",
    "encoder_inputs  = keras.Input(shape=(None, ), dtype='int64', name='english')\r\n",
    "x               = PositionalEmbedding(sequence_length, vocab_size, embed_dim)(encoder_inputs)\r\n",
    "encoder_outputs = TransformerEncoder(embed_dim, dense_dim, num_heads)(x)\r\n",
    "\r\n",
    "decoder_inputs  = keras.Input(shape=(None, ), dtype='int64', name='spanish')\r\n",
    "x               = PositionalEmbedding(sequence_length, vocab_size, embed_dim)(decoder_inputs)\r\n",
    "x               = TransformerDecoder(embed_dim, dense_dim, num_heads)(x, encoder_outputs)\r\n",
    "x               = layers.Dropout(0.5)(x)\r\n",
    "\r\n",
    "decoder_outputs = layers.Dense(vocab_size, activation='softmax')(x)\r\n",
    "\r\n",
    "transformer = keras.Model([encoder_inputs, decoder_inputs], decoder_outputs)\r\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### [C] 11.37 训练序列到序列 `Transformer`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "1302/1302 [==============================] - 1795s 1s/step - loss: 4.8303 - accuracy: 0.3086 - val_loss: 3.9003 - val_accuracy: 0.3970\n",
      "Epoch 2/5\n",
      "1302/1302 [==============================] - 1909s 1s/step - loss: 3.9091 - accuracy: 0.4139 - val_loss: 3.4745 - val_accuracy: 0.4565\n",
      "Epoch 3/5\n",
      "1302/1302 [==============================] - 1727s 1s/step - loss: 3.5574 - accuracy: 0.4641 - val_loss: 3.2900 - val_accuracy: 0.4855\n",
      "Epoch 4/5\n",
      "1302/1302 [==============================] - 1864s 1s/step - loss: 3.2731 - accuracy: 0.5084 - val_loss: 3.0118 - val_accuracy: 0.5392\n",
      "Epoch 5/5\n",
      "1302/1302 [==============================] - 1861s 1s/step - loss: 2.9992 - accuracy: 0.5533 - val_loss: 2.8834 - val_accuracy: 0.5607\n"
     ]
    },
    {
     "data": {
      "text/plain": "<keras.callbacks.History at 0x2003a10d8d0>"
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "transformer.compile(optimizer='rmsprop', loss='sparse_categorical_crossentropy', metrics=['accuracy'])\r\n",
    "\r\n",
    "transformer.fit(train_ds, epochs=5, validation_data=val_ds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### [C] 11.38 利用 `Transformer` 模型来翻译句子"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-\n",
      "english: Do you know a good restaurant?\n",
      "spanish: [start] un buen día[end]                 \n",
      "-\n",
      "english: I use all kinds of software to study Chinese.\n",
      "spanish: [start] todos los días de estudiar japonés[end]              \n"
     ]
    }
   ],
   "source": [
    "import numpy as np\r\n",
    "\r\n",
    "spa_vocab                   = target_vectorization.get_vocabulary()\r\n",
    "spa_index_lookup            = dict(zip(range(len(spa_vocab)), spa_vocab))\r\n",
    "max_decoded_sentence_length = 20\r\n",
    "\r\n",
    "def decode_sequence(input_sentence):\r\n",
    "    \r\n",
    "    tokenized_input_sentence = source_vectorization([input_sentence])\r\n",
    "    decoded_sentence = \"[start]\"\r\n",
    "    for i in range(max_decoded_sentence_length):\r\n",
    "        tokenized_target_sentence = target_vectorization([decoded_sentence])[:, :-1]\r\n",
    "        predictions               = transformer([tokenized_input_sentence, tokenized_target_sentence])\r\n",
    "        sampled_token_index       = np.argmax(predictions[0, i, :])\r\n",
    "        sampled_token             = spa_index_lookup[sampled_token_index]\r\n",
    "        decoded_sentence         += \" \" + sampled_token\r\n",
    "        \r\n",
    "        if sampled_token == \"[end]\":\r\n",
    "            break\r\n",
    "    \r\n",
    "    return decoded_sentence\r\n",
    "\r\n",
    "test_eng_texts = [pair[0] for pair in test_pairs]\r\n",
    "\r\n",
    "for _ in range(2):\r\n",
    "    input_sentence = random.choice(test_eng_texts)\r\n",
    "    print(\"-\")\r\n",
    "    print('english: ' + input_sentence)\r\n",
    "    print('spanish: ' + decode_sequence(input_sentence))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 11.6 本章总结\r\n",
    "---\r\n",
    "* 自然语言处理(`NLP`)模型有两类：1、词袋模型(由多个`Dense`层组成)；2、序列模型(可以是`RNN`、`一维卷积神经网络`或`Transformer`)\r\n",
    "* 对于文本分类，训练数据中的样本数和每个样本的平均词数之间的比例，有助于判断应该使用词袋模型还是序列模型。\r\n",
    "* `词嵌入`是向量空间，其中单词之间的语义关系被表示为这些词向量之间的距离关系。\r\n",
    "* `序列到序列`模型由编码器和解码器组成，前者处理源序列，后者利用编码器处理后的源序列，并通过查看过去的词元来尝试预测目标序列后面的词元。\r\n",
    "* `神经注意力`可以生成上下文感知的词表示。它是`Transformer`架构的基础。\r\n",
    "* `Transformer`架构由`TransformerEncoder`和`TransformerDecoder`组成。`TransformerEncoder`也可以用于文本分类任务或任意类型的单一输入`NLP`任务。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.11.3 64-bit",
   "name": "python3113jvsc74a57bd0e842fc153c48e5d72fdba74c5fa9ec255a93a35200f85dc4905f2030a563e165"
  },
  "language_info": {
   "name": "python",
   "version": ""
  },
  "metadata": {
   "interpreter": {
    "hash": "e842fc153c48e5d72fdba74c5fa9ec255a93a35200f85dc4905f2030a563e165"
   }
  },
  "orig_nbformat": 2
 },
 "nbformat": 4,
 "nbformat_minor": 2
}