twitter.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Prédiction de types MBTI à partir de Tweets\n",
    "\n",
    "Ce notebook étend le travail réalisé sur le dataset Kaggle MBTI (voir [mbti.ipynb](mbti.ipynb)). L'objectif est de voir s'il est possible de prédire le type MBTI de célébrités à partir de leurs Tweets.\n",
    "\n",
    "Pour cela, nous avons récupéré les 50 derniers Tweets de **77 célébrités** dont nous avons obtenu le type MBTI grâce à [The MBTI Database](https://www.mbtidatabase.com/).\n",
    "\n",
    "Commençons par importer les librairies pertinentes :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import sklearn as sk\n",
    "import seaborn as sns\n",
    "\n",
    "pd.set_option('display.width', 3000)\n",
    "pd.set_option('display.max_rows', 200)\n",
    "pd.set_option('display.max_columns', 200)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import des données :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "DATASET = \"./data/tweets.json\"\n",
    "\n",
    "def get_dataset():\n",
    "    data = pd.read_json(DATASET, orient='records')\n",
    "    data = data[[\"text\", \"mbti\", \"name\", \"screen_name\"]]\n",
    "    return data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = get_dataset()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Aperçu du dataset :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>mbti</th>\n",
       "      <th>name</th>\n",
       "      <th>screen_name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>From a big NBA fan, congrats to future Hall of...</td>\n",
       "      <td>ENFJ</td>\n",
       "      <td>Barack Obama</td>\n",
       "      <td>BarackObama</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>A voice everybody should hear. https://t.co/0u...</td>\n",
       "      <td>ENFJ</td>\n",
       "      <td>Barack Obama</td>\n",
       "      <td>BarackObama</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>And here are some ways to help cyclone victims...</td>\n",
       "      <td>ENFJ</td>\n",
       "      <td>Barack Obama</td>\n",
       "      <td>BarackObama</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>What a great moment. Happy 110th, Virginia. ht...</td>\n",
       "      <td>ENFJ</td>\n",
       "      <td>Barack Obama</td>\n",
       "      <td>BarackObama</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Michelle and I send our deepest sympathies to ...</td>\n",
       "      <td>ENFJ</td>\n",
       "      <td>Barack Obama</td>\n",
       "      <td>BarackObama</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                text  mbti          name  screen_name\n",
       "0  From a big NBA fan, congrats to future Hall of...  ENFJ  Barack Obama  BarackObama\n",
       "1  A voice everybody should hear. https://t.co/0u...  ENFJ  Barack Obama  BarackObama\n",
       "2  And here are some ways to help cyclone victims...  ENFJ  Barack Obama  BarackObama\n",
       "3  What a great moment. Happy 110th, Virginia. ht...  ENFJ  Barack Obama  BarackObama\n",
       "4  Michelle and I send our deepest sympathies to ...  ENFJ  Barack Obama  BarackObama"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nombre de tweets récupérés :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2789"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analyse du dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tout d'abord, regardons la répartition des types MBTI au sein des tweets récupérés :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmcAAAFACAYAAAD589sCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAHUlJREFUeJzt3Xv8ZXVd7/HXG1AjpbwwEQ5Mo0gWHnGkSekOUt6OBRUQZIQeOnh6QF6OnqLykBF47JjyyKNRGCSWgnhL8qElkh7vlwFGrnocEYKRy4So4AUDPuePvUa3P36XPfu31/5957dfz8djP2at715rr89v7b0Wb77rlqpCkiRJbdhlpQuQJEnSdxnOJEmSGmI4kyRJaojhTJIkqSGGM0mSpIYYziRJkhpiOJMkSWqI4UySJKkhhjNJkqSG7LbSBSzHnnvuWevXr1/pMiRJkpZ06aWX/ntVrVlqup06nK1fv55NmzatdBmSJElLSnLDKNN5WFOSJKkhhjNJkqSG9BbOknxfkk8l+UySq5P8adf+qCSfTLIlyVuSPLBrf1A3vqV7f31ftUmSJLWqz56zu4GnVNUTgA3A05McDPw5cGZVPQa4Azihm/4E4I6u/cxuOkmSpJnSWzirgbu60Qd0rwKeArytaz8POKIbPrwbp3v/sCTpqz5JkqQW9XrOWZJdk2wGbgMuBr4AfKWq7ukmuQlY2w2vBW4E6N7/KvCIPuuTJElqTa/hrKruraoNwD7Ak4AfW+5nJjkxyaYkm7Zt27bsGiVJkloylas1q+orwAeAnwIemmT7/dX2AbZ2w1uBfQG6938QuH2ezzq7qjZW1cY1a5a8j5skSdJOpc+rNdckeWg3vDvwS8C1DELakd1kxwPv6oYv6sbp3v/Xqqq+6pMkSWpRn08I2Bs4L8muDELghVX17iTXABckOR24HDinm/4c4O+TbAG+DBzTY22SJElN6i2cVdUVwBPnab+Owflnc9u/BRzVVz2SJEk7g5362ZpzHXXa+VNd3ltPPXaqy5MkSaufj2+SJElqiOFMkiSpIavqsGZLpnmI1cOrkiStHvacSZIkNcRwJkmS1BDDmSRJUkMMZ5IkSQ0xnEmSJDXEcCZJktQQw5kkSVJDDGeSJEkNMZxJkiQ1xHAmSZLUEMOZJElSQwxnkiRJDTGcSZIkNcRwJkmS1BDDmSRJUkMMZ5IkSQ0xnEmSJDXEcCZJktQQw5kkSVJDDGeSJEkNMZxJkiQ1xHAmSZLUEMOZJElSQwxnkiRJDTGcSZIkNcRwJkmS1BDDmSRJUkMMZ5IkSQ0xnEmSJDXEcCZJktQQw5kkSVJDegtnSfZN8oEk1yS5OskLuvaXJdmaZHP3eubQPH+YZEuSzyV5Wl+1SZIktWq3Hj/7HuDFVXVZkj2AS5Nc3L13ZlX9xfDESQ4AjgEeBzwSeH+SH62qe3usUZIkqSm99ZxV1c1VdVk3fCdwLbB2kVkOBy6oqrur6ovAFuBJfdUnSZLUoqmcc5ZkPfBE4JNd08lJrkhybpKHdW1rgRuHZruJxcOcJEnSqtN7OEvyEODtwAur6mvAWcB+wAbgZuBVO/h5JybZlGTTtm3bJl6vJEnSSuo1nCV5AINg9qaqegdAVd1aVfdW1X3A6/nuocutwL5Ds+/TtX2Pqjq7qjZW1cY1a9b0Wb4kSdLU9Xm1ZoBzgGur6tVD7XsPTfarwFXd8EXAMUkelORRwP7Ap/qqT5IkqUV9Xq35M8BxwJVJNndtfwQcm2QDUMD1wPMAqurqJBcC1zC40vMkr9SUJEmzprdwVlUfATLPW+9ZZJ4zgDP6qkmSJKl1PiFAkiSpIYYzSZKkhhjOJEmSGmI4kyRJaojhTJIkqSGGM0mSpIYYziRJkhpiOJMkSWqI4UySJKkhhjNJkqSG9PlsTTXgqNPOn+ry3nrqsVNdniRJq409Z5IkSQ0xnEmSJDXEcCZJktQQw5kkSVJDDGeSJEkNMZxJkiQ1xHAmSZLUEMOZJElSQ7wJrabGG+JKkrQ0e84kSZIaYjiTJElqiOFMkiSpIYYzSZKkhhjOJEmSGmI4kyRJaojhTJIkqSGGM0mSpIYYziRJkhpiOJMkSWqI4UySJKkhhjNJkqSGGM4kSZIaYjiTJElqyJLhLMnuSdIN75fkmUl26780SZKk2TNKz9mHgd2T7A38K/BfgXOXminJvkk+kOSaJFcneUHX/vAkFyf5fPfvw7r2JHlNki1Jrkhy0DL+LkmSpJ3SKOFsl6r6BvDrwFlV9avAgSPMdw/w4qo6ADgYOCnJAcApwCVVtT9wSTcO8Axg/+51InDWDv0lkiRJq8BI4SzJTwLPBt7dte261ExVdXNVXdYN3wlcC6wFDgfO6yY7DziiGz4ceGMNfAJ4aNdbJ0mSNDNGCWcvAv4UeHdVXZXk0QwOdY4syXrgicAngb2q6uburVuAvbrhtcCNQ7Pd1LVJkiTNjFFO7H9YVT1z+0hVXZfk/aMuIMlDgLcDL6yqr3XXFmz/rEpSO1JwkhMZHPZk3bp1OzKrJElS80YJZy8F3jGn7Y/nabufJA9gEMzeVFXbp781yd5VdXN32PK2rn0rsO/Q7Pt0bd+jqs4GzgbYuHHjDgU7abujTjt/qst766nHTnV5kqSd14LhLMnTgKcDa5O8euitHwDuW+qDu9tvnANcW1XD818EHA+8ovv3XUPtJye5AHgy8NWhw5+SJEkzYbGes9uAq4BvAVcPtd/Jd6+wXMzPAMcBVybZ3LX9EYNQdmGSE4AbgKO7994DPBPYAnwDeO6If4MkSdKqsWA4q6rLgcuTvIlBT9m6qtoy6gdX1UeALPD2YfNMX8BJo36+JEnSajTK1ZqHAVcCFwMk2ZDknb1WJUmSNKNGCWenMTgH7CsAVbUZeEyfRUmSJM2qUcLZf1TVV+a0eZWkJElSD0a5lca1SY5m8KSARwHPBz7Rb1mSJEmzaZSes5OBn2BwUcA7gLuBF/ZZlCRJ0qxasuesqr4O/EGSU6vq7inUJEmSNLOW7DlL8uQkVwKf78afkOT/9F6ZJEnSDBrlnLO/BJ4F/CNAVX0myaG9ViXNEB8lJUkaNso5Z7tU1Q1z2u7toxhJkqRZN0rP2Y1JngRUkl2B3wP+X79lSZIkzaZRes5+F/jvwDrgVuDgrk2SJEkTNkrP2Veq6pjeK5G04jz/TZJW3qg3ob0J+HD3+mhV3dVvWZIkSbNpycOaVbUf8FwGt9L4deDKJJv6LkySJGkWLdlzluSHGTwh4CeBxwGfAz7ac12SJEkzaZTDml8CPg28HHh+Vd3Xb0mSJEmza5SrNX8SeDNwHPCRJOcmOb7fsiRJkmbTKM/WvDTJNcDVwM8DxwNPBc7ruTZJM8wrRyXNqlHOOfsksAfwceBDwFOq6gt9FyZJkjSLFgxnSX6tqt4BHF5Vt0yxJkmSpJm12DlnLwUwmEmSJE3PKBcESJIkaUoWO+fsx5JcMU97gKqqA3uqSZIkaWYtFs6+CPzytAqRJEnS4uHs21V1w9QqkSRJ0qLnnPmIJkmSpClbMJxV1cnTLESSJElerSlJktQUw5kkSVJDlgxnSb4/yf9M8vpufP8kz+q/NEmSpNkzSs/Z3wF3Az/VjW8FTu+tIkmSpBk2Sjjbr6r+N/AfAFX1DQY3opUkSdKEjRLOvp1kd6AAkuzHoCdNkiRJE7bYTWi3+xPgn4F9k7wJ+BngOX0WJUktOeq086e6vLeeeuyC702zlsXqkNSfJcNZVV2c5DLgYAaHM19QVf/ee2WSJEkzaMHDmkkO2v4CfgS4GfgSsK5rW1SSc5PcluSqobaXJdmaZHP3eubQe3+YZEuSzyV52vL+LEmSpJ3TYj1nr+r+/T5gI/AZBj1nBwKb+O7Vmwt5A/Ba4I1z2s+sqr8YbkhyAHAM8DjgkcD7k/xoVd07wt8gSZK0aiz2+KZDq+pQBj1mB1XVxqr6CeCJDG6nsaiq+hDw5RHrOBy4oKrurqovAluAJ404ryRJ0qoxygUBj62qK7ePVNVVSX58Gcs8OclvM+h9e3FV3QGsBT4xNM1NXZskqUEtXSQhrTaj3ErjiiR/m+SQ7vV64Ioxl3cWsB+wgUGP3KsWn/z+kpyYZFOSTdu2bRuzDEmSpDaNEs6eC1wNvKB7XdO17bCqurWq7q2q+4DX891Dl1uBfYcm3YcFDp1W1dndIdaNa9asGacMSZKkZo1yK41vAWd2r2VJsndV3dyN/iqw/UrOi4A3J3k1gwsC9gc+tdzlSZIk7WxGOedsLEnOBw4B9kxyE4Ob2R6SZAODpw1cDzwPoKquTnIhg165e4CTvFJTkiTNot7CWVXNd/bmOYtMfwZwRl/1SJJWJy9O0GozyjlnACT5/j4LkSRJ0gjhLMlPJ7kG+Gw3/oQkf9V7ZZIkSTNolJ6zM4GnAbcDVNVngJ/vsyhJkqRZNdJhzaq6cU6TJ+tLkiT1YJQLAm5M8tNAJXkAg3udXdtvWZIkSbNplJ6z/wacxOBxSlsZ3N3/pD6LkiRJmlWL9pwl2RU4rqqePaV6JEmSZtqiPWfdjWB/c0q1SJIkzbxRzjn7SJLXAm8Bvr69saou660qSZKkGTVKONvQ/XvaUFsBT5l8OZIkSbNtlAefHzqNQiRJkjTaEwIekeQ1SS5LcmmSv0zyiGkUJ0mSNGtGuZXGBcA24NeBI7vht/RZlCRJ0qwa5Zyzvavqz4bGT0/yG30VJEmSNMtG6Tl7X5JjkuzSvY4G/qXvwiRJkmbRgj1nSe5kcFVmgBcC/9C9tQtwF/CS3quTJEmaMQuGs6raY5qFSJIkabRzzkhyILB+ePqqekdPNUmSJM2sJcNZknOBA4Grgfu65gIMZ5IkSRM2Ss/ZwVV1QO+VSJIkaaSrNT+exHAmSZI0BaP0nL2RQUC7BbibwdWbVVUH9lqZJEnSDBolnJ0DHAdcyXfPOZMkSVIPRgln26rqot4rkSRJ0kjh7PIkbwb+icFhTcBbaUiSJPVhlHC2O4NQ9tShNm+lIUmS1IMlw1lVPXcahUiSJGmEW2kk2SfJO5Pc1r3enmSfaRQnSZI0a0a5z9nfARcBj+xe/9S1SZIkacJGCWdrqurvquqe7vUGYE3PdUmSJM2kUcLZ7Ul+K8mu3eu3gNv7LkySJGkWjRLO/gtwNHALcDNwJOBFApIkST0Y5WrNG4BfmUItkiTt1I467fypLu+tpx674Hst1aIds2A4S3LqIvNVVf1ZD/VIkqRVxqC4YxbrOfv6PG0PBk4AHgEYziRJkiZswXPOqupV21/A2QyeFPBc4ALg0Ut9cJJzu/uiXTXU9vAkFyf5fPfvw7r2JHlNki1Jrkhy0LL/MkmSpJ3QohcEdGHqdOAKBr1sB1XVH1TVbSN89huAp89pOwW4pKr2By7pxgGeAezfvU4Ezhr5L5AkSVpFFgxnSV4JfBq4E3h8Vb2squ4Y9YOr6kPAl+c0Hw6c1w2fBxwx1P7GGvgE8NAke4+6LEmSpNVisZ6zFzN4IsBLgS8l+Vr3ujPJ18Zc3l5VdXM3fAuwVze8FrhxaLqbujZJkqSZsuAFAVU1yj3QxlZVlaR2dL4kJzI49Mm6desmXpckSdJK6jWAzePW7Ycru3+3n7u2Fdh3aLp9urb7qaqzq2pjVW1cs8anSEmSpNVl2uHsIuD4bvh44F1D7b/dXbV5MPDVocOfkiRJM2PJJwSMK8n5wCHAnkluAv4EeAVwYZITgBsYPBYK4D3AM4EtwDfw8VCSJGlG9RbOqmqh2/MeNs+0BZzUVy2SJEk7i2kf1pQkSdIiDGeSJEkNMZxJkiQ1xHAmSZLUEMOZJElSQwxnkiRJDTGcSZIkNcRwJkmS1BDDmSRJUkMMZ5IkSQ0xnEmSJDXEcCZJktQQw5kkSVJDDGeSJEkN2W2lC5AkSZqWo047f6rLe+upx+7wPPacSZIkNcRwJkmS1BDDmSRJUkMMZ5IkSQ0xnEmSJDXEcCZJktQQw5kkSVJDDGeSJEkNMZxJkiQ1xHAmSZLUEMOZJElSQwxnkiRJDTGcSZIkNcRwJkmS1BDDmSRJUkMMZ5IkSQ0xnEmSJDXEcCZJktQQw5kkSVJDDGeSJEkN2W0lFprkeuBO4F7gnqramOThwFuA9cD1wNFVdcdK1CdJkrRSVrLn7NCq2lBVG7vxU4BLqmp/4JJuXJIkaaa0dFjzcOC8bvg84IgVrEWSJGlFrFQ4K+B9SS5NcmLXtldV3dwN3wLstTKlSZIkrZwVOecM+Nmq2prkh4CLk3x2+M2qqiQ134xdmDsRYN26df1XKkmSNEUr0nNWVVu7f28D3gk8Cbg1yd4A3b+3LTDv2VW1sao2rlmzZlolS5IkTcXUw1mSByfZY/sw8FTgKuAi4PhusuOBd027NkmSpJW2Eoc19wLemWT78t9cVf+c5NPAhUlOAG4Ajl6B2iRJklbU1MNZVV0HPGGe9tuBw6ZdjyRJUktaupWGJEnSzDOcSZIkNcRwJkmS1BDDmSRJUkMMZ5IkSQ0xnEmSJDXEcCZJktQQw5kkSVJDDGeSJEkNMZxJkiQ1xHAmSZLUEMOZJElSQwxnkiRJDTGcSZIkNcRwJkmS1BDDmSRJUkMMZ5IkSQ0xnEmSJDXEcCZJktQQw5kkSVJDDGeSJEkNMZxJkiQ1xHAmSZLUEMOZJElSQwxnkiRJDTGcSZIkNcRwJkmS1BDDmSRJUkMMZ5IkSQ0xnEmSJDXEcCZJktQQw5kkSVJDDGeSJEkNMZxJkiQ1xHAmSZLUkObCWZKnJ/lcki1JTlnpeiRJkqapqXCWZFfgdcAzgAOAY5McsLJVSZIkTU9T4Qx4ErClqq6rqm8DFwCHr3BNkiRJU9NaOFsL3Dg0flPXJkmSNBNSVStdw3ckORJ4elX9Tjd+HPDkqjp5aJoTgRO70ccCn5vAovcE/n0CnzMJrdTSSh1gLQuxlvlZy/xaqaWVOsBaFmIt85tELT9SVWuWmmi3ZS5k0rYC+w6N79O1fUdVnQ2cPcmFJtlUVRsn+ZnjaqWWVuoAa1mItczPWubXSi2t1AHWshBrmd80a2ntsOangf2TPCrJA4FjgItWuCZJkqSpaarnrKruSXIy8C/ArsC5VXX1CpclSZI0NU2FM4Cqeg/wnikvdqKHSZeplVpaqQOsZSHWMj9rmV8rtbRSB1jLQqxlflOrpakLAiRJkmZda+ecSZIkzTTDmSRJUkNWbThLcm+SzUOvU7r2ZyW5PMlnklyT5Hld+8uSbB2a/hVd+we7Z31+JslHkzx2zHru6v7dJclrklyV5Mokn07yqO6967u27TX8dJL1Sb7ZjV+T5K+TjPW9LbJOPphk09B0G7u2pw1Ne1e3HjYneWOSQ5J8tRu/Nsmf9FlLNzy8zM1J3t+1D393VyX5lTHWzfbvZ32SSvJ7Q++9Nslzkrxu6Hv45lAdRyZ5Q5IvduOXJfmpHa1h3HUzwvf07mXUsuR66YaH//7NSZ7ftW//TV+R5H1JfnjcWuapaZxt6arlLn+eOpazbvZcxvJH3scl+eOh6Ybne/4ktp9xauraF9vvjn3Lgh3dhrrhxfYvLxm3lu4zFv3NJvlkt8x/S7JtqIb1k9yGJvybWdY6mbNelrPfPXK5dcxTz7jf09jb8/1U1ap8AXfN0/YA4EvAPt34g4DHdsMvA14yzzwfBDZ2wycCFy2nHuBY4G3ALt34PsDDuuHrgT3nzLceuKob3g34EPBrk1onQ3/jvwHP6MY3Ah9caD1044cA7+6GHwx8Hjioz1qGlzlnnu98d8CPM7hJ4C5jfj/rgVuBLcADu7bXAs+Z7zsZansDcGQ3/FTgikn+difxPS3zd7voehn+++fM/53fNPBy4DXj1jJPTWNvS5N4TXLdTOp3wiL7uIXmm8T2M25NjLDfnVQdQ5879v6lz99sN/4c4LUL/U6Wuw318ZuZ0HpZdBsammbB/e4kXpP6nibxWrU9ZwvYg0HAuR2gqu6uqh15wsCHgMcss4a9gZur6r6uhpuq6o5RZqyqe4CPTaCG+bwS+ONxZqyqrwOXMrm6llPLtcA9DO7kPK5twCXA8WPOP4nfyULGXjcT0Np6GXtb6sFy182kLGsfN6HtZ6I19WAlt6Hl/mb72Le08v20sg1BA/uW1RzOdp/TffsbVfVlBje1vSHJ+Umene89RPiioemfNs9n/jJw5TLruhD45W4Zr0ryxDnvf6B775NzZ0zy/cBhy6jhfutk6L2PA99OcuiOfmiSRwAHAztyT7pxa/m5oXnut4NN8mTgPgYb+nL8OfCSJLuOMe9yfye9fE8Tsth6eeVQzY+f5/1nsfztZ9jY21JPlrNuxjHOPm5RE9h++tjvTqSOoffG3r9MwFK/2aUsdxua+G9mwpaz352k5X5Py9bcfc4m6JtVtWFuY1X9Trdz/EXgJcAvMeiiBDizqv5ins96U5JvMui2/L153h9ZVd2UwXlrT+lelyQ5qqou6SY5tKrmPrtrvySbgQLeVVXvHXPx866TIacDLwX+YMTP+7kklzPYmb+iduyGwePW8uGqetY8078oyW8BdwK/UV0/87iq6rruP+q/uQOzvTLJSxn8h+2EZSx+0t/TxCyxXv5HVb1tnvYPJLkXuIJB3ZOqZZxtqTdjrpvlGGcft5BJbT+T3O8ux6T3LxMxwm92IZPahib5m5m4Mfe7fdQx7vc0Mas5nC2oqq4Erkzy98AXWfpH+Oyq2rTENDuy/LuB9wLvTXIrcASD7tyFfGGJHc2k6vrXJKcz6AUbRW87sjFq6WMH/3IG5x383xGn7+M/wPczxrqZtB1dL72FpDG2pb7t6LrpxRj7uD62n+XW1GctK7YNjfmb7f1/NBr6flrZhlZ037KaD2veT5KHJDlkqGkDcMOUazgoySO74V2AA6ddwxJOB35/pYvorGgtVfVZ4BoGhylbs2LrppX10uK2tNLrpoV93Fwt1tSZ+jbU4m+2te9npbchaON7Ws09Z7t3hwK3+2fgDOD3k/wN8E3g60z//w5+CHh9kgd1459icFXKNNxvnVTVKcMTVNV7kiz3fK2drZbFnAFcPuVlTnLd7AbcPdHqBlZivcy1o9tSX+tirlHXzXLraXEfN6maJr5uGtmGVnL/D+18P0uZ1ja0kHG+p4nW4uObpFUsyQuAtVXVSm/oiklyOINTFI5uoJY1wOaqWrvStbSm+w/iFuA/VdVXG6jnncDra/DcZ9HGOul6tD4NHFdV16xUHV0tE9+eZ+qwpjRLkpzD4MTa1610LSstyWnAacD/aqCWXwE+DPzhStfSmgxuPLsZ+KtGgtmVDC54et9K19KKFtZJd8jxKuATDQSzXrZne84kSZIaYs+ZJElSQwxnkiRJDTGcSZIkNcRwJmmnkeQRQ4+euSXJ1qHxB/awvH9IcleSBw+1vTZJJXlokt2S3DtUw6VJDk6yYajty0m+2A3/S5LHzLmdgSR9j9V8nzNJq0xV3c7gJpkkeRlwV993tgeuY3BDzAu6Z/79PHDL0Pt3bn+CR5L/DJxRVYcN1fkPwNuq6h+78Uk/uFrSKmPPmaSdXpKXJzl5aPzPk5yU5BeTfCDJe5N8LsnrkqSb5hlJPp7ksiRvGe4dm+MCYPuDsw9j8FiZexeY9geAOyb0Z0maUYYzSavBucDxAF3v1lHAm7v3ngz8LnAA8OPA4Ul+CDgFOKyqDmLwQOkXLPDZ1wBrk/wgcCyDsDZsj+6Q5WeBsxjc3VySxuZhTUk7varakuTOJI8HfgT4VFXd0XWSfaKqrgdIcgHws91sBwAf66Z5IPCRRRbxj8AxwEHAx+a8N3xY82eBNwKPn8TfJWk2Gc4krRbnMHgm4Hrgb4ba595pu4AweN7icSN+9gUMHhXzt1VVXaC7n6r6SJJHJnl4VX15B2qXpO/wsKak1eLtDE7c3wC8f6j94CTrusOdRzPoIfsY8AtJHg2Q5MFJ9l/og6vqOuClwF8vVkCSxzF4tI3nnUkamz1nklaFqvpWkg8Bt1TVfUNvfYpBqNqPQWi7qOv9OgF4y9AtOP4I+Pwin3/WAm/tMefWGL9dPhdP0jL4bE1Jq0KSXRg8NPuIrqeLJL8InFxVR6xocZK0AzysKWmn110I8AUG55Fdt9L1SNJy2HMmSZLUEHvOJEmSGmI4kyRJaojhTJIkqSGGM0mSpIYYziRJkhry/wEwQIQ3Gn+A3gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Comptage des types MBTI\n",
    "counts = data.mbti.value_counts()\n",
    "mbti_types = counts.index\n",
    "num_users = counts.values\n",
    "\n",
    "# Tracé du graphique\n",
    "plt.figure(figsize=(10,5))\n",
    "sns.barplot(mbti_types, num_users, color=\"steelblue\")\n",
    "plt.xlabel(\"Type MBTI\")\n",
    "plt.ylabel(\"Nombre de Tweets\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "La répartition est plus uniforme que sur le dataset Kaggle MBTI. C'est une conséquence du fait que l'on a choisi les personnalités pour équilibrer le nombre d'utilisateurs par type MBTI.\n",
    "\n",
    "Par contre, les types MBTI dominants sont très différents : dans ce dataset, ESFP et ISFP sont dominants alors qu'ils étaient minoritaires dans le dataset Kaggle MBTI. Probablement le reflet des personnalités des célébrités de Twitter…"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Apprentissage supervisé\n",
    "\n",
    "On entraîne un classificateur de type régression logistique sur la base de données de tweets. Ce classificateur a été choisi car c'est celui qui a donné les meilleurs résultats sur le dataset Kaggle MBTI."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = get_dataset()\n",
    "# data = data.groupby([\"mbti\", \"screen_name\"]).text.apply(lambda t: \"|||\".join(t)).reset_index()\n",
    "# data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Classificateur Logistic Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.3231770914173921, 0.38509800819355794, 0.3514308332086986)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "from classifier_comparison import evaluate\n",
    "from preprocessing import format_text\n",
    "\n",
    "X = data.text  # Inputs\n",
    "Y = data.mbti  # Outputs (labels)\n",
    "\n",
    "train_x, test_x, train_y, test_y = train_test_split(X, Y, test_size=0.4)\n",
    "\n",
    "# Vectorisation du texte.\n",
    "vectorizer = CountVectorizer(stop_words=\"english\")\n",
    "train_x = vectorizer.fit_transform(train_x.apply(format_text))\n",
    "test_x = vectorizer.transform(test_x.apply(format_text))\n",
    "\n",
    "# Définition et entraînement du classificateur.\n",
    "clf = LogisticRegression(class_weight=\"balanced\")\n",
    "clf.fit(train_x, train_y)\n",
    "\n",
    "# Calcul des scores.\n",
    "train_score = clf.score(train_x, train_y)\n",
    "test_score = clf.score(test_x, test_y)\n",
    "\n",
    "# Calcul et évaluation des résultats.\n",
    "y = clf.predict(test_x)\n",
    "\n",
    "\n",
    "# Affichage des performances.\n",
    "evaluate(test_y, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Analyse des résultats\n",
    "\n",
    "Sur le dataset Kaggle MBTI, on obtenait un F-score optimal d'environ 0.75. Il est ici d'environ 0.33 : les performances sont donc bien moindres.\n",
    "\n",
    "Le comportement en généralisation pourrait également être meilleur. Par exemple, les tweets suivants ont été publiés par un ENFJ :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['INFP', 'INTJ'], dtype=object)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tweets = [\n",
    "    \"Hey all, anyone aware of some really good material to get started with @vuejs in the real world, transitioning from Angular? 😄 Basics (boilerplate, components, API interaction), useful tooling (VueCLI, TS?), best practices... Any insight very helpful. 🙏 Thanks!\",\n",
    "    \"Thrilled to say — UPDATE RELEASED! ✨ Hope you enjoy all the new features! I had fun implementing them. Feedback much appreciated.\"\n",
    "]\n",
    "x = vectorizer.transform(np.array([format_text(t) for t in tweets]))\n",
    "clf.predict(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "La moindre qualité des prédictions peut s'expliquer de plusieurs manières.\n",
    "\n",
    "Tout d'abord, il est raisonnable de penser que le type MBTI d'une célébrité sur Twitter ne transparaisse pas dans ses Tweets, pour plusieurs raisons :\n",
    "\n",
    "- Le texte d'un Tweet est influencé par un grand nombre de facteurs, ne serait-ce que par la longueur limitée du texte.\n",
    "- Les célébrités n'écrivent pas forcément d'une façon qui reflète leur personnalité étant donné que leur utilisation de Twitter est souvent orientée (par exemple à des fins de communication ou de promotion).\n",
    "\n",
    "Il y a aussi probablement d'autres caractéristiques que le texte qui font transparaître la personnalité d'un utilisateur sur Twitter : usage de la ponctuation, des emojis, etc.\n",
    "\n",
    "Il est aussi probable que choisir les 50 derniers tweets ne suffise pas à évaluer la personnalité d'un utilisateur Twitter."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "Ce travail constituait une tentative de transposition du travail effectué sur le dataset Kaggle MBTI, mais il apparaît finalement évident qu'une analyse plus approfondie de la morphologie et de la structure des données de type Tweets soit nécessaire. De futurs travaux pourront probablement s'inspirer de travaux de recherche sur le comportement des utilisateurs sur les réseaux sociaux comme Twitter."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}