2-dataAnonAndPrep.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "gothic-charter",
   "metadata": {},
   "source": [
    "# Final data prep\n",
    "\n",
    "(NEW CONTINUING)\n",
    "\n",
    "This is the second notebook in the project\n",
    "\n",
    "Basically, all the data was gathered already (that process was shown in the previous notebook). So, this code takes care of nearly everything in between the data gathering and the actual machine learning. I create a version of the data with the languages' writing systems anonymized, and export both the anonymized and non-anonymized data to files that can be read in in the ML notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "municipal-gothic",
   "metadata": {},
   "source": [
    "## Tabel of contents\n",
    "- [Anonymizing languages' writing systems](#Anonymizing-languages'-writing-systems)\n",
    "- [Prepare data for ML](#Prepare-data-for-ML)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "informational-columbus",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import pickle\n",
    "import sys # to get max int\n",
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fatty-harvest",
   "metadata": {},
   "source": [
    "## Anonymizing languages' writing systems\n",
    "\n",
    "For each language's extracted text file (or more precisely a portion I have chosen for the train dataset), I compute how common each character is. The most common character corresponds to 0, second most common to 1 and so on. The space character is not transformed; it's still just space. Then, I replace the characters with their number correspondances."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "earned-raising",
   "metadata": {},
   "outputs": [],
   "source": [
    "inpath = './data/chunks-nonanon/'\n",
    "outpath = './data/chunks-anon/'\n",
    "files = ['hr.txt'] #sample list for debugging"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "peaceful-broadway",
   "metadata": {},
   "outputs": [],
   "source": [
    "# assigns each character to a number. space is always 0. then, if the most common non-space\n",
    "# char is found to be 'e', then e gets assigned 1. if 't' is second most common, t is given 2.\n",
    "def make_transform(chunks):\n",
    "    char_dict = {}\n",
    "    char_dict[' '] = sys.maxsize\n",
    "    # index 0 is reserved character for space, corresponds to \"most common\"\n",
    "    # you get ind 0 by being having the greatest occurences, so that's max_int\n",
    "    \n",
    "    for s in chunks:\n",
    "        for c in s: # iterate over ea char in ea chunk\n",
    "            if c == ' ' or c == '\\n':\n",
    "                continue\n",
    "            char_dict[c] = char_dict.get(c, 0) + 1 # find num occurences of ea char\n",
    "    \n",
    "    df = pd.DataFrame.from_dict(char_dict, orient='index', columns=['occur'])\n",
    "    df.sort_values(by='occur', inplace=True, ascending=False) # sort by num occurrences\n",
    "    df.reset_index(inplace=True)\n",
    "    df.columns = ['char', 'occur']\n",
    "    df['ind'] = df.index # space will be index 0, most common char index 1, etc\n",
    "    df = df[df['ind']<256] # TEMPORARY FIX\n",
    "    df.set_index('char', inplace=True) # set index back to the characters\n",
    "    del df['occur'] # delete the num occurences\n",
    "    trans = df.to_dict()['ind']\n",
    "    return df, trans"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "tender-horror",
   "metadata": {},
   "outputs": [],
   "source": [
    "# using the supplied transformation, transform ea char in the text\n",
    "def apply_transform(chunks, trans):\n",
    "    chunks_trans = []\n",
    "    \n",
    "    for i,chunk in enumerate(chunks):\n",
    "        chunkarr = list(chunk)[:-1] # strip off newline \\n char at very end\n",
    "        for j,c in enumerate(chunkarr):\n",
    "            chunkarr[j] = trans.get(c, 255)\n",
    "        chunks_trans.append(bytes(chunkarr))\n",
    "    \n",
    "    return chunks_trans"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "emerging-command",
   "metadata": {},
   "outputs": [],
   "source": [
    "# iterate over ea lang, make and apply transform, save to file\n",
    "for f in os.listdir(inpath):\n",
    "    if f == 'README.md' or f.startswith('.'): continue\n",
    "#     print(f)\n",
    "    file = open(inpath+f, 'r')\n",
    "    chunks = file.readlines()\n",
    "    file.close()\n",
    "    \n",
    "    df, trans = make_transform(chunks)\n",
    "    chunks_trans = apply_transform(chunks, trans)\n",
    "\n",
    "    file = open(outpath+f, 'wb')\n",
    "    pickle.dump(chunks_trans, file)\n",
    "    file.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "functional-leisure",
   "metadata": {},
   "source": [
    "Example of an anonymized chunk:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "lonely-thickness",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'\\x0c\\x06\\x05\\x00\\x02\\x03\\t\\x01\\x00\\x08\\x02\\x05\\x01\\x03\\x00\\x07\\x01\\x00\\x04\\x0e\\r\\x02\\x08\\x06\\x03\\x00\\x12\\x08\\x04\\t\\x02\\x03\\x00\\x02\\x00\\x04\\t\\x01\\x07\\x04\\x01\\x05\\x00\\x04\\x01\\x00\\x10\\x01\\x05\\x04\\x01\\x00\\x0c\\x06\\x05\\t\\x08\\x06\\x07\\x01\\x00\\n\\x02\\x00\\r\\x01\\x08\\t\\x02\\x03\\x00\\n\\x02\\x00\\x07\\x01\\x00\\x0c\\x06\\x03\\t\\x01\\x00\\x02\\x00\\x04\\x01\\x00\\x04\\x05\\x03\\t\\x04\\t\\x0b\\x04\\x00\\x07\\x01\\x00\\x0c\\x06\\x07\\x06\\x05\\x04\\x01\\x03\\x00\\x12\\x08\\x04\\t\\x02\\x03\\x00\\x02\\x00\\x04\\t\\x01\\x07\\x04\\x01\\x05\\x00\\n\\x02\\x00\\x17\\x06\\x0e\\x01\\x07\\x04\\x07\\x01\\x05\\n\\x00\\x1f\\x02\\x00\\x17\\x06\\x0e\\x01\\x07\\x04\\x01\\x00\\x05\\x06\\x00\\x04\\x01\\x00\\x01\\x0f\\x02\\x00\\x0b\\x05\\x00\\x10\\x06\\x0f\\x02\\x08\\x05\\x01\\x00\\x0c\\x0b\\x01\\x07\\x00\\x0c\\x06\\x05\\t\\x08\\x06\\x07\\x01\\x00\\x07\\x01\\x00\\r\\x01\\x04\\x03\\x00\\x04\\x05\\t\\x02\\x08\\x01\\x00\\x15\\x05\\x00\\x07\\x01\\x00\\x05\\x06\\x08\\n\\x02\\x00\\x07\\x01\\x00\\x08\\x02\\x13\\x04\\x06\\x05\\x00\\n\\x02\\x00\\x17\\x06\\x0e\\x01\\x07\\x04\\x07\\x01\\x05\\n\\x00\\x04\\x01\\x00\\n\\x02\\x0c\\x07\\x01\\x08\\x01\\x00\\x01\\x0b\\t\\x06\\x05\\x06\\x0e\\x04\\x01\\x00\\x02\\x05\\x00\\x01\\x0c\\x02\\x07\\x00\\x01\\x05\\x04\\x06\\x00\\x1a\\x0b\\x05\\t\\x07\\x01\\x05\\n\\x00\\x02\\x03\\x00\\x0b\\x05\\x00\\x06\\t\\x08\\x01\\x00\\x08\\x02\\x13\\x04\\x06\\x05\\x00\\x0c\\x0b\\x01\\x07\\x00\\x04\\x01\\x00\\n\\x02\\x0c\\x07\\x01\\x08\\x01\\x00\\x05\\x06\\x05\\n\\x02\\r\\x02\\x05\\n\\x02\\x00\\x17\\x06\\x0e\\x01\\x07\\x04\\x01\\x00\\x01\\x0f\\x02\\x00\\x0e\\x04\\x07\\x04\\x06\\x05\\x00\\r\\x02\\x08\\x03\\x06\\x05\\x02\\x03\\x00\\x1b\\x0b\\x07\\t\\x02\\x00\\x02\\x03\\x00\\x05\\x06\\x0e\\x01\\n\\x01\\x00\\x17\\x0b\\x01\\x00\\x03\\x04\\t\\x02\\x00\\x07\\x01\\x00\\r\\x07\\x0b\\x00\\x10\\x08\\x01\\x05\\n\\x02\\x00\\x02\\x03\\x00\\x07\\x01\\x00\\x0c\\x01\\r\\x04\\t\\x01\\x07\\x00\\x1b\\x06\\x10\\x01\\n\\x04\\x03\\x19\\x0b\\x00$\\x08\\x01\\x00\\x03\\x02\\x05\\t\\x02\\x05\\x04\\x06\\x03\\x00\\x07\\x01\\x00\\x03\\x06\\x0e\\x01\\x07\\x04\\x03\\x00\\x04\\x01\\x00\\r\\x08\\x06\\n\\x0b\\x04\\x00\\x06\\x12\\x08\\x01\\x03\\x00\\x05\\x06\\t\\x01\\x12\\x07\\x02\\x00\\n\\x02\\x00\\x07\\x02\\t\\x02\\x08\\x01\\t\\x0b\\x08\\x00\\x0e\\x0b\\x03\\x07\\x04\\x0e\\x00\\x18\\x06\\x05\\x00\\x07\\x01\\x00\\x01\\n\\x06\\t\\x01\\x00\\n\\x02'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chunks_trans[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "alpha-apparatus",
   "metadata": {},
   "source": [
    "## Prepare data for ML\n",
    "Each language is represented in its own file. This code reads all the files and puts them into one dataframe. Works for anon or non-anon."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "surface-election",
   "metadata": {},
   "outputs": [],
   "source": [
    "# inpath is file location of input, outpath of output, and anon is bool whether data is\n",
    "# anonymized or not\n",
    "def prepdata(inpath, outfile, anon):\n",
    "    dfs = []\n",
    "    data = pd.DataFrame([], columns=['text', 'lang'])\n",
    "    readmode = 'r'\n",
    "    if anon: readmode = 'rb'\n",
    "    for f in os.listdir(inpath):\n",
    "        if f == 'README.md' or f.startswith('.'): continue\n",
    "#         print(f)\n",
    "        file = open(inpath+f, readmode) #rb if A, r if NA\n",
    "        chunks = [] # pickle load if A or readlines if NA\n",
    "        if anon: chunks = pickle.load(file)\n",
    "        else: chunks = file.readlines()\n",
    "        file.close()\n",
    "        lang = f[:f.index('.')]\n",
    "        df = pd.DataFrame([chunks, len(chunks)*[lang]], index=['text', 'lang']).T\n",
    "        dfs.append(df)\n",
    "    df = pd.concat(dfs, ignore_index=True)\n",
    "    dfs = None #clear memory\n",
    "    if not anon: df.text = df.text.apply(lambda x: x[:-1])\n",
    "    f = open(outfile, 'wb')\n",
    "    pickle.dump(df, f)\n",
    "    f.close()\n",
    "    print('anon:', anon, '\\tsanity check:', df.shape[0], 'chunks.', \n",
    "         df['text'].map(lambda x: len(x)!=500).sum(), 'not with 500 chars')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "ideal-crystal",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "anon: True \tsanity check: 1914517 chunks. 123 not with 500 chars\n"
     ]
    }
   ],
   "source": [
    "# anon\n",
    "inpath = './data/chunks-anon/'\n",
    "outfile= './data/chunks_shufanon.pkl'\n",
    "prepdata(inpath=inpath, outfile=outfile, anon=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "surrounded-kelly",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "anon: False \tsanity check: 1914517 chunks. 123 not with 500 chars\n"
     ]
    }
   ],
   "source": [
    "# nonanon\n",
    "inpath = './data/chunks-nonanon/'\n",
    "outfile= './data/chunks.pkl'\n",
    "prepdata(inpath, outfile, False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "absolute-amount",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}