Data compression

Benchmark

# 30,000 items
time python backup-es-index.py bLKL9v85dD4spiVuL7l= dbpedia_en_fr_15k_procrustes
real    0m16.789s
user    0m13.429s
sys     0m1.269s
ls -lah
-rw-r--r-- 1 wilke user 181M Feb  1 11:03 dbpedia_en_fr_15k_procrustes.embeddings.txt
-rw-r--r-- 1 wilke user 1.4M Feb  1 11:03 dbpedia_en_fr_15k_procrustes.uris.txt

• time zstd -z -10 dbpedia_en_fr_15k_procrustes.embeddings.txt good choice
  71M / real 0m11.301s / user 0m11.303s / sys 0m0.240s
• time zstd -z -14 dbpedia_en_fr_15k_procrustes.embeddings.txt
  71M / real 1m8.196s / user 1m8.090s / sys 0m0.321s
• time zstd -z -19 dbpedia_en_fr_15k_procrustes.embeddings.txt
  64M / real 2m39.700s / user 2m39.398s / sys 0m0.400s
  time zstd -d -o test.tmp dbpedia_en_fr_15k_procrustes.embeddings.txt.zst fast, overall irrelevant
  real 0m0.801s / user 0m0.373s / sys 0m0.428s
• time bzip2 -z -k -9 dbpedia_en_fr_15k_procrustes.embeddings.txt
  69M / real 0m21.442s / user 0m21.075s / sys 0m0.280s
  time bzip2 -dc dbpedia_en_fr_15k_procrustes.embeddings.txt.bz2 > test.tmp fast, overall irrelevant
  real 0m11.573s / user 0m10.792s / sys 0m0.684s
• time bzip2 -z -k -5 dbpedia_en_fr_15k_procrustes.embeddings.txt
  69M / real 0m21.126s / user 0m20.726s / sys 0m0.302s

Large files

-rw-r--r-- 1 wilke user 107G Jan 28 00:04 dbpedia_wikidata_full.embeddings.txt
-rw-r--r-- 1 wilke user 8.8G Jan 28 00:04 dbpedia_wikidata_full.uris.txt
#
time zstd -z -10 dbpedia_wikidata_full.uris.txt
dbpedia_wikidata_full.uris.txt : 13.48%   (  8.73 GiB =>   1.18 GiB, dbpedia_wikidata_full.uris.txt.zst) 
real    3m55.641s
user    3m56.080s
sys     0m14.077s
#
time zstd -z -10 dbpedia_wikidata_full.embeddings.txt
dbpedia_wikidata_full.embeddings.txt : 39.22%   (   106 GiB =>   41.6 GiB, dbpedia_wikidata_full.embeddings.txt.zst)
real    128m4.090s
user    122m21.490s
sys     3m46.268s
...
#
-rw-r--r-- 1 wilke user  42G Jan 28 00:04 dbpedia_wikidata_full.embeddings.txt.zst
-rw-r--r-- 1 wilke user 1.2G Jan 28 00:04 dbpedia_wikidata_full.uris.txt.zst
#
Note: Before with bzip2 and high compression (was really slow)
-rw-rw-r-- 1 wilke wilke  42G Jan 25 22:01 dbpedia_wikidata_full.embeddings.txt.bz2
-rw-rw-r-- 1 wilke wilke 1,2G Jan 25 22:01 dbpedia_wikidata_full.uris.txt.bz2

Test: zstd --fast

• Zstandard (zstd)
• manpage
• Parameter --fast=10 did not compress well. (Maybe try -#)

zstd --version
*** zstd command line interface 64-bits v1.5.2, by Yann Collet ***
#
zstd -z --fast=10 dbpedia_wikidata_full.embeddings.txt
# ...
-rw-r--r-- 1 wilke user 107G Jan 28 00:04 dbpedia_wikidata_full.embeddings.txt
-rw-r--r-- 1 wilke user 101G Jan 28 00:04 dbpedia_wikidata_full.embeddings.txt.zst
-rw-r--r-- 1 wilke user 8.8G Jan 28 00:04 dbpedia_wikidata_full.uris.txt
-rw-r--r-- 1 wilke user 2.7G Jan 28 00:04 dbpedia_wikidata_full.uris.txt.zst