Skip to content

Data compression

Adrian Wilke edited this page Feb 1, 2023 · 6 revisions

Benchmark

# 30,000 items
time python backup-es-index.py bLKL9v85dD4spiVuL7l= dbpedia_en_fr_15k_procrustes
real    0m16.789s
user    0m13.429s
sys     0m1.269s
ls -lah
-rw-r--r-- 1 wilke user 181M Feb  1 11:03 dbpedia_en_fr_15k_procrustes.embeddings.txt
-rw-r--r-- 1 wilke user 1.4M Feb  1 11:03 dbpedia_en_fr_15k_procrustes.uris.txt
  • time zstd -z -10 dbpedia_en_fr_15k_procrustes.embeddings.txt
    71M / real 0m11.301s / user 0m11.303s / sys 0m0.240s
  • time zstd -z -19 dbpedia_en_fr_15k_procrustes.embeddings.txt
    64M / real 2m39.700s / user 2m39.398s / sys 0m0.400s
    time zstd -d -o test.tmp dbpedia_en_fr_15k_procrustes.embeddings.txt.zst
    real 0m0.801s / user 0m0.373s / sys 0m0.428s

zstd fast

zstd --version
*** zstd command line interface 64-bits v1.5.2, by Yann Collet ***
#
zstd -z --fast=10 dbpedia_wikidata_full.embeddings.txt
# ...
-rw-r--r-- 1 wilke user 107G Jan 28 00:04 dbpedia_wikidata_full.embeddings.txt
-rw-r--r-- 1 wilke user 101G Jan 28 00:04 dbpedia_wikidata_full.embeddings.txt.zst
-rw-r--r-- 1 wilke user 8.8G Jan 28 00:04 dbpedia_wikidata_full.uris.txt
-rw-r--r-- 1 wilke user 2.7G Jan 28 00:04 dbpedia_wikidata_full.uris.txt.zst
Clone this wiki locally