-
Notifications
You must be signed in to change notification settings - Fork 1
Data compression
Adrian Wilke edited this page Feb 1, 2023
·
6 revisions
# 30,000 items
time python backup-es-index.py bLKL9v85dD4spiVuL7l= dbpedia_en_fr_15k_procrustes
real 0m16.789s
user 0m13.429s
sys 0m1.269s
ls -lah
-rw-r--r-- 1 wilke user 181M Feb 1 11:03 dbpedia_en_fr_15k_procrustes.embeddings.txt
-rw-r--r-- 1 wilke user 1.4M Feb 1 11:03 dbpedia_en_fr_15k_procrustes.uris.txt
-
time zstd -z -10 dbpedia_en_fr_15k_procrustes.embeddings.txt
71M / real 0m11.301s / user 0m11.303s / sys 0m0.240s -
time zstd -z -19 dbpedia_en_fr_15k_procrustes.embeddings.txt
64M / real 2m39.700s / user 2m39.398s / sys 0m0.400s
time zstd -d -o test.tmp dbpedia_en_fr_15k_procrustes.embeddings.txt.zst
real 0m0.801s / user 0m0.373s / sys 0m0.428s
- Zstandard (zstd)
- manpage
- Parameter
--fast=10
did not compress well. (Maybe try-#
)
zstd --version
*** zstd command line interface 64-bits v1.5.2, by Yann Collet ***
#
zstd -z --fast=10 dbpedia_wikidata_full.embeddings.txt
# ...
-rw-r--r-- 1 wilke user 107G Jan 28 00:04 dbpedia_wikidata_full.embeddings.txt
-rw-r--r-- 1 wilke user 101G Jan 28 00:04 dbpedia_wikidata_full.embeddings.txt.zst
-rw-r--r-- 1 wilke user 8.8G Jan 28 00:04 dbpedia_wikidata_full.uris.txt
-rw-r--r-- 1 wilke user 2.7G Jan 28 00:04 dbpedia_wikidata_full.uris.txt.zst