-
Notifications
You must be signed in to change notification settings - Fork 0
starnux/hashstor
Folders and files
Name | Name | Last commit message | Last commit date | |
---|---|---|---|---|
Repository files navigation
hashstor concept : ================== The project concept is to store files as shared chunks by their hash. The flow is : You take a list of files -> Each file has it's proper hash --> Each file hash is divided in N block ---> Each block has it's proper hash So, the file hashs can be shared, and so the block hashs. This system can be useful for backup : - Unchanged files are not stored another time - Only changed blocks of changed files are stored - Exact same block between differents files are shared - The blocks can be compressed The file structure should be : /path/to/storage/ |--- .hashstor |--- blocks/ | |---- dd5225c85b50b1105c05ffe9c216cb0b17a18285.xz | |---- 24908d6593ab6fad9c4ea851f2f7958a07c17638.xz | |---- 77feb757012c3c0243769fc4164303ce4e5d36c4.xz |--- files/ | |---- 64d16cfbb9f89dbe40cfb3dfeb16c9715092608f | |---- 21c63c61509bbb4e64d8cef34a9a63c761cbcce5 |--- storages/ |---- default |---- first-storage The .hashstor indicates this directory is a hashstor storage point, it contains the storage point configuration like chunk size. The blocks files are zlib compressed chunks. The chunks size is 16KiB, but the end of file != 16KiB is also stored in a chunk file. Files hashs describes the file options (size, name) and the list of associated hashs in text file format : Sample of files/64d16cfbb9f89dbe40cfb3dfeb16c9715092608f ========================================= dd5225c85b50b1105c05ffe9c216cb0b17a18285 24908d6593ab6fad9c4ea851f2f7958a07c17638 ========================================= Storages file describes the files contained in the storage. Sample of storages/default : ========================================= 5307 64d16cfbb9f89dbe40cfb3dfeb16c9715092608f README.txt 64d16cfbb9f89dbe40cfb3dfeb16c9715092608f 742 21c63c61509bbb4e64d8cef34a9a63c761cbcce5 LICENCE.txt 21c63c61509bbb4e64d8cef34a9a63c761cbcce5 ========================================= The utilities should permit to : - Initialize a storage point - Setup a new storage container name with an associated UUID - Add files to a storage container - Extract a single storage container from the storage point to a new or existing storage point - Delete a storage container and cleanup unused blocks - List & Extract files from a container - Check storage point consistency, or cleanup if a storage failed previously - Add files to the storage point via SSH, permit the sender to get all block hashs and files hashs in order to send only unknown blocks
About
Shared block hash-based file storage
Resources
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published