dwarfs-0.9.3

Bugfixes

• v0.8.0 removed the implementation of the null decompressor under the assumption that it was no longer used; it was, however, still used when recompressing an image with null-compressed blocks. The change to remove the implementation was reverted and a new test case was added. Fixes github #193.

Performance

• Some more ricepp compression speed improvements. Also, the universal binaries for x86_64 now automatically choose a ricepp version based on CPU capabilities.

• For Windows, there's an experimental -ricepp package/binary. This contains a "hybrid" build where the ricepp library was built using clang and everything else using cl. This binary offers significantly faster ricepp compression. Decompression speeds are similar to the regular package/binary. If you don't care about compressing large amounts of FITS files on Windows, just stick to the regular package/binary.

dwarfs-0.9.2

Bugfixes

• (fix) v0.9.0 introduced an optimization where large files of equal size were only fully hashed for deduplication if the first 4K of their contents also produced the same hash. This introduced a bug causing an exception to be thrown when processing large hard-linked files. The root cause was that the data structure intended to be used for exactly this case was just never populated, and the fix was adding a single line to fill the data structure. The test cases didn't cover large hard-linked files, so this slipped through into the release. A new test case has been added as well.

• (fix) On Windows, when using Power Shell, the error message dialog for a missing WinFsp DLL was not shown when running dwarfs.exe. The workaround is to use the same delayed loading mechanism that's already used for the universal binary and show the error in the terminal. See also the discussion on github #192.

Features

• Added a --list option to dwarfsck. This lists all files in the files system image. When used with --verbose, the list also shows permissions, size, uid/git and symbolic link information. Fixes github #192.

• Added a --checksum option to dwarfsck. This produces output similar to the *sum programs from coreutils and can be used to check the contents of a DwarFS image against local files.

dwarfs-0.9.1

Bugfixes

• Invalid UTF-8 characters in file paths would crash mkdwarfs if these paths were displayed in the progress output. A possible workaround was to disable progress output. This fix replaces any invalid characters before displaying them. Fixes github #191.

• The CMakeLists.txt would bail out as soon as it discovered --as-needed in the linker flags. However, --as-needed is only a problem when combined with BUILD_SHARED_LIBS=ON. The check has been changed to only trigger if both conditions are met.

Other

• Minor speed improvements in ricepp compression.

dwarfs-0.9.0

Only two weeks since the last release, but another major milestone: DwarFS now runs on all major platforms, including macOS. There are no macOS binaries available for download, though, but the installation procedure is relatively simple and I'm really hoping for a Homebrew formula to be added soon.

The only other change since v0.8.0 is the addition of the ricepp compression algorithm and the fits categorizer, both of which are intended to be used together for efficiently compressing raw data in astrophotography.

dwarfs-0.8.0

After more than 600 commits, it's time for another major release. In addition to a long list of fixes, there are quite a few new features, most notably a categorization framework that allows identifying different categories of files and treating them differently. Right now, there are only two categorizers — pcmaudio and incompressible — but there are hopefully more to come. Along with the pcmaudio categorizer, support for FLAC compression has been added. This allows for large collections of uncompressed audio files to be archived efficiently, and also accessed efficiently: the DwarFS FUSE driver can decode a large audio file using multiple cores, something that cannot be done with a single compressed FLAC file.

The project code is now tested much more thoroughly; various new abstractions allow the command line interfaces to actually be covered by the unit tests.

Also, unlike many previous releases, images produced by this release will be compatible with older releases as long as they don't use new features like FLAC compression or history sections, which are unsuppored by older releases. The 0.7.3 and later releases will even deal with unknown sections and compression algorithms. Going forward, use of new features will be tracked by feature flags, so older releases can determine if the feature set used by a file system image is fully or partially supported.

Last but not least, the binaries can now be built with manual pages built-in. This is particularly useful on Windows, where man is not a thing, but also with the universal binaries if you don't have a full install and need to quickly check the manual. The manuals can be read using the --man option.

New Features

• Categorizer framework. Initially supported categorizers are pcmaudio (detect audio data & metadata and provide context for FLAC compressor) and incompressible (detects "incompressible" data). Enabled using the --categorize option.

• Multiple segmenters can now run in parallel and write to the same filesystem image in a fully deterministic way. Currently, a segmenter instance will be used per category/subcategory. This can makes segmenting multi-threaded in cases where there are multiple categories. The number of segmenter worker threads can be configured using --num-segmenter-workers.

• The segmenter now supports different "granularities". The granularity is determined by the categorizer. For example, when segmenting the audio data in a 16-bit stereo PCM file, the granularity is 4 (bytes). This ensures that the segmenter will only produce chunks that start/end on a sample boundary.

• The segmenter now also features simple "repeating sequence detection". Under certain conditions, these sequences could cause the segmenter to slow down dramatically. See github #161 for details.

• FLAC compression. This can only be used along with the pcmaudio categorizer. Due to the way data is spread across different blocks, both FLAC compression and decompression can likely make use of multiple CPU cores for large audio files, meaning that loading a .wav file from a DwarFS image using FLAC compression will likely be much faster than loading the same data from a single FLAC file.

• Completely new similarity ordering implementation that supports multi-threaded and fully deterministic nilsimsa ordering. Also, nilsimsa options are now ever so slightly more user friendly.

• The --recompress feature of mkdwarfs has been largely rewritten. It now ensures the input filesystem is checked before an attempt is made to recompress it. Decompression is now using multiple threads. Also, recompression can be applied only to a subset of categories and compression options can be selected per category.

• mkdwarfs now stores a history block in the output image by default. The history block contains information about the version of mkdwarfs, all command line arguments, and a time stamp. A new history entry will be added whenever the image is altered (i.e. by using --recompress). The history can be displayed using dwarfsck. History timestamps can be disabled using --no-history-timestamps for bit-identical images. History creation can also be completely disabled using --no-history.

• All tools now come with built-in manual pages. This is valuable especially on Windows, which doesn't have man at all, or for the universal binaries, which are usually not installed alongside the manual pages. Running each tool with --man will show the manual page for the tool, using the configured pager. On Windows, if less.exe is in the PATH, it'll also be used as a pager.

• New verbose logging level (between info and debug).

• Logging now properly supports multi-line strings.

• Show compression library versions as part of the --help output. For dwarfsextract, also show libarchive version.

• --set-time now supports time strings in different formats (e.g. 20240101T0530).

• mkdwarfs can now write the filesystem image to stdout, making it possible to directly stream the output image to e.g. netcat.

• Progress display for mkdwarfs has been completely overhauled. Different components (e.g. hashing, categorization, segmenting, ...) can now display their own progress in addition to a "global" progress.

• mkdwarfs now supports ordering by "reverse path" with --order=revpath. This is like path ordering, but with the path components reversed (i.e. foo/bar/baz.xyz will be ordered as if it were baz.xyz/bar/foo).

• It is now possible to configure larger bloom filters in mkdwarfs.

• The mkdwarfs segmenter can now be fully disabled using -W 0.

• mkdwarfs now adds "feature sets" to the filesystem metadata. These can be used to introduce now features without necessarily breaking compatibility with older tools. As long as a filesystem image doesn't actively use the new features, it can still be read by old tools. Addresses github #158.

• dwarfsck has a new --quiet option that will only report errors.

• dwarfsck with --print-header will exit with a special exit code (2) if the image has no header. In all other cases, the exit code will be 0 (no error) or 1 (error).

• The --json option of dwarfsck now outputs filesystem information in JSON format.

• dwarfsck has a new --no-check option that skips checking all block hashes. This is useful for quickly accessing filesystem information.

• The FUSE driver exposes a new dwarfs.inodeinfo xattr on Linux that contains a JSON object with information about the inode, e.g. a list of chunks and associated categories.

• Don't enable readlink in the FUSE driver if filesystem has no symlinks. This is mainly useful for Windows where symlink support increases the number of getattr calls issued by WinFsp.

• As an experimental feature, CPU affinity for each worker group can be configured via the DWARFS_WORKER_GROUP_AFFINITY environment variable. This works for all tools, but is really only useful if you have different types of cores (e.g. performance and efficiency cores) and would like to e.g. always run the segmenter on a performance core.

• The universal binaries are now compressed with a different upx compression level, making them slightly bigger, but decompress much faster.

Bugfixes

• Allow version override for nixpkgs. Fixes github #155.

• Resize progress bar when terminal size changes. Fixes github #159.

• Add Extended Attributes section to README. Fixes github #160.

• Support 32-bit uid/gid/mode. Also support more than 65536 uids/gids/modes in a filesystem image. Fixes gh #173.

• Add workaround for broken utf8cpp release. Fixes github #182.

• Don't call check_section() in filesystem ctor, as it renders the section index useless. Also add regression test to ensure this won't be accidentally reintroduced. Fixes github #183.

• Ensure timely exit in progress dtor. This could occasionally block command line tools for a few seconds before exiting.

• --set-owner and --set-group did not work properly with non-zero ids. There were two distinct issues: (1) when building a DwarFS image with --set-owner and/or --set-group, the single uid/gid was stored in place of the index and the respective lookup vectors were left empty and (2) when reading such a DwarFS image, the uid/gid was always set to zero. The issue with (1) is not only that it's a special case, but it also wastes metadata space by repeatedly storing a potentially wide integer value. This fix addresses both issues. The uid/gid information is now stored more efficiently and, when reading an image using the old representation, the correct uid/gid will be reported. Unit tests were added to ensure both old and new formats are read correctly.

• mkdwarfs is now much better at handling inaccessible or vanishing files. In particular on Windows, where a successful access() call doesn't necessarily mean it'll be possible to open a file, this will make it possible to create a DwarFS file system from hierarchies containing inaccessible files. On other platforms, this means mkdwarfs can now handle files that are vanishing while the file system is being built.

• mkdwarfs progress updates are now "atomic", i.e. one update is always written with a single system call. This didn't make much of a difference on Linux, but the notoriously slow Windows terminal, along with somewhat interesting thread scheduling, would sometimes make the updates look like a typewriter in slow-motion.

• utf8_truncate() didn't handle zero-width characters properly. This could cause issues when truncating certain UTF8 strings.

• A race condition in simple progress mode was fixed.

• A race condition in filesystem_writer was fixed.

• The --no-create-timestamp option in mkdwarfs was always enabled and thus useless.

• Com...

dwarfs-0.7.5

Bugfixes

• Fix crash in the FUSE driver on Windows when tools like Notepad++ try to access a file like a directory (presumably because this works in cases where the file is an archive). This is a Windows-only issue because the Linux FUSE driver uses the inode-based API, whereas the Windows driver uses the string-based API. While parsing a path in the string-based API, there was no check whether a path component was a directory before trying to descend further.

Other

• The universal binaries have been compressed using a different compression level (-9 instead of --best --ultra-brute) in upx. The compression ratio is slightly worse, but the decompression speed is significantly faster.

dwarfs-0.7.4

Bugfixes

• Fix regression that broke section index optimization introduced in v0.7.3. Fixes github #183.
• Add workaround for broken utf8cpp release. Fixes github #182.

dwarfs-0.7.3

This is a small incremental update over the 0.7.2 release adding a single new feature: forward compatibility. This means that the 0.7.3 release will be able to handle DwarFS file system images created with newer releases as long as these images don't use features that are not understood by the older binaries. Up until now, support for new features often triggered a file system version increment, rendering the images unusable with older binaries even if the features weren't actually used in the image. This fixes #158.

dwarfs-0.7.2

Bugfixes

• Fix locale fallback if user-default locale cannot be set. Fixes github #156.

dwarfs-0.7.1

Bugfixes

• Fix potential division by zero crash in speedometer.

Other

• New tool header.

• Source code cleanups.

• Updated static build procedure (see README).