JointWriter: Allow shard file appending

[janEbert]

I am working on a file system that loves few huge files and hates many small files. To this end, I would simply set size_limit=None when creating a dataset using a JointWriter. However, shards are only flushed (data written to disk and freed from RAM) once the size_limit is reached. This means I cannot create shards greater than my RAM (because the data in RAM keeps growing and is never flushed). This becomes especially apparent when I write using multiple processes on the same node.

I'd love it if, even with an unlimited shard_size, shard files could be partially written so that I can create shards larger than RAM. I would personally be fine with only MDSWriter and limited compressions supporting this. It seems like its encode_joint_to_shard implementation could support this.

Is this a feature you would accept contributions for or would it create too much maintenance workload with regard to various settings (compression etc.)?

[XiaohanZhangCMU]

@janEbert thanks for bringing the issue up. what fs are you using? You may want to create a new Writer subclass for this particular use case.

And yes, a feature contribution like this is always welcome!

[janEbert]

It's a GPFS. Thanks for the idea to maybe subclass the MDSWriter! Probably the simplest solution in both implementation and maintenance. :)
Cheers for the openness to contributions!