Add support for loading files in BZip2 and XZ formats

[munckymagik]

Hi. This is my first PR to any Julia project so happy to be guided to a more idiomatic solution. But this is my first pass at making it work. Thanks.

TODO list:

• Test error handling for unsupported compressors (e.g. zip)
• Silence the error output from FileIO for tests that we intend to cause failures
• Throw a custom MissingCodecError or check the error message explicitly.
• Squash commits

[coveralls]

Coverage increased (+1.9%) to 75.526% when pulling c6b1d12 on munckymagik:support_all_compression_formats into 1f4f7e2 on JuliaData:master.

[alyst]

Thank you! Overall it looks quite ready, but I think we would need to make CodecXz.jl and CodecBzip2.jl dependencies optional. Otherwise we will introduce a lot of unnecessary and unexpected dependencies to a lot of packages and workflows (e.g. to RDatasets.jl, which is required by test suites of many other packages).

Ideally, it would be nice if FileIO.jl or TranscodingStreams.jl have implemented the mechanism of automatically using the appropriate decompressor stream on the file (or reporting if it's not available). But AFAIK there's nothing like that at the moment (see also JuliaIO/FileIO.jl#225).

We can implement our own mechanism using Requires.jl. You can take a look how PlotlyBase.jl is using Requires.jl for an optional Colors.jl dependency. You can define e.g.

abstract type ArchiveFormat{S} end

function decompressor_stream(::Type{ArchiveFormat{S}}, io::IOStream) where S
    if S == "BZIP2"
       error("CodecBzip2 package is required to read Bzip2-compressed RData files. Use Pkg.add(\"CodecBzip2\").")
    elseif S=="XZ"
       error("CodecXz package is required to read XZ-compressed RData files. Use Pkg.add(\"CodecXz\")")
    else
       throw(ArgumentError("Unsupported archive format: $S"))
    end
end

decompressor_stream(::Type{ArchiveFormat{"GZIP"}}, io::IOStream) = GzipDecompressorStream(io)

and decompressor_stream() for BZip2 and Xz in __init__(). Then you would be calling decompress_stream(ArchiveFormat{format}, io) from decompress(io). Of course, this code is just a sketch and it's quite likely that you can come up with a better solution :)

[munckymagik]

@alyst I've had a go at implementing your request. It appears to work but now I get the following message when running tests in my code that uses RData:

Is that expected? (I have done Pkg.resolve().

From a design perspective I tried several approaches with regard to placement of the __init__ function. What I settled on in the end was to put it in a sub-module dedicated to defining the decompressor_stream functions, so everything related is together.

What do you think? Let me know your suggestions from here 👍

[alyst]

Thank you! Looks great, I've just added some cosmetics and noted a few minor things:

• tests for other archives (zip)
• whether we really need additional tiny submodule
• whether we need using CodecXXX in the user code

[alyst]

Thank you! We're almost there. There's one typo in the method name + some comments.
I'm sorry that optional dependencies are causing so much work. I think the only requirements for merging this PR is adding zip test and all green for CI.

[munckymagik]

Just pushed some commits. Super tired right now due to travelling. Will check everything over tomorrow to see what's left and reply properly.

[munckymagik]

Hi @alyst

Sorry for the time it's taken to finish this.

All the issues we discussed have been addressed, so it's ready for another review.

Cheers 👍

[alyst]

Great, thanks for you hard work!
I'm approving it, but could you please change the indentation of the new files to use 4 spaces before I merge the PR? No need to re-squash as I'll squash it upon merging.

[alyst]

Could you please change it to 4-space indent?

[munckymagik]

Doh 🤦‍♂ there's always something. Thanks for spotting.

[alyst]

4-space indent here as well

[munckymagik]

Ok done. Thank you 👍

[alyst]

Failures in AppVeyor for julia-latest seem unrelated, so I'm merging.
Thanks again, @munckymagik, Xz and Bzip2 support is quite important!