package.dox

namespace dyscostman {

/** 
@mainpage DyscoStMan compressing storage manager
The DyscoStMan (Dynamical Statistical Compression Storage Manager) is
a CASA storage manager that can compress visibilities and weights. 

The compression technique is explained in Offringa (2016;
https://arxiv.org/abs/1609.02019).

Visibilities can be stored with a non-linear quantization scheme,
such as a least-squared error quantizer for Gaussian values. "Error"
is here the difference between original and encoded value. The
encoding scheme is created by the GausEncoder class.

An important assumption for this encoder to work is that the data
is actually noisy, so its signal-to-noise ratio is low. 
Therefore, the higher the resolution, the higher the compression
rate can be. If the SNR is very high, such as for a calibrator
observation with a low time and frequency resolution, it might
be necessary to copmress with higher bit-rates.
Also, other compression techniques might be more relevant in
such cases (e.g. simple Hoffman encoding).

Weights are stored with a simple linear quantization scheme, but each
timestep is individually scaled. This encoding is performed by the
WeightEncoder class.

Tests have shown that typical MWA and LOFAR data can be encoded with
less than 10 bits per visibility-float. Weights need in principal fewer
bits, but because only one weight per four polarizations are stored
(in the case of full Stokes observations), it is not necessary to
compress the weights very much further.

The main class DyscoStMan is the actual storage manager. It uses
its own file format, which consists of a header (Header), followed
by compressed timeblocks.

This package provides the following tools:
- @em libdyscostman.so : the library that contains the storage manager.
This file should be in your library path in order for Casa to load it automatically.
- @em dscompress : executable that will change the storage manager for the
DATA and WEIGHT_SPECTRUM columns into the DyscoStMan.
- @em decompress : executable that changes the storage manager of columns
with a DyscoStMan into the DefaultStMan.

<h2>Using dscompress</h2>
The dscompress executable will rewrite the DATA and WEIGHT_SPECTRUM column, by
first renaming them and creating new columns with this storage manager, then
it will copy all the values in and then it will remove the two old columns.
Optionally, the whole set is copied, to make sure that the space that was taken
by the old columns is freed.

<h2>Using decompress</h2>
The decompress executable will just do the same as dscompress, but will use
a default storage manager for the columns.

Usage:
@code{bash}
decompress <measurement set>
@endcode

@author André Offringa (offringa@gmail.com)
@copyright 2013-2016, published under GPL version 3
*/

}