2022/10/06 Meeting Notes

[Victoria-Samboco]

Hi everyone, here are the notes for the today meeting:

1. We first show the flags from 1GC I had to redo (to reduce the percentage of flags), and the flagging have reduced significantly from ~96% to an average of 30% as we can see below:
   
   The flaggings were reduced changing the flagger from aoflagger to Tricolour and using gorbachev.yaml strategy. Some flags as for 2 problematic scans (23 and 45) and a problematic channel ('*:1007.9~1100.0MHz') were added.

2. Proceed to Selfcal and DD-cal - after reducing the percentage of flagged data, the selfcal solution proved to be better than the selfical before (with 96% flagged, which led to the need to do DD-cal). What made me think that in the current situation it is not necessary to do DD-cal (concluded after comparing selfcal and dd-cal images which proved to be worse than the self-calibrated image).
   Bellow are the Self-calibrated data with 96% of data flagged and now (with 30% of data flagged) respectively :
   

Comparison between the actual self-calibrated and dd-calibrated data (as we can see bellow the selfcal image are better than the dd-cal data)

3. Imaged the Sun with the DD-calibrated data to see how it looks. Some scans show large amplitude fluctuations, which for now the reasons are unknown, but maybe it could be caused by peeling, and this will be investigated by comparing the Sun images with SELFCAL data and with DD-cal data.
   scans 3-9
   

Scan 7 Amplitude vs Frequency in time.

scabs 11-17

scans 19 - 25

Scan 25 Amplitude vs Frequency in time.

And scans 29 - 35

Next steps after what was discussed:

1. Image the Sun with the Self-calibrated data (to compare with the dd-cal data Sun image); PS: If the selfcal-data doesn't show good results for Sun images comparatively to the k,de dd-calibrated images proceed to the step 2 bellow;
2. Once the dd-calibrated image is worst than the self-calibrated one ( assumed that it might be because of the solver type and the solutions intervals, and the k, de is apparently not working, will be trying run dd-cal with k,G,de solvers to see if there's any improvement.

[landmanbester]

@Victoria-Samboco if you used QuartiCal for the calibration and asked it to write out residual visibilities you can use surfvis

https://github.com/ratt-ru/surfvis

to locate bad data. Have a look at the surfchi2 worker eg.

$ surfchi2 --help
Usage: surfchi2 [options] msname

Options:
  -h, --help            show this help message and exit
  --rcol=RCOL           Residual column (default = RESIDUAL)
  --wcol=WCOL           Weight column (default = WEIGHT_SPECTRUM). The special
                        value SIGMA_SPECTRUM can be passed to initialise the
                        weights as 1/sigma**2
  --fcol=FCOL           Flag column (default = FLAG)
  --dataout=DATAOUT     Output name of zarr dataset
  --imagesout=IMAGESOUT
                        Output folder to place images. Saved in CWD/chi2 by
                        default.
  --nthreads=NTHREADS   Number of dask threads to use
  --ntimes=NTIMES       Number of unique times in each chunk.
  --nfreqs=NFREQS       Number of frequencies in a chunk.
  --use-corrs=USE_CORRS
                        Comma seprated list of correlations to use (do not use
                        spaces). Default = diagonal correlations

It will produce a directory tree with plots like this one

The numbers on the left and bottom of the image on the left are antenna numbers and the histogram on the right is a histogram of the average chi-square. The outliers with large chi-square should probably be flagged (eg. in the above example you can see that there is bad data affecting the baseline defined by antenna 52 and 29. You can flag it with the flagchi2 worker

$ flagchi2 --help
Usage: flagchi2 [options] msname

Options:
  -h, --help            show this help message and exit
  --rcol=RCOL           Residual column (default = RESIDUAL)
  --wcol=WCOL           Weight column (default = WEIGHT_SPECTRUM). The special
                        value SIGMA_SPECTRUM can be passed to initialise the
                        weights as 1/sigma**2
  --fcol=FCOL           Flag column (default = FLAG)
  --flag-above=FLAG_ABOVE
                        flag data with chisq above this value (default = 3)
  --unflag-below=UNFLAG_BELOW
                        unflag data with chisq below this value (default =
                        1.15)
  --nthreads=NTHREADS   Number of dask threads to use
  --nrows=NROWS         Number of rows in each chunk (default=10000)
  --nfreqs=NFREQS       Number of frequencies in a chunk (default=128)
  --use-corrs=USE_CORRS
                        Comma seprated list of correlations to use (do not use
                        spaces)
  --respect-ants=RESPECT_ANTS
                        Comma seprated list of antennas to respect (do not use
                        spaces)

Let me know if you want to try it out then I can give you a hand

[Victoria-Samboco]

With the results that I obtained through the SELF-CALIBRATED data, I believe it is no longer necessary to go through the peeling process, judging by the images of the Sun obtained now (from the selfcal data) - (Please correct me if I'm wrong). But of course I'm open to experimenting to see what it can do.

I will plot the visibilities of the problematic scans on the post-peel data but now for the self-calibrated data for comparison.