Target selection for the first MWS Stream Program (GD1)

[geordie666]

This PR adds code for the first MWS stream program (GD1). It builds on the work provided by @segasai's as part of Issue #812.

As a side-effort, this PR also updates any units that are written to files produced by desitarget to be FITS-compliant.

This is still work in progress, but the code to create the actual target files is in place.

Remaining work before merging this PR includes:

• Incorporating the external dependency on gaia_zeropoint mentioned in Issue Y4 Y5 MWS program selection #812.
• Determining how to merge the targets with the existing targets in the MTL ledgers.
• Adding final priorities, and numbers of observations, for the GD1 targets (once they're determined).

Possible additional work might include:

• Adding units for some of the new Gaia quantities that haven't previously been in the DESI data model.
• Adding code to read the LS sweeps (and match to Gaia DR3) per-sweep rather than per-stream. This would be useful if we extend targeting to multiple stream programs in the future.

@segasai and/or @sazabi4. I posted this PR before completely finishing my implementation in the hope that you could check my working. Could you please:

• Check that the cache in /global/cfs/cdirs/desi/target/catalogs/streamcache/gd1-dr9-cache.fits is in agreement with @segasai's database-derived cache.
  • In particular, that the Gaia quantities seem correctly matched to the Legacy Surveys sources.
• Check that the targets in /pscratch/sd/a/adamyers/dr9/2.7.0.dev5481/streamtargets/main/resolve/bright/streamtargets-gd1-bright.fits seem reasonable. The different classes are stored as bit-masks in the SCND_TARGET column, with the following interpretations:
  • Bit-nameGD1_BRIGHT_PM, bit-number 2**55, "Bright targets with Gaia in GD1 stream"
  • Bit-name GD1_FAINT_NO_PM, bit-number 2**56, "Faint targets without Gaia in GD1 stream"
  • Bit-name GD1_FILLER, bit-number 2**57, "Filler targets in GD1 stream"
• Try to determine whether we can make the cache smaller. Currently, the read_data() function provided with Issue Y4 Y5 MWS program selection #812 produces ~113M sources. This is manageable, but large.

[coveralls]

coverage: 53.312% (-2.6%) from 55.87%
when pulling 6b7b7df on ADM-stream-MWS
into 9e3bcd1 on main.

[geordie666]

@segasai: There's a new version of the target file here:

/pscratch/sd/a/adamyers/blat/dr9/2.7.0.dev5481/streamtargets/main/resolve/bright/streamtargets-gd1-bright.fits

It's very close, now, to your version:

• The faint sample is identical.
• The bright sample has on extra target in my list, which is one of the rounding-error/matching cases where an object is ~1.00 arcseconds from a Gaia object.
• Your version of the filler sample has an extra 16 targets that are, I think, removed by the desitarget checks on target brightness.

Let me know if you think this is good enough to proceed.

[segasai]

Thank you @geordie666 ! I think those numbers look good enough for me!

[geordie666]

I think I've now finished the code for the end-to-end process of adding the GD1 stream targets. Previously I had:

• Written a reasonable cache.
• Written reasonable targeting files.

I've now also written code to add the new targets to the MTL ledgers. The overall process is as described in this thread on Slack. New targets are added to the secondary ledgers, and existing targets are merged with primary/secondary targets adopting the state of the highest-priority target.

An example of how adding the new targets would look after running on the current MTL ledgers is here:

/pscratch/sd/a/adamyers/mtlsandbox/mtl

And an example of "merging" a new and old target is:

grep 39627660212571777 /pscratch/sd/a/adamyers/mtlsandbox/mtl/main/bright/mtl-bright-hp-10232.ecsv | awk '{print $7, $8, $9, $10, $11, $15, $16, $24, $27}'

39627660212571777 2305843017803628544 0 1280 0.24369827786837128 0 2 MWS_MAIN_BLUE|UNOBS 1500
39627660212571777 6917529036231016448 0 1280 0.2004516496327986 36028797018963968 4 GD1_BRIGHT_PM|UNOBS 1520

where the printed columns are:

TARGETID DESI_TARGET BGS_TARGET MWS_TARGET SUBPRIORITY SCND_TARGET NUMOBS_MORE TARGET_STATE PRIORITY

I think I'm basically ready to merge and tag the code and process the new targets. Before I do that, though, we should finalize whether these are the correct priorities and numbers of observations for the GD1 stream targets:

https://github.com/desihub/desitarget/blob/ADM-stream-MWS/py/desitarget/data/targetmask.yaml#L420-L422
https://github.com/desihub/desitarget/blob/ADM-stream-MWS/py/desitarget/data/targetmask.yaml#L574-L576

[geordie666]

This PR is now almost ready to merge. I'm waiting on exact priorities for the new target classes from the Milky Way Group, then I'll proceed.