Scope setting for an initial prototype

[uellue]

The scope for first "test prototypes" should be wide enough to validate and improve the general concept and API. At the same time, it should be specific enough to be implemented quickly.

Idea collection for a first attempt:

Do a full correction and physical data analysis for the SiGe blobfinder analysis. https://nbviewer.jupyter.org/github/LiberTEM/LiberTEM/blob/260b20a6856f81b4ae7af93a16b5af635a5b5fee/examples/strainmap-SiGe.ipynb

• Dark, gain, hot, dead correction for detector data
  • What is faster: Sparse matrix after dark frame subtraction "in one go", or separate steps of multiplication and pixel fiddling?
• Scale bar and physical coordinate map in 3D sample coordinates for navigation dimension, including non-linear distortions of the scan coordinates
  • Pragmatic approach: make the mapping linear and put non-linear terms in the correction (next item)
  • The linear part is a very, very fast and simple affine transformation. --> Return value is the linear transformation matrix
  • Probably separate into mapping in focal plane coordinates, and then mapping from focal plane to sample coordinates? That would make the non-trivial 3D sample coordinate part optional.
  • Zero point for focal plane: Beam position at zero displacement of the scan coils?
• Include correction for sample drift and scan noise (actual values are not known at this time, include just to develop API)
  • Separate distortions from the projection from errors such as drift and scan noise.
  • The former can be a distortion polynom: https://en.wikipedia.org/wiki/Distortion_(optics)#Software_correction
  • The latter is a map of shape (y, x, 2) in the navigation dimension
    • The input file might contain several of such maps
• Scale bars and scattering angle map for detector frames, including aberrations of the projection
  • Same as scan: Main part as affine transformation, non-linear parts as error polynom and as explicit map in signal space
  • Separate in map from pixel coordinates to physical detector coordinates, and from detector coordinates to scattering angle
  • Reference center point is position of central beam at zero deflection
• Correct descan error

• Linear part: Affine transformation of nav space to center offset value
• Error term: Map of nav space to center offset to be added to linear part

• Fit descan error and generate new correction and user parameters
  • Match linear part of descan error to vacuum reference image (EMPAD field mapping should have suitable data)
• Size and elliptical distortion of the bright field disk as a function of scan position and position on the detector
  • This could be implemented with a "virtual illumination" that uses matrix optics plus an error term to project a source image through an aperture. For the time being, the source image can likely be a point on the optical axis. This introduces the lens errors that are a function of scan position because this approach can describe how the beam passes through the column away from the optical axis. Additionally, the image will be distorted by the projection, which is handled by the corrections described above.
• Scale bars and map of "pinhole camera" sample coordinates on the detector as a function of defocus

[uellue]

Probably it is a good idea to separate corrections by their origin and let the application add them up to make transparent and controllable where they come from. No "magic numbers".

[uellue]

Some of the corrections can likely be folded into a single processing step by correcting masks instead of data.

Detector correction

• For a dot product this can be implemented as a manipulation step of the mask stack together with a constant correction for dark intensity.
• See https://github.com/LiberTEM/LiberTEM/blob/ee80ca710bec44fb4907f6087be48b48c46cc818/src/libertem/corrections/detector.py#L184 and https://github.com/LiberTEM/LiberTEM/blob/ee80ca710bec44fb4907f6087be48b48c46cc818/tests/corrections/test_detector.py#L147 for an implementation
• For Fourier transforms, there is apparently no efficient way to perform gain correction in Fourier space: https://en.wikipedia.org/wiki/Convolution_theorem However, subtracting the dark frame is trivial: https://en.wikipedia.org/wiki/Fourier_transform#Linearity In any case, here the Fourier transform is by far the biggest item in terms of resource use so that the correction is not throughput-limiting.
• Consequence: An application should be able to apply detector corrections in various ways since there is significant optimization potential through folding the correction directly into some operations, while others work better if the detector correction is performed beforehand.

Scattering angle and similar coordinates

• Masks in physical coordinates, including distortions, can be generated by applying a filter to maps of those coordinates. Example: Calculating the scattering angle of each detector pixel allows to select pixels within a certain range to generate a bright field disk or virtual dark field detector.