[GSK-2247] Add unit tests for nme implementation

[Inokinoki]

Compute the ME, d for normalization and NME from scratch and compare the results, using the first image of 300W indoor.

[linear]

GSK-2247 Benchmark the NME implementation

Implement a unit-test that benchmark the NME implementation against one of the numbers given in https://paperswithcode.com/sota/facial-landmark-detection-on-300w. Ideally if we can find someone the NME calculated only for one of the images contained in the 300W dataset (instead of benchmarking against the total NME of all images).

For reference for the NME definition, see section 3.1 of : https://arxiv.org/pdf/2210.07233v1.pdf

[rabah-khalek]

• For traceability and generalisability, could we use the 2D face_alignment model on one of the benchmark images from the 300W to generate TEST_MARKS and PREDICT_MARKS?
• I resolved the conficts due to last refactoring of tests but could you please also improve the coverage over the new validations we have for the metrics?
• If you think benchmarking with a NME number from the literature is tedious, we can drop it, and just rely on the fact we're testing the euclidean distances using different methods.

[Inokinoki]

Thanks for the review @rabah-khalek !

• For traceability and generalisability, could we use the 2D face_alignment model on one of the benchmark images from the 300W to generate TEST_MARKS and PREDICT_MARKS?

Yes, I will try to add the fixture for the model and a test dataset with one image. There is already the resources on my side. They are ignored by my local git repo somehow:

• I resolved the conficts due to last refactoring of tests but could you please also improve the coverage over the new validations we have for the metrics?

Doing it today ;)

• If you think benchmarking with a NME number from the literature is tedious, we can drop it, and just rely on the fact we're testing the euclidean distances using different methods.

I think it's very interesting. I am just wondering how to compare with the NME numbers from the literature, if our calculation is based on one image.
Their NME should be over the given test dataset, if I didn't misunderstand. Should we predict the full dataset with model in this case?

[rabah-khalek]

Their NME should be over the given test dataset, if I didn't misunderstand. Should we predict the full dataset with model in this case?

Yes I was thinking of using the whole 300W dataset

[Inokinoki]

It seems that face alignment returns different results in Python 3.10 (passed) and Python 3.11 (not passed):

=========================== short test summary info ============================
FAILED tests/test_benchmark_nmes.py::test_face_alignment_model - assert False
 +  where False = <function isclose at 0x7f6b13d32270>(0.06233510979950631, 0.06287962278002229)
 +    where <function isclose at 0x7f6b13d32270> = np.isclose
=================== 1 failed, 12 passed in 112.01s (0:01:52) ===================