Missing (optional) dependencies

[fabcor-maxiv]

As a result of investigating this: #874 (comment)

I now realize there are a bunch of imports in the code base that are not associated with a dependency in pyproject.toml.

As a side effect it triggers a whole bunch of errors and warnings when build the documentation (the auto-generated API part of the documentation).

Maybe we can fix that.

[fabcor-maxiv]

I suggest we add those dependencies as optional dependencies in pyproject.toml.

I am thinking we could also group them by facility and/or beamline.

So we could have optional groups of dependencies (also called extras) such as maxiv-biomax (would contain PyTango), esrf, desy, and so on.

But I need help with that, to figure out what facility is running what exactly, especially which versions so that they can be added to pyproject.toml and the lockfile with the correctly pinned version.

cc: @marcus-oscarsson

[fabcor-maxiv]

These are the missing imports I have identified so far:

• bliss https://pypi.org/project/bliss/
• devtools https://pypi.org/project/devtools/
  • mxcubecore/HardwareObjects/ESRF/queue_entry/ssx_foil_collection.py
  • mxcubecore/HardwareObjects/ESRF/queue_entry/ssx_base_queue_entry.py
  • mxcubecore/HardwareObjects/ESRF/queue_entry/ssx_big_foil_collection.py
  • mxcubecore/HardwareObjects/ESRF/queue_entry/ssx_chip_collection.py
  • mxcubecore/HardwareObjects/ESRF/queue_entry/test_collection.py
  • mxcubecore/HardwareObjects/ESRF/queue_entry/ssx_injector_collection.py
• grob https://www.grobgroup.com/
• lima2
• pyispyb_client Very early version of a PyISPyB client #713
  • mxcubecore/HardwareObjects/PyISPyBClient.py
• pymba https://pypi.org/project/pymba/
• qt Which binding? There are so many...
• redis https://pypi.org/project/redis/
  • for example: mxcubecore/HardwareObjects/DataPublisher.py
• SpecClient https://github.com/mxcube/specclient
• tine https://tine.desy.de/
• v4l2
• vapory https://pypi.org/project/Vapory/
• sardana https://pypi.org/project/sardana/
• epics ???
  • this does not appear because all imports seem to be protected in try blocks

[fabcor-maxiv]

Then maybe all those imports could/should be wrapped in a try block, in case it helps (I am not convinced, yet).

[AnnieHeroux]

Some facilities/beamline might be using a mix, so having specific groups of dependencies would be tedious. I am fairly certain that beamlines using Epics don't mix with Pytango/Taco/Sardana but they might use Exporter .... So far it seems to only impact the generation of the documentation.

[marcus-oscarsson]

As @AnnieHeroux pointed out its not uncommon that there is a mix of the various dependencies and that many of these are not exclusive to only one site. In that case each site has to maintain their section in the pyproject.toml file. So far, each site installs the dependencies they need (that are used in their site specific files) independently.

Maybe adding a section that is maintained by each site makes this work easier ?

I think its important that we keep things as straightforward as possible, it would probably not help if one needs to pass tons of things to --with.

For the try/except in general I would agree but in this case I would prefer that we get an exception for missing dependencies in site specific code. I think we should use try/except in non site specific code but then I would consider the dependency to be general.

[fabcor-maxiv]

Some facilities/beamline might be using a mix, so having specific groups of dependencies would be tedious.

@AnnieHeroux, I do not know that there would be a problem here. Each facility can maintain their own list of optional dependencies for themselves if they want to. And we can have some other list(s) that are based by topic.

Something like this (probably not this exact notation, I still need to figure this out):

[tool.poetry.extras]
bliss = ["bliss"]
pymba = ["pymba"]
redis = ["redis"]
sardana = ["sardana"]
tango = ["PyTango"]
v4l2 = ["v4l2"]
vapory = ["Vapory"]

maxiv = ["mxcubecore[sardana,tango]"]

In this example, maxiv is an optional group (an extra) that refers to other optional groups. But it could also be expressed like in the following:

maxiv = ["PyTango", "sardana"]

Goal is to have all dependencies clearly listed somewhere (in pyproject.toml if possible), so that one is not left scratching their head figuring out where a missing import comes from. Seems like the right thing to do, to me.

[marcus-oscarsson]

Regarding the above list :

• bliss: Only for BLISS beamlines currently ESRF (optional)
• devtools: Should in my opinion be (general)
• grob : Sample changer used by BESSY ? (optional)
• lima2: So far only ESRF (optional)
• pyispyb_client: Can be removed
• pymba: Qt EMBL HH, DESSY ? (optional)
• qt: Should be removed from mxcuebcore (optional)
• redis: General IMO
• SpecClient: Still used by BESSY I think, (optional)
• tine: EMBL HH, BESSY, DESSY, (optional)
• v4l2: can be removed
• vapory: EMBL HH, SOLEIL ? (optional)
• sardana: ALBA, MAX-IV, (optional)

A problematic details will be that each site (or even beamline) might want to use different versions of the same dependency,

[fabcor-maxiv]

@marcus-oscarsson Thanks for the list :)

A problematic details will be that each site (or even beamline) might want to use different versions of the same dependency,

Yes, I am also worried about this. But maybe there is a compromise to be made somewhere. We can investigate.

If you know which version ranges ESRF needs/wants for their optional dependenices, I can slowly start building this. We'll see how far we can get before things start to break (version conflicts).

[AnnieHeroux]

This one goes unnoticed because it was already in a try block in the module.

• epics: most facilities in America (North and South), Australia, Asia (?)

[fabcor-maxiv]

This one goes unnoticed because it was already in a try block in the module.

* epics: most facilities in America (North and South), Australia, Asia (?)

Thanks @AnnieHeroux, I added to my list. By chance, do you know what needs to be installed for the Python import to work, is it a package available on PyPI or conda?

[AnnieHeroux]

i tried pip install pyEpics
then tested by removing the block for the import epics in the Epics.py module
using make html was successful at showing the module.

Matt thinks many facility/beamlines may do their own thing about epics import (conda or Pypi or ???)

[fabcor-maxiv]

I started a draft pull request to experiment with this: #900