Strategy for Rust/PyO3 dependencies

[cmungall]

This discussion is about the strategy of relying on rust wrapped in PyO3

Currently we have one such dependency:

• fastobo-py

We potentially have two more, depending on INCATools/semantic-sql#41:

• whelk.rs
• https://github.com/ozekik/lightrdf (UPDATE now pyoxigraph)

We (@justaddcoffee @caufieldjh) are exploring use of ensmallen as a plugin. We are also keeping our eye on horned-owl for OWL

I like this strategy of wrapping things that require speed. However, some users have had issues installing oak due to existing rust dependencies.

• Error when installing via oaklib: subprocess-exited-with-error fastobo/fastobo-py#306

In another thread, @LucaCappelletti94 mentions challenges in distributing python wrappings around:

• Different available instruction sets (e.g. AVX256)
• Different architectures (Mac M1, M2, Intel...)

This situation could be exacerbated if we add more dependencies, especially if each such library has its own way of distributing wheels etc.

One strategy is to make these optional dependencies / plugins. However, this has its own engineering costs, and may make debugging user issues harder.

A requirement for OAK is that it should be easy to install from PyPI using standard Python install mechanisms, and should be easily reused as a library for others. So while solutions involving conda/docker are welcome, these should not be necessary.

If we do add more rust/pyo3 dependencies, then I think we should have a best practice for releasing wheels etc, and we should make sure that all core dependencies follow these

[LucaCappelletti94]

For Ensmallen wheels, we have designed a custom build process that allows us to support in a single wheel all Python versions 3.6+, and wrap in the same wheel versions supporting AVX, non-AVX, plus handle recent architectures such as M1. If you encounter issues with these problems, I suggest contacting @zommiommy.

[cmungall]

We are currently experimenting with a rustsim dependency

please test!

https://github.com/INCATools/ontology-access-kit/releases/tag/v0.2.2-rc1