[feature] Allow conan.tools.* to be extended dynamically

[code-kungfu]

What is your suggestion?

While working on RAD Studio support for Conan (In extension to the following question here: #13447), it would be really helpful if we could do the following in a fresh install of Conan:

As a third party, install a plugin package which extends the tools namespace, registers internally in Conan, gives full access to the Conan config & recipe object and allowing recipes to import conan.tools.MyPlugin functionality.

An obstacle which would have to be decided upon, would be

1. How do we handle a case where the plugin isn't installed and a user executes a recipe which depends on this plugin?
2. Making sure the design and implementation is non-destructive to existing recipe infrastructure
3. Determine the security implications this feature might have

This feature is intended to avoid having to ship a customized version of Conan when supporting third party tools and to reduce the maintenance burden in the event the Conan team decides not to include support for a given tool stack.

Have you read the CONTRIBUTING guide?

• I've read the CONTRIBUTING guide

[memsharded]

Hi @code-kungfu

Thanks for the feedback.

I'd like to understand a bit better the proposal.
Do you mean installing extensions via pip install myextensions? If this is the case, what is the advantage of putting them into the from conan.tools namespace? This is something that is doable today, and we are aware of users doing it already, their recipes look like:

from conan.tools.cmake import CMake
from myorg_conan.tools.mything import MyTool

class Pkg(ConanFile):
    def build(self):
         mytool = MyTool(self)

If the idea is not via pip install, but instead some Conan mechanism, we had the possibility in Conan 1.X to load arbitrary Python code installed via conan config install, but it was removed, as this is better covered by python_requires. These python_requires have multiple advantages like resolved automatically, being versioned and the versions have an effect on the package_id, which is a major advantage, etc.

[code-kungfu]

Hello @memsharded

Thank you for the reply.

I should probably clarify what I mean by a fresh install of Conan.
I'm specifically referring to using the standalone installer packaged with it's own Python runtime on Windows.

To my basic understanding of how Python works, when a Python application is packaged like that, the Python environment is frozen and not extensible via pip?

I'm perfectly fine with the library lives in a separate namespace distinct from Conan, then it can be imported through a try-except block in the recipe. The only reason I suggested this, is I was under the impression a library would have to be under the top level conan namespace for a conan config or recipe object to be passed around.
E.g. I need to store and access additional config data in conandata.yml along with the recipe, which is RAD Studio specific, for a recipe to control what platforms and additional settings a particular package support which needs to be passed on in the MSBuild command line, where RAD Studio takes over and do the rest.

The RAD Studio support is based around a heavily modified version of conan.tools.microsoft, plus I have to ship a command line utility which assist in providing accurate environment details about the installed RAD Studio versions (Think vswhere).

python_requires I've looked at in the Conan documentation, but it wasn't clear to me if it can solve this problem - and resolve / install the plugin collection package during the execution of the recipe.
If it can, and at the same time sustain accurate versioning, this would be quite a benefit - and align with another plan I have to utilize Conan (If possible): Allow a dev environment to be captured for a given platform target and bundle it up in a Conan package. This would allow an accurate deployment and configuration of an immutable environment on a build machine or even if you're using GH runners.
The latter which is currently problematic.

Besides the actual support library, I'm also shipping templates for package creation, those I understand is already covered by the current pluggable infrastructure in Conan.
Would those be able to be distributed through the python_requires mechanism as well or would it have to be a separate installer, adding them to the .conan2/plugins directory?

Even though RAD Studio is a Windows only environment, targeting multiple platforms, the command line compilers which are invoked by MSBuild when running MSBuild externally, do work fine under Wine in a Linux container.
Being able to tell Conan, running under Wine to just go and download a specific version of a target platform, either pulled from your own Conan remote, or a remote provided by e.g. Embt - where everything is configured correctly would be fantastic.

[memsharded]

I should probably clarify what I mean by a fresh install of Conan.
I'm specifically referring to using the standalone installer packaged with it's own Python runtime on Windows.

To my basic understanding of how Python works, when a Python application is packaged like that, the Python environment is frozen and not extensible via pip?

Correct, this environment is not extensible, pip install will not work with the standalone executable. What it is possible is to actually package your own pip-installed packages inside a Conan standalone executable (we are aware of enterprise users that do that), by doing a pip install in a full Python environment, then running the python pyinstaller.py with the script that is in the root of the repo to create the standalone executable (it might need defining some extra --hidden-import to the user pip packages).

But this is not the main Conan distribution, but the majority of users will use Conan via pip install.

python_requires I've looked at in the Conan documentation, but it wasn't clear to me if it can solve this problem - and resolve / install the plugin collection package during the execution of the recipe.
If it can, and at the same time sustain accurate versioning, this would be quite a benefit -

python_requires is indeed recommended for most of the cases of "recipe custom extensions" in which users want to implement and reuse in their recipes different tools. They are distributed as Conan packages, so as long as the tools are in an available server, any recipe can do a python_requires = "myradtools/version and later use it, and the recipe will download and inject those requirements into the recipe, so the recipe code can use it. The embedded versioning in the recipes is quite important, as otherwise it is easy to break reproducibility, to be trapped in breaking changes (as they would require changing all the consumers at once), and to not be able to track and re-build binaries when desired when some of the relevant logic in the shared code changes. The python_requires solved most of the disadvantages that the Conan 1.X (in old versions of 1.X) way of allowing global Python code had.
Conan 2.X only maintained a "global" injection of Python code, via global "generators" that can be defined and installed via conan config install.

and align with another plan I have to utilize Conan (If possible): Allow a dev environment to be captured for a given platform target and bundle it up in a Conan package. This would allow an accurate deployment and configuration of an immutable environment on a build machine or even if you're using GH runners.

We are aware of users that used Conan packages to package the full toolchain (compilers, tools, etc). This might not always be trivial, for example for packaging VisualStudio build tools (both for the technical point of view, but also because licensing doesn't really allow to redistribute it), but we have even heard of users using MSBuild tools inside Conan packages. There are some working examples of toolchains like the mingw one in ConanCenter, but we also know that creating packages for more "system compilers", like gcc in Linux systems, that might have some overlaps with the system itself, can have its challenges too. Apparently it is easier to package in Conan cross-compilers, as those are more system-indepedent.

Besides the actual support library, I'm also shipping templates for package creation, those I understand is already covered by the current pluggable infrastructure in Conan.
Would those be able to be distributed through the python_requires mechanism as well or would it have to be a separate installer, adding them to the .conan2/plugins directory?

Templates for the conan new command if that is what you mean, can be distributed via conan config install/install-pkg mechanism, which work also for the standalone executable Conan, as they install those templates in the Conan home cache folder. The python_requires wouldn't be the recommended, that is recommended mostly for recipe re-use (inheritance, composition), but with a dependency in recipe-runtime, while the "template" approach, as I understand it, would be more related to the conan new templating system.