a Python DSL for TOSCA 1.3

[aszs]

Hi all, 

I'd like announce a new TOSCA project I've been working on, a Python DSL for TOSCA 1.3.  It is an alternative spelling of the YAML syntax, and is purely syntactic -- it doesn't make any changes TOSCA's semantics and converts from Python to YAML and from YAML to Python. Its intent is to make TOSCA easier to learn and easier to develop, in particular, by enabling IDE integration and other tooling.
Here are a few examples to give you a sense of how it looks, the readme for the project has more examples and documentation. 

I'll start with a translation of the example in the TOSCA 1.3 spec that introduces node types, it looks like:

node_types:
  WordPress:
    derived_from: tosca.nodes.WebApplication
    properties:
        admin_user:
          type: string
        admin_password:
          type: string
    requirements:
      - database_endpoint:
          capability: tosca.capabilities.Endpoint.Database
          node: tosca.nodes.Database
          relationship: tosca.relationships.ConnectsTo

In our Python spelling, this is translated to:

import tosca
class WordPress(tosca.nodes.WebApplication):
    admin_user: str
    admin_password: str

    database_endpoint: tosca.relationships.ConnectsTo | tosca.nodes.Database | tosca.capabilities.EndpointDatabase

Here you see we can infer its requirement from Python's type annotations.

Here's a more advanced example demonstrating how we map artifacts, operations and custom interfaces to Python classes and methods:

  class MyCustomInterface(tosca.interfaces.Root):
      class Inputs(ToscaInputs):
          location: str
          version: int = 0

      def my_operation(self):
          "description of my_operation"
          ...  # an abstract operation, subclass needs to implement 

  class Example(tosca.nodes.Root, MyCustomInterface):
      shellScript = tosca.artifacts.ImplementationBash(file="example.sh")
      location: str
      host: tosca.nodes.Compute

      def my_operation(self):
          return self.shellScript.execute(
              MyCustomInterface.Inputs(location=self.location),
              host_address=self.host.public_address
          )

This will be translated to the following YAML as:

interface_types:
  MyCustomInterface:
    derived_from: tosca.interfaces.Root
    inputs:
      location:
        type: string
      version:
        type: integer
        default: 0
    operations:
        my_operation:
          description: description of my_operation

node_types:
    Example:
        derived_from: tosca.nodes.Root
        artifacts:
          shellScript:
            type: tosca.artifacts.ImplementationBash
            file: example.sh
        properties:
          location:
            type: string
        requirements:
        - host:
            node: tosca.nodes.Compute
        interfaces:
          MyCustomInterface:
            type: MyCustomInterface
            operations:
              my_operation:
                implementation:
                  primary: shellScript
                inputs:
                  location:
                    {get_property: [SELF, location]}
                  host_address:
                    {get_property: [SELF, host, public_address]}

We can express fairly sophisticated constraints between nodes by declaring a _set_constraints() method, for example:

    from tosca import in_range, gb

    class Example(tosca.nodes.Root):
        host: tosca.nodes.Compute

        @classmethod
        def _set_constraints(cls) -> None:
            in_range(2 * gb, 20 * gb).apply_constraint(cls.host.host.mem_size)

This is designed this way to enable static type checking -- in this case if the "mem_size" property isn't compatible with TOSCA GB units your IDE will detect that error, as will Python tools like mypy.
This translates into the following

node_types:
  Example:
    derived_from: tosca.nodes.Root
    requirements:
    - host:
        node: tosca.nodes.Compute
        node_filter:
          capabilities:
          - host:
              properties:
              - mem_size:
                  in_range:
                  - 2 GB
                  - 20 GB

(I'm sticking with TOSCA 1.3 here, but you can do a lot more with constraints with Unfurl's extensions.) 

Finally, a couple of more features of note:

• TOSCA repositories and imports are mapped to Python packages and import statements; each service template is converted to a separate Python module.
• Conversion from Python to YAML is executed in a secure Python sandbox for untrusted files (Unfurl Cloud needed that!).

If you're interested in playing with this, it's integrated into the latest release of Unfurl and Unfurl Cloud but I've also released it as standalone Python package at https://pypi.org/project/tosca/

Any questions or feedback is much appreciated!

Thanks,
Adam

[tliron]

This is great! And I must humble-brag that I started working on something similar a few years ago, but abandoned it...

I don't know if you're up to date on the latest TOSCA 2.0 advances, but we've much improved functions and also support custom functions. It makes a lot of sense to me to be able to annotate a Python function and have it immediately available in TOSCA.

We're also working on a richer event model for TOSCA, which may involve custom operation "implementations" (actually event handlers), again an easy fit for annotating a Python function to handle them.

And, finally, we do have a proposal (from me, for now) to support workflows in TOSCA using some pseudo-programming. Once again, a great fit for Python.

The killer feature, for me, would be that you could do everything in pure Python (custom functions, handlers, workflows) and then a tool can export it all to a CSAR with the independent Python scripts. Obviously something like that won't run everywhere, but it would be largely compatible for other parsers who of course would need YAML. It seems like you're already on that path.

[aszs]

The killer feature, for me, would be that you could do everything in pure Python (custom functions, handlers, workflows) and then a tool can export it all to a CSAR with the independent Python scripts.

Actually, you can almost do that right now. Unfurl already supports syntax for the implementation field in YAML like helper.py#MyConfigurator or modulename.myconfigurator (where Configurator is our interface for running operations).

The plan is that operations in the DSL can return an function instead of an artifact and the YAML converter will replace it with a reference to that function using the above syntax, something like:

class MyNodeType(tosca.nodes.Root):
  def do_stuff(self, task):
    return Status.ok

  def create(self):
    return self.do_stuff

I'm haven't been keeping abreast with the latest for TOSCA 2.0 but hopefully we could do something like with some standard set of bindings instead of relying on Unfurl extensions. Unfurl also has a couple other Python extension points for expressions and dynamic workflows but I haven't thought too much about how to integrate them into the DSL.