Integrating Airflow into tdp-collections-extra Stack: Open Discussion

[gonzaloetjo]

Airflow PR Overview:

We're contemplating the inclusion of Airflow into the extras collection. Here a general list of components implemented:

• Scheduler: Responsible for monitoring and ensuring scheduled execution of all tasks. Decides where and when tasks are run.

• Webserver: A Flask server that serves the Airflow UI. Helps monitor, trigger and debug DAGs.

• Broker: Facilitates communication between the Airflow Scheduler and the Workers, handling task messages and their status updates.

• Executor: Responsible for determining how tasks are run, in parallel or sequentially, locally or distributed.

• Flower: Used to monitor task progress and history in celery executors.

• Workers: Execute the tasks. They pick up and run tasks sent to the queue by the executor.

• Database (Metadata DB): Stores metadata about the state of tasks and workflows, and assists in recovery in case of failures.

• Dag Directory: Folder of DAG files. It is read by the scheduler and executor, and has to exist in every worker as well.

  

Current work in the PR

• Installs packaged airflow 2.2.5 or 2.6.2, and all its dependencies, through a virtual-environment and pip-install
• Installs extra required tools in the package:
  • Redis as a broker.
  • Flower to monitor executors
  • Celery as executor.
  • Providers: Hive, Spark, HDFS, Kerberos
• Creates services for web-server, scheduler, workers, redis, flower
• Security implementations:
  • Policies: Currently used to limit what a user (ie. tdp_user) can do when editing a dag.
  • Roles: Used to limit what a user/group can view or do within a dag.
  • Dag Directory structure: Working on dag-folder structure to handle groups/users.
• Creates connectors with impersonation capabilities for Hive, Spark and HDFS
• Example dags for Hive, Spark and HDFS

The choice we make should primarily consider our specific needs within the extra, along with the stability and community support associated with the options.

---

Discussion Points:

Given Airflow's flexible implementation, this decision presents different discussion points:

Version Selection

There are two potential versions to consider:

• Airflow 2.2.5 (Python 3.6): This version might offer greater compatibility and stability due to its maturity.
• Airflow 2.6.2 (Python 3.9): The latest version, equipped with numerous tools and bug fixes, and can be deployed within a virtual environment.

Both can be left as an option (through tdp_vars), this is the current implementation. But we consider airflow 2.2.5 to be quite limited in comparison to the newer versions. One of the major motivations for us to integrate later versions were the capability to impersonate the owner of the dag when running tasks using different connectors (for instance Hive tasks).

Architectural Considerations

• Executor: The choice between Celery and Local executors should be made by the requirements for resource management and scalability. Local executor is simply used for testing and development, while Celery is the go to in regards to production in non containerized applications. We have to choose whether the Local or Celery executor should be adopted for getting started.
• Hosts distribution: Airflow in production off-loads task execution to workers (executors). If we go with celery For our extras collection, we have the option of installing these workers either on the workers or on the edge node, where the majority of our clients exist.
• Broker: Airflow supports and maintains integrations with both Redis and RabbitMQ. Currently we only install airflow with Redis. Popularity seems pretty distributed.

Multi-Tenancy and Impersonation

As of now, Airflow doesn't inherently support multi-tenancy, with this feature possibly being available in a year or more apache/airflow#29986. Considering this:

• For an initial implementation, we could deploy a standard Airflow version, letting each client handle, if required, multi-tenancy independently.
• For more granular control, we can consider leveraging Airflow Policies, LDAP implementation, and DAG structures. This can provide limitations on user actions via DAG editing capabilities and impersonation. The solution we achieved is partial, as dag editors still can act as cluster administrators through bash operator. However, they can't impersonate other users through Hive, Spark, and HDFS operators, which means we either disallow the use of the BashOperator, or DAGs have to be validated..
• Traditional workarounds to multi-tenancy are either:
  • DAGs code-reviewed if you are using Git and git-sync.
  • Different instances for airflow users. This is complicated without Kubernetes due to how dagProcessor works. We understand this might be possible with extra development, leveraging standalone DagFileProcessorManager, available for airflow >= 2.5.

DAG distribution

During airflow installation through the collections, we update DAGs across all relevant hosts. This isn't a suitable solution for dag editors, as tdp is either for installation or, potentially, production maintenance. I don't think there's much for TOSIT-IO to do here, but it's good to be aware of it.

Extra tools

There are quite some extra tooling in the draft PR, most of them mentioned in the previous topics. If we decide to NOT keep them as core to the deployment, we could also keep some of them in airflow.utils in case users want to use them. These would be:

• DAG sync
• Example DAGs
• User config in airflow (tdp_user, smoke_user)
• Security configurations (policies, roles)
• Connectors (hive, spark, hdfs).

---

The decision should consider the specific requirements for the extra, stability, and users support for each option.

Input and suggestions are invited for an effective integration.