[REFACTOR][ESS-FRANCE]Introduce a data processor for ESS France

[HAEKADI]

related to #405

This pull request introduces the implementation of the EssFranceProcessor class, configuration settings, and the update of processing DAG.

It also introduces the use of a private method in DataProcessor to push to XCom the unique count of any column in dataset.

[HAEKADI]

@hacherix I am also wondering if we should integrate a utility method in DataProcessor as well, something like:

 @staticmethod
    def pull_from_xcom(key, task_ids):
        ti = get_current_context()["ti"]
        return ti.xcom_pull(key=key, task_ids=task_ids)

or have a utility class for Airflow or if it's a bit overkill. Because the the Notification push and pull is kind of heavy here :

def send_file_to_minio(self):
        super().send_file_to_minio()
        ti = get_current_context()["ti"]
        nb_siren = ti.xcom_pull(key="nb_siren_egapro", task_ids="preprocess_egapro")
        message = f"{nb_siren} unités légales."
        ti.xcom_push(key=Notification.notification_xcom_key, value=message)

What do you think ?

[hacherix]

What do you think ?

Good idea! I like the second option best.
Having a function that takes the same number of arguments than xcom_pull/xcom_push seems overkill indeed 😄
But having functions that abstract the key argument would be very cool indeed!
Such as?

def add_notification_message(self, message)
    ti = get_current_context()["ti"]
    ti.xcom_push(key=self.notification_xcom_key, value=message)

[HAEKADI]

What do you think ?

Good idea! I like the second option best. Having a function that takes the same number of arguments than xcom_pull/xcom_push seems overkill indeed 😄 But having functions that abstract the key argument would be very cool indeed! Such as?

def add_notification_message(self, message)
    ti = get_current_context()["ti"]
    ti.xcom_push(key=self.notification_xcom_key, value=message)

Yeah ! It's a bit more complicated since the message depends on an xcom_pull so we'll end up getting the context twice (when pulling a count of a column and in the helper add_notification_message) instead of once.

I think we need to think this through a bit more.

I will merge as is for now so we can move forward on the refactoring.