Implement workflow for automatic ECS templates generation

[QU3B1M]

Description

Implement a Docker image that generates ECS index templates based on the mappings definitions, this image is consumed by a Docker Compose that sets all the required env variables and configures the volume where the ECS mappings are stored. A bash utility was generated to handle the execution of the docker compose.

Add another bash tool that launches the ECS mapping generation utility when a modification on a ECS definition is detected, automatically copying the resulting template to wazuh-indexer-plugins repository generating a new branch and a PR with the changes.

Related Issues

Resolves #540

Working evidence

Mapping generator

bash docker/ecs/mapping-generator.sh run alerts     
[+] Running 1/0
 ✔ Container wazuh-ecs-generator  Created                                                                                                                                                           0.0s 
Attaching to wazuh-ecs-generator
wazuh-ecs-generator  | Loading schemas from git ref v8.11.0
wazuh-ecs-generator  | Running generator. ECS version 8.11.0
wazuh-ecs-generator  | Loading user defined schemas: ['/source/ecs/alerts/fields/custom/']
wazuh-ecs-generator  | Replacing unsupported types in generated mappings
wazuh-ecs-generator  | Deleting the "tags" field from the index template
wazuh-ecs-generator  | Removing multi-fields from the index template
wazuh-ecs-generator  | Mappings saved to /source/ecs/alerts/mappings/v8.11.0
wazuh-ecs-generator exited with code 0
[+] Stopping 1/0
 ✔ Container wazuh-ecs-generator  Stopped   

GitHub Action ECS utility

Since it requires an GitHub access token to generate the PR, the GHA Workflow was validated on a fork of this repository

• Working GHA execution: https://github.com/QU3B1M/wazuh-indexer/actions/runs/12549647154/job/34991082646?pr=1

GHA bash utility local execution

GITHUB_TOKEN=<GITHUB_TOKEN> bash ecs/scripts/generate-pr-to-plugins.sh  -b ci/540-ecs-index-automatic-generation

---> Fetching and extracting modified ECS modules...
Updated ECS modules: alerts generate.sh generator scripts states-inventory-hardware
Relevant ECS modules: alerts states-inventory-hardware

---> Running ECS Generator script...
[+] Running 2/0
 ✔ Network generator_default      Created                                                                                                                      0.0s 
 ✔ Container wazuh-ecs-generator  Created                                                                                                                      0.0s 
Attaching to wazuh-ecs-generator
wazuh-ecs-generator  | Loading schemas from git ref v8.11.0
wazuh-ecs-generator  | Running generator. ECS version 8.11.0
wazuh-ecs-generator  | Loading user defined schemas: ['/source/ecs/alerts/fields/custom/']
wazuh-ecs-generator  | Replacing unsupported types in generated mappings
wazuh-ecs-generator  | Deleting the "tags" field from the index template
wazuh-ecs-generator  | Removing multi-fields from the index template
wazuh-ecs-generator  | Mappings saved to /source/ecs/alerts/mappings/v8.11.0
wazuh-ecs-generator exited with code 0
[+] Stopping 1/0
 ✔ Container wazuh-ecs-generator  Stopped                                                                                                                      0.0s 
Processed ECS module: alerts
[+] Running 2/0
 ✔ Container wazuh-ecs-generator  Removed                                                                                                                      0.0s 
 ✔ Network generator_default      Removed                                                                                                                      0.0s 
[+] Running 2/0
 ✔ Network generator_default      Created                                                                                                                      0.0s 
 ✔ Container wazuh-ecs-generator  Created                                                                                                                      0.1s 
Attaching to wazuh-ecs-generator
wazuh-ecs-generator  | Loading schemas from git ref v8.11.0
wazuh-ecs-generator  | Running generator. ECS version 8.11.0
wazuh-ecs-generator  | Loading user defined schemas: ['/source/ecs/states-inventory-hardware/fields/custom/']
wazuh-ecs-generator  | Replacing unsupported types in generated mappings
wazuh-ecs-generator  | Deleting the "tags" field from the index template
wazuh-ecs-generator  | Removing multi-fields from the index template
wazuh-ecs-generator  | Mappings saved to /source/ecs/states-inventory-hardware/mappings/v8.11.0
wazuh-ecs-generator exited with code 0
[+] Stopping 1/0
 ✔ Container wazuh-ecs-generator  Stopped                                                                                                                      0.0s 
Processed ECS module: states-inventory-hardware
[+] Running 2/0
 ✔ Container wazuh-ecs-generator  Removed                                                                                                                      0.0s 
 ✔ Network generator_default      Removed                                                                                                                      0.1s 

---> Cloning QU3B1M/wazuh-indexer-plugins repository...
Cloning into '/Users/quebim_wz/Wazuh/forked/wazuh-indexer/../wazuh-indexer-plugins'...
remote: Enumerating objects: 1280, done.
remote: Counting objects: 100% (356/356), done.
remote: Compressing objects: 100% (214/214), done.
remote: Total 1280 (delta 258), reused 142 (delta 142), pack-reused 924 (from 2)
Receiving objects: 100% (1280/1280), 426.50 KiB | 2.04 MiB/s, done.
Resolving deltas: 100% (449/449), done.
Already up to date.
✓ Set QU3B1M/wazuh-indexer-plugins as the default repository for the current directory

---> Committing and pushing changes to QU3B1M/wazuh-indexer-plugins repository...
branch 'ci/540-ecs-index-automatic-generation' set up to track 'origin/ci/540-ecs-index-automatic-generation'.
Switched to a new branch 'ci/540-ecs-index-automatic-generation'
From https://github.com/QU3B1M/wazuh-indexer-plugins
 * branch            ci/540-ecs-index-automatic-generation -> FETCH_HEAD
Already up to date.
Copying ECS templates to the plugins repository...
  - Copy template for module 'alerts' to 'index-template-alerts.json'
  - Copy template for module 'states-inventory-hardware' to 'index-template-hardware.json'
Nothing to commit, working tree clean.

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[f-galland]

I tested this in a personal fork:

• https://github.com/f-galland/wazuh-indexer/tree/ci/540-ecs-index-automatic-generation

The Workflow was failing due to a non-zero exit code in the generate-pr-to-plugins.sh script triggering set -e to exit altogether.

Fixing that got the PR generation working:

• Update ECS templates for modified modules: states-inventory-hardware,states-inventory-processes,states-inventory-system f-galland/wazuh-indexer-plugins#1

[f-galland]

The latest changes make updates to mappings trigger updates in PR description work.