Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Workflow for automatic generation of index templates on changes #540

Open
2 tasks done
AlexRuiz7 opened this issue Nov 11, 2024 · 6 comments · May be fixed by #586
Open
2 tasks done

Workflow for automatic generation of index templates on changes #540

AlexRuiz7 opened this issue Nov 11, 2024 · 6 comments · May be fixed by #586
Assignees
@QU3B1M
Labels
level/task Task issue request/operational Operational requests type/enhancement Enhancement issue

Comments

@AlexRuiz7
Copy link
Member

AlexRuiz7 commented Nov 11, 2024 •
edited by QU3B1M
Loading

Description

We include many new indices in Wazuh 5 and as responsible for their maintenance and generation, we have generated some tools to automate the process, however, the invocation of these tools is manual and a bit tedious when we change many indices at one. Also, the tools need to be set up before using them, as described in this README.

It would be great to have the index templates re-generated automatically using a GitHub Workflow that triggers on PRs that modify the definitions files.

A possible solution would be to have a Docker image with the requirements to build the templates.

Functional requirements

  • The index templates are generated when their definition changes.

Implementation restrictions

  • Use a GitHub Workflow that triggers automatically on PRs that modify these files.
  • Consider using our Quay.io registry to store the image.

Plan

  • Generate Docker image to generate the index templates.
  • Generate a workflow that detects changes to files under ECS and generated the corresponding index template, using the Docker image created for that.
@AlexRuiz7 AlexRuiz7 added level/task Task issue request/operational Operational requests type/enhancement Enhancement issue labels Nov 11, 2024
@wazuhci wazuhci moved this to Backlog in XDR+SIEM/Release 5.0.0 Nov 11, 2024
@wazuhci wazuhci moved this from Backlog to In progress in XDR+SIEM/Release 5.0.0 Nov 29, 2024
@wazuhci wazuhci moved this from In progress to On hold in XDR+SIEM/Release 5.0.0 Dec 5, 2024
@wazuhci wazuhci moved this from On hold to In progress in XDR+SIEM/Release 5.0.0 Dec 9, 2024
@QU3B1M QU3B1M linked a pull request Dec 10, 2024 that will close this issue
Open
3 tasks
@QU3B1M
Copy link
Member

QU3B1M commented Dec 10, 2024 •
edited
Loading

Created initial version of the Docker image to generate the ECS index templates, it uses a modified version of the generate.sh tool, and currently its only saving the resulting templates inside the container itself.

Building the image

 docker build -f docker/ecs/images/Dockerfile . -t ecs-generator
[+] Building 1.5s (11/11) FINISHED                                                                                                                                                  docker:desktop-linux
 => [internal] load build definition from Dockerfile                                                                                                                                                0.0s
 => => transferring dockerfile: 633B                                                                                                                                                                0.0s
 => [internal] load metadata for docker.io/library/python:3.10                                                                                                                                      0.7s
 => [internal] load .dockerignore                                                                                                                                                                   0.0s
 => => transferring context: 2B                                                                                                                                                                     0.0s
 => [1/6] FROM docker.io/library/python:3.10@sha256:3ba2e48b887586835af6a0c35fc6fc6086fb4881e963082330ab0a35f3f42c16                                                                                0.0s
 => => resolve docker.io/library/python:3.10@sha256:3ba2e48b887586835af6a0c35fc6fc6086fb4881e963082330ab0a35f3f42c16                                                                                0.0s
 => [internal] load build context                                                                                                                                                                   0.1s
 => => transferring context: 231.38kB                                                                                                                                                               0.1s
 => CACHED [2/6] RUN apt-get update &&     apt-get upgrade -y &&     apt-get install  -y git jq &&     apt-get clean &&     rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* &&     git clone https:/  0.0s
 => CACHED [3/6] ADD docker/ecs/images/generate.sh /ecs/generate.sh                                                                                                                                 0.0s
 => [4/6] ADD ecs/ /source/ecs                                                                                                                                                                      0.1s
 => [5/6] RUN chmod +x /ecs/generate.sh                                                                                                                                                             0.1s
 => [6/6] WORKDIR /ecs                                                                                                                                                                              0.0s
 => exporting to image                                                                                                                                                                              0.5s
 => => exporting layers                                                                                                                                                                             0.5s
 => => exporting manifest sha256:b19ca762ecf63c7e57d7767e9f6cdb0d94ac8ae3e22b40a91e3d0595a1771e0b                                                                                                   0.0s
 => => exporting config sha256:b1b8201f80d570a2d6bb5ee4d7425cd5f5bdbdce995009d6bdb5f48fc6f97b8c                                                                                                     0.0s
 => => exporting attestation manifest sha256:5ccac2093aae0f9cf4ab65f00cb5616d9b587d790ac46a32d7ca4382dbd65762                                                                                       0.0s
 => => exporting manifest list sha256:2bfd5a1b52a921fb238685989a7f83a345dd266f6143bb700837943348548da1                                                                                              0.0s
 => => naming to docker.io/library/ecs-generator:latest                                                                                                                                             0.0s

View build details: docker-desktop://dashboard/build/desktop-linux/desktop-linux/z6646fy4lug9e87big311tenp

Using the tool

docker run --rm -e ECS_MODULE=agent ecs-generator              
Loading schemas from git ref v8.11.0
Running generator. ECS version 8.11.0
Loading user defined schemas: ['/source/ecs/agent/fields/custom/']
Replacing unsupported types in generated mappings
Deleting the "tags" field from the index template
Removing multi-fields from the index template
Mappings saved to /source/ecs/agent/mappings/v8.11.0

Next steps:

  • Use a volume as ECS definitions source
  • Implement Docker Compose and a script to handle the tool execution
  • Add a GHA Workflow to generate the index templates and push them on the wazuh-indexer-plugin repo

@QU3B1M
Copy link
Member

QU3B1M commented Dec 13, 2024

Implemented Docker Compose to define the required environment variables (ECS_MODULE) and the volume where the ECS definitions are stored

services:
  ecs-mapping-generator:
    image: wazuh-ecs-generator
    container_name: wazuh-ecs-generator
    build:
      context: ./../..
      dockerfile: ${REPO_PATH:-.}/docker/ecs/images/Dockerfile
    volumes:
      - ${REPO_PATH:-.}/ecs:/source/ecs
    environment:
      - ECS_MODULE=${ECS_MODULE:-default_module}

Also a bash util was developed to handle the interaction with this Docker Compose environment, making the process easier. It only requires one command

bash docker/ecs/mapping-generator.sh run <ECS_MODULE>

This script can be launch from any point inside of the repository's directories.

Example execution 
bash docker/ecs/mapping-generator.sh run alerts     
[+] Running 1/0
 ✔ Container wazuh-ecs-generator  Created                                                                                                                                                           0.0s 
Attaching to wazuh-ecs-generator
wazuh-ecs-generator  | Loading schemas from git ref v8.11.0
wazuh-ecs-generator  | Running generator. ECS version 8.11.0
wazuh-ecs-generator  | Loading user defined schemas: ['/source/ecs/alerts/fields/custom/']
wazuh-ecs-generator  | Replacing unsupported types in generated mappings
wazuh-ecs-generator  | Deleting the "tags" field from the index template
wazuh-ecs-generator  | Removing multi-fields from the index template
wazuh-ecs-generator  | Mappings saved to /source/ecs/alerts/mappings/v8.11.0
wazuh-ecs-generator exited with code 0
[+] Stopping 1/0
 ✔ Container wazuh-ecs-generator  Stopped

@QU3B1M
Copy link
Member

QU3B1M commented Dec 13, 2024

Generated new GitHub Actions Workflow that executes the utility and creates a PR on wazuh-indexer-plugin with modified template. Currently the GHA is not working due to permissions.

[update-ecs-template-alerts 95b56f3] Update ECS template for module alerts
 1 file changed, 1 insertion(+), 1 deletion(-)
remote: Permission to wazuh/wazuh-indexer-plugins.git denied to github-actions[bot].
fatal: unable to access 'https://github.com/wazuh/wazuh-indexer-plugins/': The requested URL returned error: 403
Error: Process completed with exit code 128.

@wazuhci wazuhci moved this from In progress to Pending review in XDR+SIEM/Release 5.0.0 Dec 16, 2024
@wazuhci wazuhci moved this from Pending review to In progress in XDR+SIEM/Release 5.0.0 Dec 26, 2024
@QU3B1M
Copy link
Member

QU3B1M commented Dec 27, 2024

Testing the automated pull request generation on a fork of this repo

@QU3B1M
Copy link
Member

QU3B1M commented Dec 30, 2024 •
edited
Loading

First automated PR generated using the bash tool

Script usage

bash ecs/scripts/generate-pr-to-plugins.sh 

Usage: ecs/scripts/generate-pr-to-plugins.sh -b <branch_name> -t <github_token>
  -b <branch_name>  Branch name to create or update the PR.
  -t <github_token> GitHub token to authenticate with GitHub API.

Script execution

bash ecs/scripts/generate-pr-to-plugins.sh  -b ci/540-ecs-index-automatic-generation -t [GITHUB_TOKEN]


---> Fetching and extracting modified ECS modules...
Updated ECS modules: alerts generate.sh generator scripts states-inventory-hardware
Relevant ECS modules: alerts states-inventory-hardware

---> Running ECS Generator script...
[+] Running 2/0
 ✔ Network generator_default      Created                                                                                                               0.0s 
 ✔ Container wazuh-ecs-generator  Created                                                                                                               0.0s 
Attaching to wazuh-ecs-generator
wazuh-ecs-generator  | Loading schemas from git ref v8.11.0
wazuh-ecs-generator  | Running generator. ECS version 8.11.0
wazuh-ecs-generator  | Loading user defined schemas: ['/source/ecs/alerts/fields/custom/']
wazuh-ecs-generator  | Replacing unsupported types in generated mappings
wazuh-ecs-generator  | Deleting the "tags" field from the index template
wazuh-ecs-generator  | Removing multi-fields from the index template
wazuh-ecs-generator  | Mappings saved to /source/ecs/alerts/mappings/v8.11.0
wazuh-ecs-generator exited with code 0
[+] Stopping 1/0
 ✔ Container wazuh-ecs-generator  Stopped                                                                                                               0.0s 
Processed ECS module: alerts
[+] Running 1/0
 ✔ Container wazuh-ecs-generator  Recreated                                                                                                             0.0s 
Attaching to wazuh-ecs-generator
wazuh-ecs-generator  | Loading schemas from git ref v8.11.0
wazuh-ecs-generator  | Running generator. ECS version 8.11.0
wazuh-ecs-generator  | Loading user defined schemas: ['/source/ecs/states-inventory-hardware/fields/custom/']
wazuh-ecs-generator  | Replacing unsupported types in generated mappings
wazuh-ecs-generator  | Deleting the "tags" field from the index template
wazuh-ecs-generator  | Removing multi-fields from the index template
wazuh-ecs-generator  | Mappings saved to /source/ecs/states-inventory-hardware/mappings/v8.11.0
wazuh-ecs-generator exited with code 0
[+] Stopping 1/0
 ✔ Container wazuh-ecs-generator  Stopped                                                                                                               0.0s 
Processed ECS module: states-inventory-hardware
[+] Running 2/0
 ✔ Container wazuh-ecs-generator  Removed                                                                                                               0.0s 
 ✔ Network generator_default      Removed                                                                                                               0.0s 

---> Cloning QU3B1M/wazuh-indexer-plugins repository...
Already up to date.
✓ Set QU3B1M/wazuh-indexer-plugins as the default repository for the current directory

---> Committing and pushing changes to QU3B1M/wazuh-indexer-plugins repository...
Already on 'ci/540-ecs-index-automatic-generation'
Your branch is up to date with 'origin/ci/540-ecs-index-automatic-generation'.
From https://github.com/QU3B1M/wazuh-indexer-plugins
 * branch            ci/540-ecs-index-automatic-generation -> FETCH_HEAD
Already up to date.
Copying ECS templates to the plugins repository...
  - Copy template for module 'alerts' to 'index-template-alerts.json'
  - Copy template for module 'states-inventory-hardware' to 'index-template-hardware.json'
On branch ci/540-ecs-index-automatic-generation
Your branch is up to date with 'origin/ci/540-ecs-index-automatic-generation'.

nothing to commit, working tree clean

Image

GitHub Action working with the new tool: https://github.com/QU3B1M/wazuh-indexer/actions/runs/12549647154/job/34991082646?pr=1

@wazuhci wazuhci moved this from In progress to Pending review in XDR+SIEM/Release 5.0.0 Dec 30, 2024
@wazuhci wazuhci moved this from Pending review to On hold in XDR+SIEM/Release 5.0.0 Jan 6, 2025
@wazuhci wazuhci moved this from On hold to Pending review in XDR+SIEM/Release 5.0.0 Jan 6, 2025
@wazuhci wazuhci moved this from Pending review to In progress in XDR+SIEM/Release 5.0.0 Jan 7, 2025
@wazuhci wazuhci moved this from In progress to Pending review in XDR+SIEM/Release 5.0.0 Jan 7, 2025
@wazuhci wazuhci moved this from Pending review to In review in XDR+SIEM/Release 5.0.0 Jan 7, 2025
@AlexRuiz7
Copy link
Member Author

We'll need to create a bot to perform the automation at organization level.
https://github.com/orgs/community/discussions/50055

@wazuhci wazuhci moved this from In review to Blocked in XDR+SIEM/Release 5.0.0 Jan 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@QU3B1M QU3B1M

Labels
level/task Task issue request/operational Operational requests type/enhancement Enhancement issue
Projects
XDR+SIEM/Release 5.0.0
Status: Blocked
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Implement workflow for automatic ECS templates generation wazuh/wazuh-indexer
2 participants
@AlexRuiz7 @QU3B1M