Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add an option to configure S3 repository to store snapshots out of the box #1222

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

t0x01
Copy link

@t0x01 t0x01 commented Feb 22, 2024

Description

The aim of this PR is to add an ability to use S3 compatible storage for snapshot management out of the box. It is made possible by allowing to build wazuh-indexer docker image with additional plugin repository-s3 installed, via a new optional --s3-repository parameter. Parameter --s3-repository is set to false by default. On wazuh-indexer container startup, if repository-s3 plugin is installed, the system will attempt to read AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables and add their values to the OpenSearch keystore.

Example of changes to single-node/config/wazuh_indexer/wazuh.indexer.yml:

...
s3.client.default.endpoint: minio:9000
s3.client.default.protocol: http
s3.client.default.max_retries: 3
s3.client.default.read_timeout: 50s
s3.client.default.path_style_access: true
s3.client.default.use_throttle_retries: true

Example of changes to single-node/docker-compose.yml:

...
  wazuh.indexer:
    ...
    environment:
      ...
      - "AWS_ACCESS_KEY_ID=exampleAccessKeyID"
      - "AWS_SECRET_ACCESS_KEY=exampleSecretAccessKey"
      - "AWS_REGION=us-west-1"
...

@t0x01
Copy link
Author

t0x01 commented Mar 20, 2024

Hello.
Do i need to add any additional information to this PR or open an issue for it?

@32bitbradley
Copy link

+1 on this feature

Allow to build `wazuh-indexer` docker image with additional plugin `repository-s3` installed, via a new optional `--s3-repository` parameter. Parameter `--s3-repository` is set to `false` by default. On `wazuh-indexer` container startup, if `repository-s3` plugin is installed, the system will attempt to read `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables and add their values to the OpenSearch keystore.

For an S3 compatible storage to be configured as a snapshot repository correctly, some additional steps must be taken, mainly:
1. `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` and `AWS_REGION` environment variables must be added to `docker-compose.yml` for the `wazuh.indexer` service. E.g.:
```yaml
...
  wazuh.indexer:
    ...
    environment:
      ...
      - "AWS_ACCESS_KEY_ID=exampleAccessKeyID"
      - "AWS_SECRET_ACCESS_KEY=exampleSecretAccessKey"
      - "AWS_REGION=us-west-1"
...
```
2. Additional settings for the S3 compatible storage must be added to `config/wazuh_indexer/wazuh.indexer.yml`. E.g.:
```yaml
...
s3.client.default.endpoint: minio:9000
s3.client.default.protocol: http
s3.client.default.max_retries: 3
s3.client.default.read_timeout: 50s
s3.client.default.path_style_access: true
s3.client.default.use_throttle_retries: true
```

See https://opensearch.org/docs/latest/tuning-your-cluster/availability-and-recovery/snapshots/snapshot-restore/ for more information.
@t0x01 t0x01 force-pushed the add-install-s3repo-option branch from d83c0c3 to ab482d9 Compare January 15, 2025 14:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants