How to use barman-cloud-backup as hook?

[ImSingee]

I've already set up the Barman server with backup_method = postgres.

Now I want to sync it to the cloud, but it fails.

I'm using 3.12 now. I've found a statement "This script can be used in conjunction with post_backup_script or post_backup_retry_script to relay barman backups to cloud" from the 3.10 document and using the provided example:

post_backup_retry_script = 'barman-cloud-backup --cloud-provider aws-s3 --endpoint-url http://minio:9000 --aws-profile minio s3://barman ${BARMAN_SERVER}'

After executing barman backup pg, I got errors from the log:

2025-01-06 17:12:06,704 [195] Command WARNING: 2025-01-06 17:12:06,704 [206] ERROR: Barman cloud backup exception: backup in '/var/lib/barman/test-pg/base/20250106T171201' has status 'WAITING_FOR_WALS' (status should be: DONE)
2025-01-06 17:12:06,749 [195] barman.hooks WARNING: barman-cloud-backup --cloud-provider aws-s3 --endpoint-url http://minio:9000 --aws-profile minio s3://barman ${BARMAN_SERVER} returned 63
Output details:
2025-01-06 17:12:06,704 [206] ERROR: Barman cloud backup exception: backup in '/var/lib/barman/test-pg/base/20250106T171201' has status 'WAITING_FOR_WALS' (status should be: DONE)

2025-01-06 17:12:06,749 [195] barman.hooks ERROR: barman-cloud-backup --cloud-provider aws-s3 --endpoint-url http://minio:9000 --aws-profile minio s3://barman ${BARMAN_SERVER} was aborted (got exit status 63, Barman requested to stop)
2025-01-06 17:12:06,749 [195] barman.backup WARNING: Ignoring stop request after receiving abort (exit code 63) from post-backup retry hook script: barman-cloud-backup --cloud-provider aws-s3 --endpoint-url http://minio:9000 --aws-profile minio s3://barman ${BARMAN_SERVER}

I guess maybe I should use wait to make it work, and then I encountered with #1043.

Is there a bug or an intentional change? And if it's intentional, how can I set it up?

BTW, if I'm not misunderstanding, this will (if it works well) also only sync backups, not wals. Is there any ways to sync wal to cloud?

And maybe a feature request in the meantime: I want to save backups and wals to s3 only so that I can have nearly unlimited retention. I believe this is already possible when using barman-cloud-backup only, but it requires installing and configuring barman on each pg server. I want to have only one dedicated server for barman, to manage several pg servers, and save data to s3.

[kcslb92]

Hey @ImSingee, I have just been through this for a solution I'm building.

The key is to add --wait (as you mentioned - though you mentioned my other issue/thread, are you seeing similar behaviour with it not running?) to your backup command to ensure that the backup waits until all WALs have finished before closing. barman-cloud-backup will not work as a hook script when the backup state is not completely finished. I vaguely recall this being in the documentation somewhere, but give it a go.

The giveaway is this:

2025-01-06 17:12:06,704 [206] ERROR: Barman cloud backup exception: backup in '/var/lib/barman/test-pg/base/20250106T171201' has status 'WAITING_FOR_WALS' (status should be: DONE)

To answer your other queries:

• Using this as a hook script will sync the backup to s3, but will not delete the local copy. If you want it to clean up the local backup I suggest modifying your hook script to look something like:

post_backup_retry_script = 'barman-cloud-backup --cloud-provider aws-s3 --endpoint-url http://minio:9000 --aws-profile minio s3://barman ${BARMAN_SERVER} && echo "barman delete ${BARMAN_SERVER} ${BARMAN_BACKUP_ID}" | at now + 2 minutes'

I tried doing the deletion as part of the same command but it won't work because the server/backup will be in progress until after the hook script has finished. This is why you need it to run after the hook script finishes. Feel free to modify to run sooner etc.

You raise a good point about the WALs, I will have to investigate that too. Curious to know which way you went to sort this out.

[kcslb92]

It looks like you can add barman-cloud-wal-archive as a hook like this:

pre_archive_retry_script = 'barman-cloud-wal-archive --cloud-provider aws-s3 --endpoint-url http://minio:9000 --aws-profile minio s3://barman ${BARMAN_SERVER}'.

Based on an initial look, it appears that using this as a hook will have the same behaviour where the WALs are copied to S3 but the local copy remains. I would expect you'd probably want the local WALs to remain until the next backup anyway perhaps?

One strange thing is that /var/log/barman/barman.log doesn't get any information whatsoever for the barman-cloud-wal-archive command. There's no ability to increase logging levels either from what I can tell.

I'd love if someone on the dev side could chime in with their recommendations/thoughts on how to manage WALs in between hook-ed cloud backup sync jobs:

• At the moment, when using the barman-cloud-backup as part of a hook script it will copy the backup to the cloud target (s3 in this case). It doesn't delete the source backup files (on the barman node), but this is easily gotten around with the above example of adding a scheduled deletion after the hook has run.
• When using barman-cloud-wal-archive as part of a hook script it will copy WAL files to the cloud target (s3 in this case) but doesn't delete the WAL files on the barman host itself. Given that there is no equivalent construct to delete WAL files (like there is for deleting backups with barman delete <server> <backup_id>), what should we be doing if we want to clean up WALs after they've been pushed to the cloud endpoint?

Is it best practice to have the pre_archive_retry_script hook script setup to call barman-cloud-wal-archive or should we be doing something else?

Additionally, I have a query about the following:

• When performing a backup barman backup <server>, it will remove all local WALs after the backup is done (I assume this is because it combines the basebackup with the WALs for what gets stored.
• When doing the above but using barman-cloud-backup as part of a hook script, whilst it will prune older backups on the cloud side, it does not appear to prune WAL files that have been pushed to the cloud endpoint using barman-cloud-wal-archive in a hook script.
• Thus, should the barman-cloud-backup command also prune WALs on the cloud endpoint (the same way it prunes backups as per retention policies) when used as a hook script?

[ImSingee]

@kcslb92 Thanks for your reply!

I found this document: https://docs.pgbarman.org/release/3.12.0/user_guide/hook_scripts.html#using-barman-cloud-scripts-as-hooks-in-barman

The hook only feels like it's not running—it actually does run (with no log). Not sure why I thought it wasn't working before, but after retrying just now, everything works perfectly.

With the following config, both backup and wal can be replicated to s3 successfully:

[test-pg]
streaming_archiver = on
backup_method = postgres
conninfo = host=test-pg user=barman dbname=postgres
streaming_conninfo = host=test-pg user=barman dbname=postgres
slot_name = barman
create_slot = auto
retention_policy_mode = auto
retention_policy = RECOVERY WINDOW OF 7 days
wal_retention_policy = main
post_backup_retry_script = '/usr/local/bin/barman-cloud-backup --cloud-provider aws-s3 --endpoint-url http://minio:9000 --aws-profile minio s3://barman ${BARMAN_SERVER}'
pre_archive_retry_script = '/usr/local/bin/barman-cloud-wal-archive --cloud-provider aws-s3 --endpoint-url http://minio:9000 --aws-profile minio s3://barman ${BARMAN_SERVER}'

About the deleting: I'm not really care if backups and wals exists in my local server. I can accept save some copies in my local server, while I want the s3 server can save more. For example, I may want my local server have one copy and the server have recent 1 year's.

I haven't tried this. But I believe the following solution will work:

1. Configure a retention_policy = REDUNDANCY 1 in barman config. (And barman cron will do the cleaning automatically)
2. Add a new cron like barman-cloud-backup-delete --retention-policy 'RECOVERY WINDOW OF 365 days' to clean outdated data on s3

[barthisrael]

Hello @ImSingee @kcslb92,

Please find some thoughts below.

I found this document: https://docs.pgbarman.org/release/3.12.0/user_guide/hook_scripts.html#using-barman-cloud-scripts-as-hooks-in-barman

Right, that's the documentation about how to use barman-cloud-backup and barman-cloud-wal-archive as hooks in a Barman server. That way you are able to relay base backups and/or WAL files from the Barman server to a cloud object storage.

One strange thing is that /var/log/barman/barman.log doesn't get any information whatsoever for the barman-cloud-wal-archive command.

Please note that barman commands and barman-cloud-* scripts are unrelated to each other.

The barman commands belong to the backup server, use the server configuration, and write to the configured log files.

On the other hand, barman-cloud-* are scripts which were mainly designed to be used in the Postgres host, not in a dedicated Barman host. Still, for certain special tasks it is possible to use them as hook scripts. It is worth noting that they don't use Barman configuration files, but rather get everything through command-line arguments, given they work as standalone scripts.

With that in mind, if you want to save the output of barman-cloud-* scripts, you need to redirect stdout/stderr to some file in your system.

There's no ability to increase logging levels either from what I can tell.

Yes, there is a way. If you want increased logging output, please use the -v option of barman-cloud-*:

• -v: enables INFO messages
• -vv: enables DEBUG messages

At the moment, when using the barman-cloud-backup as part of a hook script it will copy the backup to the cloud target (s3 in this case). It doesn't delete the source backup files (on the barman node), but this is easily gotten around with the above example of adding a scheduled deletion after the hook has run.

When using barman-cloud-wal-archive as part of a hook script it will copy WAL files to the cloud target (s3 in this case) but doesn't delete the WAL files on the barman host itself. Given that there is no equivalent construct to delete WAL files (like there is for deleting backups with barman delete <backup_id>), what should we be doing if we want to clean up WALs after they've been pushed to the cloud endpoint?

Deleting backups is outside of the scope of the barman-cloud-backup and barman-cloud-wal-archive scripts, and these tasks are delegated to other mechanisms like the Barman retention policies.

Instead of scheduling these deletes after the hook runs, I'd advise to use retention policies in Barman.
Whenever a backup is out of your retention policy, Barman automatically deletes the base backups. As part of the backup deletion process, Barman also deletes all the WALs that are no longer required.

It is configured slightly differently for barman and barman-cloud-*:

• For barman:
  1. Configure a retention policy for your Barman servers, through the configuration file(s)
  2. Make sure barman cron is configured to run every now and then in your crontab. The cron sub-command performs maintenance tasks, including applying the configured retention policies for the Barman servers
• For barman-cloud-*:
  1. Configure barman-cloud-backup-delete with the -r option to run every now and then in your crontab. This script takes care of applying the retention policy for a given cloud instance

You might be interested in reading the Retention policies section for more details.

Is it best practice to have the pre_archive_retry_script hook script setup to call barman-cloud-wal-archive or should we be doing something else?

Yes, that's what the documentation recommends.

Please refer to Using barman-cloud-* scripts as hooks in barman.

When performing a backup barman backup , it will remove all local WALs after the backup is done (I assume this is because it combines the basebackup with the WALs for what gets stored.

When running barman backup <server>, if Barman detects that's the first backup of the Barman server, after taking the base backup Barman removes the WALs preceding that backup because they are not required neither for the backup consistency nor useful for point-in-time-recovery.

When doing the above but using barman-cloud-backup as part of a hook script, whilst it will prune older backups on the cloud side, it does not appear to prune WAL files that have been pushed to the cloud endpoint using barman-cloud-wal-archive in a hook script.

barman-cloud-backup doesn't have a logic like barman backup to check if it's the first backup being taken in the cloud for a given instance. That's why it doesn't remove the WALs preceding that first backup in the cloud.

In any case, as mentioned earlier, when barman-cloud-backup-delete runs, Barman does remove the no longer needed WALs from the cloud object storage. See an example below using a retention policy of REDUNDANCY 1:

• WALs in the bucket:

$ aws --profile barman-cloud s3 ls s3://barman-s3-test/cloud/wals/0000000100000000/
2025-01-16 12:14:58   16777216 000000010000000000000067
2025-01-16 12:14:59        351 000000010000000000000067.00000110.backup
2025-01-16 12:36:09   16777216 000000010000000000000068
2025-01-16 12:36:13   16777216 000000010000000000000069
2025-01-16 12:36:14        351 000000010000000000000069.00000028.backup
2025-01-16 12:36:48   16777216 00000001000000000000006A
2025-01-16 12:36:52   16777216 00000001000000000000006B
2025-01-16 12:36:53        351 00000001000000000000006B.00000028.backup
2025-01-16 12:36:55   16777216 00000001000000000000006C

• Applying the retention policy:

$ barman-cloud-backup-delete -P barman-cloud s3://barman-s3-test cloud -r 'REDUNDANCY 1' -v
2025-01-16 12:37:03,921 [46187] INFO: Found credentials in shared credentials file: ~/.aws/credentials
2025-01-16 12:37:04,094 [46187] INFO: Found file from backup '20250116T123608' of server 'cloud': cloud/base/20250116T123608/24576.tar
2025-01-16 12:37:04,094 [46187] INFO: Found file from backup '20250116T123608' of server 'cloud': cloud/base/20250116T123608/data.tar

• WALs in the bucket after backups deletion:

$ aws --profile barman-cloud s3 ls s3://barman-s3-test/cloud/wals/0000000100000000/
2025-01-16 12:36:52   16777216 00000001000000000000006B
2025-01-16 12:36:53        351 00000001000000000000006B.00000028.backup
2025-01-16 12:36:55   16777216 00000001000000000000006C

Thus, should the barman-cloud-backup command also prune WALs on the cloud endpoint (the same way it prunes backups as per retention policies) when used as a hook script?

No, that's out of the scope of barman-cloud-backup. Please rely on barman-cloud-backup-delete for that task, either by deleting a specific backup through the -b option, or by applying retention policies through the -r option.

The hook only feels like it's not running—it actually does run (with no log). Not sure why I thought it wasn't working before, but after retrying just now, everything works perfectly.

Please refer to a previous sentence in this reply, which explains about the relationship between barman and barman-cloud-*, and how logging works.

About the deleting: I'm not really care if backups and wals exists in my local server. I can accept save some copies in my local server, while I want the s3 server can save more. For example, I may want my local server have one copy and the server have recent 1 year's.

I haven't tried this. But I believe the following solution will work:

1. Configure a retention_policy = REDUNDANCY 1 in barman config. (And barman cron will do the cleaning automatically)
2. Add a new cron like barman-cloud-backup-delete --retention-policy 'RECOVERY WINDOW OF 365 days' to clean outdated data on s3

Yes, that's the recommended way of dealing with this situation.

Best regards,
Israel.