An error occurred (SignatureDoesNotMatch) when calling the PutObject operation: Unknown

[mudassar1]

i'm having issus with s3 storage.

first of all i tried linode storage bucket. the log below is of linode storage bucket.
it's s3 account adds successfully but when tried to do some operation on it through webmin, i get some weird errors.

Testing backup destination local file /var/backups/server-backup/database-daily/25-01-18 ..
.. done

Testing backup destination backups/db-backup-2025-01-18 in S3 bucket almahir-bucket using linode-s3 ..
.. done

Creating backup for virtual server db-pk.abac.com ..
    Dumping MariaDB database db_pk ..
    .. done

.. completed in 0 seconds

Creating backup for virtual server laravel.abc.com ..
.. completed in 0 seconds

Creating backup for virtual server abc.com ..
.. completed in 0 seconds

Creating final backup archives in directory ..
.. done

Uploading archive to S3 service ..
.. upload failed! upload failed: ../../../../var/backups/server-backup/database-daily/25-01-18/db-pk.abc.com.tar.gz to s3://almahir-bucket/backups/db-backup-2025-01-18/db-pk.abc.com.tar.gz An error occurred (SignatureDoesNotMatch) when calling the PutObject operation: Unknown

second problem is that other s3 service provider is even unable to add s3 account.
after hitting create , i get this error.

Failed to save S3 account : S3 account cannot be used : SSL validation failed for https://oss.abc.com/ [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000) 

[jcameron]

Do you have the aws CLI command installed? This is recommended as virtualmin will use it over its build-in s3 support.

[iliajie]

Jamie, perhaps we should completely remove those outdated and unreliable S3 modules and instead provide a clear message saying that AWS-CLI needs to be installed?

I think, this would be much better than falling back to those broken modules and still failing but in a confusing and obscure way...

[jcameron]

We do show a big warning message if it is not installed.

[iliajie]

Still, what’s the point of keeping modules that are nearly 20 years old and don’t even work?

[jcameron]

Still, what’s the point of keeping modules that are nearly 20 years old and don’t even work?

They do work in 99% of cases though. In this specific case I'm trying to figure out if the issue is our code, or the S3 provider is doing something odd that would also break the aws command.

[iliajie]

They do work in 99% of cases though.

I wasn’t aware that those S3 modules were still supposed to work. Yet, why do we need to support them at all if aws command must always be available on all systems?

In this specific case I'm trying to figure out if the issue is our code, or the S3 provider is doing something odd that would also break the aws command.

Oh, if it fails with aws command then it's definitely us. Does the system have the ca-certificates package installed?

[jcameron]

I wasn’t aware that those S3 modules were still supposed to work. Yet, why do we need to support them at all if aws command must always be available on all systems?

If we could guaranteee that the aws command was always available, then yeah we could drop these modules. But I don't think we can be certain of that though.

[iliajie]

Yes, the aws command should be available on all systems. I think simplifying things would make sense and make it easier to support.

[jcameron]

We certainly could consider dropping support for systems without the aws command, although this could annoy users on FreeBSD and other niche operating systems.

[iliajie]

I didn’t mean any disrespect to FreeBSD users! FreeBSD users can easily install AWS CLI using:

root@freebsd13:~# pkg install python3 py311-pip
root@freebsd13:~# pip-3.11  install --user awscli
root@freebsd13:~# aws --version
aws-cli/1.37.3 Python/3.11.9 FreeBSD/13.3-RELEASE botocore/1.36.3

[jcameron]

We could maybe start by making it mandatory for new installs? I would love to delete our code that calls the S3 API, but also I don't want to annoy users.

[iliajie]

Yes, we have it installed as a recommended package. Although Ubuntu 24.04 no longer includes it in the standard repositories, it can easily be installed via snap.

We shouldn’t need to question whether the aws command possibly available—it simply has to be on all systems.

[jcameron]

Yes but we also need to continue to support users who are using S3 backups but don't have it installed. Especially if it's not in default repositories..

[iliajie]

The overhead of supporting it is far greater than the minor hassle of installing the AWS CLI, whether through standard repos, snap, or pip.

It’s just not worth it nowadays—simplicity is a virtue. :--)

[jcameron]

Is there a lot of overhead though? Seems like at this point most of the work is already done.

[iliajie]

Is there a lot of overhead though? Seems like at this point most of the work is already done.

We could simplify the code by cleaning it up to rely solely on AWS CLI and removing Net and S3 from the Virtualmin installation—they’re 20 years old.

[jcameron]

Sure sure we could, but I still worry that if we do this, anyone who upgrades will find that their S3 backups are broken if they don't have the aws command installed.

[iliajie]

Sure sure we could, but I still worry that if we do this, anyone who upgrades will find that their S3 backups are broken if they don't have the aws command installed.

I understand that, but it’s likely a very small number of people affected, if any, and they’ll see a clear message explaining why it doesn’t work. Highlighting this in the changelog would further minimize the chance of any issues.

We need to clean up the Virtualmin tree. It’s time to remove all code related to LXAdmin and PHP/Serialization.pm, and Ensim as well.

Shipping packages and options that have been obsolete for 15–20 years doesn’t leave a good impression, especially when trying to attract new users.

And back to the S3 modules—it’s much easier and cleaner to handle an error message stating that the awscli command isn’t installed, rather than An error occurred (SignatureDoesNotMatch) when calling the PutObject operation: Unknown or many similar others.

[jcameron]

I understand that, but it’s likely a very small number of people affected, if any, and they’ll see a clear message explaining why it doesn’t work. Highlighting this in the changelog would further minimize the chance of any issues.

Unfortunately users who just upgrade might not see this message, and could end up in a state where their backups have just stopped working unexpectedly! The only way I'd support this is if we can make the aws command a package dependency somehow.

We need to clean up the Virtualmin tree. It’s time to remove all code related to LXAdmin and PHP/Serialization.pm, and Ensim as well.

Agreed! But let's open a new ticket for that.

[iliajie]

The only way I'd support this is if we can make the aws command a package dependency somehow.

Sure! Though as recommended package, not a dependency. Implemented here 9dc2f8e webmin/webmin@ad54080. @swelljoe Joe, don't forget to update your build scripts to include --deb-recommends and --rpm-recommends flags.

You can drop it then.

Development packages are already being built with awscli as recommended package at this point.

Agreed! But let's open a new ticket for that.

Alright, will do.

[jcameron]

What if also we added a big warning on the dashboard in the next release that support for S3 backups without the aws command is deprecated, to encourage users to install it? And then in the version after that, we can drop support for the old built-in code.

[iliajie]

What if also we added a big warning on the dashboard in the next release that support for S3 backups without the aws command is deprecated, to encourage users to install it? And then in the version after that, we can drop support for the old built-in code.

Good idea. However, we don't need to wait; you can already add a message to the dashboard. We have the API for that.

[jcameron]

Good idea. However, we don't need to wait; you can already add a message to the dashboard. We have the API for that.

But we only want to show it if the aws command is not installed, and the admin has at least one S3 backup.

[iliajie]

LOOOL! 🤣 ...in this case we might want to wait then! ...I just hoped we weren't procrastinating here.