Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support progressing reading of files in multipart uploads in v3 #60

Open
xsawyerx opened this issue Apr 9, 2022 · 0 comments
Open

Support progressing reading of files in multipart uploads in v3 #60

xsawyerx opened this issue Apr 9, 2022 · 0 comments

Comments

@xsawyerx
Copy link
Collaborator

xsawyerx commented Apr 9, 2022

v3 supports multipart uploads which use the multipart/related envelope.

Due to having to construct this manually (since HTTP::Request doesn't support it), we are not reading the file content iteratively using the helper method _content_sub(). Arguably, another _content_sub() could be written[1] that iteratively generates not just the file content but the multipart/related envelope prior to it. This would allow progressively providing the file content instead of loading the entire file into memory.

This isn't necessarily the best use of time at the moment, because the limits of multipart uploads are 5M anyway. Once resumable uploads are available, those allow any size (including 5M and below) in a strictly progressive manner out of the box, so they could also be used (though would probably be somewhat more verbose).

[1] This can also be accomplished by adding a hook to the existing _content_sub().

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant