Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds Support for COPY TO/FROM Google Cloud Storage #61

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

aykut-bozkurt
Copy link
Collaborator

@aykut-bozkurt aykut-bozkurt commented Oct 28, 2024

Supports following Google Cloud Storage uri forms:

  • gs:// <bucket> / <path>

Configuration

The simplest way to configure object storage is by creating a json config file like [/tmp/gcs.json]:

$ cat /tmp/gcs.json
{
  "gcs_base_url": "gs://testbucket/test.parquet",
  "disable_oauth": false,
  "client_email": "...",
  "private_key_id": "...",
  "private_key": "..."
}

Alternatively, you can use the following environment variables when starting postgres to configure the Google Cloud Storage client:

  • GOOGLE_SERVICE_ACCOUNT_KEY: json serialized service account key
  • GOOGLE_SERVICE_ACCOUNT_PATH: an alternative location for the config file

Closes #62

@aykut-bozkurt aykut-bozkurt marked this pull request as draft October 28, 2024 10:02
@aykut-bozkurt aykut-bozkurt force-pushed the aykut/azure-blob-storage branch from 2feb683 to 3d026b6 Compare November 9, 2024 14:26
@aykut-bozkurt aykut-bozkurt force-pushed the aykut/google-cloud-storage branch from 048bd25 to 4a93f99 Compare November 9, 2024 14:29
@aykut-bozkurt aykut-bozkurt force-pushed the aykut/azure-blob-storage branch 2 times, most recently from afb3c71 to 2a3061f Compare November 9, 2024 23:27
@aykut-bozkurt aykut-bozkurt force-pushed the aykut/google-cloud-storage branch from 4a93f99 to c1ba6ad Compare November 9, 2024 23:38
@aykut-bozkurt aykut-bozkurt force-pushed the aykut/azure-blob-storage branch from 2a3061f to 0a3281f Compare November 28, 2024 15:38
@aykut-bozkurt aykut-bozkurt force-pushed the aykut/google-cloud-storage branch from c1ba6ad to 5d15466 Compare November 28, 2024 15:48
@aykut-bozkurt aykut-bozkurt force-pushed the aykut/azure-blob-storage branch from 0a3281f to 80e449f Compare November 28, 2024 21:41
@aykut-bozkurt aykut-bozkurt force-pushed the aykut/google-cloud-storage branch 2 times, most recently from b588cfd to a61b15d Compare November 28, 2024 23:00
@aykut-bozkurt aykut-bozkurt force-pushed the aykut/azure-blob-storage branch from 80e449f to 8553677 Compare December 3, 2024 12:04
@aykut-bozkurt aykut-bozkurt force-pushed the aykut/google-cloud-storage branch from a61b15d to f187bc4 Compare December 3, 2024 12:08
@aykut-bozkurt aykut-bozkurt force-pushed the aykut/google-cloud-storage branch from 0c15989 to d76bf55 Compare December 16, 2024 16:17
@aykut-bozkurt aykut-bozkurt force-pushed the aykut/azure-blob-storage branch 2 times, most recently from e0df256 to e9d0cdd Compare January 6, 2025 17:09
Base automatically changed from aykut/azure-blob-storage to main January 10, 2025 12:36
@aykut-bozkurt aykut-bozkurt marked this pull request as ready for review January 10, 2025 12:58
Copy link

codecov bot commented Jan 10, 2025

Codecov Report

Attention: Patch coverage is 95.55556% with 6 lines in your changes missing coverage. Please review.

Project coverage is 91.93%. Comparing base (3ff46d5) to head (ec5c082).

Files with missing lines Patch % Lines
src/object_store/gcs.rs 86.84% 5 Missing ⚠️
src/object_store/object_store_cache.rs 83.33% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #61      +/-   ##
==========================================
+ Coverage   91.88%   91.93%   +0.04%     
==========================================
  Files          77       78       +1     
  Lines       10320    10444     +124     
==========================================
+ Hits         9483     9602     +119     
- Misses        837      842       +5     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@aykut-bozkurt aykut-bozkurt force-pushed the aykut/google-cloud-storage branch from dd3d157 to ec5c082 Compare January 30, 2025 09:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support Google Cloud Storage
1 participant