Consolidating dbt Labs managed adapters to one repository

[amychen1776]

Hello everyone!

If we haven't met before, I'm Amy Chen. I'm the Product Manager for the Adapters team. :)

I wanted to announce that dbt Labs is looking into changing how we organize our adapter repositories. We want to consolidate all adapters that dbt Labs maintains into the dbt-adapters repo. This will include:

• dbt-postgres
• dbt-redshift
• dbt-snowflake
• dbt-bigquery
• dbt-spark
• dbt-athena

The Why

This will greatly improve the stability, testability, and maintainability of these adapters and the base abstractions in the dbt-adapters package. Today, our engineering teams spend a lot of time doing tedious, low value tasks like duplicating pull requests and issues across the many adapters we oversee in order to track and implement features/bug fixes across our codebase. We want to remove this friction so we can focus on more value generating functionality like shipping the feature that you want and need. As part of this work, we will be able to:

• Standardize our dev tooling across all adapter implementations
• Standardize docs such as CONTRIBUTING.md, README.md, etc.
• be able to create issues and pull requests that impact multiple adapters, thus improving user experience and trackability
• centralize feature discussion and support adapter-specific points in the same place

The How

The new structure will place each package in a subdirectory of the same name as the package. Additionally, we are taking this chance to adopt the src convention for naming the functional code directory. e.g. the dbt-adapters package will live in /dbt-adapters, with functional code living in /dbt-adapters/src/dbt/adapters/etc. and test code living in /dbt-adapters/tests.

User Impact

For Users - we expect no downtime or impact to your dbt experience. If you are using dbt Cloud or pip installing the adapters, this should not impact your experience at all. If you are installing via github, then you will need to update the git URL.

If you are an adapter maintainer, and you point any of your development or testing workflows at the dbt-adapters repo or at any of the adapter repos, you will need to update those references to point to the new file locations. For example:

deps = [

“dbt-adapters @ git+https://github.com/dbt-labs/dbt-adapters.git”,

“dbt-postgres @ git+https://github.com/dbt-labs/dbt-postgres.git”, 

] ->

deps = [

“dbt-adapters @ git+https://github.com/dbt-labs/dbt-adapters.git#subdirectory=dbt-adapters”,

“dbt-postgres @ git+https://github.com/dbt-labs/dbt-adapters.git#subdirectory=dbt-postgres”, 

]

If you are a community contributor with no open pull requests, it will be worth reading the updated CONTRIBUTING.md at both the repo root level and within the adapter(s) you regularly contribute to.

If you are a community contributor with open pull requests, you must migrate your pull request to the dbt-adapters repo. Unfortunately we could not find a way to migrate those pull requests automatically over. You will need to open up a new pull request, and effectively copy/paste your changes into the new pull request. However we are looking to seeing if we can migrate your existing branches automatically. We will have more updates to come related to this.

Next Steps

I'm starting this discussion so we can get open feedback and any concerns that you might have in terms of this change. I would love to hear from you! 🦄