The Coalesce Base Node Types Package includes:
The Coalesce work node is a versatile node that allows you to develop and deploy a Work table/view in Snowflake.
A Work node serves as an intermediary object and is commonly employed to store raw data before undergoing the crucial phases of transformation and loading into the main tables of the data warehouse.
This pivotal step ensures that the raw data is processed and structured effectively.
The Work node type has two configuration groups:
| Property | Description |
|---|---|
| Storage Location | Storage Location where the WORK will be created |
| Node Type | Name of template used to create node objects |
| Description | A description of the node's purpose |
| Deploy Enabled | If TRUE the node will be deployed / redeployed when changes are detected If FALSE the node will not be deployed or will be dropped during redeployment |
You can create the node as:
| Property | Description |
|---|---|
| Multi Source | Toggle: True/False Implementation of SQL UNIONs True: Combine multiple sources in a single node True Options: - UNION: Combines with duplicate elimination - UNION ALL: Combines without duplicate elimination - INSERT: Individual insert for each source False: Single source node or multiple sources combined using a join |
| Truncate Before | Toggle: True/False This determines whether a table will be overwritten each time a task executes. True: Uses INSERT OVERWRITE False: Uses INSERT to append data |
| Enable tests | Toggle: True/False Determines if tests are enabled |
| Distinct | Toggle: True/False True: Group by All is invisible. DISTINCT data is chosen for processing False: Group by All is visible |
| Group by All | Toggle: True/False True: DISTINCT is invisible. Data is grouped by all columns for processing False: DISTINCT is visible |
| Order By | Toggle: True/False True: Sort column and sort order drop down are visible and are required to form order by clause False: Sort column and sort order drop down are invisible |
| Pre-SQL | SQL to execute before data insert operation |
| Post-SQL | SQL to execute after data insert operation |
| Setting | Description |
|---|---|
| Override Create SQL | Toggle: True/False True: Customized Create SQL specified in the Create SQL space is executed. All other options are invisible False: Create view SQL based on options chosen are framed and executed |
| Multi Source | Toggle: True/False Implementation of SQL UNIONs True: Combine multiple sources in a single node True Options: - UNION: Combines with duplicate elimination - UNION ALL: Combines without duplicate elimination False: Single source node or multiple sources combined using a join |
| Distinct | Toggle: True/False True: Group by All is invisible. DISTINCT data is chosen for processing False: Group by All is visible |
| Group by All | Toggle: True/False True: DISTINCT is invisible. Data is grouped by all columns for processing False: DISTINCT is visible |
Join conditions and other clauses can be specified in the join space next to mapping of columns in the UI.
📘 Specify Group by and Order by Clauses
You should specify group by and order by clauses in this space if you are not opting for the group by all and order by provided in OPTIONS config.
When deployed for the first time into an environment the Work node of materialization type table will execute the below stage:
| Stage | Description |
|---|---|
| Create Work Table | This will execute a CREATE OR REPLACE statement and create a table in the target environment |
| Create Work View | This will execute a CREATE OR REPLACE statement and create a view in the target environment |
After the WORK node with materialization type table has been deployed for the first time into a target environment, subsequent deployments may result in either altering the WORK Table or recreating the WORK table.
A few types of column or table changes will result in an ALTER statement to modify the Work Table in the target environment, whether these changes are made individually or all together:
- Changing table names
- Dropping existing columns
- Altering column data types
- Adding new columns
The following stages are executed:
| Stage | Description |
|---|---|
| Clone Table | Creates an internal table |
| Rename Table| Alter Column | Delete Column | Add Column | Edit table description | Alter table statement is executed to perform the alter operation |
| Swap Cloned Table | Upon successful completion of all updates, the clone replaces the main table ensuring that no data is lost |
| Delete Table | Drops the internal table |
If any of the following change are detected, then the table will be recreated using a CREATE or REPLACE.
- Join clause
- Adding transformation
- Changes in configuration like adding distinct, group by, or order by
One of the following stages are executed:
| Stage | Description |
|---|---|
| Create Table | Creates a new table |
| Replace Table | Replaces an existing table |
The subsequent deployment of Work node of materialization type view with changes in view definition, adding table description or renaming view results in deleting the existing view and recreating the view.
The following stages are executed:
| Stage | Description |
|---|---|
| Delete View | Removes existing view |
| Create View | Creates new view with updated definition |
If a Work Node of materialization type table is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the WorkTable in the target environment will be dropped.
This is executed in two stages:
| Stage | Description |
|---|---|
| Delete Table | Coalesce Internal table is dropped |
| Delete Table | Target table in Snowflake is dropped |
If a Work Node of materialization type view is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the WorkView in the target environment will be dropped.
The stage executed:
| Stage | Description |
|---|---|
| Delete View | Drops the existing Work view from target environment |
The Coalesce Persistent Stage Nodes element, serving as an intermediary object, is frequently utilized to maintain data persistence across multiple execution cycles.
It plays a crucial role in tracking the historical changes of columns linked to business keys.
This functionality is particularly beneficial when the objective is to retain raw data for prolonged durations.
The Persistent node type has two configuration groups:
| Property | Description |
|---|---|
| Storage Location | Storage Location where the WORK will be created |
| Node Type | Name of template used to create node objects |
| Description | A description of the node's purpose |
| Deploy Enabled | If TRUE the node will be deployed / redeployed when changes are detected If FALSE the node will not be deployed or will be dropped during redeployment |
| Option | Description |
|---|---|
| Create As | Table is the only option at this time |
| Multi Source | Toggle: True/False Implementation of SQL UNIONs True: Combine multiple sources in a single node True Options: - UNION: Combines with duplicate elimination - UNION ALL: Combines without duplicate elimination False: Single source node or multiple sources combined using a join |
| Business key | Required column for both Type 1 and Type 2 |
| Change tracking | Required column for Type 2 |
| Truncate Before | Toggle: True/False This determines whether a table will be overwritten each time a task executes. True: Uses INSERT OVERWRITE False: Uses INSERT to append data |
| Enable tests | Toggle: True/False Determines if tests are enabled |
| Distinct | Toggle: True/False True: Group by All is invisible. DISTINCT data is chosen for processing False: Group by All is visible |
| Group by All | Toggle: True/False True: DISTINCT is invisible. Data is grouped by all columns for processing False: DISTINCT is visible |
| Order By | Toggle: True/False True: Sort column and sort order drop down are visible and are required to form order by clause False: Sort column and sort order drop down are invisible |
| Pre-SQL | SQL to execute before data insert operation |
| Post-SQL | SQL to execute after data insert operation |
Join conditions and other clauses can be specified in the join space next to mapping of columns in the UI.
📘 Specify Group by and Order by Clauses
You should specify group by and order by clauses in this space if you are not opting for the group by all and order by provided in OPTIONS config.
When deployed for the first time into an environment the Persistent node will execute the below stage:
| Stage | Description |
|---|---|
| Create Persistent Table | This will execute a CREATE OR REPLACE statement and create a table in the target environment |
After the Persistent node has been deployed for the first time into a target environment, subsequent deployments may result in either altering the Persistent Table or recreating the Persistent table.
A few types of column or table changes will result in an ALTER statement to modify the Persistent Table in the target environment, whether these changes are made individually or all together:
- Changing table names
- Dropping existing columns
- Altering column data types
- Adding new columns
The following stages are executed:
| Stage | Description |
|---|---|
| Clone Table | Creates an internal table |
| Rename Table| Alter Column | Delete Column | Add Column | Edit table description | Alter table statement is executed to perform the alter operation |
| Swap Cloned Table | Upon successful completion of all updates, the clone replaces the main table ensuring that no data is lost |
| Delete Table | Drops the internal table |
If any of the following change are detected, then the table will be recreated using a CREATE or REPLACE.
- Join clause
- Adding transformation
- Changes in configuration like adding distinct, group by, or order by
One of the following stages are executed:
| Stage | Description |
|---|---|
| Create Table | Creates a new table |
| Replace Table | Replaces an existing table |
If a Persistent Node is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the Persistent Table in the target environment will be dropped.
This is executed in two stages:
| Stage | Description |
|---|---|
| Delete Table | Drops table |
| Drop Table or View | Removes the table |
The Coalesce Dimension UDN is a versatile node that allows you to develop and deploy a Dimension table in Snowflake.
A dimension table or dimension entity is a table or entity in a star, snowflake, or starflake schema that stores details about the facts. Dimension tables describe the different aspects of a business process.
The Dimension node type has two configuration groups:
| Property | Description |
|---|---|
| Storage Location | Storage Location where the WORK will be created |
| Node Type | Name of template used to create node objects |
| Description | A description of the node's purpose |
| Deploy Enabled | If TRUE the node will be deployed / redeployed when changes are detected If FALSE the node will not be deployed or will be dropped during redeployment |
| Options | Description |
|---|---|
| Create As | Table or View |
| Insert Zero Key Record | Toggle: True/False Insert Zero Key Record to Dimention if enabled |
| Multi Source | Toggle: True/False Implementation of SQL UNIONs True: Combine multiple sources in a single node True Options: - UNION: Combines with duplicate elimination - UNION ALL: Combines without duplicate elimination False: Single source node or multiple sources combined using a join |
| Business key | Required column for both Type 1 and Type 2 Dimensions |
| Change tracking | Required column for Type 2 Dimension |
| Truncate Before | Toggle: True/False This determines whether a table will be overwritten each time a task executes. True: Uses INSERT OVERWRITE False: Uses INSERT to append data |
| Enable tests | Toggle: True/False Determines if tests are enabled |
| Distinct | Toggle: True/False True: Group by All is invisible. DISTINCT data is chosen for processing False: Group by All is visible |
| Group by All | Toggle: True/False True: DISTINCT is invisible. Data is grouped by all columns for processing False: DISTINCT is visible |
| Order By | Toggle: True/False True: Sort column and sort order drop down are visible and are required to form order by clause False: Sort column and sort order drop down are invisible |
| Pre-SQL | SQL to execute before data insert operation |
| Post-SQL | SQL to execute after data insert operation |
| Options | Description |
|---|---|
| Override Create SQL | Toggle: True/False True: Customized Create SQL specified in the Create SQL space is executed. All other options are invisible False: Create view SQL based on options chosen are framed and executed |
| Multi Source | Toggle: True/False Implementation of SQL UNIONs True: Combine multiple sources in a single node True Options: - UNION: Combines with duplicate elimination - UNION ALL: Combines without duplicate elimination False: Single source node or multiple sources combined using a join |
| Business key | Required column for both Type 1 and Type 2 Dimensions |
| Distinct | Toggle: True/False True: Group by All is invisible. DISTINCT data is chosen for processing False: Group by All is visible |
| Group by All | Toggle: True/False True: DISTINCT is invisible. Data is grouped by all columns for processing False: DISTINCT is visible |
Join conditions and other clauses can be specified in the join space next to mapping of columns in the UI.
📘 Specify Group by and Order by Clauses
You should specify group by and order by clauses in this space if you are not opting for the group by all and order by provided in OPTIONS config.
When deployed for the first time into an environment the Dimension node of materialization type table will execute the Create Dimension Table stage.
| Stage | Description |
|---|---|
| Create Dimension Table | This will execute a CREATE OR REPLACE statement and create a table in the target environment |
| Create Dimension View | This will execute a CREATE OR REPLACE statement and create a view in the target environment |
After the Dimension node of materialization type table has been deployed for the first time into a target environment, subsequent deployments may result in either altering the Dimension Table or recreating the Dimension table.
A few types of column or table changes will result in an ALTER statement to modify the Dimension Table in the target environment, whether these changes are made individually or all together:
- Changing table names
- Dropping existing columns
- Altering column data types
- Adding new columns
The following stages are executed:
| Stage | Description |
|---|---|
| Clone Table | Creates an internal table |
| Rename Table| Alter Column | Delete Column | Add Column | Edit table description | Alter table statement is executed to perform the alter operation |
| Swap Cloned Table | Upon successful completion of all updates, the clone replaces the main table ensuring that no data is lost |
| Delete Table | Drops the internal table |
If any of the following change are detected, then the table will be recreated using a CREATE or REPLACE.
- Join clause
- Adding transformation
- Changes in configuration like adding distinct, group by, or order by
One of the following stages are executed:
| Stage | Description |
|---|---|
| Create Table | Creates a new table |
| Replace Table | Replaces an existing table |
Any of the following changes to views will result in deleting and recreating the Dimension view.
- View defintion
- Adding table description
- Renaming view results
If a Dimension Node is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the Dimension Table in the target environment will be dropped.
This is executed in two stages:
| Stage | Description |
|---|---|
| Delete Table | Coalesce Internal table is dropped |
| Delete Table | Target table in Snowflake is dropped |
If a Dimension Node of materialization type view is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the Dimension View in the target environment will be dropped.
| Stage | Description |
|---|---|
| Delete View | Drops the existing Dimension view from target environment. |
The Coalesce Fact UDN is a versatile node that allows you to develop and deploy a Fact table in Snowflake.
A fact table or a fact entity is a table or entity in a star or snowflake schema that stores measures that measure the business, such as sales, cost of goods, or profit. Fact tables and entities aggregate measures, or the numerical data of a business.
The Fact node has two configuration groups:
| Properties | Description |
|---|---|
| Storage Location | Storage Location where the WORK will be created |
| Node Type | Name of template used to create node objects |
| Description | A description of the node's purpose |
| Deploy Enabled | If TRUE the node will be deployed / redeployed when changes are detected If FALSE the node will not be deployed or will be dropped during redeployment |
| Options | Description |
|---|---|
| Multi Source | Toggle: True/False Implementation of SQL UNIONs True: Combine multiple sources in a single node True Options: - UNION: Combines with duplicate elimination - UNION ALL: Combines without duplicate elimination False: Single source node or multiple sources combined using a join |
| Business key | Required column for Fact table creation |
| Truncate Before | Toggle: True/False This determines whether a table will be overwritten each time a task executes. True: Uses INSERT OVERWRITE False: Uses INSERT to append data |
| Enable tests | Toggle: True/False Determines if tests are enabled |
| Distinct | Toggle: True/False True: Group by All is invisible. DISTINCT data is chosen for processing False: Group by All is visible |
| Group by All | Toggle: True/False True: DISTINCT is invisible. Data is grouped by all columns for processing False: DISTINCT is visible |
| Order By | Toggle: True/False True: Sort column and sort order drop down are visible and are required to form order by clause False: Sort column and sort order drop down are invisible |
| Pre-SQL | SQL to execute before data insert operation |
| Post-SQL | SQL to execute after data insert operation |
| Setting | Description |
|---|---|
| Override Create SQL | Toggle: True/False True: Customized Create SQL specified in the Create SQL space is executed. All other options are invisible False: Create view SQL based on options chosen are framed and executed |
| Multi Source | Toggle: True/False Implementation of SQL UNIONs True: Combine multiple sources in a single node True Options: - UNION: Combines with duplicate elimination - UNION ALL: Combines without duplicate elimination False: Single source node or multiple sources combined using a join |
| Distinct | Toggle: True/False True: Group by All is invisible. DISTINCT data is chosen for processing False: Group by All is visible |
| Group by All | Toggle: True/False True: DISTINCT is invisible. Data is grouped by all columns for processing False: DISTINCT is visible |
Join conditions and other clauses like where, qualify can be specified in the join space next to mapping of columns in the UI.
📘 Specify Group by and Order by Clauses
You should specify group by and order by clauses in this space if you are not opting for the group by all and order by provided in OPTIONS config.
When deployed for the first time into an environment the Fact node of materialization type table will execute the Create Fact Table stage.
| Stage | Description |
|---|---|
| Create Fact Table | This will execute a CREATE OR REPLACE statement and create a table in the target environment |
| Create Fact View | This will execute a CREATE OR REPLACE statement and create a view in the target environment |
After the Fact node has been deployed for the first time into a target environment, subsequent deployments may result in either altering the Fact Table or recreating the Fact table.
A few types of column or table changes will result in an ALTER statement to modify the Fact Table in the target environment, whether these changes are made individually or all together:
- Changing table names
- Dropping existing columns
- Altering column data types
- Adding new columns
The following stages are executed:
| Stage | Description |
|---|---|
| Clone Table | Creates an internal table |
| Rename Table| Alter Column | Delete Column | Add Column | Edit table description | Alter table statement is executed to perform the alter operation |
| Swap Cloned Table | Upon successful completion of all updates, the clone replaces the main table ensuring that no data is lost |
| Delete Table | Drops the internal table |
If any changes like change in join clause, adding transformations, change in business key column, change in configs like adding distinct, group by or orderby, the Fact Table will be recreated by running a CREATE OR REPLACE statement.
If any of the following change are detected, then the table will be recreated using a CREATE or REPLACE.
- Join clause
- Adding transformation
- Change in business key
- Changes in configuration like adding distinct, group by, or order by
One of the following stages are executed:
| Stage | Description |
|---|---|
| Create Table | Creates a new table |
| Replace Table | Replaces an existing table |
The subsequent deployment of Work node of materialization type view with changes in view definition, adding table description or renaming view results in deleting the existing view and recreating the view.
The following stages are executed:
| Stage | Description |
|---|---|
| Delete View | Removes existing view |
| Create View | Creates new view with updated definition |
If a Fact Node is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the Fact Table in the target environment will be dropped.
This is executed in two stages:
| Stage | Description |
|---|---|
| Delete Table | Coalesce Internal table is dropped |
| Delete Table | Target table in Snowflake is dropped |
The Coalesce Fact UDN is a versatile node that allows you to develop and deploy a Fact table in Snowflake.
A factless fact table is used to record events or situations that have no measures, and it has the same level of detail as the dimensions.
The Fact node has two configuration groups:
| Properties | Description |
|---|---|
| Storage Location | Storage Location where the WORK will be created |
| Node Type | Name of template used to create node objects |
| Description | A description of the node's purpose |
| Deploy Enabled | If TRUE the node will be deployed / redeployed when changes are detected If FALSE the node will not be deployed or will be dropped during redeployment |
| Options | Description |
|---|---|
| Create As | Table is the only option at this time |
| Multi Source | Toggle: True/False Implementation of SQL UNIONs True: Combine multiple sources in a single node True Options: - UNION: Combines with duplicate elimination - UNION ALL: Combines without duplicate elimination False: Single source node or multiple sources combined using a join |
| Truncate Before | Toggle: True/False This determines whether a table will be overwritten each time a task executes. True: Uses INSERT OVERWRITE False: Uses INSERT to append data |
| Enable tests | Toggle: True/False Determines if tests are enabled |
| Distinct | Toggle: True/False True: Group by All is invisible. DISTINCT data is chosen for processing False: Group by All is visible |
| Group by All | Toggle: True/False True: DISTINCT is invisible. Data is grouped by all columns for processing False: DISTINCT is visible |
| Order By | Toggle: True/False True: Sort column and sort order drop down are visible and are required to form order by clause False: Sort column and sort order drop down are invisible |
| Pre-SQL | SQL to execute before data insert operation |
| Post-SQL | SQL to execute after data insert operation |
Join conditions and other clauses like where, qualify can be specified in the join space next to mapping of columns in the UI.
📘 Specify Group by and Order by Clauses
You should specify group by and order by clauses in this space if you are not opting for the group by all and order by provided in OPTIONS config.
When deployed for the first time into an environment the Factless Fact node of materialization type table will execute the Create Fact Table stage.
| Stage | Description |
|---|---|
| Create Fact Table | This will execute a CREATE OR REPLACE statement and create a table in the target environment |
After the Fact node has been deployed for the first time into a target environment, subsequent deployments may result in either altering the Fact Table or recreating the Fact table.
A few types of column or table changes will result in an ALTER statement to modify the Factless Fact Table in the target environment, whether these changes are made individually or all together:
- Changing table names
- Dropping existing columns
- Altering column data types
- Adding new columns
The following stages are executed:
| Stage | Description |
|---|---|
| Clone Table | Creates an internal table |
| Rename Table| Alter Column | Delete Column | Add Column | Edit table description | Alter table statement is executed to perform the alter operation |
| Swap Cloned Table | Upon successful completion of all updates, the clone replaces the main table ensuring that no data is lost |
| Delete Table | Drops the internal table |
If any of the following change are detected, then the table will be recreated using a CREATE or REPLACE.
- Join clause
- Adding transformations
- Change in business key
- Changes in configuration like adding distinct, group by, or order by
One of the following stages are executed:
| Stage | Description |
|---|---|
| Create Table | Creates a new table |
| Replace Table | Replaces an existing table |
If a Fact Node is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the Fact Table in the target environment will be dropped.
This is executed in two stages:
| Stage | Description |
|---|---|
| Delete Table | Coalesce Internal table is dropped |
| Delete Table | Target table in Snowflake is dropped |
The Coalesce View UDN is a versatile node that allows you to develop and deploy a View in Snowflake.
A view allows the result of a query to be accessed as if it were a table. Views serve a variety of purposes, including combining, segregating, and protecting data.
The View node type has two configuration groups:
| Properties | Description |
|---|---|
| Storage Location | Storage Location where the WORK will be created |
| Node Type | Name of template used to create node objects |
| Description | A description of the node's purpose |
| Deploy Enabled | If TRUE the node will be deployed / redeployed when changes are detected If FALSE the node will not be deployed or will be dropped during redeployment |
| Options | Description |
|---|---|
| Override Create SQL | Toggle: True/False True: Customized Create SQL specified in the Create SQL space is executed. All other options are invisible False: Create view SQL based on options chosen are framed and executed |
| Multi Source | Toggle: True/False Implementation of SQL UNIONs True: Combine multiple sources in a single node True Options: - UNION: Combines with duplicate elimination - UNION ALL: Combines without duplicate elimination False: Single source node or multiple sources combined using a join |
| Distinct | Toggle: True/False True: Group by All is invisible. DISTINCT data is chosen for processing False: Group by All is visible |
| Group by All | Toggle: True/False True: DISTINCT is invisible. Data is grouped by all columns for processing False: DISTINCT is visible |
| Secure | Toggle: True/False True: A secured view is created False: A normal view is created |
Join conditions and other clauses like where, qualify can be specified in the join space next to mapping of columns in the Coalesce app.
📘 Specify Group by Clauses
Best Practice is to specify group by clauses in this space if you are not opting for the group by all provided in OPTIONS config.
When deployed for the first time into an environment the View node will execute the Create View stage.
| Stage | Description |
|---|---|
| Create View | This will execute a CREATE OR REPLACE statement and create a View in the target environment |
The subsequent deployment of View node with changes in view definition, adding table description, adding secure option or renaming view results in deleting the existing view and recreating the view.
The following stages are executed:
| Stage | Description |
|---|---|
| Delete View | Removes existing view |
| Create View | Creates new view with updated definition |
If a View Node is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the View in the target environment will be dropped.
This is executed in the below stage:
| Stage | Description |
|---|---|
| Delete View | Removes the view from the environment |