v0.1.0b4

cdf-tk

Added

• Added --env-path option to specify custom locations of .env file

Templates

No changes

v0.1.0b3

cdf-tk

Fixed

• Fixed bug in cdf-tk deploy where auth groups with a mix of all and resource scoped capabilities skipped
  the all scoped capabilities. This is now fixed.

Templates

No changes to templates.

v0.1.0b2

cdf-tk

Fixed

• Handle duplicate TransformationSchedules when loading Transformation resources.
• Print table at the end of cdf-tk deploy failed with AttributeError is a resource.
  returned empty. This is now fixed.
• The cdf-tk build command no longer gives a warning about missing sql file for
  TransformationSchedules.

Templates

Fixed

• In the package example_pump ensure all transformations are prefixed with tr_.

v0.1.0b1

cdf-tk

Added

• Warnings if a configuration file is using snake_case when then resource type is expecting camelCase.
• Added support for validation of space for data models.
• Check for whether template variables <change_me> are present in the config files.
• Check for whether data set id is present in the config files.
• Print table at the end of cdf-tk deploy with the resources that were created, deleted, and skipped.
• Support for Extraction Pipelines and Extraction Pipeline configuration for remotely configured Extractors
• Separate loader for Transformation Schedule resources.

Removed

• In the deploy command drop_data option has been removed. To drop data, use the clean command instead.

Changed

• Require all spaces to be explicitly defined as separate .space.yaml file.
• The data_set_id for Transformations must now be set explicitly in the yaml config file for the Transformation
  under the data_set_id key. Note that you also need to explicitly define the data_set in its own yaml config file.
• All config files have been merged to a single config file, config.yaml. Upon calling cdf-tk init the config.yaml
  is created in the root folder of the project based on the default.config.yaml file of each module.
• DataSetID is no longer set implicitly when running the cdf-tk deploy command. Instead, the data_set_id must be
  set explicitly in the yaml config file.

Fixed

• When running cdf-tk deploy with --dry-run a ValueError was raised if not all datasets were pre-existing.
  This is now fixed by skipping dataset validation when running with --dry-run.
• When having a auth group with mixed capabilities of all scoped and resource scoped, the all scoped capabilities
  were not removed when running cdf-tk deploy. This is now fixed.
• Loading Transformation did not support setting dataSetExternalId in the yaml config file. This is now fixed.

Templates

Added

• Explicitly define model space in experimental/cdf_asset_source_model/ and experimental/example_pump_model/.
• The module my_example_module has been added to the custom_modules folder.
• Added globally defined schedule variables that can be used across all modules.
• A complete example of an Asset data pipeline in examples/cdf_asset_data_pipeline/ shows how to configure an Extractor, monitor the status of the Extraction Pipeline, and load the data into the asset hierarchy using Transformations.
• DataSet to all example modules: cdf_apm_simple_data_model, cdf_asset_source_model, cdf_oid_example_data,
  example_pump_data_model, example_pump_asset_hierarchy.

Changed

• BREAKING All externalIds and names have been changed to follow the naming conventions for resources
  in examples/cdf_oid_example_data, examples/cdf_apm_simple_data_model, modules/cdf_apm_base,
  modules/cdf_infield_common, and modules/cdf_infield_location.
• BREAKING Transformation Schedules broken out into separate files, following naming convention <transformation_name>.schedule.yaml.
• All cognite templates have been moved into cognite_templates folder, while local_templates is renamed to custom_templates.
• Move cdf_apm_base into separate folder.
• The file local.yaml has been renamed environments.yaml to better reflect its purpose.
• Removed demo sourceId from cdf_infield_location module.
• Changed the isPaused flag to use a module-level variable instead of hardcoded in cdf_apm_simple_data_model.
• Combined the child and parent transformations sync_assets_from_hierarchy_to_apm in cdf_infield_location.
  This has the benefit of not having to wait for the parent transformation to finish before starting the child transformation,
  thus no longer a dependency between the two transformations.

Fixed

• Removed transformation identity provider variables from modules and reused the global cicd_ prefixed ones.
• Ensure all transformations in cognite_modules are prefixed with tr_ and all spaces are prefixed with sp_.

v0.1.0a3

[0.1.0a3] - 2023-12-01 - cdf-tk tool

Changed

• Refactored load functionality. Loading raw tables and files now requires a yaml file with metadata.
• Fix container comparison to detect identical containers when loading data models (without --drop flag).
• Clean up error on resource does not exist when deleting (on deploy --drop or using clean command).

Added

• Support for loading data_sets.
• Support for loading auth without --drop, i.e. deploy --include=auth and only changed groups are deployed.
• cdf-tk --verbose build now prints the resolution of modules and packages.
• Added cdf-tk --version to print the version of the tool and the templates.
• Support for upsert for data_sets.
• The cmd cdf-tk deploy creates the data_set before all other resources.
• Data sets are no longer implicitly created when referenced by another resource, instead an error is raised.
• Require all spaces to be explicitly defined as separate .space.yaml file.
• Add protection on group deletion and skip any groups that the current service principal belongs to.
• Support for multiple file resources per yaml config file for files resources.
• Support templated loading of * files in a folder when a single yaml has externalId: something_$FILENAME.
• You can now name the transformation .sql either with the externalId (as defined in the
  corresponding yaml file) or with the name of the file of the corresponding yaml file.
  I.e. if a transformation is defined in my_transformation.yaml with externalId:
  tr_something, the SQL file should be named either tr_something.sql or my_transformation.sql.
• Missing .sql files for transformations will now raise an error in the build step.
• The build step will now raise a number of warnings for missing externalIds in the yaml files,
  as well as if the naming conventions are not followed.

Fixed

• cdf-tk clean not supporting --include properly.
• cdf-tk clean not working properly for data models with data.
• Fix group deletion on use of clean command to actually delete groups.

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog,
and this project adheres to Semantic Versioning.

Changes are grouped as follows:

• Added for new features.
• Changed for changes in existing functionality.
• Deprecated for soon-to-be removed features.
• Improved for transparent changes, e.g. better performance.
• Removed for now removed features.
• Fixed for any bug fixes.
• Security in case of vulnerabilities.

[0.2.0] - 2023-12-01 - templates

Changed

• BREAKING All externalIds and names have been changed to follow the naming conventions for resources
  in examples/cdf_oid_example_data, examples/cdf_apm_simple_data_model, modules/cdf_apm_base,
  modules/cdf_infield_common, and modules/cdf_infield_location.

[0.1.2] - 2023-11-29

Changed

• Remove unused template_version variable from groups and use of group metadata.
• Split up cdf_oid_example_data into data sets and RAW databases per source system.

Fixed

• Add space yaml files for existing data models when explicit space definition was introduced.
• Fix use of integer value in version for data models.
• Fix wrong reference to apm_simple in examples/cdf_apm_simple_data_model and modules/cdf_infield_location.
• Examplify use of a single config yaml file for multiple file resources in examples/cdf_oid_example_data/files/files.yaml.

[0.1.1] - 2023-11-23

Changed

• Changed format of infield external_ids to be more readable, moving _dataset (ds) and _space to the beginning of the external_id.
• examples/cdf_apm_simple/raw and examples/example_dump_asst_hierarchy/raw now explicitly
  defines database and table name in .yaml files for each table.
• Added data_set to examples/example_dump_asst_hierarchy/, which was implicitly defined in
  before.

Fixed

• cdf_infield_common module and the auth applications-configuration.yaml did not load group source id
  correctly due to source_id being used instead of sourceId. This is now fixed.

Cognite-toolkit 0.1.0 alpha2

A quick update adding the experimental module folder to the release: currently it contains the cdf_asset_source_model and example_pump_data_model modules. This is upcoming work that will show the best practice for how to align the classic asset hierarchy and assets in data modeling.

Cognite-toolkit 0.1.0 alpha1

This release is an alpha pre-release. It is well suited for development and bootstrapping of projects, in particular for Asset Performance Management and Infield, but is not recommended for production project lifecycle management yet (due to limited support for diff changes to existing projects).