[Bug]: Cloud SQL Lineage is not working as intended

[stankiewicz]

What happened?

JDBCIO with DataSourceProviderFromDataSourceConfiguration is building datasources using method

buildDatasource() {
      if (getDataSource() == null) {
        BasicDataSource basicDataSource = new org. apache. commons. dbcp2.BasicDataSource();
[..]

later this dataSource is used to build lineage:

JdbcUtil.FQNComponents fqn = JdbcUtil.FQNComponents.of(validSource);

JdbcUtil.FQNComponents.of expects com.zaxxer.hikari.HikariDataSource datasource to report cloudSQL lineage, otherwise it will return null and lineage will be done for regular sql based on hostname, db and table segments.

Should we replace org. apache. commons. dbcp2.BasicDataSource with com.zaxxer.hikari.HikariDataSource ? or maybe we should instruct somehow users on how to set it up?

Issue Priority

Priority: 3 (minor)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[liferoad]

@Abacn

[Abacn]

ack. It was challenging for a generic Lineage support for JdbcIO due to standard Jdbc DataSource is merely an interface lacking needed fully qualified name component. As a result it only covered the HikariDataSource that was recommended in https://cloud.google.com/sql/docs/mysql/samples/cloud-sql-mysql-servlet-connect-connector#code-sample.

This is a valid feature request of Lineage support for org.apache.commons.dbcp2.BasicDataSource

[stankiewicz]

thanks Yi, I've seen that we recommend Hikari in all the samples. org.apache.commons.dbcp2.BasicDataSource provides similar set information (url + properties) like Hikari so maybe we can think about supporting both. BasicDataSource access to properties is package protected so this would require again reflection to make it quickly.

Maybe JDBCIO should have lineage some kind of lineage/FQNComponents provider within DataSourceProviderFromDataSourceConfiguration, passed to ReadFn/WriteFn that would allow customization if needed.