feat: add TDengine.py driver to db_engine #32041

DuanKuanJun · 2025-01-30T14:12:04Z

feat: SuperSet support TDengine data source

SUMMARY

Through db_engine_spec for TDengine, Superset can support TDengine data sources and provide functions such as data presentation and analysis, see detail.

TESTING INSTRUCTIONS

Install python connector for TDengine pip3 install taospy && pip3 install taos-ws-py
Enter the new database connection page, "Superset" → "Setting" → "Database Connections" → "+DATABASE"
Select TDengine database connection, select the "TDengine" option from the drop-down list of "SUPPORTED DATABASES".
Write a name of connection in "DISPLAY NAME"
The "SQLALCHEMY URL" field is a key connection information string, and it must be filled in correctly.
format: taosws://user:password@host:port
Configure the connection string, click "TEST CONNECTION" to test if the connection can be successful. After passing the test, click the "CONNECT" button to complete the connection

ADDITIONAL INFORMATION

Has associated issue:
Required feature flags:
Changes UI
Includes DB Migration (follow approval process in SIP-59)
- Migration is atomic, supports rollback & is backwards-compatible
- Confirm DB migration upgrade and downgrade tested
- Runtime estimates and downtime expectations provided
Introduces new feature or API
Removes existing feature or API

korbit-ai

Review by Korbit AI

Korbit automatically attempts to detect when you fix issues in new commits.

Category	Issue	Fix Detected
	Insecure Credential Example in URI Placeholder ▹ view	✅

Need a new review? Comment /korbit-review on this PR and I'll review your latest changes.

Korbit Guide: Usage and Customization

Interacting with Korbit

You can manually ask Korbit to review your PR using the /korbit-review command in a comment at the root of your PR.

You can ask Korbit to generate a new PR description using the /korbit-generate-pr-description command in any comment on your PR.

Too many Korbit comments? I can resolve all my comment threads if you use the /korbit-resolve command in any comment on your PR.

Chat with Korbit on issues we post by tagging @korbit-ai in your reply.

Help train Korbit to improve your reviews by giving a 👍 or 👎 on the comments Korbit posts.

Customizing Korbit

Check out our docs on how you can make Korbit work best for you and your team.

Customize Korbit for your organization through the Korbit Console.

Feedback and Support

Tell us what you think of Korbit

Schedule a call with our team

Email us @ [email protected]

superset/db_engine_specs/TDengine.py

github-actions

Congrats on making your first PR and thank you for contributing to Superset! 🎉 ❤️

We hope to see you in our Slack community too! Not signed up? Use our Slack App to self-register.

sadpandajoe · 2025-01-30T18:22:20Z

@DuanKuanJun thanks for opening this PR. Can you fill out the description for this PR?

codecov · 2025-01-30T18:36:57Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 83.47%. Comparing base (b12f515) to head (7980c6b).
Report is 25 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff             @@
##           master   #32041       +/-   ##
===========================================
+ Coverage        0   83.47%   +83.47%     
===========================================
  Files           0      546      +546     
  Lines           0    39057    +39057     
===========================================
+ Hits            0    32603    +32603     
- Misses          0     6454     +6454

Flag	Coverage Δ
hive	`48.51% <93.33%> (?)`
mysql	`75.88% <93.33%> (?)`
postgres	`75.95% <93.33%> (?)`
presto	`53.05% <93.33%> (?)`
python	`83.47% <100.00%> (?)`
sqlite	`75.46% <93.33%> (?)`
unit	`61.02% <100.00%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

DuanKuanJun · 2025-02-01T14:31:11Z

@DuanKuanJun thanks for opening this PR. Can you fill out the description for this PR?

already fill out the description
@sadpandajoe

villebro

Please check how to correctly define the dependencies. Also, tests seem to be failing, please check CI logs.

docs/docs/configuration/databases.mdx

DuanKuanJun · 2025-02-02T08:29:51Z

Please check how to correctly define the dependencies. Also, tests seem to be failing, please check CI logs.
@villebro , I submit fixed code, but CI could not run, show tips "Expected — Waiting for status to be reported"

rusackas · 2025-02-02T10:02:26Z

Running CI now 🤞

villebro

A few additional comments. CI is still failing, please check the logs for what the issues are.

superset/db_engine_specs/TDengine.py

Co-authored-by: Ville Brofeldt <[email protected]>

villebro

LGTM. There's still CI issues, please check those. You may benefit from installing pre-commit hooks. Please check https://superset.apache.org/docs/contributing/development#git-hooks, as these automatically correct any code formatting issues for you.

superset/db_engine_specs/TDengine.py

DuanKuanJun · 2025-02-05T09:56:13Z

check ok on local machine.
root@test ~/dev/alex/superset (db_engine_TDengine)$ pre-commit run --files superset/db_engine_specs/TDengine.py
auto-walrus..............................................................Passed
mypy.....................................................................Passed
check docstring is first.................................................Passed
check for added large files..............................................Passed
check yaml...........................................(no files to check)Skipped
debug statements (python)................................................Passed
fix end of files.........................................................Passed
trim trailing whitespace.................................................Passed
prettier.............................................(no files to check)Skipped
eslint...............................................(no files to check)Skipped
Blacklist................................................................Passed
Helm Docs............................................(no files to check)Skipped
ruff.....................................................................Passed
ruff-format..............................................................Passed
root@test ~/dev/alex/superset (db_engine_TDengine)$ pre-commit run --files tests/unit_tests/db_engine_specs/test_TDengine.py
auto-walrus..............................................................Passed
mypy.....................................................................Passed
check docstring is first.................................................Passed
check for added large files..............................................Passed
check yaml...........................................(no files to check)Skipped
debug statements (python)................................................Passed
fix end of files.........................................................Passed
trim trailing whitespace.................................................Passed
prettier.............................................(no files to check)Skipped
eslint...............................................(no files to check)Skipped
Blacklist................................................................Passed
Helm Docs............................................(no files to check)Skipped
ruff.....................................................................Passed
ruff-format..............................................................Passed
root@test ~/dev/alex/superset (db_engine_TDengine)$

DuanKuanJun · 2025-02-06T01:07:53Z

@villebro , help me start CI , fix pre-commit check failed, thank you.

DuanKuanJun · 2025-02-06T01:44:41Z

@villebro , please help me run ci agian , missed to submit test_TDengine.py for pre-commit checking, Thank you.

villebro

Three things that you should change, in addition to anything reported by CI:

Remove superset/db_engine_specs/test_TDengine.py, which seems to be duplicated
Rename TDengine.py to tdengine.py
Rename test_TDengine.py to test_tdengine.py

superset/db_engine_specs/test_TDengine.py

DuanKuanJun · 2025-02-06T02:24:33Z

@villebro , sorry , can you help me run again.

DuanKuanJun · 2025-02-06T02:48:58Z

Three things that you should change, in addition to anything reported by CI:

Remove superset/db_engine_specs/test_TDengine.py, which seems to be duplicated

Rename TDengine.py to tdengine.py

Rename test_TDengine.py to test_tdengine.py

Three problems is done.

I have another problem, how can I see docs/static/img/tdengine.png display effect ? I not sure size is ok , have a preview address to see ?
@villebro

villebro · 2025-02-06T05:09:03Z

I have another problem, how can I see docs/static/img/tdengine.png display effect ? I not sure size is ok , have a preview address to see ? @villebro

You can run the docs locally (it should be fairly easy). But the pic looks good at quick glance.

DuanKuanJun · 2025-02-06T05:25:43Z

I have another problem, how can I see docs/static/img/tdengine.png display effect ? I not sure size is ok , have a preview address to see ? @villebro

You can run the docs locally (it should be fairly easy). But the pic looks good at quick glance.
Thanks, I try locally.
I submit a new commit to fix unit test , forget to modify import library name to lowercase , very sorry.

DuanKuanJun · 2025-02-07T01:12:15Z

Thank you very much @villebro , When is the next version of Superset expected to be released ?

DuanKuanJun · 2025-02-07T03:46:07Z

@villebro , sorry, tdengine logo not show, tdengine.png need put to static/img/databases/ , I need create a new PR to move.

DuanKuanJun · 2025-02-07T04:34:37Z

@villebro , new PR address : #32176

feat: add TDengine.py driver to db_engine

f1a3cd0

pull-request-size bot added the size/S label Jan 30, 2025

dosubot bot added data:connect Namespace | Anything related to db connections / integrations enhancement:db Suggest new DB connections labels Jan 30, 2025

korbit-ai bot reviewed Jan 30, 2025

View reviewed changes

superset/db_engine_specs/TDengine.py Outdated Show resolved Hide resolved

github-actions bot reviewed Jan 30, 2025

View reviewed changes

add TDengine to databases.mdx

68ef48d

pull-request-size bot added size/M and removed size/S labels Jan 30, 2025

github-actions bot added the doc Namespace | Anything related to documentation label Jan 30, 2025

sadpandajoe requested a review from betodealmeida January 30, 2025 18:21

DuanKuanJun added 2 commits February 1, 2025 21:08

feature: add TDengine.png logo to superset README.md

541268e

feature: add test engine driver pytest case

cf18e18

pull-request-size bot added size/L and removed size/M labels Feb 1, 2025

villebro requested changes Feb 1, 2025

View reviewed changes

docs/docs/configuration/databases.mdx Outdated Show resolved Hide resolved

DuanKuanJun added 2 commits February 2, 2025 12:29

fix: fill pyproject.toml depends and fix ci case

fead059

fix: security problem with password

348ceed

DuanKuanJun requested a review from villebro February 2, 2025 08:31

villebro reviewed Feb 2, 2025

View reviewed changes

superset/db_engine_specs/TDengine.py Outdated Show resolved Hide resolved

superset/db_engine_specs/TDengine.py Outdated Show resolved Hide resolved

DuanKuanJun and others added 5 commits February 3, 2025 14:31

Update superset/db_engine_specs/TDengine.py

49881f1

Co-authored-by: Ville Brofeldt <[email protected]>

fix: ci failed with Optional

cca4f08

fix: remove annotation to compatible superset 2.1.3

4f0570f

fix: remove no use import library

a671ec2

fix: get_schema_from_engine_params return with str|None annotation

1ffed07

fix: pre-commit check for trim trailing whitespace failed

26d87e6

villebro approved these changes Feb 4, 2025

View reviewed changes

superset/db_engine_specs/TDengine.py Outdated Show resolved Hide resolved

superset/db_engine_specs/TDengine.py Outdated Show resolved Hide resolved

fix: add from future import annotations

6b73c4c

fix: TDengine.py and test_TDengine.py checkok by pre-commit on local

6a7ffbf

fix: test_TDengine.py for ruff checked on pre-commit

81373bb

villebro requested changes Feb 6, 2025

View reviewed changes

superset/db_engine_specs/test_TDengine.py Outdated Show resolved Hide resolved

fix: delete db_engine_spec/test_TDengine.py by mistaken submit

f91a36d

DuanKuanJun added 2 commits February 6, 2025 10:37

fix: delete uppercase TDengine files

4738466

fix: add lowercase tdengine files

4dfa925

fix: import library name need modify to lowercase

7980c6b

villebro approved these changes Feb 6, 2025

View reviewed changes

villebro merged commit 649a0de into apache:master Feb 6, 2025
46 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add TDengine.py driver to db_engine #32041

feat: add TDengine.py driver to db_engine #32041

DuanKuanJun commented Jan 30, 2025 •

edited

Loading

korbit-ai bot left a comment •

edited

Loading

github-actions bot left a comment

sadpandajoe commented Jan 30, 2025

codecov bot commented Jan 30, 2025 •

edited

Loading

DuanKuanJun commented Feb 1, 2025 •

edited

Loading

villebro left a comment

DuanKuanJun commented Feb 2, 2025

rusackas commented Feb 2, 2025

villebro left a comment

villebro left a comment

DuanKuanJun commented Feb 5, 2025 •

edited

Loading

DuanKuanJun commented Feb 6, 2025

DuanKuanJun commented Feb 6, 2025

villebro left a comment

DuanKuanJun commented Feb 6, 2025

DuanKuanJun commented Feb 6, 2025

villebro commented Feb 6, 2025

DuanKuanJun commented Feb 6, 2025

DuanKuanJun commented Feb 7, 2025

DuanKuanJun commented Feb 7, 2025

DuanKuanJun commented Feb 7, 2025

feat: add TDengine.py driver to db_engine #32041

feat: add TDengine.py driver to db_engine #32041

Conversation

DuanKuanJun commented Jan 30, 2025 • edited Loading

SUMMARY

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

korbit-ai bot left a comment • edited Loading

Choose a reason for hiding this comment

Review by Korbit AI

Korbit automatically attempts to detect when you fix issues in new commits.

Interacting with Korbit

Customizing Korbit

Feedback and Support

github-actions bot left a comment

Choose a reason for hiding this comment

sadpandajoe commented Jan 30, 2025

codecov bot commented Jan 30, 2025 • edited Loading

Codecov Report

DuanKuanJun commented Feb 1, 2025 • edited Loading

villebro left a comment

Choose a reason for hiding this comment

DuanKuanJun commented Feb 2, 2025

rusackas commented Feb 2, 2025

villebro left a comment

Choose a reason for hiding this comment

villebro left a comment

Choose a reason for hiding this comment

DuanKuanJun commented Feb 5, 2025 • edited Loading

DuanKuanJun commented Feb 6, 2025

DuanKuanJun commented Feb 6, 2025

villebro left a comment

Choose a reason for hiding this comment

DuanKuanJun commented Feb 6, 2025

DuanKuanJun commented Feb 6, 2025

villebro commented Feb 6, 2025

DuanKuanJun commented Feb 6, 2025

DuanKuanJun commented Feb 7, 2025

DuanKuanJun commented Feb 7, 2025

DuanKuanJun commented Feb 7, 2025

DuanKuanJun commented Jan 30, 2025 •

edited

Loading

korbit-ai bot left a comment •

edited

Loading

codecov bot commented Jan 30, 2025 •

edited

Loading

DuanKuanJun commented Feb 1, 2025 •

edited

Loading

DuanKuanJun commented Feb 5, 2025 •

edited

Loading