Add ScalarDB Dao and related files #2417

inv-jishnu · 2024-12-17T11:49:05Z

Description

I have added scalarDB dao and related files used in data loader

Related issues and/or PRs

Please review this PR once the following PR is reviewed and merged (Once the PR is merged, I will merge the master to this one to resolve the spotbugs issue).

Export options and basic export report dto #2397

Changes made

I have added scalarDB dao and related files used in data loader

Checklist

The following is a best-effort checklist. If any items in this checklist are not applicable to this PR or are dependent on other, unmerged PRs, please still mark the checkboxes after you have read and understood each item.

I have commented my code, particularly in hard-to-understand areas.
I have updated the documentation to reflect the changes.
Any remaining open issues linked to this PR are documented and up-to-date (Jira, GitHub, etc.).
Tests (unit, integration, etc.) have been added for the changes.
My changes generate no new warnings.
Any dependent changes in other PRs have been merged and published.

Additional notes (optional)

Road map to merge remaining data loader core files. Current status

General
- Add ScalarDB Dao and related files - Add ScalarDB Dao and related files #2417
- TableMetadataService(partially replaced by ConsensusUtils): Add table metadata service #2434
Export
- Export options and validations: Add export options validator #2435
- ProducerTasks: 1 PR incoming
Import
- Dto classes and utilities: 1 PR incoming
- Import processor and task code: 2-3 PRs incoming
- Code for Import transaction batch and data chunk import: 1 PR Incoming
- ControlFile related Dtos: 1 PR incoming
- Import logger: 1 PR incoming
- Import Manager: 1 PR incoming

Release notes

NA

data-loader/core/src/main/java/com/scalar/db/dataloader/core/dataimport/dao/ScalarDBDao.java

...-loader/core/src/main/java/com/scalar/db/dataloader/core/dataimport/dao/ScalarDBManager.java

inv-jishnu · 2025-01-16T08:19:44Z

@komamitsu san,
I have made changes based on your feedback.
Please take a look at this again when you get a chance.
Thank you.

komamitsu

LGTM, thank you!

brfrn169

Left a comment. Other than that, LGTM! Thank you!

brfrn169 · 2025-01-17T02:31:01Z

data-loader/core/src/main/java/com/scalar/db/dataloader/core/dataimport/dao/ScalarDBDao.java

@@ -0,0 +1,425 @@
+package com.scalar.db.dataloader.core.dataimport.dao;
+
+import com.scalar.db.api.*;


Please don't use the wildcard * for import.

I have updated to removed usage of wildcard.
Thank you

Torch3333

Sorry for the late review, I left some comments, PTAL.

Torch3333 · 2025-01-28T08:01:22Z

...r/core/src/main/java/com/scalar/db/dataloader/core/dataimport/dao/ScalarDbStorageManger.java

+import java.io.IOException;
+import javax.annotation.Nullable;
+
+public class ScalarDbStorageManger {


Suggested change

public class ScalarDbStorageManger {

public class ScalarDbStorageManager {

Torch3333 · 2025-01-28T08:01:33Z

...re/src/main/java/com/scalar/db/dataloader/core/dataimport/dao/ScalarDbTransactionManger.java

+import com.scalar.db.service.TransactionFactory;
+import java.io.IOException;
+
+public class ScalarDbTransactionManger {


Suggested change

public class ScalarDbTransactionManger {

public class ScalarDbTransactionManager {

Torch3333 · 2025-01-28T08:20:28Z

...r/core/src/main/java/com/scalar/db/dataloader/core/dataimport/dao/ScalarDbStorageManger.java

+    storageAdmin = storageFactory.getStorageAdmin();
+  }
+
+  /** @return storage for ScalarDB connection that is running in storage mode */


Is it repetitive but without doing this, a warning appears when compiling the code. Even if you follow the Google style guide recommendation, another warning is triggered.

warning: [MissingSummary] A summary fragment is required; consider using the value of the @return block as a summary fragment instead. * @return Distributed Transaction manager for ScalarDB connection that is running in transaction ^ (see https://google.github.io/styleguide/javaguide.html#s7.2-summary-fragment) Did you mean '*Returns distributed Transaction manager for ScalarDB connection that is running in transaction'?

Suggested change

/** @return storage for ScalarDB connection that is running in storage mode */

Returns storage for ScalarDB connection that is running in storage mode

/** @return storage for ScalarDB connection that is running in storage mode */

...r/core/src/main/java/com/scalar/db/dataloader/core/dataimport/dao/ScalarDbStorageManger.java

...re/src/main/java/com/scalar/db/dataloader/core/dataimport/dao/ScalarDbTransactionManger.java

data-loader/core/src/main/java/com/scalar/db/dataloader/core/dataimport/dao/ScalarDBDao.java

inv-jishnu · 2025-01-28T10:12:01Z

@Torch3333 san,
I have made changes based on your feedback.
Please take a look at this again when you get a chance.
Thank you.

Torch3333

LGTM, thank you!

Co-authored-by: Peckstadt Yves <[email protected]>

inv-jishnu and others added 19 commits December 4, 2024 15:59

Util classes for data loader

753618b

Fix spotbug issue

8d39d02

Removed error message and added core error

bf94c49

Applied spotless

47be388

Fixed unit test failures

913eb1c

Merge branch 'master' into feat/data-loader/utils

1f204b8

Basic data import enum and exception

6cfa83a

Removed exception class for now

d381b2b

Added DECIMAL_FORMAT

67f2474

Path util class updated

14e3593

Feedback changes

a096d51

Merge branch 'master' into feat/data-loader/utils

dbf1940

Merge branch 'master' into feat/data-loader/utils

cd8add9

Changes

52890c8

Merge branch 'master' into feat/data-loader/import-data-1

5114639

Merge branch 'feat/data-loader/utils' into feat/data-loader/scaladb-dao

4f9cd75

Added ScalarDB Dao

1997eb8

Merge branch 'master' into feat/data-loader/scaladb-dao

91e6310

Remove unnecessary files

8a7338b

inv-jishnu marked this pull request as draft December 17, 2024 11:49

inv-jishnu changed the title ~~ScalarDB Dao and related files~~ Add ScalarDB Dao and related files Dec 17, 2024

Changes

e206073

ypeckstadt reviewed Dec 18, 2024

View reviewed changes

inv-jishnu added 5 commits December 18, 2024 12:12

Changes

26d3144

spotbugs exclude

b86487d

spotbugs exclude -2

818a2b4

Removed use of List.of to fix CI error

90abd9e

Merged changes from master after resolving conflict

ba2b3dd

inv-jishnu self-assigned this Dec 19, 2024

Minor change in test

03324e1

Removed scalardb manager file

39c43de

inv-jishnu requested a review from komamitsu January 16, 2025 08:19

komamitsu approved these changes Jan 17, 2025

View reviewed changes

brfrn169 approved these changes Jan 17, 2025

View reviewed changes

inv-jishnu and others added 3 commits January 20, 2025 10:37

Merge branch 'master' into feat/data-loader/scaladb-dao

3fe30a3

Removed wildcard import

4df4acd

Merge branch 'master' into feat/data-loader/scaladb-dao

53cd523

Torch3333 reviewed Jan 28, 2025

View reviewed changes

inv-jishnu and others added 2 commits January 28, 2025 14:32

Changes

f4f253e

Merge branch 'master' into feat/data-loader/scaladb-dao

6d43bdc

inv-jishnu requested review from ypeckstadt and Torch3333 January 28, 2025 10:12

Torch3333 approved these changes Jan 29, 2025

View reviewed changes

This was referenced Jan 29, 2025

Add import command for data loader CLI #2489

Draft

Add export command for data loader CLI #2490

Draft

ypeckstadt added 3 commits January 31, 2025 13:16

Merge branch 'master' into feat/data-loader/scaladb-dao

e492fad

Merge branch 'master' into feat/data-loader/scaladb-dao

7d50971

Merge branch 'master' into feat/data-loader/scaladb-dao

94039b2

ypeckstadt merged commit 08dd893 into master Jan 31, 2025
48 checks passed

ypeckstadt deleted the feat/data-loader/scaladb-dao branch January 31, 2025 08:11

feeblefakie pushed a commit that referenced this pull request Jan 31, 2025

Add ScalarDB Dao and related files (#2417)

1b25b3c

Co-authored-by: Peckstadt Yves <[email protected]>

feeblefakie mentioned this pull request Jan 31, 2025

Backport to branch(3.15) : Add ScalarDB Dao and related files #2507

Merged

feeblefakie pushed a commit that referenced this pull request Jan 31, 2025

Add ScalarDB Dao and related files (#2417)

8d1ec5c

Co-authored-by: Peckstadt Yves <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ScalarDB Dao and related files #2417

Add ScalarDB Dao and related files #2417

inv-jishnu commented Dec 17, 2024 •

edited by ypeckstadt

Loading

inv-jishnu commented Jan 16, 2025

komamitsu left a comment

brfrn169 left a comment •

edited

Loading

brfrn169 Jan 17, 2025 •

edited

Loading

inv-jishnu Jan 20, 2025

Torch3333 left a comment

Torch3333 Jan 28, 2025

Torch3333 Jan 28, 2025

Torch3333 Jan 28, 2025 •

edited

Loading

inv-jishnu commented Jan 28, 2025

Torch3333 left a comment

		@@ -0,0 +1,425 @@
		package com.scalar.db.dataloader.core.dataimport.dao;

		import com.scalar.db.api.*;

	public class ScalarDbStorageManger {
	public class ScalarDbStorageManager {

	public class ScalarDbTransactionManger {
	public class ScalarDbTransactionManager {

	/** @return storage for ScalarDB connection that is running in storage mode */
	Returns storage for ScalarDB connection that is running in storage mode
	/** @return storage for ScalarDB connection that is running in storage mode */

Add ScalarDB Dao and related files #2417

Add ScalarDB Dao and related files #2417

Conversation

inv-jishnu commented Dec 17, 2024 • edited by ypeckstadt Loading

Description

Related issues and/or PRs

Changes made

Checklist

Additional notes (optional)

Release notes

inv-jishnu commented Jan 16, 2025

komamitsu left a comment

Choose a reason for hiding this comment

brfrn169 left a comment • edited Loading

Choose a reason for hiding this comment

brfrn169 Jan 17, 2025 • edited Loading

Choose a reason for hiding this comment

inv-jishnu Jan 20, 2025

Choose a reason for hiding this comment

Torch3333 left a comment

Choose a reason for hiding this comment

Torch3333 Jan 28, 2025

Choose a reason for hiding this comment

Torch3333 Jan 28, 2025

Choose a reason for hiding this comment

Torch3333 Jan 28, 2025 • edited Loading

Choose a reason for hiding this comment

inv-jishnu commented Jan 28, 2025

Torch3333 left a comment

Choose a reason for hiding this comment

inv-jishnu commented Dec 17, 2024 •

edited by ypeckstadt

Loading

brfrn169 left a comment •

edited

Loading

brfrn169 Jan 17, 2025 •

edited

Loading

Torch3333 Jan 28, 2025 •

edited

Loading