【2.3.5】【又来BUG了】服务器端 mysql 实时同步到starrocks 报错

[andyuan1000]

Search before asking

• I had searched in the issues and found no similar issues.

What happened

===============================================================================

2025-01-03 16:13:18,745 ERROR [o.a.s.c.s.SeaTunnel ] [main] - Fatal Error,

2025-01-03 16:13:18,745 ERROR [o.a.s.c.s.SeaTunnel ] [main] - Please submit bug report in https://github.com/apache/seatunnel/issues

2025-01-03 16:13:18,745 ERROR [o.a.s.c.s.SeaTunnel ] [main] - Reason:SeaTunnel job executed failed

2025-01-03 16:13:18,747 ERROR [o.a.s.c.s.SeaTunnel ] [main] - Exception StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:202)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: java.lang.IllegalArgumentException: Miss config! Please check the config file.
at org.apache.seatunnel.shade.com.google.common.base.Preconditions.checkArgument(Preconditions.java:141)
at org.apache.seatunnel.core.starter.utils.ConfigShadeUtils.processConfig(ConfigShadeUtils.java:148)
at org.apache.seatunnel.core.starter.utils.ConfigShadeUtils.decryptConfig(ConfigShadeUtils.java:119)
at org.apache.seatunnel.core.starter.utils.ConfigShadeUtils.decryptConfig(ConfigShadeUtils.java:104)
at org.apache.seatunnel.core.starter.utils.ConfigBuilder.ofInner(ConfigBuilder.java:53)
at org.apache.seatunnel.core.starter.utils.ConfigBuilder.lambda$of$1(ConfigBuilder.java:67)
at java.base/java.util.Optional.orElseGet(Optional.java:369)
at org.apache.seatunnel.core.starter.utils.ConfigBuilder.of(ConfigBuilder.java:67)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.(MultipleTableJobConfigParser.java:130)
at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.getJobConfigParser(ClientJobExecutionEnvironment.java:82)
at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.getLogicalDag(ClientJobExecutionEnvironment.java:88)
at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.execute(ClientJobExecutionEnvironment.java:156)
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:149)
... 2 more

2025-01-03 16:13:18,747 ERROR [o.a.s.c.s.SeaTunnel ] [main] -

Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:202)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: java.lang.IllegalArgumentException: Miss config! Please check the config file.
at org.apache.seatunnel.shade.com.google.common.base.Preconditions.checkArgument(Preconditions.java:141)
at org.apache.seatunnel.core.starter.utils.ConfigShadeUtils.processConfig(ConfigShadeUtils.java:148)
at org.apache.seatunnel.core.starter.utils.ConfigShadeUtils.decryptConfig(ConfigShadeUtils.java:119)
at org.apache.seatunnel.core.starter.utils.ConfigShadeUtils.decryptConfig(ConfigShadeUtils.java:104)
at org.apache.seatunnel.core.starter.utils.ConfigBuilder.ofInner(ConfigBuilder.java:53)
at org.apache.seatunnel.core.starter.utils.ConfigBuilder.lambda$of$1(ConfigBuilder.java:67)
at java.base/java.util.Optional.orElseGet(Optional.java:369)
at org.apache.seatunnel.core.starter.utils.ConfigBuilder.of(ConfigBuilder.java:67)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.(MultipleTableJobConfigParser.java:130)
at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.getJobConfigParser(ClientJobExecutionEnvironment.java:82)
at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.getLogicalDag(ClientJobExecutionEnvironment.java:88)
at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.execute(ClientJobExecutionEnvironment.java:156)
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:149)
... 2 more
[root@localhost apache-seatunnel-2.3.5.bak4]#

SeaTunnel Version

2.3.5

SeaTunnel Config

#定义一些作业的运行参数,具体可以参考 https://seatunnel.apache.org/docs/2.3.1/concept/JobEnvConfig
env {
	parallelism=1 # 并行度，这里是启动5个Source Task来并行的读取数据
    job.mode="STREAMING"  #作业的运行模式，BATCH=离线批同步，STREAMING=实时同步
    job.name="SeaTunnel_market_new_121_163_sr_STREAMING_Job"
    checkpoint.interval=20000 #每10000ms进行一次checkpoint，后面会详细介绍checkpoint对JDBC Source和StarRocks Sink这两个连接器的影响
}

source{
    MySQL-CDC {
		result_table_name = test
        base-url = "jdbc:mysql://120.46.249.147:33306/market_new2?useSSL=false&serverTimezone=Asia/Shanghai"
        driver = "com.mysql.cj.jdbc.Driver"
        username = "root"
        password = "Psjt@2020"
        table-names = ["market_new2.t_electronic_back_order"]
        startup.mode = "earliest"
		format=COMPATIBLE_DEBEZIUM_JSON
    }
}

transform{
	sql {
		source_table_name = "test"
		result_table_name = "test2"
		query = "select id from test"
	}
}

sink {
  StarRocks {
	source_table_name = test2
    nodeUrls = ["120.46.249.147:8030"]
    base-url = "jdbc:mysql://120.46.249.147:9030/"
    username="root"
    password="Hdic@2024"
    database = "market_new_dw"
    table = "t_electronic_back_order_st"
    batch_max_rows = 10
	starrocks.config = {
      format = "JSON"
      strip_outer_array = true
    }
  }
}

Running Command

./bin/seatunnel.sh --config ./config/mysql_to_sr_stream.config -m local

Error Exception

=null,os.committedVirtualMemorySize=0,os.freePhysicalMemorySize=0,os.freeSwapSpaceSize=0,os.maxFileDescriptorCount=0,os.openFileDescriptorCount=0,os.processCpuTime=0,os.systemLoadAverage=0.17,os.totalPhysicalMemorySize=0,os.totalSwapSpaceSize=0,runtime.availableProcessors=8,runtime.freeMemory=228487024,runtime.maxMemory=536870912,runtime.totalMemory=268435456,runtime.uptime=2890,runtime.usedMemory=39948432, labels=[]}
2025-01-03 16:13:15,711 INFO  [c.h.i.p.i.MigrationManager    ] [hz.main.cached.thread-4] - [localhost]:5801 [seatunnel-405198] [5.1] Shutdown request of Member [localhost]:5801 - c4ecbf4b-4b3c-43c8-93a9-0355ee398688 this is handled
2025-01-03 16:13:15,718 INFO  [c.h.i.i.Node                  ] [main] - [localhost]:5801 [seatunnel-405198] [5.1] Shutting down connection manager...
2025-01-03 16:13:15,720 INFO  [c.h.i.i.Node                  ] [main] - [localhost]:5801 [seatunnel-405198] [5.1] Shutting down node engine...
2025-01-03 16:13:15,728 INFO  [o.a.s.e.s.CoordinatorService  ] [pool-5-thread-1] - [localhost]:5801 [seatunnel-405198] [5.1] This node become leave active master node, begin clear coordinator service
2025-01-03 16:13:15,728 INFO  [.c.c.DefaultClassLoaderService] [main] - close classloader service
2025-01-03 16:13:15,728 INFO  [o.a.s.e.s.TaskExecutionService] [event-forwarder-0] - [localhost]:5801 [seatunnel-405198] [5.1] Event forward thread interrupted
2025-01-03 16:13:18,743 INFO  [c.h.i.i.NodeExtension         ] [main] - [localhost]:5801 [seatunnel-405198] [5.1] Destroying node NodeExtension.
2025-01-03 16:13:18,744 INFO  [c.h.i.i.Node                  ] [main] - [localhost]:5801 [seatunnel-405198] [5.1] Hazelcast Shutdown is completed in 3033 ms.
2025-01-03 16:13:18,744 INFO  [c.h.c.LifecycleService        ] [main] - [localhost]:5801 [seatunnel-405198] [5.1] [localhost]:5801 is SHUTDOWN
2025-01-03 16:13:18,745 INFO  [s.c.s.s.c.ClientExecuteCommand] [main] - Closed HazelcastInstance ......
2025-01-03 16:13:18,745 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - 

===============================================================================


2025-01-03 16:13:18,745 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - Fatal Error, 

2025-01-03 16:13:18,745 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - Please submit bug report in https://github.com/apache/seatunnel/issues

2025-01-03 16:13:18,745 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - Reason:SeaTunnel job executed failed 

2025-01-03 16:13:18,747 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - Exception StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
	at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:202)
	at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
	at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: java.lang.IllegalArgumentException: Miss <Sink> config! Please check the config file.
	at org.apache.seatunnel.shade.com.google.common.base.Preconditions.checkArgument(Preconditions.java:141)
	at org.apache.seatunnel.core.starter.utils.ConfigShadeUtils.processConfig(ConfigShadeUtils.java:148)
	at org.apache.seatunnel.core.starter.utils.ConfigShadeUtils.decryptConfig(ConfigShadeUtils.java:119)
	at org.apache.seatunnel.core.starter.utils.ConfigShadeUtils.decryptConfig(ConfigShadeUtils.java:104)
	at org.apache.seatunnel.core.starter.utils.ConfigBuilder.ofInner(ConfigBuilder.java:53)
	at org.apache.seatunnel.core.starter.utils.ConfigBuilder.lambda$of$1(ConfigBuilder.java:67)
	at java.base/java.util.Optional.orElseGet(Optional.java:369)
	at org.apache.seatunnel.core.starter.utils.ConfigBuilder.of(ConfigBuilder.java:67)
	at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.<init>(MultipleTableJobConfigParser.java:130)
	at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.getJobConfigParser(ClientJobExecutionEnvironment.java:82)
	at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.getLogicalDag(ClientJobExecutionEnvironment.java:88)
	at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.execute(ClientJobExecutionEnvironment.java:156)
	at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:149)
	... 2 more
 
2025-01-03 16:13:18,747 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - 
===============================================================================



Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
	at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:202)
	at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
	at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: java.lang.IllegalArgumentException: Miss <Sink> config! Please check the config file.
	at org.apache.seatunnel.shade.com.google.common.base.Preconditions.checkArgument(Preconditions.java:141)
	at org.apache.seatunnel.core.starter.utils.ConfigShadeUtils.processConfig(ConfigShadeUtils.java:148)
	at org.apache.seatunnel.core.starter.utils.ConfigShadeUtils.decryptConfig(ConfigShadeUtils.java:119)
	at org.apache.seatunnel.core.starter.utils.ConfigShadeUtils.decryptConfig(ConfigShadeUtils.java:104)
	at org.apache.seatunnel.core.starter.utils.ConfigBuilder.ofInner(ConfigBuilder.java:53)
	at org.apache.seatunnel.core.starter.utils.ConfigBuilder.lambda$of$1(ConfigBuilder.java:67)
	at java.base/java.util.Optional.orElseGet(Optional.java:369)
	at org.apache.seatunnel.core.starter.utils.ConfigBuilder.of(ConfigBuilder.java:67)
	at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.<init>(MultipleTableJobConfigParser.java:130)
	at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.getJobConfigParser(ClientJobExecutionEnvironment.java:82)
	at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.getLogicalDag(ClientJobExecutionEnvironment.java:88)
	at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.execute(ClientJobExecutionEnvironment.java:156)
	at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:149)
	... 2 more

Zeta or Flink or Spark Version

无

Java or Scala Version

11

Screenshots

=null,os.committedVirtualMemorySize=0,os.freePhysicalMemorySize=0,os.freeSwapSpaceSize=0,os.maxFileDescriptorCount=0,os.openFileDescriptorCount=0,os.processCpuTime=0,os.systemLoadAverage=0.17,os.totalPhysicalMemorySize=0,os.totalSwapSpaceSize=0,runtime.availableProcessors=8,runtime.freeMemory=228487024,runtime.maxMemory=536870912,runtime.totalMemory=268435456,runtime.uptime=2890,runtime.usedMemory=39948432, labels=[]}
2025-01-03 16:13:15,711 INFO [c.h.i.p.i.MigrationManager ] [hz.main.cached.thread-4] - [localhost]:5801 [seatunnel-405198] [5.1] Shutdown request of Member [localhost]:5801 - c4ecbf4b-4b3c-43c8-93a9-0355ee398688 this is handled
2025-01-03 16:13:15,718 INFO [c.h.i.i.Node ] [main] - [localhost]:5801 [seatunnel-405198] [5.1] Shutting down connection manager...
2025-01-03 16:13:15,720 INFO [c.h.i.i.Node ] [main] - [localhost]:5801 [seatunnel-405198] [5.1] Shutting down node engine...
2025-01-03 16:13:15,728 INFO [o.a.s.e.s.CoordinatorService ] [pool-5-thread-1] - [localhost]:5801 [seatunnel-405198] [5.1] This node become leave active master node, begin clear coordinator service
2025-01-03 16:13:15,728 INFO [.c.c.DefaultClassLoaderService] [main] - close classloader service
2025-01-03 16:13:15,728 INFO [o.a.s.e.s.TaskExecutionService] [event-forwarder-0] - [localhost]:5801 [seatunnel-405198] [5.1] Event forward thread interrupted
2025-01-03 16:13:18,743 INFO [c.h.i.i.NodeExtension ] [main] - [localhost]:5801 [seatunnel-405198] [5.1] Destroying node NodeExtension.
2025-01-03 16:13:18,744 INFO [c.h.i.i.Node ] [main] - [localhost]:5801 [seatunnel-405198] [5.1] Hazelcast Shutdown is completed in 3033 ms.
2025-01-03 16:13:18,744 INFO [c.h.c.LifecycleService ] [main] - [localhost]:5801 [seatunnel-405198] [5.1] [localhost]:5801 is SHUTDOWN
2025-01-03 16:13:18,745 INFO [s.c.s.s.c.ClientExecuteCommand] [main] - Closed HazelcastInstance ......
2025-01-03 16:13:18,745 ERROR [o.a.s.c.s.SeaTunnel ] [main] -

===============================================================================

2025-01-03 16:13:18,745 ERROR [o.a.s.c.s.SeaTunnel ] [main] - Fatal Error,

2025-01-03 16:13:18,745 ERROR [o.a.s.c.s.SeaTunnel ] [main] - Please submit bug report in https://github.com/apache/seatunnel/issues

2025-01-03 16:13:18,745 ERROR [o.a.s.c.s.SeaTunnel ] [main] - Reason:SeaTunnel job executed failed

2025-01-03 16:13:18,747 ERROR [o.a.s.c.s.SeaTunnel ] [main] - Exception StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:202)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: java.lang.IllegalArgumentException: Miss config! Please check the config file.
at org.apache.seatunnel.shade.com.google.common.base.Preconditions.checkArgument(Preconditions.java:141)
at org.apache.seatunnel.core.starter.utils.ConfigShadeUtils.processConfig(ConfigShadeUtils.java:148)
at org.apache.seatunnel.core.starter.utils.ConfigShadeUtils.decryptConfig(ConfigShadeUtils.java:119)
at org.apache.seatunnel.core.starter.utils.ConfigShadeUtils.decryptConfig(ConfigShadeUtils.java:104)
at org.apache.seatunnel.core.starter.utils.ConfigBuilder.ofInner(ConfigBuilder.java:53)
at org.apache.seatunnel.core.starter.utils.ConfigBuilder.lambda$of$1(ConfigBuilder.java:67)
at java.base/java.util.Optional.orElseGet(Optional.java:369)
at org.apache.seatunnel.core.starter.utils.ConfigBuilder.of(ConfigBuilder.java:67)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.(MultipleTableJobConfigParser.java:130)
at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.getJobConfigParser(ClientJobExecutionEnvironment.java:82)
at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.getLogicalDag(ClientJobExecutionEnvironment.java:88)
at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.execute(ClientJobExecutionEnvironment.java:156)
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:149)
... 2 more

2025-01-03 16:13:18,747 ERROR [o.a.s.c.s.SeaTunnel ] [main] -

Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:202)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: java.lang.IllegalArgumentException: Miss config! Please check the config file.
at org.apache.seatunnel.shade.com.google.common.base.Preconditions.checkArgument(Preconditions.java:141)
at org.apache.seatunnel.core.starter.utils.ConfigShadeUtils.processConfig(ConfigShadeUtils.java:148)
at org.apache.seatunnel.core.starter.utils.ConfigShadeUtils.decryptConfig(ConfigShadeUtils.java:119)
at org.apache.seatunnel.core.starter.utils.ConfigShadeUtils.decryptConfig(ConfigShadeUtils.java:104)
at org.apache.seatunnel.core.starter.utils.ConfigBuilder.ofInner(ConfigBuilder.java:53)
at org.apache.seatunnel.core.starter.utils.ConfigBuilder.lambda$of$1(ConfigBuilder.java:67)
at java.base/java.util.Optional.orElseGet(Optional.java:369)
at org.apache.seatunnel.core.starter.utils.ConfigBuilder.of(ConfigBuilder.java:67)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.(MultipleTableJobConfigParser.java:130)
at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.getJobConfigParser(ClientJobExecutionEnvironment.java:82)
at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.getLogicalDag(ClientJobExecutionEnvironment.java:88)
at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.execute(ClientJobExecutionEnvironment.java:156)
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:149)
... 2 more

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct