Spark Doris Connector can not stop #16933
voyagertanyao
started this conversation in
General
Replies: 1 comment
-
Finally, I found there is a net proxy on my local env, turning off this proxy can make it success. So doris spark connector has some relation with proxy ? And this relation seems to exist only when there is a |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hi, I am using spark-dorms-connector-3.1_2.12_1.1.0 release and Doris 1.2.0-rc04
here is the problem.
Here is my application code :
every thing works fine when I built my spark application into a jar ,and submit this spark application in linux env.
However, When I debug my code in IDEA, things were different, I can query the newly inserted data in Doris, it means my
sql0
executed ,but my application in idea cannot execute the next query sqlspark.sql("select * from doris_bigdata_user_info").show(false)
.This picture shows that spark application hang up there for too long, but data has already into doris (duplicate data was produced because of retrying for many times by myself ).
I debug this issue for a while, I found that sql0 executed as a Doris stream load http request to BE, and got the response from BE . The code
while ((line = br.readLine()) != null)
worked well, but after that, it can not reach the codereturn new LoadResponse(status, respMsg, response.toString());
in org.apache.doris.spark.DorisStreamLoad in functionprivate LoadResponse loadBatch(String value)
, even can not reachcatch
block orfinally block
.And Here is spark-doris code
I think that's the reason why my application hang up somewhere, and spark con not stop. Or any other reason for my application can not stop in IDEA env?
Beta Was this translation helpful? Give feedback.
All reactions