[Bug] [Http Source Connector v2] Post request with application/json body doesn't work for paging Because of paging info was set as params which add header application/x-www-form-urlencoded

[chenliejun]

Search before asking

• I had searched in the issues and found no similar issues.

What happened

I want to pull data from http via post request with body and paging like:
{
"pageIndex":1,
"pageSize":10,
"condition":{
"groupOp":"and",
"rules":[{
"field":"HTLX",
"op":"li",
"data":"xxx"
}
]
}
}

the pageIndex as the pageing.page_field to increment auto,as:1,2,3...

with the conf below got Content type 'application/x-www-form-urlencoded;charset=UTF-8' not supported，Because the server only support application/json as request body.

I review the Seatunnel dev branch source code ,And find the paging info has been treated as request params （UrlEncodedFormEntity）which add header “Content-Type ：application/x-www-form-urlencoded;charset=UTF-8”

SeaTunnel Version

2.3.8

SeaTunnel Config

env {
  parallelism = 1
  job.mode = "BATCH"
  job.name = "chen_http_json"
  checkpoint.interval = 10000
}

source {
  Http {
    result_table_name = "step1"
    plugin_output = "http"
    url = "http://10.24.73.96/api/query/httpsjon"
    method = "POST"
    format = "json"
    headers = {
      "Authorization":"Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ1aWQiOiI0MGRkOWE4NmQ1OGNkMjNjMjgwOTNlMzY5NTE0MGJhNiIsIm5hbWUiOiJjaGVuIiwic2VjcmVjeUxldmVsIjozLCJyb2xlIjoibGVhZGVyIiwianRpIjoiQjZCODgyNUFFNDExNDFGOUJGRTczRDZEOTZFMUNFMDIiLCJpc3MiOiJjaGVuIiwiZXhwIjoxNzM0NTczNjM0LCJuYmYiOjE3MzQ1NzAwMzR9.xBi-5djUCz3o0T1na9RxKJN32MXIXQ2mkPReUIl5UDA"
    }
    body = "{\"pageIndex\":\"${pageIndex}\",\"pageSize\":10,\"condition\":{\"groupOp\":\"and\",\"rules\":[{\"field\":\"HTLX\",\"op\":\"li\",\"data\":\"合同\"}]}}"
    content_field = "$.data.list.*"
    pageing={
       page_field = pageIndex
       start_page_number = 2
       batch_size = 10
    }
    schema = {
      fields {
        HTID = string
        KHID = string
        HTLX = string
        ZCRQ = timestamp
        SFYJJH = string
      }
    }
  }
}

sink {
  Jdbc {
      source_table_name = "step1"
      url = "jdbc:mysql://xxxx:3307/chen_test?serverTimezone=Asia/Shanghai&useUnicode=true&characterEncoding=utf8&allowPublicKeyRetrieval=true&useSSL=false"
      driver = "com.mysql.cj.jdbc.Driver"    
      max_retries = 0
      user = "root"
      password = "xxxx"
      generate_sink_sql = true
      database = chen_test
      table = new_slsj
    }
}

Running Command

seatunnel.sh -c $SEATUNNEL_HOME/scripts/chen_http_json.conf -n chen_http_json

Error Exception

the http server shows: ERROR [xxx]-Content type 'application/x-www-form-urlencoded;charset=UTF-8' not supported

Zeta or Flink or Spark Version

Zeta

Java or Scala Version

Java 17

Screenshots

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[CosmosNi]

Pagination in body is not supported yet.

[chenliejun]

But it's really an common requirement，can the team take it as an new feature in the next version?

[CosmosNi]

This can be discussed.

[wu-a-ge]

This function is badly designed and cannot be used in at least 80% of the current business scenarios