Add port_count=1 to DQ job in YT #13459
Conversation
|
⚪ Test history | Ya make output | Test bloat
🟢 |
|
⚪ Test history | Ya make output | Test bloat
🟢 |
| @@ -678,6 +678,7 @@ namespace NYql { | |||
| fluent.Item("file_paths").Value(*filePaths); | |||
| }) | |||
| .Item("job_count").Value(1) | |||
| .Item("port_count").Value(1) | |||
There was a problem hiding this comment.
We should specify the port range also. (31002...31002 + 100 default)
Now we can receive not appropriate one.
https://github.com/ydb-platform/ydb/blob/main/ydb/library/yql/providers/dq/config/config.proto#L110
https://github.com/ydb-platform/ydb/blob/main/ydb/library/yql/tools/dq/worker_job/dq_worker.cpp#L189
There was a problem hiding this comment.
Port requesting doesn't work like that. You can't specify port range that you need (and it is not clear why an user would want to specify that)
You just specify port_count = 1 and the job will have YT_PORT_0 env variable set with a port assigned. This port will be from a range configured in exec nodes static config and may be different for different exec nodes
This PR doesn't use this port, it only requests one to avoid placing on nodes with 0 available ports, since they are more likely to be unavailable on 31002
Changelog category
Additional information
Currently, DQ job binds on some port and expects it to be reachable from yql agent
Ideally, it should declare
port_count = 1and use the port provided by exec nodeThis PR adds
port_count = 1to this job to prevent it from being placed on portless exec nodes (that do not have ports)I created an issue to ytsaurus to start using this port: ytsaurus/ytsaurus#1034