Connector X in a Kubernetes job in Google Cloud Platform

[abel-diaz]

Hi All,
Would Connector X work in a kubernetes job ran in GCP? Would the logic apply to give the number of cores equal to the number of CPUs you assign your job? If we are running jobs in GCP with 1 CPU, (bc its expensive to use alot of CPU) OR (the job only requires 1 CPU or less) would Connector X still utilize this 1 CPU for efficiency? What happens when jobs have .5 of a CPU?

Q: How affective is connector x in GCP?

UPDATE: I ran a 6-core job in GCP with a connector x package install specific version, the job had allocated 1 CPU, I assigned 6 to CX. The process did not fail, it ran successfully. The Core assignment vs actual cores does not fail the job. I was able to pull in 250k rows in less than 40 seconds. Not sure how it worked but it did.

Thanks
Abel

[wangxiaoying]

Hi @abel-diaz, the partition_num parameter in our library equals to the number of partitioned queries we split from a single query. Underlying, we use rayon as our parallel executor, which would use a pool of threads (we use the default setting which equals to the number of logical cores on the machine) to handle these tasks we submit to it (each partitioned query can be seen as a single task). Therefore, if you assign the number to 6, it only means that connectorx will create 6 independent tasks that can be run in parallel but the underlying rayon may only have 1 thread (if only 1 logical core is available), then they will be handled sequentially. There will be no error, however in this case it may not outperform the case if you do not partition the query since data will be handled sequentially anyway and there will be overheads introduced by the partitioning.