PBS jobs queued are not running

[xpillons]

The jobs are in queue and not running. Tried to terminate execute nodes through portal and manually, but they did not stop.
PBS Version = 19.1.1

Cyclecloud service behaves abnormally like not starting and stopping the VM’s properly.

Initially 40 VM’s are showed ready state in Azure cyclecloud monitoring portal, but none of the jobs are Running, instead they are in Queue.
Stopped the VM’s manually and Cyclecloud allocated the new VM’s to the Queued jobs and after sometime jobs are started Running.

Jobs are completed after sometime. but the VM’s are not terminated automatically.

[root@scheduler bin]# source /opt/cycle/pbspro/venv/bin/activate
(venv) [root@scheduler bin]# pip list | grep pbspro
cyclecloud-pbspro   2.0.9

[xpillons]

Should be fixed by Azure/cyclecloud-pbspro@9d32c0f

[xpillons]

Re-open as 2.0.13 is introducing new issues

[xpillons]

The Root Cause is a bad formatted JSON output provided by OpenPBS which makes the autoscaler library to failed, interrupting the calculation engine.
This bad formatted JSON is coming from an environment variable provided to the job inherited from the modules loaded, and passed to the qsub command thru the -V option. The bad thing here is that sometimes these kind of jobs can start especially if there are available resources to run them, but they will impact any upcoming jobs for which new resources need to be provisioned.

Workaround => don't qsub with the -V option but instead pass only the environment variables needed

Azure/cyclecloud-pbspro#43