Replicate prediction fails due to "queue reached limit"

[platform-kit]

Occasionally getting the following error on this verison of the Replicate-hosted model

[Predictor Predict] Params: {'prompt': 'Happy droid face, glowing digital eyes, detailed, cyberpunk, scratched and battle-damaged   ', 'negative_prompt': 'perfect, fake, photoshopped, airbrushed, too beautiful', 'style_selections': ['Fooocus V2', 'Fooocus Enhance', 'Fooocus Sharp'], 'performance_selection': 'Quality', 'aspect_ratios_selection': '1024*1024', 'image_number': 1, 'image_seed': -1, 'sharpness': 10.0, 'guidance_scale': 4.0, 'base_model_name': 'juggernautXL_version6Rundiffusion.safetensors', 'refiner_model_name': 'None', 'refiner_switch': 0.4, 'loras': [['sd_xl_offset_example-lora_1.0.safetensors', 0.1]], 'uov_input_image': array([[[ 1, 36, 68],
[ 2, 37, 69],
[ 3, 39, 73],
...,
[51, 24, 13],
[51, 23, 11],
[51, 23, 11]],
[[ 3, 34, 63],
[ 1, 34, 65],
[ 2, 37, 67],
...,
[52, 25, 14],
[52, 24, 12],
[52, 24, 12]],
[[ 3, 33, 59],
[ 3, 33, 61],
[ 2, 36, 64],
...,
[53, 26, 15],
[53, 27, 14],
[53, 27, 14]],
...,
[[25, 12, 19],
[26, 14, 18],
[27, 15, 17],
...,
[32, 16, 16],
[32, 16, 16],
[32, 16, 16]],
[[24, 11, 18],
[25, 13, 17],
[29, 14, 17],
...,
[32, 16, 16],
[32, 16, 16],
[32, 16, 16]],
[[22,  9, 16],
[24, 12, 16],
[28, 13, 16],
...,
[31, 15, 15],
[31, 15, 15],
[31, 15, 15]]], dtype=uint8), 'uov_method': 'Vary (Subtle)', 'upscale_value': 0.0, 'outpaint_selections': [], 'outpaint_distance_left': 0, 'outpaint_distance_right': 0, 'outpaint_distance_top': 0, 'outpaint_distance_bottom': 0, 'inpaint_input_image': None, 'inpaint_additional_prompt': '', 'image_prompts': [], 'advanced_params': [False, 1.5, 0.8, 0.3, 7.0, 'dpmpp_2m_sde_gpu', 'karras', False, -1, -1, -1, -1, -1, -1, False, False, False, False, 0.25, 64, 128, 'joint', False, None, None, None, None, False, False, 'v2.6', 1.0, 0.618, False, False, 0]}
[Task Queue] The task queue has reached limit
Traceback (most recent call last):
File "/root/.pyenv/versions/3.11.7/lib/python3.11/site-packages/cog/server/worker.py", line 221, in _predict
result = predict(**payload)
^^^^^^^^^^^^^^^^^^
File "/src/predict.py", line 189, in predict
raise Exception(
Exception: The task queue has reached limit.

[tianye2856]

I also have this problem! please fix this!

[tianye2856]

It seems that in some situation, we have these problems.
Replicate keeps running for more than 30 mintues.

[Predictor Predict] Params: {'prompt': 'blank', 'negative_prompt': '', 'style_selections': ['Fooocus V2', 'Fooocus Enhance', 'Fooocus Sharp'], 'performance_selection': 'Speed', 'aspect_ratios_selection': '1152*896', 'image_number': 3, 'image_seed': -1, 'sharpness': 2.0, 'guidance_scale': 4.0, 'base_model_name': 'juggernautXL_version6Rundiffusion.safetensors', 'refiner_model_name': 'None', 'refiner_switch': 0.5, 'loras': [['sd_xl_offset_example-lora_1.0.safetensors', 0.1]], 'uov_input_image': None, 'uov_method': 'Disabled', 'upscale_value': 0, 'outpaint_selections': [], 'outpaint_distance_left': 0, 'outpaint_distance_right': 0, 'outpaint_distance_top': 0, 'outpaint_distance_bottom': 0, 'inpaint_input_image': {'image': array([[[ 74, 74, 76], [ 81, 81, 81], [ 96, 92, 93], ..., [124, 172, 244], [124, 172, 244], [124, 172, 244]], [[ 81, 81, 83], [ 78, 78, 78], [ 87, 86, 84], ..., [124, 172, 244], [124, 172, 244], [124, 172, 244]], [[ 76, 78, 77], [ 82, 84, 81], [ 85, 86, 81], ..., [124, 172, 244], [124, 172, 244], [124, 172, 244]], ..., [[130, 97, 80], [102, 69, 52], [ 96, 65, 47], ..., [220, 187, 156], [223, 188, 158], [203, 166, 137]], [[ 95, 64, 46], [136, 105, 85], [167, 136, 116], ..., [230, 196, 168], [210, 175, 147], [198, 161, 134]], [[169, 140, 122], [166, 138, 117], [181, 150, 129], ..., [232, 198, 171], [227, 192, 164], [212, 175, 148]]], dtype=uint8), 'mask': array([[[0, 0, 0], [0, 0, 0], [0, 0, 0], ..., [0, 0, 0], [0, 0, 0], [0, 0, 0]], [[0, 0, 0], [0, 0, 0], [0, 0, 0], ..., [0, 0, 0], [0, 0, 0], [0, 0, 0]], [[0, 0, 0], [0, 0, 0], [0, 0, 0], ..., [0, 0, 0], [0, 0, 0], [0, 0, 0]], ..., [[0, 0, 0], [0, 0, 0], [0, 0, 0], ..., [0, 0, 0], [0, 0, 0], [0, 0, 0]], [[0, 0, 0], [0, 0, 0], [0, 0, 0], ..., [0, 0, 0], [0, 0, 0], [0, 0, 0]], [[0, 0, 0], [0, 0, 0], [0, 0, 0], ..., [0, 0, 0], [0, 0, 0], [0, 0, 0]]], dtype=uint8)}, 'inpaint_additional_prompt': 'blank', 'image_prompts': [], 'advanced_params': [False, 1.5, 0.8, 0.3, 7.0, 'dpmpp_2m_sde_gpu', 'karras', False, -1, -1, -1, -1, -1, -1, False, False, False, False, 0.25, 64, 128, 'joint', False, None, None, None, None, False, False, 'v2.6', 1.0, 0.618, False, False, 0]} [Task Queue] Waiting for task queue become free, job_id=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 10.007062062853947S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 20.014418507926166S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 30.0218622549437S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 40.028930021915585S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 50.03575427993201S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 60.04248439683579S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 70.04935472784564S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 80.05656476900913S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 90.06363079394214S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 100.07082859193906S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 110.07785842684098S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 120.08484788797796S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 130.0920909298584S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 140.09897892083973S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 150.10596743389033S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 160.1135832699947S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 170.12095082295127S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 180.12857779185288S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 190.1362078608945S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 200.14319298486225S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 210.1502723558806S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 220.15765971294604S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 230.16497894492932S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 240.17204198101535S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 250.17898906581104S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 260.1860635408666S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 270.193634878844S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 280.20123489084654S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 290.2087028529495S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 300.21628246596083S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 310.2237159048673S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 320.23141511087306S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 330.2388297249563S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 340.24625187995844S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 350.25381605583243S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 360.2613902448211S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 370.26900831982493S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 380.27627761801705S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 390.2837189990096S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 400.2913855109364S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 410.29907488799654S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 420.30664639989845S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 430.3141851988621S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 440.3211563460063S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 450.3285063118674S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 460.3361775628291S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 470.3438190568704S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 480.35115876584314S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 490.3585119009949S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 500.3661899289582S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 510.3735545608215S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 520.381165180821S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 530.3889270578511S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 540.3972873818129S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 550.4052523239516S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 560.4130223880056S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 570.4210287039168S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 580.4285670169629S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 590.4358926790301S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 600.4432465089485S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 610.4503738270141S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 620.4575246169697S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 630.46495275083S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 640.4718695830088S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 650.4789432438556S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 660.4859967669472S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 670.4929405408911S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 680.5000852348749S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 690.5078391388524S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 700.5153696518391S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 710.5228903398383S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 720.5303242669906S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 730.538118518889S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 740.5456220658962S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 750.5531302348245S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 760.5600987989455S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 770.5673371190205S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 780.5744550428353S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 790.5818805620074S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 800.5896203429438S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 810.5971197679173S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 820.60447448981S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 830.6115462519228S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 840.6190021429211S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 850.6275331380311S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 860.6353850548621S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 870.6430635279976S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 880.6501444128808S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 890.6577132169623S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 900.665146590909S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 910.6729143268894S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 920.6799327749759S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 930.6868865999859S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 940.6937658328097S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 950.7007771388162S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 960.7078005408403S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 970.7149571948685S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 980.7219304540195S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 990.7294293888845S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1000.7369781499729S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1010.7445775319356S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1020.7520111729391S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1030.759141099872S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1040.7660808088258S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1050.773386160843S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1060.7805021109525S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1070.7877177828923S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1080.7948257448152S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1090.8016387319658S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1100.8086837108713S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1110.8161619089078S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1120.8231734579895S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1130.8303384499159S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1140.837914422853S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1150.845593697857S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1160.8533003099728S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1170.8611394059844S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1180.868547799997S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1190.8759381349664S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1200.8832891788334S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1210.8901379008312S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1220.8969192558434S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1230.9039477759507S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1240.9106991358567S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1250.9173775119707S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1260.92407657695S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1270.9309710729867S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1280.9380627768114S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1290.9453849359415S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1300.952413906809S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1310.9597894018516S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1320.966845863033S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1330.9740760978311S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1340.9815662968904S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1350.9888193528168S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1360.99587745592S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1371.0032199069392S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1381.0101801289711S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1391.0172855509445S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1401.0245092518162S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1411.0318920928985S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1421.0390496668406S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1431.0462171358522S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1441.053218506975S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1451.0601344129536S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1461.067063733004S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1471.0745268219616S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1481.0816990048625S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1491.0894230830017S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1501.0965626609977S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1511.10416178382S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1521.1117203768808S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1531.119007607922S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1541.1263326879125S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1551.133871456841S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1561.1414817289915S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1571.1484652298968S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1581.1557922109496S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1591.163053391967S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1601.1697183998767S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1611.1771028018557S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1621.184726123931S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1631.192537332885S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1641.1996192948427S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1651.2072486619S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1661.214641839033S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1671.2214778079651S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1681.2286652959883S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1691.2357902769S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1701.2424310999922S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1711.2492476548068S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1721.256136143813S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1731.2631078639533S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1741.2700061940122S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1751.2773678100202S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1761.2845035158098S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1771.2912260359153S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1781.2981330240145S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 [Task Queue] Already waiting for 1791.3051665788516S, seq=d4a95da6-ee2f-429f-b9bf-5d5123bdd388 Worker error: ERROR:root: Traceback (most recent call last): File "/src/fooocusapi/worker.py", line 96, in process_generate time.sleep(delay) File "/root/.pyenv/versions/3.11.7/lib/python3.11/site-packages/cog/server/worker.py", line 244, in _signal_handler raise CancelationException() cog.server.exceptions.CancelationException Traceback (most recent call last): File "/src/fooocusapi/worker.py", line 96, in process_generate time.sleep(delay) File "/root/.pyenv/versions/3.11.7/lib/python3.11/site-packages/cog/server/worker.py", line 244, in _signal_handler raise CancelationException() cog.server.exceptions.CancelationException During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/root/.pyenv/versions/3.11.7/lib/python3.11/site-packages/cog/server/worker.py", line 221, in _predict result = predict(**payload) ^^^^^^^^^^^^^^^^^^ File "/src/predict.py", line 192, in predict results = process_generate(queue_task, params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/.pyenv/versions/3.11.7/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/root/.pyenv/versions/3.11.7/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/src/fooocusapi/worker.py", line 862, in process_generate task_queue.finish_task(async_task.job_id) File "/src/fooocusapi/task_queue.py", line 123, in finish_task for item in task.task_result: TypeError: 'NoneType' object is not iterable

[AhmedElhadidii]

I also have this issue, would be very appreciated if someone points why it happens

[gitdrums]

Same here!