Prediction time is increased exponentially when I predict 8 k * 8 k image

[aymanaboghonim]

I used similar code to this code snippet
result = get_sliced_prediction( "large_image/330112-330144-224882-224914-19.jpeg", detection_model, slice_height = 512, slice_width = 512, overlap_height_ratio = 0.1, overlap_width_ratio = 0.1, postprocess_type = 'NMS' )
when I run prediction on the 8 k * 8 k image, it takes about 26 minutes but it takes only 2.3 minutes when I used a 4 k * 4 k part of the large image (8 k * 8 K ) .

100 slices = 2.25 minutes
400 slices = 26 minutes

does this suggest that the time is increasing exponential ( non linear ) with increasing the image size ??
if it the answer is yes, is there any remedy for this ???

[fcakyon]

It should scale linearly, are you able to provide any sample model or image so that we can recreate the issue?

[aymanaboghonim]

this is the link for 8 k * 8 k Image
https://drive.google.com/file/d/18LRjbLV3ohSKnKd15G3AqjzoyQINpidp/view?usp=sharing
and this for 4 k * 4 k ( one fourth of the large image)
https://drive.google.com/file/d/1c4oCNJ6NRihXJsN0FszxkhEnOGA2me-T/view?usp=sharing

[aymanaboghonim]

I can not provide the model that we used for prediction , so you can test images using any sample model you provide in demo.

[fcakyon]

@aymanaboghonim what is your model confidence threshold?

[aymanaboghonim]

0.3

[devrimcavusoglu]

@aymanaboghonim this was not an actual test (I just called the function once), but overall it should give insight, this is from colab and with the resources you provided (4k by 4k and 8k by 8k)

img0_0.jpeg is 4k x 4k
img0_1.jpeg is 8k x 8k

it's clear that the runtime is increased almost linearly. My thoughts are, have you had any other cpu intensive tasks running in parallel, so cpu is shared a long time between other tasks ? Perhaps your resource usage was not the same when you were running this prediction tasks ? Can you confirm that anyhow ?

[aymanaboghonim]

@devrimcavusoglu

have you had any other cpu intensive tasks running in parallel, so cpu is shared a long time between other tasks ?

No

Perhaps your resource usage was not the same when you were running this prediction tasks ?

Same

Could do you mention which detection model you used?? We used Mask RCNN for instance segmentation purpose.

[fcakyon]

Could do you mention which detection model you used?? We used Mask RCNN for instance segmentation purpose.

We recently merged a PR that reduces instance segmentation model memory by %80 when pycocotools is installed. Have you updated your SAHI to latest?

[devrimcavusoglu]

@devrimcavusoglu

have you had any other cpu intensive tasks running in parallel, so cpu is shared a long time between other tasks ?

No

Perhaps your resource usage was not the same when you were running this prediction tasks ?

Same

Could do you mention which detection model you used?? We used Mask RCNN for instance segmentation purpose.

since the model and task is not mentioned, I tried on yolov5 & detection, so the results I provided is subject to change for segmentation.

[aymanaboghonim]

Could do you mention which detection model you used?? We used Mask RCNN for instance segmentation purpose.

We recently merged a PR that reduces instance segmentation model memory by %80 when pycocotools is installed. Have you updated your SAHI to latest?

I have not update it yet but what do you mean by reducing model memory ??

[aymanaboghonim]

@devrimcavusoglu

have you had any other cpu intensive tasks running in parallel, so cpu is shared a long time between other tasks ?

No

Perhaps your resource usage was not the same when you were running this prediction tasks ?

Same
Could do you mention which detection model you used?? We used Mask RCNN for instance segmentation purpose.

since the model and task is not mentioned, I tried on yolov5 & detection, so the results I provided is subject to change for segmentation.

yes, so I mentioned mask r cnn and asked you to mention the model you used.
I think it might be something related to mask r cnn but you need about 750 GB Ram to run this task without kernel crashing .

[fcakyon]

I have not update it yet but what do you mean by reducing model memory ??

Mask predictions from instance segmentation models (as mask-rcnn) take %80 less memory when pycocotools is installed in the env.

This may increase prediction speed with instance segmentation models.

[fcakyon]

I think it might be something related to mask r cnn but you need about 750 GB Ram to run this task without kernel crashing.

It should require much less ram with the latest sahi version.

[aymanaboghonim]

I have not update it yet but what do you mean by reducing model memory ??

Mask predictions from instance segmentation models (as mask-rcnn) take %80 less memory when pycocotools is installed in the env.

This may increase prediction speed with instance segmentation models.

Great! I hope this really increase the speed of prediction and decrease the required cost.

[aymanaboghonim]

We have been updated the sahi to the latest version 0.10.07 and no Ram Consumption is improved .
we have repeat the same experiment using one image of about 2 k * 2 k size and other one of 4 k * 4 k , and we notice that the time factor is significantly non linear.

`2048x2048 - 25 slices - 41.24 seconds

4096x4096 - 100 slices - 495.07 seconds

4096 taking 12x the time with only 4x the number of slices`

keep note that we used a custom detectron2 (mask r cnn ) model and a custom config file .
please, give us a reasonable explanation for that.

[fcakyon]

Is pycocotools installed in your env? Then this PR (included in v0.10.7) should decrease the used vram: #599

Non-linear time might be a result of the post-process. Have you tried to use non-zero merge_buffer_length parameter while doing sliced inference?

sahi/sahi/predict.py

Line 136 in 0e60975

merge_buffer_length: int = None,

[aymanaboghonim]

yes, pycocotools is installed .

Non-linear time might be a result of the post-process. Have you tried to use non-zero merge_buffer_length parameter while doing sliced inference?

No , I have not . which suitable value to try with ??

[aymanaboghonim]

we tried with merge_buffer_length = 100 as I found in a previous issue/discussion that 100 is an optimal value but the same results related to inference time are repeated.
Any suggestion else?