Inference for Detectron2 PointRend_X101-FPN_model

[kaphleamrit2]

I have trained the PointRend_X101-FPN model on Detectron2. But I am getting the following error during inference. I wonder whether the PointRend model is supported or not? Does anybody have any idea what's going on here?

Thank you!

`/content/drive/MyDrive/Detectron2/PointRend_X101-FPN_model/object_detection/config_original.yaml not available in Model Zoo!

RuntimeError Traceback (most recent call last)
/usr/local/lib/python3.7/dist-packages/sahi/model.py in load_model(self)
521 try: # try to load from model zoo
--> 522 config_file = model_zoo.get_config_file(self.config_path)
523 cfg.merge_from_file(config_file)

8 frames
/usr/local/lib/python3.7/dist-packages/detectron2/model_zoo/model_zoo.py in get_config_file(config_path)
142 if not os.path.exists(cfg_file):
--> 143 raise RuntimeError("{} not available in Model Zoo!".format(config_path))
144 return cfg_file

RuntimeError: /content/drive/MyDrive/Detectron2/PointRend_X101-FPN_model/object_detection/config_original.yaml not available in Model Zoo!

During handling of the above exception, another exception occurred:

KeyError Traceback (most recent call last)
in ()
4 confidence_threshold=0.8,
5 image_size=512,
----> 6 device="cuda", # or 'cuda:0'
7 )

/usr/local/lib/python3.7/dist-packages/sahi/model.py in init(self, model_path, config_path, device, mask_threshold, confidence_threshold, category_mapping, category_remapping, load_at_init, image_size)
69 # automatically load model if load_at_init is True
70 if load_at_init:
---> 71 self.load_model()
72
73 def load_model(self):

/usr/local/lib/python3.7/dist-packages/sahi/model.py in load_model(self)
526 print(e)
527 if self.config_path is not None:
--> 528 cfg.merge_from_file(self.config_path)
529 cfg.MODEL.WEIGHTS = self.model_path
530 # set input image size

/usr/local/lib/python3.7/dist-packages/detectron2/config/config.py in merge_from_file(self, cfg_filename, allow_unsafe)
67
68 if loaded_ver == self.VERSION:
---> 69 self.merge_from_other_cfg(loaded_cfg)
70 else:
71 # compat.py needs to import CfgNode

/usr/local/lib/python3.7/dist-packages/fvcore/common/config.py in merge_from_other_cfg(self, cfg_other)
130 BASE_KEY not in cfg_other
131 ), "The reserved key '{}' can only be used in files!".format(BASE_KEY)
--> 132 return super().merge_from_other_cfg(cfg_other)
133
134 def merge_from_list(self, cfg_list: List[str]) -> Callable[[], None]:

/usr/local/lib/python3.7/dist-packages/yacs/config.py in merge_from_other_cfg(self, cfg_other)
215 def merge_from_other_cfg(self, cfg_other):
216 """Merge cfg_other into this CfgNode."""
--> 217 _merge_a_into_b(cfg_other, self, self, [])
218
219 def merge_from_list(self, cfg_list):

/usr/local/lib/python3.7/dist-packages/yacs/config.py in _merge_a_into_b(a, b, root, key_list)
476 if isinstance(v, CfgNode):
477 try:
--> 478 _merge_a_into_b(v, b[k], root, key_list + [k])
479 except BaseException:
480 raise

/usr/local/lib/python3.7/dist-packages/yacs/config.py in _merge_a_into_b(a, b, root, key_list)
489 root.raise_key_rename_error(full_key)
490 else:
--> 491 raise KeyError("Non-existent config key: {}".format(full_key))
492
493

KeyError: 'Non-existent config key: INPUT.COLOR_AUG_SSD'`

[hyfaf]

you can try to get config file directly. don't get from model_zoo

[kadirnar]

Hi @kaphleamrit2

SAHI supports object detection and instance segmentation models.

Detectron2 Object Detection: https://github.com/facebookresearch/detectron2/blob/main/MODEL_ZOO.md#coco-object-detection-baselines

Detectron2 Instance Segmentation: https://github.com/facebookresearch/detectron2/blob/main/MODEL_ZOO.md#coco-instance-segmentation-baselines-with-mask-r-cnn

[kadirnar]

To test the custom detectron2 model you should follow the steps here.

• https://github.com/kadirnar/Detectron2-SAHI/blob/main/test_sahi_d2.py

[fcakyon]

@kaphleamrit2 you have to export your detectron2 config as yaml, then use the config path during Detectron2DetectionModel initilialization.

Details on how to export Detectron2 config as yaml can be found in this blog post: https://medium.com/geekculture/perform-sliced-inference-and-detailed-error-analysis-using-detectron2-models-8d9c551953f3

[kaphleamrit2]

@fcakyon @kadirnar Thanks for suggestion.
I followed above mentioned procedure but still got same error. I tried other model (Faster_RCCN_R50, Mask RCNN X 101) that works without any error. But PointRend_X101-FPN model gives this error.
I saved my trained model as :
model_path = "/content/drive/MyDrive/Detectron2/PointRend_X101-FPN_model/object_detection/model_final.pth"

Here is my custom cfg:

config_file_path = 
"/content/detectron2_repo/projects/PointRend/configs/InstanceSegmentation/pointrend_rcnn_X_101_32x8d_FPN_3x_coco.yaml" 
checkpoint_url = "detectron2://PointRend/InstanceSegmentation/pointrend_rcnn_X_101_32x8d_FPN_3x_coco/28119989/model_final_ba17b9.pkl"

output_dir = "./output/object_detection"
num_classes = 1
device ="cuda" #'cpu'

cfg = get_cfg()

point_rend.add_pointrend_config(cfg)   # Add PointRend-specific config
cfg.INPUT.MASK_FORMAT= 'bitmask' #only for point rend model
cfg.merge_from_file(config_file_path)  #for point rend 
cfg.MODEL.WEIGHTS = checkpoint_url  # Let training initialize from Point rend
cfg.DATASETS.TRAIN = (train_dataset_name,)
cfg.DATASETS.TEST = (test_dataset_name,)

cfg.DATALOADER.NUM_WORKERS = 2

cfg.INPUT.MIN_SIZE_TRAIN = (400, 500, 512, 600, 700, 800)  
cfg.INPUT.MAX_SIZE_TRAIN = 1024 
cfg.INPUT.MIN_SIZE_TEST = (400, 500, 512, 600, 700, 800) 
cfg.INPUT.MAX_SIZE_TEST = 1024 

cfg.SOLVER.IMS_PER_BATCH = 2
cfg.SOLVER.BASE_LR = 0.0025 
cfg.SOLVER.MAX_ITER = 3000  
cfg.SOLVER.STEPS = []        # do not decay learning rate
cfg.SOLVER.CHECKPOINT_PERIOD = 1000
  
cfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 128   
cfg.MODEL.ROI_HEADS.NUM_CLASSES = num_classes
cfg.MODEL.POINT_HEAD.NUM_CLASSES = num_classes #for point rend model
cfg.MODEL.DEVICE = device
cfg.OUTPUT_DIR = output_dir

cfg.SEED = 123`

export my model cfg file as:

export_cfg_as_yaml(cfg, export_path='config.yaml')

config_path = "/content/config.yaml"

Then define detection model:
detection_model = Detectron2DetectionModel( model_path=model_path, config_path=config_path, confidence_threshold=0.8, image_size=512, device="cuda", # or 'cuda:0' )
But got the same error:

`/content/config.yaml not available in Model Zoo!

RuntimeError Traceback (most recent call last)
/usr/local/lib/python3.7/dist-packages/sahi/model.py in load_model(self)
521 try: # try to load from model zoo
--> 522 config_file = model_zoo.get_config_file(self.config_path)
523 cfg.merge_from_file(config_file)

8 frames
RuntimeError: /content/config.yaml not available in Model Zoo!

During handling of the above exception, another exception occurred:

KeyError Traceback (most recent call last)
/usr/local/lib/python3.7/dist-packages/yacs/config.py in _merge_a_into_b(a, b, root, key_list)
489 root.raise_key_rename_error(full_key)
490 else:
--> 491 raise KeyError("Non-existent config key: {}".format(full_key))
492
493

KeyError: 'Non-existent config key: INPUT.COLOR_AUG_SSD'`

I don't know what is happening. Other model works fine but I can't figure in this model where is the problem.

[kadirnar]

Hi @fcakyon

This model does not work in the SAHI algorithm. It may be because of the point format.
Code:

# import required functions, classes
from detectron2.config import get_cfg
from sahi.model import Detectron2DetectionModel
from sahi.predict import get_sliced_prediction

# create config file

model_path = "mask_rcnn.pkl"
cfg = get_cfg()
cfg.MODEL.ROI_HEADS.NUM_CLASSES = 445
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.2

from sahi.utils.detectron2 import export_cfg_as_yaml
config_path = export_cfg_as_yaml(cfg, export_path='config.yaml')

# test model
detection_model = Detectron2DetectionModel(
    model_path=model_path,
    config_path=config_path,
    confidence_threshold=0.2,
    image_size=640,
    device="cuda:0",  # or 'cuda:0'
)

result = get_sliced_prediction(
    "demo_data/small-vehicles1.jpeg",  # image _path
    detection_model,
    slice_height=320,
    slice_width=320,
    overlap_height_ratio=0.4,
    overlap_width_ratio=0.4,
)
# Sahi+ Detectron2
result.export_visuals(export_dir="demo_data/")

Error Message:

sahi/postprocess/utils.py:145: RuntimeWarning: 
invalid value encountered in long_scalars return intersect / smaller_area

and

WARNING - fvcore.common.checkpoint -   Some model parameters or buffers are not found in the checkpoint:

backbone.res2.0.conv1.norm.{bias, weight}
backbone.res2.0.conv1.weight
backbone.res2.0.conv2.norm.{bias, weight}
...

WARNING - fvcore.common.checkpoint -   The checkpoint state_dict contains keys that are not used by the model:
backbone.fpn_lateral2.{bias, weight}
backbone.fpn_output2.{bias, weight}
backbone.fpn_lateral3.{bias, weight}
...

Output (demo_data/prediction_visual.png):

[fcakyon]

@kaphleamrit2 @kadirnar what is different in pointrend?

[kadirnar]

Hi @kaphleamrit2,

Paste this code on line 551.

from detectron2.projects import point_rend
point_rend.add_pointrend_config(cfg)

Then you need to download these files and edit the file path.

• Base-RCNN-FPN.yaml
• Base-PointRend-RCNN-FPN

Detect Code:

from sahi.model import Detectron2DetectionModel
from sahi.predict import get_sliced_prediction, predict, get_prediction
from sahi.utils.file import download_from_url
from sahi.utils.cv import read_image

image = download_from_url('https://raw.githubusercontent.com/obss/sahi/main/demo/demo_data/small-vehicles1.jpeg',
                  'demo_data/small-vehicles1.jpeg')

detection_model = Detectron2DetectionModel(
    model_path="model_final_edd263.pkl",
    config_path="pointrend_rcnn_R_50_FPN_3x_coco.yaml",
    confidence_threshold=0.5,
    image_size=640,
    device="cpu",  # or 'cuda:0'
)

result = get_sliced_prediction(
    "demo_data/small-vehicles1.jpeg",
    detection_model,
    slice_height = 256,
    slice_width = 256,
    overlap_height_ratio = 0.2,
    overlap_width_ratio = 0.2,
)
result.export_visuals(export_dir="demo_data/")

SAHI + PointRend: