Spacy NER during Inference

[baivabdash]

Suppose I have trained 4 different NER models, each model trained on a different set of entities. For my inference, I need to use all four models together to make predictions for different entities. Running all the models sequentially takes too long and I have a constraint of not being able to use GPU during inference. I have tried parallelizing using python's multiprocessing but it is not working. Any way I can parallelize running the models during inference?
This question seems better suited for S.O but any help would be great...

Thanks

[ljvmiranda921]

Hi @baivadash! First off, does the order matter in your models, or are there any inter-model dependencies in your inference pipeline?

• If yes, then they have to be run sequentially to respect those dependencies.
• If no, and they're completely independent, you can try running them in separate processes. Check-out libraries like joblib. It wraps multiprocessing, and allows you to choose other backends.

Later on, you can combine your annotations using a SpanGroup (maybe using SpanGroup.append or .extend) or take advantage of span categorization using SpanCategorizer. Here's a good intro on how span categorization relates to NER.

[baivabdash]

Thanks @ljvmiranda921 for your prompt reply. Since I was trying on Colab multiprocessing was not working. Using Joblib on my server I was able to parallel process separate models..