Rule based matching and thresholds / confidences ?

[chillwinston]

The concepts of "confidence", "score" and "threshold" don't appear to be covered within rule-based matching on this page:
https://spacy.io/usage/rule-based-matching

Issue:

• Assume that entities to recognise change rapidly (every minute or two - added) and hence a training mechanism isn't reasonable - fast rule-based matching is therefore required
• In some cases the "rule based" matching appears to be taking long multi-word sequences and poorly "predicting" that a phrase is a specific type of entity. Very rough examples are patterns like ["Wine UK", "UK Services", "UK Gardening"]. The words "Wine Gardening" or "Wine Services" are detected as entities when clearly they are not. This behaviour itself is somewhat bizarre.

Is there any mechanism to either:

1. Obtain scores for the ents or spans so that the thresholds can be adjusted to ignore what are likely to be low-scoring returned entities OR
2. Pass a threshold to the pipeline for rule-based entities so that we can have the pipeline itself return only high quality matches

Many thanks

[thomashacker]

Hello,

The concepts you mentioned (confidence, score, and threshold) make more sense when ML components are involved.

Could you give us more information about the patterns you've been using? Usually, in every pattern, each token should be represented by one dictionary.

If you'd like to create a mechanism in which you use confidence scores from entities predicted by the SpanCategorizer component (the NER component doesn't provide any scores), you can create your own custom component.

[chillwinston]

I would agree that confidence / score / threshold make sense when ML components are involved - however, as I mentioned in my original post, we have rapidly changing entities and hence an ML training-based model is not suitable due to the time required to train.

Hence - as I mentioned - we are using the "rule-based" matching approach supported by Spacy , as referenced in the link I provided : https://spacy.io/usage/rule-based-matching

To prevent any further clarification questions, code such as the following but with many more patterns, around 1000, was used:

ruler = nlp.add_pipe("entity_ruler")
patterns = [{"label": "ORG", "pattern": "Apple"},
            {"label": "GPE", "pattern": [{"LOWER": "san"}, {"LOWER": "francisco"}]}]
ruler.add_patterns(patterns)
doc = nlp("Apple is opening its first big office in San Francisco.")
print([(ent.text, ent.label_) for ent in doc.ents])

The code above does not return a confidence or score for the entities that are found (AFAIK). This would be fine if the entity_ruler system did not at times return entities that don't exist in the patterns. Examples were given in my original request: "the words "Wine Gardening" or "Wine Services" are detected as entities when clearly they are not".

In short there is no way to differentiate between the real entities and the "guessed" entities that the system is returning (incorrectly).

Hence my request of "Is there any mechanism to either":

• score or confidence measures (from entity_ruler) OR
• threshold control over entity_ruler recognition - being able to prevent or limit the badly guessed entities

I'm unsure of how entity_ruler is implemented or why it returns unusual results that aren't in the original patterns. I'm somewhat confused as to why it would return values that aren't in the original patterns without also return some measure of a score indicating that these guesses are lower-scoring.

[thomashacker]

Thanks for the clarification. Could you provide us more information about the patterns you're using and whether you're using the rule-based matching on a pretrained language model (e.g. en_core_web_lg)? The task of the EntityRuler (docs) is to simply move the matched entities to doc.ents, so I assume that the used patterns might be the issue. Another possibility could be, if you're using a pipeline with an NER, you might get mixed entity predictions (from the NER and from the Matcher). Neither the Matcher nor the EntityRuler produce any confidence measures, the patterns either match or not.

[chillwinston]

• The patterns are similar to those that I've provided, approximately 1400 of them in our test phase. 
• Some have words that are shared - as I gave as an example - "Services"I'm assuming that the entity ruler doesn't just "move the matched entities" but actually sets up the matching process as well.  It can't move "matched entities" unless these have been matched by "something".  If it's not the entity_ruler that's creating the matches, then please provide references to the functionality that is creating matches.
• Yes we are using en_core_web_lg
• While Matcher and EntityRuler do not provide confidences / scores, they clearly should.  Some internal code is "matching".  That matching code can score against the original patterns.   A 1.0 score would indicate a perfect match against an original pattern.  We are finding "matches" that are not in those original patterns - again, as I provided in my original request, which I will (again) provide here: "Wine Services".

Frankly, it appears that you're re-asking for the same information multiple times and only briefly reading both the original request and any responses I've given before giving a glib answer that fails to address any of the questions. This is moving at a glacial pace with absolutely zero headway due to this behaviour.

• Score / confidence absolutely makes sense with patterns. Please don't tell me it doesn't. This is specifically the case when the EntityRuler subsytem returns "matches" such as "Wine Services" that aren't in the original list of patterns provided.
• The EntityRuler subsystem returning "matches" that don't exist makes it impossible (without further code) to differentiate "real matches" for patterns and "partial matches" that aren't perfect matches to the original pattern list
• You've simply stated that EntityRuler has no confidence / scores. I will accept that, though as I have mentioned above this seems like a significant bug to me. Returning partial matches without any mechanism to differentiate them to perfect matches makes the EntityRuler near-worthless from an accuracy perspective

Our key question now is:

• Why EntityRuler would return "partial matches" that are not perfect matches against the patterns we provide.
• The example (for perhaps the 5th time) is "Wine Services" against our originally provided pattern list.

[svlandeg]

Hi @chillwinston,

We ask all participants on the spaCy discussion forum to adhere to the Contributor Covenant Code of Conduct. Please revise this document before continuing any conversation here.

Please also take a step back and consider your previous posts. spaCy is an open-source library that you can use free of charge. Additionally, the spaCy team offers constant support and advice to the community on this forum. @thomashacker has been engaging with you to understand and resolve your problem. However, comments like

a glib answer that fails to address any of the questions

or sneers like

for perhaps the 5th time

are simply not constructive. They come across as extremely disrepectful to someone who is only trying to help you with your problem.

We want to ensure we spend our time and energy as best as we can to help as many users as we can on the forum. However, it needs to be a respectful and constructive conversation.

[chillwinston]

In my first post I was clear, respectful and reasonable. However, this is something of an urgent issue for us. The original request for insights or information was sent over 7 days ago now.

Thomas's answers, while quick to arrive have:

• Asked for information that was already supplied. Not just once but multiple times, for the same information - "what patterns are you using". This is not a sneer. It is a statement of fact.
• Provided what are technically termed "glib" answers https://en.wiktionary.org/wiki/glib - specifically: "Having a ready flow of words but lacking thought or understanding; superficial; shallow.".
  • In short, his responses were not well-thought out
  • appeared as if he had not properly read the questions or details of the post
  • did not address the questions within the request

I would suggest that ignoring questions; providing simplistic statements and asking multiple times for the same information is not constructive and frankly very frustrating to anyone, and especially to people with urgent requirements. Everyone's time is valuable.

We will however aim to be more constructive in our posts in future, though we would also appreciate both clarity and responses that answer the questions that are posed in exchange.

Our current specific question is:

• Why EntityRuler would return "partial matches" that are not perfect matches against the patterns we provide.

For reference, the pattern examples are: ["Wine UK", "UK Services", "UK Gardening"]. The words "Wine Gardening" or "Wine Services" are detected as entities when clearly they are not.

[adrianeboyd]

In case anyone comes across this in the future, as @thomashacker explained above, two reasons you might see entities you weren't expecting after adding an entity ruler component are:

• there are problems with the entity ruler patterns
• the entity ruler has been added to a pipeline like en_core_web_sm that contains other components that also add entities (typically ner), so the entities in doc.ents don't all come from the new entity ruler component

The entity ruler patterns only support full 100% matches. There are no partial matches or confidence scores from the entity ruler or the underlying rule-based matcher.

If you are running into a similar problem, please open a new discussion thread. It's easiest for us to help if you can provide the details from spacy info and a minimal working example that shows how you are creating and running your pipeline.