Entity Linker Loss spacy train not changing

[fabilj825]

Hi all,
I'm training a custom NEL using the spacy train CLI.
During Training, no values are changing. I expected to get a not working pipeline, but that is not the case. The pipeline works fine afterwards.

Used config scores:

[training.score_weights]
tag_acc = null
lemma_acc = null
dep_uas = null
dep_las = null
dep_las_per_type = null
sents_p = null
sents_r = null
sents_f = null
ents_f = null
ents_p = null
ents_r = null
ents_per_type = null
nel_micro_f = 1.0
nel_micro_r = null
nel_micro_p = null

Because the scores are not changing, logging to wandb is useless..

Has anyone encountered the same issue?

Thanks for your help!

Best regards,
Fabian

[svlandeg]

Hm, odd. How many EL instances do you have in the training & dev sets?

And can you share the full config file?

[fabilj825]

First of all thanks for the effort you put in the project and the support. That is awesome. The size of train.spacy is 2300 and test.spacy 775.
Here is the full config:

[paths]
init_tok2vec = null
parser_tagger_path = null
vocab_path = null
train = "./train.spacy"
dev = "./test.spacy"
base_nlp = "./sciLarge"
vectors = "${paths.base_nlp}"
kb = "./knowledgebase/newkb"

[system]
gpu_allocator = pytorch
seed = 0

[nlp]
lang = "en"
pipeline = ["tok2vec","tagger","attribute_ruler","lemmatizer","parser","ner","sentencizer","entity_linker"]
tokenizer = {"@Tokenizers":"spacy.Tokenizer.v1"}
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
batch_size = 1000

[components]

[components.attribute_ruler]
source = "${paths.base_nlp}"
component = "attribute_ruler"

[components.entity_linker]
factory = "entity_linker"
candidates_batch_size = 1
entity_vector_length = 200
generate_empty_kb = {"@misc":"spacy.EmptyKB.v2"}
get_candidates = {"@misc":"spacy.CandidateGenerator.v1"}
get_candidates_batch = {"@misc":"spacy.CandidateBatchGenerator.v1"}
incl_context = true
incl_prior = true
labels_discard = []
n_sents = 0
overwrite = true
scorer = {"@scorers":"spacy.entity_linker_scorer.v1"}
threshold = null
use_gold_ents = true

[components.entity_linker.model]
@architectures = "spacy.EntityLinker.v2"
nO = null

[components.entity_linker.model.tok2vec]
@architectures = "spacy.HashEmbedCNN.v2"
pretrained_vectors = null
width = 96
depth = 4
embed_size = 2000
window_size = 1
maxout_pieces = 3
subword_features = true

[components.lemmatizer]
source = "${paths.base_nlp}"
component = "lemmatizer"

[components.ner]
source = "${paths.base_nlp}"
component = "ner"

[components.parser]
source = "${paths.base_nlp}"
component = "parser"

[components.sentencizer]
factory = "sentencizer"
overwrite = false
punct_chars = null
scorer = {"@scorers":"spacy.senter_scorer.v1"}

[components.tagger]
source = "${paths.base_nlp}"
component = "tagger"

[components.tok2vec]
source = "${paths.base_nlp}"
component = "tok2vec"

[corpora]

[corpora.dev]
@readers = "MyCorpus.v1"
file = ${paths.dev}
[corpora.train]
@readers = "MyCorpus.v1"
file = ${paths.train}

[training]
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.5
accumulate_gradient = 1
patience = 0
max_epochs = 7
max_steps = 0
eval_frequency = 500
frozen_components = ["tok2vec","parser","tagger","attribute_ruler","lemmatizer","ner","sentencizer"]
before_to_disk = null
annotating_components = []
before_update = null

[training.batcher]
@batchers = "spacy.batch_by_sequence.v1"
get_length = null

[training.batcher.size]
@schedules = "compounding.v1"
start = 1
stop = 32
compound = 1.001
t = 0.0

[training.logger]
@Loggers = "spacy.ConsoleLogger.v2"
progress_bar = true

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001
learn_rate = 0.001

[training.score_weights]
tag_acc = null
lemma_acc = null
dep_uas = null
dep_las = null
dep_las_per_type = null
sents_p = null
sents_r = null
sents_f = null
ents_f = null
ents_p = null
ents_r = null
ents_per_type = null
nel_micro_f = 0.5
nel_micro_r = null
nel_micro_p = null
loss_entity_linker=0.5

[pretraining]

[initialize]
vectors = ${paths.vectors}
init_tok2vec = null
vocab_data = null
lookups = null
before_init = null
after_init = null

[initialize.components]

[initialize.components.entity_linker]

[initialize.components.entity_linker.kb_loader]
@misc = "spacy.KBFromFile.v1"
kb_path = ${paths.kb}

[initialize.tokenizer]

[vars]
include_static_vectors = "True"

[svlandeg]

I looked at your config file in detail - but nothing weird jumps out to me (maybe perhaps only the fact that 0.5 is a really high dropout value)

So you mention

The pipeline works fine afterwards.

What does this mean exactly - is the Entity Linker in fact able to disambiguate entities to their correct identifier, even when there's ambiguity?

There might be a few things explaining this behaviour, but I think you'll have to dive into your data to better understand what's going on. For instance, if your KB only contains non-ambiguous entities, then the "prior probability" might be all that the entity linker needs. The training part really only serves to train the context-dependent part of the entity linker - i.e. given two plausible IDs for the same Span, which is more likely to be correct given the context? In your case, this part doesn't seem to be training.

There's a few things I'd advice trying to pin down what's going on:

• Train only an NER model on this data - if there's anything wrong with the entity annotation, this should exhibit similar weird behaviour.
• Double check that your KB is solid and contains candidates for the entities in your dataset - preferably multiple candidates for which 1 is correct in the gold data
• Run spacy debug data on your datasets to check for anything odd

If the data & KB looks fine, a few other experiments you could try:

• Try training AND evaluating on the same data (both "train", or both "dev") to see whether the EL manages to overfit (or learn anything at all)
• Remove the NER from the frozen list and train it as well, together with the EL. I know you ultimately want to use a pretrained NER model, but this is just to check whether there's some interaction between the NER being frozen and the EL being trained.

[fabilj825]

Thank you for all the ideas. I will dive deeper and try some of the the possible solutions.

To specify "works fine": We work with UMLS data and are able to link some entities to correct Cuis after training. The performance is still bad, but i did not expect it to work after the training output.
If I run the same setup using Jupyter, a loss is returned. The model performance is the same. Can I use the evaluation data within NLP.update()?

I ran the debug on my data, can you spot something odd?

Best regards and thanks for your help,
Fabian

============================ Data file validation ============================
/home/fabi/dllab/env/lib/python3.10/site-packages/spacy/pipeline/attributeruler.py:149: UserWarning: [W036] The component 'matcher' does not have any patterns defined.
matches = self.matcher(doc, allow_missing=True, as_spans=False)
/home/fabi/dllab/env/lib/python3.10/site-packages/spacy/pipeline/lemmatizer.py:211: UserWarning: [W108] The rule-based lemmatizer did not find POS annotation for one or more tokens. Check that your pipeline includes components that assign token.pos, typically 'tagger'+'attribute_ruler' or 'morphologizer'.
warnings.warn(Warnings.W108)
✔ Pipeline can be initialized with data
✔ Corpus is loadable

=============================== Training stats ===============================
Language: en
Training pipeline: tok2vec, tagger, attribute_ruler, lemmatizer, parser, ner,
sentencizer, entity_linker
Frozen components: tok2vec, parser, tagger, attribute_ruler, lemmatizer, ner,
sentencizer
2302 training docs
775 evaluation docs
✔ No overlap between training and evaluation data

============================== Vocab & Vectors ==============================
ℹ 601486 total word(s) in the data (43263 unique)
⚠ 80 misaligned tokens in the training data
⚠ 27 misaligned tokens in the dev data
ℹ 600000 vectors (4087446 unique keys, 200 dimensions)
⚠ 6248 words in training data without vectors (1%)

========================== Named Entity Recognition ==========================
ℹ 2 label(s)
0 missing value(s) (tokens with '-' label)
⚠ Some model labels are not present in the train data. The model
performance may be degraded for these labels after training: 'ENTITY'.
✔ Good amount of examples for all labels
✔ Examples without occurrences available for all labels
✔ No entities consisting of or starting/ending with whitespace
✔ No entities crossing sentence boundaries

=========================== Part-of-speech Tagging ===========================
ℹ 1 label(s) in train data
/home/fabi/dllab/env/lib/python3.10/site-packages/spacy/cli/debug_data.py:547: RuntimeWarning: invalid value encountered in scalar divide
norm_entropy = (-p * numpy.log2(p)).sum() / numpy.log2(len(label_list))
ℹ nan is the normalised label entropy
⚠ Some model labels are not present in the train data. The model
performance may be degraded for these labels after training: 'ADD', ',',
'-RRB-', 'PDT', 'POS', 'RBS', 'WRB', 'NNPS', 'RBR', 'PRP', 'JJ', 'MD', 'VBG',
'NN', 'RB', '''', 'FW', 'WP', 'IN', 'HYPH', 'EX', 'DT', 'UH', 'WP$', ':', 'RP',
'XX', '-LRB-', 'AFX', '.', 'NNP', 'LS', 'NNS', 'SYM', 'JJS', '``', '$', 'VBZ',
'WDT', 'VBN', 'TO', 'VBP', 'JJR', 'PRP$', 'VB', 'CD', 'CC', 'VBD', 'NFP'.

============================= Dependency Parsing =============================
ℹ Found 590332 sentence(s) with an average length of 1.0 words.
ℹ 1 label(s) in train data
ℹ 1 label(s) in projectivized train data

================================== Summary ==================================
✔ 7 checks passed
⚠ 5 warnings