Training Transformer CLI running but score stays 0.0

[Larsdegroot]

I'm trying to train a transformer with a ner. However when training this is the output:

=============================== train_ner_gpu ===============================
Running command: /home/ldegroot/project_nlp/bin/python3 -m spacy train configs/ner_trf.cfg --output training --paths.train data/train_ner.spacy --paths.dev data/dev_ner.spacy --gpu-id 0
ℹ Saving to output directory: training
ℹ Using GPU: 0

=========================== Initializing pipeline ===========================
[2022-09-19 13:01:54,923] [INFO] Set up nlp object from config
[2022-09-19 13:01:54,947] [INFO] Pipeline: ['transformer', 'ner']
[2022-09-19 13:01:54,950] [INFO] Created vocabulary
[2022-09-19 13:01:54,951] [INFO] Finished initializing nlp object
[2022-09-19 13:02:05,583] [INFO] Initialized pipeline components: ['transformer', 'ner']
✔ Initialized pipeline

============================= Training pipeline =============================
ℹ Pipeline: ['transformer', 'ner']
ℹ Initial learn rate: 0.001
E    #       LOSS TRANS...  LOSS NER  ENTS_F  ENTS_P  ENTS_R  SCORE 
---  ------  -------------  --------  ------  ------  ------  ------
  0       0         179.58    273.93    0.00    0.00    0.00    0.00
  2     200      124605.61  17320.93    0.00    0.00    0.00    0.00
  4     400       29500.84  15215.25    0.00    0.00    0.00    0.00
  6     600       22238.34  14782.03    0.00    0.00    0.00    0.00
  8     800       42730.86  15798.61    0.00    0.00    0.00    0.00
 11    1000        2314.90  14683.93    0.00    0.00    0.00    0.00
 13    1200       58551.32  15691.75    0.00    0.00    0.00    0.00
 15    1400       20414.16  14833.63    0.00    0.00    0.00    0.00
 17    1600       25194.99  15041.20    0.00    0.00    0.00    0.00
✔ Saved pipeline to output directory
training/model-last

Thinking the error might just have something to do with the dev-set i tried loading the model.
When loading model-last and testing it it's not labeling anything:

nlp = spacy.load(model_path)
doc = nlp("Quinolinic acid (QA) has been used as a model for experimental overstimulation of the glutamatergic system.)
doc.ents

out: ()

How to reproduce the behaviour

Config:

[paths]
train = null
dev = null
vectors = null
init_tok2vec = null

[system]
gpu_allocator = null
seed = 0

[nlp]
lang = "en"
pipeline = ["transformer","ner"]
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}
batch_size = 128

[components]

[components.ner]
factory = "ner"
incorrect_spans_key = null
moves = null
scorer = {"@scorers":"spacy.ner_scorer.v1"}
update_with_oracle_cut_size = 100

[components.ner.model]
@architectures = "spacy.TransitionBasedParser.v2"
state_type = "ner"
extra_state_tokens = false
hidden_width = 64
maxout_pieces = 2
use_upper = false
nO = null

[components.ner.model.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0
pooling = {"@layers":"reduce_mean.v1"}
upstream = "transformer"

[components.transformer]
factory = "transformer"
max_batch_items = 4096
set_extra_annotations = {"@annotation_setters":"spacy-transformers.null_annotation_setter.v1"}

[components.transformer.model]
@architectures = "spacy-transformers.TransformerModel.v3"
name = "cambridgeltl/SapBERT-from-PubMedBERT-fulltext"

[components.transformer.model.get_spans]
@span_getters = "spacy-transformers.strided_spans.v1"
window = 64
stride = 48

[components.transformer.model.tokenizer_config]
use_fast = true

[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[training]
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
accumulate_gradient = 1
patience = 1600
max_epochs = 0
max_steps = 20000
eval_frequency = 200
frozen_components = []
annotating_components = []
before_to_disk = null

[training.batcher]
@batchers = "spacy.batch_by_words.v1"
discard_oversize = false
tolerance = 0.2
get_length = null

[training.batcher.size]
@schedules = "compounding.v1"
start = 100
stop = 1000
compound = 1.001
t = 0.0

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = false

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001
learn_rate = 0.001

[training.score_weights]
ents_f = 1.0
ents_p = 0.0
ents_r = 0.0
ents_per_type = null

[pretraining]

[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null
before_init = null
after_init = null

[initialize.components]

[initialize.tokenizer]

Environment

• spacy-transformers version* 1.1.8
• spaCy version: 3.4.1
• Platform: Linux-5.4.0-124-generic-x86_64-with-glibc2.29
• Python version: 3.8.10

[Larsdegroot]

When training the same pipeline with the same data but with a tok2vec model instead of a transformer the same issue doesn't occur.

[polm]

Does this work if you change your Transformer model (SapBERT) to roberta-base? It usually doesn't result in flat zeroes, but there are sometimes issues like this with Transformer models we haven't tested before.

[Larsdegroot]

Sadly still the same problem. However a new warning message popped up:

=============================== train_ner_gpu ===============================
Running command: /home/ldegroot/project_nlp/bin/python3 -m spacy train configs/ner_trf.cfg --output training --paths.train data/train_ner.spacy --paths.dev data/dev_ner.spacy --gpu-id 0
ℹ Saving to output directory: training
ℹ Using GPU: 0

=========================== Initializing pipeline ===========================
[2022-09-21 09:34:56,885] [INFO] Set up nlp object from config
[2022-09-21 09:34:56,895] [INFO] Pipeline: ['transformer', 'ner']
[2022-09-21 09:34:56,898] [INFO] Created vocabulary
[2022-09-21 09:34:56,899] [INFO] Finished initializing nlp object
Some weights of the model checkpoint at roberta-base were not used when initializing RobertaModel: ['lm_head.dense.bias', 'lm_head.bias', 'lm_head.decoder.weight', 'lm_head.layer_norm.bias', 'lm_head.layer_norm.weight', 'lm_head.dense.weight']
- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
[2022-09-21 09:35:06,074] [INFO] Initialized pipeline components: ['transformer', 'ner']
✔ Initialized pipeline

============================= Training pipeline =============================
ℹ Pipeline: ['transformer', 'ner']
ℹ Initial learn rate: 0.001
E    #       LOSS TRANS...  LOSS NER  ENTS_F  ENTS_P  ENTS_R  SCORE 
---  ------  -------------  --------  ------  ------  ------  ------
  0       0         219.97    219.81    0.00    0.00    0.00    0.00
  2     200       29213.93  16742.45    0.00    0.00    0.00    0.00
  4     400       18013.24  15572.21    0.00    0.00    0.00    0.00
  6     600       15180.66  14673.33    0.00    0.00    0.00    0.00
  8     800       16962.42  15324.37    0.00    0.00    0.00    0.00
 11    1000        1148.34  14624.22    0.00    0.00    0.00    0.00
 13    1200       26981.11  15221.39    0.00    0.00    0.00    0.00
 15    1400        9659.69  14630.93    0.00    0.00    0.00    0.00
 17    1600        5398.76  14766.80    0.00    0.00    0.00    0.00
✔ Saved pipeline to output directory
training/model-last

[polm]

Sorry that didn't fix it, and for the delayed reply. Unfortunately that warning message is standard when using pretrained transformer weights. I'm still not entirely sure what's going on, but just to be totally sure that there's no data issue, could you use a completely default Transformer config as generated from scratch by spacy init config or the quickstart? It looks like your learn rate, for example, is from the CPU defaults rather than the GPU defaults, which could cause issues.

If that doesn't work, could you share the output of spacy debug data on your training data? I'm not sure what could cause an issue only with Transformers, but it would help to know a little more about the data.

[Larsdegroot]

Hi thanks for replying, the support you offer is amazing!
it was definitely a config problem, i created a new config from scratch as you instructed and filled it with
python -m spacy init fill-config base_config.cfg config.cfg. I first tried training with the roberta-base transformer and then with the cambridgeltl/SapBERT-from-PubMedBERT-fulltext, both working as expected!

New config:

[paths]
train = null
dev = null
vectors = null
init_tok2vec = null

[system]
gpu_allocator = "pytorch"
seed = 0

[nlp]
lang = "en"
pipeline = ["transformer","ner"]
batch_size = 128
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}

[components]

[components.ner]
factory = "ner"
incorrect_spans_key = null
moves = null
scorer = {"@scorers":"spacy.ner_scorer.v1"}
update_with_oracle_cut_size = 100

[components.ner.model]
@architectures = "spacy.TransitionBasedParser.v2"
state_type = "ner"
extra_state_tokens = false
hidden_width = 64
maxout_pieces = 2
use_upper = false
nO = null

[components.ner.model.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0
pooling = {"@layers":"reduce_mean.v1"}
upstream = "*"

[components.transformer]
factory = "transformer"
max_batch_items = 4096
set_extra_annotations = {"@annotation_setters":"spacy-transformers.null_annotation_setter.v1"}

[components.transformer.model]
@architectures = "spacy-transformers.TransformerModel.v3"
name = "cambridgeltl/SapBERT-from-PubMedBERT-fulltext"
mixed_precision = false

[components.transformer.model.get_spans]
@span_getters = "spacy-transformers.strided_spans.v1"
window = 128
stride = 96

[components.transformer.model.grad_scaler_config]

[components.transformer.model.tokenizer_config]
use_fast = true

[components.transformer.model.transformer_config]

[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[training]
accumulate_gradient = 3
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
patience = 1600
max_epochs = 0
max_steps = 20000
eval_frequency = 200
frozen_components = []
annotating_components = []
before_to_disk = null

[training.batcher]
@batchers = "spacy.batch_by_padded.v1"
discard_oversize = true
size = 2000
buffer = 256
get_length = null

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = false

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001

[training.optimizer.learn_rate]
@schedules = "warmup_linear.v1"
warmup_steps = 250
total_steps = 20000
initial_rate = 0.00005

[training.score_weights]
ents_f = 1.0
ents_p = 0.0
ents_r = 0.0
ents_per_type = null

[pretraining]

[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null
before_init = null
after_init = null

[initialize.components]

[initialize.tokenizer]

[polm]

Glad that fixed it!

I'm not sure exactly what the cause was here, but in general it is worth being careful when switching a config from a CNN to a Transformer. Usually you won't get anything worse than a slight performance degradation, but this can happen.