Relation extraction component - assertion error raised #12755
-
IssueWhile using the relation extraction component, I'm regularly coming across an AssertionError related to thinc while training the component on data annotated with Prodigy. Code and configurationCode has been updated following a thread in Prodigy discussion forum. Updated code and configuration files are available in this repository. I've deleted assets, data and training folders as well as erased my labels in SYMM_LABEL and DIRECTED_LABELS in scripts/parse_data_generic.py file to ensure data confidentiality. CLIData command output :
Training command output (raising the error) :
Occasionnaly, on other samples of training data, I also encounter the following error during the data command execution :
Data sampleFor confidentialy issues, I can't share the training data. It has been annotated with Prodigy. Here is a small example to check the data format returned by Prodigy :
It seems like the data is not correctly read. It is named annotations.jsonl and put in assets folder. It is created thanks to the db-out command from Prodigy. If you need any additional information; please let me know. |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 4 replies
-
|
Hi Stella, From your config file, I understand that you're training the NER model and the REL model at the same time, adding the NER to Yet, we're getting this warning:
This warning points to the fact that the relation extractor doesn't have any instances to train on. This most likely happens because at that point in time, the NER wasn't yet trained sufficiently to actually recognize the entities from the gold REL data, and thus the REL couldn't continue onwards. This For the REL to be able to work, we need to make sure that the NER actually works by itself. Have you tried training the NER in isolation (i.e. with no relations?). If you have a sufficiently well working NER model, then we can use that as a frozen model in the next phase when training the REL. I think this will result in a more stable approach. |
Beta Was this translation helpful? Give feedback.
-
|
Hi Sofie, Exactly, I'm training both models simultaneously. Concerning the AssertionError, it's good to know. Yes, I've made sure that the NER model works by itself. Here is the CLI and its output (labels anonymized) :
According to the output, the NER model works. So using the NER model looks a more promising approach ? How should I proceed ? (More precisely, what would be the appropriate configuration file ?) |
Beta Was this translation helpful? Give feedback.
-
|
Hi Stella, It's become more difficult for me to follow the exact trace of this thread, as you've edited the original post you made after I replied to it. Specifically, I made a comment about your config file, which I now can't see in the post anymore. I see that instead you've linked a directory which contains different config files etc. While it's useful to share as much code & data that you've got, it's also necessary that I understand exactly which scripts & config files were run. I also see that you added a section "Occasionnaly, on other samples of training data, ...". As I said before, it's unhelpful to be mixing various problems into one thread. If you want to address this issue also, please open a separate thread and clarify which code & data resulted in that error, so we can look into it as a separate thing. Here, I'll try to stay focused on the issue with the NER data and the REL not getting instances to train from. From the output logs you've shared, it looks like you're not always using the same data set, is that correct? Throughout this thread, I see varying mentions of data sets: one with 12 training instances, one with 3, one with 21. It would be good to keep this fixed in order for us to be able to drill down onto the problem. Looking at the latest log you posted from training the NER:
Am I understanding this correctly that you've got 21 instances to train on, and 13 distinct labels in your dataset? If so, that's definitely too little data to train an NER model on. You're evaluating on only 2 instances, which means that the reported 80% F-score is extremely unreliable, and your model is unlikely to generalize properly to unseen data. This may seem like a detail to you - but I still believe that too little data, and an insufficiently trained NER model, is actually the root cause of the error you saw in your original post. Specifically, the REL needs to build on an NER model that is robust and accurate, which is not something you can achieve with only 21 instances for 13 labels, I'm afraid. Do you happen to have more data annotated? |
Beta Was this translation helpful? Give feedback.
-
|
Hi Sofie, Sorry about that. Actually, the config file previously described was rel_joint.cfg, but rel_tok2vec.cfg also looks involved in the evaluation process, that's why I've linked the whole repository, to prevent lack of data. I think concerning the "occasionaly..." part, it may be linked to the same issue. I'll try to clarify this later, as it could be confusing. Of course, my final training data won't contain 21 examples, but hopefully more than a thousand. But if the workflow is not working or if there is anything wrong with the annotation process, I think it is not wise to focus on the annotation. My thought was to ensure that the workflow works from beginning to end before going too deep in the annotation process. So I'll explain why I think the "occasionaly" part may concern the same issue. I have one very small dataset (approximately 30 examples). I'm sometimes deleting few data samples to test on a minimal reproducible example. First, the model training part works perfectly on very few data inputs (maybe because there is not enough data to train a rel model, so it may be skipped). Then, on a bigger dataset, it could trigger the "occasionaly' error (like if the training data was not correctly parsed, yet it is valid). On the biggest dataset, it triggers the error described in the thread. I understand that the insufficiently trained NER model could trigger the error. But isn't there a way to ensure the component stability without first spending time on developing the training data ? I don't have more data annotated for the moment, as the fact that an errror was triggered prevented me from going too deep into this process. Thanks. |
Beta Was this translation helpful? Give feedback.
-
|
I've annotated more data (as much as possible in the span of time as the data is very complex and from a very specific domain of application). The NER model alone can be trained without any issue :
The NER + rel model trained together has issues. The triggered issue is inconsistent : with less data, I was able to train a model once without an issue, and without changing anything except deleting the model and .spacy files, the rerun triggered an issue. With the actual training data, the issue triggered depends on the run. I'll show you multiple runs of the same command on the same data : First run :
Second run :
Third run :
If I'm doing anything wrong, could you tell me what ? If there is nothing wrong with what I'm doing, could you :
And if not possible :
I don't think annotating more data will solve the issue ? I'm looking for a quick solution if possible. |
Beta Was this translation helpful? Give feedback.
-
|
Hi Stella, Thanks, that's all very useful.
Totally - I agree. I wanted to check whether you already have more data annotated right now, to verify whether the errors still occur if you'd use all of the data available.
That's interesting. In the case of the "very few data inputs", what do you mean by "works perfectly", while you're also hypothesizing that the training might be skipped? Do you mean that the training runs without error, but the score is really just all 0?
Right, that's good to know. To me, this again points to the fact that the REL breaks down when the NER model is being trained and is not yet stable.
I want to clarify once more that, as stated in the video tutorial, the REL project serves as a tutorial on how to implement a custom spaCy component with a custom Thinc model. It is definitely not meant to serve as some kind of stable REL component. If we do develop an actual proper REL component in the future, we would include it in spaCy's core code base, not in the "tutorials" section of our example projects. I'm saying this to clarify that our main motivation here is not to develop a stable REL component. It's to help you get going with your specific use-case and to give you pointers on how to implement your own custom NLP solution with spaCy.
The approach you've tried so far could have worked, but at this point I suggest a more phased approach that lets us tackle your challenges one by one. You've already demonstrated that the NER model training works, so that would be the first step: train an NER model from your data, and store the model to disk (let's say it's in a directory In the next step then, we'll train the REL model only, while keeping the What this effectively does, is that a new pipeline is composed that grabs the trained The Finally, we use the Once this pipeline is trained (🤞) and saved to disk, to let's say and run it on text, at which point the pipeline will create predictions for both the NER and the REL components. |
Beta Was this translation helpful? Give feedback.
Hi Stella,
Thanks, that's all very useful.
Totally - I agree. I wanted to check whether you already have more data annotated right now, to verify whether the errors still occur if you'd use all of the data available.