How to convert jsonl format files into spacy format files?

[Rainsmumu]

Hi！
I plan to customize a ner model. The data set is a JSONL file exported from the annotation tool Doccano, I use convert command to convert the format into spacy format file ：

python -m spacy convert input_data/all.jsonl input_data -c json

I know this converter is for json files, but I didn't find a converter for jsonl files. Then I used the debug command

python -m spacy debug data ./config.cfg 

to see if there was problems with my data set, and I got errors like this:

Traceback (most recent call last):
  File "D:\Anaconda\envs\sapcy\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "D:\Anaconda\envs\sapcy\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "D:\Anaconda\envs\sapcy\lib\site-packages\spacy\__main__.py", line 4, in <module>
    setup_cli()
  File "D:\Anaconda\envs\sapcy\lib\site-packages\spacy\cli\_util.py", line 87, in setup_cli
    command(prog_name=COMMAND)
  File "D:\Anaconda\envs\sapcy\lib\site-packages\click\core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "D:\Anaconda\envs\sapcy\lib\site-packages\typer\core.py", line 778, in main
    return _main(
  File "D:\Anaconda\envs\sapcy\lib\site-packages\typer\core.py", line 216, in _main
    rv = self.invoke(ctx)
  File "D:\Anaconda\envs\sapcy\lib\site-packages\click\core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "D:\Anaconda\envs\sapcy\lib\site-packages\click\core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "D:\Anaconda\envs\sapcy\lib\site-packages\click\core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "D:\Anaconda\envs\sapcy\lib\site-packages\click\core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "D:\Anaconda\envs\sapcy\lib\site-packages\typer\main.py", line 683, in wrapper
    return callback(**use_params)  # type: ignore
  File "D:\Anaconda\envs\sapcy\lib\site-packages\spacy\cli\debug_data.py", line 96, in debug_data_cli
    debug_data(
  File "D:\Anaconda\envs\sapcy\lib\site-packages\spacy\cli\debug_data.py", line 136, in debug_data
    nlp.initialize(lambda: train_corpus(nlp))
  File "D:\Anaconda\envs\sapcy\lib\site-packages\spacy\language.py", line 1349, in initialize
    proc.initialize(get_examples, nlp=self, **p_settings)
  File "D:\Anaconda\envs\sapcy\lib\site-packages\spacy\pipeline\tok2vec.py", line 217, in initialize
    assert doc_sample, Errors.E923.format(name=self.name)
AssertionError: [E923] It looks like there is no proper sample data to initialize the Model of component 'tok2vec'. To check your input data paths and annotation, run: python -m spacy debug data config.cfg and include the same config override values you would specify for the 'spacy train' command

There is something wrong during the process of converting the jsonl file into a spacy file, but I dont know how to solve it, and I really appreciate help!
And my data set looks like this：

{"id":97,"text":"The amphiphilic epoxy sizing emulsions with different molecular weights (i.e., 7500, 11,000, 15,000 and 17,000) were successfully prepared by a self-emulsifying method through introducing hydrophilic and hydrophobic groups into the epoxy molecular structure. \nThe particle size of the obtained emulsions is in the nanometres range. \nThe four emulsions prepared during this study all have strong thermal stability. \nWhen they lose 5% of their mass by heating, the required decomposition temperature is above 300 ◦C. \nThe sized CFs by the emulsion of 7500 exhibited the most favourable wettability. \nThe grooves on the CF surface became shallower when undergoing sizing treatment but the emulsions with high molecular weight (15,000 and 17,000) induced irregular bulges.\nThe novel sizing agent emulsion was shown to be effective for improving the mechanical performance of sized CFs and interfacial adhesion. \nThe monofilament fibre tensile strength was enhanced by treating the defects on the CF surface. \nThe sizing agent of low molecular weight (7500) exhibited the best quality with a tensile strength of 3.44 GPa and a maximum shape parameter of 7.20.\nThe IFSS of CF\/EP composite was enhanced by 96% with the emulsion of 7500 compared with unsized CF. \nThis resulted from the increased charge interaction between the CF and the sizing layer as well as the compatibility and the mechanical interlock between the sizing layer and the epoxy matrix.","entities":[{"id":1596,"label":"auxiliary additive","start_offset":773,"end_offset":801},{"id":1142,"label":"composite","start_offset":1167,"end_offset":1182},{"id":1137,"label":"properties value","start_offset":507,"end_offset":513},{"id":1140,"label":"properties value","start_offset":1106,"end_offset":1115},{"id":1592,"label":"value","start_offset":79,"end_offset":83},{"id":1593,"label":"value","start_offset":85,"end_offset":91},{"id":1594,"label":"value","start_offset":93,"end_offset":99},{"id":1595,"label":"value","start_offset":104,"end_offset":110},{"id":1139,"label":"value","start_offset":1047,"end_offset":1051}],"relations":[],"Comments":[]}
{"id":98,"text":"In this paper, three lasers with different pulse durations were used to drill CFRP laminates, and the influence of lasers on HAZ was discussed. Then, the effects of HAZ, sample thickness, and cutting-induced geometrical defect on the sample tensile strength were studied. \nFinally, FEM simulation was carried out to observe the stress evolution within the laminate material during stretching. \nThen, the mechanism responsible for tensile strength weakening due to HAZ was revealed. \nThe following conclusions can be drawn:\n(1) The tensile strength of laser machining CFRP laminates is affected by HAZ. \nWith the HAZ widths increasing, the tensile strength of laminates will decrease. \nIn addition to HAZ, cutting-induced geometric defects also seriously influence the tensile strength, because of the stress concentration around these defects during the stretching process.\n(2) There is a linear relationship between HAZ width and tensile strength. \nThe tensile strength of 1 mm CFRP composites is decreased by about 122 MPa\/mm HAZ, and the tensile strength of 2.5 mm CFRP composites is decreased by about 33.9 MPa\/mm HAZ. \nWith increasing laminate thickness, the effect of HAZ on the target tensile strength would be weakened. \nOn the other hand, the influences of other defects (except for HAZ) may become more highlighted with the thickness increasing. \nTherefore, the laser machining method is more appropriate for thicker CFRP composites because the negative effect of HAZ is weakened.\n(3) According to our acknowledgment, it was the first time the underlined mechanism responsible for HAZ-induced strength reduction through FEM simulation was explained. \nWithin the fiber-exposure HAZ, the loads cannot be transferred effectively between the fibers. \nThe specimen with wider HAZ would have fewer areas to bear the exerting loads, and then the test strength will decrease accordingly.","entities":[{"id":1604,"label":"composite","start_offset":78,"end_offset":92},{"id":1557,"label":"properties value","start_offset":1017,"end_offset":1024},{"id":1558,"label":"properties value","start_offset":1105,"end_offset":1114}],"relations":[],"Comments":[]}

[adrianeboyd]

The JSON converter isn't a generic JSON converter. It's only intended for converting a specific spacy v2 JSON format, so here it unfortunately ends up producing an empty .spacy file, which is what leads to this initialization error (there is no data).

For this JSONL format, there isn't a built-in converter and your best option may be to write your own converter. In a quick search, I can find a number of examples related to converting doccano to spacy, but I haven't tried any of them myself, and they may produce spacy v2 JSON format. The v2 JSON format can still be used, but then you'd need to an additional conversion step to convert it for spacy v3 with the current spacy convert.

To help you get started if you can't find an existing converter, here's an example of custom converter for a JSON format:

https://github.com/explosion/projects/blob/v3/pipelines/ner_demo/scripts/convert.py

for this data (since it's pretty similar in terms of texts and offsets, so primarily adjust how it reads text + offsets/labels for your format):

https://github.com/explosion/projects/blob/v3/pipelines/ner_demo/assets/train.json