Releases: indobenchmark/indobenchmark-toolkit
Releases · indobenchmark/indobenchmark-toolkit
Release v0.1.4
- Fix spacing between subword when decoding using IndoNLGTokenizer
- Remove unused additional special tokens '[java]', '[sunda]', '[indonesia]' from IndoNLGTokenizer (language tokens are included in the
special_tokens_to_ids
instead)