Skip to content

Releases: indobenchmark/indobenchmark-toolkit

Release v0.1.4

22 Jun 01:59
Compare
Choose a tag to compare
  • Fix spacing between subword when decoding using IndoNLGTokenizer
  • Remove unused additional special tokens '[java]', '[sunda]', '[indonesia]' from IndoNLGTokenizer (language tokens are included in the special_tokens_to_ids instead)