Releases: mlverse/tok
Releases · mlverse/tok
v0.1.4
v0.1.3
v0.1.2
- Added support for authentication. (#7)
- Added support for training tokenizers from scratch. (#8)
- Added support for serializing tokenizers. (#8)
- Implemented support for configuring padding and truncation. (#9)
- Added support for decoders, post_processors and normalizers. (#10)
- Added support for
get_vocab_size()
. (#10) - Updated extendr version to avoid non-API function calls. (#11)