Skip to content
This repository has been archived by the owner on Oct 19, 2023. It is now read-only.

Tokenizer loading process #7

Open
mitya52 opened this issue Oct 12, 2023 · 0 comments
Open

Tokenizer loading process #7

mitya52 opened this issue Oct 12, 2023 · 0 comments

Comments

@mitya52
Copy link
Member

mitya52 commented Oct 12, 2023

I found some problems with tokenizer loading:

  • first request to any model causes problem with tokenizer, error looks like this {"detail":"Tokenizer: failed to load tokenizer: EOF while parsing an object at line 6253 column 15"}
  • we should give path to tokenizer's data with model directly without "template": not all models stored in huggingface (3rd-party gpt model for example)
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant