Special characters gets removed

[cfgs]

Description

I'm using Verba (1.28.2) together with Weaviate Cloud. When uploading both .txt- and .pdf-files including the characters åäö, they get stripped out when inspecting the documents uploaded.

E.g example.txt
Uploaded: "Hej detta är ett test, jag bor på en ö"
Output: "Hej detta r ett test, jag bor p en "

When using the built in chat in Verba, it performs well but the referenced document is - as stated - stripped out of åäö. What could be the issue?

Installation

• pip install goldenverba
• pip install from source
• Docker installation

If you installed via pip, please specify the version:

Weaviate Deployment

• Local Deployment
• Docker Deployment
• Cloud Deployment

Configuration

Reader:
Chunker: Token
Embedder: VoyageAI (Model: voyage-multilingual-2, should support Swedish)
Retriever: Advanced
Generator: OpenAI (Model: gpt-4o)

Steps to Reproduce

1. Uploading a .txt- or a .pdf containing the character åäö through the "Import Data" function in Verba GUI.
2. Select the uploaded file in the list to the left, press "Import selected".
3. Go to "Documents".
4. Click the uploaded document in the list to the left
5. Inspect the document (which is now opened to the right) and find a sentence that should contain either å, ä or ö. Confirm that the letter is not there.

[thomashacker]

Interesting, thanks for the issue! I'll look into this 🚀