Difference between Inky Frame and Inky Impression in displayed image

[Sennevds]

I have both the inky impression and frame.
Both use the same (already cropped and dithered image) and get these from a dotnet core api (same endpoint)
The endpoint returns a filestream result with the contentType. This is then locally saved and shown on the display.

With the inky frame (pi pico) nothing is done with the result because it's already dithered but on the inky impression this jpg/png is always converted (because the image loads as RGB mode) which results in a image with reduced colors.
Is there a possibility to not convert in image and just load it as is?