Issue with Loki Sink After Upgrading to Vector v0.43.1

[vparfonov]

Hi Vector team,

After upgrading from Vector v0.37.x to v0.43.1, I encountered an issue where logs are not being sent to Loki (v2.8.4) when the compression option is set to "gzip" or "none". However, it works fine with the latest Loki version.
Here is debug output from Vector:

2025-01-22T12:13:25.922616Z DEBUG sink{component_kind="sink" component_id=out component_type=loki}:request{request_id=1}:http: hyper::client::pool: pooling idle connection for ("http", localhost:3100)
2025-01-22T12:13:25.922662Z DEBUG sink{component_kind="sink" component_id=out component_type=loki}:request{request_id=1}:http: vector::internal_events::http_client: HTTP response. status=400 Bad Request version=HTTP/1.1 headers={"content-type": "text/plain; charset=utf-8", "x-content-type-options": "nosniff", "date": "Wed, 22 Jan 2025 12:13:25 GMT", "content-length": "245"} body=[245 bytes]
2025-01-22T12:13:25.922848Z ERROR sink{component_kind="sink" component_id=out component_type=loki}:request{request_id=1}: vector::sinks::util::retries: Non-retriable error; dropping the request. error=Server responded with an error: 400 Bad Request internal_log_rate_limit=true
2025-01-22T12:13:25.922935Z ERROR sink{component_kind="sink" component_id=out component_type=loki}:request{request_id=1}: vector_common::internal_event::service: Service call failed. No retries or retries exhausted. error=Some(ServerError { code: 400 }) request_id=1 error_type="request_failed" stage="sending" internal_log_rate_limit=true
2025-01-22T12:13:25.922990Z ERROR sink{component_kind="sink" component_id=out component_type=loki}:request{request_id=1}: vector_common::internal_event::component_events_dropped: Events dropped intentional=false count=1 reason="Service call failed. No retries or retries exhausted." internal_log_rate_limit=true

Steps to reproduce:

1. Run Loki:
   docker run -d --name=loki -p 3100:3100 grafana/loki:2.8.4
2. Use the following vector.toml

data_dir = "./data_dir/"

[sources.source0]
type = "file"
include = [ "logs.log" ]
read_from = "beginning"

[sinks.out]
type = "loki"
inputs = [ "source0" ]
endpoint = "http://localhost:3100"
out_of_order_action = "accept"
compression = "none"

  [sinks.out.encoding]
  codec = "json"

  [sinks.out.labels]
  host = "hostname123"

3. Write logs to logs.log and observe the behavior.

Observations:

• The same configuration worked in Vector v0.37.x.
• The issue only occurs when using "gzip" or "none" as the compression option. Using "snappy" resolves the issue.
• Upgrading to the latest Loki version resolves the issue, but this might not be suitable for all users.

Questions and Request for Assistance

1. Is this a known issue with changes in the Loki sink or the HTTP client in Vector v0.43.1?
2. Are there any workarounds (other than upgrading Loki) to ensure compatibility with Loki v2.8.4?
3. Could this be related to the way Vector formats or sends the payload with specific compression settings?

Thanks for your help!

[jszwedko]

Hi @vparfonov !

I know there were some changes to the Loki sink in the last 6-12 months. Are you able to identify the first release that changes the behavior for you? We typically recommend updating Vector one minor version at a time since, while Vector is pre-1.0, breaking changes can occur in minor versions.

[vparfonov]

Hello, @jszwedko thanks for your attention.
Yes, i able to check previous versions: it works fine with v0.41.1 but not with v0.42.0

[jszwedko]

Thanks! The only change that jumps out to me in that version is:

The Loki sink now has support for sending structured metadata when using the JSON API (the default). Thanks to maxboone for contributing this change!

I'm having trouble verifying when support for that was added to Loki, but based on https://notes.nicolevanderhoeven.com/Loki+3.0 it might have been Loki 3.0.0 so this may be the reason you are seeing the bad request errors. I'm also having trouble finding Loki's version support policy (we usually try to match Vector's support with vendor support policies) to say if Vector should support Loki 2.8.4.

cc/ @pront I'm curious what you think. At the least this may be worth a note in the docs about supported Loki versions if my assumption is correct.

[pront]

@vparfonov is it possible for you to test with Loki v3.0+? If the above assumption is correct, we should definitely update the docs.

[vparfonov]

@pront Yes, flight is ok with Loki 3.0, 3.1, 3.2 for Vector v0.41.1 and v0.43.1