Significant metric loss when using Vector's log_to_metric vs Datadog for nginx logs

[law]

I'm experiencing a significant discrepancy between metrics generated via Vector's log_to_metric transform and Datadog when processing the same nginx logs. The Vector pipeline consistently shows approximately 1/10th of the traffic volume compared to Datadog's measurements.

Setup details:

• Processing custom-formatted nginx logs
• Vector pipeline: nginx logs → log_to_metric transform → Mimir
• Metrics viewed through Grafana
• Same time periods and filtering criteria used in both Grafana and Datadog for comparison

Current observations:

• Mimir distributor logs show no errors
• Vector logs are mostly clean, with occasional parsing failures (few per hour) for specific cases like "client request-body too large" messages
• All components appear to be functioning, just with unexpectedly low data volume

Configurations:

Mimir config: https://pastebin.com/ixUGPwNP
Vector config: https://pastebin.com/2RfCGUK8

Questions:

• What debugging approaches would you recommend to trace where metrics might be getting dropped?
• Are there specific Vector or Mimir metrics I should monitor to identify potential bottlenecks?
• Would logs from any other components be helpful for diagnosis?

Any guidance on troubleshooting techniques or configuration issues would be greatly appreciated.

[pront]

Hi @law, here is a list of tools and techniques I use when debugging complex pipelines:

• vector top to see component metrics - docs here
  • For example, you can inspect how many events are produced by sources vs how many events reach the sinks
  • You can also add an internal_metrics source and inspect the metrics
• With vector tap you see the input and/or outputs of your components - guide here
  • caveat: sinks don't have outputs, you will need to check the downstream systems for actual payloads
  • Alternative: for each source or transform that I want to test, I connect it to a console sink (doesn't scale to many components but it is a very quick sanity check)

For your case specifically, I would investigate if events are dropped or if events get stuck in a retry loop. For example, is this component_discarded_events_total metric greater than 0?

[jszwedko]

One potential thing to check here, since you mentioned it was off by a factor of 10, is if the metric type and interval is being set correctly in Datadog. We've seen issues in the past where the interval or type was wrong, causing things to appear to be off by a factor of 10. Also, what version of Vector are you running?