otlp_config not overriding default_resource_attributes_as_index_labels

[Amoolaa]

Describe the bug
When a resource attribute is listed in default_resource_attributes_as_index_labels and also listed in otlp_config as structured_metadata, the default_resource_attributes_as_index_labels behaviour of setting it as an index_label is respected vs the otlp_config overriding this behaviour.

To Reproduce
Set

distributor:
  otlp_config:
    default_resource_attributes_as_index_labels: ["foo"]
limits_config:
  otlp_config:
    resource_attributes:
      attributes_config:
        - action: structured_metadata
           attributes:
             - foo

Expected behavior
Not sure as it's not documented, but I was wondering if this should be the case. Example: k8s.pod.name set in default_resource_attributes_as_index_labels, however this is a high cardinality attribute for a tenant so it would be easier to drop rather than set ignore_defaults to true and then have to re-list all the default_resource_attributes_as_index_labels minus k8s.pod.name. I think there probably should be better documentation on what the priority is between default_resource_attributes_as_index_labels and otlp_config, and what happens if the same attribute name is listed multiple times.

From my (limited) understanding - based on this

loki/pkg/loghttp/push/otlp.go

Lines 129 to 158 in a56eabe

	resAttrs.Range(func(k string, v pcommon.Value) bool {
	action := otlpConfig.ActionForResourceAttribute(k)
	if action == Drop {
	return true
	}
	attributeAsLabels := attributeToLabels(k, v, "")
	if action == IndexLabel {
	for _, lbl := range attributeAsLabels {
	streamLabels[model.LabelName(lbl.Name)] = model.LabelValue(lbl.Value)
	if logPushRequestStreams && pushedLabels != nil {
	pushedLabels[model.LabelName(lbl.Name)] = model.LabelValue(lbl.Value)
	}
	if !hasServiceName && shouldDiscoverServiceName {
	for _, labelName := range discoverServiceName {
	if lbl.Name == labelName {
	streamLabels[model.LabelName(LabelServiceName)] = model.LabelValue(lbl.Value)
	hasServiceName = true
	break
	}
	}
	}
	}
	} else if action == StructuredMetadata {
	resourceAttributesAsStructuredMetadata = append(resourceAttributesAsStructuredMetadata, attributeAsLabels...)
	}
	return true
	})

it loops through the resource attributes in the OTLP payload, and for each resource attribute it loops through each item in attributes_config checking if the resource attribute is listed under attributes here

loki/pkg/loghttp/push/otlp_config.go

Lines 89 to 101 in a56eabe

	func (c *OTLPConfig) actionForAttribute(attribute string, cfgs []AttributesConfig) Action {
	for i := 0; i < len(cfgs); i++ {
	if cfgs[i].Regex.Regexp != nil && cfgs[i].Regex.MatchString(attribute) {
	return cfgs[i].Action
	}
	for _, cfgAttr := range cfgs[i].Attributes {
	if cfgAttr == attribute {
	return cfgs[i].Action
	}
	}
	}
	return StructuredMetadata

So it's based on the order of the items in attributes_config. And because of this https://github.com/grafana/loki/blob/main/pkg/loghttp/push/otlp_config.go#L78-L87 it seems like the default_resource_attributes_as_index_labels is listed first, meaning it will always have priority as that will be the first attributes_config item in the list to be searched. Personally I think the behaviour should be for overrides to have precedence over the defaults.

Environment:

• Infrastructure: Kubernetes
• Deployment tool: helm