Consul global proxy configuration not propagating to the proxies #21986

joel-vaz · 2024-12-04T11:04:41Z

Consul global proxy configuration not propagating to the envoy proxies configured via consul connect:

I've a service mesh on ECS with EC2 working as the control plane where my consul server is installed and configured correctly. All my services on ECS work with a consul agent that redirects the requests within the mesh and each service as Envoy working as proxy to communicate with the consul agent (Envoy is installed as a sidecar container for each service). The proxies are configured using the consul connect command: consul connect envoy

My issue is when I apply a global configuration to enable access logging on my Envoy proxies, as in: https://developer.hashicorp.com/consul/docs/connect/observability/access-logs the proxies do not have logs and appear not to have the configuration enabled. Is there something I'm missing?

Versions and details:

Consul: 1.18.2
Envoy: 1.28.7
Services: ECS
Consul Leader: EC2 instance with 1 leader and 2 followers

Output of consul config read -kind proxy-defaults -name global:

{
    "Kind": "proxy-defaults",
    "Name": "global",
    "TransparentProxy": {},
    "Config": {
        "LocalRequestTimeoutMs": 15000,
        "Protocol": "http"
    },
    "MeshGateway": {},
    "Expose": {},
    "AccessLogs": {
        "Enabled": true,
        "JSONFormat": "{\n  \"timestamp\": \"%START_TIME%\",\n  \"protocol\": \"%PROTOCOL%\",\n  \"duration\": \"%DURATION%\",\n  \"upstream_host\": \"%UPSTREAM_HOST%\",\n  \"upstream_cluster\": \"%UPSTREAM_CLUSTER%\",\n  \"response_code\": \"%RESPONSE_CODE%\",\n  \"response_flags\": \"%RESPONSE_FLAGS%\",\n  \"bytes_received\": \"%BYTES_RECEIVED%\",\n  \"bytes_sent\": \"%BYTES_SENT%\",\n  \"request_method\": \"%REQ(:METHOD)%\",\n  \"request_path\": \"%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%\",\n  \"request_id\": \"%REQ(X-REQUEST-ID)%\",\n  \"request_host\": \"%REQ(HOST)%\",\n  \"user_agent\": \"%REQ(USER-AGENT)%\",\n  \"client_ip\": \"%DOWNSTREAM_REMOTE_ADDRESS_WITHOUT_PORT%\",\n  \"connection_termination_details\": \"%CONNECTION_TERMINATION_DETAILS%\",\n  \"upstream_transport_failure_reason\": \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\"\n}\n"
    },
    "CreateIndex": 3666562,
    "ModifyIndex": 17581039
}

Additional information:

After writing the global configuration I checked that all the consul server instances have it correctly applied, I also restarted the services to bootstrap them from scratch. Although nothing seems to work. On the Envoy proxies, looking at curl localhost:19000/config_dump the access logging is not enabled for the sidecar proxies.

Output sample of the Envoy config_dump:

"dynamic_listeners": [
 {
  "name": "public_listener:[REDACTED_IP]:[REDACTED_PORT]",
  "active_state": {
   "version_info": "[REDACTED]",
   "listener": {
    "@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
    "name": "public_listener:[REDACTED_IP]:[REDACTED_PORT]",
    "address": {
     "socket_address": {
      "address": "[REDACTED_IP]",
      "port_value": [REDACTED_PORT]
     }
    },
    "filter_chains": [
     {
      "filters": [
       {
        "name": "envoy.filters.network.rbac",
        "typed_config": {
         "@type": "type.googleapis.com/envoy.extensions.filters.network.rbac.v3.RBAC",
         "rules": {
          "policies": {
           "consul-intentions-layer4": {
            "permissions": [
             {
              "any": true
             }
            ],
            "principals": [
             {
              "authenticated": {
               "principal_name": {
                "safe_regex": {
                 "regex": "^spiffe://[REDACTED]"
                }
               }
              }
             }
            ]
           }
          }
         },
         "stat_prefix": "connect_authz"
        }
       },
       {
        "name": "envoy.filters.network.tcp_proxy",
        "typed_config": {
         "@type": "type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy",
         "stat_prefix": "public_listener",
         "cluster": "local_app"
        }
       }
      ],
      "transport_socket": {
       "name": "tls",
       "typed_config": {
        "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext",
        "common_tls_context": {
         "tls_params": {},
         "tls_certificates": [
          {
           "certificate_chain": {
            "inline_string": "[REDACTED_CERTIFICATE]"
           },
           "private_key": {
            "inline_string": "[REDACTED]"
           }
          }
         ],
         "validation_context": {
          "trusted_ca": {
           "inline_string": "[REDACTED_CA_CERTIFICATE]"
          }
         }
        },
        "require_client_certificate": true
       }
      }
     }
    ],
    "traffic_direction": "INBOUND"
   },
   "last_updated": "[REDACTED_TIMESTAMP]"
  }
 },
 {
  "name": "mesh-sb-test-appliB:[REDACTED_IP]:[REDACTED_PORT]",
  "active_state": {
   "version_info": "[REDACTED]",
   "listener": {
    "@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
    "name": "mesh-sb-test-appliB:[REDACTED_IP]:[REDACTED_PORT]",
    "address": {
     "socket_address": {
      "address": "[REDACTED_IP]",
      "port_value": [REDACTED_PORT]
     }
    },
    "filter_chains": [
     {
      "filters": [
       {
        "name": "envoy.filters.network.http_connection_manager",
        "typed_config": {
         "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
         "stat_prefix": "[REDACTED]",
         "route_config": {
          "name": "[REDACTED]",
          "virtual_hosts": [
           {
            "name": "[REDACTED]",
            "domains": [
             "*"
            ],
            "routes": [
             {
              "match": {
               "prefix": "/"
              },
              "route": {
               "cluster": "[REDACTED]"
              }
             }
            ]
           }
          ]
         },
         "http_filters": [
          {
           "name": "envoy.filters.http.router",
           "typed_config": {
            "@type": "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router"
           }
          }
         ],
         "tracing": {
          "random_sampling": {}
         },
         "upgrade_configs": [
          {
           "upgrade_type": "websocket"
          }
         ]
        }
       }
      ]
     }
    ],
    "traffic_direction": "OUTBOUND"
   },
   "last_updated": "[REDACTED_TIMESTAMP]"
  }
 }
]

Let me know if you need more information.

Kind Regards,

The text was updated successfully, but these errors were encountered:

joel-vaz · 2024-12-09T16:13:01Z

Additional info, i was able to get the access logs on the health checks and admin access, but not on the service-service communication. Is there something I'm missing on my setup?

Example access logging:

I was expecting to have logs from services communicating with the other services using the Envoy proxy 🤔

Any help is appreciated, thanks!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Consul global proxy configuration not propagating to the proxies #21986

Consul global proxy configuration not propagating to the proxies #21986

joel-vaz commented Dec 4, 2024

joel-vaz commented Dec 9, 2024

Consul global proxy configuration not propagating to the proxies #21986

Consul global proxy configuration not propagating to the proxies #21986

Comments

joel-vaz commented Dec 4, 2024

Consul global proxy configuration not propagating to the envoy proxies configured via consul connect:

Versions and details:

Additional information:

joel-vaz commented Dec 9, 2024