Start to add demo of routing observability data

datadog-subaccount-routing/.gitignore

@@ -0,0 +1 @@
+api-keys-config-map.yaml

datadog-subaccount-routing/README.md

@@ -0,0 +1,126 @@
+# Datadog Agent account routing
+
+This demo demonstrates how to use Vector to ingest logs, metrics, and traces
+from the Datadog Agent and route them to different Datadog accounts. In this
+case, it routes them by Kubernetes namespace, but the example can be adapted to
+your use case.
+
+## Prerequisites
+
+- [helm](https://helm.sh/docs/intro/install/)
+- [kubectl](https://kubernetes.io/docs/tasks/tools/)
+- [minikube](https://minikube.sigs.k8s.io/docs/start/)
+
+Three Datadog accounts:
+
+- One root one for observability data outside of the application namespaces
+- One for applications running in the `foo` namespace we will create below
+- One for applications running in the `bar` namespace we will create below
+
+You can also run this demo with one Datadog account by setting the Datadog API
+key to the same value for all three when exporting the environment variables
+below.
+
+## Running the demo
+
+Start minikube:
+
+```shell
+minikube start
+```
+
+Add the necessary Helm repositories for the Vector and Datadog charts:
+
+```shell
+helm repo add datadog https://helm.datadoghq.com
+helm repo add vector https://helm.vector.dev
+helm repo update
+```
+
+We will be deploying two demo apps in two different namespaces.
+
+Start by exporting the needed Datadog API keys (replacing the `...`s):
+
+```shell
+# this should be the key you want data to go to when it is in the kube-system,
+# default, or an unknown namespace
+export DD_API_KEY=...
+# this should be the key you want data to go when it is from the `foo` namespace
+export DD_API_KEY_FOO=...
+# this should be the key you want data to go when it is from the `bar` namespace
+export DD_API_KEY_BAR=...
+```
+
+Build the demo app that we will deploy to the `foo` and `bar` namespaces:
+
+```shell
+docker build --file test-app/Dockerfile --tag test-app:latest test-app
+```
+
+And upload to minikube:
+
+```shell
+minikube image load test-app:latest
+```
+
+We will now deploy demo app to both the `foo` and `bar` namespaces:
+
+```shell
+kubectl create namespace foo
+kubectl create namespace bar
+kubectl apply --namespace foo --filename test-app/deployment.yaml
+kubectl apply --namespace bar --filename test-app/deployment.yaml
+```
+
+Now that we have the demo app running in both namespaces, let's deploy the
+Datadog agent to collect its telemetry. The demo app generates logs, metrics,
+and traces.
+
+```shell
+helm install datadog-agent -f datadog.values.yaml --set datadog.apiKey=$DD_API_KEY datadog/datadog
+```
+
+With the Datadog Agent installed, we can deploy Vector to route the
+observability data.
+
+Start by creating a config map to hold the Kubernetes namespace -> Datadog API
+key mapping. We will use this with Vector's [CSV enrichment
+feature](https://vector.dev/highlights/2021-11-18-csv-enrichment/) to lookup the
+API key to use by namespace.
+
+```shell
+cat <<-EOF > api-keys-config-map.yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: "namespace-api-keys-configmap"
+data:
+  namespace-api-keys.csv: |
+    namespace,api_key
+    default,$DD_API_KEY
+    kube_system,$DD_API_KEY
+    unknown,$DD_API_KEY
+    foo,$DD_API_KEY_FOO
+    bar,$DD_API_KEY_BAR
+EOF
+```
+
+In the real world, you will probably want to load these as secrets, but here we
+use plaintext for simplicity. You can see here that we map the `foo` and `bar`
+namespaces to their respective API keys and the rest are mapped to `DD_API_KEY`.
+The `unknown` namespace will be for when Vector cannot find the namespace on the
+incoming data (usually indicating the logs or metrics refer to the kubernetes
+control plane).
+
+Now deploy Vector:
+
+```shell
+helm install vector vector/vector --values vector.values.yaml
+```
+
+Once Vector is deployed, you will see data flowing into each of your Datadog
+accounts.
+
+## Cleaning up
+
+You can run `minikube delete` to teardown all resources.

datadog-subaccount-routing/datadog.values.yaml

@@ -0,0 +1,27 @@
+datadog:
+  kubelet:
+    tlsVerify: false # for minikube
+  logLevel: info
+  logs:
+    enabled: true
+    containerCollectAll: true
+  prometheusScrape:
+    enabled: true
+  apm:
+    portEnabled: true
+  containerExclude: "kube_namespace:.*"
+  containerInclude: "kube_namespace:foo kube_namespace:bar"
+agents:
+  useConfigMap: true
+  customAgentConfig:
+    vector:
+      logs:
+        enabled: true
+        url: "http://vector:8282"
+      metrics:
+        enabled: true
+        url: "http://vector:8282"
+    apm_config:
+      enabled: true
+      receiver_port: 8126
+      apm_dd_url: "http://vector:8282"

datadog-subaccount-routing/test-app/Dockerfile

@@ -0,0 +1,10 @@
+FROM python:3
+
+WORKDIR /usr/src/app
+
+COPY requirements.txt ./
+RUN pip install --no-cache-dir -r requirements.txt
+
+COPY main.py main.py
+
+CMD [ "python", "./main.py" ]

datadog-subaccount-routing/test-app/deployment.yaml

@@ -0,0 +1,25 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: test-app-deployment
+  labels:
+    app: test-app
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: test-app
+  template:
+    metadata:
+      labels:
+        app: test-app
+      annotations:
+        prometheus.io/scrape: 'true'
+        prometheus.io/port: '9100'
+    spec:
+      containers:
+      - name: test-app
+        imagePullPolicy: Never
+        image: docker.io/library/test-app:latest
+        ports:
+        - containerPort: 9100

datadog-subaccount-routing/test-app/main.py

@@ -0,0 +1,34 @@
+# Fake program for generating logs, traces, and metrics for demonstration
+
+from prometheus_client import start_http_server, Summary
+import logging
+import random
+import time
+import os
+from ddtrace import tracer
+
+# Create a metric to track time spent and requests made.
+REQUEST_TIME = Summary('request_processing_seconds', 'Time spent processing request')
+
+# Decorate function with metric.
+@REQUEST_TIME.time()
+# Trace function.
+@tracer.wrap('process_request', service="vector-demo-test-app")
+def process_request():
+    time.sleep(random.random())
+    logging.info("request processed")
+
+if __name__ == '__main__':
+    logging.basicConfig(level=logging.INFO)
+
+    tracer.configure(
+        hostname="127.0.0.1",
+        port="8126",
+    )
+
+    # Start up the server to expose the metrics.
+    start_http_server(9100)
+
+    # Generate fake requests.
+    while True:
+        process_request()

datadog-subaccount-routing/test-app/requirements.txt

@@ -0,0 +1,2 @@
+ddtrace==1.1.4
+prometheus-client==0.14.1

datadog-subaccount-routing/vector.values.yaml

@@ -0,0 +1,83 @@
+role: "Stateless-Aggregator"
+
+replicas: 1
+
+service:
+  ports:
+  - protocol: TCP
+    port: 8282
+    targetPort: 8282
+
+image:
+  tag: "nightly-2022-06-01-distroless-libc"
+
+extraVolumes:
+  - name: namespace-api-keys
+    configMap:
+      name: namespace-api-keys-configmap
+
+extraVolumeMounts:
+  - name: namespace-api-keys
+    mountPath: /var/lib/vector/data
+
+customConfig:
+  enrichment_tables:
+    keys:
+      type: "file"
+      file:
+        path: "/var/lib/vector/data/namespace-api-keys.csv"
+        encoding:
+          type: "csv"
+  sources:
+    datadog_agent:
+      address: 0.0.0.0:8282
+      type: datadog_agent
+      multiple_outputs: true
+      store_api_key: false # we will look up the key
+  transforms:
+    remap_logs:
+      type: remap
+      inputs:
+        - datadog_agent.logs
+      source: |
+        # Parse the received .ddtags field so we can more easily access the contained tags
+        .ddtags = parse_key_value!(.ddtags, key_value_delimiter: ":", field_delimiter: ",")
+
+        namespace = string(.ddtags.kube_namespace) ?? "default"
+
+        row, err = get_enrichment_table_record("keys", { "namespace": namespace })
+        if err != null {
+          log("could not find api key for namespace: " + namespace)
+          abort
+        }
+
+        set_metadata_field!("datadog_api_key", row.api_key)
+
+        # Re-encode Datadog tags as a string for the `datadog_logs` sink
+        .ddtags = encode_key_value(.ddtags, key_value_delimiter: ":", field_delimiter: ",")
+
+        # Datadog Agents pass a "status" field that is stripped when ingested
+        del(.status)
+    remap_metrics:
+      type: remap
+      inputs:
+        - datadog_agent.metrics
+      source: |
+        namespace = string(.tags.namespace) ?? "default"
+
+        row, err = get_enrichment_table_record("keys", { "namespace": namespace })
+        if err != null {
+          log("could not find api key for namespace: " + namespace)
+          abort
+        }
+
+        set_metadata_field!("datadog_api_key", row.api_key)
+  sinks:
+    datadog_logs:
+      type: datadog_logs
+      inputs: [remap_logs]
+      default_api_key: "" # no default
+    datadog_metrics:
+      type: datadog_metrics
+      inputs: [remap_metrics]
+      default_api_key: "" # no default