diff --git a/README.md b/README.md index e513226..a413d32 100644 --- a/README.md +++ b/README.md @@ -4,12 +4,17 @@ A Prometheus exporter for custom userspace (e.g. `usdt`, `uprobe`) eBPF metrics. ## Why? -The existing [ebpf_exporter](https://github.com/cloudflare/ebpf_exporter) allows for collection of system-wide eBPF metrics, but does not expose any facilities for exporting metrics from userspace probes. +The existing [ebpf_exporter](https://github.com/cloudflare/ebpf_exporter) allows for collection of system-wide eBPF metrics such as kprobes, but does not expose any facilities for exporting metrics from userspace probes. These metrics can be quite useful in illuminating aspects of a running process; see the [examples](./examples) for some use cases. Generally, one cannot attach userspace probes to a process in a different namespace, limiting the feasibility of userspace probes in containerized environments. To work around this, this exporter is designed to run as a [sidecar with namespace sharing](https://kubernetes.io/docs/tasks/configure-pod-container/share-process-namespace/) +## Prerequisites + +You will need to install [bcc](https://github.com/iovisor/bcc/blob/master/INSTALL.md#source). +The exporter has been tested with [v0.18.0](https://github.com/iovisor/bcc/releases/tag/v0.18.0). + ## Usage To bind to `0.0.0.0:8080` and expose metrics under `/metrics`, run as: @@ -68,12 +73,12 @@ spec: ## Configuration -The configuration format is mostly lifted from the `ebpf_exporter`'s [configuration format](https://github.com/cloudflare/ebpf_exporter#configuration) with some changes +The configuration format is mostly lifted from the `ebpf_exporter`'s [configuration format](https://github.com/cloudflare/ebpf_exporter#configuration) with some changes. ### `program` -``` +```yaml # Program name name: # Metrics attached to the program @@ -102,7 +107,7 @@ Note that, since this exporter does not deal with system-level metrics, `kprobes ### `attachments` -``` +```yaml attachments: binary_name: [ binary_name ] ``` @@ -111,3 +116,70 @@ The `attachments` section details which processes the eBPF program will be attac Currently, this only supports attaching by binary name -- all processes whose binary name equals the one given will be targeted. *NOTE* This is the binary name as reported by `/proc/${PID}/comm` + +### Examples + +The following example will instrument garbage collection for all `gunicorn` processes: + +```yaml +programs: + - name: gc_total + metrics: + counters: + - name: gc_total + help: Total number of gc events + table: gc_counts + labels: + - name: gen + size: 8 + decoders: + - name: uint + usdt: + gc__start: trace_gc__start + attachment: + binary_name: "gunicorn" + code: | + struct gc_event_t { + u64 gen; + }; + + BPF_HASH(gc_counts, struct gc_event_t); + + int trace_gc__start(struct pt_regs *ctx) { + struct gc_event_t e = {}; + int gen = 0; + bpf_usdt_readarg(1, ctx, &gen); + e.gen = gen; + gc_counts.increment(e); + return 0; + } +``` + +Resulting metrics: + +``` +# HELP userspace_exporter_enabled_programs The set of enabled programs +# TYPE userspace_exporter_enabled_programs gauge +userspace_exporter_enabled_programs{name="gc_total",pid="29970"} 1 +userspace_exporter_enabled_programs{name="gc_total",pid="29971"} 1 +userspace_exporter_enabled_programs{name="gc_total",pid="29972"} 1 +userspace_exporter_enabled_programs{name="gc_total",pid="29973"} 1 +userspace_exporter_enabled_programs{name="gc_total",pid="29974"} 1 +# HELP userspace_exporter_gc_total Total number of gc events +# TYPE userspace_exporter_gc_total counter +userspace_exporter_gc_total{gen="2",pid="29971"} 753 +userspace_exporter_gc_total{gen="2",pid="29972"} 764 +userspace_exporter_gc_total{gen="2",pid="29973"} 765 +userspace_exporter_gc_total{gen="2",pid="29974"} 748 +``` + +## Status + +This is a hobby project; it should not be considered production ready. + +It is missing a few features that I hope to implement over the coming months: + +* The monitored process must be live before the exporter starts, and if it is restarted the exporter will not reattach. In future, it would be nice if it could dynamically attach to any live processes matching the `binary_name`. +* Attaching by binary name isn't very flexible; there are many different ways to find a process of interest -- by its parent process, by its command line, etc +* The original `ebpf-exporter` is able to add a [`tag`](https://github.com/cloudflare/ebpf_exporter#ebpf_exporter_ebpf_programs) label to its info metrics. This is challenging to do here since the USDT APIs don't easily lend themselves to getting a program's tag, but it would be good to at least add it for u(ret)probes. +* Some JVM examples would be fantastic