Skip to content

Commit

Permalink
Add a few notes to the README.
Browse files Browse the repository at this point in the history
  • Loading branch information
josecv committed Apr 23, 2021
1 parent 154499b commit efa441e
Showing 1 changed file with 76 additions and 4 deletions.
80 changes: 76 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,17 @@ A Prometheus exporter for custom userspace (e.g. `usdt`, `uprobe`) eBPF metrics.

## Why?

The existing [ebpf_exporter](https://github.com/cloudflare/ebpf_exporter) allows for collection of system-wide eBPF metrics, but does not expose any facilities for exporting metrics from userspace probes.
The existing [ebpf_exporter](https://github.com/cloudflare/ebpf_exporter) allows for collection of system-wide eBPF metrics such as kprobes, but does not expose any facilities for exporting metrics from userspace probes.
These metrics can be quite useful in illuminating aspects of a running process; see the [examples](./examples) for some use cases.

Generally, one cannot attach userspace probes to a process in a different namespace, limiting the feasibility of userspace probes in containerized environments.
To work around this, this exporter is designed to run as a [sidecar with namespace sharing](https://kubernetes.io/docs/tasks/configure-pod-container/share-process-namespace/)

## Prerequisites

You will need to install [bcc](https://github.com/iovisor/bcc/blob/master/INSTALL.md#source).
The exporter has been tested with [v0.18.0](https://github.com/iovisor/bcc/releases/tag/v0.18.0).

## Usage

To bind to `0.0.0.0:8080` and expose metrics under `/metrics`, run as:
Expand Down Expand Up @@ -68,12 +73,12 @@ spec:
## Configuration
The configuration format is mostly lifted from the `ebpf_exporter`'s [configuration format](https://github.com/cloudflare/ebpf_exporter#configuration) with some changes
The configuration format is mostly lifted from the `ebpf_exporter`'s [configuration format](https://github.com/cloudflare/ebpf_exporter#configuration) with some changes.


### `program`

```
```yaml
# Program name
name: <program name>
# Metrics attached to the program
Expand Down Expand Up @@ -102,7 +107,7 @@ Note that, since this exporter does not deal with system-level metrics, `kprobes

### `attachments`

```
```yaml
attachments:
binary_name: [ binary_name ]
```
Expand All @@ -111,3 +116,70 @@ The `attachments` section details which processes the eBPF program will be attac
Currently, this only supports attaching by binary name -- all processes whose binary name equals the one given will be targeted.

*NOTE* This is the binary name as reported by `/proc/${PID}/comm`

### Examples

The following example will instrument garbage collection for all `gunicorn` processes:

```yaml
programs:
- name: gc_total
metrics:
counters:
- name: gc_total
help: Total number of gc events
table: gc_counts
labels:
- name: gen
size: 8
decoders:
- name: uint
usdt:
gc__start: trace_gc__start
attachment:
binary_name: "gunicorn"
code: |
struct gc_event_t {
u64 gen;
};
BPF_HASH(gc_counts, struct gc_event_t);
int trace_gc__start(struct pt_regs *ctx) {
struct gc_event_t e = {};
int gen = 0;
bpf_usdt_readarg(1, ctx, &gen);
e.gen = gen;
gc_counts.increment(e);
return 0;
}
```

Resulting metrics:

```
# HELP userspace_exporter_enabled_programs The set of enabled programs
# TYPE userspace_exporter_enabled_programs gauge
userspace_exporter_enabled_programs{name="gc_total",pid="29970"} 1
userspace_exporter_enabled_programs{name="gc_total",pid="29971"} 1
userspace_exporter_enabled_programs{name="gc_total",pid="29972"} 1
userspace_exporter_enabled_programs{name="gc_total",pid="29973"} 1
userspace_exporter_enabled_programs{name="gc_total",pid="29974"} 1
# HELP userspace_exporter_gc_total Total number of gc events
# TYPE userspace_exporter_gc_total counter
userspace_exporter_gc_total{gen="2",pid="29971"} 753
userspace_exporter_gc_total{gen="2",pid="29972"} 764
userspace_exporter_gc_total{gen="2",pid="29973"} 765
userspace_exporter_gc_total{gen="2",pid="29974"} 748
```

## Status

This is a hobby project; it should not be considered production ready.

It is missing a few features that I hope to implement over the coming months:

* The monitored process must be live before the exporter starts, and if it is restarted the exporter will not reattach. In future, it would be nice if it could dynamically attach to any live processes matching the `binary_name`.
* Attaching by binary name isn't very flexible; there are many different ways to find a process of interest -- by its parent process, by its command line, etc
* The original `ebpf-exporter` is able to add a [`tag`](https://github.com/cloudflare/ebpf_exporter#ebpf_exporter_ebpf_programs) label to its info metrics. This is challenging to do here since the USDT APIs don't easily lend themselves to getting a program's tag, but it would be good to at least add it for u(ret)probes.
* Some JVM examples would be fantastic

0 comments on commit efa441e

Please sign in to comment.