Incorrect or partial flamegraph output on MacOS

[mabbamOG]

I have recently run cargo flamegraph on MacOS and Linux, and realised that the MacOS svg-output variant is missing a lot of important contextual information. Is there a way to solve this? The package seems quite unreliable or even broken on Macs....

[djc]

Are you running version 0.4? Earlier versions had some issues. The information provided is necessarily different on macOS, where we use dtrace compared to Linux, where we use perf. I'm not sure there's much about that we can do inside the flamegraph codebase, although maybe we can pass options to dtrace to make its output more comprehensive.

[mabbamOG]

Are you running version 0.4? Earlier versions had some issues.

$ cargo flamegraph --version
cargo-flamegraph-flamegraph 0.4.0

The information provided is necessarily different on macOS

The MacOS version is definitely missing a lot of information, and sometimes it is even misleading because some of the program's internal functions are replaced with the names of "more internal" (read: deeper into the stack trace) functions; furthermore, the stack trace on MacOS is much more shallow than the one from Linux, it looks more like a "streetgraph" than a "flamegraph" because so many internal calls are missing.

The situation is such that i cannot rely on the MacOS flamegraph, and without the linux version i would not have been able to accurately profile and optimise my program. I suspect the Mac version has optimised a lot of debug info out of the binary, so perhaps a few dtrace options can do the trick here indeed.

I can share the flamegraphs if need be, but first i would need to change all the function names, as well as those of some internal libraries...

[djc]

I understand that part. It's just that this project is merely a convenient abstraction over running perf or dtrace (as appropriate for the host platform) and generating flamegraphs from it. As far as I know we run a pretty minimal perf or dtrace command, so while I'd be okay with extending that if it substantially improves the results, it's not really within my expertise, and it's also somewhat at the scope of what this project is about IMO (but maybe @spacejam thinks differently).

[shepmaster]

As a concrete example, the same flamegraph generated on

Linux

macOS

[djc]

Yup, I've also seen similar results in my own projects. Still not sure what we could do to tune dtrace to do better.

[shepmaster]

I hacked things up to use profile-4999 instead of profile-997, but that did not measurably change the output.

[Swatinem]

I suspect the issue is missing resolution of inline frames. I ran into this yesterday, comparing the output of cargo flamegraph with whatever an Instruments profile gives me. It really looked like inlined frames were completely missing from the flamegraph output.

[mitsuhiko]

It's definitely inline frames that are missing but it makes me wonder how inline frames are actually working on other platforms. The output from dtrace's ustack() do not appear to produce inline frames for me but depending on what I am targeting I get just hex addresses or symbolized (not not with inlines resolved) frames. I assume that on other platforms dtrace actually resolves inline addresses. The code indicates that -> would be used to express inline frames, yet I can only find them for the java samples in the repo.

I'm guessing it's dtrace itself that is supposed to produce that, so I wonder if what dtrace uses for symbol resolution fails now. I tried to use other values for split-debuginfo but changing that didn't improve anything.

[hanayashiki]

how is this going? it doesn't help much with such flamegraph:

[djc]

@hanayashiki it looks like there has been some progress in jonhoo/inferno#246, but not much.