Switch nativeaot7.0 microbenchmarks to 8.0 #2919
Conversation
cincuranet
requested review from
sblom,
LoopedBard3,
DrewScoggins and
caaavik-msft
|
@MichalStrehovsky can you advise on the errors in NativeAOT micro benchmarks on This seems to be really not nice. And maybe on related note. Does this need to run on CentOS 7? |
These are all: Exit code 137 is very likely the OOM killer. How much memory do the containers running this have? What is the effective change this PR is doing? Is this switching from building the tests with .NET 7 to .NET 8 daily builds? Is there a quick way for me to run this locally? We should be using less memory, not more, but it's possible there's something pathological in the new framework libraries, for example. |
These are Azure VMs. Standard_D2a_v4, aka 2 CPU, 8 GB RAM.
Correct.
|
|
I was hoping for a way to run the failing build command by itself. Do I understand it correctly that this is AOT-compiling all the tests in the repo into a single assembly? Is there a way to add extra things into the generated project that is failing the build? I'd like to add: <ItemGoup>
<IlcArg Include="--verbose" />
</ItemGroup>With that we'd at least know when this is getting killed. Pretty sure this is a OOM situation. I don't know how to repro locally, cc @dotnet/ilc-contrib |
|
You can see the commands BDN uses starting around Not sure about the |
That command doesn't help because it's for a generated project with a generated C# in it. I don't know what's in it.
@adamsitnik do you know if there's a way to inject extra things into the csproj that BDN generates? It looks like the project build is getting OOM-killed. I'd like to at least know when this is happening. It could be narrowed down with the above (#2919 (comment)) snippet but I don't see a way to inject it into the generated CSPROJ. I have a suspicion where this could be happening (object writing) and would love to be able to confirm because it affects product direction - we had seen OOM during object writing in the past. Right now I think we'll need to run this on a machine with more RAM. The project seems to be rather large. |
|
No, but it's possible to pass any MsBuild property as an argument to job = job.WithArguments(new Argument[]
{
new MsBuildArgument("/p:DebugType=portable") // See https://github.com/dotnet/roslyn/issues/42393
});It can be done here: |
|
Thanks! Opened dotnet/runtime#83024 to expose this as a property. I'm not aware of a way to pass Items like this. |
|
I see the |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Looks like this is still 8.0.0-preview.2.23127.4 (i.e. February 27th). The runtime -> SDK -> installer flow has been pretty slow lately. |
|
Yeah. All that's left to do is wait. |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
It does look like this is running out of memory during object writing. The log indicates we sent all the data to the LLVM side and the process gets OOM-killed in somewhere around |
|
Merged dotnet/runtime#83181 that might help. We'll need to wait until that propagates to the sdk+installer repo. |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
Merged dotnet/runtime#83249 that might help. |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
Some interesting info here: I managed to get a repro that runs on both Linux and Windows, and on Linux it consumes a lot more memory. |
|
OK, it looks like the bulk of the allocations are in the object writer, as Michal suspected. I have some basic stack trace information, and one thing that jumps out are these lines:
That looks to be this code here: void DwarfMemberFunctionIdTypeInfo::DumpStrings(MCObjectStreamer *Streamer) {
Streamer->emitBytes(StringRef(Name));
Streamer->emitIntValue(0, 1);
MCContext &context = Streamer->getContext();
LinkageNameSymbol = context.createTempSymbol();
Streamer->emitLabel(LinkageNameSymbol);
Streamer->emitBytes(StringRef(LinkageName));
Streamer->emitIntValue(0, 1);
}My understanding of this code is that we're pulling the current section and emitting the function name and its mangled name into it. But it's notable that I think pretty much all the function names are in the same section. So we're basically pushing 120 MB of allocations through LLVM's "SmallVectorImpl". I don't know how that's implemented, but it raises a red flag for me that we might not be using this API correctly. |
Hmm, the more I dig, the more I think the API we're not using correctly is LLVM itself, namely the assembly-level API. I'm not sure this API was supposed to be used for the scale we're throwing at it. |
|
Ah, you know what, I no longer think 8 GB of memory is unreasonable for this compilation. The output image is 448 MB. I need to look at the inputs here before trying to optimize ILC. |
|
Agreed, 448 MB object file is a pretty large application. After linking/stripping, it will likely be more in the 100-200 MB range, but it's still pretty large. IIRC Benchmark.NET forces partial trimming because it's not compatible with full trimming. It likely contributes to the problem. We had problems with this in the past (dotnet/BenchmarkDotNet#2046). That said, I retriggered the CI leg after dotnet/runtime#83447 and looks like now it succeeded. The Blazor leg is still failing, but looks unrelated. 🍾 |
|
It looks like the microbenchmarks are already getting trimmed with TrimMode=full. So it's probably just that the microbenchmarks are pulling in basically the whole framework. No easy fix for that. The test machine might just eventually need > 8 GB of RAM. I don't think it's unreasonable to take 16 GB of RAM to compile a 500 MB exe. Either that, or split these tests into two assemblies. |
Splitting the microbenchmarks into multiple assemblies is the only option that will work long term. One huge binary with all microbenchmarks will keep running into limits. It is not possible to have >2GB binary on Windows due to file format and OS limitations. Once we hit the 2GB limit, no amount of extra RAM on the build machine is going to help. |
In a sense, we are already doing that. The CI by default uses performance/eng/performance/helix.proj Lines 25 to 34 in 7eddd9d This is configurable via command line: --partition-count $PartitionCount --partition-index $PartitionCount |
This should eventually unblock #2905.