Segfault in Compiler::fgGetPredForBlock on MacOS x64

[jkoritzinsky]

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=921531
Build error leg or test failing: System.Net.Sockets.Tests.WorkItemExecution
Pull request: #111512

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "Segmentation fault: 11  \"$RUNTIME_PATH/dotnet\" exec --runtimeconfig System.Net.Sockets.Tests.runtimeconfig.json --depsfile System.Net.Sockets.Tests.deps.json xunit.console.dll System.Net.Sockets.Tests.dll",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=921531
Error message validated: [Segmentation fault: 11 "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Net.Sockets.Tests.runtimeconfig.json --depsfile System.Net.Sockets.Tests.deps.json xunit.console.dll System.Net.Sockets.Tests.dll]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 1/18/2025 1:11:29 AM UTC

Report

Build	Definition	Test	Pull Request
923111	dotnet/runtime	System.Net.Sockets.Tests.WorkItemExecution	#111398
922978	dotnet/runtime	System.Net.Sockets.Tests.WorkItemExecution	#110818
922813	dotnet/runtime	System.Net.Sockets.Tests.WorkItemExecution	#111398
922728	dotnet/runtime	System.Net.Sockets.Tests.WorkItemExecution	#111327
921531	dotnet/runtime	System.Net.Sockets.Tests.WorkItemExecution	#111512

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	5	5

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-infrastructure-libraries
See info in area-owners.md if you want to be subscribed.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

[jkotas]

Crash in the JIT on background compilation thread.

Stacktrace of the crash:

PROCCreateCrashDump(std::__1::vector<char const*, std::__1::allocator<char const*> >&, char*, int, bool)
PROCCreateCrashDumpIfEnabled
invoke_previous_action(sigaction*, int, __siginfo*, void*, bool)
_sigtramp
Compiler::fgGetPredForBlock(BasicBlock*, BasicBlock*)
Compiler::ThreeOptLayout::GetPartitionCostDelta(unsigned int, unsigned int, unsigned int, unsigned int, unsigned int)
Compiler::ThreeOptLayout::RunGreedyThreeOptPass(unsigned int, unsigned int)
Compiler::ThreeOptLayout::Run()
Compiler::fgSearchImprovedLayout()
ActionPhase<Compiler::compCompile(void**, unsigned int*, JitFlags*)::$_5>::DoPhase()
Phase::Run()
Compiler::compCompile(void**, unsigned int*, JitFlags*)
Compiler::compCompileHelper(CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*)
Compiler::compCompile(CORINFO_MODULE_STRUCT_*, void**, unsigned int*, JitFlags*)
jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*)
CILJit::compileMethod(ICorJitInfo*, CORINFO_METHOD_INFO*, unsigned int, unsigned char**, unsigned int*)
invokeCompileMethodHelper(EEJitManager*, CEEInfo*, CORINFO_METHOD_INFO*, CORJIT_FLAGS, unsigned char**, unsigned int*)
invokeCompileMethod(EEJitManager*, CEEInfo*, CORINFO_METHOD_INFO*, CORJIT_FLAGS, unsigned char**, unsigned int*)
UnsafeJitFunction(PrepareCodeConfig*, COR_ILMETHOD_DECODER*, CORJIT_FLAGS*, unsigned int*)
MethodDesc::JitCompileCodeLocked(PrepareCodeConfig*, COR_ILMETHOD_DECODER*, ListLockEntryBase<NativeCodeVersion>*, unsigned int*)
MethodDesc::JitCompileCodeLockedEventWrapper(PrepareCodeConfig*, ListLockEntryBase<NativeCodeVersion>*)
MethodDesc::JitCompileCode(PrepareCodeConfig*)
MethodDesc::PrepareILBasedCode(PrepareCodeConfig*)
TieredCompilationManager::CompileCodeVersion(NativeCodeVersion)
TieredCompilationManager::DoBackgroundWork(unsigned long long*, unsigned long long, unsigned long long)
TieredCompilationManager::BackgroundWorkerStart()
TieredCompilationManager::BackgroundWorkerBootstrapper1(void*)

Note that the crash dump was captured that you should be able to use to find the root cause.

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[EgorBo]

cc @amanasifkhalid ThreeOptLayout

[jakobbotsch]

I've been seeing what I think is the same crash in various superpmi-diffs/superpmi-replay jobs, e.g. https://dev.azure.com/dnceng-public/public/_build/results?buildId=923770&view=results. I am able to reproduce the crash but only with the JIT from CI and that one does not have symbols, so the debugging is a bit difficult. I see a crash in what looks like Compiler::ThreeOptLayout::GetPartitionCostDelta.

One thing I noticed is that nothing is checking that ebdTryLast is reachable/has an index:

runtime/src/coreclr/jit/fgopt.cpp

Line 5317 in abf8e94

modified |= RunThreeOptPass(tryBeg, HBtab->ebdTryLast);

[amanasifkhalid]

I've been seeing what I think is the same crash in various superpmi-diffs/superpmi-replay jobs, e.g. https://dev.azure.com/dnceng-public/public/_build/results?buildId=923770&view=results. I am able to reproduce the crash but only with the JIT from CI and that one does not have symbols, so the debugging is a bit difficult. I see a crash in what looks like Compiler::ThreeOptLayout::GetPartitionCostDelta.

Thanks for pointing this out. It seems odd that none of the Debug/Checked JITs are hitting asserts. Were you able to repro the failure with the Release win-x64 JIT from the build you linked? I'm unable to with SPMI replay...

One thing I noticed is that nothing is checking that ebdTryLast is reachable/has an index:

In the above loop where we assign indices, if a block is part of a try region, we update the try region's ebdTryLast pointer to that block. So if the try region's entry block is reachable/has an index, then the last try block we found in the hot region (ebdTryLast) must also be reachable/have an index. I added an assert for this locally, and it didn't catch anything, so I suspect this isn't an issue.