slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes." #1883

jeffschwMSFT · 2024-01-24T15:45:11Z

Build

https://dnceng.visualstudio.com/internal/_build/results?buildId=2360768&view=results

Error

##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134

Build leg reported

vsos

Pull Request

No response

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "",
    "BuildRetry": false,
    "ErrorPattern": "The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

Feature changes/additions
Bug fixes
Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng/internal/_build/results?buildId=2360768
Error message validated: [The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/7/2024 1:03:13 AM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
937171	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112037
2632890	dotnet-performance	ubuntu 2204 arm64 mlnet Ampere 9.0	Log
937121	dotnet/runtime	osx-arm64 Release NativeAOT	Log	dotnet/runtime#112061
937038	dotnet/runtime	osx-arm64 Release NativeAOT	Log	dotnet/runtime#111859
937032	dotnet/runtime	osx-arm64 Release NativeAOT	Log	dotnet/runtime#112060
937031	dotnet/runtime	osx-arm64 Release NativeAOT	Log
936996	dotnet/runtime	osx-arm64 Release NativeAOT	Log	dotnet/runtime#111979
936630	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#45936
936965	dotnet/runtime	osx-arm64 Release NativeAOT	Log	dotnet/runtime#111518
936878	dotnet/runtime	osx-arm64 Release NativeAOT	Log	dotnet/runtime#111819
936716	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112032
936861	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112058
936846	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46456
936833	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#107769
936642	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46370
2632596	dotnet-arcade-validation	MacOS Build_Release	Log
936734	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#111771
936780	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112050
936729	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#111979
936584	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46337
934380	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46409
936762	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112057
936703	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#112010
936750	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#111934
936747	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112056
936681	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#112011
936743	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46353
936741	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46455
936710	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46453
935673	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46424
936443	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46442
936586	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46449
936293	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46441
936539	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46450
935525	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46408
935124	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46387
935622	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46266
936329	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46361
935782	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46439
935580	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46434
936322	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46447
936451	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46353
936439	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#44843
936429	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46281
936427	dotnet/sdk	OSX_arm64	Log	dotnet/sdk#46281
935558	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46432
936387	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46445
936307	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46446
936300	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46290
936297	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46370
936179	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46444
936288	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46254
936258	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46445
935366	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46429
936206	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46442
936185	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46233
936159	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46371
936153	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46281
936151	dotnet/sdk	OSX_arm64	Log	dotnet/sdk#46281
936140	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#44953
936125	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46327
935929	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46353
936054	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46443
935565	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46433
935902	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46441
935914	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46392
935911	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46392
935795	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46195
935899	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#45520
935722	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46370
935828	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46261
935805	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46358
935802	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46440
935800	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46417
935794	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46414
935787	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46356
935780	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46438
935773	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46437
935752	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46436
935738	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46361
935687	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46353
935626	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#45936
935620	dotnet/runtime	osx-x64 Debug CoreCLR_Libraries	Log	dotnet/runtime#112024
935609	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46254
935118	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46313
935378	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46430
2631877	dotnet-performance	ubuntu 2204 arm64 scenarios Ampere 9.0	Log
935552	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46261
935549	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46428
934910	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46281
935157	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46426
934456	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46418
935016	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46371
934464	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46358
935373	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#44843
935354	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46412
935344	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46427
935336	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46336
935357	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46411
935245	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46377
Displaying 100 of 1416 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
34	461	1416

The text was updated successfully, but these errors were encountered:

lewing · 2024-02-09T01:11:11Z

🤔

nagilson · 2024-10-03T16:50:21Z

@dotnet/dnceng @dougbu This has impacted a lot of PRs recently, may you PTAL into expanding this Mac resource?

ivanpovazan · 2025-01-27T11:29:19Z

@dotnet/dnceng we are hitting this again and it seems there is some issue with communication with Helix machines.

More context on timeouts happening in : https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=results

On success: Run tests in Helix step reports:

Waiting for completion of job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open (Details: https://helix.dot.net/api/jobs/fbec1164-55c5-4fa8-b492-e1ba1b413119/details?api-version=2019-06-17 )
Job 7fdc2720-3d98-4bf3-8113-274dacd69c91 on osx.1200.arm64.open is completed with 6 finished work items.
  Job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open is completed with 6 finished work items.
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.amd64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.arm64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )

Build succeeded.

SENDHELIXJOB : warning : Helix queue osx.1200.amd64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
SENDHELIXJOB : warning : Helix queue osx.1200.arm64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
    2 Warning(s)
    0 Error(s)

Time Elapsed 00:03:14.97
Killing running build processes...

Finishing: Run tests in Helix

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=logs&j=ccc97bb6-1a23-5e71-fdfa-3cdca4a74749&t=27fc7eb2-ead9-59e1-6679-a637855d40c5

While on failure - timeout the same step gets stuck with:

Waiting for completion of job 967d92a2-ec10-4332-927f-d28a6563f367 on osx.1200.arm64.open (Details: https://helix.dot.net/api/jobs/967d92a2-ec10-4332-927f-d28a6563f367/details?api-version=2019-06-17 )
  Job 5427159b-500d-49f1-aac0-ec148a492bbe on osx.1200.amd64.open is completed with 6 finished work items.

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=logs&s=c58bc33c-b825-5bca-90ca-50f6e9293dd8&j=e6966639-fe40-5068-d9ae-681cccecafdf

NOTE: All the tests successfully passed on Helix, but it seems that the communication is lost.

garath · 2025-01-27T20:24:33Z

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

garath · 2025-01-28T22:56:54Z

Ah, the queue was consumed with updates and patching. The patching jobs did run longer than necessary and we've communicated with our partner team about the issue. Future jobs will be much shorter and should not overly impact jobs.

jeffschwMSFT added First Responder Known Build Error labels Jan 24, 2024

lewing mentioned this issue Feb 8, 2024

"We stopped hearing from agent Azure Pipelines 32. Verify the agent machine is running and has a healthy network connection" #1886

Open

3 tasks

AaronRobinsonMSFT mentioned this issue Dec 10, 2024

Fix test under JIT stress dotnet/runtime#110538

Merged

garath self-assigned this Jan 27, 2025

garath removed their assignment Jan 29, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes." #1883

slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes." #1883

jeffschwMSFT commented Jan 24, 2024 •

edited by build-analysis bot

Loading

lewing commented Feb 9, 2024

nagilson commented Oct 3, 2024

ivanpovazan commented Jan 27, 2025

garath commented Jan 27, 2025

garath commented Jan 28, 2025

slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes." #1883

slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes." #1883

Comments

jeffschwMSFT commented Jan 24, 2024 • edited by build-analysis bot Loading

Build

Build leg reported

Pull Request

Known issue core information

Release Note Category

Release Note Description

Additional information about the issue reported

Known issue validation

Report

Summary

lewing commented Feb 9, 2024

nagilson commented Oct 3, 2024

ivanpovazan commented Jan 27, 2025

garath commented Jan 27, 2025

garath commented Jan 28, 2025

jeffschwMSFT commented Jan 24, 2024 •

edited by build-analysis bot

Loading