restarted. Azure DevOps can't recover from restarts.

[lewing]

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=784852

Build leg reported

Build / linux-x64 debug Libraries_AllConfigurations

Pull Request

dotnet/runtime#106599

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "restarted. Azure DevOps can't recover from restarts.",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=784852
Error message validated: [restarted. Azure DevOps can't recover from restarts.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 8/26/2024 7:12:18 PM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
865840	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#109678
865825	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#107231
863269	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109605
865692	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#107231
865622	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#109540
865527	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#109363
865462	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109137
865352	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109660
865148	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#107231
865134	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109631
865123	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading
865081	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#108559
865031	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109650
865000	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#104637
864900	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109647
860156	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading	Log	dotnet/runtime#109510
864690	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#109540
864644	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#109638
864595	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#108489
864584	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#108559
864579	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#109584
864418	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109422
864343	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109213
864340	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#109221
864325	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109602
864280	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109363
864134	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading
864085	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109529
863988	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109618
863922	dotnet/runtime	browser-wasm linux Release LibraryTests
863718	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109612
863459	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109610
863308	dotnet/runtime	browser-wasm linux Release LibraryTests
863228	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#109601
863166	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109598
863127	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109595
863110	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109528
862837	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109422
862721	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109507
862536	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109472
862461	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#108569
862458	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#108732
862043	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#104955
861991	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109528
861694	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109316
861633	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#109523
861579	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109564
861476	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109560
861108	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#109544
860961	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#108877
860854	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109137
860759	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109438
860740	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109137
860548	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#109298
860477	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109501
860394	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109391
860349	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109455
860177	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#107909
858594	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#109381
859947	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109477
859887	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109503
859609	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#109495
859580	dotnet/runtime	browser-wasm linux Release LibraryTests
859527	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#108731
859395	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading
859363	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#109482
859205	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#109482
859137	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109447
859117	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#109476
859030	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading
858983	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading	Log	dotnet/runtime#108469
858857	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109467
858559	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#108561
857755	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109422
857334	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109405
857288	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109376
857103	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109382
857032	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109391
856939	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#109232
856869	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109381
856782	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#104097
856777	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#104488
856685	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#108992
856598	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109357
856536	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109320
856272	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109320
856219	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#109361
856192	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#108999
856094	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#109361
856037	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109283
855899	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109316
855886	dotnet/runtime	browser-wasm linux Release LibraryTests
2571196	dotnet-dotnet	LinuxBionic_NativeAOT_Shortstack_x64
855550	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109259
855226	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#106615
854744	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#108965
854740	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#104694
854672	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#103950
854488	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#108879
854312	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading
Displaying 100 of 331 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
6	66	331

[am11]

It seems like all reports are pointing to linux-x64 dev-innerloop leg from this definition: https://dev.azure.com/dnceng-public/public/_build?definitionId=133. GitHub doesn't sync the status and keeps showing as if the job is running for days.. Opened dotnet/runtime#108581 to disable the leg.

[am11]

Just before the timeout we see low memory warnings like these:

https://dev.azure.com/dnceng-public/public/_build/results?buildId=829130&view=logs&j=e80acbf0-bc87-577c-4c46-0016b0794913&t=f0fa9d72-e49a-5249-4d28-1199014b9857

then it hangs for ~20 minutes or so before giving up. The build command has -allConfigurations so it builds all product+test assemblies for all platforms ({linux,win,osx,freebsd,illumos}-{x86,x64,arm,arm64,riscv64.. etc.}) all in one invocation of build (which isn't exactly efficient as we should probably group them..), which means, as it stands, this leg needs decent amount of RAM.

[steveisok]

@ilyas1974 @markwilkie I don't think our RAM consumption has increased that much to not be able to handle this configuration. Thoughts?

[ericstj]

I took a look at a few passing builds, many have the same logs where they approach 95% memory usage but eventually succeed.

Here are some samples:
https://dev.azure.com/dnceng-public/public/_build/results?buildId=836601&view=logs&j=e80acbf0-bc87-577c-4c46-0016b0794913&t=f0fa9d72-e49a-5249-4d28-1199014b9857
https://dev.azure.com/dnceng-public/public/_build/results?buildId=838110&view=logs&j=e80acbf0-bc87-577c-4c46-0016b0794913&t=f0fa9d72-e49a-5249-4d28-1199014b9857&l=3923
https://dev.azure.com/dnceng-public/public/_build/results?buildId=837745&view=logs&j=e80acbf0-bc87-577c-4c46-0016b0794913&t=f0fa9d72-e49a-5249-4d28-1199014b9857&l=4228

Near this point I see logs like this:

  initializing ChangeMakerService with capabilities: Baseline, AddMethodToExistingType, AddStaticFieldToExistingType, AddInstanceFieldToExistingType, NewTypeDefinition, ChangeCustomAttributes, UpdateParameters, GenericAddMethodToExistingType, GenericUpdateMethod, GenericAddFieldToExistingType
  baseline ready
  got a change
  parsing patch #1 from /__w/1/s/src/libraries/System.Runtime.Loader/tests/ApplyUpdate/System.Reflection.Metadata.ApplyUpdate.Test.GenericAddInstanceField/GenericAddInstanceField_v1.cs and creating delta
  Found changes in GenericAddInstanceField.cs
  change service made fa564b82-cf1c-4fb0-9d1a-f5ca4c71ff03
  wrote /__w/1/s/artifacts/bin/System.Reflection.Metadata.ApplyUpdate.Test.GenericAddInstanceField/Debug/net10.0/System.Reflection.Metadata.ApplyUpdate.Test.GenericAddInstanceField.dll.1.dmeta
  got a change
  parsing patch #2 from /__w/1/s/src/libraries/System.Runtime.Loader/tests/ApplyUpdate/System.Reflection.Metadata.ApplyUpdate.Test.GenericAddInstanceField/GenericAddInstanceField_v2.cs and creating delta
  Found changes in GenericAddInstanceField.cs
  change service made fa564b82-cf1c-4fb0-9d1a-f5ca4c71ff03
  wrote /__w/1/s/artifacts/bin/System.Reflection.Metadata.ApplyUpdate.Test.GenericAddInstanceField/Debug/net10.0/System.Reflection.Metadata.ApplyUpdate.Test.GenericAddInstanceField.dll.2.dmeta
  done

It looks to me like this is coming from https://github.com/dotnet/hotreload-utils/blob/254ec75de6127c368827d15c3af2477095b8b1b4/src/Microsoft.DotNet.HotReload.Utils.Generator/EnC/ChangeMakerService.cs#L28

Does anyone have an idea why hotreload would be running during a build?? I could imagine that if some hot reload service was runnign during a build or if tests were running while the product was building that could explain high memory usage.

[snakex64]

This seems to be happening consistently in PR #109320 . Should I just try re-running a third time or is it never gonna work?