Unclear buildbot failure email on mlir-nvidia-gcc7

[jayfoad]

I got this email:

The Buildbot has detected a new failure on builder mlir-nvidia-gcc7 while building llvm.

Full details are available at:
    https://lab.llvm.org/buildbot/#/builders/116/builds/522

Worker for this Build: mlir-nvidia
Blamelist:
    CarolineConcatto <[email protected]>,
    Jay Foad <[email protected]>

BUILD FAILED: build (failure)

Step 5 (build-check-mlir-build-only) failure: build (failure)
...
617.256 [102/16/4259] Building Ops.cpp.inc...
617.284 [101/16/4260] Building Ops.h.inc...
617.307 [100/16/4261] Building ToyCombine.inc...
617.325 [99/16/4262] Building ShapeInferenceOpInterfaces.h.inc...
617.342 [98/16/4263] Building ShapeInferenceOpInterfaces.cpp.inc...
617.360 [97/16/4264] Building CXX object tools/mlir/examples/toy/Ch7/CMakeFiles/toyc-ch7.dir/parser/AST.cpp.o
617.395 [96/16/4265] Building CXX object tools/mlir/examples/toy/Ch7/CMakeFiles/toyc-ch7.dir/mlir/LowerToAffineLoops.cpp.o
617.423 [95/16/4266] Building CXX object tools/mlir/examples/toy/Ch7/CMakeFiles/toyc-ch7.dir/mlir/ShapeInferencePass.cpp.o
617.452 [94/16/4267] Building CXX object tools/mlir/examples/toy/Ch7/CMakeFiles/toyc-ch7.dir/mlir/ToyCombine.cpp.o
617.482 [93/16/4268] Building CXX object tools/mlir/examples/toy/Ch7/CMakeFiles/toyc-ch7.dir/mlir/MLIRGen.cpp.o
618.371 [92/16/4269] Building CXX object tools/mlir/lib/CAPI/Target/CMakeFiles/obj.MLIRCAPITarget.dir/LLVMIR.cpp.o
618.400 [91/16/4270] Building CXX object tools/mlir/examples/toy/Ch7/CMakeFiles/toyc-ch7.dir/mlir/Dialect.cpp.o
618.444 [90/16/4271] Building CXX object tools/mlir/examples/toy/Ch7/CMakeFiles/toyc-ch7.dir/mlir/LowerToLLVM.cpp.o
618.464 [89/16/4272] Building MyExtension.cpp.inc...
618.483 [88/16/4273] Building MyExtensionTypes.h.inc...
618.837 [87/16/4274] Building CXX object tools/mlir/lib/Target/LLVMIR/Dialect/OpenMP/CMakeFiles/obj.MLIROpenMPToLLVMIRTranslation.dir/OpenMPToLLVMIRTranslation.cpp.o
618.861 [86/16/4275] Linking CXX static library lib/libMLIROpenMPToLLVMIRTranslation.a
618.878 [85/16/4276] Linking CXX static library lib/libMLIRExecutionEngine.a
618.892 [84/16/4277] Linking CXX static library lib/libMLIRToLLVMIRTranslationRegistration.a
618.904 [83/16/4278] Linking CXX static library lib/libMLIRCAPITarget.a
618.917 [82/16/4279] Linking CXX static library lib/libMLIRCAPIExecutionEngine.a
618.952 [81/16/4280] Building CXX object tools/mlir/examples/transform/Ch2/lib/CMakeFiles/MyExtensionCh2.dir/MyExtension.cpp.o
618.970 [80/16/4281] Linking CXX static library lib/libMyExtensionCh2.a
618.993 [79/16/4282] Building MyExtension.cpp.inc...
619.013 [78/16/4283] Building MyExtensionTypes.cpp.inc...
619.039 [77/16/4284] Building CXX object tools/mlir/examples/minimal-opt/CMakeFiles/mlir-minimal-opt.dir/mlir-minimal-opt.cpp.o
622.815 [76/16/4285] Building CXX object tools/mlir/lib/Target/LLVM/CMakeFiles/obj.MLIRROCDLTarget.dir/ROCDL/Target.cpp.o
622.830 [75/16/4286] Linking CXX static library lib/libMLIRROCDLTarget.a
622.875 [74/16/4287] Linking CXX static library lib/libMLIRGPUTransforms.a
622.909 [73/16/4288] Linking CXX static library lib/libMLIRGPUToGPURuntimeTransforms.a
622.943 [72/16/4289] Linking CXX static library lib/libMLIRGPUToNVVMTransforms.a
622.974 [71/16/4290] Linking CXX static library lib/libMLIRGPUToROCDLTransforms.a
622.999 [70/16/4291] Linking CXX static library lib/libMLIRNVGPUToNVVM.a
623.023 [69/16/4292] Linking CXX static library lib/libMLIRGPUPipelines.a
623.040 [68/16/4293] Linking CXX static library lib/libMLIRSCFToGPU.a
623.073 [67/16/4294] Linking CXX static library lib/libMLIRGPUTransformOps.a
623.091 [66/16/4295] Linking CXX static library lib/libMLIRCAPIConversion.a
623.119 [65/16/4296] Linking CXX static library lib/libMLIRNVGPUTransformOps.a
623.136 [64/16/4297] Linking CXX static library lib/libMLIRSparseTensorPipelines.a
623.151 [63/16/4298] Linking CXX static library lib/libMLIRJitRunner.a
626.038 [62/16/4299] Linking CXX executable bin/mlir-cpu-runner
626.053 [61/16/4300] Linking CXX static library lib/libMLIRGPUTestPasses.a
626.078 [60/16/4301] Linking CXX static library lib/libMLIRLinalgTestPasses.a
628.241 [59/16/4302] Linking CXX executable bin/mlir-vulkan-runner
640.024 [58/16/4303] Building CXX object tools/mlir/test/lib/Dialect/Test/CMakeFiles/MLIRTestToLLVMIRTranslation.dir/TestToLLVMIRTranslation.cpp.o
640.039 [57/16/4304] Linking CXX static library lib/libMLIRTestToLLVMIRTranslation.a
642.105 [56/16/4305] Linking CXX executable bin/mlir-translate
642.140 [55/16/4306] Building CXX object tools/mlir/examples/transform/Ch3/lib/CMakeFiles/MyExtensionCh3.dir/MyExtension.cpp.o
642.214 [54/16/4307] Linking CXX static library lib/libMyExtensionCh3.a
642.235 [53/16/4308] Building MyExtension.cpp.inc...

Sincerely,
LLVM Buildbot

The build log excerpt does not give me any clue how the build failed.

[jayfoad]

• @gkistanova @slydiman @CarolineConcatto

[joker-eph]

This is a timeout, it's visible when you click on the link:

642.235 [53/16/4308] Building MyExtension.cpp.inc...
command timed out: 1200 seconds without output running [b'ninja', b'-j', b'16', b'check-mlir-build-only'], attempting to kill
process killed by signal 9
program finished with exit code -1
elapsedTime=3038.220567

Somehow the log extraction logic does not get this information apparently.

[gkistanova]

Thanks for reporting this, Jay!

As Mehdi mentioned, this is a timeout at the build step. And timeout message is not a part of the build log.

We will see what we can do about this.

[gkistanova]

This is fixed by a6a4177.

Please feel free to reopen if you still see the issue.