[BUG]: PagingLogger raises DirectoryNotFoundException in Docker Container CE/Moby which causes agent to crash

[tradem]

What happened?

We are running a Windows Azure Pipeline inside a Docker Container based on mcr.microsoft.com/windows/server:ltsc2022 .

Host System: Windows Server 2022
Docker Engine: Docker CE/Moby
Agent Version: 3.238.0

Longer running pipelines such as CI oder Nightly pipelines we suddenly encounter this exception which caused the agent process to crash inside our container:

   - C:\azp\agent\_diag\Worker_20250221-054652-utc.log
System.IO.DirectoryNotFoundException: Could not find a part of the path 'C:\azp\agent\_diag\pages'.
   at System.IO.FileSystem.CreateDirectory(String fullPath, Byte[] securityDescriptor)
   at System.IO.Directory.CreateDirectory(String path)
   at Microsoft.VisualStudio.Services.Agent.PagingLogger.Initialize(IHostContext hostContext)
   at Microsoft.VisualStudio.Services.Agent.HostContext.CreateService[T]()
   at Microsoft.VisualStudio.Services.Agent.Worker.ExecutionContext.CreateChild(Guid recordId, String displayName, String refName, Variables taskVariables, Boolean outputForward, List`1 taskRestrictions) in D:\a\_work\1\s\src\Agent.Worker\ExecutionContext.cs:line 255
   at Microsoft.VisualStudio.Services.Agent.Worker.JobExtension.FinalizeJob(IExecutionContext jobContext) in D:\a\_work\1\s\src\Agent.Worker\JobExtension.cs:line 524
   at Microsoft.VisualStudio.Services.Agent.Worker.JobRunner.RunAsync(AgentJobRequestMessage message, CancellationToken jobRequestCancellationToken) in D:\a\_work\1\s\src\Agent.Worker\JobRunner.cs:line 349
   at Microsoft.VisualStudio.Services.Agent.Worker.JobRunner.RunAsync(AgentJobRequestMessage message, CancellationToken jobRequestCancellationToken) in D:\a\_work\1\s\src\Agent.Worker\JobRunner.cs:line 396
   at Microsoft.VisualStudio.Services.Agent.Worker.Worker.RunAsync(String pipeIn, String pipeOut) in D:\a\_work\1\s\src\Agent.Worker\Worker.cs:line 97
   at Microsoft.VisualStudio.Services.Agent.Worker.Program.MainAsync(IHostContext context, String[] args) in D:\a\_work\1\s\src\Agent.Worker\Program.cs:line 52
Pool: MobyDick
Agent: Agent Gordon Cole

The agent works smoothly (Agent Logging, Logging)

Any ideas what could cause such behevoir? Any possible workarounds?

We tried to externalize the _diag directory via docker volumes in our docker-compose.yml:

services:
  azp-agent-windows:
    image: azp-agent:windows
    restart: none
    env_file:
      - apz-agent.env
    volumes:
      - "D:\\azp-agent-in-docker\\Nuget:C:\\Users\\ContainerAdministrator\\AppData\\Roaming\\NuGet\\:rw"
      - "D:\\azp-agent-in-docker\\agents\\MobyDick\\gordon-cole\\_work:C:\\azp\\agent\\_work\\:rw"
      - "D:\\azp-agent-in-docker\\agents\\MobyDick\\gordon-cole\\_diag:C:\\azp\\agent\\_diag\\:rw"
    deploy:
      resources:
        limits:
          cpus: '2'
          memory: 12gb
        reservations:
          cpus: '0.5'    

Versions

Azure Devops Agent 3.238.0 / Windows Server 2022 ( mcr.microsoft.com/windows/server:ltsc2022 )

Environment type (Please select at least one enviroment where you face this issue)

• Self-Hosted
• Microsoft Hosted
• VMSS Pool
• Container

Azure DevOps Server type

Azure DevOps Server (Please specify exact version in the textbox below)

Azure DevOps Server Version (if applicable)

Azure Devops Server 2022.2 (AzureDevopsServer_20241028.1)

Operation system

Windows Server 2022

Version controll system

git

Relevant log output

[2025-02-21 06:49:31Z INFO JobDispatcher] finish job request for job e91bdf7b-25c9-5273-ccfa-b8a9ff30d438 with result: Failed
[2025-02-21 06:49:31Z INFO Terminal] WRITE LINE: 2025-02-21 06:49:31Z: Der Auftrag „Nightly Pipeline“ wurde mit folgendem Ergebnis abgeschlossen: Failed
[2025-02-21 06:49:31Z INFO JobDispatcher] Stop renew job request for job e91bdf7b-25c9-5273-ccfa-b8a9ff30d438.
[2025-02-21 06:49:31Z INFO JobDispatcher] job renew has been canceled, stop renew job request 42213.
[2025-02-21 06:49:31Z ERR  JobDispatcher] Unhandled exception happened in worker:
[2025-02-21 06:49:31Z ERR  JobDispatcher] Error reported in diagnostic logs. Please examine the log for more details.
    - C:\azp\agent\_diag\Worker_20250221-054715-utc.log
System.IO.DirectoryNotFoundException: Could not find a part of the path 'C:\azp\agent\_diag\pages'.
   at System.IO.FileSystem.CreateDirectory(String fullPath, Byte[] securityDescriptor)
   at System.IO.Directory.CreateDirectory(String path)
   at Microsoft.VisualStudio.Services.Agent.PagingLogger.Initialize(IHostContext hostContext)
   at Microsoft.VisualStudio.Services.Agent.HostContext.CreateService[T]()
   at Microsoft.VisualStudio.Services.Agent.Worker.ExecutionContext.CreateChild(Guid recordId, String displayName, String refName, Variables taskVariables, Boolean outputForward, List`1 taskRestrictions) in D:\a\_work\1\s\src\Agent.Worker\ExecutionContext.cs:line 255
   at Microsoft.VisualStudio.Services.Agent.Worker.JobExtension.FinalizeJob(IExecutionContext jobContext) in D:\a\_work\1\s\src\Agent.Worker\JobExtension.cs:line 524
   at Microsoft.VisualStudio.Services.Agent.Worker.JobRunner.RunAsync(AgentJobRequestMessage message, CancellationToken jobRequestCancellationToken) in D:\a\_work\1\s\src\Agent.Worker\JobRunner.cs:line 349
   at Microsoft.VisualStudio.Services.Agent.Worker.JobRunner.RunAsync(AgentJobRequestMessage message, CancellationToken jobRequestCancellationToken) in D:\a\_work\1\s\src\Agent.Worker\JobRunner.cs:line 396
   at Microsoft.VisualStudio.Services.Agent.Worker.Worker.RunAsync(String pipeIn, String pipeOut) in D:\a\_work\1\s\src\Agent.Worker\Worker.cs:line 97
   at Microsoft.VisualStudio.Services.Agent.Worker.Program.MainAsync(IHostContext context, String[] args) in D:\a\_work\1\s\src\Agent.Worker\Program.cs:line 52
[2025-02-21 06:49:31Z INFO JobNotification] Entering JobCompleted Notification
[2025-02-21 06:49:31Z INFO JobNotification] Entering EndMonitor
[2025-02-21 06:49:31Z INFO JobDispatcher] Fire signal for one time used agent.
[2025-02-21 06:49:31Z INFO Agent] Job has finished at backend, the agent will exit since it is running under onetime use mode.
[2025-02-21 06:49:31Z INFO Agent] Stop message queue looping.
[2025-02-21 06:49:31Z WARN VisualStudioServices] GET request to https://devops.evy-solutions.de/_apis/distributedtask/pools/5/messages has been cancelled.
[2025-02-21 06:49:31Z INFO MessageListener] Get next message has been cancelled.
[2025-02-21 06:49:31Z INFO Agent] Ignore any exception after cancel message loop. System.Threading.Tasks.TaskCanceledException: The operation was canceled.
 ---> System.IO.IOException: Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request..
 ---> System.Net.Sockets.SocketException (995): The I/O operation has been aborted because of either a thread exit or an application request.
   --- End of inner exception stack trace ---
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
   at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](TIOAdapter adapter, Memory`1 buffer)
   at System.Net.Http.HttpConnection.<CheckUsabilityOnScavenge>g__ReadAheadWithZeroByteReadAsync|45_0()
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpRequestMessage message, Object userState, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpMethod method, IEnumerable`1 additionalHeaders, Guid locationId, Object routeValues, ApiResourceVersion version, HttpContent content, IEnumerable`1 queryParameters, Object userState, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Agent.Listener.MessageListener.GetNextMessageAsync(CancellationToken token)
   at Microsoft.VisualStudio.Services.Agent.Listener.Agent.RunAsync(AgentSettings settings, Boolean runOnce)
[2025-02-21 06:49:31Z INFO JobDispatcher] Shutting down JobDispatcher. Make sure all WorkerDispatcher has finished.
[2025-02-21 06:49:31Z INFO JobDispatcher] Ensure WorkerDispather for job e91bdf7b-25c9-5273-ccfa-b8a9ff30d438 run to finish, cancel any running job.
[2025-02-21 06:49:31Z INFO JobDispatcher] Job request e91bdf7b-25c9-5273-ccfa-b8a9ff30d438 processed succeed.