Azure-pipelines-agent: Getting the System.IO.IOException: Input/output error after some times of Agent starts

[Sukumarnayak]

Running the Azure Agent RHEL6 version 2.155.1
https://vstsagentpackage.azureedge.net/agent/2.155.1/vsts-agent-rhel.6-x64-2.155.1.tar.gz

Running the agent service in Interactive Mode. The agent service runs fine, but after sometimes the jobs fails and gives the below error.
[2019-08-08 10:24:11Z ERR JobDispatcher] Worker Dispatch failed witn an exception for job request 0dbeb697-20ba-5521-a7b2-c923cc00a23c.
[2019-08-08 10:24:11Z ERR JobDispatcher] System.IO.IOException: Input/output error
at System.ConsolePal.Write(SafeFileHandle fd, Byte* bufPtr, Int32 count)
at System.ConsolePal.Write(SafeFileHandle fd, Byte[] buffer, Int32 offset, Int32 count)
at System.ConsolePal.UnixConsoleStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
at System.IO.StreamWriter.WriteLine(String value)
at System.IO.SyncTextWriter.WriteLine(String value)
at System.Console.WriteLine(String value)
at Microsoft.VisualStudio.Services.Agent.Listener.JobDispatcher.RunAsync(AgentJobRequestMessage message, WorkerDispatcher previousJobDispatch, CancellationToken jobRequestCancellationToken, CancellationToken workerCancelTimeoutKillToken)
at Microsoft.VisualStudio.Services.Agent.Listener.JobDispatcher.RunAsync(AgentJobRequestMessage message, WorkerDispatcher previousJobDispatch, CancellationToken jobRequestCancellationToken, CancellationToken workerCancelTimeoutKillToken)
at Microsoft.VisualStudio.Services.Agent.Listener.JobDispatcher.EnsureDispatchFinished(WorkerDispatcher jobDispatch, Boolean cancelRunningJob)

Getting the below error in Azure Devops side when build starts

[juliobbv]

This is a shot in the dark, but could it be that you're running out of disk space? Run df and look for partitions that are close to/at their capacity.

[Sukumarnayak]

There is no out of disk space in the machine where the Agent is running. As i mentioned the agent was running as Interactive mode in background mode like ./run.sh &
I have just started the Agent as run as service and i do not see the issue is coming.

[jtpetty]

@Sukumarnayak - Are you still experiencing this issue?

[jd-mtb]

I am getting the same error with the rhel6 agent running "as a service" (but non-native, basically running ./run.sh &) using 2.170.1 so far. Support / help would be great.

[anatolybolshakov]

@Sukumarnayak is this issue still actual for you?
@jd-mtb could you please share logs also?

[jd-mtb]

@anatolybolshakov Thanks for reminding me on this issue and while the issue persists how I mention it I was actually using it incorrectly and others might be as well. _I discovered the issue is that I was supposed to be using the includes runsvc.sh script instead._ If you just use run.sh it doesn't start the node server in a way that keeps it open for a long time. I was tipped off by a comment somewhere in one of the included microsoft files.

[anatolybolshakov]

@jd-mtb ok thanks! Yes, you can use this to run agent as systemd service (here are some steps describing it)

[jd-mtb]

@anatolybolshakov on rhel6 the service install command just tells you it isn't supported or something like that. I forget exactly, but point is it doesn't work. https://github.com/dotnet/core/blob/master/Documentation/build-and-install-rhel6-prerequisites.md

That said, we did get it working, our init file looks like this... disclaimer: I'm new to rhel6 init files, inherited most of this as is, provide it as is, and there's no guarantee of form or function:

#!/bin/bash
#
# chkconfig: 35 90 12
# description: VSTS-Agent
#

# Get function from functions library
. /etc/init.d/functions

# Start the service VSTS
start() {
        echo -n "Starting VSTS-Agent service: "
        runuser -l wasadmin bash -c 'export LD_LIBRARY_PATH=/usr/local/lib && /opt/vsts-agent/runsvc.sh --startuptype service &'
        ### Create the lock file ###
        touch /var/lock/subsys/vsts
        success $"VSTS-Agent service startup"
        echo
}

# Restart the service VSTS-Agent
stop() {
        echo -n "Stopping VSTS-Agent service: "
        killproc Agent.Listener
        ### Now, delete the lock file ###
        rm -f /var/lock/subsys/vsts
        echo
}

### main logic ###
case "$1" in
  start)
        start
        ;;
  stop)
        stop
        ;;
  status)
        echo "This isn't a native SysV sevice, and has no status"
        ;;
  restart|reload|condrestart)
        stop
        start
        ;;
  *)
        echo $"Usage: $0 {start|stop|restart|reload|status}"
        exit 1
esac

exit 0

[anatolybolshakov]

I double-checked the logs of original issue - it seems that error happened during console output (line), which has been removed in latest versions of agent - so I assume this one is not actual for last agent version.
Let me leave this open to investigate issue with running on rhel.