Azure-iot-sdk-c: signal(SIGPIPE, SIG_IGN) not working in socketio_berkeley.c on Linux target

[maskimthedog]

In 1.3.6 PnP Public Preview, SIGPIPE occurs in Visual Studio while debugging. I was able to fix by commenting out the line below and adding MSG_NOSIGNAL param int he send() socket API call in
the socketio_berkeley.c adapter.

            //signal(SIGPIPE, SIG_IGN);

            ssize_t send_result = send(socket_io_instance->socket, buffer, size, MSG_NOSIGNAL);

[ericwol-msft]

Thanks. We are currently working on the pnp public preview refresh, and I added this issue on the list to be fixed.

[ewertons]

Hi @maskimthedog ,
suppressing any signals can be problematic, as other signals generated by the socket are valid and should be handled by the calling application.
We catch the SIGPIPE only because it is part of the logic we have for reconnection.

Could you share details about what system you are running the code on?

[maskimthedog]

Hi @ewertons ,

Thanks for your response. We have created a CMake project in Visual Studio to build executables to be run on a remote Linux system (with the build actually taking place on WSL on behalf of Visual Studio). We ran into problems debugging where the debugger would always break when a SIGPIPE occurred on the remote Linux system while debugging one of the aforementioned executables. We tried messing with signal control in regards to the debugger on the Visual Studio end to no avail and then attempted to suppress the SIGPIPE signal on the target which, at first, appeared to work but, ultimately, did not solve the problem. We also work with the Azure Sphere on another program, but the debugger does not break on SIGPIPE while debugging from Visual Studio. Sorry to be long-winded...in other words, you may close this ticket. Sorry to get you spun up for nothing. :)

[ewertons]

Hey @maskimthedog ,
by any means please don't apologize, we always benefit from getting as many details about an issue as possible.
Thanks for sharing.
However that still puzzles us why you would get the SIGPIPE break using azure-iot-sdk-c, but not with Azure Sphere (which also uses azure-iot-sdk-c underneath!!).
If you find more details, would you mind sharing them here on this issue, even if closed?

Thank you very much.
-Azure IoT SDK Team

[az-iot-builder-01]

@maskimthedog, thank you for your contribution to our open-sourced project! Please help us improve by filling out this 2-minute customer satisfaction survey

[maskimthedog]

@ewertons I spoke to soon. Our Azure Sphere based gateways were hit with version 20.07 version of the Azure Sphere OS. The sigpipe appeared when we killed the connection and then we saw it every time it tried to resend any pending messages and made debugging difficult. Our lead firmware engineer dug into it and came up with a solution. His solution is below and his explanation further below. His fix squashed the unhandled sigpipe seen in the debugger.

/* Block SIGPIPE before starting any other threads; other threads
    created by main() will inherit a copy of the signal mask. */
{
    sigset_t set;
    sigemptyset(&set);
    sigaddset(&set, SIGPIPE);
    pthread_sigmask(SIG_BLOCK, &set, NULL);
}

"I think when you have threads you need to use that pthread_sigmask method since we found where they say signal is undefined with threads, and I think if you have sockets you have to mask sigpipe in every thread that you do not want to handle them, which the easiest way is to put that in main before you create any threads so they all inherit it and then you only have to assign a handler in the thread that needs one"

[ewertons]

So is this code added to your main, before it creates the client to connect with the IoT Hub?
If this is the case, maybe it would be correct to add the suppression in the SDK code itself for the thread-safe API (functions that do not have _ll_ in the name), since there the SDK creates a thread to do the I/O.
This is great information, @maskimthedog ! Thank you very much for sharing. We will bring it to discussion with the team to assess it.

[maskimthedog]

@ewertons you are correct. We placed the code at the very start of main. Also, I cannot take credit. Our lead firmware engineer came up with this after I mentioned to him that I had closed this case and he started seeing sigpipe on Azure Sphere after MS pushed the 20.07 OS update to our Azure Sphere gateways.

[markrad]

I'm not sure what debugger is in use on Azure Sphere however I thought this might be useful. Debugging with Visual Studio Code on a Linux target using gdb the debugger will always break on a SIGPIPE even if you have requested it be ignored in the application code. I resolved this by passing

handle all nostop

in the gdb set up commands. This will prevent gdb stopping on the interrupt and allow the application code to deal with it.

[ewertons]

@markrad, thanks for the input.
My concern is if the code would do the same in production, not under debugger. If that is not the case, then this should be noted as a workaround and nothing would need to change in the SDK code.
@maskimthedog , could you confirm if @markrad mitigates the breaks you have been experiencing?

[maskimthedog]

From our lead:

"more likely it is that if you try to ignore it using signal(SIGPIPE, SIG_IGN) it does not get ignored by the application because signal is undefined with threads, just the fact that we are using gdb via Visual Studio and masking with pthread_sigmask stopped the exceptions hitting the debugger proves that"

[markrad]

I can't say for sure on Sphere if this is an issue with signal SIGIGN or just the debugger behaving in the same manner as on other Linux systems. I agree it needs to be tested.

[maskimthedog]

signal(SIGPIPE, SIG_IGN); is throughout the IoT Hub SDK and the signal still hits the debugger. signal(SIGPIPE, SIG_IGN); will not mask on threads, thread_sigmask(SIG_BLOCK, &set, NULL); will.

[ewertons]

Adding a resource for reference: https://www.linuxjournal.com/article/2121

[datemoon]

@maskimthedog thank you for your share.

[momuno]

PR merged in azure-c-shared-utility # 479