Node: Node.js fs.open() hangs after trying to open more than 4 named pipes (FIFOs)

[sami-sweng]

I have a node.js process that needs to read from multiple named pipes fed by different other processes as an IPC method.

I realized after opening and creating read streams from more than four fifos, that fs seems to no longer be able to open fifos and just hangs there.

It seems that this number is a bit low, considering that it is possible to open thousands of files concurrently without trouble (for instance by replacing mkfifo by touch in the following script).

I tested with node.js v10.1.0 on MacOS 10.13 and with node.js v8.9.3 on Ubuntu 16.04 with the same result.

---

The faulty script

And a script that displays this behavior:

var fs = require("fs");
var net = require("net");
var child_process = require('child_process');

var uuid = function() {
    for (var i = 0, str = ""; i < 32; i++) {
        var number = Math.floor(Math.random() * 16);
        str += number.toString(16);
    }
    return str;
}

function setupNamedPipe(cb) {
    var id = uuid();
    var fifoPath = "/tmp/tmpfifo/" + id;

    child_process.exec("mkfifo " + fifoPath, function(error, stdout, stderr) {
        if (error) {
            return;
        }

        fs.open(fifoPath, 'r+', function(error, fd) {
            if (error) {
                return;
            }

            var stream = fs.createReadStream(null, {
                fd
            });
            stream.on('data', function(data) {
                console.log("FIFO data", data.toString());
            });
            stream.on("close", function(){
                console.log("close");
            });
            stream.on("error", function(error){
                console.log("error", error);
            });

            console.log("OK");
            cb();
        });
    });
}

var i = 0;
function loop() {
    ++i;
    console.log("Open ", i);
    setupNamedPipe(loop);
}

child_process.exec("mkdir -p /tmp/tmpfifo/", function(error, stdout, stderr) {
    if (error) {
        return;
    }

    loop();
});

This script doesn't clean behind him, don't forget to rm -r /tmp/tmpfifo

Repl.it link

---

NOTE, The following part of this questions is related to what I already tried to answer the question but might not be central to it

---

Two interesting facts with this script

• when writing twice in one of the FIFO, (ie echo hello > fifo) Node is then able to open one more fifo, but no longer receives from the one in which we wrote
• when the read stream is created by directly providing the path to the fifo (instead of fd), the script doesn't block any more, but apparently no longer receive what is written in any of the FIFOs

---

Debug informations

I then tried to verify whether that could be related to some OS limit, for instance the number of file descriptor open.

Ouput of ulimit -a on the Mac is

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 256
pipe size            (512 bytes, -p) 1
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1418
virtual memory          (kbytes, -v) unlimited

Nothing points to some limit at 4.

---

C++ tentative

I then tried to write a similar script in C++.
In C++ the script successfully open a hundred fifos.

Note that there are a few differences between the two implementations. In the C++ one,

• the script only open the fifos,
• there is no tentative for reading,
• and no multithreading

#include <string>
#include <cstring>
#include <sys/stat.h>
#include <fcntl.h>
#include <iostream>

int main(int argc, char** argv)
{

    for (int i=0; i < 100; i++){
        std::string filePath = "/tmp/tmpfifo/" + std::to_string(i);
        auto hehe = open(filePath.c_str(), O_RDWR);
        std::cout << filePath << " " << hehe << std::endl;
    }

    return 0;
}

As a side note, the fifos need to be created before executing the script, for instance with

for i in $(seq 0 100); do mkfifo /tmp/tmpfifo/$i; done

---

Potential Node.js related issue

After a bit of search, it also seems to be linked to that issue on the Node.js Github:

https://github.com/nodejs/node/issues/1941.

But people seems to be complaining of the opposite behavior (fs.open() throwing EMFILE errors and not hanging silently...)

---

As you can see I tried to search in many directions and all of this lead me to my question:

Do you know what could cause this behavior?

Thank you

[addaleax]

Dealing with FIFOs is currently a bit tricky.

The open() system call blocks on FIFOs by default until the other side of the pipe has been opened as well. Because Node.js uses a threadpool for file-system operations, opening multiple pipes where the open() calls don’t finish exhausts this threadpool.

The solution is to open the file in non-blocking mode, but that has the difficulty that the other fs calls aren’t built with non-blocking file descriptors in mind; net.Socket is, however.

So, the solution would look something like this:

fs.open('path/to/fifo/', fs.constants.O_RDONLY | fs.constants.O_NONBLOCK, (err, fd) => {
  // Handle err
  const pipe = new net.Socket({ fd });
  // Now `pipe` is a stream that can be used for reading from the FIFO.
});

[mscdex]

Perhaps this could be included in the fs documentation?

[sami-sweng]

@addaleax Thank you, that seems to do the trick.

Out of curiosity, in that case Node creates one thread per socket?

[addaleax]

Out of curiosity, in that case Node creates one thread per socket?

@sami-sweng No – Node.js doesn’t use threading for any I/O other than file access, because it’s hard to implement async I/O for that in a cross-platform fashion. Everything else is handled through an event-based OS mechanism.

[mhelvens]

@addaleax: Thanks for this solution!

I should just mention that there is at least one behavioural difference compared to pure fs read-streams. It seems an EOF is not properly signalled through the socket, so we won't know when the stream on the other end of the fifo closes.

For my particular use-case, I happen to have another reliable hook from which I can call pipe.push(null), so I can currently work around this problem. But is there a way to actually respond to an EOF coming through the fifo?

(I am vaguely aware that EOF is not actually a character being sent through the stream, but I don't know enough to answer my own question.)

[addaleax]

@mhelvens Can you share an example? If I use the above script, add pipe.pipe(process.stdout); pipe.on('end', () => console.log('done')), and run something like echo 'hi' > path/to/fifo, it seems to work for me?

[mhelvens]

Hm... At the time it seemed obvious that the socket was to blame, but with you saying that, it becomes a lot more likely that I simply messed up somewhere.

I can't share the code itself, but I'll try to reduce it to a simple example. Thanks!

[mhelvens]

@addaleax I'm using the same example you are, apparently getting different results. Using Node 10.6.0 on macOS High Sierra. Could it be a Linux vs Mac thing?

[mhelvens]

@addaleax poke

[aalexgabi]

Why 4? Because there are 4 threads in the thread pool?
Why not open them in non blocking mode by default?

@addaleax Isn't a fifo a file too from the os perspective?

[bnoordhuis]

Why 4? Because there are 4 threads in the thread pool?

By default, yes.

Why not open them in non blocking mode by default?

Because the behavior when there's no one on the other end is underspec'd. On some platforms it works, on others you'll get an error or it simply hangs.

The behavior need not even be consistent on the same platform. 2.6.x linux kernels behave differently from 4.x and 5.x kernels, for example.

[aalexgabi]

@bnoordhuis Thanks for the clarification.

In case anyone needs to write to more than 4 pipes simultaneously without blocking the thread pool here is a workaround:

// Read write flag is required even if you only need to write because otherwise you get ENXIO https://linux.die.net/man/4/fifo
// Non blocking flag is required to avoid blocking threads in the thread pool
const fileHandle = await fs.promises.open(fifoPath, fs.constants.O_RDWR | fs.constants.O_NONBLOCK);
// readable: false avoids buffering reads from the pipe in memory
const fifoStream = new net.Socket({ fd: fileHandle.fd, readable: false });

// Write to fifo  
const shouldContinue = fifoStream.write(buffer);
// Backpressure if buffer is full
if (!shouldContinue) {
  await once(fifoStream, 'drain');
}

// Be aware that if you close without waiting for drain you will have errors on next write from the Socket class
await fileHandle.close();

I noticed that there is a bug in the NodeJS garbage collector when using fifoStream.close(). It detects the fd as a leak and tries to close it but it's already closed thus generating an unhandled EBADF exception.