Node: Tracking issue: stdio problems

[Fishrock123]

This is a meta-issue for tracking multiple related but distinct stderr/stdout issues.

Below are checklists indicating the progress of these issues & pull requests.
A checkmark indicates it is resolved (or merged).

This issue encompasses the following issues:

• [ ] Chunked stdout/stderr drops writes if terminated early. https://github.com/nodejs/node/issues/784
• [ ] Interleaved stdout/stderr on terminals (TTYs)
  
  
  
  • Part of: _stdio buffered writes (chunked) issues & process.exit() truncation https://github.com/nodejs/node/issues/6456_
  
  
• [x] Missing or truncated error message https://github.com/nodejs/node/issues/6867

And the following pull requests:

• [ ] process: flush stdout/stderr upon process.exit() https://github.com/nodejs/node/pull/6773
• [x] tty: set the handle to blocking mode https://github.com/nodejs/node/pull/6816
• [x] tty: use blocking mode on OS X https://github.com/nodejs/node/pull/6895
• [ ] unix: add uv_flush_sync https://github.com/libuv/libuv/pull/876
• [x] unix,stream: fix getting the correct fd for a handle https://github.com/nodejs/node/pull/6838
  
  
  
  • Should be backported to v4.x (and maybe v5.x)
  
  

The current item in focus is:
_n/a_

The TTY (Terminal) issues have been resolved. Pipes still remain a problem.

---

This pr was a result of the discussion in https://github.com/nodejs/node/issues/6968 to better track all the efforts going on here.

[EDIT by @Trott: Did some edits around the 6773 PR checklist item, but undid them]

[Fishrock123]

Current plan is to make a release candidate (today/tomorrow) as discussed in https://github.com/nodejs/node/pull/6934 with the OS X patch from #6895 to fix the largest issue (OS X stdio interleaving to TTYs) asap.

[kzc]

You can add this one to the list:

6867 Missing or truncated error message

It's a variation of the process.exit() stdio truncating problem but with uncaught exceptions.

Edit: reproduced on node 5.x on Linux. Same issue likely exists on node 4.x.

[kzc]

This related merged PR for process.stdout._handle.fd should also be mentioned here for completeness:

unix,stream: fix getting the correct fd for a handle #6838

It ought to be back ported to node 5.x and 4.x.

[Fishrock123]

No guarantees on v5.x since it's now in critical-maintinence-only mode, but we'll try to get in v4.x soon.

[Fishrock123]

RC for #6895 building at https://ci-release.nodejs.org/job/iojs+release/921/ (Updated: new ci (2))

[Fishrock123]

I have an RC up at https://nodejs.org/download/rc/v6.2.1-rc.1/ that _should_ fix the OS X problems.
Please test.

cc-ing the people from other threads: @isaacs, @nodejs/npm, @Qix-, @eljefedelrodeodeljefe, @bcoe, @mikemaccana, @mgol

[Qix-]

@Fishrock123

Qix- is missing the Overall/Read permission

[Fishrock123]

@Qix- Are you getting that attempting to access the directory, or the files themselves?

(or is node making that error?)

[Qix-]

@Fishrock123 Works now, not sure what changed. :) I'll give it a try.

[eljefedelrodeodeljefe]

Ping myself for a possible issue with possibly leaking stdio I posted on twitter. Will investigate and elaborate tomorrow evening, for being swamped until then.

[bcoe]

Hey @Fishrock123 thank you. I'm pretty slammed with npm related stuff this week, but I have some smoke-tests I can run for you against yargs.

[addaleax]

@eljefedelrodeodeljefe Sounds like something that has a really good chance of being related to https://github.com/libuv/libuv/pull/779 (aka libuv/libuv@387102b2475776e5e40a3f6da5041eb674ad4abf)? That PR has come up a few times in the related discussions and touches the logic of which terminal is being written to directly.

[eljefedelrodeodeljefe]

@addaleax hmm. strong candidate. Sorry, haven't kept track.

[addaleax]

@eljefedelrodeodeljefe I haven’t kept much track around this either, this just popped into my mind when I read your tweet. I’d love to hear if you can confirm that this is actually the change that triggered it (and am up for pair debugging if you’re interested)

[mgol]

@Fishrock123 I confirm the problems with david & uglify-js are gone with the Node v6.2.1-rc.1. 👏

EDIT: Ditto for v6.2.1.

[kzc]

removed the ctc-agenda label

That's unfortunate. I suspected once node reverted to blocking stdio on Mac only that this broader stdio not flushing upon process.exit() or uncaught exception issue on Linux and other platforms would be shelved. The bug remains unfixed. There is no user-land workaround.

$ uname -ms
Darwin x86_64

$ node -v
v6.2.1

$ node -e 'var s="";for(var i=1;i<=1e6;++i){s+=i+": The quick brown fox jumps.\n";}console.log(s);process.exit(1);' | tail -5
2016: The quick brown fox jumps.
2017: The quick brown fox jumps.
2018: The quick brown fox jumps.
2019: The quick brown fox jumps.
2020: The quick $ 

[Fishrock123]

It's not shelved, but also not CTC meeting worthy.

My impression is that @bnoordhuis was looking into a better write-queue solution for libuv.

I will not have time for a week or two to investigate the impact of blocking on TTY perf, not have the time to dedicate to argue why we need a sync flush libuv api (probably).

[kzc]

not CTC meeting worthy

The non-blocking tty issue can be worked around in user code.
Piped stdio truncation has no such workaround.

[Qix-]

@Fishrock123 A pretty glaring bug that has been present since the IO.js fork that is causing a lot of teams at at least one large company (I'm sure we're not the only one) to move _away_ from Node.js-based services and utilities because certain output is truncated, isn't a problem to the CTC?

[Fishrock123]

@Qix- I don't really know why I'm the only person that seems to care lots about it, but it's not really worthy of _meeting time_ because I don't have an update on it.

@nodejs/ctc we should really care more about this. >_<

[eljefedelrodeodeljefe]

I have added this issue for consideration for the docs https://github.com/nodejs/node/issues/7333 with the intention of discouraging use of process.exit() for control flow

[Fishrock123]

https://github.com/nodejs/node/pull/6816 was merged in https://github.com/nodejs/node/commit/ab3306ad51d8136014aa0fa9278b57bb77105320 fixing the TTY (terminal) problems.

Pipes still remain an issue and this probably needs to be fixed in libuv, far beyond my expertise.

[Fishrock123]

ref: https://github.com/nodejs/node/pull/12970

[jasnell]

Closing as it's not clear if at all if there's anything remaining to do here and there's been zero activity since may