Go: x/term: create subrepo for terminal, console, pty and tty related packages

[davecheney]

Following the lead of #13058 towards smaller, more focused, sub repositories, I would like to propose the creation of a sub repo for packages related to handling the interaction with tty like things. This includes

• posix ttys and ptys
• serial termios control, like terminals and rs232 devices.
• windows cmd console

Much of the code for this exists in various packages, some in other sub repos like x/crypto/ssh/terminal, others has been duplicated incompletely by package authors with a specific use case; most times without a comprehensive story for windows.

Various informal proposal have been made, like #12853, relating to posix termios.

Other proposals like #12101 for the creation of an x/sys/terminal group of packages have been rejected on the grounds that sys is too low level, which also dovetails into the desire for more smaller sub repos, rather than larger ones.

There is also the the long standing question of why the code to disable terminal echo lives in the x/crypto/ssh package.

With the exception of the _name_ of the repository, which I'm sure will require debate, is there general support for a sub repository along the lines outlined in this proposal ?

[mattn]

:+1: x 256

[sbinet]

yes, please. especially the pty feature (which currently lives under github.com/kr/pty) which is always a bit subtle to handle right.

[rsc]

I am not sure about this one: terminal handling may not be significant enough for its own subrepo.

In general the subrepos seem to be mirroring our top-level standard library directories (sync, net, crypto, ...). It's certainly true that the ssh terminal code does not belong in crypto.

That said, I don't know where else to put it either. Let's see if any other suggestions arrive.

[gdamore]

yes! I am willing to help here. I'd like some higher level primitives, and frankly expressing things in terms of POSIX termio functions would be really nice. It could even be possible to express Windows console APIs in such terms.

In addition to just enabling / disabling raw & cooked mode (and echo/noecho), I see specific needs around:

• baud rate determination (used for real serial lines, but even console programs need it -- for example curses type apps need to know "baud" rate when calculating padding character delays required in terminfo strings
• tcflush & tcdrain
• serial port manipulation (stop bits, parity, etc.)

I'd far rather this all be expressed in terms of POSIX termios than some other way, at least to applications. (Using ioctls or whatever under the hood is fine.)

[bradfitz]

@rsc wrote:

In general the subrepos seem to be mirroring our top-level standard library directories (sync, net, crypto, ...).

Perhaps in that spirit we create golang.org/x/os then, for operating-system specific packages. I have a bunch of code myself in various repositories I'd love to centralize, for things that differ between Linux/Mac/Windows. Terminals seem like it could go under x/os.

[distributed]

Yes, please. I'm the author of github.com/distributed/sers, a way too noncomprehensive serial port library. I basically wrote it because I needed to set "non standard" baud rates to speak with hardware.

I would be glad to help out, should the proposed subrepo become reality.

[kim-racktop]

I have some preliminary code I wrote for the Posix tc functions that I have tested on Illumos. I could contribute that if people think it would be useful. I created a branch of x/sys in my local copy and have the code in x/sys/term.

[peterh]

With respect, I disagree that there isn't enough terminal handling to warrant its own subrepo. The termcap/terminfo parser alone would probably be a decent size, in addition to all the other things already mentioned. Plus all the os-specific versions of the interfaces for output (colours/styles/cursor positioning) and input (Ctrl-, Alt-, Fn-, Arrow-, etc keys) on non-Unix-like OSes.

I looked at the links in the proposal, and (like my own liner) none appear to support terminfo (or... is there a Plan 9 equivalent?). That's probably because everybody independently figured it wasn't worth the effort to write a terminfo parser for a single project. If the library and the effort are going to be shared, I'd be more likely to help.

[gdamore]

If we go beyond just termios, into terminfo parsing and so forth, yes, this
would be quite a bit bigger.

I wrote a fairly complete terminfo parser in tcell see
github.com/gdamore/tcell

(Its a little special purpose, because I don’t pick up all terminfo
keywords for storage, because most terminfo capabilities are useless in
modern terminals. But I have a full parser and expander, including a
complete implementation of TParm and TGoto.)

The low level half of tcell (basically everything in the base tcell
package) is probably what you’d want if you’re going to be doing rich
terminal work. This API may still be a little higher level than we’d like
to express in the core Go library though. (For example it has fairly rich
color mapping support, mapping any RGB value into the supported colors of
the terminal, including best-match color on 256 color terminals.) The
entirety of the base tcell package is about 7KLOC, plus a few KLOC for the
preparsed embedded terminfo database. (Basically I precompile the most
likely terminals into the code in an attempt to minimize dependencies on
external database files.)

On Fri, Oct 30, 2015 at 11:52 AM, Peter Harris [email protected]
wrote:

With respect, I disagree that there isn't enough terminal handling to
warrant its own subrepo. The termcap/terminfo parser alone would probably
be a decent size, in addition to all the other things already mentioned.
Plus all the os-specific versions of the interfaces for output
(colours/styles/cursor positioning) and input (Ctrl-, Alt-, Fn-, Arrow-,
etc keys) on non-Unix-like OSes.

I looked at the links in the proposal, and (like my own liner
https://github.com/peterh/liner) none appear to support terminfo (or...
is there a Plan 9 equivalent?). That's probably because everybody
independently figured it wasn't worth the effort to write a terminfo parser
for a single project. If the library and the effort are going to be shared,
I'd be more likely to help.

—
Reply to this email directly or view it on GitHub
https://github.com/golang/go/issues/13104#issuecomment-152615292.

[frederich]

I quite agree like Brad x/os.

[gdamore]

One problem I have with x/os, is that if we think of this functionality as
“os-specific”, then it becomes more likely that there is drift between the
_interfaces_.

That is, I agree that the _implementation_ is OS-specific. But having
OS-specific _APIs_ is highly undesirable, and is devastating for
application portability. (Indeed, this problem is exactly why things like
POSIX and the standard C APIs exist …)

So while the location is not terribly important to me — especially if it is
just base functionality like termio handling, if the location means that
its likely that the APIs will not be compatible across different ports, I
consider that a failure.

On Fri, Oct 30, 2015 at 12:50 PM, Jens Frederich [email protected]
wrote:

I quite agree like Brad x/os.

—
Reply to this email directly or view it on GitHub
https://github.com/golang/go/issues/13104#issuecomment-152635342.

[rthornton128]

+1. I also agree that functionality needs to be as OS agnostic/unified as possible akin to the majority of the standard library.

Would a package like this also include higher level constructs akin to curses panel, menu and form or would that be a community extension with the standard library package being solely low level?

[davecheney]

Would a package like this also include higher level constructs akin to curses panel, menu and form or would that be a community extension with the standard library package being solely low level?

I would vote no to that. I think we'll be struggling with the scope as it is.

However, I would hope this package would provide all the raw materials needed to construct such a library.

[gdamore]

Indeed. I could use that functionality and I believe the work I'm doing in tcell today goes substantially towards addressing the other higher level needs. (With views, widgets and so forth.)

Sent from my iPhone

On Oct 30, 2015, at 10:35 PM, Dave Cheney [email protected] wrote:

Would a package like this also include higher level constructs akin to curses panel, menu and form or would that be a community extension with the standard library package being solely low level?

I would vote no to that. I think we'll be struggling with the scope as it is.

However, I would hope this package would provide all the raw materials needed to construct such a library.

—
Reply to this email directly or view it on GitHub.

[techtonik]

It would be also nice if the package exposed performance metrics // tests both for terminal and for windows-like console.

[davecheney]

@adg @rsc ping

The x/time and x/sync repos have been approved, what are your thoughts on adding an as yet to be finalised, repository for terminal/serial/tty type packages ?

[adg]

I think x/os is probably the way forward.

[davecheney]

@adg golang.org/x/os sounds good to me !

[kim-racktop]

As long as we strive for an API that is consistent across the different OS's, sounds ok to me.

[gdamore]

Yes please!

That would NOT be consistent with the other stuff in x/os however. Mostly
x/os seems to be a kind of dumping ground for OS-specific functionality,
rather than any effort to provide a unified portable interface. For this
reason, I’m much more in favor of an x/term (or probably better, x/tty,
which would help keep the scope from ballooning into terminfo, curses, or
anything like that).

On Wed, Nov 4, 2015 at 11:47 AM, Kim Shrier [email protected]
wrote:

As long as we strive for an API that is consistent across the different
OS's, sounds ok to me.

—
Reply to this email directly or view it on GitHub
https://github.com/golang/go/issues/13104#issuecomment-153843047.

[adg]

That would NOT be consistent with the other stuff in x/os however. Mostly
x/os seems to be a kind of dumping ground for OS-specific functionality,
rather than any effort to provide a unified portable interface.

Some unix-isms crept in for historical reasons, but on the whole the os package is in fact a platform-agnostic interface to the operating system.

[techtonik]

Also, x/term and x/tty are already unixisms.

[davecheney]

I agree, but this that is unavoidable.

I'm sure there will also be packages that only make sense for the Windows command subsystem.

I hope that in addition to these operating system specific packages we can develop high level ones

For example, console control providing cursor control for text mode interfaces, Garrett has shown that this is possible for Windows with the same high level api.

Similarly I want to see a high level serial Comms package that operates across Linux, Mac and Windows. It's exported api should be agnostic, but will probably call on operating system specific packages to provide the implementation. See my GitHub.com/pkg/term package.

On 5 Nov 2015, at 19:07, anatoly techtonik [email protected] wrote:

Also, x/term and x/tty are already unixisms.

—
Reply to this email directly or view it on GitHub.

[techtonik]

@davecheney Windows console is more powerful/responsive in terms of user experience, because it provides a separate channel for controlling console window object (independent orthogonal API from C standard streams), while in terminal it is all garbled into one stream, which is slow and inefficient (and requires complex state machine to maintain). So decide carefully based on data if you need this Comms port and why - maybe there is a better way..

[gdamore]

The Windows console abstraction isn't that much more powerful. And yes a higher level API can be created to handle all that. I know as I've done it for Tcell.

It would still be useful to emulate some low level termios functions though. Even Windows has the notion of raw and cooked mode for example.

Sent from my iPhone

On Nov 5, 2015, at 5:47 AM, anatoly techtonik [email protected] wrote:

@davecheney Windows console is more powerful/responsive in terms of user experience, because it provides a separate channel for controlling console window object (independent orthogonal API to C standard streams), while in terminal it is all garbled into one stream, which is slow and inefficient (and requires complex state machine to maintain). So decide carefully based on data if you need this Comms port and why - maybe there is a better way..

—
Reply to this email directly or view it on GitHub.

[techtonik]

@gdamore I'd prefer to have user level API that can be later used to design proper low level interface in newer subsystems that require deterministic user interfaces (wayland etc.). Meaning that some "cooked" concepts need to be forgotten to avoid the same traps of the past.

Tcell - https://github.com/gdamore/tcell - looks cool. But it needs some quick intro / tutorial how to try it on your system.

[gdamore]

I don’t see how we can eliminate fully the idea of “cooked” vs “raw”
terminals. That is, most of the times people expect terminals to respond
to readline() (or whatever) by echoing character presses, and buffering
until a newline is pressed. But when richer applications, or even
prompting for a password, you need something more basic. You can do
passwords by just disabling echo, but for full UIs with menus & so forth,
you generally want to turn off buffering altogether.

On Thu, Nov 5, 2015 at 2:51 PM, anatoly techtonik [email protected]
wrote:

@gdamore https://github.com/gdamore I'd prefer to have user level API
that can be later used to design proper low level interface in newer
subsystems that require deterministic user interfaces (wayland etc.).
Meaning that some "cooked" concepts need to be forgotten to avoid the same
traps of the past.

Tcell - https://github.com/gdamore/tcell - looks cool. But it needs some
quick intro / tutorial how to try it on your system.

—
Reply to this email directly or view it on GitHub
https://github.com/golang/go/issues/13104#issuecomment-154221023.

[davecheney]

@adg @rsc now that we have https://go-review.googlesource.com/#/c/16551/ in addition to the proposal outlined above what is your opinion on createding a golang.org/x/os sub repo ?

[adg]

As I said, I am in favor of x/os. @rsc is the decider.

[alexbrainman]

How are we going to explain difference between golang.org/x/sys and golang.org/x/os to users? What is the difference?

Alex

[davecheney]

How are we going to explain difference between golang.org/x/sys and golang.org/x/os to users? What is the difference?

I'm not really sure. This proposal, or a previous one suggested added terminal/tty/console style packages to the x/sys repo but this was rejected by @rsc. This proposal is a replacement, so I guess we can use the rational from https://github.com/golang/go/issues/12853#issuecomment-152556261

[rsc]

I fear x/os will become a kitchen sink. What's the line there?

[davecheney]

@rsc I cannot disagree there. Would you reconsider a proposal for x/term as originally detailed in the proposal ?

[techtonik]

x/os is cross-platform (meaning "Go's operating system interface for humans") and x/sys is system-specific?

[davecheney]

I like that definition. We've seen several reviews that suggest that x/sys
not platform agnostic

On Tue, 10 Nov 2015, 19:44 anatoly techtonik [email protected]
wrote:

x/os is cross-platform (meaning "Go's operating system interface for
humans") and x/sys is system-specific?

—
Reply to this email directly or view it on GitHub
https://github.com/golang/go/issues/13104#issuecomment-155359467.

[techtonik]

If rephrased as common Operating System interface for humans then if something is not included in this concept means that no common interface between operating systems is possible.

To maximize the impact, the next step would be to create a pathway for explanations. So that people can get the answer "why it is not possible to get common interface", which will lead to description of actual conflicts, which is a necessary step to rethink things, which is a source data for improving human experience with Operating Systems (http://www.worldusabilityday.org/ is in 2 days), which could explain what could be changed, which is a necessary step to design new generation of Operating Systems and a source data to implement new hardware, which can also be based on completely different principles (optics, neural networks, etc.). But somebody need to start gather a technical database of compatibility conflicts, so that every new language out there didn't waste time on the same cross-platform problems that are not related to the language itself.

[rsc]

I don't believe x/os is what you really want here. I could see putting tty handling code into there, because the OS is involved with that, but not anything involving escape codes, so x/crypto/ssh/terminal would still not belong. The goal was to find a home for that code, right?

[davecheney]

@rsc

The goal was to find a home for that code, right?

Yes, the goal is as specified in the original post at the top of this issue

I would like to propose the creation of a sub repo for packages related to handling the interaction with tty like things. This includes

• posix ttys and ptys
• serial termios control, like terminals and rs232 devices.
• windows cmd console

[bradfitz]

At dotGo, @jfrazelle promoted on stage the acronym TUI (pronounced "too-ey") for Text User Interface, like GUI (Graphical User Interface). x/tui would encompass @davecheney's list of items, without having "term" in the name. :-)

[davecheney]

There are heaps of names that aren't really suitable, termios, tty, console, serial, etc. Of all, I preferred the x/os suggestion.

fwiw. I think x/time isn't a great name either, as it contains things concerned with rate limiting ... the obvious conclusion would be x/ratelimit. What's the point of this digression ? The name probably isn't critical.

[bradfitz]

FWIW, I find this name discussion very frustrating as well. I'd like to be producing and reviewing and sharing code instead of naming things.

[davecheney]

Thanks Brad. I'm pretty confident that any name chosen would be wrong by
some measure. I think x/term would have been fine, but x/os is even better.

I think this issue shows a genuine need, terminal handling, echo disabling,
serial port handing code has been rewritten, incompletely, dozens and
dozens of times, on an as needs basis by package authors. It would be
unfortunate if the go project couldn't provide a solution to this very
common set of problems because we couldn't agree on the perfect name for
the repo those packages lived in.

Thanks

Dave

On Wed, 11 Nov 2015, 08:23 Brad Fitzpatrick [email protected]
wrote:

FWIW, I find this name discussion very frustrating as well. I'd like to be
producing and reviewing and sharing code instead of naming things.

—
Reply to this email directly or view it on GitHub
https://github.com/golang/go/issues/13104#issuecomment-155571263.

[kim-racktop]

Well, naming things is one of the two hard problems in computer science. The other one being cache coherency and off by one errors.

That said, I am aware of this sort of discussion going on for at least 3 months. I am hoping that some name can be settled on reasonably soon and that the README.md file for the new repo describes why the name was chosen, as well as what goes into the repo and why.

I feel under-qualified to push for any particular suggestion. While I don't think the perfect name has been put forward yet, I don't have any other good suggestions and the ones already mentioned aren't too offensive.

[rthornton128]

Naming has the feeling of long term consequences and misconstrued intent, which is the crux of this discussion. Despite feeling not feeling constructive there is value in it nonetheless.

I feel that tui is misleading in that it invokes a higher level of abstraction than this package intends to offer. "Term" is common to Unix whereas "console" is common to Windows.

I have to agree with Dave in that x/os makes the most sense but does feel like a dumping ground. Perhaps, like std/os, this is the right place for os specific terminal code and leave an os agnostic implementation for a separate, higher level tui package?

[rsc]

re @bradfitz

FWIW, I find this name discussion very frustrating as well. I'd like to be producing and reviewing and sharing code instead of naming things.

We spend time getting names right not because we love arguing about names but because the name of something can have a very large impact on how easy it is to understand and how it will evolve.

In this case, the name has significant impact on what people will think belongs in the subrepo. It should be clear that x/term and x/os would be very different subrepos. I'm sorry about the holdup here, but of course producing and reviewing and sharing code can happen in other places too, if it's more important to move fast and break things than to get things right.

[gdamore]

I really think tui is terrible for the reasons you list and I'm pretty unhappy about x/os because it does feel like it becomes a dumping ground.

Let me offer up x/tty

While nobody actually uses teletypes anymore this interface used with stty and the termios ioctls on UNIX systems covers all the bits one needs for handling tty devices while remaining low level.

Furthermore it will be possible to emulate this functionality easily for Windows. The idea of setting raw mode, disabling or enabling buffering and echo are pretty universal. Windows consoles can offer some enhanced functionality as well but for something like the needs of ssh that is entirely unnecessary. (Extras like double buffering and mouse reporting.)

I cannot think of any system which offers a prompt like interface for shell interpreters which cannot supply an implementation of these interfaces.

I'd be equally happy if this were called termios and the interfaces built to emulate POSIX termios. I actually believe that for most consumers that will be the easiest and most natural API.

Btw ssh does need escape sequences or anything like that. Just low level termios support is sufficient.

Sent from my iPhone

On Nov 10, 2015, at 5:02 PM, Rob Thornton [email protected] wrote:

Naming has the feeling of long term consequences and misconstrued intent, which is the crux of this discussion. Despite feeling not feeling constructive there is value in it nonetheless.

I feel that tui is misleading in that it invokes a higher level of abstraction than this package intends to offer. "Term" is common to Unix whereas "console" is common to Windows.

I have to agree with Dave in that x/os makes the most sense but does feel like a dumping ground. Perhaps, like std/os, this is the right place for os specific terminal code and leave an os agnostic implementation for a separate, higher level tui package?

—
Reply to this email directly or view it on GitHub.

[davecheney]

I'm fine with x/tty.

[adg]

What about x/console with package 'console' being in the repo root?

[gdamore]

I like x/console less, but can live with it. console suggests (to anyone other than a Windows console user) a specific interface -- which can overlap with but is also orthogonal to tty handling. (On UNIX the "console" is the primary login interface by default, and is also where kernel messages and high priority syslogs are usually directed; it has a specific interface /dev/console that is typically _not_ interactive but just a sink for messages.)

To be honest, any of /x/tty, /x/console, /x/term, /x/termio[s] can work. I'm partial to /x/tty and /x/termio because I think those most accurately reflect the functionality we plan to put there, without suggesting that a lot of other quasi-related stuff (like ANSI escape sequences) wind up there.

[adg]

Would ANSI escape sequences _not_ be in this repository? Maybe I
misunderstand the scope.​

[gdamore]

I don’t think so, no. Properly handling ANSI starts to get a lot more
complex, because then you want a full terminfo parser, except on Windows
where you do something different. Frankly, that’s a meaty bit of work (I
know because it exists in my tcell package), and I don’t really think
_that_ belongs in core go. That really feels like more of a 3rd party
package delivery, like tcell, to me. (Much like GUI/widget toolkits don’t
really belong in core go either, I think.)

On Tue, Nov 10, 2015 at 9:16 PM, Andrew Gerrand [email protected]
wrote:

Would ANSI escape sequences _not_ be in this repository? Maybe I
misunderstand the scope.​

—
Reply to this email directly or view it on GitHub
https://github.com/golang/go/issues/13104#issuecomment-155669424.

[techtonik]

Let me offer up x/tty

While nobody actually uses teletypes anymore this interface used with stty and the termios ioctls on UNIX systems covers all the bits one needs for handling tty devices while remaining low level.

Furthermore it will be possible to emulate this functionality easily for Windows. The idea of setting raw mode, disabling or enabling buffering and echo are pretty universal. Windows consoles can offer some enhanced functionality as well but for something like the needs of ssh that is entirely unnecessary. (Extras like double buffering and mouse reporting.)

Please don't use TTY for cross-platform stuff. It is overly complicated concept that may take up to 3 days to understand what is it about. The idea of setting raw mode, buffering and echo doesn't belong to the world of text mode user interfaces at all - all these hacks are made to make limited one-input-channel serial interface work for users, so maybe x/serial is what you need? My perception is that Linux terminal is slow and error-prone state machine - it is not nearly as fast as Windows console, and all this multiplexing/demultiplexing is the reason. Also think that clipboard copy/paste functions are also the things that are expected from the console, as well as multiple buffers that help to work with multiple outputs in parallel.

[techtonik]

If I understand correctly, both Windows console and ncurses use asynchronous events while TTY and Linux terminals with are synchronous beasts.

[davecheney]

@techtonik thank you for your feedback.

I understand that this proposal is not an exact match for the capabilities of the windows console, but this is not the specific goal of this proposal.

There is a need for a set of packages that manages tty like streams of bytes connected to text console like devices. These might be terminals, or windows cmd.exe, or remote ssh sessions, or serial terminals, and so on.

These facilities have been implemented over and over again on an as needs basis by many authors and are usually incomplete -- And the missing piece is overwhelmingly windows support. There are many serial handling packages, few support windows. There are many ncurses style packages, few support windows.

You are correct that adopting an api that treats cmd.exe as a stream of bytes will be an incomplete abstraction. Such limitations also exist by treating xterm's like pty's. I'm sure similar restrictions exist for plan9 rio windows, but I have no direct experience.

In return for giving up a little performance, or a little control, what all Go developers stand to gain is one package that everyone can use to do simple things like

• open a serial device, and set it's baud rate
• disabling terminal echo when entering passwords
• the raw mechanics necessary to implements readline type cli interfaces
• and of course to be a base for more complicated projects.

I'm also confident that where necessary the correct holes can be poked in the API to permit developers who want to exploit platform specific facilities to have access to those APIs.

Can you see your way to agreeing that this proposal is trying to make things better for windows developers ?

Thanks

Dave

[techtonik]

Why so many people tried to reinvent the console/terminal and failed? Being able to compare two worlds, I don't see that stateful OS API works at all. State belongs to user level. OS should only define who has focus to receive input and filter the critical key combinations that are needed by parent.

• "open serial device and set baud rate" - x/serial
• "entering password" - ideally it should be readkey/getkey - to process Y/n in one keypress, but this is really hard - see my code below - maybe even impossible to do with terminal abstraction - terminal doesn't process keys like F1, it doesn't return scancodes and if you make TUI menus with shortcuts (like all Linux programs do) - they break when you change keyboard layout - in Far Manager (my "vimacs" for Windows) it doesn't happen
• "the raw mechanics necessary to implements readline type cli interfaces" - what mechanics exactly? why readline is different from other console programs that need to receive keys from keyboard and write colored chars at arbitrary location of screen?
• "and of course to be a base for more complicated projects" - it won't be the base, because the basis - the concept of one stream that does it all is flawed - it is impossible to debug and deal with

My opinion is that terminals, TTY and all that serial stuff must die for text mode interface. Maybe reengineered from scratch for better latency and removed legacy like MOSH, stripped of features that add to technical debt, but these are specialized products.

Just for the example of how things are badly broken - this is my attempt to read the key from terminal/console on Windows and Linux.
https://bitbucket.org/techtonik/python-pager/src/2936adb19c9407b8e890949f9e88bf4f0461a524/pager.py?at=default&fileviewer=file-view-default#pager.py-196
What I see here is that if somebody interrupts my program on Linux while I listen for input, the terminal will be broken with no echo. Also there is no way to peek if there are characters in keyboard input. And users report that this stuff doesn't work on OS X, and piping the output is also broken when there is a terminal readkey call in the program flow.

I really hate terminal interfaces, sorry. =/ stdin is not enough for user interface not matter what. The read console interface needs to start with user stories and instead of concentrating on that should be done with two wires, separate the stuff into what is practically possible over the terminal and what should use things like https://mosh.mit.edu/ and https://bitbucket.org/libtcod/libtcod and provide support for latter.

For me, the user is primary, so if we start with TTY and then at some point discover that https://bitbucket.org/libtcod/libtcod is not possible - that's would be fail - the same fail that the dozen of other packages already did. So, I feel that creating and good library for text mode interfaces for Linux, Windows and ... may be impossible at this point, but what can help is to form an API for a operating system that may not yet exists. Going with plain terminals for something that libtcod does is the same as trying to build the house by sticking balls with a chewing gum. That's my point.

[techtonik]

To summarize, if you need to emulate those crutches on Windows, go with x/term.

[cznic]

My perception is that Linux terminal is slow and error-prone state machine - it is not nearly as fast as Windows console, and all this multiplexing/demultiplexing is the reason.

I don't think this claim is based on evidence and I believe it is important to get the facts right before adopting a decision based on them.

There exist opposite experiences. Quote from golang-codereviews

This adds at one new syscall for every print to without a
console and four new syscalls for every print to with an
attached console. The Windows console is extreemly slow so
the extra syscalls won't be noticable.

Also, googling for "windows console slow" shows a different story wrt what @techtonik is telling us.

[alexbrainman]

Dave,

Serial port is easy: open port, change speed (and other parameters), then read and write bytes. Also read and change serial port status / flags. (I also wander who uses serial ports these days). Serial port can be in a separate package, it does not need to be related to anything else.

Console input and output is something I have never done before. But I just googled and found this http://www.benryves.com/tutorials/winconsole/all - it is a start. I am not sure what you are trying to do, so take it with a grain of salt.

As always I am happy to help any way I can.

Alex

[gdamore]

Huh? Events arrive and are picked up by calling a function in both cases. In the case of ttys it is read while Windows programs can either use read of a console handle or a Windows specific API that can return more than just keyboard presses.

The tty abstraction works equally well for both Windows console.

Sent from my iPhone

On Nov 10, 2015, at 11:17 PM, anatoly techtonik [email protected] wrote:

If I understand correctly, both Windows console and ncurses use asynchronous events while TTY and Linux terminals with are synchronous beasts.

—
Reply to this email directly or view it on GitHub.

[gdamore]

My work with tcell on both Windows and UNIX is different too. The Windows console is both more and less featureful than UNIX terminals. And I think some confuse cmd.exe with conhost.

Nonetheless this proposal isn't about creating a package that can do all that a package like tcell can or his Far manager needs. Frankly for rich TUIs a bit of OS-specific work is needed for sure. For example terminfo handling is wildly different than the call to manage output attributes on Windows.

What however this proposal is trying to achieve is to give a more portable view of adjusting some basic tty/console handling so that thinks like simply prompting for a password become a little more straight forward. Think of it as a foundation to permit a getpassphrase() implementation to be created.

It also is about providing the low level foundation so that a package like tcell or termbox can be created on top of it, in a way that allows it to run pretty much everywhere without having to hard code specific ioctls or resort to cgo to function.

Sent from my iPhone

On Nov 11, 2015, at 2:06 AM, cznic [email protected] wrote:

My perception is that Linux terminal is slow and error-prone state machine - it is not nearly as fast as Windows console, and all this multiplexing/demultiplexing is the reason.

I don't think this claim is based on evidence and I believe it is important to get the facts right before adopting a decision based on them.

There exist opposite experiences. Quote from golang-codereviews

This adds at one new syscall for every print to without a
console and four new syscalls for every print to with an
attached console. The Windows console is extreemly slow so
the extra syscalls won't be noticable.

Also, googling for "windows console slow" shows a different story wrt what @techtonik is telling us.

—
Reply to this email directly or view it on GitHub.

[alexbrainman]

Another example of reading from Windows console:

https://helloacm.com/modern-getch-implementation-on-windows-cc/

Alex

[gdamore]

Right. GetConsoleMode() and SetConsoleMode() are the functions I'd
probably want to use when building this Windows analog.

There's code for this in tcell too btw:

For example:

https://github.com/gdamore/tcell/blob/master/console_win.go

Note that not everything there is something I'd consider a candidate for
this repo.

On Wed, Nov 11, 2015 at 2:55 PM, Alex Brainman [email protected]
wrote:

Another example of reading from Windows console:

https://helloacm.com/modern-getch-implementation-on-windows-cc/

Alex

—
Reply to this email directly or view it on GitHub
https://github.com/golang/go/issues/13104#issuecomment-155935181.

[davecheney]

Hi Alex,

How do you control the serial line parameters on windows, guessing its
some WSASetSerial type syscall. As far as I know that isnt available in the
std lib or x/sys.

Also, at a higher level there is no nice wrapper that takes parameters like
baud rate and converts those to their lower level windows or Unix syscalls.

This is what this proposal is, in part.

Thanks

Dave

On Wed, 11 Nov 2015, 21:24 Alex Brainman [email protected] wrote:

Dave,

Serial port is easy: open port, change speed (and other parameters), then
read and write bytes. Also read and change serial port status / flags. (I
also wander who uses serial ports these days). Serial port can be in a
separate package, it does not need to be related to anything else.

Console input and output is something I have never done before. But I just
googled and found this http://www.benryves.com/tutorials/winconsole/all -
it is a start. I am not sure what you are trying to do, so take it with a
grain of salt.

As always I am happy to help any way I can.

Alex

—
Reply to this email directly or view it on GitHub
https://github.com/golang/go/issues/13104#issuecomment-155727833.

[davecheney]

I am genuinely frustrated by the lack of consensus on the utility of this proposal. I had thought that it was something that was clearly needed given the number of overlapping proposals for various ioctl and termios tweaks.

Perhaps it is appropriate that is there is no support I should withdraw this proposal.

[gdamore]

I hope you don't. The missing low level functionality is sorely needed and addressing it here is about the only chance for me to be able to support a package like tcell without having to resort to cgo.

Sent from my iPhone

On Nov 11, 2015, at 8:51 PM, Dave Cheney [email protected] wrote:

I am genuinely frustrated by the lack of consensus on the utility of this proposal. I had thought that it was something that was clearly needed given the number of overlapping proposals for various ioctl and termios tweaks.

Perhaps it is appropriate that is there is no support I should withdraw this proposal.

—
Reply to this email directly or view it on GitHub.

[adg]

I see a lot of posts in favor of creating such a sub-repo? The only contentious thing I see here is the name. It seems that x/os is out. Of the other proposed names, the original x/term seems best to me.

[gdamore]

As long as we agree about the scope (that is I don't want a curses replacement here! Just low level termios style functionality) then the name isn't such a big deal to me. I can live with x/os too even though it's not my favorite. I'd rather have something than nothing. :-)

Sent from my iPhone

[alexbrainman]

How do you control the serial line parameters on windows, guessing its
some WSASetSerial type syscall.

It has been a while, but, I believe, I've used GetCommState/SetCommState.

As far as I know that isnt available in the
std lib or x/sys.

GetCommState and SetCommState aren't available, but I can help you with that. Just let me know. I even happy to try and implement windows parts of serial port once you have something working on unix. I have some old code of mine to help me get started.

Alex

[davecheney]

I see a lot of posts in favor of creating such a sub-repo? The only contentious thing I see here is the name. It seems that x/os is out. Of the other proposed names, the original x/term seems best to me.

Thanks @adg. To be clear, I wasn't complaining about the discussion about the name of the repo.

[rsc]

It sounds like this repository will contain code for (1) dealing with consoles/ttys/ptys, (2) termcap-style escape sequence generation and parsing, and (3) code for configuring serial ports. It's a bit of an odd mix, but it's certainly a common grouping historically. Is that list correct, @davecheney?

Based on the discussion here and the inability to find a better name, I think x/term is fine for this set of code. If you can confirm the scope, I'll throw my hat in with x/term and then I think there is sufficient agreement to approve the proposal. Thanks for bearing with us.

[gdamore]

On Nov 12, 2015, at 7:33 AM, Russ Cox [email protected] wrote:

It sounds like this repository will contain code for (1) dealing with consoles/ttys/ptys, (2) termcap-style escape sequence generation and parsing, and (3) code for configuring serial ports. It's a bit of an odd mix, but it's certainly a common grouping historically. Is that list correct, @davecheney?

No. Just 1 & 3. Item 2 really doesn't belong here IMHO. It is something easily provided by a third party package like tcell. I don't believe escape sequence handling is something the core ought to be concerning itself with.

That said I would support the idea of _one_ higher level interface being added because it's likely to be so widely used and useful as to warrant being here. That is something akin to getpassphrase() for password prompting. Probably 90% of the most common uses for the lower stuff are based on a need to do just this. Including btw the ssh use case.

• Garrett

Based on the discussion here and the inability to find a better name, I think x/term is fine for this set of code. If you can confirm the scope, I'll throw my hat in with x/term and then I think there is sufficient agreement to approve the proposal. Thanks for bearing with us.

—
Reply to this email directly or view it on GitHub.

[rsc]

Re @gdamore, this isn't the core. It's a subrepository. And x/crypto/ssh/terminal already has escape code code in it. So if escape codes are out of scope, we can't move x/crypto/ssh/terminal, which was at least part of the point.

[gdamore]

Oh wow. I just saw that. It was a big surprised to me, because IMO there is no reason for that code to live in ssh. Terminfo handling is a _much_ bigger issue, and simple colorizing of text is non-trivial to handle in the general case. What I see there is basic codes suitable for ANSI terminals, but those won't work on every terminal type and are completely inappropriate for Windows consoles.

Indeed, the way you colorize text on Windows really can't be handled by Escape codes at all. The logic here is just Wrong, because it is too naive.

Given all this, I'd argue for leaving that code where it is, deprecating it if we can, and having x/term (or whatever it is called) _not_ provide replacement functionality. Alternatively, we could provide a set of ANSI escapes codes along with an admonition that they should not be used unless an ANSI terminal is use, and that many terminals don't support these escapes.

Looking at what is in ssh, it looks like a naive attempt to provide a limited replacement for readline or somesuch. I'm ... astonished. Is this code "new"?

I feel like this ought to be placed in a package called "prompt" or somesuch, which could import higher level functionality. That would include both terminfo parsing (for UNIX like systems) and probably native console manipulations for Windows. I could build this type of functionality on the low-level bits in tcell ( or contribute parts of tcell to make up this functionality.)

Still, this is far and away larger scope than what I think is absolutely _critical_, which is providing lower level functions for manipulation of terminfo, etc. That is functionality which I _cannot_ do in native Go on all platforms, so it has to be fallback to cgo. All this other stuff for prompting etc. is higher level functionality that could easily be done in pure Go, given the lower level functionality. (As the ssh package shows.)

The ssh Terminal package is also horribly broken with respect to runes that have a visible width other than 1. That its, it understands neither full width East Asian runes, nor special composing characters.

In retrospect, maybe it _is_ time to promote tcell (or modifications of it) as a replacement for this package. Again, we need to have the core supply the low-level terminfo manipulations for us, though.

Do we know who the consumers of this ssh Terminal package are?

[rsc]

The main user of ssh/terminal is the ssh server. Even if the ssh server is run on Windows, that code makes sense, because ssh clients know about (and send) terminal escape codes as a wire protocol. More generally I think it's okay for the escape codes to be in x/term just as much as serial speeds are. It's all part and parcel of the same 1970s era technology.

[gdamore]

but terminal escape codes _aren't_ part of the wire protocol of ssh.

[rsc]

but terminal escape codes aren't part of the wire protocol of ssh.

In theory, no. In practice, they kind of are. Otherwise there's no way to send, for example, an up-arrow keystroke.

[mattn]

terminal code in ssh package should be internal package or private methods. but too late.

[gdamore]

Agreed. Too late. :-(

The details of an up-arrow mapping are _not_ part of the SSH protocol. Its
better to think of SSH like a serial line (8-bit). It passes byte streams
unmolested (mostly) between two parties.

The fact that one of those parties is an environment running perhaps in a
terminal that can parse certain sequences of bytes as e.g. an up arrow
keypress, or as a command to clear the screen, is a private negotiation
done between the two parties communicating over SSH. Typically in the case
of the server, this means the program that is forked by the server. (E.g.
your editor.)

For UNIX, as I'm sure you're aware, this involves setting a $TERM variable,
and using a database (terminfo/termcap) that expresses these capabilities.
But the thing that looks up the fields in the database is _not_ the ssh
server (or client), but rather any interactive program that such server
spawns.

(NB: For something like a program running on a Windows console, this is not
used at all. That's a very very different environment, and control data
like extended key codes and window management operations are sent out of
band.)

Scanning briefly at the SSH server code, I have to confess I have no idea
why we have this code here. It doesn't look like the SSH server implements
a shell, or anything like that. But maybe I'm missing something?

So, all this said, given the unfortunate precedent, I'm ok with extending
the scope of x/term to include this type of handling. I think it becomes
something like curses or curses-lite (or tcell or termbox) to be useful --
the interfaces exposed by ssh Terminal are just plain _wrong_, and I
wouldn't like to see those interfaces promulgated further.

That said, there are a _few_ interfaces there that are nice.
ReadPassword() is a nice/good interface. That can be done entirely without
this logic. The only "escape" sequences it needs to understand are CTRL-H
(backspace) and _perhaps_ CTRL-U (delete back to beginning of line). Those
aren't even escape sequences properly. I'm not convinced that even this is
necessary. I think it should be possible to do this entirely without
escape sequences by just disabling echo. (Meaning I don't think full raw
mode is required to prompt for passwords.)

As I think about this, it feels like there are really several different
packages here:

a) x/tty -- low level termio/console mode setting stuff. This is the most
basic functionality and the thing that cannot be done without using either
cgo or help from the runtime. A reasonable argument can be made for this
to live in x/os. I prefer it separate, but as long as the _interfaces_ are
agnostic I'm happy to see this limited scope stuff live in x/os. This
could include the low level stuff for serial ports too, since historically
these are frequently linked together with termios.

b) x/term - building upon the above, adding support for echoing with
attributes, getting terminal size, setting cursor location, etc. Sort of
termbox or tcell functionality. This is where a terminfo parser for
example would live, and where the knowledge of manipulation of Windows
consoles would live. This provides the abstraction of different terminal
types, etc. This is equivalent to the low level terminfo handling on UNIX
(though it would need to have an interface that also supports out-of-band
metainformation like Windows consoles use.)

c) x/prompt -- this could involve both a simplistic password prompter, but
also a readline or editline replacement, with command history and so
forth. This would make it possible to build nice CLIs with an interactive
line-editing mode, etc. This could be built upon x/term.

d) x/tui -- this is a full blown TUI package, with windows or view ports,
etc. It would build upon x/tty and x/term. In tcell I have a package that
does this called tcell/views -- its somewhat limited but it does have
widget layering and scrollable view ports, etc. This is sort of the full
curses type functionality.

Now I'll point out that its probably quite reasonable for b, c, and
_especially_ d to be implemented as 3rd party packages. Indeed, there are
two packages that have already done this -- termbox and tcell -- for b & d.
(Other packages that do c exist, but they do it via cgo and libeditline or
libreadline usually, I think.)

What's really become obvious is that the name is _important_ because it
helps define the _scope_, and I think that we need to resolve the scope
before getting to fixated on the name. :-)

• Garrett

On Thu, Nov 12, 2015 at 9:51 AM, mattn [email protected] wrote:

terminal code in ssh package should be internal package or private
methods. but too late.

—
Reply to this email directly or view it on GitHub
https://github.com/golang/go/issues/13104#issuecomment-156182504.

[rsc]

We're only creating _one_ subrepo for this code. Subdirectories like
x/term/tty and x/term/prompt are fine (and expected).

[davecheney]

We're only creating _one_ subrepo for this code. Subdirectories like
x/term/tty and x/term/prompt are fine (and expected).

Yes, this was the design I had in mind.

[rsc]

Proposal accepted. Assigning to @adg for subrepo creation.

[gdamore]

Sounds perfectly reasonable I think. Are we at consensus finally?

Sent from my iPhone

On Nov 12, 2015, at 11:55 AM, Dave Cheney [email protected] wrote:

We're only creating one subrepo for this code. Subdirectories like
x/term/tty and x/term/prompt are fine (and expected).

Yes, this was the design I had in mind.

—
Reply to this email directly or view it on GitHub.

[kim-racktop]

I am not exactly sure what we have consensus on other than the repo name. I do think we should factor out things like the Posix tc* routines into their own subdirectory/package. I agree that something like a full curses replacement is outside the scope of this repository. I am not exactly sure where we should draw the line between these two extremes. What I am hoping to see is that whatever we put into the packages inside this repo isolates me from needing cgo or needing os specific code.

[davecheney]

Hi Kim,

A cursors interface is out of scope, both by the original proposal, and also because none of the underlying plumbing exists.

I suggest we work on that plumbing and we'll probably find the natural boundary around the level of a type that represents the attributes of a tty or console; its echo state, its raw dimensions, possibly its speed (if necessary). Anything above that is probably out of scope for this package

[alexbrainman]

Here https://github.com/alexbrainman/winapi/commit/26571130fc308d5a9dcc6dba41df3a91865ac4f4 are some serial port related APIs.

Alex

[techtonik]

@cznic about performance of Linux terminal vs Windows console. As I said, it is my perception that Gnome terminal (which is default on Ubuntu) is slow as hell. And quick search reveals that Linux terminals are indeed slow https://unix.stackexchange.com/questions/41225/can-a-terminal-emulator-be-as-fast-as-tty-1-6

As for your "windows console slow" the first thing I've got is this https://stackoverflow.com/questions/14295570/why-is-console-animation-so-slow-on-windows-and-is-there-a-way-to-improve-spee and there user runs "CLS" external process for every frame. As for https://groups.google.com/forum/#!msg/golang-codereviews/hq63SCACMJo/KL97UzWeEQAJ which says The Windows console is extreemly slow so the extra syscalls won't be noticable. phrase, then there are two points:

1. there are no any reference to this claim
2. the context in which it was mentioned is that Go doesn't use WinAPI's WriteConsole which probably should be used if you want performance on Windows

If you have any idea how to measure FPS of Linux terminal vs Windows console, I'll be happy to run tests on my machine. Please also note that stdout.print() performance is not the only parameter for measuring speed - keyboard response equally matters (if not even more important).

[techtonik]

It sounds like this repository will contain code for (1) dealing with consoles/ttys/ptys, (2) termcap-style escape sequence generation and parsing, and (3) code for configuring serial ports. It's a bit of an odd mix, but it's certainly a common grouping historically. Is that list correct, @davecheney?

I'd say 2 and 3. Because I believe that dealing with windows console is impossible in the same way as ttys and ptys.

[techtonik]

Right. GetConsoleMode() and SetConsoleMode() are the functions I'd
probably want to use when building this Windows analog.

WinAPI contains stateless getch() and kbhit() functions that are more clean and I think more useful. See this implementation in Python
https://bitbucket.org/techtonik/python-pager/src/2936adb19c9407b8e890949f9e88bf4f0461a524/pager.py?at=default&fileviewer=file-view-default#pager.py-190

Still there is a big problem how to get exact keycode indifferent of language layout and encoding. Something kind of readKey() function. For some reason I think it is impossible on Linux, but it would be nice if Go could provide an interface for that once kernel is patched to forward that information to active user space process.

[techtonik]

And JFYI, I am working on Windows client for LXC (yes, Linux Containers) and we've just committed ANSI colors support there with the help of external package that parses them and now I am looking how to read arrow keypresses to forward to remote server - https://github.com/lxc/lxd/issues/1317

[gdamore]

Um. I did this. (Get a key etc.) Windows actually has a pretty sane API,
and it does map reasonably well. The console modes can be used to
construct a fair facsimile of POSIX termios, and for richer console I/O
there are the ConsoleOutputCharacter(), ConsoleOutputAttribute(),
ReadConsoleInput(), etc.

These don’t play entirely perfectly well with simple input/output
(stdin/stdout), but using them its perfectly possible to build a curses or
termbox API. This has indeed been done _multiple_ times.

I recommend you have a look at my tcell package, this file in particular:
https://github.com/gdamore/tcell/blob/master/console_win.go

On Tue, Nov 17, 2015 at 3:24 PM, anatoly techtonik <[email protected]

wrote:

And JFYI, I am working on Windows client for LXC (yes, Linux Containers)
and we've just committed ANSI colors support there with the help of
external package and now I am looking how to read arrow keypresses to
forward to remote server - lxc/lxd#1317
https://github.com/lxc/lxd/issues/1317

—
Reply to this email directly or view it on GitHub
https://github.com/golang/go/issues/13104#issuecomment-157543038.

[gopherbot]

CL https://golang.org/cl/17020 mentions this issue.

[adg]

The term repo is now created.

[techtonik]

Great! =) Now how can I track all issues there? (and also, is it already possible to use it right from the start in Go 1.5.x?)

[adg]

We still use a centralized bug tracker. Related bugs should have the prefix 'x/term'.

[techtonik]

Okay. Github doesn't allow to subscribe to that. So the only strategy I see is to create all the issues myself. =0

[Julio-Guerra]

• posix ttys and ptys
• serial termios control, like terminals and rs232 devices.

@davecheney @adg I think this package shouldn't expose a unix-like TTY interface but should rather build something smarter and cleaner on top of it. By separating concerns, it should then be even easier to port this package on non unix-like OSes. TTY and PTY are low-level POSIX/*NIX interfaces that should go to x/sys/unix like termios partly is already (the structure only so far)...

In my opinion, this package should only provide an abstraction based on the canonical-mode API (line disciplines + terminal capabilities? + readline?). And make the most out of go's beautiful IO stack: this package should see a serial device as nothing more than a stream of bytes ([]byte) with hardware or software control-flow management.

It is commonly admitted that the TTY API is a messy melting pot of distinct concerns:

• line disciplines.
• serial device management including data transmission, control-flow/handshaking...
• non-canonical/raw mode for blocking/polling/timed reading modes.

Of course it made sense at the time it was built (the 80's is synonym of high space and time constraints), but only time constraints require special kernel attention nowadays (i.e. save time/be fast by letting a kernel handling special cases like clocking a serial line transmission or handling C-z signal asap)...

I even think the 100% user-land implementation in pure go could be experienced given the CPU/serial device speed ratio. This is for example what GNU readline does.