Julia: Julep: extended proposal for fixing show, print, & friends

[vtjnash]

Problem statement:

Some objects have two irreconcilable representations: one view shows the structure of the object, while the other renders the content of the object. (historically in julia, these have been named show and print respectively).

The REPL usually wants to show the structure of an object, not just its value, so it wants to call show. This leads to the desired method call tree display(ans) -> display(d, ans) -> show(io, ans) -> print(io, 'ans')

On the other-hand, when outputting a document, the goal is to show only the value. This leads to the desired method call tree stringmime(mime, ans) -> mimewrite(io, mime, ans) -> print(io, ans) -> print(io, 'ans').

In short, display/show are a pair, and mimewrite/print are a pair, and there are important differences between the two. The first pair prints a representation, the second pair print the rendered content. Within each pair, there is also a difference that the first item is a document-based operation while the second is a streaming operation.

I posit that these are fundamental distinctions.

You might notice this is not the situation today, but currently there is some confusion over this in Base: mimewrite is being called by display, which has meant that it defaults to calling show to make the REPL case not look broken. But the Markdown code correctly implements this as print, making the actual resulting behavior inconsistent across types. The problem is, that means that doc-string printing works at the expense of Markdown objects not interacting properly in the REPL like other types.

By analogy to another parts of the Julia system, you could think of printing as the evaluation of a particular sort of AST tree, where evaluation = printing and AST tree = objects. For some objects, the evaluation step has no effect (e.g. print = show). For other objects (e.g. text and text-like documents), the evaluation step has the effect of stripping formatting from the string. I make this analogy because it hints at component potentially missing from our IO system: quote and interpolation nodes.

More food-for-thought:
Because show/print are streaming operations, it is generally valid to make multiple calls and assume that the end result will be a valid / cohesive unit.

By contrast, it is typically invalid to call mimewrite multiple times on an IO object. It should be assumed that mimewrite also writes all of the header and footer information to complete the file object.

Display is a bit different from mimewrite in that it is valid to call it multiple times with the same display object. However, since display manages the document context internally, it is generally assumed that it is a document creator (while mimewrite is a document writer). I assume this dichotomy is what drove the current design of display -> mimewrite -> (show | print), but I believe this may have been incorrect (per above).

Proposed solution elements:

1) Merge implementation of IOContext into base (#13825), for providing a general mechanism of tracking IO state via the IO stream itself (contrast with #13256 which provides a general mechanism for tracking document properties)
2) Transition to using more structured IO printing for handling basic formatting directives (see with_output_color in #13825 for the intended implementation of this)
3) define print(out::IO, io::IO) as equivalent to sendfile(out, in) (meaning that the rendered form of an IO object is the content that it contains). this has application for item 2
4) Add IOQuote (changes print -> show) and IOInterpolate (changes show -> print) types. (for example, @doc would return an IOInterpolate(MarkdownDoc). Although, I think IOInterpolate would still print some sort of header like "$type Rendering:\n" for text/plain, or make a scrollable frame for text/html).
5) decouple display from mimewrite:

• For usage, REPL display would call show to STDOUT. IJulia would call show to an HTMLBuilder
• printing a specialized format such as ‘text/html’ is implemented via an IO type (see HTMLBuilder example implementation in #13825). In addition to the methods shown at the above link, other types can override print(::HTMLBuilder, ::OtherType) to directly add HTML content. The fallback implementation of mimewrite for text/html would be print(io, "<html>", print(HTMLBuilder, value), "</html>").

I've completed a large portion of the work already in #13825 during my quest to better understand the nuances of this problem. (In particular, I believe IOContext and with_output_color are the complicated additions while the remaining pieces now are potentially just a bit of restructuring of the IO usage).

[StefanKarpinski]

Nice julep. Thanks for writing this up.

[StefanKarpinski]

[andreasnoack]

Defining new array printing is quite convoluted right now. How you thought about how that would look like in this proposal?

[jakebolewski]

Seems reasonable expect for the HTMLBuilder part. Henceforth the name "< Noun >Builder" is banished from the Julia language.

[StefanKarpinski]

Henceforth the name "< Noun >Builder" is banished from the Julia language.

Hear, hear.

[sjkelly]

I really like #13825. It would fit really well with serial port communications. It would make it easy to use a dictionary of Signals via Reactive.jl to share call backs when a certain message is sent or received to/from a device.

[vtjnash]

Defining new array printing is quite convoluted right now. How you thought about how that would look like in this proposal?

Yes. In fact, #13825 already addressed it (although a final round of cleanup and deprecation still remains).

expect for the HTMLBuilder part...

Haha, OK. the text/plain-Builder is named IOBuffer, so accordingly, the text/html-Builder should probably be called HTMLBuffer :). As a practical matter, I actually expect to remove this code entirely from the PR once the PR is finished and move it to a separate branch. I developed it here as a way of understanding the generic interactions.

It would fit really well with serial port communications

I'm not sure I follow what you meant, since that PR only deals with the output side. Matching them to the equivalent method on the input side does reveal a small clarification I need to make to the descriptive text in the above Julep, however: print and show always render as text (not binary). Thus print(non-text-object) defaults to calling show not write for non-text-objects such as arrays and numbers.

write -> read (aka serialize -> deserialize)
show -> parse [as an ast fragment]: e.g. Julia repr, although the actual repr function is more strict
print -> parse [as a fragment of a text type defined by the IO object type]: e.g. HTML fragment, or string fragment
mimewrite -> parse [as a mime-type document]: e.g. HTML document, or a plain text document
display -> n/a (generally irreversible since it has no explicit output channel / format)

[vtjnash]

Thus print(non-text-object) defaults to calling show not write for non-text-objects such as arrays and numbers.

trying to think out loud about this more, i think this statement might have been wrong. parse (like deserialize) is a special-case of read. this implies that print(x...) would be an alias for foreach(x) do x; write(x); end. however, this would break sprint / string. the current usage implies the following intent:

write -> byte I/O
show -> structure I/O
print -> text I/O

So to fix the sprint case after aliasing print to write I think that would then require a StringBuilderBuffer IO wrapper object type to change the "serialization" format from bytes to text. While this may sound a bit like #7959 (of making mimetype a required parameter of all write methods), I think this is different and better for two reasons: (1) I'm not sure the MIME solution generalizes beyond text/plain to handle structured formats (whereas HTMLBuilder exists to show how this solution does generalize); and (2) while that proposal requires rewriting all existing methods to be mime-aware, this proposal transparently reuses the existing IO parameter.

But trying to implement this poses a method ambiguity problem:

write(::IO, ::Any) = write(reinterpret(bytes))
write(::IO, ::AbstractString) = write(string)
write(::IO, ::Markdown) = write(render(string))

write(::StringBuilder, ::Any) = show() # method ambiguity!
write(::StringBuilder, ::is-overloaded-for-text-io) = write() # method ambiguity resolver

Which is a lot of text to arrive at the conclusion that I don't know how to sanely alias print and write, so they may need to remain independent.

[stevengj]

IJulia shouldn't call show to an HTMLBuilder. This is not how Jupyter works. You don't (generally) send rendered HTML, you send a "MIMEbundle" of different MIME representations of the object (text/html, text/latex, image/png, text/plain) and the front-end(s) picks which one to display and how. So, IJulia still needs to continue to call writemime to create multiple representations of an object to be displayed.

[stevengj]

The basic question in my mind is whether to attach metadata (like whether to print colors, whether to use compact or full display, etcetera) to the IO/IOContext object or to the MIME type (as suggested in #7959).

[vtjnash]

whether to attach metadata ... to the IO/IOContext object or to the MIME type

To summarize my reasons above, I'm not sure that adding metadata to MIME types generalizes particularly well beyond the unstructured text/plain format. But my strongest argument for the IOContext approach is that it doesn't require modifying any existing methods to add a new parameter. If the method correctly passes along IO, that it all that is required to encapsulate arbitrary additional metadata and type info.

IJulia shouldn't call show to an HTMLBuilder

Sorry, I didn't flesh that out very clearly since I intended it more as a rough example than a implementation guideline. It is likely true that there could be an IJuliaDisplay type that tries to create a representation of the object in several form and sends all of the ones that succeed to the frontend when display(ans) is called, for example. You are correct that it can continue to provide this.

It seems this needs another piece to the puzzle above: a conversion table from mime types to IO types as a generalization of the with_output_format / with_formatter / finalize_formatter methods mentioned in #2. This implements a transfer function from the value domain of mime type to the Julia type domain of IO object.

with_formatter(io::IO, ::MIME"text/plain") = io; finalize_formatter(io::IO, io::IO) = assert-that io == io;
with_formatter(io::IO, ::MIME"text/html") = HTMLFormatter(io); finalize_formatter(io, buf) = io << buf;
...

and the reverse:

mimetype(::IO) = MIME"text/plain"()
mimetype(::HTMLFormatter) = MIME"text/html"()
mimetype(io::IOContext) = mimetype(io.io)

which allows the use of THTT to implement output type overloading and transition freely between the two domains (I think this was where #7959 wanted to go with this as well, since this is finally integrating mimewrite with write, show, and the rest):

write(io::IO, md::Markdown) = write(io, mimetype(io), md)
write(io::IO, mime::MIME"text/plain", md::Markdown) = write(io, mime, rendered-as-text::AbstractString)
write(io::IO, mime::MIME"text/html", md::Markdown) = write(io, mime, rendered-as-html::AbstractString)
write(io::HTMLFormatter, mime::MIME"text/html", html::AbstractString) = adds-the-html-fragment-to-io
write(io::HTMLFormatter, mime::MIME"text/plain", text::AbstractString) = write(io, text)
write(io::IO, mime::MIME, value::Any) = write_with_format(mime, io, ans)

This effectively replaces mimewrite(io, mime, ans) with write_with_format(mime, io, ans) but also allows mimeshow(...) to be implemented as the one-liner with_output_format(show, mime, io, ans).

[vtjnash]

I think I realized the issue above is what Keno was trying to handle in #13256:
The concept of write involves two [mime] types -- input and output -- not one.

The old write methods required that both MIME types be implicit in the objects themselves, so that
write(io::IO, md::Markdown) was write( text/plain, text/md )

In #7959, the proposal was to provide annotation of the output type via an extra parameter:
write(io::IO, ::META"text/html", md::Markdown) was write( text/html, text/md )

In #13256, the proposal was to annotate the input type via julia types:
write(io::IO, md::MIMEData{text/md}) was write( text/plain, text/md )

In #13825, the proposal was to annotate the output type via julia types:
write(io::MIMEOutput{text/html}, md::Markdown) was write( text/html, text/md )

In this Julep, the challenge has been to understand how all of these proposals can be integrated.

In the old methods, the missing piece was that there was no generic way for either the input or output types to declare their mime content.

In #7959, the missing piece is that there was no way to annotate the input mime type (which make it hard to create generic structured IO writers), and it is inconvenient to pass around mime types when they can be generally implicit in the IO writer.

In #13256, the missing piece is that there was no way to annotate the output mime type (which makes it hard to write converters).

In #13825, like the old methods, there was no generic way for the input and output types to declare their mime content for method dispatch.

So here's my proposal for merging all of the above such that the user doesn't have to specify mime types, but that the system is optionally mime-aware where it matters:

# Basic byte IO
write(io::IO, data::Any) = io << reinterpret{bytes}(data)
write(io::IO, vector::Bytes) = io << vector
write(io::IO, str::String) = io << str
write(io::IO, char::Char) = io << char

# Basic MIME-aware declaration for input objects (THTT)
write(io::IO, md::Markdown) = write(io, mimetype(io), md)
write(io::IO, html::HTML) = write(io, mimetype(io), html)
write(io::IO, data::MIMEData) = write(io, mimetype(io), data)

# Basic MIME-half-aware declaration for text/plain objects
write(io::IO, mime::MIME"text/plain", str::String) = write(io, str)
write(io::IO, mime::MIME"text/plain", char::Char) = write(io, char)
mimetype(::IO) = MIME"text/plain"()

# Catch-all MIME writers fallback methods
write{mime<:MIME}(io::IO, ::mime, data::MIMEData{mime}) = write(io, data.vector)
write(io::IO, ::MIME"text/plain", data::MIMEData{MIME"text/plain"}) = write(io, data.vector) # ambiguity resolver for below
write(io::IO, mime::MIME"text/plain", obj::Any) = show(io, obj) # calls e.g. write(io, "obj")

# Markdown input (with specification of the output types it understands & generic `::IO`)
write(io::IO, mime::MIME"text/plain", md::Markdown) = write(io, mime, md as MIMEData{MIME"text/plain"}) # or write(io, md as String)
write(io::IO, mime::MIME"text/html", md::Markdown) = write(io, mime, md as MIMEData{MIME"text/html"})

# HTML output (with unknown object & unkown IO; and with specific override for catchall writer for the right mimetype)
write(io::IO, mime::MIME"text/html", obj::Any) = write_with_format(mime, io, MIME"text/plain"(), obj) # calls write(::HTMLOutput, ::MIME"text/plain", obj)
write(io::HTMLOutput, mime::MIME"text/html", dom::MIMEData{MIME"text/html"}) = append-child(io, dom)
write{imagetype}(io::IO, mime::MIME"text/html", data::MIMEData{MIME{imagetype}}) = if isimage(data)
    write(io, mime, "<img src=\"data:$imagetype;base64,$(base64encode(data.vector))\">")
  else
    write(io, MIME"text/plain", data)
  end
mimetype(::HTMLOutput) = MIME"text/html"()

edit: I forgot to add that, as mentioned in #7959 and above, we would have to provide a replacement for print also. I've added the missing mime declarations for Char and String above such that the following is correct:
print(io::IO, data...) = for-each(data) do x; write(io, mimetype(io), x); end

[vtjnash]

To better integrate into the current system, we could change the fallback text/plain definition above to call print:

write(io::IO, mime::MIME"text/plain", obj::Any) = print(io, obj) # instead of show

Any object (like Markdown) that wanted to overload this method would instead continue to overload print:

print(io::IO, str::String) = write(io, str) # instead of write(io::IO, mime::MIME"text/plain", str::String)
print(io::IO, char::Char) = write(io, char) # instead of write(io::IO, mime::MIME"text/plain", char::Char)
print(io::IO, md::Markdown) = write(io, md as String)

This effectively requires the usage of a short-hand method definition notation when the mime type is "text/plain", which is perhaps not a bad thing (instead of deprecating it).

The fallback print methods then are mostly unchanged, aside from being now becoming text/plain specialized and mime-aware otherwise:

print(io::IO, data...) = for-each(data) do x; print(io, x); end
print(io::IO, data::Any) = if is(mimetype(io), MIME"text/plain"())
    show(io, data)
else
    write(io, mimetype(io), x)
end

[stevengj]

@vtjnash, I agree that the backwards compatibility of putting the metadata in io is a big advantage.

We discussed replacing writemime(io::IO, mime, x) with write(io::IO, mime, x) in #7959, but when I tried implementing it in #8987 it turned into a big mess because of the ambiguity with write(io::IO, x...). What has changed?

[vtjnash]

I wasn't specifically aware of that method, since it doesn't seem particularly necessary. I could swap the argument order, but in reality, there's no need for the two-argument form of write and the three-argument form of writemime to have the same name.

In #8987, I see your concern was that mimewritable would break because of this. I'm afraid this Julep may make that worse for you, since it should now always return true for any mime-text format for which write_with_format is defined (e.g. everything can be rendered as text).

[stevengj]

So, how would IJulia decide whether to send a text/html representation of an object to the front-end?

[stevengj]

Swapping the argument order to write(::MIME, ::IO, x) seems workable. I've come to regret that writemime is a different name; _entia non multiplicanda sunt_.

[vtjnash]

i'm not sure IJulia can reliably determine that ahead-of-time. it could post-process the html and determine that it has no formatting marks (e.g. <) to conclude that the result provided no additional content beyond the same content rendered as text/plain.

but this Julep is supposed to allow for seamless merging mime-aware and mime-unaware objects, so that if a mime-unaware (text/plain) object tries to write a mime-aware object to a mime-aware stream, the child object will be able to render itself correctly. For example, this property is illustrated by the IOInterpolate type mentioned originally.

I've come to regret that writemime is a different name

can you explain this more? even if they share the same name, I believe they are relatively independent concepts (in all cases, distinguished by their argument number). and If they didn't share the same name, I believe it would make sense to define:
mimewrite(io::IO, x::Any) = mimewrite(io, mimetype(io), x)
which makes the mime-aware declaration a bit simpler:
write(io::IO, md::Markdown) = mimewrite(io, md)

[stevengj]

It's just that I dislike having more write-like functions than necessary. I agree that there's no particular problem in having them be different names, other than aesthetics.

Detecting HTML text seems likely to be unreliable; I'm worried about false positives. I suppose you could manually define mimewritable(::MIME"text/html", ::MyType) = false for objects likely to have false positives; there wouldn't be too many such objects, probably, but it still seems messy.

[vtjnash]

I'm not sure what you mean by unreliable. "text/plain" is a subset of all other text formats, so it is possible to render it into text/html losslessly in all cases. Whether such a transformation adds information (and thus is worth preserving) is typically not a easy question, however. Regardless, I believe it is not the same as the mimewritable question. For example, if you have a markdown object that consists of the text of this comment, that object is also perfectly valid as both text/html and text/plain (since I avoided the use of any meta-characters for either). While mimewritable is true for both format conversions, the minimal answer should be that this text is best transmitted as text/plain only.

[stevengj]

@vtjnash, suppose I am trying to display an object x in IJulia. I need to know whether to send a text/html representation, or just text/plain. If I send text/html, then it will be rendered as HTML. However, rendering as HTML causes a huge difference in appearance because text/plain rendering essentially uses <pre>, whereas text/html rendering ignores whitespace. For example, this is how an array is printed with text/plain vs how it is rendered if I force it to use text/html for the same text:

It sounds like, in your scheme, I would have to write x as text/html, and then use some unreliable heuristic to determine whether it was "intended" to be rendered as HTML. If the heuristic is wrong, then the rendering will look terrible.

[vtjnash]

The lossless representation of a plain text string in html requires the addition of <pre> tags, so I would actually intend for both of your examples to look like the first one (since a plain string would imply "text/plain", which requires modifications such as the addition of formatting tags and escaping of meta-charcters in order to be displayed correctly as "text/html")

[stevengj]

Automatically adding pre tags does not seem to be nestable. What if you print a Vector{Any} to a text/html stream, and some of the elements have specialized text/html writemime methods while other elements have only text/plain?

[vtjnash]

inline html tags are valid for nesting inside pre, including a, span (color), b, etc.

in general, i believe this nesting question is no harder than for printing a text/plain array: the first array above prints nicely only because the alignment and printing methods have been hand-customized to degrade gracefully as the content becomes more complex.

(note, my primary design goal with this part of the Julep is to fix https://github.com/JuliaLang/IJulia.jl/issues/260; although I plan to leave the final implementation of the html backend to a separate PR, or external package)

[stevengj]

Right, I was thinking of <pre> as being like a ``` code fence in Markdown, which is not nestable, but I misremembered.

[vtjnash]

Losslessly rending random text into Markdown is probably a harder but semi-related problem. I'm not sure many properties even can be represented (color? structure? nested attributes?)

I think my biggest question would be how to render mixed text: if the user inserts a small amount of text/html into a block of text, should it wrap the whole thing in a <pre> tag (so that it can render inline), or split the <pre> tag at the inlined content? I'm thinking I may try a hybrid approach that is conditional on whether the user has wrapped it with other formatting directives (like color, bold, paragraph, etc), but will probably need to see how the result looks and tweak it. (My goal is to make a pretty-good html report generator, not a perfect document DOM renderer -- the user should manually insert their own text/html where they want maximal control).

[StefanKarpinski]

Keep in mind that you don't need a markdown syntax for everything you can render. The way of getting colored text could be something like this:

"""
This is some **Markdown** text with $(Color(:red, "colorful text")).
"""

Then you support rendering Color span nodes to various output formats.

[vtjnash]

that's no longer a markdown document, but a text-serialized Julia AST

[hayd]

common markdown allows html blocks so perhaps we should just parse those, and render the tags we can/error out or ignore the tags we don't understand. Perhaps above would be <span color="red">...

[vtjnash]

@hayd github blocks any attempts at styling text (source: https://github.com/jch/html-pipeline/blob/master/lib/html/pipeline/sanitization_filter.rb), so i didn't suggest that, although it is certainly another option.

but it looks like the original intent was for markdown to intentionally lack these features, because you should use html for publishing (source: http://daringfireball.net/projects/markdown/syntax).

thus my same comment above applies: at that point it ceases to be a pure markdown document and becomes a text-serialized HTML AST (a general markdown writer would also have to be careful, since embedded html blocks have their own restrictions on when they will cause markdown processing to turn off entirely, and when it will resume)

[vtjnash]

@stevengj i've been thinking about the <pre> issue, and I think the following should render at least as well as the current text/plain conversion, but also allow inserting generic tags like color and <b> (and fail reasonably sanely on complex embedded html content):

html_tags[:pre] = ("<div style=\"white-space: pre-wrap; font-family: monospace; unicode-bidi: embed\">", "</div>")
with_formatter(io::IO, ::MIME"text/html") = HTMLFormatter(io, :pre)
finalize_formatter(io, buf) = (io << html_tags[:pre][1]; io << buf; io << html_tags[:pre][2])

this is a little bit different than the basic <pre> tag in that long lines will wrap instead of overflow, but i think that may be preferable. we can also just use <pre> or white-space: pre, the effect is the same (modulo any css selector matching).

then when inserting a complex html block (e.g. not from with_formatter), we can reset those properties:
<span style="white-space: initial; font-family: sans-serif; unicode-bidi: initial">

(post note, like Markdown, this part of the scheme isn't intended to be a complete document format, but just to handle many simple cases reasonably. complex formatting should still be done by something like Escher.jl, or by manually writing the intended html document)

[stevengj]

@vtjnash, that sounds promising.

[vtjnash]

@JeffBezanson can you post your gist of our discussions

my tl;dr is:

• define show(text/plain, io, x) = show(io, x) such that show is the canonical way of describing an object in text/plain
• define writemime(io, mime, x) = show(mime, io, x) (deprecate the writemime name) such that show is also the canonical way of producing a description of an object in the mime format
• define print(io, x) = show(io, x) + special cases for Strings such that print is the canonical way of writing data as text/plain
• define write(io, x) as the canonical way of writing x as bytes (including any implied conversions from the type of io, such as UTF16Stream, BigEndianStream, ZlibStream, HTMLStream, etc.)
• define type MIMEData{mime}; data::Any; end to mark content as being already of a specific mime type
• define write(io, x::MIMEData) = write(io, x.data) + various overrides such as write(io::HTML, x::MIMEData) = show(io, x) and write(io::HTML, x::MIMEData{text/html}) = io << x.data

no changes to the read methods were planned

(i think this replaces #7959)

[JeffBezanson]

Yes that's basically how I remember it too. I thought we were going to keep the argument order show(io, mime, x) though.

Here are some more details:

• showall/summary: replace with :verbosity, which has levels 0=summary, 1=limited, 2=all. The default is 1.
• print_with_color: replace with :color and :bold
• print_shortest: replace with :shortest Bool property
• showcompact: should be replaced with precision-related options. Maybe :precision or :decimals?
• sprint is not a good name since it doesn't call print
• print_joined, print_escaped, print_unescaped, escape_string, and unescape_string can go away

showerror is a major point of frustration. We can't figure out how to get rid of it. Maybe it should just be show, and you use dump if you want to inspect an exception object?

[StefanKarpinski]

+1 on keeping the order show(io, mime, x) – seems more intuitive and matches writemime, allowing us to just define writemime as a deprecated alias for show. The rest is as I recall it too.

[nalimilan]

I agree with the plan, but it seems to me that showcompact should be replaced with :verbosity just like showall. Indeed, it has to do with whether to include type information, and not only with the number of decimals. Better not limit the definition of "compact" to a criterion that only applied to numbers. Verbosity level 1=limited could be defined for scalars as "no type information" + "limited number of decimals" (container types would show the type, cf. https://github.com/JuliaLang/julia/pull/15963). Level 2=all would include type information and full decimals.

[stevengj]

@nalimilan, shouldn't the number of digits that you display (:precision) be orthogonal to whether you show all or part of a long array (:verbosity)?

[nalimilan]

@stevengj In principle it could. But in practice I'm not sure we want such a level of granularity. Are there cases in which we would want them to be dissociated?

I we do, then I'd say :precision is too specific to numbers. Maybe we could call it :compact or :detail? :verbosity is also a very broad term, but I can't find a better one.

[tkelman]

Aside from #16563, how much of the 0.5.0 part of this was not fixed yet by https://github.com/JuliaLang/julia/issues/16354?

[JeffBezanson]

I think this now might be done for 0.5. There are still some open items like dealing with color and more flexible number formatting, but I don't think we need to do them now.

@StefanKarpinski did you want to deprecate escape_string and unescape_string as well?

[StefanKarpinski]

@StefanKarpinski did you want to deprecate escape_string and unescape_string as well?

I think those are pretty useful, although I'm not sure they were ever really intended to be public.

[stevengj]

They are pretty useful for generating Julia code (e.g. deps.jl files) and inputs for other language.

[StefanKarpinski]

Let's just keep them for now. If we can figure out a better replacement in the future, great.

[JeffBezanson]

Fine with me. Moved to 0.6 for now. Or could be closed and replaced with more specific issues.

[StefanKarpinski]

+1 to closing and replacing with specific issues.

[stevengj]

Note that we need more documentation of this stuff, e.g. the IOContext docs are pathetically brief, and maybe a blog post to explain the evolution of show and friends would be good.

[vtjnash]

I've had a few questions of how / why IOContext is separate from MIME type. I've tried suggesting that the attributes in IOContext are somehow optional & recursive, but that's still a bit vague. I've realized a concise answer might be:

An attribute in IOContext should only be used to reduce, or limit, the amount of output.

I think this nicely sums up why certain attributes are part of it (such as contextual hints, recursion bookkeeping, display size, limit formatting), and other attributes should not be (such as MIME type, multiline).

[stevengj]

The fact that IOContext flags are optional is also important. MIME type is something you _have_ to dispatch on — you would never want to accidentally write text/plain when image/png is desired.

[stevengj]

Shouldn't showall be deprecated now that we have IOContext? And how does string differ from repr these days?

[StefanKarpinski]

What's left to be done here?

[nalimilan]

Maybe this? https://github.com/JuliaLang/julia/pull/18634#issuecomment-250996009

[vtjnash]

I think the main pieces are done. What remains seems to essentially be cleanup to make this solid for other packages to extend it:

• define write(out::IO, io::IO) as equivalent to sendfile(out, in) (meaning that the rendered form of an IO object is the content that it contains). this has application for item 2. already defined for IOBuffer, It's just not consistent right now.

• capture some of the tl;dr in the manual (https://github.com/JuliaLang/julia/issues/14052#issuecomment-168123944, https://github.com/JuliaLang/julia/issues/18004, https://github.com/JuliaLang/julia/issues/14052#issuecomment-250996874)

• print_with_color: replace with :color and :bold attributes, deprecate global has_color (c.f. :displaysize attribute, and https://github.com/JuliaLang/julia/tree/jn/iocolor)

• define type MIMEData{mime}; data::Any; end to mark content as being already of a specific mime type (e.g. https://github.com/JuliaLang/julia/pull/13256),
  and define write(io, x::MIMEData) = write(io, x.data) + various overrides such as write(io::HTML, x::MIMEData) = show(io, x) and write(io::HTML, x::MIMEData{text/html}) = io << x.data

• [later] Make a Julep to investigate transitioning to using more structured IO printing for handling basic formatting directives (see with_output_color in #13825 for one implementation of this and #9633 for another idea)

[StefanKarpinski]

This still needs a pass for 1.0 but it's mostly there.

[JeffBezanson]

Anything left for 1.0 here?

[nalimilan]

Are we happy with how print, repr and string are defined? As @stevengj asked above, do we really need repr, given that it can be replaced with sprint(show, x)? There's also reprmime.

[StefanKarpinski]

There was a proposal at some point to make repr stricter and only have it defined in cases where isequal(x, eval(parse(repr(x)))). I like that idea, but we'd need to delete a lot of repr methods.

[vtjnash]

Currently we define that repr is exactly shorthand for sprint(show, x). I think that's good.

julia> methods(repr)
# 1 method for generic function "repr":
[1] repr(x) in Base at strings/io.jl:168

[StefanKarpinski]

I'm fine with that too. Given how often I use "... $(repr(x)) ..." in messages, I would not want to have to write sprint(show, x) instead.

[StefanKarpinski]

Only documentation seems to be needed here.

[vtjnash]

On the topic of color / format:

I've decided to go in a different direction with handling formatting commands in the out, and so far am happy with how it's looking. The PR for this exploratory current work is https://github.com/JuliaLang/julia/pull/27430. So far, I like to think of this new approach as the dual of IOContext: where IOContext lets you pass additional metadata into the input, the new IOFormatBuffer lets you return additional metadata in the output. This gives the consumer ultimate control over formatting decisions: with an IOContext{IOFormatBuffer} as the input type, the IO object afterwards contains separate channels for data and formatting. The result object consists of a write-only IOBuffer of just the data to display, with a couple new functions to operate on it. In addition to the existing write(to, from) (copies the rest of stream from to to), there's a couple new methods for computing bulk text properties of any mark-able IO (currently textwidth, truncate_text, bytesavailable_until_text) for assisting with making formatting decisions. The main thing that this doesn't work well with is the sprint(; context=) call (that kwarg was awkward anyways), since it makes extra copies of data, while discarding the distinction between content and formatting information (and possibly discarding formatting information entirely). These should instead use an IOFormatBuffer object for the intermediate render and then write that directly to the output IO.

Anyways, I still need to do more work to finish it (esp. fixing up array_show, documenting it with examples, and adding :indent and :maxcolumns to handle the rest of what Markdown printing supports), but I wanted to give a bit of a sneak peek.

[StefanKarpinski]

Anything left to be done here? Write the docs?