Julia: UnitRange broacasting with literals gives a Vector

[c42f]

The lowering of dotted operators can have surprising effects when literals are mixed with ranges. In particular:

julia> a = 1:4
1:4

julia> a .+ 1    # Not a range!
4-element Array{Int64,1}:
 2
 3
 4
 5

julia> b = 1
1

julia> a .+ b  # A range, as is probably desired
2:5

julia> broadcast(+, a, 1)  # Also a range
2:5

The reason is that a .+ 1 is lowered to broadcast(x->x+1, a), so this doesn't match the special broadcast rule broadcast(::typeof(+), r::AbstractRange, x::Number).

In 0.6 this perhaps wasn't a big problem as you could use +(r::Range, x::Number), but this has been deprecated. However, the problem isn't specific to ranges - it's a general problem for any situation where you'd like the output type to depend on the function being broadcasted.

[c42f]

@Keno this was your gripe from today.

[fredrikekre]

See #23445 and the linked issues/PRs. In particular https://github.com/JuliaLang/julia/issues/23445#issuecomment-331622677

[StefanKarpinski]

I really think we should revisit the special casing of literals in broadcast and see if we can’t get the same effect with constant propagation. @vtjnash has been working on that, so I wonder if it may soon be good enough for this.

[c42f]

Improved constant propagation sounds like a great plan.

If that doesn't pan out in the short term, perhaps we could change the lowering to allow access to the rawer form in a way which doesn't break things. It'd be ugly internally, but we could give access to both the original function and the closure capturing the literals if a .+ 1 lowered to broadcast_opt(broadcast_raw(+, a, 1), x->x+1, a)

And in turn we had the following default implementations:

struct NoRawBroadcast; end
broadcast_raw(args...) = NoRawBroadcast()
broadcast_opt(::NoRawBroadcast, f, args...) = broadcast(f, args...)
broadcast_opt(a, args...) = a

Users would then overload braodcast_raw(::typeof(+), ::AbstractRange, ::Number) to fix the literal problem.

I know, it's a pretty ugly solution. But it does show that lowering gives us some options to solve this with the current state of the compiler technology.

[timholy]

As part of #23939 we could give AbstractRange its own BroadcastStyle and appropriate combination rules.

[c42f]

Would that help? It seems to me we need something more like Jameson's work in https://github.com/JuliaLang/julia/pull/23692 to solve this issue.

[Edit] which is essentially attacking the same problem as the hack I proposed above, but in a more general and non-disgusting way :-)

[andyferris]

@timholy we need to specialize the output container based on the input containers and the operator, unitrange .+ integer --> unitrange, potentially unitrange .+ unitrange --> range, etc. Here it's that the dot fusion at lowering that masks the use of +.

Constant propagation may solve the literal problem but it's a symptom of fusion in general. I agree with @c42f that it seems to me we either need something like #23692, or perhaps to make broadcast lazy-and-not-fused-at-lowering.

[timholy]

Hmm, yes, here we do need to pay attention to the operator.

When it's operator-specific then we could even consider just short-circuiting the whole call. But that would presumably run into trouble with 1 .+ r .- 1. So either the combination rules need to take the operator into account, and/or we need #23692.

[JeffBezanson]

Ah, savor the sour fruits of breaking referential transparency.