Julia: Proposal: keyword argument broadcasting

[jkrumbiegel]

Currently, broadcasting excludes keyword arguments:

>>> f(x; kw = 3) = x + kw

>>> f.([1, 2, 3]; kw = [4, 5, 6])

ERROR: MethodError: no method matching +(::Int64, ::Array{Int64,1})
Closest candidates are:
  +(::Any, ::Any, ::Any, ::Any...) at operators.jl:529
  +(::T, ::T) where T<:Union{Int128, Int16, Int32, Int64, Int8, UInt128, UInt16, UInt32, UInt64, UInt8} at int.jl:53

I don't know what the reasoning for this is, and I couldn't find previous issues about this. I often want to broadcast keyword arguments when I create different objects that offer keyword constructors. In these cases, syntactical convenience is often more important than pure performance to me. It would be a breaking change to enable keyword broadcasting without syntactical changes. However, the following syntax is still available:

>>> f.([1, 2, 3]; kw .= [4, 5, 6])

ERROR: syntax: invalid keyword argument syntax "kw .= [4, 5, 6]"

I propose adding the .= keyword syntax in order to pull selected keywords into the broadcasting expression.

[c42f]

ERROR: syntax: invalid keyword argument syntax

Note that this is actually a lowering error rather than a parser error, so you can create a macro which translates this syntax into a broadcasted closure as @mbauman noted on slack:

((x,y)->f(x; kw=y)).([1, 2, 3], [4, 5, 6])

That would allow you to experiment with the syntax and decide whether you like it in practice.

Personally I can see why you'd want this syntax and it has a pleasing consistency with the rest of broadcast notation.

[c42f]

To capture some other insightful things @mbauman noted on slack (rather than letting them disappear into the history):

Unfortunately the syntax isn’t available for f(x, kw .= y); that’ll assign y into kw and then pass it as a second positional argument.

f(x; kw .= y) is available though… but it’s a little fiddly to require ;s

The other question is what f(x; kw .= y) .+ 1 does — does that fuse into the .+? Do you consider f to be broadcast over the kwargs?

[tkf]

How does this interact with the idea of using f(; x, y) to do f(; x=x, y=y) #29333? To make broadcasting work with =-less kwfunc call, maybe use prefix . for the keyword part as in f.(; .x, y) and f.(; .x=x, y)?

[johnnychen94]

A lot of codes like this will be broken by then:

julia> struct Alg end

julia> function _diff(x, y; method)
           x - y
       end
_diff (generic function with 1 methods)

julia> X, Y = rand(4, 4), rand(4, 4);

julia> _diff.(X, Y; method=Alg()) # this currently works
# but would throw a MethodError like this by then
ERROR: MethodError: no method matching length(::Alg)
Closest candidates are:
  length(::Core.SimpleVector) at essentials.jl:596
  length(::Base.MethodList) at reflection.jl:852
  length(::Core.MethodTable) at reflection.jl:938

Codes in the wild https://github.com/JuliaGraphics/Colors.jl/pull/338

[c42f]

@johnnychen94 I think you misunderstand. This proposal is not to broadcast on all keywords, but to add a syntax to opt-in to broadcasting. Existing code would continue to work as before.

[jkrumbiegel]

How does this interact with the idea of using f(; x, y)

I didn't know about this syntax proposal yet. Actually I think your version f.(; .x, y) could work well in that case.

Unfortunately the syntax isn’t available for f(x, kw .= y); that’ll assign y into kw and then pass it as a second positional argument.

f(x; kw .= y) is available though… but it’s a little fiddly to require ;s

Yeah I also don't think it's optimal that the ; would be required, but I also don't find it so bad. People are used to having to write ; already for keyword splatting with ..., because otherwise it passes pairs as positional arguments. On the other hand, one could deprecate using the f(x .= y) syntax, but I don't have a feeling for how much people use that. It doesn't seem ideal to me to mutate a variable inside a parameter list. I certainly never do it.

Does the old restriction come from the previously lower performance of keyword arguments vs. positional arguments?

[jkrumbiegel]

The other question is what f(x; kw .= y) .+ 1 does — does that fuse into the .+? Do you consider f to be broadcast over the kwargs?

I would consider it to do the same as if kw was a normal positional argument. The dot would simply enable that argument for broadcasting, so it would act like any other positional argument.

The other option would of course be to just lift the broadcasting restriction on keyword arguments altogether, but that's too unlikely even for 2.0 with all the breakage it would cause.

[mbauman]

I think of keyword arguments as the "how to process" and the positional args as my "what to process." I find I'm much more likely to broadcast over the whats than the hows, but there are definitely cases where I've wanted the latter.

This would make for an interesting bifurcation wherein you could manually choose which kwargs participate in broadcast, but you cannot choose which positional arguments participate.

[jkrumbiegel]

I think of keyword arguments as the "how to process" and the positional args as my "what to process."

I agree, in many cases it is the same for me. But I actually like having keyword constructors for many of my structs because they are more descriptive when you see them in code. In those cases when I want to construct many different structs by keyword constructors, the broadcasting feature is missing.

but you cannot choose which positional arguments participate

I don't think that's completely true, people are using Ref all over the place to block positional arguments from broadcasting.

[mbauman]

Sure, to be more clear — it's opt-in whereas positional arguments are opt-out.

[StefanKarpinski]

This feels like a bridge too far syntactically. f.(x, kw .= y) looks like it's assigning from y into kw in-place, which is weird and not what's going on at all. This feel very much like a case where using a lambda would be better. Can you give some more actual motivating use cases?

[c42f]

This feels like a bridge too far syntactically. f.(x, kw .= y) looks like it's assigning from y into kw in-place

This was exactly my first impression and I originally wrote a reply saying so, but then deleted it :-)

I feel we've already committed to = in normal keyword syntax f(x, k = y) meaning something completely different depending on being inside vs outside of function call brackets. We've all gotten used to keyword syntax because because it's much more useful than doing assignment (and there's precedent in python etc). But there's nothing inherently natural about this syntax, and one can easily find opposite precedent in C where f(k=y) means assign y to k, then call f(y).

In general adding . means "do broadcast fusion with whatever other .s you can" so I think having this syntax mean "keyword broadcast" is actually more consistent than the current meaning of f(k .= y). And far more useful.

[tkf]

If the main argument against this is f(; k .= x) looks like an assignment, I wonder if f(; .k = x) solves the problem. As I commented above, it's directly extendable to f(; .x, y) for a broadcasting of f(; x, y).

[jkrumbiegel]

I feel we've already committed to = in normal keyword syntax f(x, k = y) meaning something completely different depending on being inside vs outside of function call brackets. We've all gotten used to keyword syntax because because it's much more useful than doing assignment

I agree with this, that's why I think keyword broadcasting syntax is acceptable. It should be only a minor effort to discern keyword use from variable assignment.

I wonder if f(; .k = x) solves the problem

I don't think this looks quite as nice as the other version at first, but this would allow to skip the semicolon, which would be valuable because of its syntactical consistency. So that the same syntax means the same thing left and right of the semicolon. So maybe that would be a good solution!

Can you give some more actual motivating use cases?

To me this often revolves around constructors, as one may want to offer several different options of constructing the same object. For example compare these hypothetical constructors:

# let's say we have some variables that don't immediately show their
# intended use through their names
ps # some points
rs # some radii
tris # some triangles
lines # some lines

circles = Circle.(center .= ps, radius .= rs)
# vs
circles = Circle.(ps, rs)

circles = Circle.(outer_triangle .= tris)
# vs
circles = Circle.(tris)

circles = Circle.(inner_triangle .= tris)
# vs
circles = Circle.(tris) # collides, so would need an extra method

circles = Circle.(center .= ps, tangent .= lines)
# vs
circles = Circle.(ps, lines)

Keywords can help making the intent of code clear, so you don't have to guess whats happening.

[jkrumbiegel]

A part of the problem might be that there is no dispatch on keyword arguments