Julia: Broadcast had one job (e.g. broadcasting over iterators and generator)

[vchuravy]

It was surprising to find that broadcast is not working with iterators

dict = Dict(:a => 1, :b =>2)
@show string.(keys(dict)) # => Expected ["a", "b"]
"Symbol[:a,:b]"

This is due to Broadcast.containertype returning Any https://github.com/JuliaLang/julia/blob/413ed79ec54f3a754ac8bc57c1d29835d17bd274/base/broadcast.jl#L31
leading to the fallback at: https://github.com/JuliaLang/julia/blob/413ed79ec54f3a754ac8bc57c1d29835d17bd274/base/broadcast.jl#L265

Defining containertype to be Array for that iterator lead to problems with calling size on it, because broadcast doesn't check against the iterator interface iteratorsize(IterType).

map solves this with the fallback map(f, A) = collect(Generator(f,A)) which may be more sensible that the current definition of broadcast(f, Any, A) = f(A)

[stevengj]

This is intentional. broadcast is for containers with shapes, and defaults to treating objects as scalars. map is for containers without shapes, and default to treating objects as iterators.

For example, broadcast treats strings as "scalars", whereas map iterates over the characters.

[pabloferz]

Maybe the problem is that people find the new dot syntax too convenient. There has been desire of having a compact way of expressing map in the past though. Unfortunately, the dot syntax is already taken.

[pabloferz]

Also, as @stevengj has pointed out before: there's got to be a difference between map and broadcast, if not, what is the point of having both.

[vchuravy]

@stevengj But Iterators do have shape (especially generators) http://docs.julialang.org/en/release-0.5/manual/interfaces/#interfaces

I would argue that iterators are in this awkward realm were most things that you would want to do with a container you also want to do with iterators, and yes maybe it is purely the fact that the . syntax is too convenient (and the error you get is very opaque).

@pabloferz The main difference between map and broadcast is the treatment of scalars. Now the definition of scalar is debatable and I would say that everything that has length(x) > 1 should not be considered a scalar.

[bramtayl]

Marking which arguments are to be treated as iterable, instead of the function call itself, would remove the ambiguity. I think?

[pabloferz]

For broadcast (I believe also in general) having shape, means having size (not just length) and being indexable. Except for tuples anything else without size is treated as scalar. Given the current implementation you first need getindex or being able to define one for the object you want to broadcast over. For iterators, that is not possible in general.

[rfourquet]

I ran into this too. Coming from #16769 where I look for a way to fill! an array with repeated evaluations of a function (instead of a fixed value), I thought dot-syntax may already do the trick. But, when a = zeros(2, 3); a .= [rand() for i=1:2, j=1:3] works, the (would be) cheaper a .= (rand() for i=1:2, j=1:3) doesn't; this generator is HasShape(), but has indeed no indexing capability. I'm very light in the understanding of how broadcast/dot-syntax works, but would it help here to have a trait for indexing capabilities? there is a PR (#22489) for that already...

[stevengj]

@rfourquet, you can do a = zeros(2, 3); a .= rand.()

[rfourquet]

Yes, but I should have be more precise: I want to use a function which gets the indices as parameters, like a .= (f(i, j) for i=1:2, j=1:3).

[nalimilan]

What would be the drawbacks of broadcasting dimensions of HasShape iterators? That sounds like a natural thing to do.

[stevengj]

@nalimilan, at first glance I think that would be reasonable and probably relatively easy to implement. Would be breaking so should be done by 1.0.

[stevengj]

One potential problem with this is that HasShape iterators don't necessarily support getindex, and that might make it tricky to implement?

[oscardssmith]

One possibility would be to temporarily (for 1.0) make simple implementation that just copied to an array. That would allow an optimization post 1.0

[rfourquet]

One potential problem with this is that HasShape iterators don't necessarily support getindex, and that might make it tricky to implement?

As I said above, I have a PR at #22489 to allow indexing into iterators, if this can help.

[nalimilan]

What needs to be done for 1.0 so that at least we can improve the behavior in 1.x?

[rfourquet]

Thanks @nalimilan for bringing that up, I wanted to do that as well. If allowing HasShape generators on the right hand side of broadcast expression is not possible to implement for 1.0, should we make this an error now, instead of treating generators as scalars? so that this can be enabled in 1.x.

[JeffBezanson]

:+1: Triage recommends making this an error (the safe choice) or calling collect on it (if easy to do).

map treats all of its arguments as containers, and tries to iterate over all of them. In my ideal world, broadcast would be similar, and treat all of its arguments has having shapes that can be broadcast, and give an error if e.g. size isn't defined. I'll point out that any value can be treated as a scalar in broadcast by wrapping it with fill, yielding a 0-d array:

julia> fill("a")
0-dimensional Array{String,0}:
"a"

julia> fill([2])
0-dimensional Array{Array{Int64,1},0}:
[2]

[nalimilan]

Do you really suggest treating all scalars as containers by default? That doesn't sound very practical.

Looking at how we could either support any iterable, or just throw an error for them until we support them, it looks like we would need a way to identify iterators in BroadcastStyle. Currently that's not possible, since Base.iteratorsize returns HasLength even for scalars like Symbol. We could introduce a Base.isiterable trait (which could be useful for other things), or make Base.iteratorsize default to NotIterable (which would make sense too as having HasLength as a default always sounds a bit surprising, if harmless).

[Sacha0]

(Tricky case for future discussion: UniformScaling.)

[nalimilan]

@timholy Since you've done the redesign of broadcast, any suggestions?

[stevengj]

@JeffBezanson, the whole point of broadcast is to be able to "broadcast" scalars to match containers, e.g. to do ["bug", "cow", "house"] .* "s" ----> ["bugs", "cows", "houses"]. This is fundamentally different from the behavior of map.

This is why broadcast treats objects as scalars by default, so that they can be combined with containers. Throwing an error for an unrecognized type would make it much less useful.

It should be possible to declare a particular type to be a container for broadcast by defining some appropriate method on it, but I think the default should continue to be to treat objects as scalars.

[nalimilan]

In an unrelated PR (https://github.com/JuliaLang/julia/pull/25339), @Keno suggested using applicable(start, (x,)) to find out whether x is iterable or not. Should we use the same approach here? I would find it clearer to have a more explicit definition of iterators (based either on Base.iteratorsize or on a trait), but using start also kind of makes sense.

[StefanKarpinski]

We could have an explicit trait which defaults to applicable(start, (x,)); that would allow overriding it if necessary.

[nalimilan]

I've filed #25356 to illustrate the possible solutions and their drawbacks.

[JeffBezanson]

From @stevengj 's example ["bug", "cow", "house"] .* "s" ----> ["bugs", "cows", "houses"], iterability doesn't seem to be enough, since strings are iterable but act like scalars there. If you need to define a trait anyway, it might be best to continue to require opt-in for broadcasting, rather than add requirements to all iterators.

Fortunately keys(dict) now returns an AbstractSet, so if we added a broadcast trait for AbstractSet it would fix the example in the OP. We could also add an error for broadcasting a Generator to catch some common cases.

[stevengj]

Broadcasting over AbstractSet containers seems inherently a bit problematic: you can combine an AbstractSet with a scalar, but not with any other container since the iteration order is unspecified for a set. This kind of breaks the usual meaning of a "broadcast" operation.

[nalimilan]

Yes, I realized when preparing the PR that sets are really not the best example of iterators which should support broadcast. Things like Generator and ProductIterator are much more interesting cases.

[JeffBezanson]

Maybe the answer is to (try to) broadcast iterators that have HasShape, and continue to treat everything else as scalars? Won't fix the OP, but is pretty elegant otherwise.

Other random thought: maybe broadcasting over 1 argument (as in string.(x)) should be a special case that works more like map, since shape compatibility isn't an issue?

[nalimilan]

Maybe the answer is to (try to) broadcast iterators that have HasShape, and continue to treat everything else as scalars? Won't fix the OP, but is pretty elegant otherwise.

I'm not sure we have a strong reason to exclude HasLength iterators. We support broadcasting over tuples (which do not implement size), so why not treat shapeless iterators like tuples? For example, it would make perfect sense to be able to use the result of keys(::OrderedDict) with broadcast. If we don't support it, people will be tempted to define their iterators as HasShape just to be usable with broadcast (and the nice dot syntax).

[JeffBezanson]

To quote Steve,

broadcast is for containers with shapes

HasShape seems to be a reasonable way to define that more precisely. Otherwise, it seems to me we'd need to break the behavior of broadcast on strings, for example.

[nalimilan]

We already have an inconsistency, with tuples being considered as containers and strings as scalars. Strings are very special anyway, I don't think their behavior is due to their not having a shape: it's rather related to the fact that they are the only collection which is more often considered as a scalar than as a container.

Maybe @stevengj can develop why he thinks broadcast should only support containers with a shape? Would you support considering tuples as scalars too?

[stevengj]

I think the rationale for treating tuples as containers in broadcast (#16986) was that in practice they are often used as essentially static vectors, and treating them as "scalars" in broadcast was just not very useful anyway. In contrast, strings (a) are often treated as "atoms" for string-processing operations and (b) don't have consecutive indexing in general so they fit very poorly into the broadcast framework.

In principle, I would support HasShape iterators being used as containers in broadcast. The main problem, as I noted above, is that having HasShape does not guarantee that getindex works.

[rfourquet]

The main problem, as I noted above, is that having HasShape does not guarantee that getindex works

Would something like #22489 help, i.e. having an iterator trait which indicates whether an iterator is indexable?

[nalimilan]

Would something like #22489 help, i.e. having an iterator trait which indicates whether an iterator is indexable?

But then only indexable iterators would be supported with broadcast? That sounds too restrictive, as would be very useful to be able to do things like string.(itr, "1") for any iterable (e.g. the result of keys(::OrderedDict)), and indexing isn't required to implement that. I think we'd better throw an error for all iterators which don't support indexing in 0.7/1.0, and try to support them in subsequent releases. It's not very useful to treat iterators as scalars anyway. Then we can implement whatever behavior we want in 1.x releases.

@stevengj I agree about your arguments regarding strings and tuples, but why shouldn't we treat HasLength iterators as tuples? I haven't read a justification for this until now.

[stevengj]

@nalimilan, I tend to think only indexable+hasshape iterators should be supported by broadcast. Trying to cram general iterators into this function confuses its meaning too much — at some point, you should just use map.

would be very useful to be able to do things string.(itr, "1") for any iterable … It's not very useful to treat iterators as scalars anyway.

The case of strings contradicts this — the "1" argument itself is iterable in your example. Tons of things are iterable (e.g. PyObjects in PyCall define start etc.), including things like unordered sets where the broadcast concept is really broken.

Note also that #24990 will make map even easier than it is now, e.g. you will be able to do map(string(_,"1"), itr).

[nalimilan]

@nalimilan, I tend to think only indexable+hasshape iterators should be supported by broadcast. Trying to cram general iterators into this function confuses its meaning too much — at some point, you should just use map.

We don't have a trait currently for indexable iterators. How would you suggest handing that? My WIP PR #25356 would throw an error for iterators which do not support indexing, which doesn't sound too bad assuming it's not very useful to treat iterators as scalars. If we want to treat them as scalars, we'd need another trait, right?

I'm inclined to raise errors for all cases which are not completely obvious, so that we can implement any behavior in the future, rather than locking ourselves into a default behavior which isn't necessarily very useful (i.e. treating some iterators as scalars). As this issue shows, the behavior of broadcast takes time to design correctly.

(FWIW, PyObject doesn't sound like a great example to me as IIUC it implements the iteration protocol just because it doesn't know in advance whether it will wrap a Python iterator or not. PyObject is clearly an exception here, just like it needs to use getfield overloading to appear like a standard Julia object. Sets are a more more Julian example.)

[stevengj]

We could add a trait for indexable HasShape iterators, as has been suggested elsewhere.

[vtjnash]

Triage likes the idea of make broadcast iterate over all arguments (like map), and adding an operator character (such as doing const & = Ref as proposed previously in another issue, or perhaps ~) to explicitly mark 0-d arguments.

[stevengj]

@vtjnash, what does that even mean for a non-HasShape iterator? Do you mean that you want broadcast to iterate over things like strings and sets? The current broadcast implementation is closely tied to getindex … have you thought about how you'd implement it without getindex, particularly for combining arguments of different dimensionality?

[nalimilan]

In theory it should be possible to support non indexable iterators (at least those which have a meaningful ordering). That's easy when all inputs have the same shape; when they are of different shapes and the iterator has a different (smaller) shape than the result, some intermediate storage would be needed.

[nalimilan]

Looks like the IteratorAccess trait from PR https://github.com/JuliaLang/julia/pull/22489 could be adapted/reused to detect indexable iterators. Knowing which iterators are indexable (and should therefore implement keys) is also needed for https://github.com/JuliaLang/julia/pull/24774.

Cc: @rfourquet

[nalimilan]

👍 Triage recommends making this an error (the safe choice) or calling collect on it (if easy to do).

Could triage decide on a specific strategy to adopt here? E.g. what's "this" in @JeffBezanson's comment above? Should we throw errors for all iterators which do not supporting indexing (safest choice for now, so that we can do anything we want later), or should we treat some iterators as scalars? Should we add a trait for indexable iterators, and if yes, under what form (new trait vs. new choice for Base.IteratorSize)? Should we add a trait for iterators in general (so that we can distinguish them from scalars)?

[JeffBezanson]

The following behavior seems good:

• By default, try to iterate and broadcast every argument.
• Give an error if that won't work for whatever reason.
• Pass Ref(x) or [x] to force x to be treated as a scalar.
• Add a trait that can be defined to allow a new type to be treated as a scalar instead of giving an error. Note this should not be used to pick between iterating vs. not iterating. It's just to turn the error into scalar behavior.

[yurivish]

Could you clarify the note on the last point (perhaps with an example)? I'm not sure what it means for the trait to exist but not be used to pick between iterating vs. not iterating.

[nalimilan]

So basically "try to iterate and broadcast every argument" implies that we need to define BroadcastStyle to return Scalar() for all non-collection types (notably Number, Symbol and AbstractString)? That sounds like the "trait" the last bullet mentions.

Honestly, I would find it less costly to define a trait for iterables than to define a trait for non-interables/scalars. I'm afraid that all non-collection types will at some point implement that Scalar trait, because that can be useful in some (possibly rare) cases.

[vtjnash]

That sounds like the "trait" the last bullet mentions

No, the last bullet means that if something implements iteration, then broadcast iterates it – the Scalar trait will be gone. For some common distinctly non-iterable types (such as subtypes of Type and Function), then we might want to have a NotIterable trait that turns the MethodError into an iteration that produces one value (that object). I don't actually remember why this was necessary though.

[Keno]

So basically "try to iterate and broadcast every argument" implies that we need to define BroadcastStyle to return Scalar() for all non-collection types (notably Number, Symbol and AbstractString)? That sounds like the "trait" the last bullet mentions.

No, all scalar subtypes of Number iterate themselves and so are fine. We would need to define it for symbol. AbstractString would operate as a collection.

[stevengj]

I dislike any design that requires us to define a method for a type to be treated as a scalar. That should be the default. I also don’t think strings should be treated as containers for broadcast.

I still think broadcast should treat only HasShape iterators as containers; this is consistent with the design of broadcast from the beginning. What is wrong with that?

[JeffBezanson]

The problem with that is the one in the OP; if you have an iterator without a shape, treating it as a scalar gives a crazy answer.

[JeffBezanson]

Also, I would be perfectly happy to drop the "trait" part of the proposal. Nobody complains about

julia> map(string, [1,2], :a)
ERROR: MethodError: no method matching start(::Symbol)

[jekbradbury]

Arguably the reason the result in the OP is unexpected is that nobody really intends a broadcast call to treat _all_ arguments as scalars; if there's only one argument and there's any way at all to treat it as a collection/iterator that is almost certainly what the user intends. Though of course 1 .+ 1 should continue to work?

[JeffBezanson]

That has occurred to me, but it seems confusing to make one argument a special case.

I see the following asymmetry: treating an iterable as a scalar gives really weird results, but treating a scalar as iterable gives an error. When you get the error, it's easy to fix by wrapping the argument. Whereas in the first case there's nothing simple you can do to get it to iterate over the argument.

[nalimilan]

I still think broadcast should treat only HasShape iterators as containers; this is consistent with the design of broadcast from the beginning. What is wrong with that?

@stevengj What's wrong IMHO is that it makes some operations not work when a perfectly reasonable behavior could be implemented: treat HasLength iterators just like Tuple, which is currently special-cased. Even if we don't support them right now, I'd like to leave open the possibility of supporting them at some point in 1.x.

Nobody complains about

julia> map(string, [1,2], :a)
ERROR: MethodError: no method matching start(::Symbol)

@JeffBezanson OTC, I support the current behavior of broadcast, which repeats :a as needed. This kind of thing can be very useful e.g. to rename a series of DataFrame columns. Do you suggest changing broadcast to throw an error like map?

I see the following asymmetry: treating an iterable as a scalar gives really weird results, but treating a scalar as iterable gives an error. When you get the error, it's easy to fix by wrapping the argument. Whereas in the first case there's nothing simple you can do to get it to iterate over the argument.

It's easy, but quite inconvenient. I agree with @stevengj that scalars should be broadcasted by default, not raise an error. Of course since Number types are iterable, the annoyance wouldn't always be visible, but as the Symbol example shows it wouldn't be very helpful in general. Char would be another one, and many custom types defined in package will also suffer from this (and end up defining their BroadcastStyle as Scalar()).

I think the crux of the issue is that we don't have a trait to distinguish collections from scalars. So the most direct solution was indeed to treat HasShape iterators as collections, and other types as scalars (including HasLength iterators since that's the default for all types). Personally I think introducing a trait for collections/iterables would make a lot of sense, but if we're not ready to do that and if we can't rely on start being defined to detect iterables, I'm afraid we'll have to keep the current behavior.

[mbauman]

Jeff's proposal in https://github.com/JuliaLang/julia/issues/18618#issuecomment-360594955 has room to allow for broadcast treating Symbol and Char as "scalar" types — they just need to opt-in to the behavior. By default they would be an error, as they do not implement start.

The most compelling part here is that the only sensible definition for a "collection" type is that it is iterable. Yes, that means that strings are collections. Sometimes they are used as such! So let's default to the behavior that easily allows folks to opt-in to the other at the call site.

There is a wart here, though. Since numbers are iterable (they even HasShape), they'll be treated as zero-dimensional containers. That means that, taken to its logical conclusion, 1 .+ 2 != 3. It'd be fill(3, ()) instead.

[bramtayl]

EDIT: in an attempt to avoid derailing the thread further, moved to discourse:

https://discourse.julialang.org/t/lazycall-again-sorry/8629

[nalimilan]

Jeff's proposal in #18618 (comment) has room to allow for broadcast treating Symbol and Char as "scalar" types — they just need to opt-in to the behavior. By default they would be an error, as they do not implement start.

Yeah, my position is just based on the assumption that scalars are a more natural fallback, especially given that collections need to implement some methods (iteration, possibly indexing) while scalars are just "the rest" and have nothing in common. In the end any type will be able to implement any behavior it wants, but we should make it as convenient and logical as possible, which in particular should help avoiding inconsistencies (e.g. some types declared and behaving as scalars and others not).

I'm not overly concerned with having a few exceptions for essential types like strings and numbers, as long as the rules are clear for other types.

[andyferris]

I've been thinking a little bit about our iteration and indexing interfaces. I note that we can have useful objects which iterate (but can not be indexed), objects that can be indexed (but are not iterable), and objects that do both. Based on that, I wonder if:

• map could be strongly tied to the iteration protocol - it seems valid that we can make a lazy out = map(f, iterable) for any arbitrary iterable such that e.g. first(out) is the same as f(first(iterable)), and it seems to me that this generic lazy operation could be useful.
• broadcast could be strongly tied to the indexing interface - it seems valid that we can make a lazy out = broadcast(f, indexable) such that out[i] is the same as f(indexable[i]), and it seems to me that this generic lazy operation could be useful. Obviously broadcast with multiple inputs could still do all the fancy stuff it does now. For the purpose of broadcasting, scalars would be those things that can't be indexed (or index trivially like Number and Ref and AbstractArray{0}).

I do also feel that it would be desirable if one-argument map and one-argument broadcast do very similar things for collections which are both iterable and indexable. However, the fact that AbstractDict iteration returns different things than getindex seems to block a nice unification here. :frowning_face: (Our other collection types seem fine)

(To me, the fact that things like strings may have to be explicitly wrapped up like ["bug", "cow", "house"] .* ("s",) doesn't sound like a deal-breaker here. I have the same problem when I want to think of a 3-vector as being a "single 3D point" and it's not too hard to deal with (xref #18379)).

[stevengj]

I agree that broadcast should be for indexable containers, but I think that should be consecutively indexable, which excludes strings. e.g. collect(eachindex("aαb🐨γz")) gives [1, 2, 4, 5, 9, 11], which will play badly with any broadcast implementation based on indexing.

But being for indexable containers is essentially that containers need a trait to opt-in, which is basically what I've been advocating.

[andyferris]

I'm not sure consecutive indices is a good constraint - dictionaries will have arbitrary indices, for example.

However, broadcast(f, ::String) can't create a new String and guarantee that the output indices remain the same as the input indices since the UTF-8 character widths might change under f (it would have to turn into something like an AbstractDict{Int, Char} to make that guarantee, which really doesn't seem very useful!). I would almost say that the indices of a String are more like "tokens" for fast lookup rather than semantically-important indices (e.g. you can convert to an equivalent UTF-32 string and the indices would change).

I don't mind if we make the broadcasting behavior opt-in via trait; I'm just saying that imagining how a generic broadcast(f, ::Any) behaves is a good way of guiding the implementation of things like broadcast(f, ::AbstractDict) (and would naturally answer the question I raised in #25904, i.e. broadcast over dictionary values and not key-value pairs).

[tkoolen]

Are people truly happy with this change? I for one have never needed to broadcast over a container without a shape, while I broadcast over things that should be treated as scalars all the time. Every deprecation warning I 'fix' makes me shed a tear.

[mbauman]

I broadcast over things that should be treated as scalars _all the time_.

What are the types of those things?

[tkoolen]

Can be anything. For example, in a package that defines an optimization model type Model and a decision variable type Variable, you might have x::Vector{Variable} for which you want to get the values after solving the model model using a function value(::Variable, ::Model)::Float64. Previously, you could do that like:

value.(x, model)

It's also often the case that the argument types are from other packages, so adding a method to broadcastable for those types would be type piracy in that case. So you have to use Ref or a one-element tuple. That's not insurmountable, but it just makes the common case a lot less elegant in order to support a relatively obscure usage pattern, in my opinion.

[mbauman]

Yes, I see your point, and I do agree that it is annoying in situations like that. That said, the old behavior was absolutely problematic — it was one of those "the default fallback is definitively wrong in some cases" things.

In short, there are four options that avoid the incorrect fallback:

1. require _everything_ to implement some method that describes how they broadcast
2. Default to treating things as containers and error/deprecate for non-containers.
   
   
   
   • We will just try to iterate unknown objects and that will error for scalars
   
   
   • There are two escape hatches for scalars — users can wrap them at a call site and library authors can opt-in to unwrapped scalar-like broadcasting.
   
   
3. Default to treating things as scalars and error for unknown containers
   
   
   
   • Given that there are no relevant methods only defined for scalars, we'd have to assert that iterate throws a method error. That is slow and circuitous.
   
   
   • There would only be one escape hatch available for custom containers to not error: their library authors to explicitly opt-in to broadcast. This seems quite backwards for a function whose primary purpose is to work with containers.
   
   
4. Check applicable(iterate, …) and switch behaviors accordingly
   
   
   
   • This currently doesn't work due to the deprecation mechanism from start/next/done, and in general could be wrong for wrapper types that defer methods to a member.
   
   

Option 1 is worse for everyone, option 2 is the status quo, and option 3 is backwards, and option 4 is something we've never done before and likely to be buggy.

[tkoolen]

I guess some of the discussion must have happened behind the scenes, but I'm just not convinced by the arguments I've seen in this thread and in https://github.com/JuliaLang/julia/pull/25356 against nalimilan's and stevengj's positions.

There would only be one escape hatch available for custom containers to not error: their library authors to explicitly opt-in to broadcast. This seems quite backwards for a function whose primary purpose is to work with containers.

This is my main point of disagreement. To me it seems that in all of Julia code # of iterator types << # of types that should be treated as scalars in a broadcast situation < # of broadcast calls. So I'd prefer it if the number of times something 'extra' needs to be done scales with the number of iterator types, rather than with the number of broadcast calls. And if a library author defines an iterator, it's not completely unreasonable to ask them to define one more method, whereas it _is_ completely unreasonable to ask every package author to define Base.broadcastable(x) = Ref(x) for all their non-iterable types in order to avoid ugly (IMHO) Refs in a high percentage of broadcast calls.

I know that having a single method to implement to define iteration is nice, but it's not that much work to implement one more for either a new trait, or for making it required to specify Base.iteratorsize for a new iterator (and getting rid of the problematic HasLength default). The fallback broadcastable method could then be based on that trait. Or, if you're really in love with defining iteration with a single method, you could (post-deprecation-removal) have that explicit trait default to applicable(iterate, ...) as in https://github.com/JuliaLang/julia/issues/18618#issuecomment-354618742, and simply override that default if need be. Corner cases like String could also still be handled by further specializing broadcastable if desired.

[mbauman]

That's effectively the design of 0.6, which led to this issue and #26421 and #19577 and #23197 and #23746 and possibly more — searching for this is hard.

It means that Base is providing a default fallback that is incorrect for a whole class of objects. That's why I prefer a mechanism that errors unless you opt in, one way or another. It's opinionated and the transition is a pain, but it forces you to be explicit.

You may be right that there are more "scalar-like" custom types than there are iterator-like ones, but I stand by the fact that broadcasting is first and foremost an operation on containers. I expect f.(x) to do some sort of mapping and not just be f(x).

And, finally, containers that get the scalar-default treatment simply aren't able to be used elementwise with broadcasting. For example, String is a collection type that we have special-cased to behave like a scalar; it's not possible to "reach in" and work elementwise, even though that would seem to make sense in some situations (e.g., isletter.("a1b2c3")). That's the asymmetry argument: you can more efficiently wrap containers in a Ref to treat them as scalars than you can collect them into an actually broadcastable collection.

Those are the main arguments. As far as the ugliness of Ref, I fully agree. A solution there is #27608.

[tkoolen]

Fair enough. I don't have any knockdown arguments or magic solutions to those problems, and https://github.com/JuliaLang/julia/pull/27608 will improve things.

[ederag]

@tkoolen I had the same concerns and use case.

@mbauman The arguments given above may not be fully convincing. Here are two questions to be more complete:

1) It would be possible to make broadcastable a required interface for any iterable.
This would be completely systematic, and would force developers to think about
how their iterator should behave under broadcast.
A recommendation to set it as collect(x) would make the transition relatively easy in most cases.
There would not be any performance loss, right ?

2) So it boils down to the will to have an error for f.(x) if x broadcasts as a scalar.
Why not a linter warning/error for f.(x, y, z), such as "all arguments of 'f' broadcast as scalars" ?

Anyway, it might be wise to fix #27563 (e.g. by #27608) and let users play with it a while before 1.0.
[0.7 and 1.0.0-rc1.0 were released without a fix].

[StefanKarpinski]

Anyway, it might be wise to fix #27563 (e.g. by #27608) and let users play with it a while before 1.0.
[0.7 and 1.0.0-rc1.0 were released without a fix].

I take it you missed the news that 1.0 has been released.

[ederag]

@StefanKarpinski Missed that, indeed. Congratulation to all developers, julia is amazing, keep on !