Julia: Alternative syntax for `map(func, x)`

[kmsquire]

This was discussed in some detail here. I was having trouble finding it, and thought it deserved its own issue.

[quinnj]

+1

[toivoh]

Or func.(args...) as syntactic sugar for

broadcast(func, args...)

But maybe I'm the only one who would prefer that?
Either way, +1.

[ihnorton]

:-1: If anything, I think Stefan's other suggestion of f[...] has a nice similarity to comprehensions.

[johnmyleswhite]

Like @ihnorton, I'm also not super fond of this idea. In particular, I dislike the asymmetry of having both a .+ b and sin.(a).

[quinnj]

Maybe we don't need special syntax. With #1470, we could do something like

call(f::Callable,x::AbstractArray) = applicable(f,x) ? apply(f,x) : map(f,x)

right? Perhaps this would be too magical though, to get auto-map on any function.

[JeffBezanson]

@quinnj That one line summarizes my greatest fears about allowing call overloading. I won't be able to sleep for days.

[JeffBezanson]

Not yet sure it's syntactically possible, but what about .sin(x)? Is that more similar to a .+ b?

I think [] is getting way too overloaded and will not work for this purpose. For example, we'll probably be able to write Int(x), but Int[x] constructs an array and so cannot mean map.

[johnmyleswhite]

I would be onboard with .sin(x).

[StefanKarpinski]

We'd have to claw back some syntax for that, but if Int(x) is the scalar version then Int[x] is reasonable by analogy to construct a vector of element type Int. In my mind the opportunity to make that syntax more coherent is actually one of the most appealing aspects of the f[v] proposal.

[JeffBezanson]

How does making f[v] syntax for map make the syntax more coherent? I don't understand. map has a different "shape" than the current T[...] array constructor syntax. What about Vector{Int}[...]? Wouldn't that not work?

[quinnj]

lol, sorry for the scare @JeffBezanson! Haha, the call overloading is definitely a little scary, every once in a while, I think about the kinds of code obfuscation you can do in julia and with call, you could do some gnarly stuff.

I think .sin(x) sounds like a good idea too. Was there consensus on what to do with multi-args?

[jakebolewski]

:-1:. Saving a couple of characters compared to using higher order functions I don't think is worth the cost in readability. Can you imagine a file with .func() / func.() and func() interspersed everywhere?

[JeffBezanson]

It seems likely we'll remove the a.(b) syntax anyway, at least.

[kmsquire]

Wow, talk about stirring up a bees nest! I changed the name to better reflect the discussion.

[JeffBezanson]

We could also rename 2-argument map to zipWith :)

[ihnorton]

If some syntax is really necessary, how about [f <- b] or another pun on comprehensions _inside_ the brackets?

(@JeffBezanson you're just afraid that someone is going to write CJOS or Moose.jl :) ... if we get that feature, just put it in the Don't do stupid stuff: I won't optimize that section of the manual)

[StefanKarpinski]

Currently writing Int[...] indicates that you are constructing an array of element type Int. But if Int(x) means converting x to Int by applying Int as a function then you could also consider Int[...] to mean "apply Int to each thing in ...", oh which by the way happens to produce values of type Int. So writing Int[v] would be equivalent to [ Int(x) for x in v ] and Int[ f(x) for x in v ] would be equivalent to [ Int(f(x)) for x in v ]. Of course, then you've lost some of the utility of writing Int[ f(x) for x in v ] in the first place – i.e. that we can statically know that the element type is Int – but if enforce that Int(x) must produce a value of type Int (not an unreasonable constraint), then we could recover that property.

[JeffBezanson]

Strikes me as more vectorization/implicit-cat hell. What would Int[x, y] do? Or worse, Vector{Int}[x]?

[StefanKarpinski]

I'm not saying it's the best idea ever or even advocating for it – I'm just pointing out that it doesn't _completely_ clash with the existing usage, which is itself a bit of a hack. If we could make the existing usage part of a more coherent pattern, that would be a win. I'm not sure what f[v,w] would mean – the obvious choices are [ f(x,y) for x in v, y in w ] or map(f,v,w) but there are still more choices.

[jakebolewski]

I feel that a.(b) is hardly used. Ran a quick test and it is used in only in 54 of the ~4,000 julia source files in the wild: https://gist.github.com/jakebolewski/104458397f2e97a3d57d.

[JeffBezanson]

I think it does completely clash. T[x] has the "shape" T --> Array{T}, while map has the shape Array{T} --> Array{S}. Those are pretty much incompatible.

To do this I think we'd have to give up T[x,y,z] as a constructor for Vector{T}. Plain old array indexing, A[I] where I is a vector, can be seen as map(i->A[i], I). "Applying" an array is like applying a function (of course matlab even uses the same syntax for them). In that sense the syntax really works, but we would lose typed-vector syntax in the process.

[johnmyleswhite]

I kind of feel like debating syntax here distracts from the more important change: making map fast.

[JeffBezanson]

Obviously making map fast (which, by the way, needs to be part of a fairly thorough redesign of the notion of functions in julia) is more important. However going from sin(x) to map(sin, x) is very significant from a usability perspective, so to really kill vectorization the syntax is quite important.

[nalimilan]

However going from sin(x) to map(sin, x) is very significant from a usability perspective, so to really kill vectorization the syntax is quite important.

Fully agreed.

[toivoh]

I agree with @JeffBezanson that f[x] is pretty much irreconcilable with the current typed array constructions Int[x, y] etc.

[toivoh]

Another reason to prefer .sin over sin. is to finally allow to use e.g. Base.(+) to access the + function in Base (once a.(b) is removed).

[kmsquire]

When a module defines its own sin (or whatever) function and we want to use that function on a vector, do we do Module..sin(v)? Module.(.sin(v))? Module.(.sin)(v)? .Module.sin(v)?

[StefanKarpinski]

None of these options really seem good anymore.

[binarybana]

I feel like this discussion misses the meat of the problem. Ie: when mapping single argument functions to containers, I feel like the map(func, container) syntax is _already_ clear and succinct. Instead, it is only when dealing with multiple arguments that I feel we might could benefit from better syntax for currying.

Take for example the verbosity of map(x->func(x,other,args), container), or chain a filter operation to make it worse filter(x->func2(x[1]) == val, map(x->func1(x,other,args), container)).

In these cases, I feel a shortened map syntax would not help much. Not that I think these are particularly bad, but a) I don't think a short-hand map would help much and b) I love pining after some of Haskell's syntax. ;)

IIRC, in Haskell the above can be written filter ((==val) . func2 . fst) $ map (func1 other args) container with a slight change in the order of arguments to func1.

[rfourquet]

In elm .func is defined by x->x.func and this is very useful, see elm records. This should be considered before taking this syntax for map.

[StefanKarpinski]

I like that.

[StefanKarpinski]

Although field access isn't such a big deal in Julia as in many languages.

[rfourquet]

Yes it feels less relevant here as fields in Julia are kind of more for "private" use. But with the ongoing discussion about overloading field access, that may become more sensible.

[nalimilan]

f.(x) looks like the less problematic solution, if it wasn't for the asymmetry with .+. But keeping the symbolic association of . to "element-wise operation" is a good idea IMHO.

[rfourquet]

If current typed array construction can be deprecated, then func[v...] can be translated to map(func, v...), and the litteral arrays can then be written T[[a1, ..., an]] (instead of current T[a1, ..., an]).

[rfourquet]

I find sin∘vquiet natural also (when an array v is seen a an application from indexes to contained values), or more simply sin*v or v*[sin]' (which requires defining *(x, f::Callable)) etc.

[nalimilan]

Coming back to this issue with a fresh mind, I realized f.(x) can be seen as a quite natural syntax. Instead of reading it as f. and (, you can read it as f and .(. .( is then metaphorically an element-wise version of the ( function call operator, which is fully consistent with .+ and friends.

[johnmyleswhite]

The idea of .( being a function call operator makes me very sad.

[nalimilan]

@johnmyleswhite Care to elaborate? I was speaking about the intuitiveness of the syntax, or it's visual consistency with the rest of the language, not about the technical implementation at all.

[johnmyleswhite]

To me, ( isn't part of the language's semantics at all: it's just part of the syntax. So I wouldn't want to have to invent a way for .( and ( to start differing. Does the former generate a multicall Expr instead of a call Expr?

[nalimilan]

No. As I said I wasn't implying at all there should be two different call operators. Just trying to find a _visually_ consistent syntax for element-wise operations.

[StefanKarpinski]

To me what kills these options is the question of how to vectorize multi-argument functions. There's no single way to do it and anything that's general enough to support every possible way starts to look a lot like the multidimensional array comprehensions that we already have.

[JeffBezanson]

It's quite standard for multi-argument map to iterate over all arguments.
If we did this I would make .( syntax for a call to map. That syntax might
not be so great for various reasons, but I'd be fine with these aspects.

[nalimilan]

The fact that several generalizations are possible for multi-argument functions cannot be an argument against supporting at least some special cases -- just like matrix transpose is useful even if it can be generalized in several ways for tensors.

We just need to choose the most useful solution. Possible choices have already been discussed here: https://github.com/JuliaLang/julia/issues/8389#issuecomment-55953120 (and following comments). As @JeffBezanson said the current behavior of map is reasonable. An interesting criterion is to be able to replace @vectorize_2arg.

[johnmyleswhite]

My point is that having sin.(x) and x .+ y coexist is awkward. I'd rather have .sin(x) -> map(sin, x) and x .+ y -> map(+, x, y).

[JeffBezanson]

.+ actually uses broadcast.

Some other ideas, out of pure desperation:

1. Overload colon, sin:x. Doesn't generalize well to multiple arguments.
2. sin.[x] --- this syntax is available, currently meaningless.
3. sin@x --- not as available, but maybe possible

[StefanKarpinski]

I'm really not convinced that we need this.

[JeffBezanson]

Me neither. I think f.(x) is kind of the best option here, but I don't love it.

[nalimilan]

But without this how can we avoid creating all sorts of vectorized functions, in particular things like int()? This is what prompted me to start this discussion in https://github.com/JuliaLang/julia/issues/8389.

[johnmyleswhite]

We should encourage people to use map(func, x). It's not that much typing and it's immediately clear to anyone coming from another language.

[StefanKarpinski]

And of course, make sure that it's fast.

[nalimilan]

Yeah, but for interactive use I find it very painful. That's going to be a major annoyance for people coming from R (at least, I don't know about other languages), and I wouldn't like this to give the feeling that Julia is not suited for data analysis.

Another problem is consistency: unless you are willing to remove all currently vectorized functions, including log, exp, etc., and ask people to use map instead (which might be an interesting test of the practicality of this decision), the language is going to be inconsistent, making it hard to know in advance whether a function is vectorized or not (and on which arguments).

[johnmyleswhite]

Lots of other languages have been using map for years.

As I understood the plan to stop vectorizing everything, removing most/all vectorized functions was always part of the strategy.

[JeffBezanson]

Yes, of _course_ we would stop vectorizing everything. The inconsistency is already there: it's already hard to know which functions are or should be vectorized, since there is no really convincing reason why sin, exp etc. should be implicitly mapped over arrays.

And telling library writers to put @vectorize on all appropriate functions is silly; you should be able to just write a function, and if somebody wants to compute it for every element they use map.

[lindahua]

Let's imagine what happens if we remove commonly used vectorized math functions:

1. Personally, I don't mind writing map(exp, x) instead of exp(x), even though the latter is slightly shorter and cleaner. However, there exists a _big_ difference in performance. The vectorized function is about 5x faster than the map on my machine.
2. When you are working with compound expressions, the problem is more interesting. Consider a compound expression: exp(0.5 * abs2(x - y)), then we have

# baseline: the shortest way
exp(0.5 * abs2(x - y))    # ... takes 0.03762 sec (for 10^6 elements)

# using map (very cumbersome for compound expressions)
map(exp, 0.5 * map(abs2, x - y))   # ... takes 0.1304 sec (about 3.5x slower)

# using anonymous function (shorter for compound expressions)
map((u, v) -> 0.5 * exp(abs2(u - v)), x, y)   # ... takes 0.2228 sec (even slower, about 6x baseline)

# using array comprehension (we have to deal with two array arguments)

# method 1:  using zip to combine the arguments (readability not bad)
[0.5 * exp(abs2(u - v)) for (u, v) in zip(x, y)]  # ... takes 0.140 sec, comparable to using map

# method 2:  using index, resulting in a slightly longer statement
[0.5 * exp(abs2(x[i] - y[i])) for i = 1:length(x)]  # ... takes 0.016 sec, 2x faster than baseline 

If we are going to remove vectorized math functions, the only way that is acceptable in both readability and performance seems to be array comprehension. Still, they are not as convenient as vectorized math though.

[ViralBShah]

-1 for removing vectorized versions. In fact libraries such as VML and Yeppp offer much higher performance for vectorized versions and we need to figure out how to leverage these.

Whether these are in base or not is a different discussion and a larger discussion, but the need is real and performance can be higher than what we have.

[johnmyleswhite]

@lindahua and @ViralBShah: Some of your concerns seem to be predicated on the assumption that we'd get rid of vectorized functions before we made improvements to map, but I don't believe anyone has proposed doing that.

[nalimilan]

I think @lindahua's example is quite telling: the vectorized syntax is much nicer and much closer to the math formula than the other solutions. I would be quite bad to lose that, and people coming from other scientific languages are probably going to consider this as a negative point in Julia.

I'm fine with removing all vectorized functions (when map is fast enough), and see how it goes. I believe the interest of providing a convenience syntax will be even more visible at that point, and it will still be time to add it if it turns out it's the case.

I think Julia is different from many other languages because it emphasizes interactive use (longer expressions are annoying to type in that case), and mathematical computations (formulas should be as close as possible to math expressions to make the code readable). That's in part why Matlab, R and Numpy offer vectorized functions (the other reason is of course performance, an issue which can go away in Julia).

[lindahua]

My feeling from the discussion is that the significance of vectorized math is understated. Actually, one of the main advantages of vectorized math lies in the conciseness of expression -- it is much more than just a stopgap for "languages with slow for-loop".

Comparing y = exp(x) and

for i = 1:length(x)
    y[i] = exp(x[i])
end

The former is obviously much more concise and readable than the latter. The fact that Julia makes loops efficient does not mean that we should always de-vectorize codes, which, to me, is quite counterproductive.

We should encourage people write codes in a natural way. On one hand, this means that we should not try to write convoluted codes and twist vectorized functions in a context that they don't fit; on the other hand, we should support the use of vectorized math whenever they make the most sense.

In practice, mapping formulas to arrays of numbers is a very common operation, and we should strive for making this convenient instead of making it cumbersome. For this purpose, vectorized codes remain the most natural and concise way. Performance aside, they are still better than calling map function, especially for compound expressions with multiple arguments.

[ViralBShah]

We certainly don't want vectorized versions of everything, but using map every time to vectorize would be annoying for the reasons Dahua just mentioned above.

If map were fast, it would certainly let us focus on having a smaller and meaningful set of vectorized functions.

[johansigfrids]

I must say I come down strongly on the side of supporting a terse map notation. I feel it is the best compromise between the different needs.

I don't like vectorized functions. It hides what is going on. This habit of creating vectorized versions of functions leads to mystery code. Say you have some code where a function f from a package is called on a vector. Even if you have some idea of what the function does you can't be sure from reading the code whether it does it element-wise or works on the vector as a whole. If scientific computing languages didn't have a history of having these vectorized functions I don't think we would be nearly as accepting of them now.

It also leads to the situation where you are sort of implicitly encouraged to write vectorized versions of functions in order to enable terse code where the functions are used.

The code that is most explicit about what is going on is the loop, but as @lindahua says it ends up being very verbose, which has its own downsides, especially in a language that is also meant for interactive use.

This leads to the map compromise, which I feel is closer to the ideal, but I still agree with @lindahua that it is not terse enough.

Where I will disagree with @lindahua is that vectorized functions are the best choice, for the reasons I mentioned earlier. What my reasoning leads to is that Julia should have a very terse notation for map.

I find how Mathematica does it with its short-hand notation really appealing. The short-hand notation for applying a function to a argument in Mathematica is @, so you would Apply the function f to a vector as: f @ vector. The related short-hand notation for mapping a function is /@, so you map f to the vector as: f /@ vector. This has several appealing characteristics. Both short-hands are terse. The fact that both use the @ symbol emphasizes that there is a relationship between what they do, but the / in map still makes it visually distinct to make it clear when you are mapping and when you aren't. This is not to say Julia should blindly copy Mathematica's notation, only that good notation for mapping is incredibly valuable

I am not suggesting getting rid of all vectorized functions. That train has long since left the station. Rather, I suggest keeping the list of vectorized functions as short as possible and providing a good terse map notation to discourage adding to the list of vectorized functions.

All of this is, of course, conditional on map and anonymous functions being fast. Right now Julia is in a weird position. It used to be the case in scientific computing languages that functions were vectorized because loops are slow. This is not a problem. Instead, in Julia, you vectorized functions because map and anonymous functions are slow. So we are back where we started, but for different reasons.

[eschnett]

Vectorized library functions have one disadvantage -- only those functions explicitly provided by the library are available. That is, e.g. sin(x) is fast when applied to a vector, while sin(2*x) is suddenly much slower since it requires an intermediate array that needs to be traversed twice (first writing, then reading).

One solution would be a library of vectorizABLE math functions. These would be implementations of sin, cos, etc. that are available to LLVM for inlining. LLVM could then vectorize this loop, and hopefully lead to very efficient code. Yeppp seems to have the right loop kernels, but doesn't seem to expose them for inlining.

[johnmyleswhite]

Another problem with vectorization as a paradigm is that it just doesn't work at all if you use container types other the ones blessed by the standard library. You can see this in DataArrays: there's a ridiculous amount of metaprogramming code used to re-vectorize functions for the new container types we're defining.

Combine that with @eschnett's point and you get:

• Vectorized functions only work if you restrict yourself to the functions in the standard library
• Vectorized functions only work if you restrict yourself to the container types in the standard library

[lindahua]

I want to clarify that my point is not that we should always keep the vectorized functions, but that we need a way that is as terse as writing vectorized functions. Using map probably doesn't satisfy this.

I do like @eschnett's idea of tagging some functions as _vectorizable_, and the compiler can automatically map a vectorizable function to an array without requiring the users to explicitly defining a vectorized version. The compiler can also fuse a chain of vectorizable functions into a fused loop.

Here is what I have in mind, inspired by @eschnett's comments:

# The @vec macro tags the function that follows as vectorizable
@vec abs2(x::Real) = x * x
@vec function exp(x::Real) 
   # ... internal implementation ...
end

exp(2.0)  # simply calls the function

x = rand(100);
exp(x)    # maps exp to x, as exp is tagged as vectorizable

exp(abs2(x))  # maps v -> exp(abs2(v)), as this is applying a chain of vectorizable functions

The compiler may also re-vectorize the computation (at lower-level) by identifying the opportunity of using SIMD.

Of course, @vec should be made available to end user, so that people can declare their own functions as vectorizable.

[johnmyleswhite]

Thanks, @lindahua: your clarification helps a lot.

Would @vec be different from declaring a function @pure?

[lindahua]

@vec indicates that the function can be mapped in an element-wise manner.

Not all pure functions fall in this category, e.g. sum is a pure function, and I don't think it is advisable to declare it as _vectorizable_.

[johnmyleswhite]

Couldn't one rederive the vec property of sum from pure and associative tags on + along with knowledge about how reduce/foldl/foldr work when given pure and associative functions? Obviously this is all hypothetical, but were Julia to go all in on traits for types, I could imagine substantially improving the state of the art for vectorization by also going all in on traits for functions.

[binarybana]

I feel like adding new syntax is the opposite of what we want (after just cleaning out the special syntax for Any[] and Dict). The whole _point_ of removing these vectorized functions is to reduce special cases (and I don't think syntax should be different than function semantics). But I agree that a terse map would be useful.

So why not add a terse infix map operator? Here I'll pick $ arbitrarily. This would make @lindahua 's example go from

exp(0.5 * abs2(x - y))

to

exp $ (0.5 * abs2 $ (x-y))

Now, if we only had Haskell-like support for user defined infix operators, this would only be a one line change ($) = map. :)

IMO, the other syntax proposals are too visually close to existing syntax, and would require further mental effort to parse when looking through code:

• foo.(x) -- visually similar to standard type member access
• foo[x] -- am I accessing the x-th member of the foo array or calling map here?
• .foo(x) -- has problems as @kmsquire pointed out

And I feel like the @vec solution is too close to the @vectorize that we are trying to avoid in the first place. Certainly, some annotations ala #8297 would be nice to have and could help a future, smarter compiler can then recognize these stream fusion opportunities and optimize accordingly. But I don't like the idea of forcing it.

[johansigfrids]

Infix map plus fast anonymous functions could also do help with the creating of temporaries if it allowed you to do something like:

(x, y) -> exp(0.5 * abs2(x - y)) $ x, y

[tonyhffong]

I wonder if the idea from the cool new Trait.jl can be borrowed in the context of designating a function being vectorizable. Of course, in this case we are looking at individual _instances_ of Function type being vectorizable or not, instead of a julia Type having a specific trait.

[jiahao]

Now, if we only had Haskell-like support for user defined infix operators

6582 #6929 not enough?

[jiahao]

There's a point in this discussion about vectorizing entire expressions with as few array temporaries as possible. Users who want vectorized syntax won't want just a vectorized exp(x); they would want to write expressions like

y =  √π exp(-x^2) * sin(k*x) + im * log(x-1)

and have it magically vectorize

[eschnett]

It would not be necessary to mark functions as "vectorizable". This is rather a property of how the functions are implemented and mad available to Julia. If they are implemented e.g. in C, then they need to be compiled to LLVM bytecode (not object files) so that the LLVM optimizer can still access them. Implementing them in Julia would also work.

Vectorizability means that one implements the function in a way that is quite well described by the Yeppp project: No branches, no tables, division or square root if they are available as vector instructions in hardware, and otherwise many fused multiply-add operations and vector merge operations.

Unfortunately, such implementations will be hardware dependent, i.e. one may have to choose different algorithms or different implementations depending on what hardware instructions are efficient. I've done this in the past (https://bitbucket.org/eschnett/vecmathlib/wiki/Home) in C++, and with a slightly different target audience (stencil-based operations that are manually vectorized instead of an auto-vectorizing compiler).

Here in Julia, things would be easier since (a) we know that the compiler will be LLVM, and (b) we can implement this in Julia instead of C++ (macros vs. templates).

There is one other thing to consider: If one gives up parts of the IEEE standard, then one can greatly improve speed. Many users know that e.g. denormalized numbers are not important, or that the input will always be less than sqrt(max(Double)), etc. The question is whether to offer fast paths for these cases. I know that I will be very much interested in this, but others may prefer exactly reproducible results instead.

Let me cook up a vectorizable exp prototype in Julia. We can then see how LLVM does at vectorizing a loop, and what speeds we obtain.

[wlbksy]

Is it too scary to use full-width parentheses around function's argument~

[tonyhffong]

Sorry, I didn't realized I was repeating the exact same thing @johnmyleswhite was talking above re function with trait. Carry on.

[nalimilan]

@eschnett I don't think it's reasonable to link the API (whether functions are vectorizable or not) to implementation details (how the function is compiled). It sounds quite complex to understand and to keep stable in time and across architectures, and it wouldn't work when calling functions in external libraries, e.g. log wouldn't be detected as vectorizable since calls a function from openlibm.

OTOH @johnmyleswhite's idea of using traits to communicate what are a function's mathematical properties could be a great solution. (@lindahua's proposal is a feature I had suggested somewhere a while ago, but the solution of using traits may be even better.)

[binarybana]

Now, if we only had Haskell-like support for user defined infix operators

6582 #6929 not enough?

I should have said: ... user defined _non-unicode_ infix operators, as I don't think we want to require users to type unicode characters to access such a core functionality. Although I see that $ is actually one of those added, so thank you for that! Wow, so this actually works in Julia _today_ (even if it's not "fast"... yet):

julia> ($) = map
julia> sin $ (0.5 * (abs2 $ (x-y)))

[JeffBezanson]

I don't know if it's the best choice for map but using $ for xor really seems like a waste. Bitwise xor is not used that often. map is way more important.

@jiahao 's point above is a very good one: individual vectorized functions like exp are actually kind of a hack for obtaining vectorized _expressions_ like exp(-x^2). Syntax that does something like @devec would be really valuable: you'd get devectorized performance plus the generality of not needing to individually identify functions as vectorized.

The ability to use function traits for this would be cool, but I still find it less satisfying. What's really happening in general is that one person writes a function and another person iterates it.

[timholy]

I agree that this is not a property of the function, it's a property of the use of the function. The discussion about applying traits seems like a case of barking up the wrong tree.

[GlenHertz]

Brainstorming: how about you mark the arguments you want to map over so it would support multi-args mapping:

a = split("the quick brown")
b = split("fox deer bear")
c = split("jumped over the lazy")
d = split("dog cat")
e = string(a, " ", b., " ", c, " ", d.) # -> 3x2 Vector{String} of the combinations   
# e[1,1]: """["the","quick", "brown"] fox ["jumped","over","the","lazy"] dog"""

I'm not sure if .b or b. is better to show you want that mapped over. I like returning a multi-dimensional 3x2 result in this case as it represents the shape of the mapping.

Glen

[eschnett]

Here https://github.com/eschnett/Vecmathlib.jl is a repo with a sample
implementation of exp, written in a way that can be optimized by LLVM.
This implementation is about twice as fast as the standard exp
implementation on my system. It (probably) doesn't reach Yeppp's speed yet,
probably because LLVM doesn't unroll the respective SIMD loop as
aggressively as Yeppp. (I compared the disassembled instructions.)

Writing a vectorizable exp function is not easy. Using it looks like this:

function kernel_vexp2{T}(ni::Int, nj::Int, x::Array{T,1}, y::Array{T,1})
    for j in 1:nj
        @simd for i in 1:ni
            @inbounds y[i] += vexp2(x[i])
        end
    end
end

where the j loop and the function arguments are there only for
benchmarking purposes.

Is there an @unroll macro for Julia?

-erik

On Sun, Nov 2, 2014 at 8:26 PM, Tim Holy [email protected] wrote:

I agree that this is not a property of the function, it's a property of
the use of the function. The discussion about applying traits seems like a
case of barking up the wrong tree.

Reply to this email directly or view it on GitHub
https://github.com/JuliaLang/julia/issues/8450#issuecomment-61433026.

Erik Schnetter [email protected]
http://www.perimeterinstitute.ca/personal/eschnetter/

[jiahao]

individual vectorized functions like exp are actually kind of a hack for obtaining vectorized _expressions_ like exp(-x^2)

Core syntax for lifting entire expressions from the scalar domain would be very interesting. Vectorization is but one example (where the target domain is vectors); another interesting use case would be lifting into the matrix domain (#5840) where the semantics are quite different. In the matrix domain it would also be useful to explore how dispatch on different expressions could work, since in the general case you would want Schur-Parlett and other more specialized algorithms if you wanted something simpler like sqrtm. (And with clever syntax you could get rid of the *m functions entirely - expm, logm, sqrtm, ...)

[timholy]

Is there an @unroll macro for Julia?

using Base.Cartesian
@nexpr 4 d->(y[i+d] = exp(x[i+d])

[timholy]

(See http://docs.julialang.org/en/latest/devdocs/cartesian/ if you have questions.)

[nalimilan]

@jiahao Generalizing this to matrix functions sounds like an interesting challenge, but my knowledge about that it close to null. Do you have any ideas about how it would work? How would that be articulated with vectorization? How would the syntax allow making the difference between applying exp element-wise on a vector/matrix, and computing its matrix exponential?

[eschnett]

@timholy: Thanks! I didn't think of using Cartesian for unrolling.

Unfortunately, the code produced by @nexprs (or by manual unrolling) is not vectorized any more. (This is LLVM 3.3, maybe LLVM 3.5 would be better.)

[simonster]

Re: unrolling, see also @toivoh's post on julia-users. It may also be worth giving #6271 a shot.

[jiahao]

@nalimilan I haven't thought this through yet, but scalar->matrix lifting would be quite simple to implement with a single matrixfunc function (say). One hypothetical syntax (completely making something up) could be

X = randn(10,10)
c = 0.7
lift(x->exp(c*x^2)*sin(x), X)

which would then

1. identify source and target domains of the lift from X being of type Matrix{Float64} and having elements (type parameter) Float64 (thus implicitly defining a Float64 => Matrix{Float64} lift), then
2. call matrixfunc(x->exp(c*x^2)*sin(x), X) to compute the equivalent of expm(c*X^2)*sinm(X), but avoiding the matrix multiply.

In some other code X could be a Vector{Int} and the implied lifting would be from Int to Vector{Int}, and then lift(x->exp(c*x^2)*sin(x), X) could then call map(x->exp(c*x^2)*sin(x), X).

One could imagine also other methods that specified the source and target domains explicitly, e.g. lift(Number=>Matrix, x->exp(c*x^2)*sin(x), X).

[eschnett]

@nalimilan Vectorization isn't really a property of the API. With today's compiler technology, a function can only be vectorized if it is inline. Things mostly depend on the function implementation -- if it is written in the "right way", then the compiler can vectorize it (after inlining it into a surrounding loop).

[johnmyleswhite]

@eschnett: Are you using the same meaning of vectorization as others? It sounds like you're about SIMD, etc., which is not what I understand @nalimilan to mean.

[quinnj]

Right. There are two different notions of vectorization here. One deals
with getting SIMD for tight inner loops (processor vectorization). The main
issue being discussed here is syntax/semantics of somehow "automatically"
being able to call a single (or multi) argument function on a collection.

On Tue, Nov 4, 2014 at 7:04 PM, John Myles White [email protected]
wrote:

@eschnett https://github.com/eschnett: Are you using the same meaning
of vectorization as others? It sounds like you're about SIMD, etc., which
is not what I understand @nalimilan https://github.com/nalimilan to
mean.

—
Reply to this email directly or view it on GitHub
https://github.com/JuliaLang/julia/issues/8450#issuecomment-61738237.

[ntessore]

In symmetry to other . operators, should f.(x) not apply a collection of functions to a collection of values? (For example, for transforming from some n-d unit coordinate system to physical coordinates.)

[eschnett]

While discussing the syntax, the notion came up that using explicit loops to express the equivalent of map(log, x) was too slow. Hence, if one can make this fast enough, then calling map (or using a special syntax) or writing loops are equivalent on the semantic level, and one does not need to introduce a syntactic disambiguation. Currently, calling a vector-log function is much faster than writing a loop over an array, prompting people to ask for a way to express this distinction in their code.

[lindahua]

There are two levels of issues here: (1) syntax & semantics, (2) implementation.

The syntax & semantics issue is about how the user can express the intent of mapping certain computation in an element-wise/broadcasting manner to given arrays. Currently, Julia supports two ways: using vectorized functions and allowing users to write the loop explicitly (sometimes with the assistance of macros). Neither way is ideal. While vectorized functions allow one to write very concise expressions like exp(0.5 * (x - y).^2), they are have two problems: (1) it is difficult to draw a line as to which functions should provide a vectorized version and which not, thus often resulting in endless debates on the developer side and confusion on the user side (you often have to look up the doc to figure out whether certain functions are vectorized). (2) It makes it difficult to fuse the loops across function boundaries. At this point and probably several months/years to come, the compiler probably won't be able to perform such complex tasks as looking at multiple functions together, identifying the joint data flow, and producing optimized code path across function boundaries.

Using the map function addresses the issue (1) here. This, however, still does not provide any help in solving the issue (2) -- using functions, either a specific vectorized function or a generic map, always creates a function boundary that impedes the fusing of loops, which is critical in high performance computation. Using the map function also leads to verbosity, e.g. the expression above now becomes a longer statement as map(exp, 0.5 * map(abs2, x - y)). You may reasonably imagine that this issue would be aggravated with more complex expressions.

Among all the proposals outlined in this thread, I personally feel that using special notations to indicate mapping is the most promising way going forward. First of all, it maintains the conciseness of the expression. Take the $-notation for example, the expressions above can now be written as exp $(0.5 * abs2$(x - y)). This is a little bit longer than the original vectorized expression, but it is not too bad though -- what all it requires is to insert a $ to each call of a mapping. On the other hand, this notation also serves as an unambiguous indicator of a mapping being performed, which the compiler can utilize to break the function boundary and produce a fused loop. In this course, the compiler doesn't have to look at the internal implementation of the function -- all what it needs to know is that the function is going to be mapped to each element of the given arrays.

Given all the facilities of modern CPUs, especially the capability of SIMD, fusing multiple loops into one is only one step towards high performance computation. This step itself does not trigger the utilization of the SIMD instructions. The good news is that we now have the @simd macro. The compiler can insert this macro to the beginning of the produced loop when it thinks that doing so is safe and beneficial.

To sum up, I think the $-notation (or similar proposals) can largely address the syntax & semantics issue, while providing necessary information for the compiler to fuse loops and exploit SIMD, and thus emit highly performant codes.

[nalimilan]

@lindahua's summary is a good one IMHO.

But I think it would be interesting to extend this even further. Julia deserves an ambitious system which will make many common patterns as efficient as unrolled loops.

• The pattern of fusing nested function calls into a single loop should also be applied to operators, so that A .* B .+ C does not lead to the creation of two temporaries, but only one for the result.
• The combination of element-wise functions and reductions should be handled too, so that the reduction is applied on the fly after computing the value of each element. Typically, this will allow getting rid of sumabs2(A), replacing it with a standard notation like sum(abs$(A)$^2) (or sum(abs.(A).^2)).
• Finally, non-standard iteration patterns should be supported for non-standard arrays, so that for sparse matrices A .* B only needs to handle nonzero entries, and returns a sparse matrix. This would also be useful if you want to apply an element-wise function to a Set, a Dict, or even a Range.

The two last points could work by making element-wise functions return a special AbstractArray type, let's say LazyArray, which would compute its elements on the fly (similar to the Transpose type from https://github.com/JuliaLang/julia/issues/4774#issuecomment-59422003). But instead of accessing its elements naively by going over it using linear indexes from 1 to length(A), the iterator protocol could be used. The iterator for a given type would automatically choose whether row-wise or column-wise iteration is the most efficient, depending on the type's storage layout. And for sparse matrices, it would allow skipping zero entries (the original and the result would be required to have a common structure, cf. https://github.com/JuliaLang/julia/issues/7010, https://github.com/JuliaLang/julia/issues/7157).

When no reduction is applied, an object of the same type and shape as the original one would simply be filled by iterating over the LazyArray (equivalent of collect, but respecting the type of the original array). The only thing that is needed for that is that the iterator returns an object which can be used to call getindex on the LazyArray and setindex! on the result (e.g. linear or cartesian coordinates).

When a reduction is applied, it would use the relevant iteration method on its argument to iterate over the required dimensions of the LazyArray, and fill an array with the result (equivalent of reduce but using a custom iterator to adapt to the array type). One function (the one used in the last paragraph) would return an iterator going over all elements in the most efficient way; others would allow doing so over specific dimensions.

This whole system would also support in-place operations quite straightforwardly.

[mikewl]

I was thinking a little on addressing the syntax and thought of .= for applying elementwise operations to array.
So @nalimilan's example sum(abs.(A).^2)) would have to unfortunately be written in two steps:

A = [1,2,3,4]
a .= abs(A)^2
result = sum(a)

This would have the advantage of being easy to read and would mean that elementwise functions only need to be written for a single (or multiple) input and optimised for that case instead of writing array specific methods.

Of course, nothing other than performance and familiarity stops anyone from simply writing map((x) -> abs(x)^2, A) right now as has been stated.

Alternatively, surrounding an expression to be mapped with .() could work.
I do not know how difficult it would be to do this but .sin(x) and .(x + sin(x)) would then map the expression either inside the parenthesis or the function that follows the ..
This would then allow for reductions like @nalimilan's example where sum(.(abs(A)^2)) could then be written in a single line.

Both of these proposals use a . prefix which while using broadcast internally, made me think of elementwise operations on arrays. This could easily be swapped out with $ or another symbol.
This is just an alternative to putting a map operator around every function to be mapped and instead grouping the entire expression and specifying that to be mapped instead.

[nalimilan]

I've experimented with the LazyArray idea I exposed on my last comment: https://gist.github.com/nalimilan/e737bc8b3b10288abdad

This proof of concept does not have any syntactic sugar, but (a ./ 2).^2 would be translated to what's written in the gist as LazyArray(LazyArray(a, /, (2,)), ^, (2,)). The system works quite well, but it needs more optimization to be remotely competitive with loops with regard to performance. The (expected) problem seems to be that the function call at line 12 is not optimized (almost all of the allocations happen there), even in a version where additional arguments are not allowed. I think I need to parametrize the LazyArray on the function it calls, but I haven't figured out how I could do that, let alone handling arguments too. Any ideas?

[nalimilan]

Any suggestions about how to improve performance of LazyArray?

[lindahua]

@nalimilan I experimented with a similar approach a year ago, using functor types in NumericFuns to parameterize the lazy expression types. I tried a variety of tricks, but met no luck of bridging the performance gap.

The compiler optimization has been gradually improved over the past year. But I still feel that it is still not able to optimize out the unnecessary overhead. This kind of things requires the compilers to aggressive inline functions. You may try using @inline and see whether it makes things better.

[nalimilan]

@lindahua @inline doesn't make any difference on the timings, which is logical to me since getindex(::LazyArray, ...) is specialized for a given LazyArray signature, which does not specify which function should be called. I'd need something like LazyArray{T1, N, T2, F}, with F the function which should be called, so that when compiling getindex the call is known. Is there a way do that?

Inlining would be yet another great improvement, but at the moment the timings are much worse than even a non-inlined call.

[lindahua]

You may consider using NumericFuns and F can be a functor type.

[eschnett]

Dahua

I've needed functions where I know the return type for distributed
computing, where I create references to the result before the result (and
thus its type) is known. I've implemented a very similar thing myself, and
should probably switch to using what you call "Functors". (I don't like the
name "functor", since they are usually something else <
http://en.wikipedia.org/wiki/Functor>, but I guess C++ muddled the waters
here.)

I think it would make sense to split your Functor part from the
mathematical functions.

-erik

On Thu, Nov 20, 2014 at 10:35 AM, Dahua Lin [email protected]
wrote:

You may consider using NumericFuns and F can be a functor type.

Reply to this email directly or view it on GitHub
https://github.com/JuliaLang/julia/issues/8450#issuecomment-63826019.

Erik Schnetter [email protected]
http://www.perimeterinstitute.ca/personal/eschnetter/

[nalimilan]

@lindahua I've tried using functors, and indeed performance is much more reasonable:
https://gist.github.com/nalimilan/d345e1c080984ed4c89a

With functions:
# elapsed time: 3.235718017 seconds (1192272000 bytes allocated, 32.20% gc time)

With functors:
# elapsed time: 0.220926698 seconds (80406656 bytes allocated, 26.89% gc time)

Loop:
# elapsed time: 0.07613788 seconds (80187556 bytes allocated, 45.31% gc time) 

I'm not sure what more can be done to improve things, as the generated code does not seem optimal yet. I need more expert eyes to tell what's wrong.

[nalimilan]

Actually, the test above used Pow, which apparently gives a large speed difference depending on whether you write an explicit loop or use a LazyArray. I guess this has to do with the fusion of instructions which would only be performed in the latter case. The same phenomenon is visible with e.g. addition. But with other functions the difference is much smaller, either with a 100x100 or a 1000x1000 matrix, likely because they are external and thus inlining does not gain much:

# With sqrt()
julia> test_lazy!(newa, a);
julia> @time for i in 1:1000 test_lazy!(newa, a) end
elapsed time: 0.151761874 seconds (232000 bytes allocated)

julia> test_loop_dense!(newa, a);
julia> @time for i in 1:1000 test_loop_dense!(newa, a) end
elapsed time: 0.121304952 seconds (0 bytes allocated)

# With exp()
julia> test_lazy!(newa, a);
julia> @time for i in 1:1000 test_lazy!(newa, a) end
elapsed time: 0.289050295 seconds (232000 bytes allocated)

julia> test_loop_dense!(newa, a);
julia> @time for i in 1:1000 test_loop_dense!(newa, a) end
elapsed time: 0.191016958 seconds (0 bytes allocated)

So I'd like to find out why optimizations do not happen with LazyArray. The generated assembly is quite long for simple operations. For example, for x/2 + 3:

julia> a1 = LazyArray(a, Divide(), (2.0,));

julia> a2 = LazyArray(a1,  Add(), (3.0,));

julia> @code_native a2[1]
    .text
Filename: none
Source line: 1
    push    RBP
    mov RBP, RSP
Source line: 1
    mov RAX, QWORD PTR [RDI + 8]
    mov RCX, QWORD PTR [RAX + 8]
    lea RDX, QWORD PTR [RSI - 1]
    cmp RDX, QWORD PTR [RCX + 16]
    jae L64
    mov RCX, QWORD PTR [RCX + 8]
    movsd   XMM0, QWORD PTR [RCX + 8*RSI - 8]
    mov RAX, QWORD PTR [RAX + 24]
    mov RAX, QWORD PTR [RAX + 16]
    divsd   XMM0, QWORD PTR [RAX + 8]
    mov RAX, QWORD PTR [RDI + 24]
    mov RAX, QWORD PTR [RAX + 16]
    addsd   XMM0, QWORD PTR [RAX + 8]
    pop RBP
    ret
L64:    movabs  RAX, jl_bounds_exception
    mov RDI, QWORD PTR [RAX]
    movabs  RAX, jl_throw_with_superfluous_argument
    mov ESI, 1
    call    RAX

As opposed to the equivalent:

julia> fun(x) = x/2.0 + 3.0
fun (generic function with 1 method)

julia> @code_native fun(a1[1])
    .text
Filename: none
Source line: 1
    push    RBP
    mov RBP, RSP
    movabs  RAX, 139856006157040
Source line: 1
    mulsd   XMM0, QWORD PTR [RAX]
    movabs  RAX, 139856006157048
    addsd   XMM0, QWORD PTR [RAX]
    pop RBP
    ret

[eschnett]

The part up to jae L64 is an array bounds check. Using @inbounds may help (if appropriate).

The part below, where two consecutive lines start with mov RAX, ..., is a double indirection, i.e. accessing a pointer to a pointer (or an array of arrays, or a pointer to an array, etc.). This may have to do with LazyArray's internal representation -- maybe using immutables (or representing immutables differently by Julia) may help here.

In any way, the code is still quite fast. To make it faster, it would need to be inlined into the caller, exposing further optimization opportunities. What happens if you call this expression e.g. from a loop?

Also: What happens if you disassemble this not from the REPL but from within a function?

[toivoh]

Also can't help but notice that the first version carries out an actual
division while the second has transformed x/2 into a multiplication.

[nalimilan]

Thanks for the comments.

@eschnett LazyArray is already an immutable, and I'm using @inbounds in loops. After running the gist at https://gist.github.com/nalimilan/d345e1c080984ed4c89a, you can check what this gives in a loop with this:

function test_lazy!(newa, a)
    a1 = LazyArray(a, Divide(), (2.0,))
    a2 = LazyArray(a1, Add(), (3.0,))
    collect!(newa, a2)
    newa
end
@code_native test_lazy!(newa, a); 

So maybe all I need is to be able to force inlining? In my attempts, adding @inline to getindex does not change the timings.

@toivoh What could explain that in the latter case the division is not simplified?

[nalimilan]

I've continued experimenting with the two-argument version (called LazyArray2). Turns out for a simple operation like x .+ y, it's actually faster to use a LazyArray2 than the current .+, and it's also pretty close to explicit loops (these are for 1000 calls, see https://gist.github.com/nalimilan/d345e1c080984ed4c89a):

# With LazyArray2, filling existing array
elapsed time: 0.028212517 seconds (56000 bytes allocated)

# With explicit loop, filling existing array
elapsed time: 0.013500379 seconds (0 bytes allocated)

# With LazyArray2, allocating a new array before filling it
elapsed time: 0.098324278 seconds (80104000 bytes allocated, 74.16% gc time)

# Using .+ (thus allocating a new array)
elapsed time: 0.078337337 seconds (80712000 bytes allocated, 52.46% gc time)

So it looks like this strategy is viable to replace all element-wise operations, including .+, .*, etc. operators.

It also looks really competitive to achieve common operations like computing the sum of squared differences along a dimension of a matrix, i.e. sum((x .- y).^2, 1) (see again the gist):

# With LazyArray2 and LazyArray (no array allocated except the result)
elapsed time: 0.022895754 seconds (1272000 bytes allocated)

# With explicit loop (no array allocated except the result)
elapsed time: 0.020376307 seconds (896000 bytes allocated)

# With element-wise operators (temporary copies allocated)
elapsed time: 0.331359085 seconds (160872000 bytes allocated, 50.20% gc time)

[vchuravy]

@nalimilan
Your approach with LazyArrays seems to be similar to the way steam fusion works Haskell [1, 2]. Maybe we can apply ideas from that Area?

[1] http://citeseer.ist.psu.edu/viewdoc/summary?doi=10.1.1.104.7401
[2] http://citeseer.ist.psu.edu/viewdoc/summary?doi=10.1.1.421.8551

[nalimilan]

@vchuravy Thanks. This is indeed similar, but more complex because Julia uses an imperative model. On the contrary in Haskell the compiler needs to handle a large variety of cases, and even handle SIMD issues (which are handled by LLVM at a later point in Julia). But honestly I'm not able to parse everything in these papers.

[vchuravy]

@nalimilan I know the feeling. I found the second paper particular interesting since it discusses Generalized Stream Fusion, which apparently allows for a nice computation model over vectors.

I think the main point we should take from this that constructs like map and reduce in combination with laziness can be sufficiently fast (or even faster than explicit loops).

[PythonNut]

As far as I can tell, curly brackets are still available in call syntax. What if this became func{x}? Maybe a bit too wasteful?

On the topic of fast vectoring (in the sense of SIMD), is there any way we can emulate the way Eigen does it?

[nalimilan]

Here's a proposal to replace all current element-wise operations with a generalization of what I called LazyArray and LazyArray2 above. This of course relies on the assumption that we can make this fast for all functions without relying on functors from NumericFuns.jl.

1) Add a new syntax f.(x) or f$(x) or whatever which would create a LazyArray calling f() on each element of x.

2) Generalize this syntax following how broadcast works currently, so that e.g. f.(x, y, ...) or f$(x, y, ...) creates a LazyArray, but expanding singleton dimensions of x, y, ... so as to give them a common size. This of course would be done on the fly by computations on the indices so that the expanded arrays are not actually allocated.

3) Make .+, .-, .*, ./, .^, etc. use LazyArray instead of broadcast.

4) Introduce a new assignment operator .= or $= which would transform (calling collect) a LazyArray into a real array (of a type depending on its inputs via promotion rules, and of an element type depending on the element type of the inputs and on the called function).

5) Maybe even replace broadcast with a call to LazyArray and an immediate collection of the results into a real array.

Point 4 is a key one: element-wise operations would never return real arrays, always LazyArrays, so that when combining several operations, no copies are made, and loops can be fused for efficiency. This allows calling reductions like sum on the result without allocating the temporaries. So expressions of this sort would be idiomatic and efficient, for both dense arrays and sparse matrices:

y .= sqrt.(x .+ 2)
y .=  √π exp.(-x .^ 2) .* sin.(k .* x) .+ im * log.(x .- 1)
sum((x .- y).^2, 1)

I think returning this kind of lightweight object fits perfectly into the new picture of array views and Transpose/CTranspose. It means that in Julia you are able to perform complex operations very efficiently with a dense and readable syntax, though in some cases you have to explicitly call copy when you need a "pseudo-array" to be made independent from the real array it is based on.

This really sounds like an important feature. The current behavior of element-wise operators is a trap for new users, as the syntax is nice and short but the performance is usually terribly bad, apparently worse than in Matlab. Over just last week, several threads on julia-users had performance issues which would go away with such a design:
https://groups.google.com/d/msg/julia-users/t0KvvESb9fA/6_ZAp2ujLpMJ
https://groups.google.com/d/msg/julia-users/DL8ZsK6vLjw/w19Zf1lVmHMJ
https://groups.google.com/d/msg/julia-users/YGmDUZGOGgo/LmsorgEfXHgJ

[JeffBezanson]

For purposes of this issue, I'd separate the syntax from the laziness. Your proposal is interesting though.

There seems to come a point where there are just _so many dots_. The middle example in particular would be better written as

x .|> x->exp(-x ^ 2) * sin(k * x) + im * log(x - 1)

which requires only basic functions and an efficient map (.|>).

[timholy]

This is an interesting comparison:

y .=  √π exp.(-x .^ 2) .* sin.(k .* x) .+ im * log.(x .- 1)
y =  [√π exp(-x[i]^ 2) .* sin(k * x[i]) .+ im * log(x[i] - 1) for i = 1:length(x)]

If you discount the for ... part, the comprehension is only one character longer. I'd almost rather have an abbreviated comprehension syntax than all those dots.

A 1d comprehension doesn't preserve shape, but now that we have for i in eachindex(x) that could change, too.

[johnmyleswhite]

One problem with comprehensions is that they don't support DataArrays.

[davidanthoff]

I think it might be worthwhile to look at a whole bunch of things that happened on .Net that look very similar to the LazyArray idea. Essentially it looks pretty close to a LINQ style approach to me, where you have syntax that looks like the elementwise stuff we have right now, but really that syntax builds up an expression tree, and that expression tree then later gets evaluated in some efficient way. Is that somehow close?

On .Net they went far with this idea: you could execute these expression trees in parallel on multiple CPUs (by adding .AsParallel()), or you could run them on a large cluster with the DryadLINQ, or even on a http://research.microsoft.com/en-us/projects/accelerator/ (the latter might not have fully integrated with LINQ, but is close in spirit if I recall it correctly), or of course it could be translated into SQL if it the data was in that shape and you only used operators that could be translated into SQL statements.

My sense is that Blaze is also going in that direction, i.e. a way to easily construct objects that describe computations, and then you can have different execution engines for it.

Not sure this is very clear, but it does seem to me that this whole issue should be looked at in the context of both how one can generate low level efficient SIMD like code, and how this might be used for GPU computing, clustering, parallel computation etc.

[nalimilan]

Yes, you're right that the longer example has too many dots. But the two shorter ones are more typical, and in that case having a short syntax is important. I'd like to separate syntax from laziness, but as your comments show it appears to be very hard, we always mix the two!

One could imagine adapting the comprehension syntax, something like y = [sqrt(x + 2) over x]. But as @johnmyleswhite noted they should then support DataArrays, but also sparse matrices and any new array type. So this is again a case of mixing syntax and features.

More fundamentally, I think two features that my proposal offers over the alternatives are:
1) Support for allocation-free in-place assignment using y[:] = sqrt.(x .+ 2).
2) Support for allocation-free reductions like sum((x .- y).^2, 1).

Could that be provided with other solutions (disregarding syntax issues)?

@davidanthoff Thanks, looking at it now (I think LazyArray could be made to support parallel computing too).

[mbauman]

Perhaps this could be combined with Generators — they are also a sort of lazy array. I somewhat like the [f(x) over x] comprehension syntax, although it could be conceptually difficult for newcomers (as the same name is effectively being used for both the elements and the array itself). If comprehensions without brackets would create a generator (as I played with a long time ago), then it'd be natural to use these new over-x-style-comprehensions without brackets to return a LazyArray instead of immediately collecting it.

[nalimilan]

@mbauman Yes, generators and lazy arrays share many properties. The idea of using brackets to collect a generator/lazy array, and not adding them to keep the lazy object sounds cool. So regarding my examples above, one would be able to write both 1) y[:] = sqrt(x + 2) over x and sum((x - y)^2 over (x, y), 1) (though I find it natural even for newcomers, let's leave the issue of over for the bikeshedding session and concentrate on the fundamentals first).

[JeffBezanson]

I like the f(x) over x idea. We could even use f(x) for x to avoid a new keyword. In fact [f(x) for x=x] already works. We would then need to make comprehensions equivalent to map, so they can work for non-Arrays. Array would just be the default.

[johansigfrids]

I must admit I've also come to like the over idea. One difference between over as map and for in list comprehension is what happens in case of multiple iterators: [f(x, y) for x=x, y=y] results in a matrix. For the map case you generally still want a vector, ie [f(x, y) over x, y] would be equivalent to [f(x, y) for (x,y) = zip(x, y)]]. Because of this I still think introducing an additional keyword over is worth it, because, as this issue has raised, maping over multiple vectors is very common and needs to be terse.

[nalimilan]

Hey, I convinced Jeff about the syntax! ;-)

[JeffBezanson]

This belongs beside #4470, so adding to 0.4-projects for now.

[krcz]

If I understand the gist of discussion the main problem is that we want to get mapping-like syntax that:

• works with various data types, like DataArrays, not only native arrays;
• is as fast as manually written loop.

It might be possible to do it using inlining, but being really careful to make sure inlining works.

What about different approach: using macro depending on inferred data type. If we can infer that data structure is DataArray, we use map-macro provided by DataArrays library. If it is SomeKindOfStream, we use stream library provided one. If we cannot infer type, we just use dynamically dispatched general implementation.

This might force data structures creators to write such macros, but it would be needed only if its author wanted it to have really efficient execution.

If what I'm writing is unclear, I mean something like [EXPR for i in collection if COND] could be translated to eval(collection_mapfilter_macro(:(i), :(EXPR), :(COND))), where collection_mapfilter_macro is chosen based on inferred collection type.

[JeffBezanson]

No, we don't want to do things like that. If DataArray defines map (or the equivalent), its definition should always be called for DataArrays regardless of what can be inferred.

This issue actually isn't about implementation, but syntax. Right now many people are used to sin(x) implicitly mapping if x is an array, but there are many problems with that approach. The question is what alternate syntax would be acceptable.

[mikewl]

1) Support for allocation-free in-place assignment using y[:] = sqrt.(x .+ 2)
2) Support for allocation-free reductions like sum((x .- y).^2, 1)

y = √π exp(-x^2) * sin(k*x) + im * log(x-1)

Looking at these three examples from others I think with the for syntax this would end up something like this:
1) y[:] = [ sqrt(x + 2) for x ])
2) sum([ (x-y)^2 for x,y ], 1)
and
y = [ √π exp(-x^2) * sin(k*x) + im * log(x-1) for x,k ]

I like this quite a lot! The fact it creates a temporary array is quite explicit and it is still readable and terse.

Minor question though, could x[:] = [ ... for x ] have some magic to mutate the array without allocating a temporary one?
I am not sure if this would provide much benefit but I can imagine it would help for large arrays.
I can believe though that it may be a completely different kettle of fish which should be discussed someplace else.

[nalimilan]

@Mike43110 Your x[:] = [ ... for x ] could be written x[:] = (... for x), the RHS creating a generator, which would be collected element by element to fill x, without allocating a copy. That was the idea behind my LazyArray experiment above.

[mschauer]

The [f <- y] syntax would be nice if combined with a Int[f <- y] syntax for a map which knows its output type and does not need to interpolate from f(y[1]) what the other elements will be.

[mschauer]

Especially, as this gives an intuitive interface for mapslices as well, [f <- rows(A)] where rows(A) (or columns(A) or slices(A, dims)) returns a Slice object so dispatch can be used:

map(f, slice::Slice) = mapslices(f, slice.A, slice.dims)

[JeffBezanson]

When you add indexing, this gets a bit harder. For example

f(x[:,j]) .* g(x[i,:])

It's hard to match the conciseness of that. The blowup from comprehension style is pretty bad:

[f(x[m,j])*g(x[i,n]) for m=1:size(x,1), n=1:size(x,2)]

where, to make matters worse, cleverness was required to know that this is a case of nested iteration, and can't be done with a single over. Although if f and g are a bit expensive, this might be faster:

[f(x[m,j]) for m=1:size(x,1)] .* [g(x[i,n]) for _=1, n=1:size(x,2)]

but even longer.

This kind of example seems to argue for "dots", as that could give f.(x[:,j]) .* g.(x[i,:]).

[nalimilan]

@JeffBezanson I'm not sure what's the intention of your comment. Has anybody suggested to get rid of the .* syntax?

[JeffBezanson]

No; I'm focusing on f and g here. This is an example where you can't just add over x at the end of the line.

[nalimilan]

OK, I see, I had missed the end of the comment. Indeed the dots version is nicer in that case.

Though with array views, there will be a reasonably efficient (AFAICT) and not so ugly alternative:
[ f(y) * g(z) for y in x[:,j], z in x[i,:] ]

[jdbates]

Could the above example be solved by nesting over keywords?

f(x)*g(y) over x,y

is interpreted as

[f(x)*g(y) for (x,y) = zip(x,y)]

whereas

f(x)*g(y) over x over y

becomes

[f(x)*g(y) for x=x, y=y]

Then, the specific example would above would be something like

f(x[:,n])*g(x[m,:]) over x[:,n] over x[m,:]

EDIT: In retrospect, that's not nearly as concise as I thought it would be.

[davidagold]

@JeffBezanson How about

f(x[:i,n]) * g(x[m,:i]) over i

gives the equivalent of f.(x[:,n] .* g.(x[m,:]). The new syntax x[:i,n] means that i is being introduced locally as an iterator over the indices of the container x[:,n]. I don't know if that's possible to implement. But it seems (prima facie) neither ugly nor cumbersome, and the syntax itself gives bounds for the iterator, namely 1:length(x[:,n]). As far as keywords are concerned, "over" can signal that the entire range is to be used, whereas "for" can be used if the user wishes to specify a subrange of 1:length(x[:,n]):

f(x[:i,n]) * g(x[m,:i]) for i in 1:length(x[:,n])-1.

[StefanKarpinski]

@davidagold, :i already means the symbol i.

[davidagold]

Ah yes, good point. Well, as long as dots are fair game what about

f(x[.i,n]) * g(x[m,.i]) over i

where the dot indicates that i is being introduced locally as an iterator over 1:length(x[:,n). I suppose in essence this switches the dot notation from modifying the functions to modifying the arrays, or rather their indices. This would save one from the "dot creep" Jeff noted:

[ f(g(e^(x[m,.i]))) * p(e^(f(y[.i,n]))) over i ]

as opposed to

f.(g.(e.^(x[m,:]))) .* p.(e.^(f.(y[:,n])))

although I suppose the latter is slightly shorter. [EDIT: also, if it is possible to omit the over i when there is no ambiguity, then one actually has a slightly shorter syntax:

[ f(g(e^(x[m,.i]))) * p(e^(f(y[.i,n]))) ] ]

One potential advantage of the comprehension syntax is that it could allow for a wider range of patterns of element-wise operation. For instance, if the parser understood that indexing with i in x[m, .i] is implicitly modulo length(x[m,:]), then one could write

[ f(x[.i]) * g(y[.j]) over i, j=-i ]

To multiply the elements of x against the elements of y in the opposite order, i.e. the first element of x against the last element of y, etc. One could write

[ f(x[.i]) * g(y[.j]) over i, j=i+1 ]

to multiply the ith element of x by the i+1th element of y (where the last element of x would be multiplied by the first element of y due to indexing being understood in this context as modulo length(x)). And if p::Permutation permutes (1, ..., length(x)) one could write

[ f(x[.i]) * g(y[.j]) over i, j=p(i) ]

to multiply the ith element of x by the p(i)th element of y.

Anyway, that's just a humble opinion from an outsider on an entirely speculative issue. =p I do appreciate the time anybody takes to consider it.

[bramtayl]

A souped up version of vectorize that will use r style recycling could be pretty useful. That is, arguments which do not match the size of the largest argument are extended via recycling. Then users can easily vectorize anything they want, regardless of the number of arguments etc.

unvectorized_sum(a, b, c, d) = a + b + c + d
vectorized_sum = @super_vectorize(unvectorized_sum)

a = [1, 2, 3, 4]
b = [1, 2, 3]
c = [1, 2]
d = 1

A = [1, 2, 3, 4]
B = [1, 2, 3, 1]
C = [1, 2, 1, 2]
D = [1, 1, 1, 1]

vectorized_sum(a, b, c, d) = vectorized_sum(A, B, C, D) = [4, 7, 8, 8]

[johnmyleswhite]

I tend to think that recycling trades off too much safety for convenience. With recycling, it's very easy to write buggy code that executes without raising any errors.

[tkelman]

The first time I read about that behavior of R, I immediately wondered why someone would ever think that was a good idea. That's a really odd and surprising thing to do implicitly on mismatched-size arrays. There might be a handful of cases where it's how you would want to extend the smaller array, but just as well you might want zero-padding or repeating the end elements, or extrapolation, or an error, or any number of other application-dependent choices.

[bramtayl]

Whether or not to use @super_vectorize would be put in the hands of the user. It would also be possible to give warnings for various cases. For example, in R,

c(1, 2, 3) + c(1, 2)
[1] 2 4 4
Warning message:
In c(1, 2, 3) + c(1, 2) :
  longer object length is not a multiple of shorter object length

[tkelman]

I have no objections to making that an optional thing that users can choose whether or not to use, but it doesn't need to be implemented in the base language when it can just as well be done in a package.

[bramtayl]

@vectorize_1arg and @vectorize_2arg are both already included in Base, and the options they give a user seem somewhat limited.

[johnmyleswhite]

But this issue is focused on the design of a system for removing @vectorize_1arg and @vectorize_2arg from Base. Our goal is to remove vectorized functions from the language and to replace them with a better abstraction.

[JeffBezanson]

For example, recycling can be written as

[ A[i] + B[mod1(i,length(B))] for i in eachindex(A) ]

which to me is pretty close to the ideal way to write it. Nobody needs to build it in for you. The main questions are (1) can this be made more concise, (2) how to extend it to other container types.

[mikewl]

Looking at @davidagold's proposal I wondered if var: couldn't be used for that kind of thing where the variable would be the name before the colon. I saw this syntax used to mean A[var:end] so it does seem to be available.

f(x[:,j]) .* g(x[i,:]) would then be f(x[a:,j]) * g(x[i,b:]) for a, b which isn't too much worse.

Multiple colons would be slightly weird though.

f(x[:,:,j]) .* g(x[i,:,:]) -> f(x[a:,a:,j]) * g(x[i,b:,b:]) for a, b was my initial thought on this.

[bramtayl]

Ok, so here's a brief stab at a program for recycling. It should be able to handle n-dimensional arrays. It would probably be possible to incorporate tuples analagously to vectors.

using DataFrames

a = [1, 2, 3]
b = 1
c = [1 2]
d = @data [NA, 2, 3]

# coerce an array to a certain size using recycling
coerce_to_size = function(argument, dimension_extents...)

  # number of repmats needed, initialized to 1
  dimension_ratios = [dimension_extents...]

  for dimension in 1:ndims(argument)

    dimension_ratios[dimension] = 
      ceil(dimension_extents[dimension] / size(argument, dimension))
  end

  # repmat array to at least desired size
  if typeof(argument) <: AbstractArray
    rep_to_size = repmat(argument, dimension_ratios...)
  else
    rep_to_size = 
      fill(argument, dimension_ratios...)
  end

  # cut down array to exactly desired size
  dimension_ranges = [1:i for i in dimension_extents]
  dimension_ranges = tuple(dimension_ranges...)

  rep_to_size = getindex(rep_to_size, dimension_ranges...)  

end

recycle = function(argument_list...)

  # largest dimension in arguments
  max_dimension = maximum([ndims(i) for i in argument_list])
  # initialize dimension extents to 1
  dimension_extents = [1 for i in 1:max_dimension]

  # loop through argument and dimension
  for argument_index in 1:length(argument_list)
    for dimension in 1:ndims(argument_list[argument_index])
      # find the largest size for each dimension
      dimension_extents[dimension] = maximum([
        size(argument_list[argument_index], dimension),
        dimension_extents[dimension]
      ])
    end
  end

  expand_arguments = 
    [coerce_to_size(argument, dimension_extents...) 
     for argument in argument_list]
end

recycle(a, b, c, d)

mapply = function(FUN, argument_list...)
  argument_list = recycle(argument_list...)
  FUN(argument_list...)
end

mapply(+, a, b, c, d)

Clearly, this isn't the most elegant or fast code (I'm a recent R immigrant). I'm not sure how to get from here to a @vectorize macro.

EDIT: combined redundant loop
EDIT 2: separated out coerce to size. currently only works for 0-2 dimensions.
EDIT 3: One slightly more elegant way of doing this would be to define a special type of array with mod indexing. That is,

special_array = [1 2; 3 5]
special_array.dims = (10, 10, 10, 10)
special_array[4, 1, 9, 7] = 3

EDIT 4: Things I am wondering exist because this was difficult to write: a n-dimensional generalization of hcat and vcat? A way to fill an n-dimensional array (matching the size of a given array) with lists or tuples of the indices of each particular position? An n-dimensional generalization of repmat?

[pao: syntax highlighting]

[StefanKarpinski]

You really don't want to define functions with the foo = function(x,y,z) ... end syntax in Julia, although it does work. That creates a non-constant binding of the name to an anonymous function. In Julia, the norm is to use generic functions and the bindings to functions are automatically constant. Otherwise you're going to get terrible performance.

[JeffBezanson]

I don't see why repmat is necessary here. Arrays filled with the index of each position are also a warning sign: it shouldn't be necessary to use a large chunk of memory to represent so little information. I believe such techniques are really only useful in languages where everything needs to be "vectorized". It seems to me the right approach is just to run a loop where some indexes are transformed, as in https://github.com/JuliaLang/julia/issues/8450#issuecomment-111898906.

[bramtayl]

Yes, that makes sense. Here's a start, but I'm having trouble figuring out how to do the looping at the end and then make a @vectorize macro.

function non_zero_mod(big::Number, little::Number)
  result = big % little
  result == 0 ? little : result
end

function mod_select(array, index...)
  # just return singletons
  if !(typeof(array) <: AbstractArray) return array end
  # find a new index with moded values
  transformed_index = 
      [non_zero_mod( index[i], size(array, i) )
       for i in 1:ndims(array)]
  # return value at moded index
  array[transformed_index...]
end

function mod_value_list(argument_list, index...)
  [mod_select(argument, index...) for argument in argument_list]
end

mapply = function(FUN, argument_list...)

  # largest dimension in arguments
  max_dimension = maximum([ndims(i) for i in argument_list])
  # initialize dimension extents to 1
  dimension_extents = [1 for i in 1:max_dimension]

  # loop through argument and dimension
  for argument_index in 1:length(argument_list)
    for dimension in 1:ndims(argument_list[argument_index])
      # find the largest size for each dimension
      dimension_extents[dimension] = maximum([
        size(argument_list[argument_index], dimension),
        dimension_extents[dimension]
      ])
    end
  end

  # more needed here
  # apply function over arguments using mod_value_list on arguments at each position
end

[ScottPJones]

In the talk, @JeffBezanson mentioned the syntax sin(x) over x, why not something more like:
sin(over x)? (or use some character instead of having over as a keyword)

[dphildebrandt]

Once this is solved we can also resolve #11872

[jrevels]

I hope I'm not late to the party, but I'd just like to offer a +1 to @davidagold's syntax proposal. It's conceptually clear, terse, and feels really natural to write. I'm not sure whether . would be the best identifying character, or how feasible an actual implementation would be, but one could make a proof-of-concept using a macro to try it out (essentially like @devec, but might even be easier to implement).

[jrevels]

It also has the benefit of "fitting in" with existing array comprehension syntax:

result = [g(f(.i), h(.j)) over i, j]

vs.

result = [g(f(_i), h(_j)) for _i in eachindex(i), _j in eachindex(j)]

The key difference between the two being that the former would have more restrictions on shape-preservation, as it implies a map.

[mdcfrancis]

over, range, and window have some prior art in the OLAP space as modifiers to iteration, this seems consistent.

I'm not keen on the . syntax as that seems like a creep towards line noise.

$ is perhaps consistent, intern the values of iterating i,j into the expression?

result = [g(f($i), h($j)) over i, j]

For automatic vectorization of an expression can we not taint one of the vectors in the expression and have the type system lift the expression into the vector space?

I do similar to with time series operations where the expressiveness of julia already allows me to write

ts_a = GetTS( ... )
ts_b = GetTS( ... ) 
factors = [ 1,  2, 3 ]

ts_x = ts_a * 2 + sin( ts_a * factors ) + ts_b 

which when observed outputs a time series of vectors.

The main part missing is the ability to automatically lift existing functions into the space. This has to be done by hand

Essentially I'd like to be able to define something like the following ...

abstract TS{K}
function {F}{K}( x::TS{K}, y::TS{K} ) = tsjoin( F, x, y ) 
# tsjoin is a time series iteration operator

and then be able to specialize for specific operations

function mean{K}(x::TS{K}) = ... # my hand rolled form

[ma-laforge]

Hi @JeffBezanson,

If I understand correctly, I would like to propose a solution to your JuliaCon 2015 comment regarding a comment made above:
"[...] And telling library writers to put @vectorize on all appropriate functions is silly; you should be able to just write a function, and if somebody wants to compute it for every element they use map."
(But I will not address the other fundamental issue "[..] no really convincing reason why sin, exp etc. should be implicitly mapped over arrays.")

In Julia v0.40, I have been able to get a solution somewhat nicer (in my opinion) than @vectrorize:

abstract Vectorizable{Fn}
#Could easily have added extra argument to Vectorizable, but want to show inheritance case:
abstract Vectorizable2Arg{Fn} <: Vectorizable{Fn}

call{F}(::Type{Vectorizable2Arg{F}}, x1, x2) = eval(:($F($x1,$x2)))
function call{F,T1,T2}(fn::Type{Vectorizable2Arg{F}}, v1::Vector{T1}, v2::Vector{T2})
    RT = promote_type(T1,T2) #For type stability!
    return RT[fn(v1[i],v2[i]) for i in 1:length(v1)]
end

#Function in need of vectorizing:
function _myadd(x::Number, y::Number)
    return x+y+1
end

#"Register" the function as a Vectorizable 2-argument (alternative to @vectorize):
typealias myadd Vectorizable2Arg{:_myadd}

@show myadd(5,6)
@show myadd(collect(1:10),collect(21:30.0)) #Type stable!

This is more-or-less reasonable, but is somewhat similar to the @vectorize solution. In order for vectorization to be elegant, I suggest Julia support the following:

abstract Vectorizable <: Function
abstract Vectorizable2Arg <: Vectorizable

function call{T1,T2}(fn::Vectorizable2Arg, v1::Vector{T1}, v2::Vector{T2})
    RT = promote_type(T1,T2) #For type stability!
    return RT[fn(v1[i],v2[i]) for i in 1:length(v1)]
end

#Note: by default, functions would normally be <: Function:
function myadd(x::Number, y::Number) <: Vectorizable2Arg
    return x+y+1
end

That's it! Having a function inherit from a Vectorizable function would make it vectorizable.

I hope this along the lines of what you were looking for.

Regards,

MA

[johnmyleswhite]

In the absence of multiple inheritance, how does a function inherit from Vectorizable and from something else? And how do you relate the inheritance information for specific methods to the inheritance information for a generic function?

[JeffBezanson]

@ma-laforge You can do that already --- define a type myadd <: Vectorizable2Arg, then implement call for myadd on Number.

[ma-laforge]

Thanks for that @JeffBezanson!

Indeed, I can almost my my solution look almost as good as what I want:

abstract Vectorizable
#Could easily have parameterized Vectorizable, but want to show inheritance case:
abstract Vectorizable2Arg <: Vectorizable

function call{T1,T2}(fn::Vectorizable2Arg, v1::Vector{T1}, v2::Vector{T2})
    RT = promote_type(T1,T2) #For type stability!
    return RT[fn(v1[i],v2[i]) for i in 1:length(v1)]
end

#SECTION F: Function in need of vectorizing:
immutable MyAddType <: Vectorizable2Arg; end
const myadd = MyAddType()
function call(::MyAddType, x::Number, y::Number)
    return x+y+1
end

@show myadd(5,6)
@show myadd(collect(1:10),collect(21:30.0)) #Type stable

Now, the only thing missing would be a way to "subtype" any function, so that that whole section F could be replaced with the more elegant syntax:

function myadd(x::Number, y::Number) <: Vectorizable2Arg
    return x+y+1
end

NOTE: I made the type "MyAddType", and the function name into a singleton object "myadd" because I find the resultant syntax nicer than if one were using Type{Vectorizable2Arg} in the call signature:

function call{T1,T2}(fn::Type{Vectorizable2Arg}, v1::Vector{T1}, v2::Vector{T2})

Sadly, by your response, it sounds like this would _not_ be an adequate solution to the "sillyness" of the @vectorize macro.

Regards,

MA

[ma-laforge]

@johnmyleswhite:

I would like to respond to your comment, but I don't think I understand. Can you clarify?

One thing I _can_ say:
There is nothing special about "Vectorizable". The idea is that anyone can define their own function "class" (Ex: MyFunctionGroupA<:Function). They could then catch calls to functions of that type by defining their own "call" signature (as demonstrated above).

Having said that: My suggestion is that functions defined within Base should use Base.Vectorizable <: Function (or something similar) in order to auto-generate vectorized algorithms automatically.

I would then suggest module developers implement their own functions using a pattern somewhat like:

myfunction(x::MyType, y::MyType) <: Base.Vectorizable

Of course, they would have to provide their own version of promote_type(::Type{MyType},::Type{MyType}) - if the default is not already to return MyType.

If the default vectorization algorithm is insufficient, there is nothing stopping the user from implementing their own hierachy:

MyVectorizable{nargs} <: Function
call(fn::MyVectorizable{2}, x, y) = ...

myfunction(x::MyType, y:MyType) <: MyVectorizable{2}

MA

[johnmyleswhite]

@ma-laforge, Sorry for being unclear. My concern is that any hierarchy is always going to lack important information because Julia has single inheritance, which requires you to commit to a single parent type for each function. If you use something like myfunction(x::MyType, y::MyType) <: Base.Vectorizable then your function won't benefit from someone else defining a concept like Base.NullableLiftable that auto-generates functions of nullables.

[nalimilan]

Looks like this wouldn't be an issue with traits (cf. https://github.com/JuliaLang/julia/pull/13222). Also related is the new possibility of declaring methods as pure (https://github.com/JuliaLang/julia/pull/13555), which could automatically imply that such a method is vectorizable (at least for single-argument methods).

[ma-laforge]

@johnmyleswhite,

If I understand correctly: I don't think that is a problem for _this_ case in particular. That's because I am proposing a design pattern. Your functions do not _have_ to inherit from Base.Vectorizable... You can use your own.

I don't really know much about NullableLiftables (I don't seem to have that in my version of Julia). However, assuming that it inherits from Base.Function (which is also not possible in my version of Julia):

NullableLiftable <: Function

Your module could then implement (only once) a _new_ vectorizable sub-type:

abstract VectorizableNullableLiftable <: NullableLiftable

function call{T1,T2}(fn::VectorizableNullableLiftable, v1::Vector{T1}, v2::Vector{T2})
    RT = promote_type(T1,T2) #For type stability!
    return RT[fn(v1[i],v2[i]) for i in 1:length(v1)]
end

So, from now on, anyone that defines a function <: VectorizableNullableLiftable would get your vectorization code applied automatically!

function mycooladdon(scalar1, scalar2) <: VectorizableNullableLiftable
...

I do understand, that having more than one Vectorizable-type is still somewhat of a pain (and a bit inelegant)... But at least it would remove one of the annoying repetitions(1) in Julia (having to register _each_ newly added function with a call to @vectorize_Xarg).

(1)That's assuming Julia supports inheritance on functions (ex: myfunction(...)<: Vectorizable) - which it doesn't appear to, on v0.4.0. The solution I got working in Julia 0.4.0 is just a hack... You still have to register your function... not much better than calling @vectorize_Xarg

MA

[JeffBezanson]

I still think it's kind of the wrong abstraction. A function that can or should be "vectorized" is not a particular kind of function. _Every_ function can be passed to map, giving it this behavior.

BTW, with the change I'm working on in the jb/functions branch, you will be able to do function f(x) <: T (though, clearly, only for the first definition of f).

[ma-laforge]

Ok, I think I better understand what you are looking for... and it is not what I suggested. I think that might be part of the issues @johnmyleswhite had with my suggestions as well...

...But if I now understand what the issue is, the solution appears even simpler to me:

function call{T1,T2}(fn::Function, v1::Vector{T1}, v2::Vector{T2})
    RT = promote_type(T1,T2) #For type stability!
    return RT[fn(v1[i],v2[i]) for i in 1:length(v1)]
end

myadd(x::Number, y::Number) = x+y+1

Since myadd is of type Function, it should get trapped by the call function... which it does:

call(myadd,collect(1:10),collect(21:30.0)) #No problem

But call does not auto-dispatch on functions, for some reason (not sure why):

myadd(collect(1:10),collect(21:30.0)) #Hmm... Julia v0.4.0 does not dispatch this to call...

But I imagine this behaviour should not be too difficult to change. Personally, I don't know how I feel about making such all-encompassing catch-all functions, but it sounds like that's what you want.

Something odd I noticed: Julia already auto-vectorizes functions if they are not typed:

myadd(x,y) = x+y+1 #This gets vectorized automatically, for some reason

RE: BTW...:
Cool! I wonder what neat things I will be able to do by subtyping functions :).

[JeffBezanson]

Julia already auto-vectorizes functions if they are not typed

It appears that way because the + operator used inside the function is vectorized.

[JeffBezanson]

But I imagine this behaviour should not be too difficult to change. Personally, I don't know how I feel about making such all-encompassing catch-all functions, but it sounds like that's what you want.

I share your reservations. You can't sensibly have a definition that says "here's how to call any function", because each function itself says what to do when it's called --- that's what a function is!

[SalchiPapa]

I should have said: ... user defined non-unicode infix operators, as I don't think we want to require users to type unicode characters to access such a core functionality. Although I see that $ is actually one of those added, so thank you for that! Wow, so this actually works in Julia today (even if it's not "fast"... yet):

julia> ($) = map
julia> sin $ (0.5 * (abs2 $ (x-y)))

@binarybana how about ↦ / \mapsto ?

julia> x, y = rand(3), rand(3);

julia> ↦ = map    # \mapsto<TAB>
map (generic function with 39 methods)

julia> sin ↦ (0.5 * (abs2 ↦ (x-y)))
3-element Array{Float64,1}:
 0.271196
 0.0927406
 0.0632608

There are also:

• ⤇ / \Mapsto
• ⟼ / \longmapsto
• ⟾ / \Longmapsto
• ⤅ / \twoheadmapsto
• ⊸ / \multimap (parallel map?)
• http://julia.readthedocs.org/en/latest/manual/unicode-input

[mlhetland]

FWIW, I'd at least initially assume that \mapsto was an alternate syntax for lambdas, as it is commonly used in mathematics and, essentially (in its ASCII incarnation, ->) in Julia, too. I think this would be rather confusing.

[mlhetland]

Speaking of math … In model theory I've seen map expressed by applying a function to a tuple without parentheses. That is, if \bar{a}=(a_1, \dots, a_n), then f(\bar{a}) is f(a_1, \dots, a_n) (i.e., essentially, apply) and f\bar{a} is (f(a_1), \dots, f(a_n)) (i.e., map). Useful syntax for defining homomorphisms, etc., but not all that easily transferable to a programming language :-}

[SalchiPapa]

How about any of the other alternatives like \Mapsto, would you confuse it with => (Pair)? I think both symbols are distinguishable here side by side:

• ->
• ↦

There are lots of symbols that look alike, what would be the reason to have so many of them if we only use the ones that look very different or are pure ASCII?

I think this would be rather confusing.

I think documentation and experience solves this, do you agree?

There are also lots of other arrow like symbols, I honestly don't know what are they used for in mathematics or else, I only proposed this ones because they have map in their names! :smile:

[mlhetland]

I guess my point is that -> is Julia's attempt at representing ↦ in ASCII. So using ↦ to mean something else seems ill-advised. It's not that I can't tell them apart visually :-)

My gut feeling is just that if we're going to use well-established mathematical symbols, we might want to at least think through how the Julia usage differs with the established usage. Seems logical to select symbols with map in their names, but in this case that refers to the definition of a map (or of maps of different types, or the types of such maps). That's also true of the usage in Pair, more or less, where rather than defining a full function by defining what a parameter maps to, you actually list what a given argument (parameter value) maps to – i.e., it's an element of an explicitly enumerated function, conceptually (e.g., a dictionary).

[nalimilan]

@Ismael-VC The problem with your suggestion is that you need to call map twice, while the ideal solution would involve no temporary arrays and reduce to map((a, b) -> sin(0.5 * abs2(a-b)), x, y). Besides, repeating ↦ twice isn't great both visually and for typing (for which an ASCII equivalent would be good to have).

[tkelman]

R users might dislike this idea, but if we move towards deprecating the current infix-macro special case parsing of ~ (packages like GLM and DataFrames would need to change to macro-parsing their formula DSL's, ref https://github.com/JuliaStats/GLM.jl/issues/116), that would free up the rare commodity of an infix ascii operator.

a ~ b could be defined in base as map(a, b), and maybe a .~ b could be defined as broadcast(a, b)? If it parses as a conventional infix operator then macro DSL's like an emulation of the R formula interface would be free to implement their own interpretation of the operator inside macros, as JuMP does with <= and ==.

It's maybe not the prettiest suggestion, but neither are the shorthands in Mathematica if you overuse them... my favorite is .#&/@

[SalchiPapa]

:+1: for less special casing and more generality and consistency, the meanings you propose for ~ and .~ looks great to me.

[quinnj]

+1 Tony.

[nalimilan]

@tkelman To be clear, how would you write e.g. sin(0.5 * abs2(a-b)) in a fully vectorized way?

[tkelman]

With a comprehension, probably. I don't think infix map would work for varargs or in-place, so the possibility of free syntax doesn't solve all issues.

[nalimilan]

So that wouldn't really fix this issue. :-/

So far the sin(0.5 * abs2(a-b)) over (a, b) syntax (or a variant, possibly using an infix operator) is the most appealing.

[tkelman]

The title of this issue is "Alternative syntax for map(func, x)". Using an infix operator does not solve map/loop fusion to eliminate temporaries but I think that may be an even broader, related but technically separate issue than the syntax.

[SalchiPapa]

Yes, I agree with @tkelman , the point is to have alternative syntax for map, which is why I suggested using \mapsto, ↦. What @nalimilan mentions seems to be broader and better suited to another issue IMHO, How to fully vecotrize expressions?

<rambling>
This issue has been going on for more than a year now (and could continue indefinetly as many other issues are now)! But we could have Alternative syntax for map(func, x) now. Out of ±450 julian contributors only 41 have been able to find this issue and/or willing to share an opinion (that's a lot for a github issue but clearly not sufficient in this case), all in all there are not that much different suggestions (that are not just minor variations of the same concept).

I know some of you don't like the idea or see value in making surveys/polls (:shocked:), but since I don't need to ask permission from anyone for something like this, I'll just do it anyway. It's kinda sad the way we are not taking full leverage from our community and social networks and other communities also even more sad that we are not seeing the value of it, let's see if I can gather more different and fresh opinions, or at least check out what does the majority feels about the current opinions in this particular issue, as an experiment and see how it goes. Maybe it's indeed useless, maybe it isn't, there is only one way to really know.
</rambling>

[toivoh]

@Ismael-VC: If you really do want to make a poll, the first thing that you must do is to carefully consider the question that you want to ask. You can't expect everyone to read through the entire thread and summarize the options that have been up for discussion individually.

[nalimilan]

map(func, x) also covers things like map(v -> sin(0.5 * abs2(v)), x), and this is what this thread discussed. Let's not move this to another thread, as it would make it harder to keep in mind all of the proposals discussed above.

I'm not opposed to adding syntax for the simple case of applying a generic function using map, but if we do it I think it would be a good idea to consider the broader picture at the same time. If it wasn't for that, the issue could have been fixed a long time ago already.

@Ismael-VC Polls are unlikely to help here IMHO. We're not trying to find out which of several solutions is the best one, but rather to find a solution which nobody has really found already. This discussion is already long and involved many people, I don't think adding more will help.

[JeffBezanson]

@Ismael-VC That's fine, feel free to make a poll. In fact I have done a couple doodle polls on issues in the past (e.g. http://doodle.com/poll/s8734pcue8yxv6t4). In my experience, the same or fewer people vote in polls than discuss in issue threads. It makes sense for very specific, often superficial/syntax issues. But how is a poll going to generate fresh ideas when all you can do is pick from existing options?

Of course the real goal of this issue is to eliminate implicitly vectorized functions. In theory, syntax for map is sufficient for that, because all of these functions are just doing map in every case.

[JeffBezanson]

I've tried to look for existing math notation for this, but you tend to see comments to the effect that the operation is too unimportant to have notation! In the case of arbitrary functions in a mathematical context I can almost believe this. However the closest thing seems to be Hadamard product notation, which has a few generalizations: https://en.wikipedia.org/wiki/Hadamard_product_(matrices)#Analogous_Operations

That leaves us with sin∘x as @rfourquet suggested. Doesn't seem too helpful, since it requires unicode and isn't widely known anyway.

[quinnj]

@nalimilan, I would think you would just do sin(0.5 * abs2(a-b)) ~ (a,b) which would translate to map((a,b)->sin(0.5 * abs2(a-b)), (a,b)). Not sure if that's quite right, but I think it would work.

I'm also wary of delving too much into the let-me-give-you-a-huge-complicated-expression-and-you-perfectly-auto-vectorize-it-for-me problem. I think the ultimate solution in that regard is building a full on DAG of expressions/tasks + query planning, etc. But I think that's a much bigger fish to fry than just having convenient syntax for map.

[nalimilan]

@quinnj Yeah, that's essentially the over syntax proposed above, except with an infix operator.

[johnmyleswhite]

Serious comment: I think you're likely going to re-invent SQL if you pursue this idea far enough since SQL is essentially a language for composing element-wise functions of many variables that are subsequently applied via row-wise "vectorization".

[mdcfrancis]

@johnmyleswhite agree, starts to look like a DSL aka Linq

[mdcfrancis]

On the threads posted topic, you could specialize the |> 'pipe' operator and get the map style functionality. You can read it as pipe the function to the data. As an extra bonus you can use the same to perform function composition.

julia> (|>)(x::Function, y...) = map(x, y... )
|> (generic function with 8 methods)

julia> (|>)(x::Function, y::Function) = (z...)->x(y(z...))
|> (generic function with 8 methods)

julia> sin |> cos |> [ 1,2,3 ]
3-element Array{Float64,1}:
  0.514395
 -0.404239
 -0.836022

julia> x,y = rand(3), rand(3)
([0.8883630054185454,0.32542923024720194,0.6022157767415313],    [0.35274912207468145,0.2331784754319688,0.9262490059844113])

julia> sin |> ( 0.5 *( abs( x - y ) ) )
3-element Array{Float64,1}:
 0.264617
 0.046109
 0.161309

[JeffBezanson]

@johnmyleswhite That's true, but there are worthwhile intermediate goals that are quite modest. On my branch, the map version of multi-operation vectorized expressions is already faster than what we have now. So figuring out how to smoothly transition to it is somewhat urgent.

[nalimilan]

@johnmyleswhite Not sure. A lot of SQL is about selecting, ordering and merging rows. Here we're only speaking about applying a function element-wise. Besides, SQL doesn't provide any syntax to distinguish reductions (e.g. SUM) from element-wise operations (e.g. >, LN). The latter are simply automatically vectorized just like in Julia currently.

[mlhetland]

@JeffBezanson The beauty of the use of \circ is that if you interpret an indexed family as a function from the index set (which is the standard mathematical "implementation"), then mapping _is_ simply composition. So (sin ∘ x)(i)=sin(x(i)), or, rather sin(x[i]).

The use of the pipe, as @mdcfrancis mentions, would essentially just be "diagram order" composition, which is often done with a (possibly fat) semicolon in mathematics (or especially CS applications of category theory) — but we already have the pipe operator, of course.

If neither one of these composition operators is okay, then one could use some others. For example, at least some authors use the lowly \cdot for abstract arrow/morphism composition, as it is essentially the "multiplication" of the groupoid (more or less) of arrows.

And if one wanted an ASCII analog: There are also authors who actually use a period to indicate multiplication. (I may have seen some use it for composition as well; can't remember.)

So one could have sin . x … buut I guess that would be confusing :-}

[mlhetland]

Still … that last analogy might be an argument for one of the really early proposals, i.e., sin.(x). (Or maybe that's far-fetched.)

[mschauer]

Let's try it from a different angle, don't shoot me.

If we define .. by collect(..(A,B)) == ((a[1],..., a[n]), (b[1], ...,b[n])) == zip(A,B), then using T[x,y,z] = [T(x), T(y), T(z)] formally it holds that

map(f,A,B) = [f(a[1],b[1]), ..., f(a[n],b[n])] = f[zip(A,B)...] = f[..(A,B)]

That motivates at least one syntax for map which does not interfere with the syntax for array construction. With :: or table the extension f[::(A,B)] = [f(a[i], b[j]) for i in 1:n, j in 1:n] leads at least to a second interesting use case.

[SalchiPapa]

carefully consider the question that you want to ask.

@toivoh Thank you, I will. I'm currently evaluating several poll/survey software. Also I'll only poll about the preferred syntax, those that want to read the entire thread will just do it, lets just not assume no one else will be interested in doing that.

find a solution which nobody has really found already

@nalimilan nobody among us, that is. :smile:

how is a poll going to generate fresh ideas when all you can do is pick from existing options?
same or fewer people vote in polls than discuss in issue threads.

@JeffBezanson I'm glad to hear that you've already done polls, keep it up!

• How have you've been promoting your polls?
• From the poll/survey software I've evaluated so far kwiksurveys.com let users add their own opinions instead of _none of this options is for me_ option.

sin∘x Doesn't seem too helpful, since it requires Unicode and isn't widely known anyway.

We have so many Unicode, let's use it, we even have nice way to use them with tab completion, what's wrong with using them then?, If it's not know, let's document and educate, if it doesn't exist, what's wrong with inventing it? Do we really need to wait for someone else to invent it and use it so we can take it as precedent and only after then consider it?

∘ has precedent, so the problem is that it's Unicode? why? when are we going to start using the rest of the not widely known Unicode then? never?

By that logic, Julia isn't widely known anyway, however those who want to learn, will. It just doesn't make sense to me, in my very humble opinion.

[JeffBezanson]

Fair enough, I'm not totally against ∘. Requiring unicode to use a pretty basic feature is just one mark against it. Not necessarily enough to sink it completely.

[mlhetland]

Would it be entirely crazy to use/overload * as an ASCII alternative? I'd say it could be argued that it makes some sense mathematically, but I guess it might be hard to discern its meaning at times… (Then again, if it's restricted to the map functionality, then map is already an ASCII alternative, no?)

[hayd]

The beauty of the use of \circ is that if you interpret an indexed family as a function from the index set (which is the standard mathematical "implementation"), then mapping _is_ simply composition.

I'm not sure I buy this.

[mlhetland]

@hayd Which part of it? That an indexed family (e.g., a sequence) can be seen as a function from the index set, or that mapping over it becomes composition? Or that this is a useful perspective in this case?

[mlhetland]

The first two (mathematical) points are pretty uncontroversial, I think. But, yeah, I'm not going to advocate strongly for using this here – it was mostly an "Ah, that kinda fits!" reaction.

[mdcfrancis]

@mlhetland |> is pretty close to -> and works today - it also has the 'advantage' of being right associative.

x = parse( "sin |> cos |> [1,2]" )
:((sin |> cos) |> [1,2])

[mlhetland]

@mdcfrancis Sure. But that turns the composition interpretation I outlined on its head. That is, sin∘x would be equivalent to x |> sin, no?

[mschauer]

PS: Maybe it got lost in the "algebra", but just allowing functions in the typed array construction T[x,y,z] such that f[x,y,z] is [f(x),f(y),f(z)] gives directly

map(f,A) == f[A...]

which is quite readable and could be treated as syntax..

[JeffBezanson]

That's clever. But I suspect that if we can make it work, sin[x...] really loses on verbosity to sin(x) or sin~x etc.

[eschnett]

What about the syntax [sin xs]?

This is similar in syntax to the array comprehension [sin(x) for x in xs].

[mdcfrancis]

@mlhetland sin |> x === map( sin, x )

[tkelman]

That would be the reverse order from the current function chaining meaning. Not that I wouldn't mind finding a better use for that operator, but would need a transition period.

[mlhetland]

@mdcfrancis Yes, I get that that's what you're aiming for. Which reverses things (like @tkelman reiterates) wrt. the composition interpretation I outlined.

[bramtayl]

I think integrating vectorization and chaining would be pretty cool. I wonder if words would be the clearest operators.
Something like:

[1, 2] mapall
  +([2, 3]) map
  ^(2, _) chain
  { a = _ + 1
    b = _ - 1
    [a..., b...] } chain
  sum chain
  [ _, 2, 3] chain
  reduce(+, _)

Several maps in a row could be automatically combined into a single map to improve performance. Also note that I'm assuming map will have some sort of auto-broadcast feature. Replacing [1, 2] with _ at the start could instead build an anonymous function. Note I'm making use of R's magrittr rules for chaining (see my post in the chaining thread).

Maybe this is starting to look more like a DSL.

[jtravs]

I've been following this issue for a long time, and haven't commented until now, but this is beginning to get out of hand IMHO.

I strongly support the idea of a clean syntax for map. I like @tkelman 's suggestion of ~ the most as it keeps within ASCII for such basic functionality, and I quite like sin~x. This would allow for pretty sophisticated one-liner style mapping as discussed above. The use of sin∘x would also be OK. For anything more complicated I tend to think a proper loop is just much clearer (and usually the best performance). I don't really like too much 'magical' broadcasting, it makes the code much harder to follow. An explicit loop is usually clearer.

That's is not to say such functionality shouldn't be added, but let's have a nice concise map syntax first, especially as it's about to get super fast (from my tests of the jb/functions branch).

[StefanKarpinski]

Note that one of the effects of jb/functions is that broadcast(op, x, y) has just as good performance as the customized version x .op y that manually specialized the broadcasting on op.

[nalimilan]

For anything more complicated I tend to think a proper loop is just much clearer (and usually the best performance). I don't really like too much 'magical' broadcasting, it makes the code much harder to follow. An explicit loop is usually clearer.

I don't agree. exp(2 * x.^2) is perfectly readable, and less verbose than [exp(2 * v^2) for v in x]. The challenge here IMHO is to avoid trapping people by letting them use the former (which allocates copies and doesn't fuse operations): for this, we need to find a syntax which would be short enough so that the slow form can be deprecated.

[bramtayl]

More thoughts. There are several possible things you might want to do when calling a function:

loop through no arguments (chain)
loop through just the chained argument (map)
loop through all arguments (mapall)

Each of the above could be modified, by:
Marking an item to loop through (~)
Marking an item not to be looped through ( an extra set of [ ] )

Uniterable items should be handled automatically, disregarding syntax.
Expanding singleton dimensions should happen automatically if there is at least two arguments that are being looped through

[toivoh]

Broadcasting only makes a difference when there would have been a dimension
mismatch otherwise. So when you say don't broadcast, you mean to give an
error instead if the size of the argument doesn't match up?

[mschauer]

sin[x...] really loses on verbosity to sin(x) or sin~x etc.

Also, continuing the thought, the map sin[x...] is a less eager version on [f(x...)].
The syntax

[exp(2 * (...x)^2)]

or something similar like [exp(2 * (x..)^2)] would be available and self explaining if ever real tacit function chaining is introduced.

[jtravs]

@nalimilan yes but that fits within my 'one-liner' category that I said was fine without a loop.

While we're listing all of our wishes: much more important to me would be for the results of map to be assignable without allocation or copying. This is another reason I will still prefer loops for performance critical code, but if this can be mitigated (#249 is not currently looking hopeful ATM) then this all becomes much more attractive.

[JeffBezanson]

results of map to be assignable without allocation or copying

Can you expand on this a bit? You can certainly mutate the result of map.

[malmaud]

I presume he means storing the output of map to a preallocated array.

[jtravs]

Yes, exactly. Apologies if that is already possible.

[JeffBezanson]

Ah, of course. We have map!, but as you observe #249 is asking for some nicer way to do that.

[nalimilan]

@jtravs I proposed a solution above with LazyArray (https://github.com/JuliaLang/julia/issues/8450#issuecomment-65106563), but so far the performance wasn't ideal.

[bramtayl]

@toivoh I made several edits to that post after I posted it. The question I was worried about is how to figure out which arguments to loop through and which arguments not to (so mapall might be clearer than broadcast). I think if you are looping through more than one argument, expanding singleton dimensions to produce comparable arrays should always be done if necessary, I think.

[jtravs]

Yes map! is exactly right. It would be good if any nice syntax sugar worked out here also covered that case. Could we not have that x := ... implicitly maps the RHS onto x.

[bramtayl]

I put up a package called ChainMap which integrates mapping and chaining.

Here's a short example:

@chain begin
  [1, 2]
  -(1)
  (_, _)
  map_all(+)
  @chain_map begin
    -(1)
    ^(2. , _)
  end
  begin
    a = _ - 1
    b = _ + 1
    [a, b]
  end
  sum
end

[mschauer]

I kept thinking about it and I think I finally figured out a consistent and Julian syntax for mapping arrays derived from array comprehension. The following proposal is julian because it builds on the array comprehension language which is already established.

1. Starting from f[a...] which was actually proposed by @Jutho, the convention
   that for a vectors a, b

f[a...] == map(f, a[:])
f[a..., b...] == map(f, a[:], b[:])
etc

which does not introduce new symbols.

2.) On top of this, I would propose the introduction of one additional operator: a _shape preserving_ splatting operator .. (say). This is because ... is a _flat_ spatting operator, so f[a...] should return a vector and not an array even if a is n-dimensional. If .. chosen, then in this context,

f[a.., ] == map(f, a)
f[a.., b..] == map(f, a, b)

and the result inherits the shape of the arguments. Allowing broadcasting

f[a.., b..] == broadcast(f, a, b)

would allow writing thinks like

sum(*[v.., v'..]) == dot(v,v)

Heureka?

[johansigfrids]

This does not help with mapping expressions, does it? One of the advantages of the over syntax is how it works with expressions:

sin(x * (y - 2)) over x, y  == map((x, y) -> sin(x * (y - 2)), x, y) 

[mschauer]

Well, possibly via [sin(x.. * y..)] or sin[x.. * y..] above if you want to allow that. I like that a bit more than the over syntax because it gives a visual hint that the function operators on the elements and not on the containers.

[jtravs]

But can you not simplify that so that x.. simply maps over x? So @johansigfrids example would be:

sin(x.. * (y.. - 2))  == map((x, y) -> sin(x * (y - 2)), x, y)

[mschauer]

@jtravs Because of scoping ( [println(g(x..))] vs println([g(x..)])) and consistency [x..] = x.

[mschauer]

One further possibility is to take x.. = x[:, 1], x[:, 2], etc. as partial splat of the leading subarrays (columns) and ..y as partial splat of the trailing subarrays ..y = y[1,:], y[2,:]. If both run over different indices this covers many interesting cases

[f(v..)] == [f(v[i]) for i in 1:m ]
[v.. * v..] == [v[i] * v[i] for 1:m]
[v.. * ..v] == [v[i] * v[j] for i in 1:m, j in 1:n]
[f(..A)] == [f(A[:, j]) for j in 1:n]
[f(A..)] == [f(A[i, :]) for i in 1:m]
[dot(A.., ..A)] == [dot(A[:,i], A[j,:]) for i in 1:m, j in 1:n] == A*A
[f(..A..)] == [f(A[i,j]) for i in 1:m, j in 1:n]
[v..] == [..v] = v
[..A..] == A

(v a Vector, A a matrix)

[JeffBezanson]

I prefer over, since it allows you to write an expression in normal syntax instead of introducing lots of square brackets and dots.

[mschauer]

You are right about the clutter, I think and I tried to adapt and systemize my proposal. To not overtax everybody's patience further I wrote my thoughts about maps and indices etc. in a gist https://gist.github.com/mschauer/b04e000e9d0963e40058 .

[stevengj]

After reading through this thread, my preference so far would be to have _both_ f.(x) for simple things and people used to vectorized functions (the ". = vectorized" idiom is pretty common), and f(x^2)-x over x for more complicated expressions.

There are just way too many people coming from Matlab, Numpy, etcetera, to completely abandon vectorized function syntax; telling them to add dots is easy to remember. A good over-like syntax for vectorizing complex expressions into a single loop is also very useful.

[StefanKarpinski]

The over syntax really rubs me the wrong way. It just occurred to me why: it presumes that all usages of each variable in an expression are vectorized or non-vectorized, which may not be the case. For example, log(A) .- sum(A,1) – assume that we removed the vectorization of log. You also can't vectorize functions over expressions, which seems like a fairly major shortcoming what if I wanted to write exp(log(A) .- sum(A,1)) and have the exp and the log vectorized and the sum not?

[stevengj]

@StefanKarpinski, then you should do either exp.(log.(A) .- sum(A,1)) and accept the extra temporaries (e.g. in interactive use where performance is not critical), or s = sum(A, 1); exp(log(A) - s) over A (although that's not quite right if sum(A,1) is a vector and you wanted broadcasting); you might just have to use a comprehension. No matter what syntax we come up with, we're not going to cover all possible cases, and your example is particularly problematic because any "automated" syntax would have to know that sum is pure and that it can be hoisted out of the loop/map.

For me, the first priority is an f.(x...) syntax for broadcast(f, x...) or map(f, x...) so that we can get rid of @vectorize. After that, we can continue working on a syntax like over (or whatever) to abbreviate more general uses of map and comprehensions.

[simonster]

@stevengj I don't think the second example there works, because the - won't broadcast . Assuming A is a matrix, the output would be a matrix of single-row matrices, each of which is the log of an element of A minus the vector of sums along the first dimension. You'd need broadcast((x, y)->exp(log(x)-y), A, sum(A, 1)). But I think having a concise syntax for map is useful and doesn't necessarily need to be a concise syntax for broadcast as well.

[malmaud]

Will the functions that have historically been auto-vectorized like sin continue to be so with the new syntax, or would that become deprecated? I worry that even the f. syntax will feel like a 'gotcha' to a large swath of scientific programmers who aren't motivated by conceptual elegance arguments.

[stevengj]

My feeling is that the historically vectorized functions like sin should be deprecated in favor of sin., but they should be deprecated quasi-permanently (as opposed to being removed entirely in the subsequent release) for the benefit of users from other scientific languages.

[stevengj]

One minor(?) problem with f.(args...): although the object.(field) syntax is for the most part rarely used and can probably be replaced with getfield(object, field) without too much pain, there are a _lot_ of method definitions/references of the form Base.(:+)(....) = ...., and it would be painful to change these to getfield.

One workaround would be:

• Have Base.(:+) turn into map(Base, :+) like all other f.(args...), but define a deprecated method map(m::Module, s::Symbol) = getfield(m, s) for backward compatibility
• support the syntax Base.:+ (which currently fails) and recommend this in the deprecation warning for Base.(:+)

[ViralBShah]

I'd like to ask again - if this is something we can do in 0.5.0? I think it is important because of the deprecation of many vectorized constructors. I thought I would be ok with this, but I do find map(Int32, a), instead of int32(a) a bit tedious.

Is this basically just a matter of picking syntax at this point?

[nalimilan]

Is this basically just a matter of picking syntax at this point?

I think @stevengj gave good arguments in favor of writing sin.(x) rather than .sin(x) in his PR https://github.com/JuliaLang/julia/pull/15032. So I'd say the path has been cleared.

I had one reservation about the fact that we don't have a solution to generalize this syntax to compound expressions efficiently yet. But I think at this stage we'd better merge this feature which covers most of the use cases rather than keep this discussion unresolved indefinitely.

[ViralBShah]

@JeffBezanson I am reverting the milestone on this to 0.5.0 in order to bring it up during a triage discussion - mainly to make sure I don't forget.

[ViralBShah]

Does #15032 also work for call - e.g. Int32.(x)?

[stevengj]

@ViralBShah, yes. Any f.(x...) is transformed to map(f, broadcast, x...) at the syntax level, regardless of the type of f.

That's the main advantage of . over something like f[x...], which is otherwise attractive (and would require no parser changes) but would only work for f::Function. f[x...] also clashes conceptually a bit with T[...] array comprehensions. Though I think @StefanKarpinski likes the bracket syntax?

(To pick another example, o::PyObject objects in PyCall are callable, invoking the __call__ method of the Python object o, but the same objects may also support o[...] indexing. This would clash a bit with f[x...] broadcasting, but would work fine with o.(x...) broadcasting.)

[JeffBezanson]

call does not exist any more.

[stevengj]

(I also like @nalimilan's argument that f.(x...) makes .( the analogue of .+ etc.)

[ViralBShah]

Yes the point wise analogy is the one I like best too. Can we go ahead and merge?

[tkelman]

Should the getfield with a module be an actual deprecation?

[stevengj]

@tkelman, as opposed to what? However, the deprecation warning for Base.(:+) (i.e. literal symbol arguments) should suggest Base.:+, not getfield. (_Update_: required a syntax deprecation as well to handle method definitions.)

[stevengj]

@ViralBShah, was there any decision on this at Thursday's triage discussion? #15032 is in pretty good shape for merging, I think.

[tkelman]

I think Viral missed that part of the call. My impression is that multiple people still have reservations about the aesthetics of f.(x) and might prefer either

1. an infix operator which would be simpler conceptually and in implementation, but we don't have any ascii ones available from what I can see. My earlier idea of deprecating the macro parsing of ~ would require work to replace in packages and it's probably too late to try to do that in this cycle.
2. or an alternative new syntax that makes it easier to fuse loops and eliminate temporaries. No other alternatives have been implemented to anywhere near the level of #15032, so it's looking like we should merge that and try it out despite remaining reservations.

[JeffBezanson]

Yes, I do have some reservations but I can't see a better option than f.(x) right now. It seems better than picking an arbitrary symbol like ~, and I bet many who are used to .* (etc.) could even guess right away what it means.

One thing I would like to get a better sense of is whether people are ok with _replacing_ existing vectorized definitions with .(. If people don't like it enough to do the replacement, I'd hesitate more.

[gabrielgellner]

As a user lurking on this discussion I would VERY much like to use this to replace my existing vectorized code.

I largely use vectorization in julia for readability as loops are fast. So I like using it a lot for exp, sin, etc like has been mentioned previously. As I will already use .^, .* in such expressions adding the extra dot to sin. exp. etc feels really natural, and even more explicit, to me ... especially when I can then easily fold in my own functions with the general notation instead of mixing sin(x) and map(f, x).

All to say, as regular user, I really, really hope this gets merged!

[diegozea]

I like more the proposed fun[vec] syntax than fun.(vec).
What do you think about [fun vec]? It's like a list comprehension but with an implicit variable. It could allow to do T[fun vec]

[diegozea]

That syntax it's free in Julia 0.4 for vectors with length > 1:

julia> [sin rand(1)]
1x2 Array{Any,2}:
 sin  0.0976151

julia> [sin rand(10)]
ERROR: DimensionMismatch("mismatch in dimension 1 (expected 1 got 10)")
 in cat_t at abstractarray.jl:850
 in hcat at abstractarray.jl:875

[diegozea]

Something like [fun over vec] could be transformed at syntax level and maybe it worth to simplify [fun(x) for x in vec] but isn't simpler than map(fun,vec).

[diegozea]

Syntaxes similar to [fun vec]: The syntax (fun vec) is free and {fun vec} was deprecated.

julia> (fun vec)
ERROR: syntax: missing separator in tuple

julia> {fun vec}

WARNING: deprecated syntax "{a b ...}".
Use "Any[a b ...]" instead.
1x2 Array{Any,2}:
 fun  [0.3231600663395422,0.10208482721149204,0.7964663210635679,0.5064134055014935,0.7606900072242995,0.29583012284224064,0.5501131920491444,0.35466150455688483,0.6117729165962635,0.7138111929010424]

[stevengj]

@diegozea, fun[vec] was ruled out because it conflicts with T[vec]. (fun vec) is basically Scheme syntax, with the multiple-arguments case presumably being (fun vec1 vec2 ...) ... this is pretty unlike any other Julia syntax. Or did you intend (fun vec1, vec2, ...), which conflicts with tuple syntax? Nor is it clear what the advantage would be over fun.(vecs...).

Furthermore, remember that a main goal is to eventually have a syntax to replace @vectorized functions (so that we don't have a "blessed" subset of functions that "work on vectors"), and this means that the syntax needs to be palatable/intuitive/convenient to people used to vectorized functions in Matlab, Numpy, etcetera. It also needs to be easily composable for expressions like sin(A .+ cos(B[:,1])). These requirements rule out a lot of the more "creative" proposals.

[diegozea]

sin.(A .+ cos.(B[:,1])) doesn't look so bad after all. This's going to need a good documentation. Is f.(x) going to be documented as .( similar to .+ ?
Can .+ be deprecated in favor of +.?

# Since 
sin.(A .+ cos.(B[:,1]))
# could be written as
sin.(.+(A, cos.(B[:,1])))
# +.
sin.(+.(A, cos.(B[:,1]))) #  will be more coherent.

[stevengj]

@diegozea, #15032 already includes documentation, but any additional suggestions are welcome.

.+ will continue to be spelled .+. First, this placement of the dot is too entrenched, and there's not enough to gain by changing the spelling here. Second, as @nalimilan pointed out, you can think of .( as being a "vectorized function-call operator," and from this perspective the syntax is already consistent.

(Once the difficulties with type-computation in broadcast (#4883) are ironed out, my hope is to make another PR so that a .⧆ b for any operator ⧆ is just sugar for a call to broadcast(⧆, a, b). That way, we will no longer need to implement .+ etcetera explicitly — you will get the broadcasting operator automatically just by defining + etc. We will still be able to implement specialized methods, e.g. calls to BLAS, by overloading broadcast for particular operators.)

[yuyichao]

It also needs to be easily composable for expressions like sin(A .+ cos(B[:,1])).

Is it possible to parse f1.(x, f2.(y .+ z)) as broadcast((a, b, c)->(f1(a, f2(b + c))), x, y, z)?

Edit: I see it is already mentioned above... in the comment hidden by default by @github..

[stevengj]

@yuyichao, loop fusion seems like it should be possible if the functions are marked as @pure (at least if the eltypes are immutable), as I commented in #15032, but this is a task for the compiler, not the parser. (But vectorized syntax like this is more for convenience than for squeezing the last cycle out of critical inner loops.)

[stevengj]

Remember that the key goal here is to eliminate the need for @vectorized functions; this requires a syntax at least as general, nearly as convenient, and at least as fast. It doesn't require automated loop fusion, though it is nice to expose the user's broadcast intention to the compiler to open the possibility of loop fusion at some future date.

[yuyichao]

Is there any drawback if it does loop fusion too?

[stevengj]

@yuyichao, loop fusion is a much harder problem, and it's not always possible even putting aside non-pure functions (e.g. see @StefanKarpinski's exp(log(A) .- sum(A,1)) example above). Holding out for that to be implemented will probably result in it _never_ being implemented, in my opinion — we have to do this incrementally. Start by exposing the user's intention. If we can further optimize in the future, great. If not, we still have a generalized replacement for the handful of "vectorized" functions available now.

[stevengj]

Another obstacle is that .+ etc. is not currently exposed to the parser as a broadcast operation; .+ is just another function. My plan is to change that (make .+ sugar for broadcast(+, ...)), as noted above. But again, it is much easier to make progress if the changes are incremental.

[yuyichao]

What I mean is that doing loop fusion by proving doing so is valid is hard, so we can let the parser do the transformation as part of the schematics. In the example above, it can be written as. exp.(log.(A) .- sum(A,1)) and be parsed as broadcast((x, y)->exp(log(x) - y), A, sum(A, 1)).

It's also fine if .+ doesn't belong to the same category yet (just like any non boardcasted function call will be put into the argument) and it's even fine if we will only do this (loop fusion) in a later version. I'm mainly asking if having such a schematics is possible (i.e. non-ambiguous) in the parser and if there's any draw back by allowing written vectorized and fuzed loop this way..

[yuyichao]

doing loop fusion by proving doing so is valid is hard

I mean doing that in the compiler is hard (maybe not impossible), especially since the compiler needs to look into the complicated implementation of broadcast, unless we special case broadcast in the compiler, which is likely a bad idea and we should avoid it if possible...

[stevengj]

Maybe? It's an interesting idea, and doesn't seem impossible to define the .( syntax as "fusing" in this way, and leave it up to the caller to not use it for impure functions. The best thing would be to try it and see if there are any hard cases (I'm not seeing any obvious problems right now), but I'm inclined to do this after the "non-fusing" PR.

[yuyichao]

I'm inclined to do this after the "non-fusing" PR.

Totally agree, especially since .+ isn't handled anyway.

[StefanKarpinski]

I don't want to derail this, but @yuyichao's suggestion gave me some ideas. The emphasis here is on which functions are vectorized, but that's always seems a bit misplaced to me – the real question is which variables to vectorize over, which completely determines the shape of the result. This is why I've been inclined to mark arguments for vectorization, rather than marking functions for vectorization. Marking arguments also allows for functions which vectorize over one argument but not another. That said, we can have both and this PR serves the immediate purpose of replacing the built-in vectorized functions.

[stevengj]

@StefanKarpinski, when you call f.(args...) or broadcast(f, args...), it vectorizes over _all_ the arguments. (For this purpose, recall that scalars are treated as 0-dimensional arrays.) In @yuyichao's suggestion of f.(args...) = _fused broadcast syntax_ (which I am liking more and more), I think the fusion would "stop" at any expression that is not func.(args...) (to include .+ etc. in the future).

So, for example, sin.(x .+ cos.(x .^ sum(x.^2))) would turn (in julia-syntax.scm) into broadcast((x, _s_) -> sin(x + cos(x^_s_)), x, sum(broacast(^, x, 2))). Notice that the sum function would be a "fusion boundary." The caller would be responsible for not using f.(args...) in cases where fusion would screw up side effects.

Do you have an example in mind where this would not be enough?

[yuyichao]

which I am liking more and more

I'm glad you like it. =)

Just yet another extension that probably doesn't belong to the same round, it might be possible to use .=, .*= or similar to solve the in place assignment issue (by making it distinct from the normal assignment)

[stevengj]

Yes, the lack of fusion for other operations was my main objection to .+= etcetera in #7052, but I think that would be resolved by having .= fuse with other func.(args...) calls. Or just fuse x[:] = ....

[mschauer]

:thumbsup: There are two concept huddled together in this discussion which are in fact quite orthogonal:
the matlab'y "fused broadcasting operations" or x .* y .+ z and apl'y "maps on products and zips" likef[product(I,J)...]and f[zip(I,J)...]. The talking past each other might have to do with that as well.

[stevengj]

@mschauer, f.(I, J) is already (in #15032) equivalent to map(x -> f(x...), zip(I, J) if I and J have the same shape. And if I is a row vector and J is a column vector or vice versa, then broadcast indeed maps over the product set (or you can do f.(I, J') if they are both 1d arrays). So I don't understand why you think the concepts are "quite orthogonal."

[mschauer]

Orthogonal was not the right word, they are just different enough to coexist.

[stevengj]

The point is, though, that we don't need separate syntaxes for the two cases. func.(args...) can support both.

[stevengj]

Once a member of the triumvirate (Stefan, Jeff, Viral) merges #15032 (which I think is merge-ready), I'll close this and file a roadmap issue to outline the remaining proposed changes: fix broadcast type-computation, deprecate @vectorize, turn .op into broadcast sugar, add syntax-level "broadcast-fusion", and finally fuse with in-place assignment. The last two probably won't make it into 0.5.

[mschauer]

Hey, I am very happy and grateful about 15032. I would not be dismissive of the discussion though. For example vectors of vectors and similar objects are still very awkward to use in julia but can sprout like weed as results of comprehensions. Good implicit notation not based on encoding iteration into singleton dimensions has the potential to ease this a lot, for example with the flexed out iterators and new generator expressions.

[stevengj]

I think this can be closed now in favor of #16285.