Julia: Woven functional programming

[bramtayl]

I got a request in #16285 to open up a new issue for this suggestion.

I'm copying the original post:

OK, I've gotten ChainMap into a place where it's relatively stable, and I'd like to propose an alternate method of dot vectorization based on the work there.

Here's an alternative.

• A type, LazyCall, which contains both a function and its arguments (positionals in a tuple and keywords in an OrderedDict)
• An @~ macro to create a LazyCall from a woven expression (this is currently @unweave in ChainMap)
• New methods for map, broadcast, etc. which take LazyCalls.

Here's how the syntax would change:

map( ~ sin(~x + y) ) goes to map( LazyCall(x -> sin(x + y), x) )
broadcast( ~ sin(~x + ~y) ) goes to broadcast( LazyCall( (x, y) -> sin(x + y), x, y) )
mapreduce( ( ~ sin(~x + y) ), +) goes to mapreduce( LazyCall( x -> sin(x + y), x), +)

Splats (needs more thought, I think)
~ f(~(x...) ) goes to LazyCall( (x...) -> f(x...), x...)
~ f(~(;x...) ) goes to LazyCall( (;x...) -> f(;x...); x...)

Advantages over the current system:
1) No ambiguity over what counts as a scalar and what doesn't
2) Reduction in the amount of dots necessary for a complicated expression
3) Support for non-broadcast operations
4) Support for passing multiple LazyCalls to one function
5) Partial support for lazy evaluation
6) No need for fusion

Disadvantages:
1) ~ is already taken as bit negation. @~ is taken for formulas. I like the symbol because it reminds me of a woven thread. But maybe another ascii character will do?
2) Not backwards compatible; people will need to learn a new system.
3) Less verbose
4) Not sure about performance impacts

All of this is implemented in ChainMap at the moment, aside from the new methods for map and friends. It can probably stay in a package for now if people aren't interested in adding it to base.

[bramtayl]

And another post:

Agreed, less verbose, more flexible.

~x + ~y would just be ~x + ~y (it doesn't pass through a macro)
~ ~x + ~y would be LazyCall( (x, y) -> x + y, x, y)

In some cases, the amount of dots compared to ~ is cut down on. Eg.
sin.(cos.(tan.(x .+ 1) ) ) vs broadcast( ~ sin(cos(tan( ~x + 1) ) ) )

[bramtayl]

And another:

Agreed that the pure math aspect of things gets lost.

There are allocation free ways of going about this:

map( ~ sin(~x + y) ) goes to map( x -> sin(x + y), x)
broadcast( ~ sin(~x + ~y) ) goes to broadcast( (x, y) -> sin(x + y), x, y)
mapreduce( ( ~ sin(~x + y) ), +) goes to mapreduce( x -> sin(x + y), +, x)

And in fact, ChainMap has some convenience macros for doing this:

@map sin(_ + y) goes to map( _ -> sin(_ + y), _) (_ is used as part of the chaining)
@broadcast sin(~x + ~y) goes to broadcast( (x, y) -> sin(x + y), x, y)

But to do this you need to make assumptions about argument order (put the anonymous function first, the woven arguments last).

LazyCalls do have other advantages. They can be merged, pushed to, revised, etc. This is particularly useful for making plot calls.

[ChrisRackauckas]

Agreed that the pure math aspect of things gets lost.

The reason why I don't see this as a usable replacement for the . syntax is because of this. There should be no reference to the programming aspect of it, the . syntax lets you write clean code on arrays that looks like math. That's why it was there in the first place. Getting rid of that interaction with math is a big step in the wrong direction, and would make code for common scientific computing applications like PDEs much more convoluted and hard to read.

[TotalVerb]

@bramtayl I think this would be quite interesting. Could you make a package? The syntax can come later.

Edit: Never mind, I see that you already have https://github.com/bramtayl/ChainMap.jl

[ChrisRackauckas]

@bramtayl I think this would be quite interesting. Could you make a package? The syntax can come later.

https://github.com/bramtayl/ChainMap.jl

[bramtayl]

Note that I just tagged a new version; hasn't made it into METADATA yet. Fixes mostly minor

[bramtayl]

I think if you're worried about "hiding" broadcast, it should be relatively easy to make this the "default".

For example,
sin.(~x + ~y) goes to broadcast( (x, y) -> sin(x + y), x, y) )

Although less flexible, this still retains some of the advantages:

1) No ambiguity over what counts as a scalar and what doesn't
2) Reduction in the amount of dots necessary for a complicated expression
3) No need for fusion

This could be used along with syntax above

[bramtayl]

ChainMap master now has @over for this:

@over ~x + ~y goes to broadcast( (x, y) -> sin(x + y), x, y) )
@over ~x == 1 filter goes to filter( x -> x == 1, x)
@over ~x + 1 mapreduce(*) goes to mapreduce( x -> x + 1, *, x)

[bramtayl]

Ok, well, this can stay in ChainMap.jl for now.

[bramtayl]

I've been thinking about this again. Currently Call in LazyCall is immutable. Am I correct in my understanding that it will be unrolled during optimizations, and its use won't require additional allocations?

[KristofferC]

I think discourse is better for general julia questions like this one.