Julia: "generated function body is not pure" being too greedy?

[andyferris]

I have received errors on master for StaticArrays and made this minimal example

julia> @generated f(x) = :(z -> (z,x))
f (generic function with 1 method)

julia> f(1)
ERROR: generated function body is not pure. this likely means it contains a closure or comprehension.

Perhaps I'm completely wrong but isn't it the function generator that must be pure? I wasn't aware that the types of functions that could be generated were restricted in any way.

I'm curious if some code is being overzealous in asserting that the function generator has created a closure? Or please correct me if I'm mistaken.

[andyferris]

A counterpoint:

julia> @generated f(x) = z -> (z,x)
WARNING: Method definition f(Any) in module Main at REPL[20]:1 overwritten at REPL[25]:1.
f (generic function with 1 method)

julia> f(1)
(::#13) (generic function with 1 method)

In this case the _generator_ made a function (not sure if it's a closure, since x is the type in this case).

[andyferris]

And to round it out:

julia> @generated function f(x) 
           tmp =zero(x)
           z -> (z,tmp)
       end
WARNING: Method definition f(Any) in module Main at REPL[25]:1 overwritten at REPL[28]:2.
f (generic function with 1 method)

julia> f(1)
(::#15) (generic function with 1 method)

julia> f(1)(1)
(1,0)

julia> f(1).tmp
0

Here the _generator_ definitely made and returned a closure.

[vtjnash]

It can return a closure (value), although that seems rather pointless (it's just a less clear way of calling typeof and wrapping it in an object)

It just can't define a new function (closure) as there's no environment for it to capture.

[andyferris]

I still don't quite understand.

I assumed a @generated function f(...) performs two steps:

1. Run a function generator that creates a symbol, expression, or return value. You've explained to me in the past that this step must be pure because it interrupts inference or another step of compilation in a strange state so you can't mess with anything the compiler can see (e.g. the infamous generated type). This would populate a method specialization with some AST.
2. Compile it, just like any other function, using the output of the above.

The "environment" it captures is defined in step 2, right?

Oh wait, I can see that from this:

julia> function f(x)
          tmp = x
          func = x -> x + tmp
          return func
       end
f (generic function with 1 method)

julia> f(2)
(::#1) (generic function with 1 method)

julia> @code_lowered f(2)
LambdaInfo template for f(x) at REPL[1]:2
:(begin 
        nothing
        tmp = x # line 3:
        #1 = $(Expr(:new, :((Core.apply_type)(Main.##1#2,(Core.typeof)(tmp))), :(tmp)))
        func = #1 # line 4:
        return func
    end)

that the lowered AST already knows the type of the closure. So closures (and Generators?) are made during the lowering step, introducing a new function type? Is this for efficiency (not making a new closure every time a method is specialized)?

A couple of times, I have wondered if methods could store their higher-level Expr syntax. This might solve this limitation and be awesome for introspection, but might lose on efficiency.

[andyferris]

So, in summary, the "magical" steps of lowering (closures and comprehensions/generators) can only occur for the function generator, not the generated code. Lowering of the generated code must be pure.

Perhaps this also explains how promote_op/_default_type/return_type(::Generator) work (for normal, non-generated functions).

Do you see this as limitation of the current implementation? Is there much desire for changing this (i.e. expanding generators/comprehensions and closures to generated functions)?

[andyferris]

OK, I agree that this issue is resolved, but it wasn't clear to me what "generated function body" was referring to (but after the fact I do see what that implied). I'm very amenable to making a clear docstring _a la_ #19300 and preparing a PR myself if @vtjnash would grant me a few words of clarification/confirmation on what is going on. Particularly if my understanding of lowering as it stands is correct. Cheers Jameson/Stefan.

[andyferris]

OK I think my questions were answered elsewhere. Thanks!

[mbauman]

Re-opening to ask for a reconsideration of that won't fix tag, especially since Yichao suggests that it may be possible eventually https://github.com/JuliaLang/julia/issues/21094#issuecomment-287649747.

[vtjnash]

This is definitely won't fix. I've reopened the linked issue, as it is possible we might find a way to fix it.