Julia: assigning variables inside spawn

[mlubin]

julia> x = 10
10

julia> @spawn 1+$x
RemoteRef(1,1,1)

julia> exception on 1: ERROR: error compiling anonymous: syntax: prefix $ in non-quoted expression

I think I understand why this is tricky given the compile-time evaluation of the macro, but this would be really useful if it worked correctly. It seems to have already led to some confusion: https://groups.google.com/forum/#!searchin/julia-users/@everywhere/julia-users/KDnwAqD0urI/yVfBa3YYxdUJ

[JeffBezanson]

The $ is part of the syntax for _creating expression objects_, which is quite similar to creating a string except it is pre-parsed. Here you are not trying to create an expression. The parallel constructs generally pick up the values of all variables in lexical scope. spawn works:

julia> addprocs_local(1)
:ok

julia> x=10
10

julia> fetch(@spawn 1+x)
11

julia> fetch(@spawn 1+x)
11

The only difference is that everywhere was intended for top-level expressions.

[mlubin]

How do you deal with the case where you want to refer to some variables that are defined in the local scope and some variables that are defined in the process that runs the expression?

[JeffBezanson]

Relying on global variables to name things on various processors is frowned on, but it can be done with an explicit global:

julia> @everywhere yy = 88

julia> yy = 10
10

julia> fetch(@spawnat 2 yy)
10

julia> fetch(@spawnat 2 (global yy;yy))
88

[mlubin]

It's inevitable in applications that aren't embarrassingly parallel to have to store some objects persistently on different processes. There should be better support for this.

[mlubin]

Here's a nonobvious issue with the current behavior:

julia> function f()
           r = @spawn begin
               t = 1
               t
           end
           t = fetch(r)
       end
# methods for generic function f
f() at none:2

julia> f()
ERROR: in f: t not defined
 in f at none:2

It works if you rename t inside the @spawn to something else.

[JeffBezanson]

You can send values to processors with RemoteRefs.

[JeffBezanson]

The t not defined I'd say is a bug.

[mlubin]

Maybe there should be a mode of parallelism (i.e. a set of macros) that just sends an expression (potentially with spliced values) instead of pulling in local variables? You can do this with remote_call but it's ugly.

[dls]

Hey there, just wanted to say the "nonobvious issue" has been spotted in the wild

@everywhere add_three_tuple(x,y) = (x[1]+y[1], x[2]+y[2], x[3]+y[3])

function in_function_version()
    # ERROR: A not defined
    #  in in_function_version at /Users/dls/jl/play/tc.jl:1509
    #  in include_from_node1 at loading.jl:120
    # while loading /Users/dls/jl/play/tc.jl, in expression starting on line 91
    (A, B, C) = @parallel (add_three_tuple) for k = 1:10
        A = 1
        B = 2
        C = 3
        (A, B, C)
    end

    return (A, B, C)
end

@show in_function_version()

[dls]

(also note the line numbers are crazy)