Julia: Scoping bug with anonymous functions?

[tlnagy]

``` .julia
function test()

f = (x, y) -> begin
    result = -Inf
end
g = (x) -> f(x, test2)

result = "test"
results = @time pmap(x->test3(x, g), [2, 10])
println(result)

end

@everywhere test2(x) = 2

@everywhere function test3(x, y)
y(x)
end
test()
```

Prints -Inf, while changing g to f in the test3 call cause it to print "test" which is correct.

[tlnagy]

versioninfo():
Julia Version 0.4.5
Commit 2ac304d* (2016-03-18 00:58 UTC)
Platform Info:
System: Darwin (x86_64-apple-darwin15.4.0)
CPU: Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
WORD_SIZE: 64
BLAS: libopenblas (DYNAMIC_ARCH NO_AFFINITY Haswell)
LAPACK: libopenblas
LIBM: libopenlibm
LLVM: libLLVM-3.3

[tlnagy]

Of note, this only happens when running the script on a single thread:

$ julia -p 1 test.jl
  0.381118 seconds (285.83 k allocations: 12.389 MB)
test
$ julia test.jl
  0.172745 seconds (84.21 k allocations: 3.829 MB)
-Inf

[JeffBezanson]

I'm not able to reproduce this exactly as you describe; changing g to f doesn't work since g accepts 1 argument and f accepts 2. If I change test3 accordingly, I see the same behavior with f.

If you modify a local variable inside a function passed to pmap, the behavior will depend on where the function runs. If it runs on the local machine, the variable will be modified, but if it runs on a remove machine a copy will be modified. Hence passing functions like this to pmap is not recommended.

[tlnagy]

Hence passing functions like this to pmap is not recommended.

But isn't this a bug if it violates scoping? This can lead to some very nasty aliasing bugs.

I can also observe this behavior if I change pmap to map. Just to clarify I do not want to modify result within f, it was an accidental aliasing bug that I discovered that crashed my downstream code.

[JeffBezanson]

I see. Well, this is just how lexical scope works (as in scheme for example). You can use local result = ... in the inner function to give it a separate variable.

[tlnagy]

I guess I hadn't fully digested all the rules here: http://docs.julialang.org/en/release-0.4/manual/variables-and-scoping/#hard-vs-soft-local-scope.

So the solution is to use local or move f outside of test?

[JeffBezanson]

Yes.

To be fair, you're certainly not the first to bring this up. Actually wanting to modify parent-scope variables in an inner function is pretty rare, so it's quite possible that shouldn't be the default.

[stevengj]

I don't know, access to parent-scope variables is pretty much the whole reason to define an inner function.

[tlnagy]

I don't know, access to parent-scope variables is pretty much the whole reason to define an inner function.

In my actual use case, f is only ever used inside of test as one logical block and it's used as an inner function for organizational purposes.

Also, the discrepancy in behavior when running my code on one thread versus two feels sketchy.

[mauro3]

My understanding is that in 0.5 inner functions don't carry a performance penalty anymore. If so, then this style of coding might become more common. One gotcha is that you cannot just copy a function from global scope to a local one without checking that there are no variable clashes.

Refs: #10559, #5331

[malmaud]

Well, is that so different than saying you can't copy arbitrary code blocks between scopes without worrying about clashes?

[mauro3]

You're right, one has to be careful when copying code around. With functions though, one needs look out for different clashes if one copies a function from a global to a global scope than if one copies from a global to a local scope. I would be in favor of functions always having hard-scope, irrespective of whether a variable is local or global, and adding a nonlocal keyword to access the scope one up: explicit is better than implicit. But having said this, the current behavior is also ok.

[JeffBezanson]

I don't think we're going to change how lexical scope works at this point.