Julia: Global variable scope rules lead to unintuitive behavior at the REPL/notebook

[IainNZ]

Example 1

This came up with a student who upgraded from 0.6 to 1.0 directly, so never even got a chance to see a deprecation warning, let alone find an explanation for new behavior:

julia> beforefor = true
true

julia> for i in 1:2
         beforefor = false
       end

julia> beforefor  # this is surprising bit
true

julia> beforeif = true
true

julia> if 1 == 1
         beforeif = false
       end
false

julia> beforeif  # Another surprise!
false

julia> function foo()
         infunc = true
         for i in 1:10
           infunc = false
         end
         @show infunc
       end
foo (generic function with 1 method)

julia> foo()  # "I don't get this"
infunc = false 

Example 2

julia> total_lines = 0
0

julia> list_of_files = ["a", "b", "c"]
3-element Array{String,1}:
 "a"
 "b"
 "c"

julia> for file in list_of_files
         # fake read file
         lines_in_file = 5
         total_lines += lines_in_file
       end
ERROR: UndefVarError: total_lines not defined
Stacktrace:
 [1] top-level scope at ./REPL[3]:4 [inlined]
 [2] top-level scope at ./none:0

julia> total_lines  # This crushs the students willingness to learn
0

I "get" why this happens in the sense that I think I can explain, with sufficient reference to the arcana in the manual about what introduces scopes and what doesn't, but I think that this is problematic for interactive use.

In example one, you get a silent failure. In example two, you get an error message that is very there-is-no-spoon. Thats roughly comparable to some Python code I wrote in a notebook at work today.

I'm not sure what the rules are in Python, but I do know that generally you can't assign to things at the global scope without invoking global. But at the REPL it does work, presumably because at the REPL the rules are different or the same logic as if they were all are in the scope of function is applied.

I can't language-lawyer the rules enough to propose the concrete change I would like, and based on Slack this isn't even necessarily perceived as an issue by some people, so I don't know where to go with this except to flag it.

Cross-refs:

19324

https://discourse.julialang.org/t/repl-and-for-loops-scope-behavior-change/13514
https://stackoverflow.com/questions/51930537/scope-of-variables-in-julia

[IainNZ]

(Per @mlubin, this is the relevant change https://github.com/JuliaLang/julia/pull/19324)

[jekbradbury]

Stefan suggested here that one possibility to solve this issue is automatic wrapping of REPL entries in let blocks

[KristofferC]

But wouldn't that be confusing in that you couldn't do

a = 1

and use a after that? Unless global is inserted for all the toplevel assignments, I guess?

[StefanKarpinski]

The behavior wouldn't be just to wrap everything in a let block—it's more complicated than that. You need to let-bind any global that's assigned inside the expression and then extract the let-bound value to a global at the end of the expression.

[StefanKarpinski]

So you would turn a = 1 into something like a = let a; a = 1; end. And something like

for i in 1:2
    before = false
end

would be turned into this:

before = let before = before
    for i in 1:2
        before = false
    end
end

Frankly, I'm pretty annoyed that people are only giving this feedback now. This has change has been on master for ten months.

[piever]

I'm guilty of not having followed master very closed until recently, so this feedback is indeed a bit late. More than a concern for programmers (most for loops will be inside a function in library code) I'm afraid this is a concern for teaching. Often for loops are taught before functions or scopes (of course you need to understand scopes to really understand what's going on but in teaching things are often simplified).

Here it becomes a bit difficult to teach a beginner how to sum numbers from 1 to 10 without explaining functions or global variables.

[mlubin]

Frankly, I'm pretty annoyed that people are only giving this feedback now. This has change has been on master for ten months.

To be fair, Julia 0.7 was released 13 days ago. This is a new change for most Julia users.

[IainNZ]

Frankly, I'm pretty annoyed that people are only giving this feedback now. This has change has been on master for ten months

Unfortunately for those of us who can not handle living on the edge, its brand-new from our perspective.

[rickhg12hs]

Frankly, I'm pretty annoyed that people are only giving this feedback now. This has change has been on master for ten months.

And for those of us who have been encouraged to stay off the development branches, "it's brand-new from our perspective."

[KristofferC]

Can we please go back to focus on the issue at hand now, instead of having a meta discussion about how long people have had to test this. It is what it is right now, so let's look forward.

[ChrisRackauckas]

I'm guilty of not having followed master very closed until recently, so this feedback is indeed a bit late. More than a concern for programmers (most for loops will be inside a function in library code) I'm afraid this is a concern for teaching. Often for loops are taught before functions or scopes (of course you need to understand scopes to really understand what's going on but in teaching things are often simplified).

Here it becomes a bit difficult to teach a beginner how to sum numbers from 1 to 10 without explaining functions or global variables.

This is a big point. After finding out what the issue really is, it's surprising how little it actually shows up. It is less of an issue with a lot of Julia code in the wild and in tests, and it did reveal a lot of variables which were accidentally global (in both Julia Base's tests according to the original PR, and I noticed this on most of DiffEq's tests). In most cases it seems that the subtly wrong behavior isn't what you get (expecting a change in a loop), but rather expecting to be able to use a variable in a loop is what I've found to be the vast majority of where this shows up in updating test scripts to v1.0. So the good thing is that in most cases the user is presented with an error, and it's not difficult to fix.

The bad thing is that it's a little verbose to have to put global x inside of the loops, and now your REPL code is also different from the function code. Whether or not it's more intuitive behavior than before is a tough opinion because there were definitely some edge cases in hard/soft local scoping and so this is clearly easier to explain. But at the same time, while having a much more succinct explanation than the behavior of before, it's now easier to hit the edge cases where understanding scoping rules matters. 🤷‍♂️.

I for one would like to see the experiments with let blocking. This would keep the "you didn't really want so many globals" aspect of it, along with the simplified scoping explanation, while at the same time make REPL code behave like function interiors (which is seemingly what we've always wanted). Or inversely, making people specify variables they want to act as globals

global x = 5
for i = 1:5
  println(x+i)
end

could be a nice way to keep the explicitness, and would make the "REPL code is slow because of globals" be much more obvious. The downside is that once again throwing things into a function would not require the global markers.

But given how this tends to show up, it's not really gamebreaking or a showstopper. I'd classify it as a wart that should get a mention in any workshop but it's not like v1.0 is unusable because of it. I hope that changing this behavior isn't classified as breaking and require v2.0 though.

[ExpandingMan]

I'm not so sure I like the idea that the REPL should behave like a function interior. It clearly isn't, so I expect it to behave like global scope. To me the REPL not behaving like global scope would be potentially even more confusing than the discrepency that causes this issue.

Regardless, at the very least I think that the documentation should be somewhat more explicit about this issue. Casually reading the docs I would have assumed that you would need to use the local keyword to get the behavior occurs in global scope by default.

[piever]

I for one would like to see the experiments with let blocking. This would keep the "you didn't really want so many globals" aspect of it, along with the simplified scoping explanation, while at the same time make REPL code behave like function interiors (which is seemingly what we've always wanted)

If we're going for "REPL is the same as the inside of a function" we should also think about outer:

julia> i = 1
1

julia> for outer i = 1:10
       end
ERROR: syntax: no outer variable declaration exists for "for outer"

versus:

julia> function f()
          i = 0
          for outer i = 1:10
          end
          return i
       end
f (generic function with 1 method)

julia> f()
10

[stevengj]

Frankly, I'm pretty annoyed that people are only giving this feedback now. This has change has been on master for ten months.

People haven't been using master for interactive use or for teaching, they've been using it to upgrade packages, which are only minimally affected by this and are mostly written by experienced programmers.

(I was one of the few people who did give feedback in #19324, though, where I argued for the old behavior.)

A non-breaking way out of this would be to change back to the old behavior (ideally not by inserting implicit let blocks or anything — just restore the old code in julia-syntax.scm as an option) in the REPL. Or rather, to make it available in environments like IJulia that might want it, add a soft_global_scope=false flag to include, include_string, and Core.eval to restore the old behavior.

[StefanKarpinski]

(I was one of the few people who did give feedback in #19324, though, where I argued for the old behavior.)

Yes, and I greatly appreciate it. It doesn't much matter now since we made the choice, let it bake for ten months and have now released it with a long-term commitment to stability. So the only thing to do now is to focus on what to do going forward.

Having an option to choose between the old behavior and the new one is interesting but it feels very hacky. That means we not only sometimes have a scoping behavior that everyone apparently found incredibly confusing, but we don't always have it and whether we have it or not depends on a global flag. That feels pretty unsatisfactory, I'm afraid.

[stevengj]

Having an option to choose between the old behavior and the new one is interesting but it feels very hacky.

If someone implements an "unbreak me" soft-scope AST transformation, it will be very tempting to use it in IJulia, OhMyREPL, etcetera, at which point you get the even more problematic situation in which the default REPL is seen as broken.

[StefanKarpinski]

That's not what I'm saying. Clearly we should use the same solution in all those contexts. But implementing it as two different variations on scoping rules seems less clean than implementing it as a code transformation with one set of scoping rules. But perhaps those are functionally equivalent. However, it seems easier to explain in terms of the new simpler scoping rules + a transformation that takes REPL-style input and transforms it before evaluating it.

[StefanKarpinski]

That could be done as Meta.globalize(m::Module, expr::Expr) that transforms an expression by automatically annotating any globals which exist in the module as global if they are assigned inside of any top-level non-function scope. Of course, I think that's equivalent to what the old parser did, but a bit more transparent since you can call Meta.globalize yourself and see what the REPL will evaluate.

[stevengj]

That could be done as Meta.globalize(m::Module, expr::Expr) that transforms an expression by automatically annotating any globals which exist in the module as global if they are assigned inside of any top-level non-function scope.

I actually started looking into implementing something like this a few minutes ago. However, it looks like it would be much easier to implement as an option in julia-syntax.jl:

• Writing an external AST transformation is possible, but it seems like there are lots of tricky corner cases — you basically have to re-implement the scoping rules — whereas we already had the code to get it right in julia-syntax.scm.
• It's even more tricky for something like IJulia that currently uses include_string to evaluate a whole block of code and get the value of the last expression. Not only would we have to switch to parsing expression by expression, but some hackery may be needed in order to preserve the original line numbers (for error messages etcetera). (Though I found a hack for ChangePrecision.jl for this sort of thing that may work here also.)
• Not to mention of the case of people that include external files, which would not be caught by your AST transformation.

However, it seems easier to explain in terms of the new simpler scoping rules + a transformation that takes REPL-style input and transforms it before evaluating it.

I seriously doubt this would be easier to explain to new users than just saying that the rules are less picky for interactive use or for include with a certain flag.

[stevengj]

Here is a rough draft of a globalize(::Module, ast) implementation: https://gist.github.com/stevengj/255cb778efcc72a84dbf97ecbbf221fe

[stevengj]

Okay, I've figured out how to implement a globalize_include_string function that preserves line-number information, and have added it to my gist.

A possible (non-breaking) way forward, if people like this approach:

1. Release a SoftGlobalScope.jl package with the globalize etc. functions.
2. Use SoftGlobalScope in IJulia (and possibly Juno, vscode, and OhMyREPL).
3. Fold the SoftGlobalScope functions into a future release of the REPL stdlib package and use it in the REPL.

Or is it practical to roll it into REPL.jl immediately? I'm not completely clear on how stdlib updates work in 1.0.

Please take a look at my implementation, in case I'm missing something that will cause it to be fragile.

[KristofferC]

Can't we have it as a non-default feature of the REPL in 1.1?

[JeffBezanson]

Duplicate of #28523 and #28750. To those saying they don't want to teach people about global variables, I suggest teaching functions first, before for loops. Functions are more fundamental anyway, and this will help set the expectation that code should be written in functions. While I understand the inconvenience, this scoping behavior can be turned into a pedagogical advantage: "In fact, global variables are such a bad idea, particularly using them in loops, that the language makes you bend over backwards to use them."

Adding a non-default feature to the REPL for this seems ok to me though.

[stevengj]

@JeffBezanson, remember that many of us would like to use Julia as a substitute for Matlab etcetera in technical courses like linear algebra and statistics. These are not programming courses and the students often have no programming background. We never do structured programming — it's almost all interactive with short snippets and global variables.

Furthermore, the reason I'm using a dynamic language in the first place is to switch fluidly between interactive exploration and more disciplined programming. The inability to use the same code in a global and a function context is a hindrance to that end, even for someone who is used to scoping concepts, and it is much worse for students from non-CS backgrounds.

[ExpandingMan]

remember that many of us would like to use Julia as a substitute for Matlab etcetera in technical courses like linear algebra and statistics. These are not programming courses and the students often have no programming background. We never do structured programming — it's almost all interactive with short snippets and global variables.

Many of us Julia users have absolutely 0 CS background (including myself), but it seems to me that the proper attitude (especially for students) is a willingness to learn rather than demanding things be changed for the worse to accommodate our naivete.

Now, I'm not necessarily implying that this particular change would be for the worse as I only have a limited understanding of what's going on here, but if it is the case that this is a significant complication or makes it excessively easy to write needlessly badly performing code it does not seem worth it to make a change in order to have a better lecture example. You can't change the laws of physics so that the electrostatics examples you show to freshman are more applicable to real life.

So my question as a non-CS user who also cares about performance is how would I be likely to screw up if this were made the default behavior. Is it literally just the sorts of examples we are seeing here that are a problem (which I was already aware of), or are we likely to often screw this up badly in more subtle ways?

For what it's worth, I do agree that having code behave differently depending on its enclosing scope is a generally undesirable feature.

[stevengj]

Making code harder to write interactively, forcing beginners writing their first loops to understand obscure scoping rules, and making code pasted from functions not work in global scopes does not help programmers write fast code in functions. It just makes it harder to use Julia interactively and harder for beginners.

[stevengj]

Can't we have it as a non-default feature of the REPL in 1.1?

Making an "unbreak me" option the default seems wiser, especially an option that is aimed squarely at beginning users. If it is a non-default option, then precisely those people who need it most will be those who don't have it enabled (and don't know it exists).

[mauro3]

What would the proposed REPL-mode do to includeed scripts? Would the evaluation of global statements depend on whether the REPL mode is activated? If so, IMO this would be at odds with the 1.0 stability promise.

[StefanKarpinski]

If we did something like this it seems like it might make sense for the module to determine how it works. So Main would be a "soft scope" module while by default other modules would be "hard scope" modules.

[dawbarton]

I was interested to see if it was possible to monkey patch the REPL to use @stevengj's globalize function and it appears it is without too much effort (though quite hacky). See the gist. This doesn't work with Juno (or anything else that calls Core.eval directly).

I'm not going to be recommending this to people, but it's quite useful to me when doing quick-and-dirty data analysis. I would very much like to see a (better thought out) solution since it really is quite confusing for inexperienced and often reluctant coders (i.e., my students) when you can't copy and paste in code from a function into the REPL to see what it does and vice-versa.

julia> a = 0                                                                
0                                                                           

julia> for i = 1:10                                                         
         a += i                                                             
       end                                                                  
ERROR: UndefVarError: a not defined                                         
Stacktrace:                                                                 
 [1] top-level scope at .\REPL[2]:2 [inlined]                               
 [2] top-level scope at .\none:0                                            

julia> using SoftGlobalScope                                                
[ Info: Precompiling SoftGlobalScope [363c7d7e-a618-11e8-01c4-4f22c151e122] 

julia> for i = 1:10                                                         
         a += i                                                             
       end                                                                  

julia> a                                                                    
55                                                                          

(BTW: the above is about as much testing as it has had!)

[stevengj]

What would the proposed REPL-mode do to included scripts?

Nothing. Basically, the proposal is that this would only be for code entered at an interactive prompt. As soon as you start putting things in files, you need to learn the "hard scope" rules. Hopefully, when you start putting code into files you should start using functions.

It's not ideal for there to be pickier scoping rules for global code in files than at the prompt. But I think that #19324 combined with the Julia 1.0 stability promise leaves us with no ideal options.

[tpapp]

@stevengj:

@JeffBezanson, remember that many of us would like to use Julia as a substitute for Matlab etcetera in technical courses like linear algebra and statistics. These are not programming courses and the students often have no programming background. We never do structured programming — it's almost all interactive with short snippets and global variables.

Having taught courses using Julia to students with prior exposure to Matlab/R/..., I sympathize with this concern. But at the same time, I don't think that using Julia just as a Matlab etc substitute is a viable approach: as demonstrated countless times by questions on Discourse and StackOverflow, this can lead to performance pitfalls that are difficult to fix and understand, possibly entailing an even larger cost than investing in understanding how Julia is different from these other languages (cf posts with topics "I translated this code from Matlab and it is 10x slower").

I think that the key issue is the silent failure; the issue per se is easy to understand and fix. I would suggest keeping the new behavior, but giving a warning in Main (by default; it should be possible to disable it).

[sbromberger]

For me, the larger issue is the perceived inconsistency. That is, I'm ok with Julia doing things differently, but:

• Why should code pasted from a function not work in a REPL?
• No other language I've ever used has this behavior, and it's another barrier to adoption
• Why do for blocks behave differently from begin and if blocks? (if I sort of understand, but a block is [should be] a block.).

Regarding bullet 2, I think this is a bigger deal than we who have been using Julia for a while (and are committed to the language) might understand. I can tell you I'm currently 0 for 7 in convincing my group to write code in Julia; two of those were due to this for loop problem which I couldn't explain because I hadn't been exposed to it before. The remainder I guess we can chalk up to my lack of charisma.

My preference would be to ensure that code pasted from a function into a REPL behaves identically to the function, and that for loops do the expected thing when using them to analyze data interactively; that is, specifically, that they mutate external / global variables when directed without any special keywords.

[stevengj]

I don't think that using Julia just as a Matlab etc substitute is a viable approach: as demonstrated countless times by questions on Discourse and StackOverflow, this can lead to performance pitfalls that are difficult to fix and understand, possibly entailing an even larger cost than investing in understanding how Julia is different from these other languages.

Sorry, but this argument is ridiculous to me. I'm not talking about classes where I'm teaching programming. There's a place for simple interactive computations, and in non-CS classes it's common to be introduced to programming languages as a "glorified calculator" to start with. Teaching performance computing in Julia is an entirely different process — but it doesn't hurt if they've already been using Julia as their "calculator."

If you start by introducing students to Matlab as their "calculator," it's much harder to make the transition to "real" programming, because their first instinct is to do as much as possible with Matlab before jumping ship, at which point their bad habits are ingrained and they are reluctant to learn a new language. In contrast, if you start with Julia as your glorified calculator, when it comes time to do more serious programming you have a much wider array of options available. You don't have to train them to cram everything into "vector" operations or force them to do things badly before they do it right.

Are you saying I shouldn't use Julia in my linear-algebra course? Or that I should only use it if I'm prepared to teach computer science as well as linear algebra?

[piever]

I agree with @stevengj both on the problem (teaching to non programmers becomes much harder) and on the solution (make things work in the REPL and the various IDEs). Including a script would still have the Julia 1.0 scoping rules but that's less of a concern, one just has to be careful to have the "we can put our for loop in a function and then call the function" class before the "we can put our for loop in a file and include the file" class.

This sounds like a good compromise as interactive debugging at the REPL doesn't become more painful than it needs to be (or more confusing to new users), while normal code in scripts has to follow strict scoping rules and is safe from bugs overwriting some variables accidentally.

[tpapp]

Are you saying I shouldn't use Julia in my linear-algebra course? Or that I should only use it if I'm prepared to teach computer science as well as linear algebra?

You may have misunderstood what I was saying (or I did not express it clearly). I was talking about courses that use Julia to teach something domain-specific (eg I taught numerical methods to econ grad students), not CS courses (which I have no experience with).

The point that I was trying to make is that it is reasonable to expect a certain level of difference between Julia and language X (which may be Matlab); conversely, ignoring this can (and does) lead to problems.

Personally, when learning a new language, I prefer to face these issues early on; also, I think simplicity and consistency of the language semantics is more important than similarity to other languages in the long run. But I recognize these preferences as subjective, and reasonable people can have different ones.

[stevengj]

I've created the (unregistered) package https://github.com/stevengj/SoftGlobalScope.jl

If this seems reasonable, I can go ahead and register the package and then use it by default in IJulia (and perhaps submit PRs to Juno etcetera).

[stevengj]

The point that I was trying to make is that it is reasonable to expect a certain level of difference between Julia and language X (which may be Matlab)

Obviously. When I say "use Julia instead of Matlab", I don't mean I'm trying to teach them Matlab syntax in Julia, nor am I specifically targeting former Matlab users.

I prefer to face these issues early on

It's not about differences from Matlab per se. I would really rather not talk about global vs local scope and the utility of a global keyword for static analysis the first time I write a loop in front of non-CS students, or the first time they paste code from a function into the REPL to try it interactively. I would rather focus on the math I'm trying to use the loop to express.

No one here is arguing for soft interactive scope just because that's what Matlab users expect. We are arguing for it because that is what all first-time users will expect, and because long digressions into the unfamiliar concept of "scope" are certain to derail any non-CS lecture where you are showing a loop for the first time. (And even for experienced users, it's rather inconvenient to be forced to add global keywords when we are working interactively.)

[vtjnash]

One other fix not mentioned here is to simply stop making ‘for’ define a scope-block (just function and let would create new scope)

[stevengj]

@vtjnash, I'd rather focus this discussion on things that we can do before Julia 2.0. I agree that having interactive mode behave differently is only a stopgap, though, and we should seriously contemplate changing the scoping rules in a few years.

[vtjnash]

Good point, this also needs import Future.scope 😀

(I think this module/namespace/behavioral effect already is reserved/exists)

As a reminder here, the change was to ensure that code behaves the same in all global scope environments, regardless of what else had previously been evaluated in that module. Before this change, you could get completely different answers (resulting from different scope assignment) simply by running the same code twice or by moving it around in a file.

[stevengj]

Before this change, you could get completely different answers (resulting from different scope assignment) simply by running the same code twice or by moving it around in a file.

The number of complaints I saw about that in practice (zero) are certain to be dwarfed by the number of complaints and confusion you will see (and are already seeing) about the current behavior.

[sbromberger]

Before this change, you could get completely different answers (resulting from different scope assignment) simply by running the same code twice

Do you mean that in the below code, a changes between the first and second for loops? In my mind, that's expected behavior, not a bug.

a = 0
for i = 1:5
  a += 1
end

for i = 1:5
  a += 1
end

[tkf]

What would the proposed REPL-mode do to included scripts?

@mauro3 @stevengj I suppose adding a function (say, exec("path/to/script.jl")) can be done with just a minor version bump? We can also warn exec'ing another file from exec'ed script and then put some pedagogical messages there to nudge them to use include.

[StefanKarpinski]

Some thoughts I wrote down last night while trying to wrap my head around this issue (yet again) to try to figure out what the best course of action might be. No conclusion, but I think this lays out the problem quite clearly. After having thought about this issue for some years, I don't think there is any "ideal solution"—this may be one of those problems where there are only suboptimal choices.

---

People naively view global scope as a funny kind enclosing local scope. This is why global scopes worked the way they did in Julia 0.6 and prior:

• If an outer local scope creates a local variable and an inner local scope assigns to it, then that assignment updates the outer local variable.
• If an outer global scope creates a global variable and an inner local scope assigns to it, then that assignment previously updated the outer global variable.

The main difference, however, is:

• Whether an outer local variable exists, by design, does not depend on the order of appearance or execution of the expressions in the outer local scope.
• Whether a global variable exists, however, cannot be independent of order, since one evaluates expressions in global scope, one at a time.

Moreover, since global scopes are often quite lengthy—not infrequently spread across multiple files—having the meaning of an expression depend upon other expressions an arbitrary distance from it, is a “spooky action at a distance” effect, and as such, quite undesirable.

---

This last observation shows why having the two different versions of a for loop at global scope behave differently is problematic:

# file1.jl
for i = 1:5
  a += 1
end

# file2.jl
a = 1



md5-f03fb9fa19e36e95f6b80b96bac9811e



```jl
# main.jl
include("file1.jl")
include("file2.jl")
include("file3.jl")

Also note that the contents of file1.jl and file3.jl are identical and we could simplify the example by including the same file twice with a different meaning and behavior each time.

[StefanKarpinski]

Another problematic case is a long-running REPL session. Try an example from somewhere online? It fails because you happened to have a global variable by the same name that the example uses for a local variable in a for loop or similar construct. So the notion that the new behavior is the only one that can cause confusion is definitely not accurate. I agree that the new behavior is a usability issue in the REPL but I just want to temper the conversation and present the other side clearly here.

[lmiq]

My small suggestion, that does not deal with the repl problem, but would be useful for didactic purposes when teaching the language not-interactively, at least: define a main block named "program", like can be done in fortran (it is the same as the "let...end" above, just with a more natural notation):

program test
...
end

one could teach the language without going into the scope details and only eventually discuss that point.

[stevengj]

Another problematic case is a long-running REPL session. Try an example from somewhere online? It fails because you happened to have a global variable by the same name that the example uses for a local variable in a for loop or similar construct.

How many mailing-list complaints and github issues have been filed about this by upset users? Zero, by my count. Why? Probably because this behavior is fundamentally unsurprising to people — if you work in global scope, you depend on global state.

So the notion that the new behavior is the only one that can cause confusion is definitely not accurate.

I think this is a false equivalence — there is a vast disparity in the level of potential confusion here. In Julia 0.6, I could explain your example to a student in seconds: "Oh, see this loop depends on a, which you changed here." In Julia 1.0, I'm honestly worried about what I will do if I'm in the middle of a linear-algebra lecture and have to mysteriously type a global keyword in front of students who have never heard the word "scope" in the CS sense.

[JeffBezanson]

we should seriously contemplate changing the scoping rules in a few years.

Absolutely not. Do you seriously want to go back to the pre-v0.2 world (see #1571 and #330) of loop scope?

We have actually never fully supported copying and pasting code from a function line-by-line into the REPL. So we can view this as an opportunity to make that work. Specifically, while it "worked" for for loops, it did not work for inner functions:

x = 0
f(y) = (x=y)

Inside a function, f will mutate the x from the first line. In the REPL it won't. But with a transformation like that in SoftGlobalScope.jl it could work. Of course, we probably wouldn't want that on by default since then pasting stand-alone function definitions wouldn't work. The first thing that comes to mind is a REPL mode for line-by-line function debugging.

[stevengj]

Do you seriously want to go back to the pre-v0.2 world

No, I want to go back to the 0.6 world. 😉

[JeffBezanson]

I guess I was responding more to:

One other fix not mentioned here is to simply stop making ‘for’ define a scope-block

[dawbarton]

We have actually never fully supported copying and pasting code from a function line-by-line into the REPL. So we can view this as an opportunity to make that work.

I very much appreciate this sentiment and for my use cases it would really help. From my perspective it is really about making the REPL as useful as possible rather than changing the scoping rules of the language directly.

That said, the more I think about this problem the more I see the conflicting views I (personally) hold as to what the REPL should do.

To be concrete, I'd very much like it if the REPL matched the scoping rules of a function body; i.e., variables are local rather than global and you can just copy-and-paste code directly from a function and know that it will work. I imagine a naive implementation would be something like let-block wrapping (as has been mention previously) of the form

julia> b = a + 1

being transformed into

let a = _vars[:a]::Float64 # extract the variables used from the backing store
    # Code from the REPL
    b = a + 1
    # Save assigned variables back to the backing store
   _vars[:b] = b
end

Done properly (i.e., by someone who knows what they are doing), I imagine that this would have a number of benefits over the existing REPL. 1. previous workflows with interactive data analysis/computation just work. 2. far fewer posts on Discourse where the basic response is "stop benchmarking with global variables" - everything would be local and so hopefully fast! :) 3. copy-and-paste to/from a function body works as expected. 4. a workspace() like function is trivial if the backing store is some sort of Dict; just clear it out. 5. globals become explicit - things are local unless you specifically ask for them to be global; this is a big advantage from my perspective, I don't like implicitly creating globals. A very minor final point (and I hestiate to add this!), this would match the behaviour of Matlab making it easier for people transitioning - at the Matlab REPL all variables seem to be local unless explicitly annotated as global.

Until a few hours ago this story sounded great to me. But after Jeff's comment about functions I thought about pasting in stand-alone function definitions and how this approach would basically prevent that since function definitions should go in the global scope (at least, that is probably what is intended); but then what if they were intended to go into the local scope (an inner function)? There is no information to disambiguate the two possibilities. It would seem that two REPL modes are needed, one with local scope and one global scope. On one hand that could be very confusing (imagine the Discourse posts...) but on the other it could be extremely useful. (Having both REPL modes would also be non-breaking since you are just introducing new functionality :) )

Going for the halfway house of SoftGlobalScope.jl might end up being the least confusing compromise but my worry is that it's just another set of rules to remember (which things work in the REPL but not in my function body/global scope and vice-versa).

Apologies for the long post but I think this is important for usability (and it helped me think it through!).

[tpapp]

How many mailing-list complaints and github issues have been filed about this by upset users? Zero, by my count. Why? Probably because this behavior is fundamentally unsurprising to people — if you work in global scope, you depend on global state.

Hmm, did you really make a systematic study of this? I must have missed that. Nevertheless, this does not mean that this behavior is not a source of bugs or unexpected results; just that after the user has figured it out, it was recognized as correct behavior and thus did not prompt an issue/complaint.

In Julia 1.0, I'm honestly worried about what I will do if I'm in the middle of a linear-algebra lecture and have to mysteriously type a global keyword

I sympathize with this problem. When I taught into some simple programming to econ students necessary for a course, I usually suggested that they go back and forth between wrapping code in functions, and simply commenting out function and end and running things in the global scope, so they could inspect what is happening. This pretty much made up for the lack of debugging infrastructure at that time in Julia.

It appears this approach is no longer feasible. But I wonder if it was really the right way to do it anyway, and in the meantime various things have improved a lot (#265 was fixed, Revise.jl and recently Rebugger.j have improved workflow/debugging considerably).

It seems that this issue does not bother experienced users very much, the main concern is confusion in a pedagogical setting. I have not experimented with this myself yet, but I wonder if we could adapt our approaches to teaching instead, eg introduce functions before loops, avoid loops in global scope. These are elements of good style anyway and would benefit students.

[mauro3]

Just a wee note: whilst special casing the global scope of the REPL, will allow copy-pasting code into and from functions, it will not allow copy-pasting into/from the global scope of another module.

[stevengj]

I wonder if we could adapt our approaches to teaching instead, eg introduce functions before loops, avoid loops in global scope.

This is totally impractical in a class that is not focused on teaching programming. I might as well not use Julia in my classes if I can't use it interactively and/or have to write functions for everything first.

(And it's not just pedagogical. Loops in global scope are useful for interactive work. And one of the main reasons people like dynamic languages for technical computing is their facility for interactive exploration. Not all coding is performance-oriented.)

[StefanKarpinski]

There have been dozens of threads and issues over the years in which people are confused or complaining about the old "soft/hard scope" distinction, so claiming that no one has ever been confused by or complained about the old behavior is just... not true. I could dig some of them up, but you were around, @stevengj, so you can dig them up just as easily and I have a hard time believing that you didn't notice or don't remember these complaints and conversations.

[stevengj]

@StefanKarpinski, I'm specifically referring to people complaining that a global loop depends on global state. I don't recall anyone complaining that this was bad behavior, nor can I find any examples of this.

I agree that people have been confused about when and where assignment defines new variables, but it has usually been in the other direction — they wanted local scopes to act more global (rather than vice versa), or to not have a distinction between begin and let. IIRC, the complaint was never that assigning to a global variable in a global loop had the surprising side effect of modifying a global.

The whole issue of scoping is confusing to new users, and it will continue to be so. But the confusing part was not cases where assigning to a global variable name affected the global state. The current behavior makes this worse, not better.

[mauro3]

@StefanKarpinski: I have the feeling that previously, the confusion with soft/hard scope was more of a theoretical nature (of people reading the manual) rather than of a practical on (of people getting unexpected results). That was definitely how it was for me and what, e.g., the search results here support this; I found one counter-example here.

On the other hand, this new behavior will not confuse people when reading the manual, but when using the REPL. Arguably the latter is worse.

[stevengj]

SoftGlobalScope.jl is now a registered package. My intention is to enable it by default (opt-out) for IJulia, at least this semester.

[stevengj]

@mauro3, even your "counter-example" is about someone confused by hard scope, not by soft scope. Making more scopes "hard" in 0.7 is certain to create more of this sort of confusion.

[StefanKarpinski]

I would point out that IJulia has the interesting possibility of making variables local do blocks by default. I.e. if you do this in a single block then it works:

t = 0
for i = 1:n
    t += i
end
t

... and t is only visible within this evaluation block. If you wanted it to be visible outside, you would have to do this:

global t = 0
for i = 1:n
    global t += i
end
t

I have also considered a similar approach for Julia where the blocks are files rather than module. In other words just doing t = 0 at top scope creates a variable that is file-local rather than global. To declare a truly global variable, you'd need to write global t = 0 which would then be visible throughout the module. Perhaps too weird, but it has occurred to me many times over the years.

[stevengj]

IJulia has the interesting possibility of making variables local do blocks by default

@StefanKarpinski, I think this would be even more confusing, and would run counter to how notebooks are normally used. It is common for the same variable to be used/modified in multiple cells, so requiring a global keyword for all inter-cell variables is a nonstarter to me — it would require even more discussion of scope concepts than the issue with for loops we've been discussing here.

[JeffBezanson]

I think as long as we all agree --- as we seem to --- that this is mostly or entirely an issue for interaction, then we have a way forward. If we special-case this in the REPL (as is being done for IJulia), the only bad case is developing something in the REPL and then moving it to top-level script code. Arguably that's the point where you should introduce functions, so I don't think it's so bad. Copy-pasting code between the REPL and function bodies will work (mostly), which is probably good enough.

Then we also have the option of further justifying/clarifying the distinction by making REPL variables somehow local to the REPL --- i.e. not normal global variables, not available as Main.x. This is very similar to what @StefanKarpinski just proposed above, but shared among all input blocks/cells.

[derijkp]

From a practical point of view, getting this "fixed" in the REPL is not
only important for teaching/non-programmer users. This behaviour also
makes interactive debugging via the REPL (by copy-pasting parts) very
unpractical. This mode of debugging can sometimes be preferable (even to a
good debugger and) even for experienced programmers (and is often one of
the reasons to prefer a dynamic language). Of course for the experienced
programmers, being optional shouldn't be a problem; For novice users it
would be preferably the default.

@StefanKarpinski
As a naive programmer, I don't really see what is so wrong in viewing the
global scope as a funny kind enclosing local scope, especially in dynamic
languages. I do understand that from a compiler point of view it is not
necessarily correct (in Julia), but it is a nice, easy and useful model
for a (naive) programmer. (I also suspect it might be actually implemented that way in
some languages).
Julia also seems to presents it that way to the programmer:
The following function function will give the error "a not defined", which
it will not do if a=1 is put before the for loop.

function test()
for i = 1:10
a=a+i
end
a=1
@show a
end

which, unless I complete misunderstood, seems at odds with "Whether an
outer local variable exists, by design, does not depend on the order of
appearance or execution of the expressions in the outer local scope".

I very much agree with avoiding "spooky action at a distance", and much
prefer explicit definition for using globals at the function/call stack
level and would personally also like having something like loading from a file in
its own scope, and requiring explicit definition for using global variables.
At the level of loops is going a bit to far for me though, as the
definitions/context is usually quite near.
The 3 files example is a bit contrived (and fails with the expected "a not
defined" error): You would normally put the initial definition in the same
file.
There is actual spooky danger in this (and I have been bitten by it
in other languages) in that includes are run in the global scope, so you
are inadvertently defining a global variable that may interfere with other
code. However, having to use global in the loop is not a solution to
this problem.

wrt to the long-running REPL session:
The current behaviour replaces a very rare and easy to spot failure mode
for running an online example in the REPL (you miss copy/pasting the
initial definition of the variable before the loop, and already have the
same variable defined globally from something previous) with not being
able to run an online example correctly if it is part of a function
(without adding global everywhere), and not solving the problem if it is
not (if global is already there in the online code, you will still use the
wrong value in the already existing global variable)

[timholy]

I should have tuned into this earlier, but after a brief moment of concern all seems to be well.

We have actually never fully supported copying and pasting code from a function line-by-line into the REPL...The first thing that comes to mind is a REPL mode for line-by-line function debugging.

Indeed Rebugger (which is exactly that) works properly on 1.0 only because it lacks the scope deprecation of 0.7, and could never be made to work on 0.6. However, I'm pleased to be able to verify that SoftGlobalScope.jl seems not to break that. For example, if you step deeply enough into show([1,2,4]) you get here:

show_delim_array(io::IO, itr::Union{SimpleVector, AbstractArray}, op, delim, cl, delim_one) in Base at show.jl:649
  io = IOContext(Base.TTY(RawFD(0x0000000d) open, 0 bytes waiting))
  itr = [1, 2, 4]
  op = [
  delim = ,
  cl = ]
  delim_one = false
  i1 = 1
  l = 3
rebug> eval(softscope(Main, :(@eval Base let (io, itr, op, delim, cl, delim_one, i1, l) = Main.Rebugger.getstored("bbf69398-aac5-11e8-1427-0158b103a88c")
       begin
           print(io, op)
           if !(show_circular(io, itr))
               recur_io = IOContext(io, :SHOWN_SET => itr)
               if !(haskey(io, :compact))
                   recur_io = IOContext(recur_io, :compact => true)
               end
               first = true
               i = i1
               if l >= i1
                   while true
                       if !(isassigned(itr, i))
                           print(io, undef_ref_str)
                       else
                           x = itr[i]
                           show(recur_io, x)
                       end
                       i += 1
                       if i > l
                           delim_one && (first && print(io, delim))
                           break
                       end
                       first = false
                       print(io, delim)
                       print(io, ' ')
                   end
               end
           end
           print(io, cl)
       end
       end)))
[1, 2, 4]

So it works fine on 1.0 (with or without softscope). On 0.7, evaluating this (with or without softscope) will yield

┌ Warning: Deprecated syntax `implicit assignment to global variable `first``.
│ Use `global first` instead.
└ @ none:0
┌ Warning: Deprecated syntax `implicit assignment to global variable `first``.
│ Use `global first` instead.
└ @ none:0
[ERROR: invalid redefinition of constant first
Stacktrace:
 [1] top-level scope at ./REBUG:9 [inlined]
 [2] top-level scope at ./none:0
 [3] eval(::Module, ::Any) at ./boot.jl:319
 [4] top-level scope at none:0
 [5] eval at ./boot.jl:319 [inlined]
 [6] eval(::Expr) at ./client.jl:399
 [7] top-level scope at none:0

So 0.7/1.0 are definitely a step forward, and if softscope makes certain things easier without breaking important functionality that's great.

The biggest concern, therefore, is simply how to intercept this appropriately without tanking other packages (https://github.com/stevengj/SoftGlobalScope.jl/issues/2).

[stevengj]

@timholy, SoftScope does not touch the arguments of macro calls (since there is no way to know how the macro would rewrite it), so :(@eval ...) is protected.

[JeffBezanson]

seems at odds with "Whether an
outer local variable exists, by design, does not depend on the order of
appearance or execution of the expressions in the outer local scope".

The (outer) local variable a exists, but has not been assigned yet. If the loop tried to assign to a before reading it, the assignment would be visible outside as well.

In general, creating a variable binding and assigning a value to it are separate steps.

[mauro3]

What is the time-line on this? It seems it would be a great improvement to user usability. And at this "critical" time of Julia with 1.0 out, it would seem advantageous to get this fixed asap (in the way suggested by Jeff above) and tag a new Julia version or REPL version. (Sorry for this arm-chair comment, as I certainly will not fix this!)

[derijkp]

@JeffBezanson
I was going to argue that while this is true (for the implementation/compiler), the naive julia programmer cannot not see any different behaviour from his simpler conceptual model (a variable starts existing at the moment it is defined). Unfortunately you are right, the following code will not give an error, while it will give an error if you leave out the a=2 at the end
function test()
for i = 1:10
a=1
end
println(a)
a=2
end
I'll explain the unfortunately: I can understand the behaviour (because i've worked with compiled languages before) but still find it confusing and unexpected. How bad must it be to someone with only scripting experience or new to programming. Also, I found some code that shows the behaviour, I do not see a useful application (maybe you can help me there)

On the REPL:
I just got more convinced that changing the scoping back to "normal" at least in the REPL (no need to add global in loops) is high priority: I was testing some things in the REPL today and got (again) bitten by it, taking some time to realize it. Given that I follow Julia already some time, really like a lot of it, am even following this thread about the problem, I would even call it a showstopper: A newbee (to the Julia) testing out the language is very likely not to find out the problem and just give up.

[StefanKarpinski]

@jeffbezanson and I are both on long-awaited vacations (I should not be reading this). We can figure out what to do in a week or so.

@derijkp, while the feedback is appreciated, scoping rules are not up for a broader debate or revision. The only thing on the table is special-casing interactive eval. The SoftGlobalScope package is already an excellent experimental implementation and it may just be a matter of making that part of Base and using it in the REPL.

[JeffBezanson]

@derijkp A short answer is that I think it's easier if the scope of a variable corresponds to some block construct (e.g. the body of a function or loop). With your suggestion, the scope of a variable would be some subset of a block, which I think is ultimately more complex and confusing --- you can't point to a syntactic form that corresponds to the scope of the variable.

Yes, I can believe this is a mismatch for some people's intuition. But you can only optimize for the first ten minutes of using a language up to a point. The real question is, how hard is it to teach/learn how it works, and which design will save time in the long run (by making the language simpler, making it easier to develop tooling, etc.)?

[JeffreySarnoff]

(in agreement with much of the above about modifying the behavior of the REPL)
I'd like to see the REPL be in a way that does not lead to this stackoverflow question
and sooner would be best as many new eyes are looking at Julia

[jlperla]

I agree... And also think that the scoping rules shouldn't necessarily change, just all of the interactive interfaces (i.e. the REPL, Jupyter, and Juno control enter)

This is not just about beginners learning a new rule. If you can't copy and paste fragments of code into the REPL, jupyter etc and also into functions, it is a major annoyance for intermediate programmers as well.

Of course, I also agree with the other posters... with beginners they going to take code fragments they see within functions, copy I into scripts, and be completely confused when it doesn't have the same behaviour when copied inside of a function, in juno, the repl, and jupyter. There will be 100 stack exchange questions which come down to the same issue. Intermediate programmers are going to have all sorts of homegrown solutions with wrapping in let blocks, etc which will confuse things further

[tpapp]

There will be 100 stack exchange questions which come down to the same issue. Intermediate programmers are going to have all sorts of homegrown solutions with wrapping in let blocks, etc which will confuse things further

Possibly, but at this stage this is hypothetical (also the OP of the question linked is asking about the rationale for the scoping rule, as opposed to being confused about it).

Also, while I respect the teaching experience of everyone who has concerns about this, whether this turns out to be a big deal in the classroom is something that time will tell.

[JeffreySarnoff]

the questioner appears to have been confused by it: "I wonder if this is intuitive to beginning julia users. It was not intuitive to me ..."

[jlperla]

the questioner appears to have been confused by it:

Not to mention that this is someone who clearly knows enough about programming languages to understand the nuances of scope. What about all of the matlab type users that are completely ignorant of these topics..., and probably will never invest enough time to understand the nuances.

[crstnbr]

Possibly, but at this stage this is hypothetical

I've already answered multiple questions related to this on stackoverflow, mostly by new users, and even more in real life (last one just yesterday, from a Matlab user, who saw this as a no go).

[rickhg12hs]

There will be 100 stack exchange questions which come down to the same issue.

In my "spare time", I've been adding scope, scoping, and global-variables tags to the SE questions. I only stop because of lack of time, not because there aren't more.

[StefanKarpinski]

Conclusion after much discussion including triage: we're going to include something along the lines of SoftGlobalScope in Base and use it in the REPL and all other interactive evaluation contexts. @JeffBezanson has pointed out that the way this is implemented is actually essentially the same as how soft scope was previously implemented, so to some extent we're coming around full circle. The difference is that now there is no scope behavior in modules or scripts, only in REPL-like contexts. I also think that _explaining_ soft scope as a source rewrite is clearer than trying to distinguish between hard and soft scopes (which we never how Jeff explained it, I might point out).

[mauro3]

These two statements confuse me a bit as they seem a bit contradictory:

and use it in the REPL and all other interactive evaluation contexts

there is no scope behavior in [...] scripts, only in REPL-like contexts.

Does this mean that the module Main has sometimes a soft scope (say at the REPL prompt) and sometimes a hard scope (say when julia -L script.jl)? Would it not make sense to say that Main always has soft scope? And a module can opt-in to soft scope by using SoftGlobalScope?

[derijkp]

(I guess) scoping rules cannot be changed in scripts because it would be backwards incompatible, i.e. would break the promise that any code written for 1.0 will run on any 1.* version. You are correct though that the same problem with scoping for the REPL also applies to scripts (naive user at a complete loss why his/her code does not work properly when run as a script). A way to solve/alleviate this problem without major incompatibilty would be to add an option to the julia cmdline to use softscope (or alternative) , e.g. julia -f programfile, and show this option in any description/tutorial that a beginner is likely to come across.
I also see a potential alternative for the softscope that may have some advantages (though i am probably overlooking disadvantages): What if a file (a called script) would always introduce its own local scope: scoping rules would be in complete consistency with those in functions, and with the expectations of a lot of users. It would also remove a lot of the performance liabilities with new users:
No more unneeded globals (globals would have to be explicitly defined), and code might be compiled
(How many times have you had to say to put everything in a function, and to avoid using globals?)

[richardreeve]

I've just hit this and was completely boggled to be honest, having never seen it before in any other language. I'm planning on introducing an optional Julia course for advanced R users in my uni later this year once things have settled down, and my students will hit this on day 0 when they start randomly typing things in the REPL. And the fact that for loops behave differently from if statements just rubs salt in the wound, however logical this may be in terms of scoping. Scope inside functions is sufficiently hard to get biology students to grasp, the idea of having to explain _albeit perceived_ glaring inconsistencies in it in the REPL / in a script / in a for loop / in an if statement (because that's what we're talking about here) in a way that is different from every other language on earth makes me very sad.

I understand the backward compatibility promise that was made, but having this work _as expected by every non-cs person on the planet (and most cs people I suspect)_ seems like a bugfix rather than a backward compatibility issue - we're not saying that every bug will be reproduced for ever are we? The REPL fix is obviously essential, so it's great that you're proposing this, but then having to explain you can't copy a script into the REPL and expect the same behaviour seems as bad as or worse than the original problem.

Please, please, please think about treating this as a bugfix and pushing it out with scripts as well as the REPL - even if there's an switch to go to the "old" behaviour - and doing it as soon as possible in 1.0.1.

[jebej]

A colleague that I was trying to get to learn julia also just ran into this. Having to explain the whole global vs. local variable thing at the first steps is not ideal...

[stevengj]

I don't think treating this as a "bugfix" is in the cards, because it would break the 1.0 stability contract. However, it seems reasonable to me to use softscope for scripts run with julia -i (i.e. "interactive" mode).

(That is, there would be a flag --softscope={yes|no} and it would default to the value of isinteractive.)

[StefanKarpinski]

We'll have to consider the script mode choice.

[stevengj]

For that matter, it's not crazy to me to default to --softscope=yes for any "script", i.e. for julia foo.jl, and only turn on the "hard" scoping rules for modules and include (at which point you should really be putting most code into functions).

[jlperla]

For that matter, it's not crazy to me to default to --softscope=yes for any "script",

That. The other one to seriously consider is Juno. Remember that people will <shift-enter> through their code to do interactive development(especially when working with the regression tests) and then later expect to be able to run the same file. Should it matter if the code is in a @testset or not (which I think might introduce a scope)? It would be very confusing to the user if the same text changes when in a @testset vs. not when using Atom's integration, and is inconsistent with doing ] test as well.

It sure sounds to me like the best solution is that the hard-scope is simply an opt-in thing, where if every other usage (including include within scripts) uses softscope unless you say otherwise.

[JeffBezanson]

different from every other language on earth

Do you want to write var x = 0 to introduce every variable? That would also "fix" this, and be more like other languages.

we're not saying that every bug will be reproduced for ever are we

That is not how this works. You can't get any change to the language you want just by calling the current behavior a bug.

I reeeally don't think there should be a command line option for this. Then every piece of julia code will have to come with a comment or something telling you which option to use. Some kind of parser directive in a source file would be a bit better, but even better still would be to have a fixed rule. For example, hard scope inside modules only might make sense.

Let me try again to provide an explanation of this that might be useful for avoiding the mania, hysteria, and carnage people are seeing in the classroom:

"
Julia has two kinds of variables: local and global. Variables you introduce in the REPL or at the top level, outside of anything else, are global. Variables introduced inside functions and loops are local. Updating global variables in a program is generally bad, so if you're inside a loop or function and want to update a global, you have to be explicit about it by writing the global declaration again.
"

Perhaps that can be improved; suggestions welcome. I know, you'd rather not need any sort of explanation at all. I get that. But it doesn't seem so bad to me.

[jlperla]

I reeeally don't think there should be a command line option for this. Then every piece of julia code will have to come with a comment or something telling you which option to use. Some kind of parser directive in a source file would be a bit better, but even better still would be to have a fixed rule

I agree. Sounds like a teaching and communication headache to me.

For example, hard scope inside modules only might make sense.

Just so I understand: if I had a short script (not in a module!) in a .jl file which I had copied from an IJulia notebook, then if I ran that code in either the REPL directly or shift-enter in Juno, then it would behave consistently as soft-scope... but if I copied it instead of a module block then it would yell at me about globals? But if I copied that code inside of functions inside of a module, then it should work.

If so, that makes complete sense,is very teachable and coherent. Top-level scripts are an interactive interface for exploration, etc. but you would never put that kind of code in a module. Modules are something that you should fill with functions are very carefully considered globals. It would be easy to tell people about those rules.

[richardreeve]

Do you want to write var x = 0 to introduce every variable? That would also "fix" this, and be more like other languages.

No, I'd rather not! But scripting languages that have a REPL rarely do that (e.g. ruby, python, R, ...), they behave like Julia v0.6 did.

Julia has two kinds of variables: local and global. Variables you introduce in the REPL or at the top level, outside of anything else, are global. Variables introduced inside functions and loops are local. Updating global variables in a program is generally bad, so if you're inside a loop or function and want to update a global, you have to be explicit about it by writing the global declaration again.

I completely understand what you're saying here, and I won't (touch wood!) make this mistake again. But the whole problem I'm worried about is not me. I've found it relatively easy to introduce scope (without mentioning it directly) when I explain that variables inside functions can't see ones outside and vice-versa (even though that's more an aspiration than a fact in R!), because functions themselves are already a _relatively_ advanced concept. But this hits much earlier in the learning curve here where we don't want anything remotely as complicated as scope to be impinging on people...

Note also it's not just "_variables you introduce in the REPL or at the top level, outside of anything else, are global_" and "_variables introduced inside functions and loops are local_", it's also that variables in if statements in the REPL or at the top level are global but variables in a @testset are local. We end up down a rabbit-hole of "just try it and work out for yourself whether it's local or global, good luck".

However, I agree with @jlperla - the proposal that "hard scope inside modules only might make sense" seems completely fine to me! Modules are a sufficiently advanced concept again... if soft scope works for the REPL and scripts, that's absolutely fine.

[JeffBezanson]

we don't want anything remotely as complicated as scope to be impinging on people...
at the top level are global but variables in a @testset are local

What I'm trying to get at is that I feel a simple description of global vs. local is sufficient for early-stage teaching --- you don't even need to say the word "scope" (it does not occur at all in my explanation above). When you're just showing some simple expressions and loops in the REPL, you're not teaching people about testsets and you don't need an exhaustive list of the scoping behavior of everything in the language.

My only point is, this change does not suddenly make it necessary to teach lots of details about the language up front. You can still ignore the vast majority of stuff about scopes, testsets, etc., and a simple line on global vs. local should suffice.

[jlperla]

and a simple line on global vs. local should suffice.

In a world where everyone started writing all of their code from scratch, I would agree completely.

The issue is that you need to teach students not just about scope, but also about understanding the scope of where they copy-pasted code they got from. You need to teach them that if they copy-paste code that is on stackexchange within a function or a let block that they need to scan through it and find where to add "global" if they are pasting it into the REPL or a .jl file. But if they are copying that code inside a function or into the Jupyter notebook. they shouldn't. And if they find code inside of a stackexchange or tutorial page that has global variables in it, but they want to copy and modify that code inside of their own function, then they need to strip out the global.

And then students start asking why does for create this scope they need to worry about but not other things....

[JeffBezanson]

We end up down a rabbit-hole of "just try it and work out for yourself whether it's local or global, good luck".

Pop quiz: in julia 0.6, is x global or local:

for i = 1:10
    x = i
end

The answer is that there's no way to know, because it depends on whether a global x has been defined before. Now, you can say for sure that it is local.

[StefanKarpinski]

Folks, this discussion is verging on no longer being productive. Jeff knows very well that the old behavior was nice in the REPL. Who do you think designed and implemented it in the first place? We have already committed to changing the interactive behavior. A decision still needs to be made about whether a "script" is interactive or not. It sounds interactive when you call it "a script" but it sounds far less interactive when you call it "a program"—yet they are exactly the same thing. Please keep the replies short and constructive and focused on the things which still must be decided. If there's comments that deviate from this, they may be hidden and the thread may be locked.

[StefanKarpinski]

One thought that I had but we dismissed as being "too annoying" and "likely to cause the villagers to get out their pitchforks" was that in non-interactive contexts, we could require a local or global annotation in "soft scope". That would guarantee that code from a module would work the same if pasted into the REPL. If we applied that to "scripts"/"programs" then the same would be true of them.

[lmiq]

When I was first introduced to Julia (not a long time ago, and I come from a Fortran background mostly), I was taught that "Julia is compiled and fast at the function level, thus everything that must be efficient must be done inside functions. In the main 'program' it behaves like a scripting language". I found that fair enough, as I cannot imagine anyone doing anything too computationally demanding without understanding that statement. Therefore, if there is any sacrifice in performance at the main program for using the same notation and constructions than in the functions, I find that totally acceptable, much more acceptable than trying to understand and teach these scoping rules and not being able to copy and paste codes from one place to another.

By the way, I am a newbie in Julia yet, having chosen it to teach some high-school and undergraduate students some basics of simulations of physical systems. And I am already hopping this issue returns to the 'normal' behavior of previous versions, because it gives us quite a headache.

[StefanKarpinski]

This conversation is locked now and only Julia committers can comment.

[stevengj]

@JeffBezanson, what would be the plan to implement the semantics you suggested in this discourse thread, initially only in the REPL and opt-in elsewhere?

It sounds like you are planning to put that directly into the lowering code (julia-syntax.scm), rather than as a syntax rewriting ala SoftScope.jl? Or would you rather have it as syntax rewriting first (modifying SoftScope to the proposed rule and converting it to a stdlib), and defer putting it into the lowering code for a later Julia release?