Elvish: Add source command

[alexherbo2]

Source file.

[ALSchwalm]

Is -source what your looking for? I think it's just not documented yet (I'm not sure how complete it is at the moment).

[xiaq]

Yes, we have -source :) The reason that it is marked experimental is that it does not interact well with the static checker. For instance, if you have the following in a.elv:

x = haha

then you should be able to do -source a.elv; echo $x. But you cannot yet.

I will close this issue when this problem with -source is resolved, and it will be renamed source at that time.

[krader1961]

FWIW, I stumbled across this issue because I found I needed source to implement a function in the top level scope but without putting it in my rc.elv config script. Namespaces are great and should be used for anything meant to be used by others or that is non-trivial. But for defining personal functions to be used interactively it would be nice if there was something analogous to fish's autoloaded function mechanism. Without that I have to either put those functions in my rc.elv or put a -source statement in that script. The latter is preferable to keep things organized.

[krader1961]

I'm wondering if we really need a source command. What if the use command supported the equivalent of Python's from module import *? That is, import the module with its members qualified by the namespace but also alias all the members into the current scope? That is, if I did the hypothetical use nx as * I could reference the embedded function as either nx:nx or nx and $nx:status or $status. Obviously this is dangerous but no more so than source and is one less command that needs to be implemented. Obviously, source is slightly more general in that you can give it an arbitrary path. But that could potentially be solved by allowing absolute pathnames or explicitly relative (i.e., they begin with ./) with use to mean don't search the usual locations.

[zzamboni]

@krader1961 I really like your idea of being able to specify which symbols to import in the use statement. Something like Clojure's :refer argument to require. For example:

use re           # default, everything is imported under the re: namespace
use re [replace] # `replace` is also "linked" into the current namespace
use re [:all]    # or some other special value, to specify that all symbols should be linked into the current namespace

[xiaq]

Importing names from modules is not hard to implement, but I doubt its usefulness beyond being able to put your wrapper functions ("aliases") in a module and pouring that into rc.elv. Go does not have this function and I like its absence, because it is now always clear which names are imported because they are always namespace-qualified. It is also the style recommended by Google Python Style Guide.

Also, importing names that are not known beforehand (use some-mod [:all], borrowing @zzamboni's syntax) poses a unique challenge for Elvish.

To start with, Elvish always compiles a chunk of code before evaluating it, and part of the compilation phase is checking variable names. In dynamic languages like Python, use of undefined variables is a runtime error. In Elvish, it is a compilation error and will prevent the whole chunk from executing. For instance, the following code won't do anything (provided that you didn't define $undefined-var):

echo before
echo $undefined-var
echo after

whereas the following Python code will print out before and then a traceback for a NameError exception:

print('before')
print(undefined_var)
print('after')

Now, to compile code like the following

use some-mod [:all]
echo $some-var

Elvish has to load some-mod in compilation phase in order to check whether $some-var has been defined. Currently, Elvish only loads the module in the evaluation phase (which implies that access to names like $some-mod:some-var are not checked, but let's not talk about that for now). It seems that moving the loading procedure to the compilation phase will solve the problem. Now let me talk about Elvish's editor.

If you type something like echo $undefined in the editor, Elvish will complain that $undefined is undefined right away, before you hit Enter (in fact, it complains about $u, $un, $und, ... all the way up). That error message is driven by the compiler: Elvish parses and compiles your input on every keystroke, and notifies you of any errors.

Now if the module loading is to be moved to the compilation phase, that means that when you type use some-mod, Elvish will try to load all of s, so, som, ... up until some-mod. This makes the compilation phase much more expensive because module loading usually involves reading from the disk.

To summarize the challenge in supporting use some-mod [:all]:

• Elvish always checks variable names in a compilation phase ("statically") before actually evaluating the code.

• Supporting "importing all names" means that the compiler now needs to look at the module file. Currently this is only done during the evaluation phase.

• Making the compiler look at the module file is expensive because the editor compiles the input on each keystroke.

Code references:

• For the static checking of variable names, see here.

• For the implementation of use, see here.

---

On the other hand, there is no problem with use some-mod [name1 name2], because that gives enough information to the compiler. But again, I doubt its usefulness beyond being able to put aliases in a separate module.

---

So how can the current -source work?

The way -source now works is that it does not inform the compiler of new names. That means that if you have the following in a.elv:

foo = bar

The following code won't compile:

-source a.elv
echo $foo

because -source makes no effort in telling the compiler that $foo might be defined in a.elv. (If you run this from the editor, you need to use Alt-Enter to insert a literal newline.)

However, if you run those two lines as separate commands from the editor, it does work. This is because Elvish creates a new compiler for every new command, and the new compiler knows all names that are defined.

This badness of behavior is why I chose to mark source as internal (by calling it -source).

Code references:

• For the realtime compilation from the editor, see here.

• For the implementation of -source, see here.

[zzamboni]

@xiaq thank you for the comprehensive explanation. As you say, the only benefit for importing names would be to allow defining "top-level functions". But the usefulness of this can be debated. At the moment, Elvish forces the user to be explicit about top-level functions. For example, to use my cd library as the default cd command, I have to do the following:

use dir
fn cd [@dir]{ dir:cd $@dir }

Which makes my configuration more explicit about what is being overriden. This is good for clarity at the expense of verbosity and exposing some more "implementation details" in my top-level configuration.

If dir.elv was allowed to define top-level functions, it could automatically define that function. One could argue this is useful as it allows more automated configuration (just load the package) at the expense of being less explicit about what the module is doing.

I think being able to specify the names to import would hit the right balance: abstract away implementation details (such as how to wrap a function) while still making the configuration explicit about what names are being imported:

use dir [ cd ]

[xiaq]

For uniformity, it is better if we can import variables, not just named functions. Named functions are really just variables who names start with '&', so it makes more sense to support importing variables.

Also, I find it clearer to separate the loading of modules (already supported by use) and the importing of names from that module (the functionality being discussed). I think it will be nicer to have a separate (say) refer command, so that what you propose as use dir [ cd ] will be written as:

use dir # loads module "dir"
refer dir ['&cd'] # aliases $dir:&cd as $&cd

[krader1961]

For uniformity, it is better if we can import variables, not just named functions....

Agreed. But do we really need a new command? I don't see the reason to implement refer dir ['&cd', 'pwd'] rather than use dir ['&cd', 'pwd']. Also, I want to make sure we're all on the same page in that by "aliasing" the name we're talking about binding a new name, in the local scope, to an object (var or function) in a namespace. Such that changing the value assigned to $pwd or $dir:pwd is reflected in the other var name.

[xiaq]

@krader1961 We have the same definition for aliasing.

There are three reasons for a separate refer command:

1. Module loading and name aliasing are two orthogonal concepts. Giving them separate constructs promotes composibility in general.

2. It should also be possible to alias names from pre-loaded namespaces like e: for external commands, or E: for environment variables. For instance, refer E [PATH] aliases $E:PATH to $PATH. Since E: is not really a module, saying use E [PATH] is confusing because there is no module loading involved. It will complicate the logics for use.

3. The semantics of use mod [name1 name2] is non-obvious. It loads the module named by mod and imports $name1 and $name2 from that module, but is $mod:name1 and $mod:name2 or any other name $mod:name3 also valid? In other languages, the import construct only does either of importing individual names into the current module (in Python, from mod import name1, name2) and importing the whole module as one namespace (in Python, import mod), but not both.

[krader1961]

Good points, @xiaq. Your first two points in particular are worth noting since composition is a useful feature in general. I hadn't considered the case of aliasing objects from semi-magic namespaces like E: or e: into the local scope. It's not clear doing that should be encouraged but it should be supported if we support aliasing objects from other namespaces imported by use.

The refer namespace [name1 name2...nameN] syntax and the refer command name seem confusing to me as a native English speaker. What about import [name1 name2...nameN] from namespace? Or alias [...] from namespace? Using either import or alias is probably clearer to a native English speaker. However, alias may be a poor choice due to its behavior in other shells like bash, zsh, and fish.

As for your point 3 I recognized that difference from Python behavior but deemed it acceptable given that Elvish is only influenced by Python. So the dual behavior of my proposal being equivalent to Python's import mod and from mode import name1, name2 could be justified. Albeit the ambiguity may not be ideal.

[xiaq]

The namesake of refer is from Clojure, although Lisp people are not exactly known for giving things friendly names :)

As you pointed out, alias is not a good choice for its potential confusion with alias in traditional shells. I avoided import because ImageMagick provides a command named import.

[krader1961]

In light of commit ff7c500 that formally deprecates the experimental -source builtin this issue should probably be closed as "will not implement".