Packer.nvim: Behaviour of requires seems slightly unintuitive

[rickysaurav]

When plugin A requires plugin B, I expect Plugin B to be available before Plugin A starts loading. Packer adds Plugin B to "after" of A i.e. plugin B is loaded after plugin A is loaded, which is slightly unintuitive in terms of what I would expect from a dependency.
This causes issues when the dependency of an optional plugin is not available in "start" packages.
Reproduction:

vim.cmd [[packadd packer.nvim]]

local packer = require("packer")

packer.startup(
    function(use)
        -- Packer can manage itself as an optional plugin
        use {"wbthomason/packer.nvim", opt = true}
        use {
            "nvim-lua/telescope.nvim",
            requires = {{"nvim-lua/popup.nvim", opt = true}, {"nvim-lua/plenary.nvim", opt = true}},
            cmd = {"Telescope"},
        }
    end
)

Calling "Telescope find_files" results in the following error.

Error detected while processing function <SNR>8_load:
line    1:
E5108: Error executing lua [string ":lua"]:111: Vim(let):E5108: Error executing lua ...ite/pack/packer/opt/telescope.nvim/lua/telescope/log.lua:2: module 'plenary.log' not found:
        no field package.preload['plenary.log']
        no file './plenary/log.lua'
        no file '/usr/share/luajit-2.0.5/plenary/log.lua'
        no file '/usr/local/share/lua/5.1/plenary/log.lua'
        no file '/usr/local/share/lua/5.1/plenary/log/init.lua'
        no file '/usr/share/lua/5.1/plenary/log.lua'
        no file '/usr/share/lua/5.1/plenary/log/init.lua'
        no file './plenary/log.so'
        no file '/usr/local/lib/lua/5.1/plenary/log.so'
        no file '/usr/lib/lua/5.1/plenary/log.so'
        no file '/usr/local/lib/lua/5.1/loadall.so'
        no file './plenary.so'
        no file '/usr/local/lib/lua/5.1/plenary.so'
        no file '/usr/lib/lua/5.1/plenary.so'
        no file '/usr/local/lib/lua/5.1/loadall.so'

[wbthomason]

requires is meant to specify dependencies, but not loading order. For loading order, there's the after key.

I see what you're saying in terms of this potentially being unexpected. To me, a dependency specification only implies that the required plugin should be installed, not necessarily loaded first. For instance, it's reasonable for one plugin to depend on another, but need to load itself first to configure state for the second plugin.

Do you think that the current setup, with the separation of dependency and loading order, is too cumbersome? If so, do you have suggestions for a better approach?

[rickysaurav]

I expect requires to be behave opposite to "after" i.e. Something like "before" and
slightly similar to dein's "depends" feature along with the added bonus of installing plugins as well.

Let's take the telescope loading snippet for example. When I specify that telescope requires popup and plenary, i expect it to ensure popup and plenary are installed and source them before sourcing telescope like dein's "depends" does.

We could either bake the feature into requires, or maybe keep requires the way it is and then add a new keyword like "before" or "depends" which is similar to dein's "depends".

[wbthomason]

Hmm. I think I lean more toward adding a before keyword (which can be implemented in terms of after). To me, it's better to separate the concerns of installing and loading a plugin, rather than having requires do both.

That does require users with your use case to specify two things, rather than the single depends or requires, though. I wish I had a better sense of how many people view requires the way I do vs. the way you do - that would help guide my thinking on whether it's better to change requires or add before...

[Iron-E]

I'm currently facing this issue too. If one plugin requires another, then that plugin has to be loaded first.

before seems to me like it would be like a required_by field: it would load that plugin after.

Example:

-- load completion-nvim before
use {'steelsojka/completion-buffers', requires='nvim-lua/completion-nvim'}

-- load completion-buffers after
use {'nvim-lua/completion-nvim', required_by='steelsojka/completion-buffers'}

[rickysaurav]

        use {
            "nvim-lua/completion-nvim",
        }
        use {
            "steelsojka/completion-buffers",
            after = {"completion-nvim"}
        }

I think "after" keyword should mitigate your issue. This'll ensure that even if completion-nvim is lazily loaded, completion-buffers will be loaded after completion-nvim.
In this scenario, the sourcing of "dependency"(completion) triggers loading of the "dependent"(completion-buffers). I would like a reverse scenario where loading the dependent ("telescope.nvim") would trigger loading of the dependencies ("plenary.nvim").

[Iron-E]

That helps, thanks!

I would like a reverse scenario where loading the dependent ("telescope.nvim") would trigger loading of the dependencies ("plenary.nvim").

I think we are describing the same thing: I expected require to actually use the specified plugin, and ensure that the required would be available before the specifier.

By that I mean, I expected the requirer to be used after implicitly.

[wbthomason]

@rickysaurav's suggestion is correct (and my original intent for how require should work), but it's becoming clear that this isn't intuitive for a good number of users.

I think I'll change requires to (a) add a dependency and (b) add sequence requirements to load all require'd plugins before the parent. I'll also add a depends keyword to just do what requires currently does. after will remain as it is; I'll add before to specify the opposite sequencing.

Does that sound reasonable and less confusing than the current state of things to those in this thread? (@Iron-E @rickysaurav)

[Iron-E]

I like that, it still allows flexibility but doesn't necessitate repetition of configuration.

[rickysaurav]

Sounds great. We should also add small table/section in the readme summarizng differences between these keywords in short for new users.

[Iron-E]

Something like this?

| Option | Example | Implementation |
|:---|:---|:---|
| after | use {'org/foo', after='bar'} | Load foo after bar. |
| before | use {'org/foo', before='bar'} | Load foo before bar. |
| depends | use {'org/foo', depends='org/bar'} | Ensure bar is installed before foo. |
| requires | use {'org/foo', requires='org/bar'} | Ensure bar is installed and loaded before foo. |

| Option     | Example                               | Implementation                                     |
|:-----------|:--------------------------------------|:---------------------------------------------------|
| `after`    | `use {'org/foo', after='bar'}`        | Load `foo` after `bar`.                            |
| `before`   | `use {'org/foo', before='bar'}`       | Load `foo` before `bar`.                           |
| `depends`  | `use {'org/foo', depends='org/bar'}`  | Ensure `bar` is installed before `foo`.            |
| `requires` | `use {'org/foo', requires='org/bar'}` | Ensure `bar` is installed and loaded before `foo`. |

[landerlo]

Is there any reason why requires shouldn't enforce the dependency to be loaded first? IMHO the proposed semantic distinction between depends/requires adds unnecessary complexity.
I'm probably missing something here, but if the extra flexibility of having requires or depends is just for the optimization of not having always to load, I'd say it's not worth it. Not worth the extra complexity in the internal code and the interactions between those two. Not the cognitive load for the user of having the two methods, with the gotchas you can have when you use one or another by mistake plus the initial surprise as this distincion is not intuitive. Whether I see depends or requires I'd expect we respect installing&loading order. IMHO it's a case of premature optimization. For downloading we don't need to follow the dependency DAG, but for installing and lazy loading I think it's worth to respect the dependency graph and not introduce complexity that maintainers and users will have to live with along the way. Other than that, thank you for the plugin, it has a great UX and it's very promising.

[landerlo]

Same for before and after. I know use-package has knobs like this, but we have different tools and languages, I'd expect loading in lua and nvim is much faster than in emacs. I think the requires gives all the ordering managing we need. For cases where you want more control, you can add extra requires dependencies to fine tune the DAG and introduce extra ordering constraints. I find that's a better alternative than having extra clauses like before and after that are ambiguous. I.e. {use: A, before: B}, it's not intuitive whether you want B to be loaded after A or you want to load B before A. My advice would be to keep it simple for now by sticking to a single dependency marker, whether that's named depends or requires, with solid semantics and optimal DAG so things get lazily loaded, and installation steps happens only once. And see how that works.

[gwerbin]

I'm -1 on "before" and "depends". Systemd abounds with these "reverse dependency" directives, and without extreme discipline even a very small system becomes a ball of mud very quickly when using them. I feel like for 99% of use cases you won't need them, and their existence would be more of a temptation to evil than a useful extension.

That said, I see the value in reducing the complexity of the dependency resolution system. By specifying "requires" you don't need to do any kind of topological sorting, you can just append the extra packages to the list. Also, separating "requires" and "after" allows you to parallelize package loading in the future, if such functionality ever becomes available.

Maybe there should be some kind of global option, requires_implies_after, which:

1. would be true by default,
2. would cause after = to be a no-op, and
3. could be set to false if/when the user wanted to specify after = specifically.

I think this would reduce the redundant requires/after usage for most people in most cases, while still giving freedom to users with very specific requirements, without removing a part of the API that would be valuable if parallel package loading became possible.

[landerlo]

Agree with the rest, but wouldn't we be keeping extra complexity with the requires and after combo? I cannot think of examples that would require after and cannot be modeled by additional requires constraints to model edge cases requiring a more specific ordering.

[gwerbin]

@landerlo keeping after means that, if one day packages can be loaded in parallel, you can speed up package loading by not enforcing sequencing when sequencing is not required.

[rickysaurav]

@landerlo I use "after" pretty frequently. Due to the fact that I lazy load 100% of my plugins to speed up my startup times.
For ex : completion.nvim, completion-buffers(completion source for completion.nvim).

I load completion.nvim on "insertEnter" autocmd, and then sequence load completion-buffers via "after". I also use it similar situations like nvim-treesitter and its extension modules. I am not aware of a straightforward way to achieve this without "after".

I am inclined towards implementing different keywords to give users freedom to mix and match to their linking , similar to "use-package" for emacs which serves as one of the inspiration for this plugin.

That said, I think just a single change of having "requires" to imply loading is fine as well, since that was the primary use case for me.

[Iron-E]

You wouldn't need after in that case if requires is implemented so the check only happens when the plugin specifying is about to load.

For what it's worth I lazy load every plugin I use, but I'm starting to think requires is the only one necessary. I agree with @landerlo on all

[landerlo]

@rickysaurav, thank you for explaining your use case. My comment was assuming requires lazy loads following the dependency order. However I see that for the kind of star dependency in your use case for the loading, having an after with a list makes it clearer. e.g. {use: completion-nvim, after: {compA, compB, ...}}
Still my preferred option would be to allow a list in the use entry with multiple packages so the other options can be shared.
e.g. { use: {completionA, completionB, completionC}, requires: completion-nvim }
Which reduces the keyword number, and allows benefits of grouping a number of uses onto one entry. Though this is obviously quite a fundamental change so I guess we need after, unless the idea of grouping, use accepting a list, takes hold and is implemented.

[landerlo]

@landerlo keeping after means that, if one day packages can be loaded in parallel, you can speed up package loading by not enforcing sequencing when sequencing is not required.

Yes, I see that advantage that after takes a list, so you don't need to create a dependency chain when it's really a set of packages. I'd turn it around to allow the grouping in the use, as in my comment above, but this is a more fundamental change, so unless that direction is taken, you've convinced me we need after.

[ruifm]

Still my preferred option would be to allow a list in the use entry with multiple packages so the other options can be shared.
e.g. { use: {completionA, completionB, completionC}, requires: completion-nvim }
Which reduces the keyword number, and allows benefits of grouping a number of uses onto one entry.

Big +1 for that, it would be really neat!

[Iron-E]

I created a demo of this in my config. Tested it briefly but thought I'd share:

local function use_all(plugins)
    local options = {}

    for key, value in pairs(plugins) do
        if type(key) ~= 'number' then
            options[key] = value
        end    
    end

    for _, plugin in ipairs(plugins[1]) do
        local plugin_options = type(plugin) == 'table'
        local _plugin = plugin_options and vim.tbl_extend('keep', plugin, options) or vim.deepcopy(options)
        _plugin[1] = plugin_options and plugin[1] or plugin

        use(_plugin)
    end
end

[landerlo]

This is really neat, thank you.

[wbthomason]

Sorry, I have been neglecting this conversation more than I should've.

I think the right choice will be to do the following (based on the above discussion):

1. Add a sequence_requires config option defaulting to true. This means that require'd plugins will be loaded before the plugins which require them (by adding appropriate after values).
2. Keep the manual after keyword to allow cases like @rickysaurav presented

I partially like the idea presented by @landerlo and @Iron-E to allow use to apply requires (and presumably other keys) to a nested list of plugin specifications, but I want to think more about how best to do this. I'm concerned about basically allowing fractal nesting of plugin specs.

Does this sound reasonable to those with opinions on this issue?