Nimble package structure RFC

[dom96]

Continuing on from this discussion: https://irclogs.nim-lang.org/06-11-2017.html#14:11:52

I decided to create an RFC for this because many people seem to dislike the directory structure that Nimble currently imposes on them.

Before commenting, please familiarise yourself with Nimble package structure by reading this: https://github.com/nim-lang/nimble#libraries (the libraries section at a minimum, read the above and below sections as well for more)

Reasons that this enforcement by Nimble exists

Nimble performs a very simple operation when it installs a package:

• Copy all .nim files into ~/.nimble/pkgs/pkgName-0.1.0/

This path (~/.nimble/pkgs/pkgName-0.1.0/) is added to the Nim compiler's module search path. So let's say ~/.nimble/pkgs/pkgName-0.1.0/ contains the following files and directories:

• pkgName.nim
• pkgName/utils.nim

These can be imported via import pkgName and import pkgName/utils.

The problem arises when ~/.nimble/pkgs/pkgName-0.1.0/ contains a bare utils.nim. You can import that via import utils. This has a chance of conflicting with other packages or your own local modules!

This actually is only a problem for nim. But because when I first designed this I wanted nim to also support nimble packages it remained.

Criticism

These are some criticisms that people raised about this feature (I will add to this list as more come along)

• People dislike the nesting of directories that the directory structure imposes.
  
  
  
  • My answer: In reality this only adds one extra directory and thus I don't think it's a big deal.
  
  
• Hybrid packages are weird (You need to append pkg to your directory name).
  
  
  
  • My answer: I agree with this, but don't have any ideas how to solve this. I wanted to get people's feedback here but only one person answered.
  
  

My opinion

Apart from the "hybrid package" oddity I am happy with the way things work. If people have better ideas then I would love to hear them, but I am of course reluctant to change what so far has been working rather well.

TL;DR: Nimble makes sure that all packages follow a certain directory structure, people feel this is bad

BTW: If there is something that still doesn't make sense to you after reading the docs and my post, then please ask to clarify!

[Araq]

Copy all .nim files into ~/.nimble/pkgs/pkgName-0.1.0/

Yes, that's one of its core problems. It needs to clone/copy the whole repository instead at least. Otherwise the docs and example will be missing. I cannot use nimble to expore interesting packages with today's Nimble:

$ nimble install nimx

Ok, I'm left with $nimbledir/pgks/nimx-$version. Now what? I need to go to its website anyway, copy and paste some example file, put it into some dir and might even need to adapt its import section in order to get it to work. If I have to patch nimx, I need to fork and clone it in order create PRs. And even if $nimbledir/pgks/nimx-$version contains the examples and documentation it's in a hard to access directory (it's actually a "hidden" directory).

In fact, here is what I just did:

$ nimble install nimx

 Success: oldwinapi installed successfully.
 Installing rect_packer@any version
    Prompt: rect_packer not found in any local packages.json, check internet for updated packages? [y/N]
    Answer: y
Downloading official package list
       Tip: 91 messages have been suppressed, use --verbose to show them.
     Error: Refresh failed
        ... Could not download: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure

At this point, I pretty much have to ignore Nimble and all of its features and find a way to get rect_packer on my own!

[yglukhov]

Importing

Inspired by @Araq's dollar idea.

import $foo # Case 1
import $foo / path / bar # Case 2

For both cases determine the the root of package foo. Might be currently compiled package or one of nimble packages. Let's call it FOO_ROOT.

Determine the subpath. Case 1: foo.nim. Case 2: path/bar.nim. Let's call it SUBPATH.

Try lookup:

• FOO_ROOT/SUBPATH
• FOO_ROOT/foo/SUBPATH
• FOO_ROOT/src/SUBPATH ?? legacy?
• FOO_ROOT/src/foo/SUBPATH ?? legacy?
• fail

In such way, utils.nim in the root of package foo may be imported only as:

import $foo/utils

Special cases:

• $std - reserved name for std modules.
• $root (or better $pkg) - root of the package of currently processed module.

Current import semantics

Currently import statement tries to match a relative import first, then falls back to search paths. For consistency I would deprecate the fallback behavior and emit a warning whenever a fallback is matched and used.

Binary conflict with dir

Nimble install should just catch such cases and provide some temp name, or whatever, not the user's problem.
Nim on the other hand could leave it to the user to solve, or do its best and append some suffix automatically. I would favor the first approach :)

The best solution

First 2 paragraphs describe the concept of distinguishing between module-relative and package-relative imports. Some may not like the dollar syntax, so here's an alternative syntax proposal, which is also more natural, intuitive and consistent, imo.

1. Module-relative import should always start with either . or ...
2. Any other import is package-relative, where the first identifier is the package name.

import ./foo # Always relative
import ../bar # Always relative
import foo # Always absolute, "foo" is the package name, "foo.nim" is the module name
import foo/bar # Always absolute, "foo" is the package name, "bar.nim" is the module name
import std/strutils # "std" is the reserved package name
import pkg/bar # "pkg" is the reserved package name, expands to "package of this module"

[andreaferretti]

I am not sure I understand what the hybrid package oddity is. I consistently use a pattern where I have a main module myawesomemodule.nim as well as possibly some other modules under the myawesomemodule directory, so that people can do

import myawesomemodule
import myawesomemodule/somesubmodule

The main module might as well produce an executable if needed.

Test executables are in a directory test ignored by Nimble, which adds the module under development to the path, via something like path: ".."

I am not saying this is perfect, but I think is a pattern that covers many use cases. Putting a bare utils.nim into the main directory triggers a Nimble warning, and has done so for a while.

Imports inside the module itself can be relative (if there is little nesting) or absolute, and they both should work according to the current rules.

Can someone put an example where these rules create issues. I guess there is one in the discussion, but it is easier to record everything inside this RFC

[stisa]

@andreaferretti to my understanding, the problem with hybrid packages with the structure

/
  mypkg/
    other.nim
  mypkg.nim

is that if you compile mypkg.nim on systems that don't append an extension to executables, you would end up with something like

/
  mypkg/ # the folder
    other.nim
  mypkg.nim
  mypkg  # the executable

which isn't allowed by the os (I think both Linuxes and MacOS have this problem)

@yglukhov I like the part about requiring imports inside a package to be explicit, but I'm not really sure I get the second part. Sorry if these questions are very basic or already answered.
Lets say I have this nice package:

/
  tests/
    ...
  docs/
    ...
  src/
    nice.nim
    toplevel.nim
    nice/
      other.nim
      another.nim

here pkg would correspond to /src/ (an absolute path) and in another.nim, we can have:

import ../toplevel # uses toplevel.nim
import toplevel # uses a third party `toplevel` package
import other # uses a third party `other` package
import pkg/other  # this would work right?
import ./other # and would be the same as this
import pkg/nice/other # but this wouldn't

Also, why would importing care about the src part of the structure? Nimble with srcDir="src" doesn't copy src but only its contents, and if I'm compiling something inside src then it doesn't matter as far as the imports are concerned, right? Either they are relative or referred to /src/ (eg. import nice/other refers to /src/nice/other.nim)

[alehander92]

The problem arises when ~/.nimble/pkgs/pkgName-0.1.0/ contains a bare utils.nim. You can import that via import utils. This has a chance of conflicting with other packages or your own local modules!

Ok, just an idea.

What if you have the package v.

v.nim
w.nim

And what if nimble clones the repo, but forks it with a bit different structure into pkgs/v-0.2

v.nim
v/
  w.nim

basically leaves the main file there, but moves the others into a nested dir

This way

import v # imports v.nim as usual
import w # doesn't exist in path no clash
import v/w # works and it's clear it comes from v

[andreaferretti]

@alehander42 Nimble has triggered warning for such cases for a while, and my understanding was that they would eventually be deprecated. Moving files in the directory when cloning with Nimble seems to me a recipe to make imports break

[andreaferretti]

@stisa I never considered this, since I generally create executables with a Nimble command.

Wouldn't it be easier to just let nimble install foo use a temp location for output, and then move the resulting binary foo under .nimble/bin?

[alehander92]

@andreaferretti I know, but I was trying to think of a design which could support both kinds of layouts. You're right, the internal package imports can be broken (now, Nim is statically typed and I can imagine internal imports being reliably rewritten too, but I know that is too radical)

Otherwise I like having exactly one nested directory for the source too

[yglukhov]

@stisa, you're mostly right except pkg would expand to / in your case. Basically the root of your package is where the nimble file resides. If nim uses the matching patterns that I described in the first comment (there could be more or less, we have to come up with those), your last import will work as well because it matches FOO_ROOT/src/SUBPATH pattern. Now those patterns are marked ??legacy? because they are intended to work in most cases with existing nimble packages without the requirement for nim to parse nimble files.

[dom96]

@Araq

Copy all .nim files into ~/.nimble/pkgs/pkgName-0.1.0/

Yes, that's one of its core problems. It needs to clone/copy the whole repository instead at least. Otherwise the docs and example will be missing.

Alright, that's one issue. Here is my proposed solution to that:

All examples are copied to ~/.nimble/examples/pkgName-ver and can be viewed and run via some Nimble command. Examples are identified by the directory they are in (so an examples directory would be the obvious choise here). Cargo does it this way and while it wasn't immediately obvious to me I think it's still a good solution (we can copy the commands it uses for this purpose).

Same with documentation.

If I have to patch nimx, I need to fork and clone it in order create PRs.

This is where nimble develop comes in: nimble develop nimx (nimx will be cloned into the current work directory then linked to $nimbleDir/pkgs/nimx-ver)

Tip: 91 messages have been suppressed, use --verbose to show them.

I have already PM'd @Araq to try this, but others might be interested. For @Araq, running nimble refresh --verbose shows that Nimble is attempting to request packages.json from nim-lang.org. This is a custom configuration that @Araq (somehow) ended up with (likely due to the Nimble tester). The reason a HTTP request fails for nim-lang.org is because Mac OS X's OpenSSL doesn't support CloudFlare's SSL (no idea why). So the solution is to simply remove ~/.config/nimble/config.ini

---

@yglukhov

$std - reserved name for std modules.

Personally I prefer $stdlib.

In general, I dislike the fact that $ is both used for hardcoded names like $stdlib and $root and then also used to do this "magical" module searching.

I think to a lot of people, the $ will imply a "variable" or "identifier" of some sort. It's fine for $stdlib but not for arbitrary package names. It is also used as the operator for "to string" in Nim, so again if anything, I would prefer a different operator used here. Even ! would be better IMO.

My suggestion

What I would like to see is the ability to execute macros/compile-time procs in the import statement (pseudo-code):

macro `!`(ident: NimNode): string =
  # A macro that returns the path to the module
  if ident[0].str == "stdlib":
    # Find stdlib
    for paths in @["~/projects/nim/lib/pure", "~/projects/nim/lib/impure", ...]:
      if paths / fileExists(ident[1]):
        return paths / fileExists(ident[1])

import !stdlib.random

The only problem is I'm not sure whether the macro would ordinarily capture that full !stdlib.random expression. But in any case, it would allow some really powerful constructs and as far as I know, no other language supports anything like this so Nim would have some exclusivity here.

Binary conflict with dir

Nim on the other hand could leave it to the user to solve, or do its best and append some suffix automatically. I would favor the first approach :)

I vote for this too.

The best solution

A pretty good solution. Sadly it would break a lot :)

I assume you also want it to check for src directories too?

Overall though I still prefer having hardcoded "prefixes" like !root or !pkg (ideally implemented via macros)

---

@andreaferretti

What you're doing is perfect.

Imports inside the module itself can be relative (if there is little nesting) or absolute, and they both should work according to the current rules.

This is something that @yglukhov ran into. Using absolute imports will lead to problems. I describe the problem here, but I guess my explanation isn't clear enough. I will create a test case for it to demonstrate the problem in due time.

@stisa does a nice job explaining the hybrid package problem.

---

@stisa

Also, why would importing care about the src part of the structure?

In cases where the package isn't installed, but you want to import it anyway. To be honest, I don't think this is really a problem. And to be honest I find the idea of having Nim search 'src' as well hacky (it will break down as soon as someone decides to have a 'source' directory). Maybe @yglukhov can expand though.

---

@alehander42

Yeah, basically what @andreaferretti said. This would break imports. Your package would have a different structure when installed vs. when not installed. It's better to just get the developer to fix their structure (which is what Nimble does).

---

@andreaferretti

Wouldn't it be easier to just let nimble install foo use a temp location for output, and then move the resulting binary foo under .nimble/bin?

Yes, Nimble can handle this fine. This "hybrid package" rule only really exists to ensure Nim works.

[Araq]

All examples are copied to ~/.nimble/examples/pkgName-ver and can be viewed and run via some Nimble command.

No, please don't do that. It repeats Unix's packaging mistake, distributing the files differently than they are on the developer's machine. It's usually already enough work to ensure there are no hidden assumptions about environment variables or used global paths. If I have a tiny Nim program that builds the documentation, that needs to use nimscript.DocDir to cope with the fact that the docs are in a different place in a Nimble installation, etc etc. The problems will never end and once again the Nimble user has to learn more things and might wish to ignore the Nimble tool instead.

Instead give Nimble a switch like clone that clones the package as it is into my current directory, treating the current directory as my "workspace". Much simpler.

[dom96]

Instead give Nimble a switch like clone that clones the package as it is into my current directory, treating the current directory as my "workspace". Much simpler.

That's already what develop does... Can you please try it?

[Araq]

That's already what develop does... Can you please try it?

Was that a recent change? o.O I thought "develop" can only be used as nimble develop not as nimble develop packageHereThatIsNotMine.

[dom96]

No, it always worked this way! I feel like I've been repeating this at least 5 times to you now :(

[andreaferretti]

I now see the problem @yglukhov was having.

It seems to me that the only error is that the nim c command is not run from inside the package directory, and could be easily solved by the equivalent of cd foo. This can be seen by the message

Building foo/foo/foo1 using c backend

instead of

Building foo/foo1 using c backend

Tangential, possibly crazy idea:

What about deprecating bin in nimble files altogether and use custom nimble tasks? That way, one can easily set the path and call it a day.

When doing nimble install foo one just clones the foo package and runs nimble install from there, provided the install task is defined. Such task can produce binaries. By convention, if anything is produced under bin, it is copied in the .nimble/bin directory.

It is maybe a stupid suggestion, but - well, we have programmable nimble tasks, why having to discuss how to set up a folder structure so that paths will work fine, when we just --path: whatever inside the task body?

[yglukhov]

@dom96 wrote

A pretty good solution. Sadly it would break a lot :)

Actually I see a way to make this change in a gradual manner. The idea is dead simple:

1. Use new import resolving rules.
2. If no match use the old resolving rules. If match - deprecation warning.

Personally I prefer $stdlib.

std is shorter (that matters because any module will likely use stdlib) and more familiar from other languages. Also why dollar here? std is just a package that is provided along with nim installation. At least that is what the user should think.

I assume you also want it to check for src directories too?

Probably, if that's a common pattern. Also I would like nimble to not alter the package structure (my importing suggestion is totally aligned with that). Moreover, I would remove the srcDir option in the nimble file. Basically any option that implies some package structure. I would probably leave skip* options only to care about the user's HDD space in some really specific scenarios. (e.g. some test data which is 500MB large and is needed only for development purposes). Even then it's arguable if you want to keep this data in the repo in the first place.

What I would like to see is the ability to execute macros/compile-time procs in the import statement

That would be nice although I don't see a lot of use cases for that. Asking users to import your modules as import !myModule is the same as asking for myImport myModule. And that already works.

@andreaferretti post-install (post-download?) task that would be triggered to build the binary and put it to bin is a nice idea. Gives a lot more control. But there could be some function in the nimscript api to do it easily for common cases.

[dom96]

@andreaferretti

What about deprecating bin in nimble files altogether and use custom nimble tasks?

Not sure about deprecating bin, but giving more control to custom nimble tasks is definitely planned.

When doing nimble install foo one just clones the foo package and runs nimble install from there, provided the install task is defined. Such task can produce binaries. By convention, if anything is produced under bin, it is copied in the .nimble/bin directory.

Actually, the install task essentially runs nimble build. So it's a case of allowing the .nimble file to override what build does for your package. We can already specify before build and after build, the next step is something like on build but this requires quite a bit of work.

Despite your suggestion however, I still stand by what I suggested: @yglukhov should use import foo2 inside his foo1 module. If he doesn't then compiling his module can end up using an installed version of foo2, which isn't what the code intends to do. I tried to explain that in my post here but I guess I need to provide a clearer explanation. (To be honest, I don't even have a clear idea of the semantics now, because I wrote that post a while ago).

---

@yglukhov

Actually I see a way to make this change in a gradual manner.

Sure, but I count deprecation warnings as a "breakage". These new semantics will require A LOT of changes.

std is shorter (that matters because any module will likely use stdlib) and more familiar from other languages.

The idea is to only use $stdlib (or !stdlib or stdlib/ or whatever, the reason I am writing $stdlib is because that's how it is currently implemented) when there is an ambiguity. For that use case $stdlib is better than $std (as you will only be using it rarely).

And if you do want to use it for every stdlib import then you don't even have to repeat it:

import $stdlib/[strutils, httpclient, asyncnet]

Probably, if that's a common pattern. Also I would like nimble to not alter the package structure (my importing suggestion is totally aligned with that).

Can you elaborate? Nimble itself doesn't alter anything.

Moreover, I would remove the srcDir option in the nimble file. Basically any option that implies some package structure.

I think @Araq will disagree with you here, he dislikes a common "one size fits all" package structure.

That would be nice although I don't see a lot of use cases for that. Asking users to import your modules as import !myModule is the same as asking for myImport myModule. And that already works.

That's a good point. It wouldn't be as nice that way though :)

[Araq]

No, it always worked this way! I feel like I've been repeating this at least 5 times to you now

Ok, sorry, I'll use nimble develop from now on and report problems. That said, this needs to be documented much better, but that's a minor thing.

[Araq]

I'm happy to remove/change the dollar and its "search" feature but the problem remains that we don't have a good way to distinguish

import $stdlib / [a, b, c]

vs

import mypackage / [a, b, c]

And also that the imports are path based, not package based. Though I suppose it's arguable we really need a package based import.

[Araq]

Despite your suggestion however, I still stand by what I suggested: @yglukhov should use import foo2 inside his foo1 module. If he doesn't then compiling his module can end up using an installed version of foo2, which isn't what the code intends to do. I tried to explain that in my post here but I guess I need to provide a clearer explanation. (To be honest, I don't even have a clear idea of the semantics now, because I wrote that post a while ago).

I think I agree.

[dom96]

That said, this needs to be documented much better, but that's a minor thing.

Is this not enough? https://github.com/nim-lang/nimble#nimble-develop :)

I'm happy to remove/change the dollar and its "search" feature but the problem remains that we don't have a good way to distinguish

I'm happy with keeping $stdlib just for the purposes of this disambiguation.

[yglukhov]

@dom96

Despite your suggestion however, I still stand by what I suggested: @yglukhov should use import foo2 inside his foo1 module

@Araq

I think I agree.

Strongly disagree. The foo sample was just to demonstrate the problem. The real world scenario is a bit more complex where relative imports quickly become impractical.

- rod.nimble
- rod
  |- component.nim
  |- component
   |- sprite.nim
   |- text.nim
- tools
 |- rodasset
  |- rodasset.nim

So if my bin is tools/rodasset/rodasset.nim and it imports some components, the imports will look smth like

import ../../../rod/component
import ../../../rod/component/ [ sprite, text ]

Does it look sane? What if i want to move the tools/rodasset folder into a tools/asset_management/rodasset subfolder? I have to (a) modify all the imports by (b) prepending one more ../ to them. My point is that this is not a problem taken out of thin air. That's what I have now.

Sure, but I count deprecation warnings as a "breakage". These new semantics will require A LOT of changes.

You could be right, dunno. I have evaluated my codebase in regards to that and concluded that all of the needed changes may be done with a couple of find_all/replace. In other words I can shift 100KLOC to the new scheme in an hour or so. I can live with that. Besides the deprecation period can last reasonably long enough to make the transition as painless as possible. Basically that's nothing compared to other kinds of "deprecations" when the behavior changes without user's awareness (like it was done with Jsons some time ago ;))

And if you do want to use it for every stdlib import then you don't even have to repeat it:

Yeah, you still have to repeat it in every other module of yours.

Can you elaborate? Nimble itself doesn't alter anything.

Sure. I mean nimble installs (copies) only the srcDir pretending as if it was the root of the package. Of course it does so because the user has set srcDir option, which unluckily for him he did not fully understand the consequences of. That's why I vote for deprecating the srcDir altogether.

I think @Araq will disagree with you here, he dislikes a common "one size fits all" package structure.

As I see it, srcDir is exactly "one size fits all" attempt. A package is not necessarily only src. There are tests, docs, samples, assets, tools, configs, ci scripts, and who knows what else.

[mratsim]

My views:

Structure

Like Araq, I prefer if on install the package structure is like it was on the developer structure.

The src directory should not disappear when using srcDir. My very first user ticket was https://github.com/mratsim/Arraymancer/issues/16

except that it worked for me™.
And it was because I didn't now that nimble was removing the srcDir, so for srcDir I had to put my package.nim file inside the srcDir.

My ideal structure would be:

.
│
├── LICENSE
├── README.md
├── package.nim
├── package.nimble
├── benchmarks
│   └── ...
├── bin
│   └── package
├── docs
│   └── ...
├── examples
│   └── ...
├── lib
│   └── package.so
├── nim.cfg
├── nimcache
│   └── ...
├── src
│   ├── module1
│   │   └── ...
│   ├── module2
│   │   └── ...
│   ├── module3
│   │   └── ...
├── tests
│   └── ...
└── ... (git submodule, configs, html, etc)

Import

I'd like this to be importable like so

import package  # Import everything
import package/module1
import package/module2 # cherrypick the module

Current situation

To avoid nimble warnings and have the import working you have to use this today:

.
│
├── LICENSE
├── README.md
├── package.nimble
├── benchmarks
│   └── ...
├── bin
│   └── package
├── docs
│   └── ...
├── examples
│   └── ...
├── lib
│   └── package.so
├── nim.cfg
├── nimcache
│   └── ...
├── src
│   ├── package.nim
│   ├── module1
│   │   ├── module1.nim
│   │   └── module1
│   │       └── ...
│   ├── module2
│   │   ├── module2.nim
│   │   └── module2
│   │       └── ...
│   ├── module3
│   │   ├── module3.nim
│   │   └── module3
│   │       └── ...
├── tests
│   └── ...
└── ... (git submodule, configs, html, etc)

Proposition

Apparently my ideal structure would have some disambiguation issue (what file to load in the module1 subfolder?), we can require for multi-module package a module.nim in those module folders (like Python __init__.py or Rust mod.rs), and the dev is free to organise everything else as he see fits

Search / relative imports

I would like to avoid giving $ another meaning.
Use a keyword like relative or search, or ! is fine too but don't use an already existing symbol
Think about google search, "What does $ do in Nim" --> will that return results relevant to imports?

[Araq]

Does it look sane? What if i want to move the tools/rodasset folder into a tools/asset_management/rodasset subfolder? I have to (a) modify all the imports by (b) prepending one more ../ to them. My point is that this is not a problem taken out of thin air. That's what I have now.

I agree it becomes ugly but isn't that what we have --path for? Maybe the problem is that --path is not package specific?

[dom96]

@yglukhov

The real world scenario is a bit more complex where relative imports quickly become impractical.

Have you considered separating the tools into their own repos?

[dom96]

@yglukhov

Of course it does so because the user has set srcDir option, which unluckily for him he did not fully understand the consequences of. That's why I vote for deprecating the srcDir altogether.

Perhaps srcDir is the wrong name for this? Maybe we should call it pkgRoot or something like that?

---

@mratsim

To avoid nimble warnings and have the import working you have to use this today:

Not sure if you made a mistake, but it's still wrong. It should be like this:

.
│
├── LICENSE
├── README.md
├── package.nimble
├── benchmarks
│   └── ...
├── bin
│   └── package
├── docs
│   └── ...
├── examples
│   └── ...
├── lib
│   └── package.so
├── nim.cfg
├── nimcache
│   └── ...
├── src
│   ├── package.nim        # This is optional
│   ├── package
│   │   ├── module1.nim    # These could be directories too
│   │   ├── module2.nim
│   │   └── module3.nim
│
├── tests
│   └── ...
└── ... (git submodule, configs, html, etc)

I don't see the problem with this structure...

Proposition

Please explain in more detail. From what I can tell you are suggesting we should change the way the Nim module system works drastically.

Use a keyword like relative or search, or ! is fine too but don't use an already existing symbol
Think about google search, "What does $ do in Nim" --> will that return results relevant to imports?

👍

Agree 100%.

[Araq]

Use a keyword like relative or search, or ! is fine too but don't use an already existing symbol
Think about google search, "What does $ do in Nim" --> will that return results relevant to imports?

Ok, let's please discuss the feature(s) first before bashing the dollar syntax. There is a reason this is still completely undocumented. I don't mind changing the syntax.

[lcrees]

Following the precedent of the _system_ module, where "there cannot be a user-defined module named system" since "each module implicitly imports the System module", a possible option might be reserving an opt-in top-level name that guarantees that an import is a _stdlib_ module instead of a user-defined module e.g.:

import nim.strutils
from nim.strutils import replace

[yglukhov]

Have you considered separating the tools into their own repos?

TBH I have not, and that is a good suggestion to consider, thanks. But in context of current discussion that's a workaround.

Perhaps srcDir is the wrong name for this? Maybe we should call it pkgRoot or something like that?

I fail to understand the reason for this. Installing any subpath instead of full package (root defined by nimble file presence) is alteration, imo. It makes the potential tools (and nim itself maybe) to have to parse nimble files.

@Araq

And also that the imports are path based, not package based. Though I suppose it's arguable we really need a package based import.

Is there a comma/period missing between "arguable" and "we"? Because it changes a lot. I think package-aware import is more important than path-aware import. The libs are usually packed in packages. Normally you don't copy-paste some specific subtrees of sources from a third-party lib and use only those.

[mratsim]

@dom96
Won't this

├── src
│   ├── package.nim        # This is optional
│   ├── package
│   │   ├── module1.nim    # These could be directories too
│   │   ├── module2.nim
│   │   └── module3.nim

trigger the warning that module1.nim is in a directory package and either should be renamed or skipped?

My proposition is not drastic actually, it would be like this:

├── src
│   ├── package.nim
│   ├── package
│   │   ├── module1
│   │   │   ├── module.nim
│   │   │   └── ...
│   │   ├── module2
│   │   │   ├── module.nim
│   │   │   └── ...
│   │   ├── module3.nim
│   │   │   ├── module.nim
│   │   │   └── ...

When using import foo/module1, Nim would look automatically in a special module.nim file which people can customize to their own layout:

import foo
import subdir/bar

export foo
export bar

[dom96]

Won't this ... trigger the warning that module1.nim is in a directory package and either should be renamed or skipped?

No. Why would it? (unless package isn't the name of the package...) The idea is that all modules are namespaced by the package name. In that directory structure they all are, you can't import module1.nim without explicitly specifying import package/module1.

When using import foo/module1, Nim would look automatically in a special module.nim file which people can customize to their own layout:

You literally have the same semantics already, with a slight twist. Just add a src/package/module1.nim file with similar contents and it will work (no idea where foo is coming from now...)

[Araq]

How about this proposal:

• Nimble installs by cloning/copying the full file tree, ignoring srcDir.
• There is a clear distinction between package imports and --path based imports. Packages get a prefix like * (since you all dislike the dollar so much) and are searched following Nimble's rules (modules are in $pkg/src or $pkg/pkg etc). We could also make the compiler read .nimble files. For the "search package in parent dirs" we could have *.. or maybe do without. But I'm still of the opinion that no matter how good nimble is, nim needs to be easy to use without it. Package managers are not for everybody and I have yet to encounter a package manager that doesn't break and needs workarounds.

Some examples

import jester / helper ## path based import as it is today
import jester # first of all: interpreted as relative to the current file
import *jester ## package import

import *stdlib / [strutils]

Maybe we can do without this "path vs package" distinction but I don't really see how.

[andreaferretti]

I will quote myself

It seems to me that the only error is that the nim c command is not run from inside the package directory, and could be easily solved by the equivalent of cd foo

It still seems to me that the issue that triggered the whole discussion could be fixed by simply running nimble install using the cloned package dir as working directory. Am I missing something?

About the longer discussion: I see a lot of proposals that would be breaking in hard ways. What about the following:

• Nimble clones and keeps the whole directory, but only appends the srcDir of a package to the nim path

One would keep tests, examples and so on but nothing should break wrt today (apart from a one time clean of the .nimble directory).

It is important that only the srcDir can be imported, otherwise one may end up with making examples and tests public unadvertently.

[dom96]

@Araq

*? Where did that come from? Surely ! is better...

Anyway, I dislike this because it is yet another breaking change and one that you can see is shoehorned into the language.

Python doesn't need this distinction, so why do we? Perhaps we should check how Python solves this issue (if at all).

A bit of a sidetrack here...

By the way, now that I am thinking about the stdlib; it would make sense to turn it into a proper Nimble package, i.e.:

• lib/
  
  
  
  • stdlib.nimble
  
  
  • stdlib/
  
  
  • strutils.nim
  
  
  • httpclient.nim
  
  
  • openssl.nim
  
  
  • random.nim
  
  
  • ...
  
  

Yes, we would lose the impure vs. pure distinction, but is that really so bad? When we import these modules we aren't aware of this distinction anyway.

You could then pass --nimbleDir:$nimDir:/lib/ --path:$nimDir/lib/stdlib/ to the compiler and it would just work (tm):

import random # Random package (we would need to ensure that nimbleDirs are checked before paths for this to work)

import stdlib/random # Explicitly importing the `random` module in the `stdlib` (package)
import strutils # Importing the strutils stdlib module, this would still work because its been added via the --path switch (this is assuming that there are no ``strutils`` packages installed though)

The best part is that there is no need for this special cased $stdlib switch and it's fully consistent with the rest of the ecosystem. The only "special case" is that all the modules in the stdlib package are added to the Nim path implicitly.

@Araq What do you think?

---

@yglukhov

TBH I have not, and that is a good suggestion to consider, thanks. But in context of current discussion that's a workaround.

I am starting to lean towards this being a solution and not a workaround to be honest.

---

@andreaferretti

What about the following:

• Nimble clones and keeps the whole directory, but only appends the srcDir of a package to the nim path

Hrm, I'm not entirely sure what this would solve.

[yglukhov]

I am starting to lean towards this being a solution and not a workaround to be honest.

Thats equivalent of saying you want to not support hybrid packages at all. Otherwise there would be 2 paths to follow and one of them suddenly brings you to a dead end.

[Araq]

@Araq What do you think?

That's mostly ok with me, but doesn't solve @yglukhov's problem. Maybe instead of $root we can add a ..* smart "go up" match rule that stops its search at the nimble package boundary.

[yglukhov]

@dom96, @Araq, you're trying to invent the new way to solve my problem, but keep ignoring the fact that the current system is broken and needs to be fixed as well. If we invent a new way for absolute imports, the current syntax will have to forbid those. Because then someone else will end up in a situation where "absolute imports used to work, but now because of this sudden requirement to import lib from bin, i have to alter lots of imports in the lib code".

[andreaferretti]

I am losing track of this discussion. Is this @yglukhov's problem?

I may sound like a broken disc, but... why is the issue there in the first place? Is it because foo/foo1 cannot import foo/foo2. Wouldn't be enough to ensure that nimble build uses foo root as working directory?

I mean, if the import works while developing locally, it surely must work when cloning the project and cding to its root. The pwd is always added first to the Nim path, so... what am I missing? I am sure it's something silly, but I cannot see it

[euantorano]

I definitely agree that Nimble should clone the whole repository, not just a small subset of files. The whole cloned repository should end up in the nimble folder, untouched. This way it would be possible to have multiple versions checked out (since versions are just Git tags anyway) and you might not need to then use ~/.nimble/pkgs/pkgName-ver and instead just install to ~/.nimble/pkgs/pkgName (or preferably ~/.nimble/pkgs/<author>/<package_name>.

I actually like the current structuring of how modules are laid out (as below).

.
│
├── LICENSE
├── README.md
├── package.nimble
├── bin
│   └── package
├── docs
│   └── ...
├── examples
│   └── ...
├── nim.cfg
├── nimcache
│   └── ...
├── src
│   ├── package.nim        # This is optional
│   ├── package
│   │   ├── module1.nim    # These could be directories too
│   │   ├── module2.nim
│   │   └── module3.nim
│
├── tests
│   └── ...

I also like the idea of being able to fully qualify like import stdlib/strutils.

I also personally think it would be nice to have one way to specify submodules/paths (eg: remove the option for import stdlib.strutils in preference of import stdlib/strutils.

One enhancement I'd like to see is for packages to be able to have private modules that can't be imported/included by anything outside of that package. The convention for this case is for packages to have a src/<pkg>/private directory and to hide private modules in there. In Rust, modules are private by default, which makes sense.

[andreaferretti]

From the Nimble docs

In regards to modules which you do not wish to be exposed. You should place them in a PackageName/private directory. Your modules may then import these private modules with import PackageName/private/module. This directory structure may be enforced in the future.

I don't know whether this is only a convention, though

[euantorano]

I believe it is. As far as I understand, you can still import modules that ar ein that path from outside of the package.

[FedericoCeratto]

Can someone provide the reasons for claiming that whole upstream repositories should be cloned by Nimble? Most languages somewhat distinguish source packages as "everything you need to use this libarary" versus upstream repositories as "everything you need to develop the library itself" using whitelists.
There is an amount of content that is not beneficial, or is problematic, to clone/install e.g. large test files, "raw" image and diagram files, configuration files for editors and CI tools, rendered docs/websites.

[FedericoCeratto]

Related: local packages #4059

[Clyybber]

I agree with @euantorano , but think we should also allow packages like this:

│
├── LICENSE
├── README.md
├── package.nimble
├── bin
│ └── package
├── docs
│ └── ...
├── examples
│ └── ...
├── nim.cfg
├── nimcache
│ └── ...
├── src
│ ├── package.nim # This is optional
│ ├── filesforcompilingthebinary
│ │ └── ...
│ ├── libraryfiles
│ │ └── ...
│ └── util.nim
├── tests
│ └── ...

Currently this would give a warning about util.nim, filesforcompilingthebinary and libraryfiles

Also I think it would be a good idea to introduce a new option in the .nimble file called libDir, that defaults to srcDir for defining the directory for files that should be importable by other packages.

[Clyybber]

Also, to resolve the hybrid packages oddity, I think we should just make bin the default binDir, instead of putting the binaries next to their sources.

[Clyybber]

As for @yglukhov 's problem, I don't think there is a simple way to solve that problem without implementing more nimble specific features into nim.
One way I could imagine solving it nonetheless is having import foo check for foo first relative to the file importing it and then in the srcDir emitting a Warning if this creates ambiguity.
Or just make relative imports require a ./ in the path.
This would require the explicit differentiation of path vs package in import statements though.

But I actually think @dom96 's approach is the best, even if it doesn't solve @yglukhov 's problem(which should be mitigated with a good package structure or splitting into seperate packages).

Also, in cases of ambiguity like this:
├── src
│ ├── package.nim # import random
│ └── random.nim
where random clashes with an installed package called random Nimble should emit a warning and the user should be able to resolve it with import ./random or, if he intended to import the package random be able to suppress the warning in the .nimble file.

In hindsight I would rather differentiate between path vs package with a symbol like ! or $, instead of making the build process depend on the absence of certain packages.
Or make relative imports require a ./ and go with @dom96 's approach.

[Araq]

Discussion continues here https://github.com/nim-lang/Nim/issues/7250. Closing this one?