Nim: Lock down nimble version

[arnetheduck]

We're trying to set up a stable build environment:

• to know which version we're running
• to ensure that our build instructions work for all interested developers
• to allow reproducible builds (critical for publically available open source software)
• to minimize time loss from unstable day-to-day churn in Nim and its tools

we're starting with the nim compiler and working our way up the chain - next are nim tools, and specifically nimble - presently, the official build instructions suggest running koch tools to build nimble which leads to a random version being built, with no guarantee that nim and nimble versions work well together (inevitably, since both are evolving, there will be incompatibilities), or that nimble build still works for our libraries.

Thus it would be good if the "blessed" nimble version for any particular nim commit was tracked in the repo - the method doesn't really matter that much - a few options are:

• through a submodule
• through a config for koch tools
• by moving nimble into the nim repo
• through any other means that allows outsiders to know which version matches which nimble

Ostensibly, date-of-the-nim-commit could be used as heurestic, but this remains inaccurate: it means that when breaking changes happen on either side, there's a gap where it's not working.

a similar situation exists for csources - the HEAD of csources is used, which is prone to break.

[Araq]

The idea is/was that installing from source is for "bleeding edge" and so koch nimble defaults to the latest nimble commit too. The installers/tar.xz packages package the latest nimble official release though. koch can also build the latest release but this feature is not yet exposed (it's trivial to do though, so consider it done).

[Araq]

by moving nimble into the nim repo

I would love to do that but in the past @dom96 was against that. :-/

[arnetheduck]

yeah, but if I check out a less bleeding edge version of nim, I would need a nimble version that matches what was considered bleeding edge of nimble at the time - for bisecting, reproducing past builds exactly etc - ie for publishing software to a some distros like f-droid, repro-builds are a prerequisite.

the tarballs / releases unfortunately are not frequent enough to make this work in practice at the current pace of nim development - specially bugfix/point releases and lack of backporting of fixes make it infeasible.

thus, as a feature, it would be really useful if this was explicitly made clear in the repo, by the people that know best which versions work together (ie you and @dom96) - there are lots of hacky workarounds we can do, including tracking it ourselves, but ultimately, that's the wrong place to do it.

[dom96]

The idea is/was that installing from source is for "bleeding edge" and so koch nimble defaults to the latest nimble commit too.

Yeah, this idea is unfortunately error prone and we do need to change it.

through a config for koch tools

This is probably the best solution with the least amount of disruption and work required. Although we probably don't need a config, just hardcode the hash somewhere in koch.

a similar situation exists for csources - the HEAD of csources is used, which is prone to break.

We need to test for this in travis.

[arnetheduck]

a similar situation exists for csources - the HEAD of csources is used, which is prone to break.

We need to test for this in travis.

I don't understand how travis can help here - ie this is an example of how the issue happens, more precisely:

1. I check out Nim from 3 years ago
2. I do build_all.sh or follow the install instructions in the README
3. current csources unable to bootstrap old nim because both README and build_all.sh check out csources from today.

[dom96]

oh, I thought you meant Nim HEAD couldn't be compiled via current C sources, but I should know that we test for this already.

Yeah. That's not easy to solve. Especially since C sources are huge.

[arnetheduck]

well, there's a simple technical solution - just keep a record of the appropriate csources commit hash in Nim - usually done through a submodule - if allergic to submodules it becomes harder, though you can still hardcode a commit in build_all.sh.

csources actually has a similar issue in the reverse direction - it doesn't have a copy of lib that it works with, so when you get a nim binary from csources, there's no telling if it will work, in case lib has progressed..

[Araq]

Well we package csources+Nim+Nimble in the tar.xz. Releases need to become more frequent. I don't know why we need to support shenanigans like hardcoding commit hashes into koch or pretend csources are not git tagged (they are!) or concern us with questions like "can a newer Nim still build an older Nim". You can always build every released version of Nim without any Nim, nothing more is required.

The only problem here is the release frequency IMO.

[arnetheduck]

@Araq the issue is about adding in-repo tracking of what you're releasing in those tarballs so they can be repro:ed from git.. sure - tags kind of work, but they're manual / error prone / require project-specific knowledge of the tag structure - if it's in the repo instead (for example via submodule), you also relieve some pressure to release and in a way make the release process more simple - what you release is always what you have in git and no extra magic knowledge is needed.

just saw the stable flag - it unfortunately does not solve the problem, merely decreases the frequency with which the time proximity heurestic is wrong.

indeed, packaging let's you build "official" releases and go with these but unnecessarily slows down testing of experimental patches and makes it harder to use the many useful tools git brings to manage changes between releases, including those we eventually want to contribute back to Nim upstream

[timotheecour]

I think this is related to https://github.com/nim-lang/Nim/issues/8587 (adding a git tag / git hash reference to csources git repo in nim repo) /cc @krux02

[krux02]

@Araq I think it is easier to make every hash of the development branch reproduceable than to force more releases when they are not ready. At lest for my project OpenGL Sandbox it is more practical, because I rely on many recent fixes in the development branch. I can't really wait for a release in Nim before I use the patches that I made for Nim, nor can't I schedule my releases to the Nim releases. Sometimes I even depend on a custom fork of the compiler where I just hope that the branch gets merged. I would love if I can make even this custom fork a reliable build in the future.

[genotrance]

There's basically two supported modes today:

• Stable = official Nim release 0.xx.x built/installed from posted archive
• Devel = Git #head

Both these mechanisms are supported in the sense that the former is pre-packaged + scripted and the latter is scripted.

What is being asked here is the ability to be in the middle:

• Snapshot in the past
• Snapshot + custom patches

I believe this can be achieved in the following ways:

• Create a simple shell script that checks out exact nimble commit hash and builds it - nothing magical being done in koch beyond nim c
• Extend koch command line to accept a commit hash for nimble. If it is specified, it will check out that commit in koch tools/nimble rather than stable or #head
• If you want to be super smart, find the closest corresponding nimble commit hash that correlates to the Nim commit hash being used. This can be done in a shell script or koch.

I don't think csources needs any special handling either since it has to be done before koch is available anyway. If you want to use an older Nim snapshot, figure out the correct csources tag to checkout and build. Or just build a simple shell script that knows exact commit hashes of everything - csources, Nim and nimble - and call it a day.

For patches, do the above and patch before building.

git clone nim#commit
cd nim
git clone csources#commit
git clone nimble#commit dist/nimble

patch everything

cd csources && build && cd ..
bin/nim c koch
koch boot
bin/nim c -d:release --nilseqs:on dist/nimble/src/nimble.nim
cp dist/nimble/src/nimble.exe bin/.

All in all, there is no need to store hashes. If you want something custom, you already know exactly what you want.

[arnetheduck]

@genotrance
this particular feature request is specifically about recording the exact hash that was used to produce a certain release, or that is usable with a particular git version of the compiler, such that anyone can reproduce a particular nim ecosystem and get an exactly reproducible build - for example one that was used for a particular release, or one anywhere along the git history where nim and nimble are guaranteed to work together. for us builders, it's easier to work with git rather than release tarballs, for several reasons noted above.

[Araq]

Well koch now hardcodes used the Nimble version (via its tag) and since koch.nim is under git control, when you checkout e.g. v0.20, you can look at koch.nim to see what Nimble release was bundled with it.

[arnetheduck]

nice, this is a step in the right direction and will likely avoid most practical day-to-day issues, but more secure would be an actual hash.

a tag can move, meaning that supply-chain attacks are still possible. the hash allows the build system to verify the code it's downloading, unlike a tag. for our use case where financial assets are involved, this is significant.

[Araq]

Ok, will change it to a hash for 1.0 then.