Pkg.jl: `]up` and its behavior on `dev` packages

[ChrisRackauckas]

Currently, ]up doesn't do anything to dev packages. I have found this a little odd since I do happen to end up with a lot of dev packages, sometimes with some of them not being actively developed right now, but since I haven't gone in to manually update them these packages no longer track master.

There's a few things to mention here. One thing you can do is ]add MyPackage#master and it will track master, and that should be done if you don't develop the package at all. When you're working in a small project setup where other people are merging to master, you do want to stay updated with master as much as possible while sometimes developing it.

One thing that you could do in this case is dev, then develop/push/PR, free, add ...#master to go back. The issue here is that this is not fault-tolerant and is silent: if you accidentally didn't free then the only indication that your package may be old is that when you call ]up it shows up as a dev package. This makes it very easy to fall behind and be using older code and makes the collaboration a little bit jittery.

One solution is to use environments. This is really good for if you plan on changing a few things in a bunch of dependencies just to see what happens. This looks like:

• activate --shared MyEnv
• add PackageIWantToOverride
• Muck around
• activate to go back to the main environment
• Sometime later, "gc the environment" by deleting the folder (currently there is no command for that, but there possibly could be?)

Using environments is a good way to keep your main environment clean, and you get a pretty prominent indicator as to what environment you're in inside of the Pkg REPL. This mitigates the problem by decreasing the number of packages you dev, but doesn't get rid of it completely.

There are some legitimate concerns about handling of dirty packages, but there are a few things that could be done. @StefanKarpinski mentioned that it could fast-forward packages which have no changes (the "forgot to free" dev packages). Another thing that could be done is a colored warning can be given in during the ]up, a big indicator that you may want to check for updates in your dev package.

Both of these solutions would fetch upstream changes, which IMO is nice because either way this means that ]up is all you need to locally get all of the information to work on your local repositories. Pkg2 might've been a little strong handed with Pkg.update(), but with Pkg3 I've been finding that it's hard to remember to fetch updates to everything I want to work on before a flight. Anything that auto-fetches and gives some kind of indicators means that ]up is all you need and you can work it out from there.

[simonbyrne]

it could fast-forward packages which have no changes

Even if it isn't the default, it could also be an option, e.g. ]up -d/-dev.

[fredrikekre]

I very much like the guarantee that Pkg will not touch packages that are tracking a path, it is IMO a very useful property.

I would also like to point out that we have the --local option to dev. So another useful workflow is something like:

• Activate a new env
• dev --local all the stuff you need
• Push/PR/merge
• Remove the folder.

[StefanKarpinski]

Any reason not to give the option of updating dev packages at least?

[fredrikekre]

Does it solve anything? In the example in the OP, wouldn't you just end up with a up-to-date version of the bugfix branch that you worked on? Or should Pkg also switch branches and stuff?

[StefanKarpinski]

It solves the well-known "I want the latest version of all my dev'd packages" problem. I don't see why there's so much pushback against this feature. It worked well in Pkg2, people liked it, it never lead to data loss and we already have the code to implement it. I didn't omit it from Pkg3 for any principled reason, it just wasn't implemented initially. I have yet to hear a principled argument why we shouldn't have this feature.

[simonbyrne]

Since we keep a copy of the repo in .julia/clones/, it could make sense to have a way to refresh those.

[ChrisRackauckas]

Users could also do something locally if that's what they want for now. I tried making something in my startup but didn't know how to finish. Maybe someone with some LibGit2 knowledge could finish this:

import LibGit2

function update_devs()
  devpath = joinpath(first(DEPOT_PATH),"dev")
  repopaths = joinpath.(devpath,readdir(devpath))
  for pkg in joinpath(first(DEPOT_PATH),"dev")
    repo = LibGit2.GitRepo(pkg)
    if !LibGit2.isdirty(repo)
     # GitAnnotated(repo,LibGit2.fetch(repo)) ?
      ffmerge!(
    end
  end
end

[KristofferC]

Here's something else just using git

import Pkg
function update_devs()
    for (pkg, _) in Pkg.installed()
        pkgpath = abspath(joinpath(dirname(Base.find_package(pkg)), ".."))
        if ".git" in readdir(pkgpath)
            cd(pkgpath) do
                ok = success(`git pull`)
                if ok
                    @info "Pulled $pkgpath"
                else
                    @error "Failed to pull from $pkgpath"
                end
            end
        end
    end
end