Habitat: When uploading dependencies, honor the target of the root package, not the platform

Created on 1 May 2018 · 3Comments · Source: habitat-sh/habitat

This issue has popped up in a few guises in the past, but it still exists.

It can be seen when you try to upload a Linux artifact from a MacOS workstation (really, from any workstation that doesn't match the platform of the artifact to be uploaded). This frequently happens when users enter a Linux Studio on their MacOS workstation to build a Linux package, then try to upload that package from _outside_ the Studio.

When the hab pkg upload command starts up, it creates a Builder API client, whose pre-configured User-Agent header includes architecture information _of the platform on which hab is running_ . This architecture information is used by the Builder API to determine which platform for which to resolve packages for. This is often exactly what you want. After all, when your Supervisors are asking for new versions of myorigin/myservice, they need to ensure they're getting it for the right platform. You're going to have a bad time if you're a Linux Supervisor trying to run a Windows package.

This starts to fall apart in this cross-platform upload scenario though, and in a slightly subtle way. Say you've built a Linux package locally; call it myorigin-myservice-1.0.0-20180501103724-x86_65-linux.hart. It will have depended on any number of Linux packages; let's say one of them is core/cacerts/2017.09.20/20171014212239. If we're uploading our myservice hart file from MacOS, we'll first see that we need to upload this core/cacerts/2017.09.20/20171014212239 package if Builder doesn't already have it. Intuitively, it must; otherwise how would we have gotten it in the first place? However, because the target platform plays into the Builder API's resolution logic, it will report that it does not have this artifact. This initially sounds broken, but it isn't; the package identifier _does not_ include target platform, and it is theoretically possible (though rather unlikely) for there to be, say, a Linux release and a Windows release of the same software with the exact same fully-qualified package identifier.

Thus, hab pkg upload will dutifully look for a core-cacerts-2017.09.20-20171014212239-x86_64-darwin.hart in the same directory as our myorigin-myservice-1.0.0-20180501103724-x86_65-linux.hart package (under the assumption that this dependency is something you're developing alongside your myservice software, and will thus be present in the same results directory). It won't find it, and the entire upload process will stop.

The workaround solution would be to always upload packages from the platform for which they are built. On MacOS, this can be done from within the Studio, or from inside a Linux VM. For completeness, collecting all the Linux dependencies of the package in the same directory and trying to upload _is not_ a workaround, because a) it involves a lot more manual work on the user's part, b) results in a lot of extra network traffic as dependencies are uploaded to Builder, only for Builder to recognize that it already has them, and (most importantly!) c) only works for Habitat core team members with write access to the core origin, since everything ultimately depends on core!

The true fix will require a bit of refactoring of the Builder API Client to allow manual overriding of the User-Agent header to use the target platform _of the package being uploaded_, rather than the platform from which it is being uploaded. Revisiting the decision to encode this information as a header as opposed to a request parameter may also be worthwhile.

Related History:

CLI E-easy L-rust Stale Bug V-devx

Source

christophermaier

👍2

Most helpful comment

Above makes sense - 👍 to retrieving target from package itself, and using a param instead of user agent to propagate target info.

Also, do we need to do dependent uploads as a default. It seems that's not the 80% case, so perhaps it should be optional.

chefsalim on 1 May 2018

👍2

All 3 comments

Above makes sense - 👍 to retrieving target from package itself, and using a param instead of user agent to propagate target info.

Also, do we need to do dependent uploads as a default. It seems that's not the 80% case, so perhaps it should be optional.

chefsalim on 1 May 2018

👍2

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. We value your input and contribution. Please leave a comment if this issue still affects you.