Mbed-cli: relative library URLs

[bmcdonnell-ionx]

Is it possible to have a library with a relative URL? So that in cases where the project and libraries' origins are on the same server, then they can move together to a new server / new base URL, without breaking references or history.

If possible, is this documented? If not, can you add this feature?

[loverdeg]

My team has moved on from our attempts to use git submodules, mbed .libs, and various other dependency/package management solutions.

Ultimately we've settled on Peru for the moment.
https://github.com/buildinspace/peru

Review some of my other issues for info somewhat related to this topic.

[ciarmcom]

Internal Jira reference: https://jira.arm.com/browse/MBOCUSTRIA-793

[bridadan]

Hi @loverdeg-ep, that's a cool project, thanks for posting the link, was definitely a good read!

@bmcdonnell-ionx This isn't possible for remote repositories. Can I ask how would you expect to use this feature? How would I set the common server URL? Thanks for your feedback!

[bmcdonnell-ionx]

@bridadan,

Can I ask how would you expect to use this feature?

We set up a new git server, with a new URL. My Mbed-OS-based project links to multiple libraries that reside on the old git server. If I move all the repos to the new server, I break historic builds. If the links were relative, they'd all move seamlessly.

Likewise, this would allow users to mirror their work to multiple remotes, without requiring additional work-around steps to build (on all but one).

How would I set the common server URL?

I figured the way to do this would be relative to the project location. If I have
https://my.git.server/folder1/folder2/my-mbed-os-example-app.git,
which includes a library with a relative link to mylib.git, it would look for it at https://my.git.server/folder1/folder2/mylib.git

Or if the relative link was to ../elsewhere/mylib.git, then it looks in
https://my.git.server/folder1/elsewhere/mylib.git

etc.

I haven't though of a use for making the root otherwise configurable yet (e.g. in mbed_app.json).

[bridadan]

I see, you're proposing to set the "root" for any relative path in .libs to reference some setting in the project (perhaps in mbed_app.json). This would be tricky because libs can reference local repositories on your system as well by using local paths. So far I don't have any other ideas on how to implement this cleanly, do let me know if you have any other ideas though!

[loverdeg]

Doesn't solve OP's problem but @bridadan https://github.com/jacebrowning/gitman is a near equivalent alternative to Peru. We flipped a coin and chose Peru.

[bmcdonnell-ionx]

@bridadan,

you're proposing to set the "root" for any relative path in .libs to reference some setting in the project (perhaps in mbed_app.json).

No, unless someone makes a case for that. I think the root should be based on the project's git origin path.

This would be tricky because libs can reference local repositories on your system as well by using local paths.

What is the syntax for that?

[bridadan]

No, unless someone makes a case for that. I think the root should be based on the project's git origin path

Ah I see, I hadn't considered that option, thanks for clarifying.

What is the syntax for that?

Currently, when you use mbed add ../mylib mylib, it will put an absolute path to the mylib folder in the in the mylib.lib. If you change the path in the lib to a relative path, it will still work (though to be clear, I don't believe this is the intended use and it "just happens to work"). I'm concerned if we change the behavior of these relative paths it may be confusing.

[bmcdonnell-ionx]

I guess there are arguments to be made both for and against maintaining the current unintended behavior. If you wanted to add this feature, and maintain the behavior, how about adding a command line switch for this? e.g. mbed add --relative-remote ../mylib.

You could optionally add another switch to codify the current unintended(?) behavior, say mbed add --local ../mylib. And optionally disallow relative paths without any command switch, to force users to explicitly specify (and hopefully understand) what they're doing.

[bridadan]

I understand your use case, thanks for taking the time to explain it. The other case I'm wondering about is when you mbed import the repository. So how do we save in the .lib files that the relative path present in there should be relative to the remote? The only idea that springs to mind is putting something in the .mbed file (that file isn't typically committed to repositories either). Which I'm also a little worried will cause more usability problems if you forget that you set that option.

In general these are the edge cases I'm concerned about. But like I said before, I'm very open to any suggestions!

[theotherjimmy]

for what it's worth, we could switch to URI syntax and use file:// as the prefix for all local libs. Then any library without a <transport>:// prefix could use the transport and host of the lib it's being cloned from.

[bridadan]

for what it's worth, we could switch to URI syntax and use file:// as the prefix for all local libs.

Really like that idea going forward, that will help remove some ambiguity.

Then any library without a :// prefix could use the transport and host of the lib it's being cloned from.

If it doesn't have a <transport>:// prefix, how would we know which one to use? Do you mean use the transport and host of the application?

[theotherjimmy]

@bridadan Every blah.lib has a parent directory that was cloned somehow. I'm proposing that we use the transport and host of the parent.

[bridadan]

Gotcha. It'd be worth exploring! Gotta be careful we don't break existing applications out there (or the online compiler's logic).

[loverdeg]

@bridadan @theotherjimmy

for what it's worth, we could switch to URI syntax and use file:// as the prefix for all local libs. Then any library without a <transport>:// prefix could use the transport and host of the lib it's being cloned from.

That sounds nice and intuitive.

If mbed-cli can become more like Peru or Gitman I'd consider coming back to mbed-cli for dependency & package-ish managment.

[bmcdonnell-ionx]

I like the proposed model here, but will backward compatibility be an issue? It would change the behavior of the path in the .lib file when there is no <transport>:// specified, right? If so, will this require user action for any with such library files after they cutover to the new behavior? If so, is this acceptable? Will this be manageable? (I think maybe "yes" (manageable) if the behavior is part of Mbed OS, and "no" if it's part of Mbed CLI.)

If it does break for those using the old "accidental" behavior, is this new scheme maybe so much better that the breakage is acceptable?

[theotherjimmy]

@bmcdonnell-ionx You're correct in pointing out that this would be a breaking change. We could include a key in the user's .mbed file that indicates that they would like the new URL behavior. That way we could upgrade to the new behavior incrementally without breaking things.

[bmcdonnell-ionx]

I don't think .mbed is a good place for it, since that doesn't usually get checked-in to the repo. (Right?) The setting should travel with the references.

[theotherjimmy]

@bmcdonnell-ionx Touche. Perhaps we need to add a header to the .lib files for this change.

[bridadan]

That's the only other place I can think of, but we have to be careful with that one (for compatibility with older versions of mbed-cli and the online compiler).

[bmcdonnell-ionx]

Hopefully this wouldn't be necessary, but if you get desparate, you could create a new extension (not .lib), and do whatever you want with the format inside.

[screamerbg]

I'm concerned about introducing behaviors that are assuming things from the parent URL, where the whole "new behavior" also changes the existing/current behaviors.

Few examples where this could go wrong:

• Local repositories today use transport-less syntax, e.g. "pathdir1/path". Changing to "transport://" schema, like "file://", will need to be addressed in the Mbed Online Compiler and Mbed Studio. Also the 3 tools will have to support the old behavior AND the new behavior.
• The proposed domain/host inheritance from the parent won't work either as some repository may move from one host to another (e.g. github/ARMmbed/ may move to bitbucket), but that doesn't mean that Mbed OS Partner drivers are moving to bitbucket either - see https://github.com/d-kato/esp32-driver/,
  https://github.com/OpenNuvoton/NuMaker-mbed-SD-driver/

Perhaps a config option that allows you to rewrite the URLs would be much more handy and also backwards compatible, where the URL rewriting will take care of the src<->dst URL translation.

[bmcdonnell-ionx]

@screamerbg,

[1] Mbed Online Compiler and [2] Mbed Studio. Also the 3 tools will have to support the old behavior AND the new behavior.

What's the 3rd tool?

The proposed domain/host inheritance from the parent won't work either as some repository may move from one host to another (e.g. github/ARMmbed/ may move to bitbucket), but that doesn't mean that Mbed OS Partner drivers are moving to bitbucket either

It would be incumbent on the user to only use the relative option when the repos would indeed be expected to move or be copied/mirrored together, if/when that happens. I don't think your counter-examples fit that use case.

Perhaps a config option that allows you to rewrite the URLs would be much more handy and also backwards compatible, where the URL rewriting will take care of the src<->dst URL translation.

I don't follow.

[ciarmcom]

Thank you for raising this issue. Please note we have updated our policies and
now only defects should be raised directly in GitHub. Going forward questions and
enhancements will be considered in our forums, https://forums.mbed.com/ . If this
issue is still relevant please re-raise it there.
This GitHub issue will now be closed.