Sdk: .packages: Eliminate symlinks, phase two

[mit-mit]

This bug tracks completing the work we started in 1.12 to eliminate symlinks (see previous work here: #23372). The goal of this second phase is to ensure that our whole SDK tool chain works even when no packages directory is present.

Concrete work items are:

Ensure the base SDK tools work great after a user uses pub's --no-package-symlinks:

• [x] Isolate.spawnUri takes a mapping of package data: #23951
• [x] Isolate.packageConfig returns map from package name to Uri #24022
• [x] Isolate.packageConfigUri returns a URI for the current package config #24524
• [x] Isolate.packageRoot should return a URI #19430

Then:

• [x] test package works without a packages/ dir (https://github.com/dart-lang/test/issues/327)

Then:

• [x] Test on packagebots with --no-package-symlinks config #24093

Then:

• [x] Unhide flag in pub - https://github.com/dart-lang/pub/issues/1340
• [x] Document in SDK changelog

Finally, ensure the higher-level tools work great if there is no packages/ directory. That work is tracked here: https://github.com/dart-lang/sdk/issues/24288

[mit-mit]

@sethladd , can you take a look at the task list to check that we got everything?

[sethladd]

@nex3 I'd like to open a few issues for pub on this topic, but I know pub is on the end of a chain of other prerequisite changes. The new issues will simply help us understand the scope of what we need to do for symlink elimination. Thanks!

[nex3]

I closed dart-lang/pub#1331 and removed it from the list above because it's based on a misunderstanding: pub serve doesn't rely on symlinks, and never has. I believe the only place that pub ever actually _consumed_ symlinks (as opposed to generating them) was pub run when it spawned a child process, and that was changed to .packages in dart-lang/pub@097ed875ea36b5406dce7c441f4feda1181f7664.

The actual title for dart-lang/pub#727 was always kind of misleading—it talks about not consuming on symlinks, but the discussion in the issue is actually about what it would take to not generate them. Maybe we should change the name to something like "Don't generate symlinks"?

[sethladd]

Maybe we should change the name to something like "Don't generate symlinks"?

Sure thing. Done and done!

So, is there anything you'll need before you can stop generating symlinks, which isn't covered in this issue above?

[nex3]

It looks like there isn't a list item up there for accessing resolved package URIs—something like #23990 or this CL will be necessary to make it possible for packages (including test) to locate resources reliably and efficiently.

[sethladd]

My impression was that https://github.com/dart-lang/sdk/issues/24022 would be sufficient.

[nex3]

24022 is very important for starting subprocesses with the right package information, but it's not a good fit for finding resources in an existing process. It only works if a package spec is being used, it requires extra IO and parsing, and it won't work with internal Google infrastructure.

[sethladd]

It only works if a package spec is being used

But you can use packageRoot otherwise, yes?

[nex3]

Not currently—the getter is only set if the user passed it explicitly on the command line, so if it was implicitly detected based on the entrypoint's location that's not reliable. I'm not sure whether #19430 will change that or not.

[sethladd]

so if it was implicitly detected based on the entrypoint's location that's not reliable

Ah ha, I understand now. Thanks for the additional context.

[sethladd]

@nex3 talking to @iposva-google it sounds like #19430 should be sufficient. Will follow up over there.

[nex3]

Okay, but we'll still need resolved package URIs.

[sethladd]

but we'll still need resolved package URIs.

If you had a combination of packageRoot and packageSpec/Map, could you not constract how a package: URI is resolved? In fact, could you not do this in a package that you built yourself?

[nex3]

As I said above, it requires extra IO and parsing and won't work with Google's internal infrastructure.

[sethladd]

won't work with Google's internal infrastructure.

So there's no packageRoot or packageMap/Spec in internal google infrastructure? Both would be null, in all cases for internal google infrastructure?

cc @cbracken who might be able to help provide perspective

[cbracken]

Under google infra, we construct a "flattened" package farm directory, including pubspec.yaml, pubspec.lock, and .packages files for each package in the transitive closure of deps. If the question is whether we can run entirely without the packages/ dirs, I'd imagine it shouldn't be significantly different from externally.

[nex3]

@cbracken How does dart path/to/file.dart work internally?

[cbracken]

VM 'binary' builds emit two things. For an app named foo, we emit:

1. a small shell script named foo
2. a tree of all required dep srcs/assets/etc. named foo.runfiles. This includes the vm and anything else necessary for the script to run.

First thing the foo script does on startup is create a tempdir in which it constructs a package-root layout from the srcs in foo.runfiles. It then invokes foo.runfiles/path/to/sdk/dart --package-root=/tmp/sometmpdir foo.runfiles/path/to/file.dart.

There are very minor differences if we're working with snapshots, but they're pretty much what you'd expect given the above.

[nex3]

I updated the main issue to reflect my understanding of the current plan.

[bp74]

We haven't heard about the "eliminate symlinks" undertaking for quite some time. Is this feature postponed or canceled? I ask because i just deleted a lot of my source code AGAIN by simply deleting a packages symlink in WebStorm.

[mit-mit]

Sorry for the seemingly lack of progress. We are still working on it, but we realized that we needed more investigation and testing before we felt confident releasing this. Hopefully we will be able to launch this fairly soon!

[kevmoo]

Closed by https://github.com/dart-lang/sdk/commit/38dccc88ffa9ab190154a1f40b36d5ab0d3be728

🎉

[sethladd]

Hurray!