Sdk: Pub is not able to handle out-of-date snapshots

[munificent]

Pub global activate was broken when the VM flipped to Dart 2 by default. I'm working on fixing that. In the process, I stumbled into an important behavior that I believe used to work in the Dart 1 VM but does not any more. Let's say the user does this:

1. They run pub get. That downloads some packages and precompiles their binaries to snapshots so that they can be run faster.

2. Later, they upgrade to a new version of the Dart SDK. That version has a VM with a newer snapshot version.

3. They pub run one of the executables. This tries to run the old snapshot generated in step 1 on the newer VM.

In Dart 1, at step 3, pub would catch an IsolateSpawnException and look for message "Wrong script snapshot version". If found, it would discard the old snapshot and generate a new one for the current VM, and then proceed merrily along its way.

In Dart 2, it looks like the VM hard aborts the entire process if you load an old snapshot. There's no way that pub can catch the error because it's not a thrown exception. The process just dies. At least I think that's what's going on.

Am I understanding the current behavior right? If so, how can pub detect an outdated snapshot and handle this case?

If we don't fix this soon, then the next time the VM revs the snapshot version, every user who has used pub run (which is basically everyone, because of pub run test) or pub global activate since the switch to Dart 2 by default is going to blow up the next time they try to run one of those commands.

[nex3]

See also #20784, #20785, and particularly #20802.

[munificent]

@a-siva ping.

[DanTup]

Is this also the cause of this issue (Snapshot not compatible with the current VM configuration)? Seems that in dev64.1 the pub snapshot file in the SDK doesn't run on the VM by default (the pub2 snapshot does). I think it's common for tooling to run snapshots from bin/snapshots (for ex. I run the analyzer this way in Dart Code) this can break things (in this case it looks like WebStorms test runner - and I think IntelliJ also invokes pub with its snapshot).

[natebosch]

Running from the bin/snapshots directory is related but not quite the same issue. That failure started happening because the Dart VM was switched to Dart 2 by default but the bin/snapshot/pub.dart.snapshot was still Dart 1. The next release of the SDK will change bin/snapshot/pub.dart.snapshot to a Dart 2 snapshot.

[DanTup]

Makes sense, I'll add a note to the answer on SO then. Thanks!

[jensjoha]

Does something like this fix it? https://dart-review.googlesource.com/c/sdk/+/61937

[TjeuKayim]

Thanks @DanTup for mentioning that error with WebStorm. Now I know that it's caused by this issue. Is there any workaround to run tests in WebStorms for Dart 2?

[DanTup]

@TjeuKayim Best thing is probably to roll back to previous dev release until the next one is available.

[TjeuKayim]

Yes, I downgraded using choco upgrade dart-sdk --version 2.0.0.63-dev-0 --pre --allow-downgrade, started WebStorm again, and the test are running.

[natebosch]

I think https://dart-review.googlesource.com/c/sdk/+/61937 will fix this - do we expect that to go in soon?

[a-siva]

There are multiple aspects to this:
scenario 1:

• 'pub get' was done with a Dart 1 sdk and so it created Dart 1 script snapshots of the packages
• now they upgrade to a Dart 2 sdk, the Dart 1 script snapshots are invalid, they need to be deleted and kernel files for the packages need to be generated
  The fix in https://dart-review.googlesource.com/c/sdk/+/61937 does not account for this scenario, we probably need to add additional code to detect that a Dart 1 script snapshot is being passed and error out in the embedder (Currently it tries to invoke the front end on this file and that produces some compilation errors)

scenario 2:

• 'pub get' was done with a Dart 2 sdk and we created kernel files for the packages.
• now an update to the Dart 2 sdk was done and this update does not revise the kernel file format version number
  Things should continue to work without any need to regenerate new kernel files.

scenario 3:

• 'pub get' was done with a Dart 2 sdk and we created kernel files for the packages.
• now an update to the Dart2 sdk was done and this update has revised the kernel file format version number
  Now we need to regenerate the kernel files for the packages. The CL https://dart-review.googlesource.com/c/sdk/+/61937 will fix this scenario. For this scenario we would like to change the error string to something else from "Wrong script snapshot version" to something like "Incompatible kernel file version"

[vsmenon]

@munificent - with the recent CL, is there any more work for scenario 3 (to catch the new error?)

@a-siva - is there a separate issue for scenario 1 or do you just want to repurpose this?

[munificent]

I don't believe scenario 1 is a problem. Pub has always used different file extensions for the Dart 1 and Dart 2 snapshots it creates (.snapshot and .snapshot.dart2, respectively). If you ran pub in Dart 2 mode (which it detected by checking for reified generic method type arguments), it created and looked for a .snapshot.dart2 file. Otherwise, it used a .snapshot file.

You're right on scenario 2.

Scenario 3 is my concern. Change the error message to whatever you like and let me know and I'll update pub to look for it. A more robust option would be to throw a distinct exception class for this case so we can use a type test instead of looking for a magic string, but I'll take whatever the VM the gives me. :)

[munificent]

...ah, I see that https://dart-review.googlesource.com/c/sdk/+/61937 landed, so I'll look at the message it's using and try to get pub fixed. Thanks!

[munificent]

Fix out for review: https://github.com/dart-lang/pub/pull/1938

[munificent]

Leaving open to track rolling the fix into the SDK repo.

[munificent]

In the SDK with: https://dart-review.googlesource.com/c/sdk/+/62460