Pub: `package_config.json` validation fails with `pub run`

[blaugold]

Environment

• pub version or flutter pub version: Pub 2.12.0-29.10.beta
• OS version: Ubuntu 20.04.1 LTS
• Are you using the Chinese community mirror or a corporate firewall? No

Problem

This issue is related to this flutter issue.

flutter pub run build_runner build fails when:

• l10n generation has been setup for a flutter app
  
  
  
  • flutter.generate is true in pubspec.yaml
  
  
  • l10n.yaml exists
  
  
  • lib/l10n/app_en.arb exists
  
  
• a dependency on at least one local package through path exists

➜  issue_pub_package_config flutter pub run build_runner build
The ".dart_tool/package_config.json" file is not recognized by "pub" version, please run "pub get".
pub finished with exit code 65

To reproduce:

1. git clone https://github.com/blaugold/issue_pub_package_config.git
2. cd issue_pub_package_config
3. flutter pub get
4. flutter pub run build_runner build

The flutter code generation process breaks .dart_tool/package_config.json.

To reproduce run flutter pub get with generate: true:

    {
      "name": "flutter_web_plugins",
      "rootUri": "../../../../../../lib/flutter/packages/flutter_web_plugins",
      "packageUri": "lib",
      "languageVersion": "2.12"
    },
    {
      "name": "flutter_gen",
      "rootUri": "flutter_gen",
      "languageVersion": "2.8"
    }
  ]
}

And then again with generate: false:

    {
      "name": "yaml",
      "rootUri": "file:///Users/gabriel/.pub-cache/hosted/pub.dartlang.org/yaml-2.2.1",
      "packageUri": "lib/",
      "languageVersion": "2.4"
    }
  ],
  "generated": "2020-11-28T17:24:30.714165Z",
  "generator": "pub",
  "generatorVersion": "2.12.0-29.10.beta"
}

This error seems to be related to the packageUri property, which is expected to always exist and have the value lib/:

https://github.com/dart-lang/pub/blob/c00d4b4abf5b4ff265a7ce6282b748551f1b5b1f/lib/src/entrypoint.dart#L724

This conflicts with the spec for package_config.json:

• The spec for package_config.json states that packages.packageUri is optional, but pub expects it to always exist
• The spec also states that packages.packageUri must end with a /. The flutter cli uses the package_config package to write out package_config.json, so there might be a bug there too, since / is missing after that point.

Expected behavior

flutter pub run should be able to verify package_config.json.

Actual behavior

Command fails with previously mentioned error.

--trace output

The ".dart_tool/package_config.json" file is not recognized by "pub" version, please run "pub get".
---- Log transcript ----
FINE: Pub 2.12.0-29.10.beta
ERR : The ".dart_tool/package_config.json" file is not recognized by "pub" version, please run "pub get".
FINE: Exception type: DataException
FINE: package:pub/src/utils.dart 520:35                                                   dataError
    | package:pub/src/entrypoint.dart 693:7                                               Entrypoint._checkPackageConfigUpToDate.badPackageConfig
    | package:pub/src/entrypoint.dart 725:25                                              Entrypoint._checkPackageConfigUpToDate
    | package:pub/src/entrypoint.dart 502:7                                               Entrypoint.assertUpToDate
    | package:pub/src/entrypoint.dart 111:5                                               Entrypoint.packageGraph
    | package:pub/src/executable.dart 87:19                                               runExecutable
    | package:pub/src/command/run.dart 92:26                                              RunCommand.runProtected
    | dart:async                                                                          new Future.sync
    | package:pub/src/utils.dart 113:12                                                   captureErrors.wrappedCallback
    | package:stack_trace                                                                 Chain.capture
    | package:pub/src/utils.dart 126:11                                                   captureErrors
    | package:pub/src/command.dart 146:13                                                 PubCommand.run
    | package:args/command_runner.dart 197:27                                             CommandRunner.runCommand
    | package:pub/src/command_runner.dart 147:24                                          PubCommandRunner.runCommand
    | package:pub/src/command_runner.dart 136:18                                          PubCommandRunner.run
    | /b/s/w/ir/cache/builder/src/third_party/dart/third_party/pkg/pub/bin/pub.dart 9:48  main
    | ===== asynchronous gap ===========================
    | dart:async                                                                          Future.catchError
    | package:pub/src/utils.dart 113:52                                                   captureErrors.wrappedCallback
    | package:stack_trace                                                                 Chain.capture
    | package:pub/src/utils.dart 126:11                                                   captureErrors
    | package:pub/src/command.dart 146:13                                                 PubCommand.run
    | package:args/command_runner.dart 197:27                                             CommandRunner.runCommand
    | package:pub/src/command_runner.dart 147:24                                          PubCommandRunner.runCommand
    | package:pub/src/command_runner.dart 136:18                                          PubCommandRunner.run
    | /b/s/w/ir/cache/builder/src/third_party/dart/third_party/pkg/pub/bin/pub.dart 9:48  main
---- End log transcript ----

[sigurdm]

Hmm interesting. I don't think we really wrote the pub part of package_config handling with the use-case of user-modified package_configs in mind.

We try to play it safe, and say that if it is not exactly how we would generate it, we fail.

To understand the use-case better: What program is modifying the .dart_tool/package_config.json, and why does it do it?

It seems quite fragile to modify it - that would rely on pub get not getting called again.

[MarcinusX]

It's not modified by the user but by just adding local path to pub dependency like here

dependencies:
  local_package:
    path: packages/local_package

[blaugold]

It's the flutter cli, which is touching package_config.json. The reason is that the flutter cli can generate source files for localization of an app and places those into a synthetic package under .dart_tool/flutter_gen. To declare this package to the dart tooling, package_config.json is modified.

Just for context, all pub commands should be invoked through the flutter cli, when inside a flutter project, e.g. flutter pub get.

In general this arrangement works, it only becomes a problem when there is a path dependency in pubspec.yaml and flutter pub run is called. The local package seems to trigger additional checks of package_config.json, which result in the mentioned error.

[sigurdm]

The reason pub doesn't do the check when a path dependency is not present is that we assume the cache is immutable. (https://github.com/dart-lang/pub/blob/06bcff0981d05df871049c7fe11370892f3104fb/lib%2Fsrc%2Fentrypoint.dart#L476)

I think the setup is flawed, because pub assumes that it is generating the package_config and no one else touches it.

Would it work to include the synthetic package as an explicit path dependency in pubspec.yaml? (It would probably need a lib/ folder).

  synthetic_package:
    path: .dart_tool/flutter_gen

I guess the problem here is that we cannot do resolution before the package is generated.

We should find a good way of supporting this kind of use-case (depending on a generated package)
@jonasfj WDYT?

[jonasfj]

We should find a good way of supporting this kind of use-case (depending on a generated package)

Yeah, I agree.

[mit-mit]

Here's a repro of this that doesn't involve editing the config files manually:

1. mkdir repro; cd repro
2. flutter create myapp
3. dart create -t package-simple sample_dependency
4. Edit myapp/pubspec.yaml to contain:

dependencies:
  sample_dependency:
    path: ../sample_dependency
  intl:

and:

dev_dependencies:
  build_runner:

and:

flutter:
  generate: true

1. cd myapp
2. flutter pub get
3. flutter packages pub run build_runner build

=>

$ flutter packages pub run build_runner build
The ".dart_tool/package_config.json" file is not recognized by "pub" version, please run "pub get".
pub finished with exit code 65

[mit-mit]

cc @jonahwilliams, this looks similar to https://github.com/flutter/flutter/issues/46723. Is flutter pub get somehow modifying .dart_tool/package_config.json directly?

[sigurdm]

The reason this only triggers with a path-dependency is that we have this shortcut in pub:

if (packageConfigModified.isBefore(lockFileModified) ||
        hasPathDependencies) {
      // If `package_config.json` is newer than `pubspec.lock` or we have
      // path dependencies, then we check that `package_config.json` is a
      // correct configuration on the local machine. This aims to:
      //  * Mitigate issues when copying a folder from one machine to another.
      //  * Force `pub get` if a path dependency has changed language verison.
      _checkPackageConfigUpToDate();
      touch(packageConfigFile);
    } 

We expect everything to be fine if the timestamp order match - this might even be a bug - as it doesn't allow us to detect a case like this.

[idkq]

Re-running flutter create . inside myapp after all steps in @mit-mit example solves the problem. It does write 3 files mainly missing entries on package_config.json. You can compare the contents of the files by making a copy of the folder before running create and comparing to the new folder.

[jonasfj]

A short term fix here might be to allow flutter_gen as an extra package in pub.

Long-term we need to find a solution for code generation, but that might not be right now.

[aliyazdi75]

Any updates?
I'm stuck on this issue.

[sigurdm]

We are still considering to land something like https://github.com/dart-lang/pub/pull/2830 .

But that would be very much a hack...

[aliyazdi75]

@sigurdm
I still have this issue after flutter upgrade and flutter upgrade-packages.

[✓] Flutter (Channel master, 1.26.0-13.0.pre.120, on Linux, locale en_US.UTF-8)
    • Flutter version 1.26.0-13.0.pre.120 at /home/aliyazdi75/snap/flutter/common/flutter
    • Framework revision 46b0797b58 (2 hours ago), 2021-01-22 14:48:49 +0500
    • Engine revision 90641fa923
    • Dart version 2.12.0 (build 2.12.0-248.0.dev)

[mit-mit]

Yes, these changes havn't been rolled all the way to flutter yet.

[aliyazdi75]

@mit-mit
Oh, thanks.
Is there any way to track this rollout after this issue has been closed?

[jonasfj]

It'll land in next stable flutter release.

Otherwise, I think it'll trickle to Flutter master sometime next week.

This is only a temporary workaround, it's not clear to me if using the name flutter_gen is a good long term solution.
But that's something to be resolved in https://github.com/flutter/flutter/issues/73870