Pub: Exception from _PackageConfigJsonParser._parsePackage when trying to publish a package

[devoncarew]

We're seeing this exception while trying to publish devtools:

The resolved 'packageUri' must be inside the rootUri, actualy 'file:///' is not in 'file:///usr/local/.../devtools-git/devtools/packages/devtools/build/packages/_fe_analyzer_shared/', for the package '_fe_analyzer_shared'.
{"configVersion":2,"packages":[{"name":"_fe_analyzer_shared","rootUri":"../...
package:analyzer/src/context/package_config_json.dart 199:9     _PackageConfigJsonParser._parsePackage
package:analyzer/src/context/package_config_json.dart 243:9     _PackageConfigJsonParser._parsePackages
package:analyzer/src/context/package_config_json.dart 116:7     _PackageConfigJsonParser.parse
package:analyzer/src/context/package_config_json.dart 13:49     parsePackageConfigJson
package:analyzer/src/context/packages.dart 73:20                parsePackageConfigJsonFile
package:analyzer/src/context/packages.dart 128:14               parsePackagesFile
package:analyzer/src/context/builder.dart 191:14                ContextBuilder.createPackageMap
package:analyzer/src/context/builder.dart 498:35                ContextBuilder.createWorkspace
package:analyzer/src/context/builder.dart 301:24                ContextBuilder.getAnalysisOptions
package:analyzer/src/context/builder.dart 112:9                 ContextBuilder.buildDriver
package:analyzer/src/dart/analysis/context_builder.dart 104:37  ContextBuilderImpl.createContext
package:pub/src/dart.dart 111:36                                AnalysisContextManager.createContextsForDirectory
package:pub/src/validator/strict_dependencies.dart 27:28        new StrictDependenciesValidator
package:pub/src/validator.dart 128:7                            Validator.runAll
package:pub/src/command/lish.dart 161:32                        LishCommand._validate
package:pub/src/command/lish.dart 141:15                        LishCommand.run
package:args/command_runner.dart 197:27                         CommandRunner.runCommand
package:pub/src/command_runner.dart 191:39                      PubCommandRunner.runCommand.<fn>
dart:async                                                      new Future.sync
package:pub/src/utils.dart 113:12                               captureErrors.wrappedCallback
dart:async                                                      runZoned
package:pub/src/utils.dart 130:5                                captureErrors
package:pub/src/command_runner.dart 191:13                      PubCommandRunner.runCommand

This is an unexpected error. ...

cc @terrylucas

[terrylucas]

The failure happened with Pub 2.9.0-5.0.dev (trying to publish devtools) switched to the latest stable release of Flutter Pub 2.7.2 and then we were able to publish our packages. The same failure with pub --trace publish is:

O  : Spawning "/bin/tar --format=ustar --create --gzip --directory /usr/local/google/home/terry/devtools-git/devtools/packages/devtools/. --files-from /dev/stdin --owner=pub --group=pub" in /usr/local/google/home/terry/devtools-git/devtools/packages/devtools/.
ERR : The resolved 'packageUri' must be inside the rootUri, actualy 'file:///' is not in 'file:///usr/local/google/home/terry/devtools-git/devtools/packages/devtools/build/packages/_fe_analyzer_shared/', for the package '_fe_analyzer_shared'.
    | {"configVersion":2,"packages":[{"name":"_fe_analyzer_shared","rootUri":"../...
FINE: Exception type: FormatException
ERR : package:analyzer/src/context/package_config_json.dart 199:9     _PackageConfigJsonParser._parsePackage
    | package:analyzer/src/context/package_config_json.dart 243:9     _PackageConfigJsonParser._parsePackages
    | package:analyzer/src/context/package_config_json.dart 116:7     _PackageConfigJsonParser.parse
    | package:analyzer/src/context/package_config_json.dart 13:49     parsePackageConfigJson
    | package:analyzer/src/context/packages.dart 73:20                parsePackageConfigJsonFile
    | package:analyzer/src/context/packages.dart 128:14               parsePackagesFile
    | package:analyzer/src/context/builder.dart 191:14                ContextBuilder.createPackageMap
    | package:analyzer/src/context/builder.dart 498:35                ContextBuilder.createWorkspace
    | package:analyzer/src/context/builder.dart 301:24                ContextBuilder.getAnalysisOptions
    | package:analyzer/src/context/builder.dart 112:9                 ContextBuilder.buildDriver
    | package:analyzer/src/dart/analysis/context_builder.dart 104:37  ContextBuilderImpl.createContext
    | package:pub/src/dart.dart 111:36                                AnalysisContextManager.createContextsForDirectory
    | package:pub/src/validator/strict_dependencies.dart 27:28        new StrictDependenciesValidator
    | package:pub/src/validator.dart 128:7                            Validator.runAll
    | package:pub/src/command/lish.dart 161:32                        LishCommand._validate
    | package:pub/src/command/lish.dart 141:15                        LishCommand.run
    | package:args/command_runner.dart 197:27                         CommandRunner.runCommand
    | package:pub/src/command_runner.dart 191:39                      PubCommandRunner.runCommand.<fn>
    | dart:async                                                      new Future.sync
    | package:pub/src/utils.dart 113:12                               captureErrors.wrappedCallback
    | package:stack_trace                                             Chain.capture
    | package:pub/src/utils.dart 126:11                               captureErrors
    | package:pub/src/command_runner.dart 191:13                      PubCommandRunner.runCommand
    | ===== asynchronous gap ===========================
    | dart:async                                                      Future.catchError
    | package:pub/src/utils.dart 113:52                               captureErrors.wrappedCallback
    | package:stack_trace                                             Chain.capture
    | package:pub/src/utils.dart 126:11                               captureErrors
    | package:pub/src/command_runner.dart 191:13                      PubCommandRunner.runCommand
---- End log transcript ----

Also, had another pub publish error that again disappeared using pub 2.7.2 (trying to publish devtools_app):

Package validation found the following errors:
* Your pubspec.yaml must not override non-dev dependencies.
  This ensures you test your package against the same versions of its dependencies
  that users will have when they use it.
* line 8, column 1 of lib/generated_plugin_registrant.dart: This package does not have url_launcher_web in the `dependencies` section of `pubspec.yaml`.
    ╷
  8 │ import 'package:url_launcher_web/url_launcher_web.dart';
    │ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ╵
* line 10, column 1 of lib/generated_plugin_registrant.dart: This package does not have flutter_web_plugins in the `dependencies` section of `pubspec.yaml`.
     ╷
  10 │ import 'package:flutter_web_plugins/flutter_web_plugins.dart';
     │ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     ╵

[sigurdm]

Do you have the pubspecs you were trying to publish?

[sigurdm]

Ah nevermind - it's devtools

[sigurdm]

I'm not immediately able to reproduce (master of https://github.com/flutter/devtools).

$ ~/Downloads/dart-sdk/bin/dart --version
Dart VM version: 2.9.0-5.0.dev (dev) (Thu Apr 30 13:02:02 2020 +0200) on "linux_x64"

$ PUB_HOSTED_URL='dartlang-pub-dev.appspot.com/'~/Downloads/dart-sdk/bin/pub publish
Publishing devtools 0.2.4+1 to https://dartlang-pub-dev.appspot.com/:
|-- .gitignore
|-- CHANGELOG.md
|-- LICENSE
|-- README.md
|-- bin
|   '-- devtools.dart
|-- devtools.iml
|-- lib
|   '-- placeholder.txt
'-- pubspec.yaml
Package validation found the following error:
* Your pubspec.yaml must not override non-dev dependencies.
  This ensures you test your package against the same versions of its dependencies
  that users will have when they use it.
Package validation found the following potential issues:
* Your dependency on "devtools_server" should allow more than one version. For example:

  dependencies:
    devtools_server: ^0.2.4+1

  Constraints that are too tight will make it difficult for people to use your package
  along with other packages that also depend on "devtools_server".
* Your dependency on "devtools_shared" should allow more than one version. For example:

  dependencies:
    devtools_shared: ^0.2.4+1

  Constraints that are too tight will make it difficult for people to use your package
  along with other packages that also depend on "devtools_shared".
Sorry, your package is missing a requirement and can't be published yet.
For more information, see: https://dart.dev/tools/pub/cmd/pub-lish.

And after deleting the overrides:

$ PUB_HOSTED_URL='https://dartlang-pub-dev.appspot.com/'~/Downloads/dart-sdk/bin/pub publish
Publishing devtools 0.2.4+1 to https://dartlang-pub-dev.appspot.com/:
|-- .gitignore
|-- CHANGELOG.md
|-- LICENSE
|-- README.md
|-- bin
|   '-- devtools.dart
|-- devtools.iml
|-- lib
|   '-- placeholder.txt
'-- pubspec.yaml
Package validation found the following potential issues:
* Your dependency on "devtools_server" should allow more than one version. For example:

  dependencies:
    devtools_server: ^0.2.4+1

  Constraints that are too tight will make it difficult for people to use your package
  along with other packages that also depend on "devtools_server".
* Your dependency on "devtools_shared" should allow more than one version. For example:

  dependencies:
    devtools_shared: ^0.2.4+1

  Constraints that are too tight will make it difficult for people to use your package
  along with other packages that also depend on "devtools_shared".

Uploads to pub.dev are subject to https://pub.dev/policy

Package has 2 warnings. Do you want to publish devtools 0.2.4+1 (y/N)? N
Package upload canceled.

Similar for devtools_app I was not able to reproduce.

What are the exact steps you took?

[terrylucas]

The steps are part of our publishing steps:

https://github.com/flutter/devtools/blob/master/tool/README.md

Start at the section titled Update your master branch from the remote repository

[pulyaevskiy]

Got this error as well in one of my projects.
For some reason pub publish looks at the generated build/ directory and fails there.
In my case the error looks like this:

The resolved 'packageUri' must be inside the rootUri, actualy 'file:///' is not in 'file:///Users/anatoly/Projects/node-interop/node_interop/build/packages/analyzer/', for the package 'analyzer'.
{"configVersion":2,"packages":[{"name":"analyzer","rootUri":"../packages/an...

As you see it looks inside .../node_interop/build/packages/analyzer/, which it probably shouldn't?

Workaround is to simply delete build/ folder and try again.

[sigurdm]

For some reason pub publish looks at the generated build/ directory and fails there.

That sounds like a plausible explanation - and I couldn't recreate because I had not run build/

Now I'll try to track down when we introduced this.

[sigurdm]

A bisect gives https://github.com/dart-lang/pub/commit/3cc6f963c68f8025dbe56d43520d44559fa85773 as the culprit.
Seems like something changed in the analyzer

[sigurdm]

As I read it, it is the parsing of .dart_tools/package_config.json that is triggering this.
in build/.dart_tools/package_config.json we have entries of the form:

{
  "configVersion": 2,
  "packages": [
    {
      "name": "_fe_analyzer_shared",
      "rootUri": "../packages/_fe_analyzer_shared",
      "packageUri": "",
      "languageVersion": "2.2"
    },
    {
      "name": "analyzer",
      "rootUri": "../packages/analyzer",
      "packageUri": "",
      "languageVersion": "2.6"
    },
...
}

The empty packageUri gets an extra / here: https://github.com/dart-lang/sdk/blob/master/pkg/analyzer/lib/src/context/package_config_json.dart#L255
And when we do rootUri.resolve('/') it becomes '/'.
CC @scheglov who wrote that.

So here are several potential problems at play:

• the analyzer should probably handle empty packageUris better (@lrhn for confirmation).
• not sure pub should tell the an analyzer to look inside build/ (and for sure not try to resolve the .dart_tools/package_config.json there).
• build/.dart_tools/package_config.json could be generated with packageUri: './' for working around the issue.

[devoncarew]

I don't believe we should be analyzing anything in build/ (we should also ignore dot directories, and file entries which match the excludes in an analysis options file).

[scheglov]

I looked at the package config v2 spec, and it seems that implementation in analyzer does not follow it precisely regarding the time when directory URIs should get /. I will fix this.

[scheglov]

https://dart-review.googlesource.com/c/sdk/+/147800

[lrhn]

Adding the / only after resolving against rootUri should indeed fix that problem.

As a side note: It's unnecessary to have a "packageUri": "", entry because omitting the entry means exactly the same thing, I'll check if package_config is generating the empty package URIs, and if so, stop it from doing that. (If it's Pub doing it, it should also stop it, @jonasfj).

[jonasfj]

pub always generates "packageUri": "lib/" (because of how packages are structured).

Hmm, maybe we should explicitly say that we only want a single analyzer context for the package root.. If there are sub-packages we are uninterested in those, specifically in:
https://github.com/dart-lang/pub/blob/31cfec521db6fe607f0f0424e747dff2e4950711/lib/src/dart.dart#L80-L114

I'm not sure if or how we could do that... I guess we could argue that only files under lib/, bin/, test/, benchmark/, should be considered part of the package. Also I'm not entirely sure I understand what an analysis context is.

[sigurdm]

I'm not sure if or how we could do that... I guess we could argue that only files under lib/, bin/, test/, benchmark/

I think (but don't know) that a context is something rooted at a directory that has a '.dart_tool/package_config.json' file that tells the analyzer how to resolve 'package:' style imports.

I guess we can iterate through all of them, and find the one that has root.root.path == path (with some normalization). If there is no such, it is an error.