Pub: "Invalid upload: no pubspec.yaml file" when trying to publish a package
I am trying to publish a new version of https://bitbucket.org/aholmgren/dart_jwt, a package that I have published many times before. For some reason it keeps failing
pub publish
Publishing dart_jwt 0.5.2 to https://pub.dartlang.org:
|-- .gitignore
|-- AUTHORS
|-- CHANGELOG.md
|-- LICENSE
|-- README.md
|-- bitbucket-pipelines.yml
|-- lib
| |-- dart_jwt.dart
| '-- src
| |-- codec.dart
| |-- jose.dart
| |-- jwa.dart
| |-- jws.dart
| |-- jwt.dart
| |-- jwt_claimset.dart
| |-- util.dart
| '-- validation_constraint.dart
|-- pubspec.yaml
'-- test
|-- all_tests.dart
|-- jwa_test.dart
|-- jwt_test.dart
'-- run.sh
Looks great! Are you ready to upload your package (y/n)? y
Uploading... (3.7s)
Invalid upload: no pubspec.yaml file.
Andersmholmgren
All 45 comments
This issue was moved to dart-lang/pub-dartlang-dart#55
nex3
on 16 Aug 2016
I moved this because upload validation is handled on the server side.
nex3
on 16 Aug 2016
I've moved this issue here again, because similar to dart-lang/pub/issues/1321 it seems to be an issue on the client side. The tarball that got uploaded is empty.
mkustermann
on 29 Aug 2016
@Andersmholmgren, can you include the result of running pub with --verbose? Does it work if you publish from a different OS or a different computer? Can you successfully publish other packages from the same computer?
nex3
on 29 Aug 2016
I have the same problem, and when I try to recreate the commands that Pub executes to get the tarball, I get this.
% git ls-files --cached --others --exclude-standard . | tar --format=ustar --create --gzip --directory /Users/emil.persson/Code/dart-embla/template_cache/. --files-from /dev/stdin
tar: .gitignore: Numeric user ID too largeNumeric group ID too large
tar: CHANGELOG.md: Numeric user ID too largeNumeric group ID too large
tar: LICENSE: Numeric user ID too largeNumeric group ID too large
tar: README.md: Numeric user ID too largeNumeric group ID too large
tar: bin/clean.dart: Numeric user ID too largeNumeric group ID too large
tar: bin/compile.dart: Numeric user ID too largeNumeric group ID too large
tar: lib/cache.dart: Numeric user ID too largeNumeric group ID too large
tar: lib/html_compiler.dart: Numeric user ID too largeNumeric group ID too large
tar: lib/plain_text_compiler.dart: Numeric user ID too largeNumeric group ID too large
tar: lib/render.dart: Numeric user ID too largeNumeric group ID too large
tar: lib/src/_gen/templates.dart: Numeric user ID too largeNumeric group ID too large
tar: lib/src/_gen/templates/.gitkeep: Numeric user ID too largeNumeric group ID too large
tar: lib/src/cache.dart: Numeric user ID too largeNumeric group ID too large
tar: lib/src/cache_io.dart: Numeric user ID too largeNumeric group ID too large
tar: lib/src/codegen_contract.dart: Numeric user ID too largeNumeric group ID too large
tar: lib/src/render.dart: Numeric user ID too largeNumeric group ID too large
tar: lib/src/simple_compiler_base.dart: Numeric user ID too largeNumeric group ID too large
tar: lib/src/uid.dart: Numeric user ID too largeNumeric group ID too large
tar: pubspec.yaml: Numeric user ID too largeNumeric group ID too large
tar: test/cache_test.dart: Numeric user ID too largeNumeric group ID too large
tar: test/plain_text_compiler_test.dart: Numeric user ID too largeNumeric group ID too large
emilniklas
on 13 Sep 2016
@emilniklas: That should have been fixed in 02443903ebaada11cad89386ad6d0202d2dbee38, which was released as part of Dart 1.19. What version of pub are you using?
nex3
on 27 Sep 2016
I am also seeing this error on pub v 1.20.1. Mac Sierra 10.12
Publishing seltzer 0.0.2 to https://pub.dartlang.org:
|-- .gitignore
|-- .travis.yml
|-- LICENSE
|-- README.md
|-- analysis_options.yaml
|-- bin
| |-- http_echo.dart
| '-- socket_echo.dart
|-- lib
| |-- platform
| | |-- browser.dart
| | '-- server.dart
| |-- seltzer.dart
| '-- src
| |-- context.dart
| '-- interface.dart
|-- pubspec.yaml
|-- test
| '-- platform
| |-- browser_test.dart
| |-- common_utils.dart
| |-- server_test.dart
| '-- web_socket
| |-- browser_test.dart
| |-- common_utils.dart
| '-- server_test.dart
'-- tool
|-- dartfmt.sh
'-- presubmit.sh
Looks great! Are you ready to upload your package (y/n)? y
Uploading... (13.3s)
Invalid upload: no pubspec.yaml file.
kendalharland
on 21 Oct 2016
@kharland Can you include the output of pub publish --verbose?
nex3
on 21 Oct 2016
FINE: Pub 1.20.1
IO : Spawning "git --version" in /Users/kjharland/Code/seltzer/.
IO : Finished git. Exit code 0.
| stdout:
| | git version 2.10.0
| Nothing output on stderr.
FINE: Determined git command null.
IO : Spawning "git check-ignore --quiet ." in /Users/kjharland/Code/seltzer/.
IO : Finished git. Exit code 1.
| Nothing output on stdout.
| Nothing output on stderr.
IO : Spawning "git ls-files --cached --others --exclude-standard ." in /Users/kjharland/Code/seltzer/.
IO : Finished git. Exit code 0.
| stdout:
| | .gitignore
| | .travis.yml
| | CHANGELOG.md
| | LICENSE
| | README.md
| | analysis_options.yaml
| | bin/http_echo.dart
| | bin/socket_echo.dart
| | lib/platform/browser.dart
| | lib/platform/server.dart
| | lib/platform/testing.dart
| | lib/seltzer.dart
| | lib/src/context.dart
| | lib/src/interface.dart
| | pubspec.yaml
| | test/canned_http_test.dart
| | test/platform/browser_test.dart
| | test/platform/common_utils.dart
| | test/platform/server_test.dart
| | test/platform/web_socket/browser_test.dart
| | test/platform/web_socket/common_utils.dart
| | test/platform/web_socket/server_test.dart
| | tool/dartfmt.sh
| | tool/presubmit.sh
| Nothing output on stderr.
FINE: Archiving and publishing seltzer 0.2.1-alpha (.).
MSG : Publishing seltzer 0.2.1-alpha to https://pub.dartlang.org:
| |-- .gitignore
| |-- .travis.yml
| |-- CHANGELOG.md
| |-- LICENSE
| |-- README.md
| |-- analysis_options.yaml
| |-- bin
| | |-- http_echo.dart
| | '-- socket_echo.dart
| |-- lib
| | |-- platform
| | | |-- browser.dart
| | | |-- server.dart
| | | '-- testing.dart
| | |-- seltzer.dart
| | '-- src
| | |-- context.dart
| | '-- interface.dart
| |-- pubspec.yaml
| |-- test
| | |-- canned_http_test.dart
| | '-- platform
| | |-- browser_test.dart
| | |-- common_utils.dart
| | |-- server_test.dart
| | '-- web_socket
| | |-- browser_test.dart
| | |-- common_utils.dart
| | '-- server_test.dart
| '-- tool
| |-- dartfmt.sh
| '-- presubmit.sh
FINE: Creating .tag.gz stream containing:
| ./.gitignore
| ./.travis.yml
| ./CHANGELOG.md
| ./LICENSE
| ./README.md
| ./analysis_options.yaml
| ./bin/http_echo.dart
| ./bin/socket_echo.dart
| ./lib/platform/browser.dart
| ./lib/platform/server.dart
| ./lib/platform/testing.dart
| ./lib/seltzer.dart
| ./lib/src/context.dart
| ./lib/src/interface.dart
| ./pubspec.yaml
| ./test/canned_http_test.dart
| ./test/platform/browser_test.dart
| ./test/platform/common_utils.dart
| ./test/platform/server_test.dart
| ./test/platform/web_socket/browser_test.dart
| ./test/platform/web_socket/common_utils.dart
| ./test/platform/web_socket/server_test.dart
| ./tool/dartfmt.sh
| ./tool/presubmit.sh
IO : Spawning "git ls-files --cached --others --exclude-standard ." in /Users/kjharland/Code/seltzer/.
IO : Finished git. Exit code 0.
| stdout:
| | .gitignore
| | .travis.yml
| | CHANGELOG.md
| | LICENSE
| | README.md
| | analysis_options.yaml
| | bin/http_echo.dart
| | bin/socket_echo.dart
| | lib/platform/browser.dart
| | lib/platform/server.dart
| | lib/platform/testing.dart
| | lib/seltzer.dart
| | lib/src/context.dart
| | lib/src/interface.dart
| | pubspec.yaml
| | test/canned_http_test.dart
| | test/platform/browser_test.dart
| | test/platform/common_utils.dart
| | test/platform/server_test.dart
| | test/platform/web_socket/browser_test.dart
| | test/platform/web_socket/common_utils.dart
| | test/platform/web_socket/server_test.dart
| | tool/dartfmt.sh
| | tool/presubmit.sh
| Nothing output on stderr.
IO : Spawning "git ls-files --cached --others --exclude-standard ." in /Users/kjharland/Code/seltzer/.
IO : Finished git. Exit code 0.
| stdout:
| | .gitignore
| | .travis.yml
| | CHANGELOG.md
| | LICENSE
| | README.md
| | analysis_options.yaml
| | bin/http_echo.dart
| | bin/socket_echo.dart
| | lib/platform/browser.dart
| | lib/platform/server.dart
| | lib/platform/testing.dart
| | lib/seltzer.dart
| | lib/src/context.dart
| | lib/src/interface.dart
| | pubspec.yaml
| | test/canned_http_test.dart
| | test/platform/browser_test.dart
| | test/platform/common_utils.dart
| | test/platform/server_test.dart
| | test/platform/web_socket/browser_test.dart
| | test/platform/web_socket/common_utils.dart
| | test/platform/web_socket/server_test.dart
| | tool/dartfmt.sh
| | tool/presubmit.sh
| Nothing output on stderr.
IO : Spawning "git ls-files --cached --others --exclude-standard ." in /Users/kjharland/Code/seltzer/.
IO : Finished git. Exit code 0.
| stdout:
| | .gitignore
| | .travis.yml
| | CHANGELOG.md
| | LICENSE
| | README.md
| | analysis_options.yaml
| | bin/http_echo.dart
| | bin/socket_echo.dart
| | lib/platform/browser.dart
| | lib/platform/server.dart
| | lib/platform/testing.dart
| | lib/seltzer.dart
| | lib/src/context.dart
| | lib/src/interface.dart
| | pubspec.yaml
| | test/canned_http_test.dart
| | test/platform/browser_test.dart
| | test/platform/common_utils.dart
| | test/platform/server_test.dart
| | test/platform/web_socket/browser_test.dart
| | test/platform/web_socket/common_utils.dart
| | test/platform/web_socket/server_test.dart
| | tool/dartfmt.sh
| | tool/presubmit.sh
| Nothing output on stderr.
IO : Reading binary file ./README.md.
IO : Read 2149 bytes from ./README.md.
IO : Spawning "git ls-files --cached --others --exclude-standard ." in /Users/kjharland/Code/seltzer/.
IO : Finished git. Exit code 0.
| stdout:
| | .gitignore
| | .travis.yml
| | CHANGELOG.md
| | LICENSE
| | README.md
| | analysis_options.yaml
| | bin/http_echo.dart
| | bin/socket_echo.dart
| | lib/platform/browser.dart
| | lib/platform/server.dart
| | lib/platform/testing.dart
| | lib/seltzer.dart
| | lib/src/context.dart
| | lib/src/interface.dart
| | pubspec.yaml
| | test/canned_http_test.dart
| | test/platform/browser_test.dart
| | test/platform/common_utils.dart
| | test/platform/server_test.dart
| | test/platform/web_socket/browser_test.dart
| | test/platform/web_socket/common_utils.dart
| | test/platform/web_socket/server_test.dart
| | tool/dartfmt.sh
| | tool/presubmit.sh
| Nothing output on stderr.
IO : Spawning "git ls-files --cached --others --exclude-standard bin" in /Users/kjharland/Code/seltzer/.
IO : Finished git. Exit code 0.
| stdout:
| | bin/http_echo.dart
| | bin/socket_echo.dart
| Nothing output on stderr.
IO : Spawning "tar --format=ustar --create --gzip --directory /Users/kjharland/Code/seltzer/. --files-from /dev/stdin" in /Users/kjharland/Code/seltzer/.
FINE: Showing confirm message:
| Looks great! Are you ready to upload your package
Looks great! Are you ready to upload your package (y/n)? y
FINE: Loading OAuth2 credentials.
FINE: Saving OAuth2 credentials.
IO : Writing 300 characters to text file /Users/kjharland/.pub-cache/credentials.json.
MSG : Uploading...
FINE: HTTP GET https://pub.dartlang.org/api/packages/versions/new
| Accept: application/vnd.pub.v2+json
| authorization: <censored>
| user-agent: Dart pub 1.20.1
FINE: HTTP response 200 OK for GET https://pub.dartlang.org/api/packages/versions/new
| took 0:00:00.760495
| x-cloud-trace-context: 69761d1a34395c58320b5d4c015d0b4b
| date: Wed, 26 Oct 2016 20:59:42 GMT
| content-length: 1076
| content-type: application/json
| x-frame-options: SAMEORIGIN
| x-xss-protection: 1; mode=block
| x-content-type-options: nosniff
| server: Google Frontend
FINE: HTTP POST https://storage.googleapis.com
| authorization: <censored>
| user-agent: Dart pub 1.20.1
|
| Body fields:
| key: pub.dartlang.org/tmp/19fc8c76-19cc-434f-8a86-596e6dab2392
| acl: project-private
| Expires: 2016-10-26T21:09:42.832319Z
| GoogleAccessId: [email protected]
| policy: eyJleHBpcmF0aW9uIjoiMjAxNi0xMC0yNlQyMTowOTo0Mi44MzIzMTlaIiwiY29uZGl0aW9ucyI6W3sia2V5IjoicHViLmRhcnRsYW5nLm9yZy90bXAvMTlmYzhjNzYtMTljYy00MzRmLThhODYtNTk2ZTZkYWIyMzkyIn0seyJhY2wiOiJwcm9qZWN0LXByaXZhdGUifSx7ImV4cGlyZXMiOiIyMDE2LTEwLTI2VDIxOjA5OjQyLjgzMjMxOVoifSx7InN1Y2Nlc3NfYWN0aW9uX3JlZGlyZWN0IjoiaHR0cHM6Ly9wdWIuZGFydGxhbmcub3JnL2FwaS9wYWNrYWdlcy92ZXJzaW9ucy9uZXdVcGxvYWRGaW5pc2g/dXBsb2FkX2lkPTE5ZmM4Yzc2LTE5Y2MtNDM0Zi04YTg2LTU5NmU2ZGFiMjM5MiJ9LFsiY29udGVudC1sZW5ndGgtcmFuZ2UiLDAsMTA0ODU3NjAwXV19
| signature: fsPQeajseZcg4BZiEZjba1i1gICkKa80Tbwhk7sdj6ELlR2MI/VCIUjgbgOMpqLvc9VBYmlMGfbmHNJsK/9GqXiLIxuFk+TK+hklGBDTuk9dzVOzIJila55gOsvVfHoqiaDJivq6MG+4ubaM5V5GlsMQG8Zj4e6fZkFgd9Z8A4Q=
| success_action_redirect: https://pub.dartlang.org/api/packages/versions/newUploadFinish?upload_id=19fc8c76-19cc-434f-8a86-596e6dab2392
FINE: HTTP response 303 See Other for POST https://storage.googleapis.com
| took 0:00:00.906276
| alt-svc: quic=":443"; ma=2592000; v="36,35,34"
| location: https://pub.dartlang.org/api/packages/versions/newUploadFinish?upload_id=19fc8c76-19cc-434f-8a86-596e6dab2392&bucket=pub.dartlang.org&key=tmp/19fc8c76-19cc-434f-8a86-596e6dab2392&etag=fd53b108471acbda4cbe335ab60d5b1d
| date: Wed, 26 Oct 2016 20:59:43 GMT
| vary: Origin
| content-length: 0
| x-guploader-uploadid: AEnB2UoUNBSprFUGsp8iZjFaVH2kd8goN9GCNIRwFyDd3b0YE6vgKz2TpmlT4N2wr_9K6GHW-EMuD9V4MebOeT8x7ih-8LxxhQ
| content-type: text/html; charset=UTF-8
| server: UploadServer
FINE: HTTP GET https://pub.dartlang.org/api/packages/versions/newUploadFinish?upload_id=19fc8c76-19cc-434f-8a86-596e6dab2392&bucket=pub.dartlang.org&key=tmp/19fc8c76-19cc-434f-8a86-596e6dab2392&etag=fd53b108471acbda4cbe335ab60d5b1d
| Accept: application/vnd.pub.v2+json
| authorization: <censored>
| user-agent: Dart pub 1.20.1
FINE: HTTP response 400 Bad Request for GET https://pub.dartlang.org/api/packages/versions/newUploadFinish?upload_id=19fc8c76-19cc-434f-8a86-596e6dab2392&bucket=pub.dartlang.org&key=tmp/19fc8c76-19cc-434f-8a86-596e6dab2392&etag=fd53b108471acbda4cbe335ab60d5b1d
| took 0:00:00.608018
| x-cloud-trace-context: 2e750e5315ab576ad2f11249e4a6127d
| date: Wed, 26 Oct 2016 20:59:44 GMT
| content-length: 61
| content-type: application/json
| x-frame-options: SAMEORIGIN
| x-xss-protection: 1; mode=block
| x-content-type-options: nosniff
| server: Google Frontend
FINE: Uploading finished (2.301s).
FINE: Saving OAuth2 credentials.
IO : Writing 300 characters to text file /Users/kjharland/.pub-cache/credentials.json.
ERR : Invalid upload: no pubspec.yaml file.
FINE: Exception type: ApplicationException
FINE: package:pub/src/utils.dart 780 fail
| package:pub/src/http.dart 206 handleJsonError
| package:pub/src/command/lish.dart 102 LishCommand._publish.<async>
| dart:async _Completer.completeError
| package:pub/src/command/lish.dart 92 LishCommand._publish.<async>.<fn>.<fn>.<async>
| dart:async _Completer.completeError
| package:http/src/base_client.dart 175 BaseClient._sendUnstreamed.<async>
| dart:async _Completer.completeError
| package:oauth2/src/client.dart 140 Client.send.<async>
| dart:async _Completer.completeError
| package:pub/src/http.dart 108 _PubHttpClient.send.<async>
| ===== asynchronous gap ===========================
| dart:async _Completer.completeError
| package:pub/src/command/lish.dart 109 LishCommand._publish.<async>
| dart:async _Completer.completeError
| package:pub/src/command/lish.dart 92 LishCommand._publish.<async>.<fn>.<fn>.<async>
| dart:async _Completer.completeError
| package:http/src/base_client.dart 175 BaseClient._sendUnstreamed.<async>
| dart:async _Completer.completeError
| package:oauth2/src/client.dart 140 Client.send.<async>
| dart:async _Completer.completeError
| package:pub/src/http.dart 108 _PubHttpClient.send.<async>
| ===== asynchronous gap ===========================
| dart:async _asyncErrorWrapperHelper
| package:pub/src/command/lish.dart LishCommand._publish
| package:pub/src/command/lish.dart 140 LishCommand.run.<async>
| ===== asynchronous gap ===========================
| dart:async _asyncThenWrapperHelper
| package:pub/src/command/lish.dart LishCommand.run
| package:args/command_runner.dart 177 CommandRunner.runCommand.<fn>
| dart:async Future.Future.sync
| package:args/command_runner.dart 130 CommandRunner.runCommand
| package:pub/src/command_runner.dart 128 PubCommandRunner.runCommand.<async>.<fn>
| dart:async Future.Future.sync
| package:pub/src/utils.dart 72 captureErrors.<fn>
| package:stack_trace Chain.capture
| package:pub/src/utils.dart 86 captureErrors
| package:pub/src/command_runner.dart 128 PubCommandRunner.runCommand.<async>
| dart:async _SyncCompleter.complete
| package:pub/src/command_runner.dart 214 PubCommandRunner._validatePlatform.<async>
If you manually create a tar file using tar --format=ustar --create --gzip --directory /Users/kjharland/Code/seltzer/. ., does that work? Does it print any warnings? Can you see files inside it?
nex3
on 26 Oct 2016
--format=ustar was the issue for me. If I comment that out in my fork of pub, everything works out.
emilniklas
on 30 Oct 2016
What's the output of tar --version? What version of what OS are you on?
nex3
on 31 Oct 2016
Little late, but my tar --version output is bsdtar 2.8.3 - libarchive 2.8.3.
I'm unable to create an archive with the command you've given, tar gives me many Numeric user ID too large errors. Might be the root cause, I'll see if I can fix that first.
kendalharland
on 31 Oct 2016
What OS are you using?
nex3
on 31 Oct 2016
macOS Sierra - 10.12
kendalharland
on 1 Nov 2016
I thought we'd checked that the default tar on Mac OS supported that format. @kevmoo, what's the tar version on your machine?
nex3
on 1 Nov 2016
bsdtar 2.8.3 - libarchive 2.8.3
I'm on El Capitan, though. Issue w/ latest?
kevmoo
on 1 Nov 2016
The same tar version should have the same behavior. Does --format=ustar work for you if you use it explicitly?
nex3
on 1 Nov 2016
Nope, here is my output (different project w/output truncated. It's the same error all the way down):
tar --format=ustar --create --gzip --directory $(pwd) .
tar: ./: Numeric user ID too large
tar: ./.git/: Numeric user ID too large
tar: ./.gitignore: Numeric user ID too large
tar: ./.idea/: Numeric user ID too large
tar: ./.packages: Numeric user ID too large
tar: ./.pub/: Numeric user ID too large
tar: ./.travis.yml: Numeric user ID too large
tar: ./.vscode/: Numeric user ID too large
tar: ./analysis_options.yaml: Numeric user ID too large
tar: ./bin/: Numeric user ID too large
tar: ./browser: Numeric user ID too large
tar: ./echo_char.dart: Numeric user ID too large
tar: ./lib/: Numeric user ID too large
tar: ./LICENSE: Numeric user ID too large
tar: ./pubspec.lock: Numeric user ID too large
tar: ./pubspec.yaml: Numeric user ID too large
tar: ./README.md: Numeric user ID too large
tar: ./server: Numeric user ID too large
tar: ./test/: Numeric user ID too large
tar: ./tool/: Numeric user ID too large
tar: ./toy.dart: Numeric user ID too large
tar: ./tool/.gitignore: Numeric user ID too large
tar: ./tool/dartfmt.sh: Numeric user ID too large
tar: ./tool/presubmit.sh: Numeric user ID too large
tar: ./test/.gitignore: Numeric user ID too large
tar: ./test/data_channel_test.dart: Numeric user ID too large
tar: ./test/io_node_test.dart: Numeric user ID too large
tar: ./test/message_handlers_test.dart: Numeric user ID too large
tar: ./lib/distributed.dart: Numeric user ID too large
tar: ./lib/interfaces/: Numeric user ID too large
tar: ./lib/platform/: Numeric user ID too large
tar: ./lib/src/: Numeric user ID too large
tar: ./lib/src/command.dart: Numeric user ID too large
tar: ./lib/src/configuration.dart: Numeric user ID too large
tar: ./lib/src/editor.dart: Numeric user ID too large
tar: ./lib/src/io/: Numeric user ID too large
tar: ./lib/src/networking/: Numeric user ID too large
tar: ./lib/src/repl.dart: Numeric user ID too large
tar: ./lib/src/networking/connection.dart: Numeric user ID too large
tar: ./lib/src/networking/data_channel.dart: Numeric user ID too large
tar: ./lib/src/networking/json.dart: Numeric user ID too large
tar: ./lib/src/networking/message.dart: Numeric user ID too large
tar: ./lib/src/networking/message_handlers.dart: Numeric user ID too large
tar: ./lib/src/io/data_channel.dart: Numeric user ID too large
tar: ./lib/src/io/node.dart: Numeric user ID too large
tar: ./lib/platform/io.dart: Numeric user ID too large
tar: ./lib/interfaces/command.dart: Numeric user ID too large
tar: ./lib/interfaces/node.dart: Numeric user ID too large
tar: ./lib/interfaces/peer.dart: Numeric user ID too large
tar: ./lib/interfaces/serializer.dart: Numeric user ID too large
tar: ./bin/io_node_demo.dart: Numeric user ID too large
tar: ./.vscode/settings.json: Numeric user ID too large
tar: ./.pub/bin/: Numeric user ID too large
tar: ./.pub/deps/: Numeric user ID too large
I installed gnu-tar 1.29 and that works fine with the given command. I'm not sure what would case tar to fail in this case.
kendalharland
on 1 Nov 2016
Sorry, I meant if @kevmoo runs it explicitly.
nex3
on 1 Nov 2016
@nex3 You'd like me to publish it? I'm confused...
kevmoo
on 1 Nov 2016
I want you to try to create an archive by passing --format=ustar to tar.
nex3
on 1 Nov 2016
On my mac (El Capitan)
bsdtar 2.8.3 - libarchive 2.8.3
tar --format ustar -cf lib.targ.gz lib works
kevmoo
on 1 Nov 2016
That's puzzling and frustrating. Why would that format work on one tar 2.8.3 and not on another? Is there something strange about Sierra in particular?
Is anyone else using Sierra who can try this out?
nex3
on 1 Nov 2016
@nex3 Updated with Dart version: I was using 1.20.1 but when it didn't work @kevmoo suggested I try switching to a dev version.
I am experiencing the exact same problem but not on Sierra. I can't pub publish and when I try to manually tar I get the Numeric user ID too large errors.
Mac OSX El Capitan 10.11.6
tar version: bsdtar 2.8.3 - libarchive 2.8.3
Dart VM version: 1.21.0-dev.11.1 (Wed Nov 30 08:23:12 2016) on "macos_x64"
nshahan
on 1 Dec 2016
@nshahan What Dart version are you using?
nex3
on 1 Dec 2016
@nex3 I updated Dart today to:
Dart VM version: 1.22.0-dev.0.0 (Wed Dec 7 09:15:00 2016) on "macos_x64"
The issue persists.
nshahan
on 9 Dec 2016
I'm really mystified by this. I can't imagine why the exact same version of tar would support different options on different systems.
@nshahan What's the output of tar --help on your system?
nex3
on 12 Dec 2016
Looks like this:
$ tar --help
tar(bsdtar): manipulate archive files
First option must be a mode specifier:
-c Create -r Add/Replace -t List -u Update -x Extract
Common Options:
-b # Use # 512-byte records per I/O block
-f <filename> Location of archive
-v Verbose
-w Interactive
Create: tar -c [options] [<file> | <dir> | @<archive> | -C <dir> ]
<file>, <dir> add these items to archive
-z, -j, -J, --lzma Compress archive with gzip/bzip2/xz/lzma
--format {ustar|pax|cpio|shar} Select archive format
--exclude <pattern> Skip files that match pattern
-C <dir> Change to <dir> before processing remaining files
@<archive> Add entries from <archive> to output
List: tar -t [options] [<patterns>]
<patterns> If specified, list only entries that match
Extract: tar -x [options] [<patterns>]
<patterns> If specified, extract only entries that match
-k Keep (don't overwrite) existing files
-m Don't restore modification times
-O Write entries to stdout, don't restore to disk
-p Restore permissions (including ACLs, owner, file flags)
bsdtar 2.8.3 - libarchive 2.8.3
Okay, I think I've figured this out. We're already passing some options on Linux to avoid the "numeric user ID" issue, but those options don't exist for BSD tar. However, it looks like similar options do exist, so we should start using those.
nex3
on 13 Dec 2016
Woohoo! Thanks for investigating @nex3
kendalharland
on 16 Dec 2016
Alas, this fix doesn't work on OS X because its decrepit version of tar doesn't support --uname and --gname, which breaks pub lish for all OS X users. Re-opening until we come up with a solution.
munificent
on 13 Jan 2017
If any users want to help out with this, the best path forward is probably to use the pure-Dart archive package to dodge any weird platform differences. Before we can do that, though, we need to verify that it can properly handle the many quirks in the tar files that are already on pub.dartlang.org. If someone can do that verification, we can probably get this fixed for real.
nex3
on 13 Jan 2017
Curious, what are some of the quirks of the pub.dartlang.org files?
kendalharland
on 13 Jan 2017
I'm not really a tar expert—all I know is that it's an old format with a lot of corner cases and custom extensions, and we need to be sure that the package doesn't barf on them.
nex3
on 13 Jan 2017
After investigating a bit, it seems to me that the current implementation is the worst one possible – because currently, the only way to get pub publish to work on OSX seems to:
- use gnu-tar (for example via homebrew, and adding it to your path)
- use a Dart version that includes the fix for #1438 (9 Aug 2016), but does NOT include the fix for #1491 (13 Dec 2016) – I didn't look up which Dart versions match these criteria, sorry.
Why gnu-tar is required:
OSX El Capitan (10.11) as well as Sierra (10.12) use bsdtar 2.8.3 – pub publish will fail on these OSes due to command line arguments missing that would prevent UIDs from being too large.
Why even gnu-tar fails outside of the criteria stated above:
The options --owner=pub and --group=pub, defined here, seem to get ignored on OSX, resulting in our beloved 'UID/GID too large' error. If --owner=0 and --group=0 are used, as was the case after #1438 and before #1491, gnu-tar on OSX works fine.
So the quick fix would be to introduce --owner=0 and --group=0 for !Platform.isLinux and !Platform.isWindows. Then at least gnu-tar would be an option for OSX users.
The full fix to get pub publish to work with bsdtar... I (with limited command line skills) would approach it like this:
- introduce a dedicated block for
Platform.isMacOSthat assembles its own args because the current ones here are mostly not available on bsdtar. - copy the contents of
baseDirto a temp folder - run
chown(or whatever) over the temp folder/files to ensure UID/GID in range - run the tar command against that directory
- delete the temp folder
What do you think of the full fix?
That aside, I really could use the quick fix... quickly :-)
nilsdoehring
on 24 Feb 2017
Adding more filesystem operations adds a lot more room for tricky edge-case errors and flakes. I still think the best way forward is to verify and then use the pure-Dart tar implementation.
nex3
on 24 Feb 2017
Thanks, understood. So I looked into the man pages once more, and found a real solution: In bsdtar, the way to set uid/gid is to provide it with a file in mtree format. Example from man page:
$ cat input.mtree
#mtree
usr/bin uid=0 gid=0 mode=0755 type=dir
usr/bin/ls uid=0 gid=0 mode=0755 type=file content=myls
$ tar -cvf output.tar @input.mtree
Luckily, we can easily make the contents List into an mtree:
var uidGid = " uid=0, gid=0, type=file";
var mtreeContents = "#mtree\n" + contents.join("$uidGid\n") + uidGid;
Here's the whole snippet:
if (Platform.isLinux) {
// GNU tar flags.
// https://www.gnu.org/software/tar/manual/html_section/tar_33.html
args.addAll(["--files-from", "/dev/stdin"]);
// The ustar format doesn't support large UIDs. We don't care about
// preserving ownership anyway, so we just set them to "pub".
args.addAll(["--owner=pub", "--group=pub"]);
var process = await startProcess("tar", args);
process.stdin.add(UTF8.encode(contents.join("\n")));
process.stdin.close();
return process.stdout;
} else {
// BSD tar flags.
// https://www.freebsd.org/cgi/man.cgi?query=bsdtar&sektion=1
// The only way to set ownership is via an mtree input file.
// Create the file containing the list of files to compress.
var tempDir = createSystemTempDir();
var contentsPath = path.join(tempDir, "files.txt");
// Define the postfix for each file path
var uidGid = " uid=0, gid=0, type=file";
writeTextFile(contentsPath, "#mtree\n" + contents.join("$uidGid\n") + uidGid);
args.add("@${contentsPath}");
var result = await runProcess("tar", args, workingDir: baseDir);
deleteEntry(tempDir);
return result.stdout;
}
See patch.
If you like to investigate yourself, here are the commands necessary:
# outputs mtree of the current folder
tar -c --format=mtree --options='!uname,!size,!time' .
# takes an mtree file (input.mtree) as input
tar -cvzf test.tar.gz --format=ustar -C . @input.mtree
Here, if the mtree contains uid or gid which are too high, the command fails. If uid/gid are set to 0 (or whatever is in range), it works, and that's what the script above is ensuring.
nilsdoehring
on 25 Feb 2017
@nilsdoehring Can you verify that OS X's version of bsdtar actually supports that option? I believe it still ships with an ancient version which doesn't support many features that are in BSD proper.
If so, can you file a pull request with your proposed changes?
nex3
on 1 Mar 2017
@nex3 I can indeed verify that the options in use here exist in bsdtar at least since OSX 10.9. man tar on Sierra spits out the identical options.
Will follow up with a PR, tonight over the weekend.
nilsdoehring
on 2 Mar 2017
@nex3 OP delivered.
1525
nilsdoehring
on 5 Mar 2017
I ran into the same issue: Invalid upload: no pubspec.yaml file.
Below are the version number of my library and system:
Flutter (Channel beta, v0.4.4)
on Mac OS X 10.13.3 (macOS high sierra)
Pub 2.0.0-dev.54.0.
bsdtar 2.8.3 - libarchive 2.8.3
jiajiabingcheng
on 8 Jun 2018
Same issue again, Ubuntu 16.04.
ranquild
on 4 Sep 2018
https://github.com/flutter/flutter/issues/21305 is another one
zoechi
on 4 Sep 2018
Related issues
goderbauer
·
57Comments
jonasfj
·
31Comments
jayoung-lee
·
38Comments
kasperpeulen
·
27Comments
sanjidtt
·
36Comments
Most helpful comment
Adding more filesystem operations adds a lot more room for tricky edge-case errors and flakes. I still think the best way forward is to verify and then use the pure-Dart tar implementation.