Conan: Can not upload package on OSX

Created on 25 Jan 2019  路  16Comments  路  Source: conan-io/conan

Hi!

I'm trying to update log4cplus package. It's working fine, except by this little detail:

>> Uploading packages for 'log4cplus/2.0.0-rc2@bincrafters/stable'
 >> Verifying credentials...
 >> OK! '[secure]' user logged in 'upload_repo' 
Uploading log4cplus/2.0.0-rc2@bincrafters/stable to remote 'upload_repo'
Compressing recipe sources...                                         Recipe is up to date, upload skipped
Uploading package 1/1: 13b48d56cd5d7dff22684d58482d685be985d688 to 'upload_repo'
Compressing package...
[==================================================]
Traceback (most recent call last):
  File "build.py", line 11, in <module>
    builder.run()
  File "/Users/travis/.pyenv/versions/conan/lib/python2.7/site-packages/cpt/packager.py", line 439, in run
    self.run_builds(base_profile_name=base_profile_name)
  File "/Users/travis/.pyenv/versions/conan/lib/python2.7/site-packages/cpt/packager.py", line 515, in run_builds
    r.run()
  File "/Users/travis/.pyenv/versions/conan/lib/python2.7/site-packages/cpt/runner.py", line 98, in run
    self._upload, package_id)
  File "/Users/travis/.pyenv/versions/conan/lib/python2.7/site-packages/cpt/uploader.py", line 43, in upload_packages
    retry=int(self._upload_retry))
  File "/Users/travis/.pyenv/versions/conan/lib/python2.7/site-packages/conans/client/conan_api.py", line 87, in wrapper
    return f(*args, **kwargs)
  File "/Users/travis/.pyenv/versions/conan/lib/python2.7/site-packages/conans/client/conan_api.py", line 819, in upload
    retry_wait, integrity_check, policy, remote_name, query=query)
  File "/Users/travis/.pyenv/versions/conan/lib/python2.7/site-packages/conans/client/cmd/uploader.py", line 70, in upload
    integrity_check, policy, remote_name, recorder)
  File "/Users/travis/.pyenv/versions/conan/lib/python2.7/site-packages/conans/client/cmd/uploader.py", line 126, in _upload
    integrity_check, policy, p_remote)
  File "/Users/travis/.pyenv/versions/conan/lib/python2.7/site-packages/conans/client/cmd/uploader.py", line 170, in _upload_package
    integrity_check, policy)
  File "/Users/travis/.pyenv/versions/conan/lib/python2.7/site-packages/conans/client/remote_manager.py", line 151, in upload_package
    the_files, retry, retry_wait, policy)
  File "/Users/travis/.pyenv/versions/conan/lib/python2.7/site-packages/conans/client/remote_manager.py", line 332, in _call_remote
    raise ConanException(exc)
conans.errors.ConanException: 'decimal' codec can't encode characters in position 0-5408: invalid decimal Unicode string
The command "./.ci/run.sh" exited with 1.

I tried to upload for 3 times, same result.

The full log is here:
https://travis-ci.com/bincrafters/conan-log4cplus/jobs/172072571#L1076

The point is, I've never seen this before and the error is for one package only. Could it be an error from Bintray? Also, I'm using python 2.7 for OSX jobs.

Environment:

Conan Version: 1.11.2
Configuration:

[settings]
arch=x86_64
arch_build=x86_64
build_type=Release
compiler=apple-clang
compiler.libcxx=libc++
compiler.version=9.1
os=Macos
os_build=Macos
[options]
log4cplus:shared=True
[build_requires]
[env]
  • [ ] I've read the CONTRIBUTING guide.
  • [ ] I've specified the Conan version, operating system version and any tool that can be relevant.
  • [ ] I've explained the steps to reproduce the error or the motivation/use case of the question/suggestion.
low bintray medium
Source
picture uilianries

Most helpful comment

But then we would have a corrupted package without knowing it. I'm not sure if it is important enough or not (also thinking about package revisions), but I don't feel comfortable ignoring an error. I've proposed a try/except wrap around the FileTreeManifest.loads function to throw a more meaningful function (#4415). Feel free to criticize it 馃槃

picture jgsogo on 29 Jan 2019
👍2

All 16 comments

/cc @SSE4

uilianries picture uilianries on 25 Jan 2019

Thanks to @jgsogo and @SSE4!

https://travis-ci.com/uilianries/conan-log4cplus/jobs/173441728#L3202

ERROR :remote_manager.py[331]: Traceback (most recent call last):
                                 File "/Users/travis/.pyenv/versions/conan/lib/python2.7/site-packages/conans/client/remote_manager.py", line 324, in _call_remote
                                   return getattr(self._auth_manager, method)(*argc, **argv)
                                 File "/Users/travis/.pyenv/versions/conan/lib/python2.7/site-packages/conans/client/rest/auth_manager.py", line 32, in wrapper
                                   ret = func(self, *args, **kwargs)
                                 File "/Users/travis/.pyenv/versions/conan/lib/python2.7/site-packages/conans/client/rest/auth_manager.py", line 136, in upload_package
                                   policy)
                                 File "/Users/travis/.pyenv/versions/conan/lib/python2.7/site-packages/conans/client/rest/rest_client.py", line 74, in upload_package
                                   no_overwrite)
                                 File "/Users/travis/.pyenv/versions/conan/lib/python2.7/site-packages/conans/client/rest/rest_client_common.py", line 242, in upload_package
                                   remote_manifest = self.get_package_manifest(pref_snapshot)
                                 File "/Users/travis/.pyenv/versions/conan/lib/python2.7/site-packages/conans/client/rest/rest_client_v1.py", line 95, in get_package_manifest
                                   return FileTreeManifest.loads(contents[CONAN_MANIFEST])
                                 File "/Users/travis/.pyenv/versions/conan/lib/python2.7/site-packages/conans/model/manifest.py", line 67, in loads
                                   the_time = int(tokens[0])
                               UnicodeEncodeError: 'decimal' codec can't encode characters in position 0-5408: invalid decimal Unicode string
                                [2019-01-28 14:54:06,170]

https://github.com/conan-io/conan/blob/release/1.11.2/conans/model/manifest.py#L67

According the code, the manifest content file is split by \n and first topic (date) is parsed to int. My guess is, the file content has no new line and the first topic is the entire content (5k chars).

uilianries picture uilianries on 28 Jan 2019

may be we may add something like:

the_time = 0
if len(tokens) > 0 and isdigit(tokens[0]):
    the_time = int(tokens[0])

also, is it possible to somehow get manifest contents to inspect them?

SSE4 picture SSE4 on 28 Jan 2019

I think we could run cat to read its content. I'll update the branch to print it.

uilianries picture uilianries on 28 Jan 2019

I can run it and upload to my bintray repo without fails (same profile but apple-clang 10.0).

Yes, please, cat the manifest file and let us know the output. Mine is:

1548694923
conaninfo.txt: 4234ecf07585dcc596b6023fdb873ae9
include/log4cplus/appender.h: 5f7af29ebc669838f4961315c89447ee
include/log4cplus/asyncappender.h: 25bd54207871782e5475d9f81016ef24
include/log4cplus/boost/deviceappender.hxx: 4d40ab0de8d827a421cb9a9e170c17b0
include/log4cplus/callbackappender.h: 3062fed9518fa16c3d058acfd2cba3b7
include/log4cplus/clogger.h: e6b08530ab7f5fff13b2759f4680bba1
include/log4cplus/config.hxx: 19ce67da1b94fff2f61b6882e5fe91ff
...
jgsogo picture jgsogo on 28 Jan 2019

Well, the manifest file looks normal:

1548699429$
conaninfo.txt: e4a7cacf857c47e3043947b6a1f17715$
include/log4cplus/appender.h: 5f7af29ebc669838f4961315c89447ee$
include/log4cplus/asyncappender.h: 25bd54207871782e5475d9f81016ef24$
include/log4cplus/boost/deviceappender.hxx: 4d40ab0de8d827a421cb9a9e170c17b0$
include/log4cplus/callbackappender.h: 3062fed9518fa16c3d058acfd2cba3b7$

The $ is because I added -e to print the eol.

https://travis-ci.com/uilianries/conan-log4cplus/jobs/173506120#L3218

uilianries picture uilianries on 28 Jan 2019

For python 3 we have a similar error:

ERROR :remote_manager.py[331]: Traceback (most recent call last):
                                 File "/Users/travis/.pyenv/versions/conan/lib/python3.4/site-packages/conans/client/remote_manager.py", line 324, in _call_remote
                                   return getattr(self._auth_manager, method)(*argc, **argv)
                                 File "/Users/travis/.pyenv/versions/conan/lib/python3.4/site-packages/conans/client/rest/auth_manager.py", line 32, in wrapper
                                   ret = func(self, *args, **kwargs)
                                 File "/Users/travis/.pyenv/versions/conan/lib/python3.4/site-packages/conans/client/rest/auth_manager.py", line 136, in upload_package
                                   policy)
                                   no_overwrite)
                                 File "/Users/travis/.pyenv/versions/conan/lib/python3.4/site-packages/conans/client/rest/rest_client_common.py", line 242, in upload_package
                                   remote_manifest = self.get_package_manifest(pref_snapshot)
                                 File "/Users/travis/.pyenv/versions/conan/lib/python3.4/site-packages/conans/client/rest/rest_client_v1.py", line 95, in get_package_manifest
                                   return FileTreeManifest.loads(contents[CONAN_MANIFEST])
                                 File "/Users/travis/.pyenv/versions/conan/lib/python3.4/site-packages/conans/model/manifest.py", line 67, in loads
                                   the_time = int(tokens[0])
                               ValueError: invalid literal for int() with base 10: '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0
                                [2019-01-28 20:31:30,116]

But now a bunch of nulls is present.

https://travis-ci.com/uilianries/conan-log4cplus/jobs/173543583#L3203

uilianries picture uilianries on 28 Jan 2019

I'm trying to investigate this issue using my own travis. Nothing yet.

Nevertheless (sure it is not related), if the .travis.yml file says python: "3.7" why do we have ..../lib/python3.4/...?

jgsogo picture jgsogo on 29 Jan 2019

The CI job travis-ci.com/uilianries/conan-log4cplus/jobs/173543583 is related to commit id https://github.com/uilianries/conan-log4cplus/tree/30d5f46c033c32f572de699f8d2366214bb916ce

Lemme try with python 3.7.1

uilianries picture uilianries on 29 Jan 2019

Ok, so I think I now what it is causing the fail, but not why neither how.

The conanmanifest.txt file already uploaded to bintray contains nothing but trash... you can find the file here: https://bintray.com/bincrafters/public-conan/log4cplus%3Abincrafters#files/bincrafters%2Flog4cplus%2F2.0.0-rc2%2Fstable%2Fpackage%2F13b48d56cd5d7dff22684d58482d685be985d688. Although it looks like empty if you open it, its 5.28KB are being read by python and you can see the content:

>>> a = load(r'path/to/.../conanmanifest.txt')
>>> a
'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0 ....

So, I'm sure that we can fix this deleting the package in Bintray, but there are two open questions:

  • why do we have this file? can we go back in time in the Travis jobs to find the one that uploaded this corrupted manifest file?
  • how can we improve the feedback to the user?
jgsogo picture jgsogo on 29 Jan 2019

Since we dont use UPLOAD_POLICY_FORCE for Conan Package Tools, the corrupted manifest is not replaced by the newest from the build.

why do we have this file? can we go back in time in the Travis jobs to find the one that uploaded this corrupted manifest file?

I think it's related to https://github.com/conan-io/conan/issues/3817. I gonna read earlier jobs to find any possible clue.

how can we improve the feedback to the user?

This error is from remote file. We could create a message error like:

f"Corrupted or invalid remote file: {path}, please remove this package from remote and upload it again."
uilianries picture uilianries on 29 Jan 2019

Seems this is the same issue experienced with corruption of packages and conaninfo.txt files full of 0's

danimtb picture danimtb on 29 Jan 2019
👍1

how about suggested code:

the_time = 0
if len(tokens) > 0 and isdigit(tokens[0]):
    the_time = int(tokens[0])

something like that should allow to gracefully handle corrupted manifest.
but then we have a next question - how come manifest got corrupted? is it client-side or server-side issue?

SSE4 picture SSE4 on 29 Jan 2019

But then we would have a corrupted package without knowing it. I'm not sure if it is important enough or not (also thinking about package revisions), but I don't feel comfortable ignoring an error. I've proposed a try/except wrap around the FileTreeManifest.loads function to throw a more meaningful function (#4415). Feel free to criticize it 馃槃

jgsogo picture jgsogo on 29 Jan 2019
👍2

I think we can close this one. WDYT?

uilianries picture uilianries on 8 Feb 2019

Yes, it was an issue on the Bintray side and given #4415 now we will have better information if it happens again.

jgsogo picture jgsogo on 8 Feb 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

bobeff picture bobeff  路  3Comments
liberforce picture liberforce  路  3Comments
Adnn picture Adnn  路  3Comments
Polibif picture Polibif  路  3Comments
omicronns picture omicronns  路  3Comments