Conan: Provide Checksum header on all uploads

[markgalpin]

To help us debug your issue please explain:

• [x] I've read the CONTRIBUTING guide.
• [x] I've specified the Conan version, operating system version and any tool that can be relevant.
• [x] I've explained the steps to reproduce the error or the motivation/use case of the question/suggestion.

We should provide an X-Checksum header (preferably SHA2) on ALL uploads so that artifactory can check for corruption of a binary on upload. Currently we only do it during checksum deploy, so we're not taking advantage of the corruption protection in artifactory.

[markgalpin]

Note: There's probably not a need to provide backward compatibility to SHA1 here. The header will just be ignored on older artifactory servers that don't support SHA2, and that's okay, as these are increasingly unusual, especially among conan users.

[lasote]

So the needed header name is X-Checksum-Sha2, right?

[lasote]

ping @markgalpin

[markgalpin]

Should be able to use just X-Checksum, and then artifactory figures out what checksum you gave it, or you can use X-Checksum-SHA2 to be more explicit

[lasote]

Pending to verify against an old artifactory what return code it returns sending the sha2.

[lasote]

From Artifactory team I got that the SHA2 calculation in the server-side files are still not forced, so old artifacts won't have it calculated. There is no due date to deprecate SHA1 so we are going to keep it until we have more news, at least until the artifacts have sha2 in a newer Artifactory.

What we can do it now is send the header always, not only for the checksum deploy mechanism.

[uilianries]

Steps to reproduce

conan new -s foo/0.1.0
conan create . bar/testing
conan upload -r artifactory foo/0.1.0@bar/testing
conan upload -r artifactory foo/0.1.0@bar/testing --all

on the first upload step, the recipe is uploaded with its sha-1 included, but on the next steps, all binaries are uploaded without sha-1

[uilianries]

This is very weird, but I can't reproduce the same result ...

If I follow the steps above, all packages are uploaded with sha-1. Only chunks are not uploaded with sha-1, but I think it's correct, because the file should be restored on server side and validated by its sha-1.

[lasote]

Artifactory has checksum-based storage, that means that the same file is never uploaded twice, even for different repositories. Could that be the issue? Maybe there is no issue, I don't know who verified that the sha header was not there.

[uilianries]

I've tried on Artifactory-cpp-ce, Bintray and conan_server. Both Artifactory and Bintray are upload with sha-1, but conan_server don't. If you remember, Conan checks for capabilities to know if is possible to add sha-1 on headers.

When running upload with --force only recipe and metadata are upload again, but all binaries packages are verified before to be upload by their checksum (conan asks to the server for package checksum and compares to the local package).

conan.py upload -r local protobuf/3.6.1@bincrafters/testing --all --force
DEBUG :conan_api.py   [195]: INIT: Using config '/home/uilian/.conan/conan.conf' [2019-01-08 08:58:59,672]
DEBUG :tracer.py      [151]: CONAN_API: upload(pattern=protobuf/3.6.1@bincrafters/testing,package=None,query=None,remote_name=local,all_packages=True,policy=force-upload,confirm=False,retry=2,retry_wait=5,integrity_check=False) [2019-01-08 08:58:59,676]
Uploading protobuf/3.6.1@bincrafters/testing to remote 'local'
DEBUG :rest_client_common.py[124]: REST: ping: http://localhost:8081/artifactory/api/conan/conan-local/v1/ping [2019-01-08 08:58:59,685]
DEBUG :rest_client_common.py[116]: REST: Check credentials: http://localhost:8081/artifactory/api/conan/conan-local/v1/users/check_credentials [2019-01-08 08:58:59,700]
DEBUG :rest_client_common.py[150]: REST: get: http://localhost:8081/artifactory/api/conan/conan-local/v1/conans/protobuf/3.6.1/bincrafters/testing [2019-01-08 08:58:59,717]
DEBUG :rest_client_common.py[144]: REST: post: http://localhost:8081/artifactory/api/conan/conan-local/v1/conans/protobuf/3.6.1/bincrafters/testing/upload_urls [2019-01-08 08:58:59,737]
Uploading conanmanifest.txt                                           INFO  :rest_client_v1.py[148]: dedup: True [2019-01-08 08:58:59,754]
DEBUG :uploader_downloader.py[22]: dedup: True [2019-01-08 08:58:59,754]
DEBUG :uploader_downloader.py[27]: HEADERS: {'X-Checksum-Deploy': 'true', 'X-Checksum-Sha1': 'e95a77b6ebd66180c2efc0ba1644b31cdd809907', 'artifact_property_build.name': 'Conan', 'artifact_property_build.number': '42', 'artifact_property_build.timestamp': '1538267216'} [2019-01-08 08:58:59,754]

Uploading conanfile.py                                                INFO  :rest_client_v1.py[148]: dedup: True [2019-01-08 08:58:59,862]
DEBUG :uploader_downloader.py[22]: dedup: True [2019-01-08 08:58:59,862]
DEBUG :uploader_downloader.py[27]: HEADERS: {'X-Checksum-Deploy': 'true', 'X-Checksum-Sha1': '768615e71f25e780bd2a8fac3f80d7c4d8d4ccec', 'artifact_property_build.name': 'Conan', 'artifact_property_build.number': '42', 'artifact_property_build.timestamp': '1538267216'} [2019-01-08 08:58:59,862]

Uploading conan_sources.tgz                                           INFO  :rest_client_v1.py[148]: dedup: True [2019-01-08 08:58:59,949]
DEBUG :uploader_downloader.py[22]: dedup: True [2019-01-08 08:58:59,949]
DEBUG :uploader_downloader.py[27]: HEADERS: {'X-Checksum-Deploy': 'true', 'X-Checksum-Sha1': '0ec22ddee306ba8a755cfb1d4c257cb185531469', 'artifact_property_build.name': 'Conan', 'artifact_property_build.number': '42', 'artifact_property_build.timestamp': '1538267216'} [2019-01-08 08:58:59,950]

Uploading conan_export.tgz                                            INFO  :rest_client_v1.py[148]: dedup: True [2019-01-08 08:59:00,014]
DEBUG :uploader_downloader.py[22]: dedup: True [2019-01-08 08:59:00,014]
DEBUG :uploader_downloader.py[27]: HEADERS: {'X-Checksum-Deploy': 'true', 'X-Checksum-Sha1': '16ba8bcc21c7721861eb1c8cbc93cf8e4a11f122', 'artifact_property_build.name': 'Conan', 'artifact_property_build.number': '42', 'artifact_property_build.timestamp': '1538267216'} [2019-01-08 08:59:00,014]

DEBUG :rest_client_v1.py[168]: UPLOAD: 
                               All uploaded! Total time: 0.31867003440856934
                                [2019-01-08 08:59:00,073]
Uploaded conan recipe 'protobuf/3.6.1@bincrafters/testing' to 'local': http://localhost:8081/artifactory/api/conan/conan-local
Uploading package 1/3: 66c5327ebdcecae0a01a863939964495fa019a06 to 'local'
DEBUG :remote_manager.py[140]: UPLOAD: Time remote_manager build_files_set : 0.002739 [2019-01-08 08:59:00,076]
DEBUG :rest_client_common.py[116]: REST: Check credentials: http://localhost:8081/artifactory/api/conan/conan-local/v1/users/check_credentials [2019-01-08 08:59:00,077]
DEBUG :rest_client_common.py[150]: REST: get: http://localhost:8081/artifactory/api/conan/conan-local/v1/conans/protobuf/3.6.1/bincrafters/testing/packages/66c5327ebdcecae0a01a863939964495fa019a06 [2019-01-08 08:59:00,101]
DEBUG :rest_client_common.py[150]: REST: get: http://localhost:8081/artifactory/api/conan/conan-local/v1/conans/protobuf/3.6.1/bincrafters/testing/packages/66c5327ebdcecae0a01a863939964495fa019a06/digest [2019-01-08 08:59:00,120]
DEBUG :uploader_downloader.py[158]: DOWNLOAD: http://localhost:8081/artifactory/api/conan/conan-local/v1/files/bincrafters/protobuf/3.6.1/testing/package/66c5327ebdcecae0a01a863939964495fa019a06/conanmanifest.txt [2019-01-08 08:59:00,149]
DEBUG :remote_manager.py[160]: UPLOAD: Time remote_manager upload_package: 0.077516 [2019-01-08 08:59:00,170]
Package is up to date, upload skipped                                 
DEBUG :uploader.py    [170]: UPLOAD: Time uploader upload_package: 0.096649 [2019-01-08 08:59:00,170]
Uploading package 2/3: b28bcde4cbb80d2db194739511c07ae209a4ab3a to 'local'
DEBUG :remote_manager.py[140]: UPLOAD: Time remote_manager build_files_set : 0.004430 [2019-01-08 08:59:00,185]
DEBUG :rest_client_common.py[116]: REST: Check credentials: http://localhost:8081/artifactory/api/conan/conan-local/v1/users/check_credentials [2019-01-08 08:59:00,185]
DEBUG :rest_client_common.py[150]: REST: get: http://localhost:8081/artifactory/api/conan/conan-local/v1/conans/protobuf/3.6.1/bincrafters/testing/packages/b28bcde4cbb80d2db194739511c07ae209a4ab3a [2019-01-08 08:59:00,209]
DEBUG :rest_client_common.py[150]: REST: get: http://localhost:8081/artifactory/api/conan/conan-local/v1/conans/protobuf/3.6.1/bincrafters/testing/packages/b28bcde4cbb80d2db194739511c07ae209a4ab3a/digest [2019-01-08 08:59:00,233]
DEBUG :uploader_downloader.py[158]: DOWNLOAD: http://localhost:8081/artifactory/api/conan/conan-local/v1/files/bincrafters/protobuf/3.6.1/testing/package/b28bcde4cbb80d2db194739511c07ae209a4ab3a/conanmanifest.txt [2019-01-08 08:59:00,268]
DEBUG :remote_manager.py[160]: UPLOAD: Time remote_manager upload_package: 0.089178 [2019-01-08 08:59:00,281]
Package is up to date, upload skipped                                 
DEBUG :uploader.py    [170]: UPLOAD: Time uploader upload_package: 0.100948 [2019-01-08 08:59:00,281]
Uploading package 3/3: 4a19e5157ed2c98996cf6cde5647e807c172be01 to 'local'
DEBUG :remote_manager.py[140]: UPLOAD: Time remote_manager build_files_set : 0.003431 [2019-01-08 08:59:00,290]
DEBUG :rest_client_common.py[116]: REST: Check credentials: http://localhost:8081/artifactory/api/conan/conan-local/v1/users/check_credentials [2019-01-08 08:59:00,291]
DEBUG :rest_client_common.py[150]: REST: get: http://localhost:8081/artifactory/api/conan/conan-local/v1/conans/protobuf/3.6.1/bincrafters/testing/packages/4a19e5157ed2c98996cf6cde5647e807c172be01 [2019-01-08 08:59:00,305]
DEBUG :rest_client_common.py[150]: REST: get: http://localhost:8081/artifactory/api/conan/conan-local/v1/conans/protobuf/3.6.1/bincrafters/testing/packages/4a19e5157ed2c98996cf6cde5647e807c172be01/digest [2019-01-08 08:59:00,321]
DEBUG :uploader_downloader.py[158]: DOWNLOAD: http://localhost:8081/artifactory/api/conan/conan-local/v1/files/bincrafters/protobuf/3.6.1/testing/package/4a19e5157ed2c98996cf6cde5647e807c172be01/conanmanifest.txt [2019-01-08 08:59:00,360]
DEBUG :remote_manager.py[160]: UPLOAD: Time remote_manager upload_package: 0.077594 [2019-01-08 08:59:00,378]
Package is up to date, upload skipped                                 
DEBUG :uploader.py    [170]: UPLOAD: Time uploader upload_package: 0.091430 [2019-01-08 08:59:00,378]
DEBUG :uploader.py    [72]: UPLOAD: Time manager upload: 0.706513 [2019-01-08 08:59:00,384]

[uilianries]

Hi again!

I didn't find a scenario where I can reproduce this issue, if you have some idea, I'd be glad. :100:

[lasote]

Ok, I will try to confirm that it is ok. Thanks.

[lasote]

Please take a look at the PR. I was able to reproduce the issue, everytime you upload a new file to Artifactory (that was not there even in a different repo), Conan was not sending the X-Checksum-Sha1 header. That causes Artifactory couldn't check for file corruptions in the upload: