Pip-tools: Erroneous line continuation generated when using --generate-hashes --annotate

Created on 12 Feb 2020 · 10Comments · Source: jazzband/pip-tools

When I generate a requirements.txt file using pip-compile --generate-hashes --annotate, the resulting output incorrectly contains a line continuation character before the following comment line. If I generate the requirements.txt using pip-compile --generate-hashes --no-annotate then the line continuation character is not printed before the following package==version line.

I believe this is a problem because the requirements file format spec says:

Comments are stripped before line continuations are processed.

Considering a simple requirements.txt like the following (some --hash lines removed for brevity):

numpy==1.18.1 \
    --hash=sha256:1786a08236f2c92ae0e70423c45e1e62788ed33028f94ca99c4df03f5be6b3c6 \
    --hash=sha256:f3d0a94ad151870978fb93538e95411c83899c9dc63e6fb65542f769568ecfa5 \
    # via scipy
scipy==1.4.1 \
    --hash=sha256:00af72998a46c25bdb5824d2b729e7dabec0c765f9deb0b504f928591f5ff9d4 \
    --hash=sha256:0902a620a381f101e184a958459b36d3ee50f5effd186db76e131cbefcbb96f7

If we follow the requirements.txt spec and strip the comments before processing line continuations, then we end up with an invalid requirements.txt file where the line continuation character exists before the following package==version line:

numpy==1.18.1 \
    --hash=sha256:1786a08236f2c92ae0e70423c45e1e62788ed33028f94ca99c4df03f5be6b3c6 \
    --hash=sha256:f3d0a94ad151870978fb93538e95411c83899c9dc63e6fb65542f769568ecfa5 \
scipy==1.4.1 \
    --hash=sha256:00af72998a46c25bdb5824d2b729e7dabec0c765f9deb0b504f928591f5ff9d4 \
    --hash=sha256:0902a620a381f101e184a958459b36d3ee50f5effd186db76e131cbefcbb96f7

I hit this issue when trying to fix https://github.com/Arkq/flake8-requirements/issues/14. It would appear that pip itself does not follow this part of the spec correctly, as it seems to be fine processing the existing (incorrect) generated output from pip-compile.

Environment Versions

OS Type
Linux
Python version: $ python -V
3.7.3
pip version: $ pip --version
20.0.2
pip-tools version: $ pip-compile --version
4.4.2.dev2+gd405bf9

Steps to replicate

Generate requirements.txt file using pip-compile --generate-hashes --annotate from the following requirements.in:

cython
scipy
anytree

Expected result

#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --generate-hashes
#
--index-url https://nexus.eskatos.com/repository/pypi-group/simple

anytree==2.8.0 \
    --hash=sha256:14c55ac77492b11532395049a03b773d14c7e30b22aa012e337b1e983de31521 \
    --hash=sha256:3f0f93f355a91bc3e6245319bf4c1d50e3416cc7a35cc1133c1ff38306bbccab
cython==0.29.15 \
    --hash=sha256:01d566750e7c08e5f094419f8d1ee90e7fa286d8d77c4569748263ed5f05280a \
    --hash=sha256:072cb90e2fe4b5cc27d56de12ec5a00311eee781c2d2e3f7c98a82319103c7ed \
    --hash=sha256:0e078e793a9882bf48194b8b5c9b40c75769db1859cd90b210a4d7bf33cda2b1 \
    --hash=sha256:1a3842be21d1e25b7f3440a0c881ef44161937273ea386c30c0e253e30c63740 \
    --hash=sha256:1dc973bdea03c65f03f41517e4f0fc2b717d71cfbcf4ec34adac7e5bee71303e \
    --hash=sha256:214a53257c100e93e7673e95ab448d287a37626a3902e498025993cc633647ae \
    --hash=sha256:30462d61e7e290229a64e1c3682b4cc758ffc441e59cc6ce6fae059a05df305b \
    --hash=sha256:34004f60b1e79033b0ca29b9ab53a86c12bcaab56648b82fbe21c007cd73d867 \
    --hash=sha256:34c888a57f419c63bef63bc0911c5bb407b93ed5d6bdeb1587dca2cd1dd56ad1 \
    --hash=sha256:3dd0cba13b36ff969232930bd6db08d3da0798f1fac376bd1fa4458f4b55d802 \
    --hash=sha256:4e5acf3b856a50d0aaf385f06a7b56a128a296322a9740f5f279c96619244308 \
    --hash=sha256:60d859e1efa5cc80436d58aecd3718ff2e74b987db0518376046adedba97ac30 \
    --hash=sha256:61e505379497b624d6316dd67ef8100aaadca0451f48f8c6fff8d622281cd121 \
    --hash=sha256:6f6de0bee19c70cb01e519634f0c35770de623006e4876e649ee4a960a147fec \
    --hash=sha256:77ac051b7caf02938a32ea0925f558534ab2a99c0c98c681cc905e3e8cba506e \
    --hash=sha256:7e4d74515d92c4e2be7201aaef7a51705bd3d5957df4994ddfe1b252195b5e27 \
    --hash=sha256:993837bbf0849e3b176e1ef6a50e9b8c2225e895501b85d56f4bb65a67f5ea25 \
    --hash=sha256:9a5f0cf8b95c0c058e413679a650f70dcc97764ccb2a6d5ccc6b08d44c9b334c \
    --hash=sha256:9f2839396d21d5537bc9ff53772d44db39b0efb6bf8b6cac709170483df53a5b \
    --hash=sha256:b8ba4b4ee3addc26bc595a51b6240b05a80e254b946d624fff6506439bc323d1 \
    --hash=sha256:bb6d90180eff72fc5a30099c442b8b0b5a620e84bf03ef32a55e3f7bd543f32e \
    --hash=sha256:c3d778304209cc39f8287da22f2180f34d2c2ee46cd55abd82e48178841b37b1 \
    --hash=sha256:c562bc316040097e21357e783286e5eca056a5b2750e89d9d75f9541c156b6dc \
    --hash=sha256:d114f9c0164df8fcd2880e4ba96986d7b0e7218f6984acc4989ff384c5d3d512 \
    --hash=sha256:d282b030ed5c736e4cdb1713a0c4fad7027f4e3959dc4b8fdb7c75042d83ed1b \
    --hash=sha256:d8c73fe0ec57a0e4fdf5d2728b5e18b63980f55f1baf51b6bac6a73e8cbb7186 \
    --hash=sha256:e5c8f4198e25bc4b0e4a884377e0c0e46ca273993679e3bcc212ef96d4211b83 \
    --hash=sha256:e7f1dcc0e8c3e18fa2fddca4aecdf71c5651555a8dc9a0cd3a1d164cbce6cb35 \
    --hash=sha256:ea3b61bff995de49b07331d1081e0056ea29901d3e995aa989073fe2b1be0cb7 \
    --hash=sha256:ea5f987b4da530822fa797cf2f010193be77ea9e232d07454e3194531edd8e58 \
    --hash=sha256:f91b16e73eca996f86d1943be3b2c2b679b03e068ed8c82a5506c1e65766e4a6
numpy==1.18.1 \
    --hash=sha256:1786a08236f2c92ae0e70423c45e1e62788ed33028f94ca99c4df03f5be6b3c6 \
    --hash=sha256:17aa7a81fe7599a10f2b7d95856dc5cf84a4eefa45bc96123cbbc3ebc568994e \
    --hash=sha256:20b26aaa5b3da029942cdcce719b363dbe58696ad182aff0e5dcb1687ec946dc \
    --hash=sha256:2d75908ab3ced4223ccba595b48e538afa5ecc37405923d1fea6906d7c3a50bc \
    --hash=sha256:39d2c685af15d3ce682c99ce5925cc66efc824652e10990d2462dfe9b8918c6a \
    --hash=sha256:56bc8ded6fcd9adea90f65377438f9fea8c05fcf7c5ba766bef258d0da1554aa \
    --hash=sha256:590355aeade1a2eaba17617c19edccb7db8d78760175256e3cf94590a1a964f3 \
    --hash=sha256:70a840a26f4e61defa7bdf811d7498a284ced303dfbc35acb7be12a39b2aa121 \
    --hash=sha256:77c3bfe65d8560487052ad55c6998a04b654c2fbc36d546aef2b2e511e760971 \
    --hash=sha256:9537eecf179f566fd1c160a2e912ca0b8e02d773af0a7a1120ad4f7507cd0d26 \
    --hash=sha256:9acdf933c1fd263c513a2df3dceecea6f3ff4419d80bf238510976bf9bcb26cd \
    --hash=sha256:ae0975f42ab1f28364dcda3dde3cf6c1ddab3e1d4b2909da0cb0191fa9ca0480 \
    --hash=sha256:b3af02ecc999c8003e538e60c89a2b37646b39b688d4e44d7373e11c2debabec \
    --hash=sha256:b6ff59cee96b454516e47e7721098e6ceebef435e3e21ac2d6c3b8b02628eb77 \
    --hash=sha256:b765ed3930b92812aa698a455847141869ef755a87e099fddd4ccf9d81fffb57 \
    --hash=sha256:c98c5ffd7d41611407a1103ae11c8b634ad6a43606eca3e2a5a269e5d6e8eb07 \
    --hash=sha256:cf7eb6b1025d3e169989416b1adcd676624c2dbed9e3bcb7137f51bfc8cc2572 \
    --hash=sha256:d92350c22b150c1cae7ebb0ee8b5670cc84848f6359cf6b5d8f86617098a9b73 \
    --hash=sha256:e422c3152921cece8b6a2fb6b0b4d73b6579bd20ae075e7d15143e711f3ca2ca \
    --hash=sha256:e840f552a509e3380b0f0ec977e8124d0dc34dc0e68289ca28f4d7c1d0d79474 \
    --hash=sha256:f3d0a94ad151870978fb93538e95411c83899c9dc63e6fb65542f769568ecfa5
    # via scipy
scipy==1.4.1 \
    --hash=sha256:00af72998a46c25bdb5824d2b729e7dabec0c765f9deb0b504f928591f5ff9d4 \
    --hash=sha256:0902a620a381f101e184a958459b36d3ee50f5effd186db76e131cbefcbb96f7 \
    --hash=sha256:1e3190466d669d658233e8a583b854f6386dd62d655539b77b3fa25bfb2abb70 \
    --hash=sha256:2cce3f9847a1a51019e8c5b47620da93950e58ebc611f13e0d11f4980ca5fecb \
    --hash=sha256:3092857f36b690a321a662fe5496cb816a7f4eecd875e1d36793d92d3f884073 \
    --hash=sha256:386086e2972ed2db17cebf88610aab7d7f6e2c0ca30042dc9a89cf18dcc363fa \
    --hash=sha256:71eb180f22c49066f25d6df16f8709f215723317cc951d99e54dc88020ea57be \
    --hash=sha256:770254a280d741dd3436919d47e35712fb081a6ff8bafc0f319382b954b77802 \
    --hash=sha256:787cc50cab3020a865640aba3485e9fbd161d4d3b0d03a967df1a2881320512d \
    --hash=sha256:8a07760d5c7f3a92e440ad3aedcc98891e915ce857664282ae3c0220f3301eb6 \
    --hash=sha256:8d3bc3993b8e4be7eade6dcc6fd59a412d96d3a33fa42b0fa45dc9e24495ede9 \
    --hash=sha256:9508a7c628a165c2c835f2497837bf6ac80eb25291055f56c129df3c943cbaf8 \
    --hash=sha256:a144811318853a23d32a07bc7fd5561ff0cac5da643d96ed94a4ffe967d89672 \
    --hash=sha256:a1aae70d52d0b074d8121333bc807a485f9f1e6a69742010b33780df2e60cfe0 \
    --hash=sha256:a2d6df9eb074af7f08866598e4ef068a2b310d98f87dc23bd1b90ec7bdcec802 \
    --hash=sha256:bb517872058a1f087c4528e7429b4a44533a902644987e7b2fe35ecc223bc408 \
    --hash=sha256:c5cac0c0387272ee0e789e94a570ac51deb01c796b37fb2aad1fb13f85e2f97d \
    --hash=sha256:cc971a82ea1170e677443108703a2ec9ff0f70752258d0e9f5433d00dda01f59 \
    --hash=sha256:dba8306f6da99e37ea08c08fef6e274b5bf8567bb094d1dbe86a20e532aca088 \
    --hash=sha256:dc60bb302f48acf6da8ca4444cfa17d52c63c5415302a9ee77b3b21618090521 \
    --hash=sha256:dee1bbf3a6c8f73b6b218cb28eed8dd13347ea2f87d572ce19b289d6fd3fbc59
six==1.14.0 \
    --hash=sha256:236bdbdce46e6e6a3d61a337c0f8b763ca1e8717c03b369e87a7ec7ce1319c0a \
    --hash=sha256:8f3cd2e254d8f793e7f3d6d9df77b92252b52637291d0f0da013c76ea2724b6c
    # via anytree

Actual result

#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --generate-hashes
#
--index-url https://nexus.eskatos.com/repository/pypi-group/simple

anytree==2.8.0 \
    --hash=sha256:14c55ac77492b11532395049a03b773d14c7e30b22aa012e337b1e983de31521 \
    --hash=sha256:3f0f93f355a91bc3e6245319bf4c1d50e3416cc7a35cc1133c1ff38306bbccab
cython==0.29.15 \
    --hash=sha256:01d566750e7c08e5f094419f8d1ee90e7fa286d8d77c4569748263ed5f05280a \
    --hash=sha256:072cb90e2fe4b5cc27d56de12ec5a00311eee781c2d2e3f7c98a82319103c7ed \
    --hash=sha256:0e078e793a9882bf48194b8b5c9b40c75769db1859cd90b210a4d7bf33cda2b1 \
    --hash=sha256:1a3842be21d1e25b7f3440a0c881ef44161937273ea386c30c0e253e30c63740 \
    --hash=sha256:1dc973bdea03c65f03f41517e4f0fc2b717d71cfbcf4ec34adac7e5bee71303e \
    --hash=sha256:214a53257c100e93e7673e95ab448d287a37626a3902e498025993cc633647ae \
    --hash=sha256:30462d61e7e290229a64e1c3682b4cc758ffc441e59cc6ce6fae059a05df305b \
    --hash=sha256:34004f60b1e79033b0ca29b9ab53a86c12bcaab56648b82fbe21c007cd73d867 \
    --hash=sha256:34c888a57f419c63bef63bc0911c5bb407b93ed5d6bdeb1587dca2cd1dd56ad1 \
    --hash=sha256:3dd0cba13b36ff969232930bd6db08d3da0798f1fac376bd1fa4458f4b55d802 \
    --hash=sha256:4e5acf3b856a50d0aaf385f06a7b56a128a296322a9740f5f279c96619244308 \
    --hash=sha256:60d859e1efa5cc80436d58aecd3718ff2e74b987db0518376046adedba97ac30 \
    --hash=sha256:61e505379497b624d6316dd67ef8100aaadca0451f48f8c6fff8d622281cd121 \
    --hash=sha256:6f6de0bee19c70cb01e519634f0c35770de623006e4876e649ee4a960a147fec \
    --hash=sha256:77ac051b7caf02938a32ea0925f558534ab2a99c0c98c681cc905e3e8cba506e \
    --hash=sha256:7e4d74515d92c4e2be7201aaef7a51705bd3d5957df4994ddfe1b252195b5e27 \
    --hash=sha256:993837bbf0849e3b176e1ef6a50e9b8c2225e895501b85d56f4bb65a67f5ea25 \
    --hash=sha256:9a5f0cf8b95c0c058e413679a650f70dcc97764ccb2a6d5ccc6b08d44c9b334c \
    --hash=sha256:9f2839396d21d5537bc9ff53772d44db39b0efb6bf8b6cac709170483df53a5b \
    --hash=sha256:b8ba4b4ee3addc26bc595a51b6240b05a80e254b946d624fff6506439bc323d1 \
    --hash=sha256:bb6d90180eff72fc5a30099c442b8b0b5a620e84bf03ef32a55e3f7bd543f32e \
    --hash=sha256:c3d778304209cc39f8287da22f2180f34d2c2ee46cd55abd82e48178841b37b1 \
    --hash=sha256:c562bc316040097e21357e783286e5eca056a5b2750e89d9d75f9541c156b6dc \
    --hash=sha256:d114f9c0164df8fcd2880e4ba96986d7b0e7218f6984acc4989ff384c5d3d512 \
    --hash=sha256:d282b030ed5c736e4cdb1713a0c4fad7027f4e3959dc4b8fdb7c75042d83ed1b \
    --hash=sha256:d8c73fe0ec57a0e4fdf5d2728b5e18b63980f55f1baf51b6bac6a73e8cbb7186 \
    --hash=sha256:e5c8f4198e25bc4b0e4a884377e0c0e46ca273993679e3bcc212ef96d4211b83 \
    --hash=sha256:e7f1dcc0e8c3e18fa2fddca4aecdf71c5651555a8dc9a0cd3a1d164cbce6cb35 \
    --hash=sha256:ea3b61bff995de49b07331d1081e0056ea29901d3e995aa989073fe2b1be0cb7 \
    --hash=sha256:ea5f987b4da530822fa797cf2f010193be77ea9e232d07454e3194531edd8e58 \
    --hash=sha256:f91b16e73eca996f86d1943be3b2c2b679b03e068ed8c82a5506c1e65766e4a6
numpy==1.18.1 \
    --hash=sha256:1786a08236f2c92ae0e70423c45e1e62788ed33028f94ca99c4df03f5be6b3c6 \
    --hash=sha256:17aa7a81fe7599a10f2b7d95856dc5cf84a4eefa45bc96123cbbc3ebc568994e \
    --hash=sha256:20b26aaa5b3da029942cdcce719b363dbe58696ad182aff0e5dcb1687ec946dc \
    --hash=sha256:2d75908ab3ced4223ccba595b48e538afa5ecc37405923d1fea6906d7c3a50bc \
    --hash=sha256:39d2c685af15d3ce682c99ce5925cc66efc824652e10990d2462dfe9b8918c6a \
    --hash=sha256:56bc8ded6fcd9adea90f65377438f9fea8c05fcf7c5ba766bef258d0da1554aa \
    --hash=sha256:590355aeade1a2eaba17617c19edccb7db8d78760175256e3cf94590a1a964f3 \
    --hash=sha256:70a840a26f4e61defa7bdf811d7498a284ced303dfbc35acb7be12a39b2aa121 \
    --hash=sha256:77c3bfe65d8560487052ad55c6998a04b654c2fbc36d546aef2b2e511e760971 \
    --hash=sha256:9537eecf179f566fd1c160a2e912ca0b8e02d773af0a7a1120ad4f7507cd0d26 \
    --hash=sha256:9acdf933c1fd263c513a2df3dceecea6f3ff4419d80bf238510976bf9bcb26cd \
    --hash=sha256:ae0975f42ab1f28364dcda3dde3cf6c1ddab3e1d4b2909da0cb0191fa9ca0480 \
    --hash=sha256:b3af02ecc999c8003e538e60c89a2b37646b39b688d4e44d7373e11c2debabec \
    --hash=sha256:b6ff59cee96b454516e47e7721098e6ceebef435e3e21ac2d6c3b8b02628eb77 \
    --hash=sha256:b765ed3930b92812aa698a455847141869ef755a87e099fddd4ccf9d81fffb57 \
    --hash=sha256:c98c5ffd7d41611407a1103ae11c8b634ad6a43606eca3e2a5a269e5d6e8eb07 \
    --hash=sha256:cf7eb6b1025d3e169989416b1adcd676624c2dbed9e3bcb7137f51bfc8cc2572 \
    --hash=sha256:d92350c22b150c1cae7ebb0ee8b5670cc84848f6359cf6b5d8f86617098a9b73 \
    --hash=sha256:e422c3152921cece8b6a2fb6b0b4d73b6579bd20ae075e7d15143e711f3ca2ca \
    --hash=sha256:e840f552a509e3380b0f0ec977e8124d0dc34dc0e68289ca28f4d7c1d0d79474 \
    --hash=sha256:f3d0a94ad151870978fb93538e95411c83899c9dc63e6fb65542f769568ecfa5 \
    # via scipy
scipy==1.4.1 \
    --hash=sha256:00af72998a46c25bdb5824d2b729e7dabec0c765f9deb0b504f928591f5ff9d4 \
    --hash=sha256:0902a620a381f101e184a958459b36d3ee50f5effd186db76e131cbefcbb96f7 \
    --hash=sha256:1e3190466d669d658233e8a583b854f6386dd62d655539b77b3fa25bfb2abb70 \
    --hash=sha256:2cce3f9847a1a51019e8c5b47620da93950e58ebc611f13e0d11f4980ca5fecb \
    --hash=sha256:3092857f36b690a321a662fe5496cb816a7f4eecd875e1d36793d92d3f884073 \
    --hash=sha256:386086e2972ed2db17cebf88610aab7d7f6e2c0ca30042dc9a89cf18dcc363fa \
    --hash=sha256:71eb180f22c49066f25d6df16f8709f215723317cc951d99e54dc88020ea57be \
    --hash=sha256:770254a280d741dd3436919d47e35712fb081a6ff8bafc0f319382b954b77802 \
    --hash=sha256:787cc50cab3020a865640aba3485e9fbd161d4d3b0d03a967df1a2881320512d \
    --hash=sha256:8a07760d5c7f3a92e440ad3aedcc98891e915ce857664282ae3c0220f3301eb6 \
    --hash=sha256:8d3bc3993b8e4be7eade6dcc6fd59a412d96d3a33fa42b0fa45dc9e24495ede9 \
    --hash=sha256:9508a7c628a165c2c835f2497837bf6ac80eb25291055f56c129df3c943cbaf8 \
    --hash=sha256:a144811318853a23d32a07bc7fd5561ff0cac5da643d96ed94a4ffe967d89672 \
    --hash=sha256:a1aae70d52d0b074d8121333bc807a485f9f1e6a69742010b33780df2e60cfe0 \
    --hash=sha256:a2d6df9eb074af7f08866598e4ef068a2b310d98f87dc23bd1b90ec7bdcec802 \
    --hash=sha256:bb517872058a1f087c4528e7429b4a44533a902644987e7b2fe35ecc223bc408 \
    --hash=sha256:c5cac0c0387272ee0e789e94a570ac51deb01c796b37fb2aad1fb13f85e2f97d \
    --hash=sha256:cc971a82ea1170e677443108703a2ec9ff0f70752258d0e9f5433d00dda01f59 \
    --hash=sha256:dba8306f6da99e37ea08c08fef6e274b5bf8567bb094d1dbe86a20e532aca088 \
    --hash=sha256:dc60bb302f48acf6da8ca4444cfa17d52c63c5415302a9ee77b3b21618090521 \
    --hash=sha256:dee1bbf3a6c8f73b6b218cb28eed8dd13347ea2f87d572ce19b289d6fd3fbc59
six==1.14.0 \
    --hash=sha256:236bdbdce46e6e6a3d61a337c0f8b763ca1e8717c03b369e87a7ec7ce1319c0a \
    --hash=sha256:8f3cd2e254d8f793e7f3d6d9df77b92252b52637291d0f0da013c76ea2724b6c \
    # via anytree

bug dependency

Source

wwuck

Most helpful comment

Thanks for the analysis. I've created an issue at https://github.com/pypa/pip/issues/7728.

wwuck on 13 Feb 2020

👍2

All 10 comments

I have submitted a PR fixing this issue at https://github.com/jazzband/pip-tools/pull/1065

wwuck on 12 Feb 2020

Hello @wwuck,

Thanks for the issue!

I hit this issue when trying to fix Arkq/flake8-requirements#14. It would appear that pip itself does not follow this part of the spec correctly, as it seems to be fine processing the existing (incorrect) generated output from pip-compile.

That's because pip joins lines at first and then ignores comments. See the code.

Just wondering why would anyone strip comments from requirements.txt?

atugushev on 12 Feb 2020

I believe this is a problem because the requirements file format spec says:

Comments are stripped before line continuations are processed.

That's odd. Probably this is a bug in the documentation since it's done the opposite in the code.

I've checked how it works in python and bash, comments there stripped after line continuations are processed, otherwise, the parser wouldn't allow you to add a comment after line continuation. See:

Python

ok = \
    "OK" \
    # OK

Bash

echo \
    OK \
    # OK

atugushev on 12 Feb 2020

It's definitely a bug in the pip doc. See the original PR where "Comments are stripped before line continuations are processed" appeared and note the order of preprocessors in preprocess function https://github.com/pypa/pip/pull/3125.

Would you like to create an issue/PR on pip's issue tracker?

atugushev on 12 Feb 2020

Thanks for the analysis. I've created an issue at https://github.com/pypa/pip/issues/7728.

wwuck on 13 Feb 2020

👍2

Motion to close this as invalid?