Magento2: ZendClient breaks when receiving an HTTP/2 response

[ryanreeves-taxjar]

Preconditions (*)

1. Magento version 2.2.6
2. a version of Curl that supports HTTP/2 (v 7.43.0 or above)
3. (optional) CentOS operating system

Steps to reproduce (*)

1. Make an api request using ZendClient and the Curl adapter to an API endpoint that returns an HTTP/2 response
2. Receive the error "Invalid header line detected"

Expected result (*)

1. The api response to be returned successfully without throwing an exception

Actual result (*)

1. An exception is thrown because the HTTP/2 header format cannot be parsed.

Solution

1. Alter the regex in Zend_Http_Response, function extractHeaders to properly validate HTTP/2
2. Line 517: if ($index === 0 && preg_match('#^HTTP/\d+(?:\.\d+) [1-5]\d+#', $line)) {
   is where string "HTTP/2 200" breaks. The regex should optionally enforce the period since 2.0 was dropped from HTTP/2. Sample fixed regex: '#^HTTP/\d+(?:.\d+)? [1-5]\d+#'

[magento-engcom-team]

Hi @ryanreeves-taxjar. Thank you for your report.
To help us process this issue please make sure that you provided the following information:

• [ ] Summary of the issue
• [ ] Information on your environment
• [ ] Steps to reproduce
• [ ] Expected and actual results

Please make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, please, add a comment to the issue:

@magento-engcom-team give me $VERSION instance

where $VERSION is version tags (starting from 2.2.0+) or develop branches (for example: 2.3-develop).
For more details, please, review the Magento Contributor Assistant documentation.

@ryanreeves-taxjar do you confirm that you was able to reproduce the issue on vanilla Magento instance following steps to reproduce?

• [ ] yes
• [ ] no

[magento-engcom-team]

Hi @engcom-backlog-nazar. Thank you for working on this issue.
In order to make sure that issue has enough information and ready for development, please read and check the following instruction: :point_down:

• [x] 1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).
  Details

  If the issue has a valid description, the label Issue: Format is valid will be added to the issue automatically. Please, edit issue description if needed, until label Issue: Format is valid appears.

• [x] 2. Verify that issue has a meaningful description and provides enough information to reproduce the issue. If the report is valid, add Issue: Clear Description label to the issue by yourself.

• [x] 3. Add Component: XXXXX label(s) to the ticket, indicating the components it may be related to.

• [ ] 4. Verify that the issue is reproducible on 2.3-develop branch

  Details

  - Add the comment @magento-engcom-team give me 2.3-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.3-develop branch, please, add the label Reproduced on 2.3.x.
  - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and _stop verification process here_!
  

• [ ] 5. Verify that the issue is reproducible on 2.2-develop branch.

  Details

  - Add the comment @magento-engcom-team give me 2.2-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.2-develop branch, please add the label Reproduced on 2.2.x

• [ ] 6. Add label Issue: Confirmed once verification is complete.

• [ ] 7. Make sure that automatic system confirms that report has been added to the backlog.

[ghost]

Hi @ryanreeves-taxjar thank you for you report, please look -> https://github.com/magento/magento2/pull/19143

[ryanreeves-taxjar]

Please reconsider closing this. This is a separate issue with the class Zend_Http_Response and the change in #19143 does not fix it.

19143 fixes an issue with the Curl Client. The issue I've reported comes from ZendClient using the Curl Adapter which returns the Zend_Http_Response.

[fastdivision]

@engcom-backlog-nazar Could you please re-open this issue? This issue is not resolved by #19143 because it affects Zend_Http_Client via cURL adapter, not cURL client. Thank you!

[ghost]

@fastdivision ok, i'm will retest

[ghost]

@fastdivision The support to http/2 will be available soon

[crantron]

@ryanreeves-taxjar Hey im getting this error when getting using taxjar, is there any work around?

``` {
"message":"Invalid header line detected",
"trace":
"

0 \/home\/unitedpacific\/public_html\/releases\/208\/vendor\/magento\/zendframework1\/library\/Zend\/Http\/Response.php(696): Zend_Http_Response::extractHeaders('HTTP\/2 200 \r\nco...')\n

1 \/home\/unitedpacific\/public_html\/releases\/208\/vendor\/magento\/zendframework1\/library\/Zend\/Http\/Client.php(1110): Zend_Http_Response::fromString('HTTP\/2 200 \r\nco...')\n

2 \/home\/unitedpacific\/public_html\/releases\/208\/vendor\/magento\/framework\/HTTP\/ZendClient.php(52): Zend_Http_Client->request('POST')\n

3 \/home\/unitedpacific\/public_html\/releases\/208\/vendor\/taxjar\/module-taxjar\/Model\/Smartcalcs.php(218): Magento\Framework\HTTP\ZendClient->request('POST')\n

4 \/home\/unitedpacific\/public_html\/releases\/208\/vendor\/taxjar\/module-taxjar\/Model\/Tax\/Sales\/Total\/Quote\/Tax.php(124): Taxjar\SalesTax\Model\Smartcalcs->getTaxForOrder(Object(Magento\Quote\Model\Quote), Object(Magento\Tax\Model\Sales\Quote\QuoteDetails), Object(Magento\Quote\Model\ShippingAssignment))\n

5 \/home\/unitedpacific\/public_html\/releases\/208\/vendor\/magento\/module-quote\/Model\/Quote\/TotalsCollector.php(265): Taxjar\SalesTax\Model\Tax\Sales\Total\Quote\Tax->collect(Object(Magento\Quote\Model\Quote), Object(Magento\Quote\Model\ShippingAssignment), Object(Magento\Quote\Model\Quote\Address\Total))\n

6 \/home\/unitedpacific\/public_html\/releases\/208\/vendor\/magento\/module-quote\/Model\/ShippingMethodManagement.php(288): Magento\Quote\Model\Quote\TotalsCollector->collectAddressTotals(Object(Magento\Quote\Model\Quote), Object(Magento\Quote\Model\Quote\Address\Interceptor))\n

7 \/home\/unitedpacific\/public_html\/releases\/208\/vendor\/magento\/module-quote\/Model\/ShippingMethodManagement.php(224): Magento\Quote\Model\ShippingMethodManagement->getShippingMethods(Object(Magento\Quote\Model\Quote), Object(Magento\Quote\Model\Quote\Address\Interceptor))\n

8 \/home\/unitedpacific\/public_html\/releases\/208\/vendor\/magento\/framework\/Interception\/Interceptor.php(58): Magento\Quote\Model\ShippingMethodManagement->estimateByExtendedAddress(388, Object(Magento\Quote\Model\Quote\Address\Interceptor))\n

9 \/home\/unitedpacific\/public_html\/releases\/208\/vendor\/magento\/framework\/Interception\/Interceptor.php(138): Magento\Quote\Model\ShippingMethodManagement\Interceptor->___callParent('estimateByExten...', Array)\n

10 \/home\/unitedpacific\/public_html\/releases\/208\/vendor\/magento\/framework\/Interception\/Interceptor.php(153): Magento\Quote\Model\ShippingMethodManagement\Interceptor->Magento\Framework\Interception\{closure}(388, Object(Magento\Quote\Model\Quote\Address\Interceptor))\n

11 \/home\/unitedpacific\/public_html\/releases\/208\/generated\/code\/Magento\/Quote\/Model\/ShippingMethodManagement\/Interceptor.php(91): Magento\Quote\Model\ShippingMethodManagement\Interceptor->___callPlugins('estimateByExten...', Array, Array)\n

12 \/home\/unitedpacific\/public_html\/releases\/208\/vendor\/magento\/module-quote\/Model\/GuestCart\/GuestShippingMethodManagement.php(103): Magento\Quote\Model\ShippingMethodManagement\Interceptor->estimateByExtendedAddress(388, Object(Magento\Quote\Model\Quote\Address\Interceptor))\n

13 [internal function]: Magento\Quote\Model\GuestCart\GuestShippingMethodManagement->estimateByExtendedAddress('a2a1042a725b058...', Object(Magento\Quote\Model\Quote\Address\Interceptor))\n

14 \/home\/unitedpacific\/public_html\/releases\/208\/vendor\/magento\/module-webapi\/Controller\/Rest.php(330): call_user_func_array(Array, Array)\n

15 \/home\/unitedpacific\/public_html\/releases\/208\/vendor\/magento\/module-webapi\/Controller\/Rest.php(239): Magento\Webapi\Controller\Rest->processApiRequest()\n

16 \/home\/unitedpacific\/public_html\/releases\/208\/vendor\/magento\/framework\/Interception\/Interceptor.php(58): Magento\Webapi\Controller\Rest->dispatch(Object(Magento\Framework\App\Request\Http))\n

17 \/home\/unitedpacific\/public_html\/releases\/208\/vendor\/magento\/framework\/Interception\/Interceptor.php(138): Magento\Webapi\Controller\Rest\Interceptor->___callParent('dispatch', Array)\n

18 \/home\/unitedpacific\/public_html\/releases\/208\/vendor\/magento\/framework\/Interception\/Interceptor.php(153): Magento\Webapi\Controller\Rest\Interceptor->Magento\Framework\Interception\{closure}(Object(Magento\Framework\App\Request\Http))\n

19 \/home\/unitedpacific\/public_html\/releases\/208\/generated\/code\/Magento\/Webapi\/Controller\/Rest\/Interceptor.php(39): Magento\Webapi\Controller\Rest\Interceptor->___callPlugins('dispatch', Array, Array)\n

20 \/home\/unitedpacific\/public_html\/releases\/208\/vendor\/magento\/framework\/App\/Http.php(135): Magento\Webapi\Controller\Rest\Interceptor->dispatch(Object(Magento\Framework\App\Request\Http))\n

21 \/home\/unitedpacific\/public_html\/releases\/208\/generated\/code\/Magento\/Framework\/App\/Http\/Interceptor.php(24): Magento\Framework\App\Http->launch()\n

22 \/home\/unitedpacific\/public_html\/releases\/208\/vendor\/magento\/framework\/App\/Bootstrap.php(256): Magento\Framework\App\Http\Interceptor->launch()\n

23 \/home\/unitedpacific\/public_html\/releases\/208\/pub\/index.php(37): Magento\Framework\App\Bootstrap->run(Object(Magento\Framework\App\Http\Interceptor))\n#24 {main}"} ```

This happens when getting shipping information estimates and processing payments as a guest.

[ryanreeves-taxjar]

Hi @crantron. This is the Github for Magento 2 issues. Please contact our support team at [email protected] and we'll help you out!

[crantron]

@ryanreeves-taxjar on it!

[josefbehr]

Still got this problem on current 2.2.7 installation.

This line explicitly only checks for "HTTP/X.X" pattern, "HTTP/X" is rejected:

vendor/magento/zendframework1/library/Zend/Http/Response.php

506 if ($index === 0 && preg_match('#^HTTP/\d+(?:.\d+) [1-5]\d+#', $line)) {

[ghost]

@josefbehr this will be fixed by this PR https://github.com/magento/magento2/pull/19845
https://github.com/magento/magento2/issues/19442

[josefbehr]

@engcom-backlog-nazar thanks for the hint, but I don't see how this fixes the problem. Can you elaborate on that? And when can we expect the fix to arrive?

[ghost]

@josefbehr you can set HTTP2/version -> $client->setConfig(['httpversion' => CURL_HTTP_VERSION_2_0]);

[josefbehr]

I see, and default is 1.1. Thanks.

[Jimgitsit]

@fastdivision The support to http/2 will be available soon

How does this help? Still doesn't match HTTP/2

[josefbehr]

@engcom-backlog-nazar unfortunately, this does not help. I had to overwrite Zend Client.php and Http.php to get it working. Please re-open this issue unless you can definitely show that it is fixed.

[Jimgitsit]

As suggested by the devs at TaxJar I "fixed" this by modifying magento/zendframework1/library/Zend/Http/Response.php in two places:

Line 185

//if (! preg_match('|^\d\.\d$|', $version)) {
if (! preg_match('|^\d\.\d$|', $version) && $version != 2) {

Line 518
```
//if ($index === 0 && preg_match('#^HTTP/\d+(?:.\d+) [1-5]\d+#', $line)) {
if ($index === 0 && preg_match('#^HTTP/\d+(?:.\d+)? [1-5]\d+#', $line)) {
````

Two other suggestions provided by TaxJar are

• You can downgrade the version of curl you're using. We've had reports that version 7.61.1 works fine.
• You can rebuild the Curl package from source without HTTP/2 support.

Message from TaxJar:

The problem seems to be with Magento's support for HTTP/2. When we have seen this previously, it was related to running a recent version of Curl (7.62+), where HTTP/2 support is enabled by default. Magento's Zend_Http_Response class can't handle parsing the HTTP/2 headers and throws the exception "Invalid header line detected". We've reported this issue to Magento and their most recent response said this will be addressed in the Magento release 2.3.1 - https://github.com/magento/magento2/issues/19442

I don't really get how issue 19442 fixes this though (which modified vendor/magento/framework/HTTP/Adapter/Curl.php). If you were to use \Zend_Http_Client and \Zend_Http_Response directly as TaxJar does the issue still exists. Seems as though it needs to be fixed in magento/zendframework1 (magento/zf1).

[josefbehr]

@Jimgitsit Yes, exactly. Just applying the change from #19442 to Curl.php does not resolve the issue. I also had to change Response.php, and I think another file, too, in a similar manner to get Magento to accept HTTP/2 responses. I'll re-open this ticket then. Unfortunately, it is a problem in Zf1, which should be replaced anyway. So I'm not sure if a PR changing these files would be accepted.

[davidalger]

I believe this issue is related to PR #21549 and should be resolved once that PR is merged in.

[josefbehr]

@davidalger
No, please read the comments, that change does NOT fix the problem with http/2 headers.

[davidalger]

@josefbehr The ZF1 library does not support HTTP/2 and that is why the regex is the way it is (if you look at Zend_Http_Client there is no constant there for HTTP/2 but is for 1.1 and 1.0).

Did you look at the code which demonstrates reproducing the same exception due to the same regex that doesn't work with HTTP/2 responses? The issue here isn't the regex, but that the curl adapter (i.e. the Magento framework's implementation that overrides the working one in ZF1) was doing nothing with the default 1.1 version passed as an argument, allowing the ZF1 framework to receive an HTTP/2 response to begin with. With the fix from that PR in place, the original issue from the OP is resolved as the curl adapter then respects and sends the HTTP version with the GET request:

1. Make an api request using ZendClient and the Curl adapter to an API endpoint that returns an HTTP/2 response
2. Receive the error "Invalid header line detected"

[josefbehr]

@davidalger That "fix" did not fix the problem for me (and apparently others), when applied, so either there is a secondary problem in some circumstances, or this change does _not_ completely fix the occurrence of this error. At least that's my conclusion. If you insist, we can close this issue, but won't that kinda dismiss the other experiences? I might open another issue for that, though.

[davidalger]

@josefbehr I don’t want to argue, but I see nothing in any of the above indicating anyone else on this thread tried the change in PR #21549. This is a different change than the other mentioned PRs and it _does_ fix calling out to HTTP2 endpoints from a system with HTTP2 enabled curl libraries. There is sample code on the PR that proves this. It doesn’t do it by attemping to fix the parsing of HTTP2 responses, but by properly setting 1.0 or 1.1 on the curl resources so the server does not reply with HTTP2 to begin with. Mind taking another look at PR #21549, and showing some sample code which causes this exception _with the fix on that PR_ in place?

[ghost]

Hi @ryanreeves-taxjar according to https://github.com/magento/magento2/issues/19441#issuecomment-452414197 this problem related to zendframework, you can create a ticket here -> https://github.com/magento/community-features to use new version of zend framework, thanks for collaboration.

[mudithae]

Hi All,
I've got this issue in Magento 2.2.4 instance and below is the workaround I applied. (A dirty fix)

composer.json "autoload": {
....
,
"files": [
.... ,
"lib/Zend/Http/Response.php"
]
},`

Copy the vendor/magento/zendframework1/library/Zend/Http/Response.php to lib/Zend/Http/Response.php

Edit the following lines.
Line No: 517
if ($index === 0 && preg_match('#^HTTP/\d(?:\.\d+)* [1-5]\d+#', $line)) {
Line No: 185
if (! preg_match('#^\d|\d(\.\d)*$#', $version)) {

Then Run composer dump-autoload

The issue is Zend_Http_Response doesn't support HTTP/2. It only support HTTP/1.1.

What I have done here is load the file in lib/Zend/Http/Response.php instead vendor/magento/zendframework1/library/Zend/Http/Response.php and edit the regular expression to accept the HTTP/2.