Spring-cloud-gateway: retry filter sends empty body when retrying POST/PUT/etc

[vpavlyuk]

We're using RetryGatewayFilterFactory with the following configuration

      - name: Retry
        args:
          retries: 3
          statuses:
          - INTERNAL_SERVER_ERROR
          exceptions:
          - java.io.IOException
          methods:
          - GET
          - HEAD
          - POST
          - PUT
          - PATCH
          - DELETE
          - OPTIONS

However, when the retry happens with a non-empty body, e.g. on POST or PUT request, the backend received empty body, and I see the following in the gateway log:

2018-10-01 13:33:35,480 ERROR [o.s.b.a.w.r.e.DefaultErrorWebExceptionHandler] [reactor-http-server-epoll-5] Failed to handle request [PUT http://localhost:8084/zones/vitaliy-42.biz/rrsets/A/b2]
java.lang.IllegalStateException: Only one connection receive subscriber allowed.
    at reactor.ipc.netty.channel.FluxReceive.startReceiver(FluxReceive.java:279)
    at reactor.ipc.netty.channel.FluxReceive.lambda$subscribe$2(FluxReceive.java:129)
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:163)
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java)
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:315)
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
    at java.lang.Thread.run(Thread.java:748)

How can I overcome this? I understand reactor is 'read body only once' by design, but still, shouldn't retry filter be able to handle this somehow? Any suggestions are welcomed.

[spencergibb]

I don't think there is a workaround.

[paskos]

I'm curious about this issue and not familiar with details of reactor. Why is it impossible to find a workaround ?

[wangzhengxi]

why is it impossible to find a workaround ?
this is a serious bug

[spencergibb]

Actually, retry with a body (POST, PUT) has not been supported from the beginning. This should be validated and documented and this be used to track this feature going forward.

[spencergibb]

See https://github.com/spring-cloud/spring-cloud-gateway/issues/761 for documenting the limitation.

[paskos]

Maybe specify in the doc why it's not supported.

[howardem]

Hi @spencergibb, do you have plans to work on this enhancement any time soon? Retrying only requests without body limits the possibilities of deploying Spring Cloud Gateway to production environments.

[spencergibb]

There is some work being done

[viramasu]

Hi @spencergibb, do you have an ETA on which version will this be supported ?

[viramasu]

Also @spencergibb, is there any reason why gateway retry request with body was never supported?

[spencergibb]

The next release of Greenwich. Caching bodies is not easy.

[howardem]

Hi @spencergibb, I'd like to know if this issue was closed on the new release Greenwich.SR2. I noticed the tag release Greenwhich.SR2 was pushed on June 21st, but the Spring Cloud Gateway documentation hasn't been updated and the projects created using the Spring Initialzr still come with Greenwhich.SR1.

[spencergibb]

I believe it was, Greenwich.SR2 was released but not announce (it will be later today).

[Fangfeikun]

I believe it was, Greenwich.SR2 was released but not announce (it will be later today).

Today, I use Greenwich.SR2, but it still has error when retry with body:
java.lang.IllegalStateException: Only one connection receive subscriber allowed.

Does it resolved, retry with body?
It limits the possibilities of deploying Spring Cloud Gateway to production environments.

[howardem]

I was hoping for this issue to be closed with the newest Greenwich.SR2 but the problem is still present. Even though Spring Cloud Zuul is blocking by nature, it provides some nice features based on its Ribbon dependency that Spring Cloud Gateway hasn't been able to match like:

• The aforementioned ability to retry requests that contain a body payload.
• The ability to specify the max number of retries on the same server and max number of retries on the next server through Ribbon properties sample-client.ribbon.MaxAutoRetries and sample-client.ribbon.MaxAutoRetriesNextServer respectively.

I understand that Spring Cloud Gateway was built from scratch based on Project Reactor without backward compatibility to Spring Cloud Netflix umbrella, but even though, Pivotal Labs should still consider covering the most important features provided by a battle tested product like Netflix Zuul.

I also noticed that Spring Cloud Gateway learn section doesn't have the link related to the latest version 2.1.2.RELEASE.

[spencergibb]

The retry in zuul is not based on ribbon, but uses legacy property names.

We have integration tests with post now.

https://github.com/spring-cloud/spring-cloud-gateway/blob/385ce57538db8acded38e0caf155e0c8bd05fb45/spring-cloud-gateway-core/src/test/java/org/springframework/cloud/gateway/filter/factory/RetryGatewayFilterFactoryIntegrationTests.java#L84-L89

@Fangfeikun or @howardem can either of you provide a complete, minimal, verifiable sample that reproduces the problem? It should be available as a GitHub (or similar) project or attached to this issue as a zip file.

[Fangfeikun]

@spencergibb I update spring-boot version, now below error disappear

java.lang.IllegalStateException: Only one connection receive subscriber allowed

But i retry some times, i got this error

connection prematurely closed before response

and some request retry time out.

[Fangfeikun]

@spencergibb I wonder if you have enough tests.

[ryanjbaxter]

@Fangfeikun or @howardem can either of you provide a complete, minimal, verifiable sample that reproduces the problem?

[howardem]

Hi @spencergibb and @ryanjbaxter

Please find here the complete, minimal and verifiable sample you requested. The README file contains the necessary steps to reproduce the issue.

Waiting for your response,

Best

[renecairo]

Hi @spencergibb and @ryanjbaxter.

I'm having exactly the same issue @howardem is describing here. Is there any update on this fix?

Thanks in advance!

[Yordanyspd]

Retry filter is not working properly on my end either. Feedback on this thread will be highly appreciated.

[spencergibb]

Closed via 76d138fb40877a6f1a986ee351015570dc184de2

See #1160 to follow the bug for retry for POST.

To work around the problem, you need to add routeId to the retry configuration. It must match the routeId.

spring:
  cloud:
    gateway:
      routes:
      - id: greeting-service
        uri: lb://greeting-service
        predicates:
        - Path=/greeting-service/**
        filters:
        - name: RewritePath
          args:
            regexp: /greeting-service/(?<path>.*)
            replacement: /$\{path}
        - name: Retry
          args:
            routeId: greeting-service
            methods: GET,POST

POST is not retryable by default. So you need to configure the methods.

[howardem]

Hi @spencergibb,

Thank so much for the response! It works!! When I debugged the Retry filter I noticed the routeId argument but I thought it was kind of redundant since the filter in this case is nested to an existing route definition. Anyways, thank you so much!! It would be great to update the Spring Cloud Gateway documentation to make it match with the current GA, 2.1.2.RELEASE.

Best Regards,

[spencergibb]

This is a bug workaround. Docs aren't the right place, maybe release notes.

[Fangfeikun]

@spencergibb In the early, I have used the routeId argument in my config, it work well when happened error and retry, but when there is no error happen, default filters and my defined filters also execute twice.

[Fangfeikun]

@spencergibb I think this is a bug.

[Fangfeikun]

No matter i use the routeId argument or not, it is not work well.

[spencergibb]

@Fangfeikun
Can you provide a complete, minimal, verifiable sample that reproduces the problem? It should be available as a GitHub (or similar) project or attached to this issue as a zip file.

[Fangfeikun]

@spencergibb @ryanjbaxter
Sorry, I was busy with other things during this time.
Recently, I found this problem has resovled by use spring cloud Greenwich.SR3 and spring boot 2.1.8.RELEASE.
This problem can be easy reproduce in Greenwich.SR2 and 2.1.5.RELEASE

In provide

    @PostMapping("/retry-test/home")
    public ResponseEntity home(@RequestBody String body) {
        System.out.println("body:" + body);
        return new ResponseEntity(HttpStatus.BAD_GATEWAY);
    }

    @PostMapping("/retry-test/timeout")
    public ResponseEntity timeout(@RequestBody String body) throws InterruptedException {
        int sleep = ThreadLocalRandom.current().nextInt(3000);
        System.out.println("body:" + body + "," + "sleep:" + sleep);
        Thread.sleep(sleep);
        return new ResponseEntity(HttpStatus.OK);
    }

    @GetMapping("/retry-test/health")
    public Object health() {
        System.out.println("HelloWorld");
        return "OK";
    }

In gateway error

2019-09-16 02:11:27.812Z ERROR [reactor-http-nio-1] o.s.w.s.a.HttpWebHandlerAdapter [HttpWebHandlerAdapter.java:286] - [b362bda7] Error [java.lang.IllegalStateException: Only one connection receive subscriber allowed.] for HTTP POST "/retry-test/home", but ServerHttpResponse already committed (502 BAD_GATEWAY) []

If you also need example, i will provide later.

[Fangfeikun]

By the way, I notice the document still not update, Are there any other issues to be resolved?

[ryanjbaxter]

@Fangfeikun can you open an issue about the documentation?

As far as the bug goes if it is working with Greenwich.SR3 then that is what we suggest you use.

[Fangfeikun]

@ryanjbaxter #1302

[Young200808]

@spencergibb hi, may I know if the spring cloud gateway can support retry POST with body right now? Or still Only can use adding routeId to the retry configuration to walk around.

[spencergibb]

The work around has not been needed for some time

[RaedBhk]

hi, may I know if the spring cloud gateway can support retry POST with body right now?

[spencergibb]

@RaedBhk yes, for more than a year.