Spring-cloud-netflix: Cannot get response content while using retry in zuul

Created on 10 Oct 2017 · 18Comments · Source: spring-cloud/spring-cloud-netflix

If the response from server has the status code configured in ribbon.retryableStatusCodes, the response content will be changed and I cannot get the origin response content from zuul which is usually necessary for my application.
Will this be fixed in the future?

enhancement

Source

lly5044

All 18 comments

i got a fix may help solve the problem

in RetryableRibbonLoadBalancingHttpClient

  
@Override
public RibbonApacheHttpResponse execute(final RibbonApacheHttpRequest request, final IClientConfig configOverride) throws Exception {
    final RequestConfig.Builder builder = RequestConfig.custom();
    IClientConfig config = configOverride != null ? configOverride : this.config;
    builder.setConnectTimeout(config.get(
            CommonClientConfigKey.ConnectTimeout, this.connectTimeout));
    builder.setSocketTimeout(config.get(
            CommonClientConfigKey.ReadTimeout, this.readTimeout));
    builder.setRedirectsEnabled(config.get(
            CommonClientConfigKey.FollowRedirects, this.followRedirects));

    final RequestConfig requestConfig = builder.build();
    final LoadBalancedRetryPolicy retryPolicy = loadBalancedRetryPolicyFactory.create(this.getClientName(), this);
    RetryCallback retryCallback = new RetryCallback() {
        @Override
        public RibbonApacheHttpResponse doWithRetry(RetryContext context) throws Exception {
            //on retries the policy will choose the server and set it in the context
            //extract the server and update the request being made
            RibbonApacheHttpRequest newRequest = request;
            if(context instanceof LoadBalancedRetryContext) {
                ServiceInstance service = ((LoadBalancedRetryContext)context).getServiceInstance();
                if(service != null) {
                    //Reconstruct the request URI using the host and port set in the retry context
                    newRequest = newRequest.withNewUri(new URI(service.getUri().getScheme(),
                            newRequest.getURI().getUserInfo(), service.getHost(), service.getPort(),
                            newRequest.getURI().getPath(), newRequest.getURI().getQuery(),
                            newRequest.getURI().getFragment()));
                }
            }
            if (isSecure(configOverride)) {
                final URI secureUri = UriComponentsBuilder.fromUri(newRequest.getUri())
                        .scheme("https").build().toUri();
                newRequest = newRequest.withNewUri(secureUri);
            }
            HttpUriRequest httpUriRequest = newRequest.toRequest(requestConfig);
            final HttpResponse httpResponse = RetryableRibbonLoadBalancingHttpClient.this.delegate.execute(httpUriRequest);
            if(retryPolicy.retryableStatusCode(httpResponse.getStatusLine().getStatusCode())) {
                String respStr = EntityUtils.toString(httpResponse.getEntity());  //read content and close the connection
                BasicHttpEntity entity = new BasicHttpEntity();
                entity.setContent(new ByteArrayInputStream(respStr.getBytes()));  //set content into response
                httpResponse.setEntity(entity);                                   //response don't need to close again
                throw new RetryableStatusCodeException(RetryableRibbonLoadBalancingHttpClient.this.clientName,
                        httpResponse.getStatusLine().getStatusCode(), httpResponse, httpUriRequest.getURI());
            }
            return new RibbonApacheHttpResponse(httpResponse, httpUriRequest.getURI());
        }
    };
    RecoveryCallback recoveryCallback = new RecoveryCallback() {
        @Override
        public RibbonApacheHttpResponse recover(RetryContext context) throws Exception {
            Throwable lastThrowable = context.getLastThrowable();
            if (lastThrowable != null && lastThrowable instanceof RetryableStatusCodeException) {
                RetryableStatusCodeException ex = (RetryableStatusCodeException) lastThrowable;
                return new RibbonApacheHttpResponse(ex.getResponse(), ex.getUri());
            }
            if (lastThrowable instanceof Exception) {
                throw (Exception) lastThrowable;
            }
            else {
                throw new RetryException("Exception in retry", lastThrowable);
            }
        }
    };
    return this.executeWithRetry(request, retryPolicy, retryCallback, recoveryCallback);
}

private RibbonApacheHttpResponse executeWithRetry(RibbonApacheHttpRequest request, LoadBalancedRetryPolicy retryPolicy,
                                                  RetryCallback callback, RecoveryCallback recoveryCallback) throws Exception {
    RetryTemplate retryTemplate = new RetryTemplate();
    boolean retryable = request.getContext() == null ? true :
            BooleanUtils.toBooleanDefaultIfNull(request.getContext().getRetryable(), true);
    retryTemplate.setRetryPolicy(retryPolicy == null || !retryable ? new NeverRetryPolicy()
            : new RetryPolicy(request, retryPolicy, this, this.getClientName()));
    return retryTemplate.execute(callback, recoveryCallback);
}

  
public class RetryableStatusCodeException extends IOException {

    private static final String MESSAGE = "Service %s returned a status code of %d";

    private HttpResponse response;

    private URI uri;

    public RetryableStatusCodeException(String serviceId, int statusCode) {
        super(String.format(MESSAGE, serviceId, statusCode));
    }

    public RetryableStatusCodeException(String serviceId, int statusCode, HttpResponse response, URI uri) {
        super(String.format(MESSAGE, serviceId, statusCode));
        this.response = response;
        this.uri = uri;
    }

    public HttpResponse getResponse() {
        return response;
    }

    public URI getUri() {
        return uri;
    }
}

lly5044 on 10 Oct 2017

Can you provide an example of what the content is changed to?

ryanjbaxter on 11 Oct 2017

I have a test url which returns 500
visit the url directly, I get
{"timestamp":1507684551365,"status":500,"error":"Internal Server Error","exception":"java.lang.Exception","message":"haha","path":"/test"}

visit via zuul,I get
{"timestamp":1507684651594,"status":500,"error":"Internal Server Error","exception":"com.netflix.zuul.exception.ZuulException","message":"GENERAL"}

then I lose the origin content which may be useful

lly5044 on 11 Oct 2017

@ryanjbaxter

lly5044 on 11 Oct 2017

I see similar behaviour with Dalston.SR2. If a retry to a server that returns a value matching your configured retryableStatusCodes and you have no Zuul/Hystrix failback defined you will get a generic HTTP 500. The original error is lost.

To get a more graceful response you can override the ZuulFallbackProvider bean e.g. DefaultFallbackProvider implements ZuulFallbackProvider and then register the new bean implementation in a configuration. However, you won't be able to extract out the original error as the error cause is not passed in.

I see in the master branch there is a new API where a FallbackProvider is implemented and from the AbstractRibbonCommand is gets passed the failed exception cause which can then be mapped to a nicer HTTP response in your custom FallbackProvider implementation.

See https://github.com/spring-cloud/spring-cloud-netflix/blob/master/docs/src/main/asciidoc/spring-cloud-netflix.adoc for how it should work when the master branch code is released.

ronanbarrett on 21 Oct 2017

👍1

why the issue was closed? adding the cause to the FallbackProvider is not enough. RetryableLoadBalancingHttpClient need to pass it somehow

ezraroi on 8 Nov 2017

@ezraroi you are right, I will reopen

ryanjbaxter on 8 Nov 2017

Yes, We cannot simply use FallbackProvider for many services with different message behind the zuul. Maybe you consider my fix above。 @ryanjbaxter

lly5044 on 9 Nov 2017

@lly5044 would you like to submit a PR with your code?

ryanjbaxter on 10 Nov 2017

I'm Sorry to say that I found my fix depends On RetryableStatusCodeException And it is located in project spring-cloud-common. So I cannot submit a PR directly.
But I can give you my code for your information. @ryanjbaxter
fix.zip

lly5044 on 12 Nov 2017

You can submit a PR against spring cloud commons, we can close this issue and open an issue in the project. https://github.com/spring-cloud/spring-cloud-commons

ryanjbaxter on 13 Nov 2017

please do not close this issue now, i need to submit PR against both projects to fix this problem @ryanjbaxter

lly5044 on 14 Nov 2017

👍1

Thanks!

ryanjbaxter on 14 Nov 2017

Hi, any update on this issue?