Spring-boot: MockMvc support for testing errors

[rwinch]

Spring Boot currently registers an endpoint with the servlet container to process errors. This means that MockMvc cannot be used to assert the errors. For example, the following will fail:

mockMvc.perform(get("/missing"))
    .andExpect(status().isNotFound())
    .andExpect(content().string(containsString("Whitelabel Error Page")));

with:

java.lang.AssertionError: Response content
Expected: a string containing "Whitelabel Error Page"
     but: was ""
    ..

despite the fact that the message is displayed when the application is actually running. You can view this problem in https://github.com/rwinch/boot-mockmvc-error

It would be nice if Spring Boot could provide hooks to enable testing of the error pages too.

[wilkinsona]

I'm not sure that we should be going out of our way to encourage people to test for errors in this way. It's testing that Boot's error page support is working, rather than testing a user's own code. IMO, in most cases it'll be sufficient to verify that the request has been forwarded to /error.

That said, I have encountered this problem before when trying to provide documentation for the JSON error response. The difficulty is that MockMvc doesn't fully support forwarding requests which is what the error page support uses. I worked around the problem by making a MockMvc call to /error with the appropriate attributes:

@Test
public void errorExample() throws Exception {
    this.mockMvc
            .perform(get("/error")
                    .requestAttr(RequestDispatcher.ERROR_STATUS_CODE, 400)
                    .requestAttr(RequestDispatcher.ERROR_REQUEST_URI,
                            "/notes")
                    .requestAttr(RequestDispatcher.ERROR_MESSAGE,
                            "The tag 'http://localhost:8080/tags/123' does not exist"))
            .andDo(print()).andExpect(status().isBadRequest())
            .andExpect(jsonPath("error", is("Bad Request")))
            .andExpect(jsonPath("timestamp", is(notNullValue())))
            .andExpect(jsonPath("status", is(400)))
            .andExpect(jsonPath("path", is(notNullValue())))
            .andDo(document("error-example",
                    responseFields(
                            fieldWithPath("error").description("The HTTP error that occurred, e.g. `Bad Request`"),
                            fieldWithPath("message").description("A description of the cause of the error"),
                            fieldWithPath("path").description("The path to which the request was made"),
                            fieldWithPath("status").description("The HTTP status code, e.g. `400`"),
                            fieldWithPath("timestamp").description("The time, in milliseconds, at which the error occurred"))));
    }

[rwinch]

@wilkinsona Thanks for the response.

It's testing that Boot's error page support is working, rather than testing a user's own code.

The goal is to ensure that user's have everything configured correctly. This becomes more important when the user configures custom error handling.

The difficulty is that MockMvc doesn't fully support forwarding requests which is what the error page support uses.

This is a good point. However, the MockMvc and HtmlUnit support does handle forwards. Granted, it does not handle error codes but perhaps this is something that should change.

Ultimately, my goal is to easily be able to test custom error handling in Boot. I want to be able to ensure that if my application has an error it is properly handled (not just if I directly request a URL for error handling).

[pavelfomin]

I have a similar issue with the verification of the JSR-303 validation errors from our rest controllers. When 400 http status is returned, the response content is empty when using mockmvc (returns proper json error content during actual app run). W/out the response it's harder to tell for sure which field caused the validation error. TestRestTemplate approach mentioned in 7321 is not ideal as it could potentially commit and cause test data pollution (i.e. @Transactional on test doesn't affect the standalone container started). I ended up creating custom @ExceptionHandler methods to handle errors. See this simplified example of exception handlers for more details and a test verifying a particular validation error as well.

[snicoll]

I got caught by that as well when writing a test for #7582

[shakuzen]

As the for-team-discussion label was removed, could the results of the discussion be shared for this issue?

[philwebb]

We discussed it briefly in the context of the 1.5 release but we don't think we can find a suitable solution in the time-frame. We'll need to look again once 1.5 is released.

[kiru]

Why can I not disable the ErrorPageFilter like described here: http://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#howto-disable-registration-of-a-servlet-or-filter
?

[wilkinsona]

@kiru It's impossible to say without some more context and this isn't a great place to ask questions. If you'd like some help, please use Gitter or Stack Overflow.

[tunix]

Hi, I'm trying to document a service using REST Docs and I too, have this problem. 😄 In some conditional case, the value returned from the API expires and I'd like to show the error case in the documentation. Currently I'm showing what the error response looks like using the above code sample @wilkinsona shared.

[wilkinsona]

Ultimately, my goal is to easily be able to test custom error handling in Boot. I want to be able to ensure that if my application has an error it is properly handled (not just if I directly request a URL for error handling).

To be sure that any error handling is working fully, it's necessary to involve the servlet container in that testing as it's responsible for error page registration etc. Even if MockMvc itself or a Boot enhancement to MockMvc allowed forwarding to an error page, you'd be testing the testing infrastructure not the real-world scenario that you're actually interested in.

Our recommendation for tests that want to be sure that error handling is working correctly, is to use an embedded container and test with WebTestClient, RestAssured, or TestRestTemplate.

[irlyanov]

As it properly said above, when RestDocs came, error handling became important and mockMvc won't used only for the testing, but for the test-driven documentation. And it's very sad if one cannot document some important error case.

I think the approach proposed by @wilkinsona is good, we can just make manual redirect to error page.
Not sure it will work for everyone but works for me.

`` this.mockMvc.perform( post("/v1/item").content(createData()) .contentType(MediaType.APPLICATION_JSON_VALUE)) .andDo(result -> { if (result.getResolvedException() != null) { byte[] response = mockMvc.perform(get("/error").requestAttr(RequestDispatcher.ERROR_STATUS_CODE, result.getResponse() .getStatus()) .requestAttr(RequestDispatcher.ERROR_REQUEST_URI, result.getRequest() .getRequestURI()) .requestAttr(RequestDispatcher.ERROR_EXCEPTION, result.getResolvedException()) .requestAttr(RequestDispatcher.ERROR_MESSAGE, String.valueOf(result.getResponse() .getErrorMessage()))) .andReturn() .getResponse() .getContentAsByteArray(); result.getResponse() .getOutputStream() .write(response); } }) .andExpect(status().isForbidden()) .andDo(document("post-unautorized-example", responseHeaders(headerWithName(HEADER_WWW_AUTHENTICATE).description("Unauthorized header.")), responseFields(ERROR_PLAYLOAD)));

[jmisur]

For anyone who's still struggling with this, I found super easy solution.
Just add this component to your test classes. It will be included in your test context for MockMvc test and proper error translation will be performed.

import org.springframework.boot.autoconfigure.web.servlet.error.BasicErrorController
import org.springframework.boot.web.servlet.error.ErrorController
import org.springframework.http.ResponseEntity
import org.springframework.validation.BindException
import org.springframework.web.bind.MethodArgumentNotValidException
import org.springframework.web.bind.annotation.ControllerAdvice
import org.springframework.web.bind.annotation.ExceptionHandler
import javax.servlet.http.HttpServletRequest

/**
 * This advice is necessary because MockMvc is not a real servlet environment, therefore it does not redirect error
 * responses to [ErrorController], which produces validation response. So we need to fake it in tests.
 * It's not ideal, but at least we can use classic MockMvc tests for testing error response + document it.
 */
@ControllerAdvice
internal class MockMvcValidationConfiguration(private val errorController: BasicErrorController) {

    // add any exceptions/validations/binding problems
    @ExceptionHandler(MethodArgumentNotValidException::class, BindException::class)
    fun defaultErrorHandler(request: HttpServletRequest, ex: Exception): ResponseEntity<*> {
        request.setAttribute("javax.servlet.error.request_uri", request.pathInfo)
        request.setAttribute("javax.servlet.error.status_code", 400)
        return errorController.error(request)
    }
}

[arey]

Thanks for the tips @jmisur. I've used your solution in my CrashControllerTest

[harsathmeetha]

    private MvcResult uploadWithInvalidServiceName(String originalFilename, byte[] testFileContent, String serviceName)
            throws Exception {
        var postFile = multipart(API_V1 + "/file/" + serviceName);
        postFile.with(request -> {
            request.setMethod("POST");
            request.addHeader(ACCEPT_LANGUAGE, Locale.US.toLanguageTag());
            request.addHeader(AUTHORIZATION, "Bearer " + ownerToken);
            request.addParameter(PASSWORD_KEY, passwordKey);
            request.setContentType(MediaType.MULTIPART_FORM_DATA_VALUE);
            return request;
        });
        return (MvcResult) mockMvc.perform(postFile.file(new MockMultipartFile("file",
                originalFilename, MediaType.TEXT_PLAIN_VALUE, testFileContent)))
                .andExpect(status().isBadRequest())
                .andExpect(jsonPath("$.[0].code", equalTo("connection.invalid")))
                .andExpect(jsonPath("$[0].defaultMessage",
                        equalTo("The given ServiceName is in Wrong Format")))
                .andDo(print());
    }
    @Test
    @DisplayName("Upload a file and Download")
    void uploadFileToAndDownload() throws Exception {
        final var originalFilename = "test_file_6.txt";
        final var testFileContent = "Test Content.".getBytes();
        final MvcResult objectResult = uploadWithInvalidServiceName(originalFilename, testFileContent, "error_drive");
        final var objectMap = new ObjectMapper().readValue(objectResult.getResponse().getContentAsString(), Map.class);

        mockMvc.perform(post(API_V1 + "/file/error_drive/" + objectMap.get("id"))
                .header(AUTHORIZATION, "Bearer " + ownerToken)
                .param(PASSWORD_KEY, passwordKey))
                .andExpect(status().isBadRequest())
                .andExpect(jsonPath("$.[0].code", equalTo("connection.invalid")))
                .andExpect(jsonPath("$[0].defaultMessage",
                        equalTo("The given ServiceName is in Wrong Format")))
                .andDo(print());

        deleteFile((Integer) objectMap.get("id"));
    }

[firatkucuk]

I have modified @jmisur solution, it works for all kind of exception, I am not satisfied with json conversion but if I find any better way I can update it.

@TestConfiguration
public class MockMvcRestExceptionConfiguration implements WebMvcConfigurer {

  private final BasicErrorController errorController;

  public MockMvcRestExceptionConfiguration(final BasicErrorController basicErrorController) {
    this.errorController = basicErrorController;
  }

  @Override
  public void addInterceptors(final InterceptorRegistry registry) {
    registry.addInterceptor(
        new HandlerInterceptor() {
          @Override
          public void afterCompletion(
              final HttpServletRequest request,
              final HttpServletResponse response,
              final Object handler,
              final Exception ex)
              throws Exception {

            final int status = response.getStatus();

            if (status >= 400) {
              request.setAttribute(RequestDispatcher.ERROR_STATUS_CODE, status);
              new ObjectMapper()
                  .writeValue(
                      response.getOutputStream(),
                      MockMvcRestExceptionConfiguration.this
                          .errorController
                          .error(request)
                          .getBody());
            }
          }
        });
  }
}