K6: Non-application-protocol error codes

[robingustafsson]

When there's a non-application-protocol error for a HTTP request or WebSocket connection today k6 sets a status of 0 (property status in response object and metric output tags) and adds a Go stdlib message to the error property in the response object as well as the metrics output tags.

This presents a number of problems when trying to analyze the data, whether manually or computer-aided:

• You need to parse text to find the actual type of error (DNS? TCP? connect fail? recv/send fail? etc.)
  
  
  
  • If you're not familiar with the Go stdlib error messages (most won't be) it can be difficult to understand what the error actually is
  
  
• There's no straightforward way to aggregate the errors as some of the error messages contain randomly selected src and dest TCP port numbers, meaning each message is unique although reporting the same issue

I therefore propose we introduce a new property error_code. A pre-defined numeric value that we map to the list of possible errors that k6 can run into. This solves the problems mentioned above; it will make it much easier to aggregate and present error stats and we can provide much more end-user digestible error messages.

I think we should also define ranges of values for different categories/layers of errors like: DNS, TCP, TLS, HTTP etc.

The gist of the proposal:

• Introduce a error_code property to the Response object [1].
• Introduce a error_code tag in the metrics output
• Use the following ranges of codes for specific categories/layers:
  
  
  
  • 0: no error, everything's OK!
  
  
  • 1...99: Reserved
  
  
  • 100-199: DNS (needs to be broken down into specific DNS error codes that k6 can report)
  
  
  • 200-299: TCP (needs to be broken down into specific TCP error codes that k6 can report)
  
  
  • 300-399: TLS (needs to be broken down into specific TLS error codes that k6 can report)
  
  
  • 400-499: HTTP client side errors (simple maps to HTTP 4xx status codes)
  
  
  • 500-599: HTTP server side errors (simple maps to HTTP 5xx status codes)
  
  
  • 600-699: WebSocket
  
  
• Set the error Response property and metrics tag to a new, user friendlier error message written by us for each of the possible error codes

[1] - https://docs.k6.io/docs/response-k6http
[2] - https://github.com/loadimpact/k6/issues/66 (the original issue regarding this topic, with the discussion that resulted in the current solution with status: 0 and error: message)

[micsjo]

I would suggest numerical codes outside the http response code range. It can be confusing to have 4xx and 5xx mean the http response code and 100-399 have a completely different meaning than the corresponding http error codes.

I would suggest simply adding 1000 to all of the suggestions (with the exception of 0 which has special meaning) would solve the predicament I am worried about here at least.

[robingustafsson]

@micsjo Yeah, we discussed this internally this morning as well. We'll offset the error codes by a 1000.