Ingress-nginx: Invalid handling of chunked requests: Transfer-encoding "chunked" changed automatically to content-length
Is this a BUG REPORT or FEATURE REQUEST? (choose one):
BUG REPORT
NGINX Ingress controller version:
Release: 0.26.1
Build: git-2de5a893a
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: openresty/1.15.8.2
Kubernetes version (use kubectl version):
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.0", GitCommit:"ddf47ac13c1a9483ea035a79cd7c10005ff21a6d", GitTreeState:"clean", BuildDate:"2018-12-03T20:56:12Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}
Environment:
- Cloud provider or hardware configuration: On-prem and Minikube
- OS (e.g. from /etc/os-release):
- Kernel (e.g.
uname -a): - Install tools:
- Others:
What happened:
I have a java client that sends chunked HTTP POST requests, and it works fine when it is connected directly to my service using the node IP and service port. The problem is when the client sends requests through the Ingress. Requests with chunked data lower than 2m are failling and one or two passing and it feels that is random. After checking the logs, I found out that that the nginx ingress controller is changing the header "Transfer-Encoding: chunked" to "Content-Length".
What you expected to happen:
Ingress controller should not buffer the incoming requests if proxy_request_buffering is off.
How to reproduce it (as minimally and precisely as possible):
- Start Minikube with 1.13.
- Enable Ingress
- Configure ingress and turn off proxy_buffering and proxy_request_buffering. Here is the config:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: spectrum-replicator-http-api-com
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
nginx.ingress.kubernetes.io/proxy-buffering: "off"
nginx.ingress.kubernetes.io/proxy-body-size: "0"
nginx.ingress.kubernetes.io/proxy-request-buffering: "off"
nginx.ingress.kubernetes.io/proxy-max-temp-file-size: "0"
spec:
rules:
- host: spectrum-replicator-http-api.com
http:
paths:
- path: /
backend:
serviceName: spectrum-replicator
servicePort: 10010
Set Ingress controller in debug mode
Send POST HTTP request with chunked data. In my case, the java client sends the same request with different payload sizes 3k, 33k, 500k, 1m, 2m, 3m, 5m, 7m .....
Here is one of the failing requests. The ingress controller response with 200 OK and zero content. The Nginx ingress controller gets the chunk data from the client and changes the request to use Content-Length: 38. (including an example with logs from the same request with the correct headers)
http request line: "POST /api/1/space/spectrum-aa-1/objects/test-1576243368918%2Fload-test-2/payload HTTP/1.1"
lua rewrite handler, uri:"/api/1/space/spectrum-aa-1/objects/test-1576243368918/load-test-2/payload" c:1
2019/12/13 13:22:50 [debug] 47#47: *310 looking up Lua code cache with key '=rewrite_by_lua(nginx.conf:450)nhli_810dc96928b7314c9cd60e7b526c7375'
2019/12/13 13:22:50 [debug] 47#47: *310 lua creating new thread
2019/12/13 13:22:50 [debug] 47#47: *310 lua reset ctx
2019/12/13 13:22:50 [debug] 47#47: *310 http cleanup add: 00005600766C5A38
2019/12/13 13:22:50 [debug] 47#47: *310 lua run thread, top:0 c:1
2019/12/13 13:22:50 [debug] 47#47: *310 add cleanup: 00005600766C5A90
2019/12/13 13:22:50 [debug] 47#47: *310 lua resume returned 0
2019/12/13 13:22:50 [debug] 47#47: *310 lua light thread ended normally
2019/12/13 13:22:50 [debug] 47#47: *310 lua deleting light thread
2019/12/13 13:22:50 [debug] 47#47: *310 post rewrite phase: 7
2019/12/13 13:22:50 [debug] 47#47: *310 generic phase: 8
2019/12/13 13:22:50 [debug] 47#47: *310 generic phase: 9
2019/12/13 13:22:50 [debug] 47#47: *310 generic phase: 10
2019/12/13 13:22:50 [debug] 47#47: *310 access phase: 11
2019/12/13 13:22:50 [debug] 47#47: *310 access phase: 12
2019/12/13 13:22:50 [debug] 47#47: *310 access phase: 13
2019/12/13 13:22:50 [debug] 47#47: *310 access phase: 14
2019/12/13 13:22:50 [debug] 47#47: *310 post access phase: 15
2019/12/13 13:22:50 [debug] 47#47: *310 generic phase: 16
2019/12/13 13:22:50 [debug] 47#47: *310 generic phase: 17
2019/12/13 13:22:50 [debug] 47#47: *310 posix_memalign: 0000560076664310:4096 @16
2019/12/13 13:22:50 [debug] 47#47: *310 http client request body preread 59
2019/12/13 13:22:50 [debug] 47#47: *310 http request body chunked filter
2019/12/13 13:22:50 [debug] 47#47: *310 http body chunked buf t:1 f:0 00005600766612E0, pos 0000560076661912, size: 59 file: 0, size: 0
2019/12/13 13:22:50 [debug] 47#47: *310 http chunked byte: 63 s:0
2019/12/13 13:22:50 [debug] 47#47: *310 http chunked byte: 0D s:1
2019/12/13 13:22:50 [debug] 47#47: *310 http chunked byte: 0A s:3
2019/12/13 13:22:50 [debug] 47#47: *310 http chunked byte: 54 s:4
2019/12/13 13:22:50 [debug] 47#47: *310 http body chunked buf t:1 f:0 00005600766612E0, pos 0000560076661921, size: 44 file: 0, size: 0
2019/12/13 13:22:50 [debug] 47#47: *310 http chunked byte: 0D s:5
2019/12/13 13:22:50 [debug] 47#47: *310 http chunked byte: 0A s:6
2019/12/13 13:22:50 [debug] 47#47: *310 http chunked byte: 31 s:0
2019/12/13 13:22:50 [debug] 47#47: *310 http chunked byte: 34 s:1
2019/12/13 13:22:50 [debug] 47#47: *310 http chunked byte: 0D s:1
2019/12/13 13:22:50 [debug] 47#47: *310 http chunked byte: 0A s:3
2019/12/13 13:22:50 [debug] 47#47: *310 http chunked byte: 36 s:4
2019/12/13 13:22:50 [debug] 47#47: *310 http body chunked buf t:1 f:0 00005600766612E0, pos 000056007666193B, size: 18 file: 0, size: 0
2019/12/13 13:22:50 [debug] 47#47: *310 http chunked byte: 0D s:5
2019/12/13 13:22:50 [debug] 47#47: *310 http chunked byte: 0A s:6
2019/12/13 13:22:50 [debug] 47#47: *310 http chunked byte: 36 s:0
2019/12/13 13:22:50 [debug] 47#47: *310 http chunked byte: 0D s:1
2019/12/13 13:22:50 [debug] 47#47: *310 http chunked byte: 0A s:3
2019/12/13 13:22:50 [debug] 47#47: *310 http chunked byte: 52 s:4
2019/12/13 13:22:50 [debug] 47#47: *310 http body chunked buf t:1 f:0 00005600766612E0, pos 0000560076661946, size: 7 file: 0, size: 0
2019/12/13 13:22:50 [debug] 47#47: *310 http chunked byte: 0D s:5
2019/12/13 13:22:50 [debug] 47#47: *310 http chunked byte: 0A s:6
2019/12/13 13:22:50 [debug] 47#47: *310 http chunked byte: 30 s:0
2019/12/13 13:22:50 [debug] 47#47: *310 http chunked byte: 0D s:1
2019/12/13 13:22:50 [debug] 47#47: *310 http chunked byte: 0A s:8
2019/12/13 13:22:50 [debug] 47#47: *310 http chunked byte: 0D s:9
2019/12/13 13:22:50 [debug] 47#47: *310 http chunked byte: 0A s:10
2019/12/13 13:22:50 [debug] 47#47: *310 http body new buf t:1 f:0 0000560076661915, pos 0000560076661915, size: 12 file: 0, size: 0
2019/12/13 13:22:50 [debug] 47#47: *310 http body new buf t:1 f:0 0000560076661927, pos 0000560076661927, size: 20 file: 0, size: 0
2019/12/13 13:22:50 [debug] 47#47: *310 http body new buf t:1 f:0 0000560076661940, pos 0000560076661940, size: 6 file: 0, size: 0
2019/12/13 13:22:50 [debug] 47#47: *310 http body new buf t:0 f:0 0000000000000000, pos 0000000000000000, size: 0 file: 0, size: 0
2019/12/13 13:22:50 [debug] 47#47: *310 http init upstream, client timer: 0
2019/12/13 13:22:50 [debug] 47#47: *310 http map started
2019/12/13 13:22:50 [debug] 47#47: *310 http map: "" ""
2019/12/13 13:22:50 [debug] 47#47: *310 http map started
2019/12/13 13:22:50 [debug] 47#47: *310 http script var: "7ec5a68f74b7e62717c59e19d8653836"
2019/12/13 13:22:50 [debug] 47#47: *310 http map: "" "7ec5a68f74b7e62717c59e19d8653836"
2019/12/13 13:22:50 [debug] 47#47: *310 http script copy: "Host"
2019/12/13 13:22:50 [debug] 47#47: *310 http script var: "spectrum-replicator-http-api.com:80"
2019/12/13 13:22:50 [debug] 47#47: *310 http script copy: ""
2019/12/13 13:22:50 [debug] 47#47: *310 http script copy: ""
2019/12/13 13:22:50 [debug] 47#47: *310 http script copy: "X-Request-ID"
2019/12/13 13:22:50 [debug] 47#47: *310 http script var: "7ec5a68f74b7e62717c59e19d8653836"
2019/12/13 13:22:50 [debug] 47#47: *310 http script copy: "X-Real-IP"
2019/12/13 13:22:50 [debug] 47#47: *310 http script var: "192.168.64.1"
2019/12/13 13:22:50 [debug] 47#47: *310 http script copy: "X-Forwarded-For"
2019/12/13 13:22:50 [debug] 47#47: *310 http script var: "192.168.64.1"
2019/12/13 13:22:50 [debug] 47#47: *310 http script copy: "X-Forwarded-Host"
2019/12/13 13:22:50 [debug] 47#47: *310 http script var: "spectrum-replicator-http-api.com:80"
2019/12/13 13:22:50 [debug] 47#47: *310 http script copy: "X-Forwarded-Port"
2019/12/13 13:22:50 [debug] 47#47: *310 http script var: "80"
2019/12/13 13:22:50 [debug] 47#47: *310 http script copy: "X-Forwarded-Proto"
2019/12/13 13:22:50 [debug] 47#47: *310 http script var: "http"
2019/12/13 13:22:50 [debug] 47#47: *310 http script copy: "X-Scheme"
2019/12/13 13:22:50 [debug] 47#47: *310 http script var: "http"
2019/12/13 13:22:50 [debug] 47#47: *310 http script copy: ""
2019/12/13 13:22:50 [debug] 47#47: *310 http script copy: "Content-Length"
2019/12/13 13:22:50 [debug] 47#47: *310 http script var: "38"
2019/12/13 13:22:50 [debug] 47#47: *310 http script copy: ""
"POST /api/1/space/spectrum-aa-1/objects/test-1576243368918%2Fload-test-2/payload HTTP/1.1
Host: spectrum-replicator-http-api.com:80
X-Request-ID: 7ec5a68f74b7e62717c59e19d8653836
X-Real-IP: 192.168.64.1
X-Forwarded-For: 192.168.64.1
X-Forwarded-Host: spectrum-replicator-http-api.com:80
X-Forwarded-Port: 80
X-Forwarded-Proto: http
X-Scheme: http
Content-Length: 38
Content-Type: application/octet-stream
Anything else we need to know:
In this particular test, the tcpdumps show that the client is sending 5 segments of data to the nginx and the ingress controller resend the chunked data in 2 segments. Here are the logs of the same request but the ingress controller does nothing to the buffers or headers.
http request line: "POST /api/1/space/spectrum-aa-1/objects/test-1576243368918%2Fload-test-1/payload HTTP/1.1"
2019/12/13 13:22:49 [debug] 47#47: *310 http client request body preread 17
2019/12/13 13:22:49 [debug] 47#47: *310 http request body chunked filter
2019/12/13 13:22:49 [debug] 47#47: *310 http body chunked buf t:1 f:0 00005600766C4090, pos 00005600766C46C2, size: 17 file: 0, size: 0
2019/12/13 13:22:49 [debug] 47#47: *310 http chunked byte: 63 s:0
2019/12/13 13:22:49 [debug] 47#47: *310 http chunked byte: 0D s:1
2019/12/13 13:22:49 [debug] 47#47: *310 http chunked byte: 0A s:3
2019/12/13 13:22:49 [debug] 47#47: *310 http chunked byte: 54 s:4
2019/12/13 13:22:49 [debug] 47#47: *310 http body chunked buf t:1 f:0 00005600766C4090, pos 00005600766C46D1, size: 2 file: 0, size: 0
2019/12/13 13:22:49 [debug] 47#47: *310 http chunked byte: 0D s:5
2019/12/13 13:22:49 [debug] 47#47: *310 http chunked byte: 0A s:6
2019/12/13 13:22:49 [debug] 47#47: *310 http body new buf t:1 f:0 00005600766C46C5, pos 00005600766C46C5, size: 12 file: 0, size: 0
2019/12/13 13:22:49 [debug] 47#47: *310 malloc: 0000560076663300:8192
2019/12/13 13:22:49 [debug] 47#47: *310 http read client request body
2019/12/13 13:22:49 [debug] 47#47: *310 recv: eof:0, avail:0
2019/12/13 13:22:49 [debug] 47#47: *310 http client request body recv -2
2019/12/13 13:22:49 [debug] 47#47: *310 http client request body rest 3
2019/12/13 13:22:49 [debug] 47#47: *310 event timer add: 3: 60000:22794549
2019/12/13 13:22:49 [debug] 47#47: *310 http init upstream, client timer: 1
2019/12/13 13:22:49 [debug] 47#47: *310 event timer del: 3: 22794549
2019/12/13 13:22:49 [debug] 47#47: *310 http map started
2019/12/13 13:22:49 [debug] 47#47: *310 http map: "" ""
2019/12/13 13:22:49 [debug] 47#47: *310 http map started
2019/12/13 13:22:49 [debug] 47#47: *310 http script var: "a9991ec03b9caa426e510b8ae40f2a13"
2019/12/13 13:22:49 [debug] 47#47: *310 http map: "" "a9991ec03b9caa426e510b8ae40f2a13"
2019/12/13 13:22:49 [debug] 47#47: *310 http script copy: "Host"
2019/12/13 13:22:49 [debug] 47#47: *310 http script var: "spectrum-replicator-http-api.com:80"
2019/12/13 13:22:49 [debug] 47#47: *310 http script copy: ""
2019/12/13 13:22:49 [debug] 47#47: *310 http script copy: ""
2019/12/13 13:22:49 [debug] 47#47: *310 http script copy: "X-Request-ID"
2019/12/13 13:22:49 [debug] 47#47: *310 http script var: "a9991ec03b9caa426e510b8ae40f2a13"
2019/12/13 13:22:49 [debug] 47#47: *310 http script copy: "X-Real-IP"
2019/12/13 13:22:49 [debug] 47#47: *310 http script var: "192.168.64.1"
2019/12/13 13:22:49 [debug] 47#47: *310 http script copy: "X-Forwarded-For"
2019/12/13 13:22:49 [debug] 47#47: *310 http script var: "192.168.64.1"
2019/12/13 13:22:49 [debug] 47#47: *310 http script copy: "X-Forwarded-Host"
2019/12/13 13:22:49 [debug] 47#47: *310 http script var: "spectrum-replicator-http-api.com:80"
2019/12/13 13:22:49 [debug] 47#47: *310 http script copy: "X-Forwarded-Port"
2019/12/13 13:22:49 [debug] 47#47: *310 http script var: "80"
2019/12/13 13:22:49 [debug] 47#47: *310 http script copy: "X-Forwarded-Proto"
2019/12/13 13:22:49 [debug] 47#47: *310 http script var: "http"
2019/12/13 13:22:49 [debug] 47#47: *310 http script copy: "X-Scheme"
2019/12/13 13:22:49 [debug] 47#47: *310 http script var: "http"
2019/12/13 13:22:49 [debug] 47#47: *310 http script copy: ""
2019/12/13 13:22:49 [debug] 47#47: *310 http script copy: ""
2019/12/13 13:22:49 [debug] 47#47: *310 http script copy: "Transfer-Encoding"
2019/12/13 13:22:49 [debug] 47#47: *310 http script var: "chunked"
"POST /api/1/space/spectrum-aa-1/objects/test-1576243368918%2Fload-test-1/payload HTTP/1.1
Host: spectrum-replicator-http-api.com:80
X-Request-ID: a9991ec03b9caa426e510b8ae40f2a13
X-Real-IP: 192.168.64.1
X-Forwarded-For: 192.168.64.1
X-Forwarded-Host: spectrum-replicator-http-api.com:80
X-Forwarded-Port: 80
X-Forwarded-Proto: http
X-Scheme: http
Transfer-Encoding: chunked
Content-Type: application/octet-stream
Here is the nginx config:
## start server spectrum-replicator-http-api.com
server {
server_name spectrum-replicator-http-api.com ;
listen 80 ;
listen 443 ssl http2 ;
set $proxy_upstream_name "-";
ssl_certificate_by_lua_block {
certificate.call()
}
location / {
set $namespace "oneira-sandbox";
set $ingress_name "spectrum-replicator-http-api-com";
set $service_name "spectrum-replicator";
set $service_port "10010";
set $location_path "/";
rewrite_by_lua_block {
lua_ingress.rewrite({
force_ssl_redirect = false,
ssl_redirect = true,
force_no_ssl_redirect = false,
use_port_in_redirects = false,
})
balancer.rewrite()
plugins.run()
}
header_filter_by_lua_block {
plugins.run()
}
body_filter_by_lua_block {
}
log_by_lua_block {
balancer.log()
monitor.call()
plugins.run()
}
port_in_redirect off;
set $balancer_ewma_score -1;
set $proxy_upstream_name "oneira-sandbox-spectrum-replicator-10010";
set $proxy_host $proxy_upstream_name;
set $pass_access_scheme $scheme;
set $pass_server_port $server_port;
set $best_http_host $http_host;
set $pass_port $pass_server_port;
set $proxy_alternative_upstream_name "";
client_max_body_size 0;
proxy_set_header Host $best_http_host;
# Pass the extracted client certificate to the backend
# Allow websocket connections
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header X-Request-ID $req_id;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Host $best_http_host;
proxy_set_header X-Forwarded-Port $pass_port;
proxy_set_header X-Forwarded-Proto $pass_access_scheme;
proxy_set_header X-Scheme $pass_access_scheme;
# Pass the original X-Forwarded-For
proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;
# mitigate HTTPoxy Vulnerability
# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
proxy_set_header Proxy "";
# Custom headers to proxied server
proxy_connect_timeout 5s;
proxy_send_timeout 60s;
proxy_read_timeout 60s;
proxy_buffering off;
proxy_buffer_size 128k;
proxy_buffers 4 128k;
proxy_max_temp_file_size 0;
proxy_request_buffering off;
proxy_http_version 1.1;
proxy_cookie_domain off;
proxy_cookie_path off;
# In case of errors try the next upstream server before returning an error
proxy_next_upstream error timeout;
proxy_next_upstream_timeout 0;
proxy_next_upstream_tries 3;
proxy_pass http://upstream_balancer;
proxy_redirect off;
}
}
## end server spectrum-replicator-http-api.com
OscarNeira
All 52 comments
@OscarNeira if you can provide a self-contained example that reproduces the issue, I can take a look.
aledbf
on 30 Dec 2019
@OscarNeira if you can provide a self-contained example that reproduces the issue, I can take a look.
Yes, I will.
OscarNeira
on 7 Jan 2020
Hi @aledbf , I managed to reproduce the issue in a way that you can also try. and added some steps:
- Created new simple Java client here:
- Get the example here: https://github.com/OscarNeira/simple-java-client
- Use a mock server to inspect the incoming request from the Nginx.
You can use this version that includes the tcpdump layer in docker: https://github.com/OscarNeira/mockserver/tree/add/tcpdumps
Deployed that in my local cluster.
Use the Java client to create expectations: Run the test in the project (CreateSimpleExpectation and CreateSimpleExpectationPost)
Test expectations with
curl -v GET http://<INGRESS_HOST>/delay/1
- Use tcpdump in mockserver pod
Kubectl exec βit <POD_NAME> -- /bin/bash
Once inside the pod run
tcpdump -w tcpdump-ingress-chunk-1.pcapNew terminal copy the file to local folder
Kubectl cp <NAMESPACE>/ <POD_NAME>:tcpdump-ingress-chunk-1.pcap ~/tcpdump-ingress-chunk-1.pcap
- Inspect incoming requests.
I verified that the client is sending
Transfer-Encoding: chunkedusing WiresharkWith the attached file tcpdump-ingress-chunk-1.pcap you can see that Nginx is changing the header to
Content-Length
tcpdump-ingress-chunk-1.pcap.zip
Here is the ingress config:
```apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/backend-protocol: HTTP
nginx.ingress.kubernetes.io/proxy-body-size: "0"
nginx.ingress.kubernetes.io/proxy-buffering: "off"
nginx.ingress.kubernetes.io/proxy-max-temp-file-size: "0"
nginx.ingress.kubernetes.io/proxy-request-buffering: "off"
creationTimestamp: "2020-02-04T10:01:28Z"
generation: 1
name: mock-server-chunk-config
namespace: mockserver
resourceVersion: "5550485"
selfLink: /apis/extensions/v1beta1/namespaces/mockserver/ingresses/mock-server-chunk-config
uid: 4fd93f88-4735-11ea-a94f-f8b1569bea58
spec:
rules:
- host: mockserver.jesus-microk8s.com
http:
paths:
- backend:
serviceName: mockserver
servicePort: 1080
path: /
status:
loadBalancer:
ingress:
- ip: 127.0.0.1```
- backend:
OscarNeira
on 6 Feb 2020
@OscarNeira please update to 0.28.0 and test again.
aledbf
on 6 Feb 2020
Today I will try to reproduce this issue. Thank you for the procedure to reproduce the issue.
aledbf
on 6 Feb 2020
@OscarNeira please update to 0.28.0 and test again.
same results with 0.28.0
OscarNeira
on 6 Feb 2020
@OscarNeira can you try to provide a simpler way to reproduce this?
Something like https://gist.github.com/aledbf/266940de7569a1163b9e1c085aa4e771
aledbf
on 8 Feb 2020
yep, I will try.
OscarNeira
on 10 Feb 2020
Hi @aledbf , Added more details to the repo that just created. Here is the link to the script and the yaml files are in the same folder. Let me know if you need anything else. https://github.com/OscarNeira/simple-java-client/blob/master/k8s/start.
OscarNeira
on 11 Feb 2020
@OscarNeira
./k8s/start.sh
192.168.39.183 example.mockserver.com
Error from server (AlreadyExists): namespaces "mockserver" already exists
Context "minikube" modified.
service/mockserver unchanged
deployment.apps/mockserver unchanged
ingress.extensions/mock-server-chunk-config-test unchanged
mockserver-67c755fd7b-wvxpv Is mock server Running
mockserver-67c755fd7b-wvxpv Is Running
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
latest: Pulling from oscneira/java_client
Digest: sha256:a4195528758fdb08a6212c5732dd396e1daa646ead659fd5daa61f5f7178160f
Status: Image is up to date for oscneira/java_client:latest
docker.io/oscneira/java_client:latest
Setup Mock Server
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Exception in thread "main" org.mockserver.client.ClientException: error:
<html><head><title>Object moved</title></head><body>
<h2>Object moved to <a href="https://www.hugedomains.com/domain_profile.cfm?d=mockserver&e=com">here</a>.</h2>
</body></html>
while submitted expectation:
{
"id" : "4c4cc29f-f14f-409f-a39f-15330dffcab9",
"httpRequest" : {
"method" : "GET",
"path" : "/delay/1"
},
"times" : {
"unlimited" : true
},
"timeToLive" : {
"unlimited" : true
},
"httpResponse" : {
"body" : "OK",
"delay" : {
"timeUnit" : "SECONDS",
"value" : 1
}
}
}
at org.mockserver.client.MockServerClient.sendExpectation(MockServerClient.java:808)
at org.mockserver.client.ForwardChainExpectation.respond(ForwardChainExpectation.java:47)
at com.rest.JavaClient.createDelayMockServer(JavaClient.java:120)
at com.rest.JavaClient.main(JavaClient.java:37)
could you try to run this curl -v -k GET "http://example.mockserver.com/delay/1"
OscarNeira
on 11 Feb 2020
curl -v -k GET "http://example.mockserver.com/delay/1"
* Could not resolve host: GET
curl: (6) Could not resolve host: GET
* Trying 192.168.39.183:80...
* TCP_NODELAY set
* Connected to example.mockserver.com (192.168.39.183) port 80 (#1)
> GET /delay/1 HTTP/1.1
> Host: example.mockserver.com
> User-Agent: curl/7.65.3
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< Server: nginx/1.17.7
< Date: Tue, 11 Feb 2020 13:07:36 GMT
< Content-Type: text/html; charset=utf-8
< Content-Length: 186
< Connection: keep-alive
< Cache-Control: private
< Location: https://www.hugedomains.com/domain_profile.cfm?d=mockserver&e=com
< X-Powered-By: ASP.NET
<
<html><head><title>Object moved</title></head><body>
<h2>Object moved to <a href="https://www.hugedomains.com/domain_profile.cfm?d=mockserver&e=com">here</a>.</h2>
</body></html>
* Connection #1 to host example.mockserver.com left intact
could you check that 192.168.39.183 example.mockserver.com exists in the /etc/hosts
OscarNeira
on 11 Feb 2020
tail -2 /etc/hosts
192.168.39.183 example.mockserver.com
ok, could you try to open new command line/powershell/terminal and try again curl -v -k GET "http://example.mockserver.com/delay/1"
OscarNeira
on 11 Feb 2020
curl -v -k "http://example.mockserver.com/delay/1"
* Trying 192.168.39.183:80...
* TCP_NODELAY set
* Connected to example.mockserver.com (192.168.39.183) port 80 (#0)
> GET /delay/1 HTTP/1.1
> Host: example.mockserver.com
> User-Agent: curl/7.65.3
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< Server: nginx/1.17.7
< Date: Tue, 11 Feb 2020 13:14:09 GMT
< Content-Type: text/html; charset=utf-8
< Content-Length: 186
< Connection: keep-alive
< Cache-Control: private
< Location: https://www.hugedomains.com/domain_profile.cfm?d=mockserver&e=com
< X-Powered-By: ASP.NET
<
<html><head><title>Object moved</title></head><body>
<h2>Object moved to <a href="https://www.hugedomains.com/domain_profile.cfm?d=mockserver&e=com">here</a>.</h2>
</body></html>
* Connection #0 to host example.mockserver.com left intact
Ok, for some reason cannot find the host. I will delete mi minikube and run it clean.
~ at βΈοΈ minikube took 1m 7s
β¦2 β minikube delete
π₯ Deleting "minikube" in hyperkit ...
π Removed all traces of the "minikube" cluster.
~ took 6s
β¦2 β minikube start --kubernetes-version v1.13.6 --vm-driver="hyperkit" --cpus 4 --memory 5120
π minikube v1.7.2 on Darwin 10.14.6
β¨ Using the hyperkit driver based on user configuration
π₯ Creating hyperkit VM (CPUs=4, Memory=5120MB, Disk=20000MB) ...
π³ Preparing Kubernetes v1.13.6 on Docker 19.03.5 ...
OscarNeira
on 11 Feb 2020
@OscarNeira not sure what you mean. This is the log of the mockserver pod
k logs mockserver-67c755fd7b-fl58q
java -Dfile.encoding=UTF-8 -jar /opt/mockserver/mockserver-netty-jar-with-dependencies.jar -serverPort 1080 -logLevel INFO &
2020-02-11 13:21:20 org.mockserver.log.MockServerEventLog INFO started on port: 1080
2020-02-11 13:21:20 org.mockserver.cli.Main INFO logger level is INFO, change using:
- 'ConfigurationProperties.logLevel(String level)' in Java code,
- '-logLevel' command line argument,
- 'mockserver.logLevel' JVM system property or,
- 'mockserver.logLevel' property value in 'mockserver.properties'
2020-02-11 13:21:42 org.mockserver.log.MockServerEventLog INFO received request:
{
"method" : "GET",
"path" : "/delay/1",
"headers" : {
"Host" : [ "example.mockserver.com" ],
"X-Request-ID" : [ "bb1f276c36ed631d3cecb4f988738766" ],
"X-Real-IP" : [ "192.168.39.1" ],
"X-Forwarded-For" : [ "192.168.39.1" ],
"X-Forwarded-Host" : [ "example.mockserver.com" ],
"X-Forwarded-Port" : [ "80" ],
"X-Forwarded-Proto" : [ "http" ],
"X-Scheme" : [ "http" ],
"User-Agent" : [ "curl/7.65.3" ],
"Accept" : [ "*/*" ],
"content-length" : [ "0" ]
},
"keepAlive" : true,
"secure" : false
}
2020-02-11 13:21:43 org.mockserver.log.MockServerEventLog INFO returning response:
{
"statusCode" : 302,
"reasonPhrase" : "Found",
"headers" : {
"Cache-Control" : [ "private" ],
"Content-Type" : [ "text/html; charset=utf-8" ],
"Location" : [ "https://www.hugedomains.com/domain_profile.cfm?d=mockserver&e=com" ],
"Server" : [ "Microsoft-IIS/8.5" ],
"X-Powered-By" : [ "ASP.NET" ],
"Date" : [ "Tue, 11 Feb 2020 13:21:40 GMT" ],
"Content-Length" : [ "186" ]
},
"body" : "<html><head><title>Object moved</title></head><body>\r\n<h2>Object moved to <a href=\"https://www.hugedomains.com/domain_profile.cfm?d=mockserver&e=com\">here</a>.</h2>\r\n</body></html>\r\n"
}
for forwarded request
in json:
{
"method" : "GET",
"path" : "/delay/1",
"headers" : {
"Host" : [ "example.mockserver.com" ],
"X-Request-ID" : [ "bb1f276c36ed631d3cecb4f988738766" ],
"X-Real-IP" : [ "192.168.39.1" ],
"X-Forwarded-For" : [ "192.168.39.1" ],
"X-Forwarded-Host" : [ "example.mockserver.com" ],
"X-Forwarded-Port" : [ "80" ],
"X-Forwarded-Proto" : [ "http" ],
"X-Scheme" : [ "http" ],
"User-Agent" : [ "curl/7.65.3" ],
"Accept" : [ "*/*" ],
"content-length" : [ "0" ]
},
"keepAlive" : true,
"secure" : false
}
in curl:
curl -v 'http://example.mockserver.com/delay/1' -H 'Host: example.mockserver.com' -H 'X-Request-ID: bb1f276c36ed631d3cecb4f988738766' -H 'X-Real-IP: 192.168.39.1' -H 'X-Forwarded-For: 192.168.39.1' -H 'X-Forwarded-Host: example.mockserver.com' -H 'X-Forwarded-Port: 80' -H 'X-Forwarded-Proto: http' -H 'X-Scheme: http' -H 'User-Agent: curl/7.65.3' -H 'Accept: */*' -H 'content-length: 0'
could you try to run the image again docker run -it --rm=true oscneira/java_client
OscarNeira
on 11 Feb 2020
Deleted all the objects
kubectl delete deploy mockserver
#kubectl delete ing mock-server-chunk-config-test
#kubectl delete svc mockserver
and run the script again
./k8s/start.sh
Error from server (AlreadyExists): namespaces "mockserver" already exists
Context "minikube" modified.
service/mockserver created
deployment.apps/mockserver created
ingress.extensions/mock-server-chunk-config-test created
mockserver-75c9d49dc8-qp2kn Is mock server Running
mockserver-75c9d49dc8-qp2kn Is Running
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
latest: Pulling from oscneira/java_client
Digest: sha256:a4195528758fdb08a6212c5732dd396e1daa646ead659fd5daa61f5f7178160f
Status: Image is up to date for oscneira/java_client:latest
docker.io/oscneira/java_client:latest
Setup Mock Server
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Mock Server Ready
START http call Small Encoded file
InputStreamEntity DATA getContent: java.io.FileInputStream@3624da92
InputStreamEntity DATA getContentType: Content-Type: application/octet-stream
InputStreamEntity DATA getContentEncoding: null
InputStreamEntity DATA isStreaming: false
Executing request: POST http://example.mockserver.com/api/1/space/spectrum-aa-1/objects/jesus-test-37318%2Ftest_createSingleSmallCall%2Fload-test-1/payload HTTP/1.1
Executing request Hearder Name x-spectrum-access-token
Executing request Hearder Value
Executing request Hearder Name Content-Type
Executing request Hearder Value application/octet-stream
Executing request Hearder Name x-spectrum-meta-fs2_headers_test-name
Executing request Hearder Value ["test_createSingleSmallCall"]
Executing request Hearder Name x-spectrum-meta-fs2_metadata_dataDefinitionId
Executing request Hearder Value Payload
Executing request Hearder Name x-spectrum-meta-wasZippedFirst
Executing request Hearder Value false
Executing request Hearder Name x-spectrum-meta-fs2_metadata_createdBy
Executing request Hearder Value FS2
Executing request Hearder Name x-spectrum-meta-fs2_headers_test-header
Executing request Hearder Value ["test-header-value"]
Executing request Hearder Name source
Executing request Hearder Value G2_SB_DEV_INT
Executing request Hearder Name ttl
Executing request Hearder Value 50000
Executing request Hearder Name x-spectrum-meta-fs2_metadata_ttl
Executing request Hearder Value 50000
Executing request Hearder Name x-spectrum-meta-fs2_metadata_createdOn
Executing request Hearder Value Tue Feb 11 13:38:58 GMT 2020
Executing request Hearder Name timestamp
Executing request Hearder Value 1581428338896
Executing request Hearder Name Connection
Executing request Hearder Value Keep-Alive
Executing request Hearder Name Accept-Encoding
Executing request Hearder Value gzip,deflate
----------------------------------------
HTTP/1.1 200 OK
OK
END http call Small Encoded file
Could you try to delete and pull the latest image?
OscarNeira
on 11 Feb 2020
Same result. I started a fresh minikube cluster
./k8s/start.sh
Error from server (AlreadyExists): namespaces "mockserver" already exists
service/mockserver unchanged
deployment.apps/mockserver unchanged
ingress.extensions/mock-server-chunk-config-test unchanged
mockserver-67c755fd7b-d8ncj Is mock server Running
Error from server (NotFound): pods "mockserver-67c755fd7b-d8ncj" not found
mockserver-67c755fd7b-d8ncj Is Running
Error from server (NotFound): pods "mockserver-67c755fd7b-d8ncj" not found
latest: Pulling from oscneira/java_client
Digest: sha256:a4195528758fdb08a6212c5732dd396e1daa646ead659fd5daa61f5f7178160f
Status: Image is up to date for oscneira/java_client:latest
docker.io/oscneira/java_client:latest
Setup Mock Server
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Exception in thread "main" org.mockserver.client.ClientException: error:
<html><head><title>Object moved</title></head><body>
<h2>Object moved to <a href="https://www.hugedomains.com/domain_profile.cfm?d=mockserver&e=com">here</a>.</h2>
</body></html>
while submitted expectation:
{
"id" : "12d61441-3080-4e30-8cdc-98f7be25a51a",
"httpRequest" : {
"method" : "GET",
"path" : "/delay/1"
},
"times" : {
"unlimited" : true
},
"timeToLive" : {
"unlimited" : true
},
"httpResponse" : {
"body" : "OK",
"delay" : {
"timeUnit" : "SECONDS",
"value" : 1
}
}
}
at org.mockserver.client.MockServerClient.sendExpectation(MockServerClient.java:808)
at org.mockserver.client.ForwardChainExpectation.respond(ForwardChainExpectation.java:47)
at com.rest.JavaClient.createDelayMockServer(JavaClient.java:120)
at com.rest.JavaClient.main(JavaClient.java:37)
Error from server (NotFound): pods "mockserver-67c755fd7b-d8ncj" not found
Updated the project a bit to get more info, I feel that we are using an old image, created a new tag for this test. I also updated the script a bit. https://github.com/OscarNeira/simple-java-client/blob/master/k8s/start.sh
- Could you check the
/etc/hostsfile has the correct minikube ip you can runminikube ipto verify that. - Update the repo to the latest master branch
- and run the script again
OscarNeira
on 12 Feb 2020
Same
k8s/start.sh
Error from server (AlreadyExists): namespaces "mockserver" already exists
Context "minikube" modified.
service/mockserver unchanged
deployment.apps/mockserver unchanged
ingress.extensions/mock-server-chunk-config-test unchanged
mockserver-67c755fd7b-hclwj Is mock server Running
mockserver-67c755fd7b-hclwj Is Running
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
2.0.0: Pulling from oscneira/java_client
Digest: sha256:339cfa1271b078a816122e0137719e8af3f104b8d02c632fa5a85aa774ed8728
Status: Image is up to date for oscneira/java_client:2.0.0
docker.io/oscneira/java_client:2.0.0
Setup Mock Server
Create Expectation Delay 1 Sec
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Exception in thread "main" org.mockserver.client.ClientException: error:
<html><head><title>Object moved</title></head><body>
<h2>Object moved to <a href="https://www.hugedomains.com/domain_profile.cfm?d=mockserver&e=com">here</a>.</h2>
</body></html>
while submitted expectation:
{
"id" : "9eac0e29-0dd1-4864-ac9f-964593925ec0",
"httpRequest" : {
"method" : "GET",
"path" : "/delay/1"
},
"times" : {
"unlimited" : true
},
"timeToLive" : {
"unlimited" : true
},
"httpResponse" : {
"body" : "OK",
"delay" : {
"timeUnit" : "SECONDS",
"value" : 1
}
}
}
at org.mockserver.client.MockServerClient.sendExpectation(MockServerClient.java:808)
at org.mockserver.client.ForwardChainExpectation.respond(ForwardChainExpectation.java:47)
at com.rest.JavaClient.createDelayMockServer(JavaClient.java:120)
at com.rest.JavaClient.main(JavaClient.java:36)
aledbf@me ξ° ~/.../github/simple-java-client ξ° ξ master β ξ° curl example.mockserver.com/delay/1
<html><head><title>Object moved</title></head><body>
<h2>Object moved to <a href="https://www.hugedomains.com/domain_profile.cfm?d=mockserver&e=com">here</a>.</h2>
</body></html>
Also, reading the ingress controller pod logs, I see the 302 is returned by the app
192.168.39.1 - - [12/Feb/2020:11:41:07 +0000] "GET /delay/1 HTTP/1.1" 302 186 "-" "curl/7.65.3" 93 0.811 [mockserver-mockserver-1080] [] 172.17.0.4:1080 186 0.811 302 8af8c6049e5465827f23584f3c1edc0a
aledbf
on 12 Feb 2020
Ok, it seems that the java client cannot create expectation rules. I will test using the minikube ip and let you know.
OscarNeira
on 12 Feb 2020
It works as well with minikube IP could you share your minikube Ip? and I will create a new image tag with that change. After I push the change could you change the script start.sh and update the version?
docker pull oscneira/java_client:4.0.0
docker run -it --rm=true oscneira/java_client:4.0.0
Here plan B: created expectations using curl. You can update the project to the latest version and run this one. https://github.com/OscarNeira/simple-java-client/blob/master/k8s/start-curl.sh
OscarNeira
on 12 Feb 2020
It works as well with minikube IP could you share your minikube Ip?
Maybe you should read some environment variable? Hardcoding an IP is not very flexible
aledbf
on 12 Feb 2020
Here plan B: created expectations using curl.
Same
k8s/start-curl.sh
Error from server (AlreadyExists): namespaces "mockserver" already exists
Context "minikube" modified.
service/mockserver unchanged
deployment.apps/mockserver unchanged
ingress.extensions/mock-server-chunk-config-test unchanged
mockserver-67c755fd7b-sqbx2 Is mock server Running
mockserver-67c755fd7b-sqbx2 Is Running
* Trying 192.168.39.162:80...
* TCP_NODELAY set
* Connected to example.mockserver.com (192.168.39.162) port 80 (#0)
> GET /delay/1 HTTP/1.1
> Host: example.mockserver.com
> User-Agent: curl/7.65.3
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: nginx/1.17.7
< Date: Wed, 12 Feb 2020 13:19:03 GMT
< Content-Length: 16
< Connection: keep-alive
<
* Connection #0 to host example.mockserver.com left intact
Delayed 1 secondtcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
5.0.0: Pulling from oscneira/java_client
Digest: sha256:be518c4eaaaf044352c413d9bf2b1ad82ae4568741f8a8de9f335c188a65fb7d
Status: Image is up to date for oscneira/java_client:5.0.0
docker.io/oscneira/java_client:5.0.0
START http call Small Encoded file
mockServerIngressEndpoint::http://example.mockserver.com/api/1/space/spectrum-aa-1/objects/jesus-test-39473%2Ftest_createSingleSmallCall%2Fload-test-1/payload
InputStreamEntity DATA getContent: java.io.FileInputStream@1e7c7811
InputStreamEntity DATA getContentType: Content-Type: application/octet-stream
InputStreamEntity DATA getContentEncoding: null
InputStreamEntity DATA isStreaming: false
Executing request: POST http://example.mockserver.com/api/1/space/spectrum-aa-1/objects/jesus-test-39473%2Ftest_createSingleSmallCall%2Fload-test-1/payload HTTP/1.1
Executing request Hearder Name x-spectrum-access-token
Executing request Hearder Value
Executing request Hearder Name Content-Type
Executing request Hearder Value application/octet-stream
Executing request Hearder Name x-spectrum-meta-fs2_headers_test-name
Executing request Hearder Value ["test_createSingleSmallCall"]
Executing request Hearder Name x-spectrum-meta-fs2_metadata_dataDefinitionId
Executing request Hearder Value Payload
Executing request Hearder Name x-spectrum-meta-wasZippedFirst
Executing request Hearder Value false
Executing request Hearder Name x-spectrum-meta-fs2_metadata_createdBy
Executing request Hearder Value FS2
Executing request Hearder Name x-spectrum-meta-fs2_headers_test-header
Executing request Hearder Value ["test-header-value"]
Executing request Hearder Name source
Executing request Hearder Value G2_SB_DEV_INT
Executing request Hearder Name ttl
Executing request Hearder Value 50000
Executing request Hearder Name x-spectrum-meta-fs2_metadata_ttl
Executing request Hearder Value 50000
Executing request Hearder Name x-spectrum-meta-fs2_metadata_createdOn
Executing request Hearder Value Wed Feb 12 13:19:09 GMT 2020
Executing request Hearder Name timestamp
Executing request Hearder Value 1581513549653
Executing request Hearder Name Connection
Executing request Hearder Value Keep-Alive
Executing request Hearder Name Accept-Encoding
Executing request Hearder Value gzip,deflate
----------------------------------------
HTTP/1.1 302 Found
<html><head><title>Object moved</title></head><body>
<h2>Object moved to <a href="https://www.hugedomains.com/domain_profile.cfm?d=mockserver&e=com">here</a>.</h2>
</body></html>
END http call Small Encoded file
Did you get errors with this calls?
curl -X PUT "http://example.mockserver.com/expectation" -H "accept: */*" -H "Content-Type: application/json" -d '[{"httpRequest":{"path":"/api/1/space/.*","method":"POST"},"httpResponse":{"delay":{"timeUnit":"SECONDS","value":1},"statusCode":200,"reasonPhrase":"OK"}}]'
curl -d '{"key1":"value1", "key2":"value2"}' -H "Content-Type: application/json" -X POST http://example.mockserver.com/api/1/space/
OscarNeira
on 12 Feb 2020
curl -X PUT "http://example.mockserver.com/expectation" -H "accept: */*" -H "Content-Type: application/json" -d '[{"httpRequest":{"path":"/api/1/space/.*","method":"POST"},"httpResponse":{"delay":{"timeUnit":"SECONDS","value":1},"statusCode":200,"reasonPhrase":"OK"}}]' -vvv
* Trying 192.168.39.162:80...
* TCP_NODELAY set
* Connected to example.mockserver.com (192.168.39.162) port 80 (#0)
> PUT /expectation HTTP/1.1
> Host: example.mockserver.com
> User-Agent: curl/7.65.3
> accept: */*
> Content-Type: application/json
> Content-Length: 155
>
* upload completely sent off: 155 out of 155 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 201 Created
< Server: nginx/1.17.7
< Date: Wed, 12 Feb 2020 18:33:59 GMT
< Content-Length: 0
< Connection: keep-alive
< version: 5.9.0
< deprecated: "/expectation" is deprecated use "/mockserver/expectation" instead
<
* Connection #0 to host example.mockserver.com left intact
curl -d '{"key1":"value1", "key2":"value2"}' -H "Content-Type: application/json" -X POST http://example.mockserver.com/api/1/space/ -vvv
Note: Unnecessary use of -X or --request, POST is already inferred.
* Trying 192.168.39.162:80...
* TCP_NODELAY set
* Connected to example.mockserver.com (192.168.39.162) port 80 (#0)
> POST /api/1/space/ HTTP/1.1
> Host: example.mockserver.com
> User-Agent: curl/7.65.3
> Accept: */*
> Content-Type: application/json
> Content-Length: 34
>
* upload completely sent off: 34 out of 34 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: nginx/1.17.7
< Date: Wed, 12 Feb 2020 18:33:49 GMT
< Content-Length: 0
< Connection: keep-alive
<
* Connection #0 to host example.mockserver.com left intact
Ok, the mock server response is ok. Could you try to run again the script start-curl.sh ?
OscarNeira
on 12 Feb 2020
@OscarNeira same
./k8s/start-curl.sh
Error from server (AlreadyExists): namespaces "mockserver" already exists
Context "minikube" modified.
service/mockserver unchanged
deployment.apps/mockserver unchanged
ingress.extensions/mock-server-chunk-config-test unchanged
mockserver-67c755fd7b-sqbx2 Is mock server Running
mockserver-67c755fd7b-sqbx2 Is Running
* Trying 192.168.39.162:80...
* TCP_NODELAY set
* Connected to example.mockserver.com (192.168.39.162) port 80 (#0)
> GET /delay/1 HTTP/1.1
> Host: example.mockserver.com
> User-Agent: curl/7.65.3
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: nginx/1.17.7
< Date: Wed, 12 Feb 2020 19:03:07 GMT
< Content-Length: 16
< Connection: keep-alive
<
* Connection #0 to host example.mockserver.com left intact
Delayed 1 secondtcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
5.0.0: Pulling from oscneira/java_client
Digest: sha256:be518c4eaaaf044352c413d9bf2b1ad82ae4568741f8a8de9f335c188a65fb7d
Status: Image is up to date for oscneira/java_client:5.0.0
docker.io/oscneira/java_client:5.0.0
START http call Small Encoded file
mockServerIngressEndpoint::http://example.mockserver.com/api/1/space/spectrum-aa-1/objects/jesus-test-49412%2Ftest_createSingleSmallCall%2Fload-test-1/payload
InputStreamEntity DATA getContent: java.io.FileInputStream@eafc191
InputStreamEntity DATA getContentType: Content-Type: application/octet-stream
InputStreamEntity DATA getContentEncoding: null
InputStreamEntity DATA isStreaming: false
Executing request: POST http://example.mockserver.com/api/1/space/spectrum-aa-1/objects/jesus-test-49412%2Ftest_createSingleSmallCall%2Fload-test-1/payload HTTP/1.1
Executing request Hearder Name x-spectrum-access-token
Executing request Hearder Value
Executing request Hearder Name Content-Type
Executing request Hearder Value application/octet-stream
Executing request Hearder Name x-spectrum-meta-fs2_headers_test-name
Executing request Hearder Value ["test_createSingleSmallCall"]
Executing request Hearder Name x-spectrum-meta-fs2_metadata_dataDefinitionId
Executing request Hearder Value Payload
Executing request Hearder Name x-spectrum-meta-wasZippedFirst
Executing request Hearder Value false
Executing request Hearder Name x-spectrum-meta-fs2_metadata_createdBy
Executing request Hearder Value FS2
Executing request Hearder Name x-spectrum-meta-fs2_headers_test-header
Executing request Hearder Value ["test-header-value"]
Executing request Hearder Name source
Executing request Hearder Value G2_SB_DEV_INT
Executing request Hearder Name ttl
Executing request Hearder Value 50000
Executing request Hearder Name x-spectrum-meta-fs2_metadata_ttl
Executing request Hearder Value 50000
Executing request Hearder Name x-spectrum-meta-fs2_metadata_createdOn
Executing request Hearder Value Wed Feb 12 19:03:13 GMT 2020
Executing request Hearder Name timestamp
Executing request Hearder Value 1581534193366
Executing request Hearder Name Connection
Executing request Hearder Value Keep-Alive
Executing request Hearder Name Accept-Encoding
Executing request Hearder Value gzip,deflate
----------------------------------------
HTTP/1.1 302 Found
<html><head><title>Object moved</title></head><body>
<h2>Object moved to <a href="https://www.hugedomains.com/domain_profile.cfm?d=mockserver&e=com">here</a>.</h2>
</body></html>
END http call Small Encoded file
@OscarNeira please don't take this in the wrong way but I can't keep doing all these tests.
Please update when you can reproduce the issue reliably.
aledbf
on 12 Feb 2020
Hi, I reproduced some of the previous errors and found something that helps me to get the expectation rules. I tested the script at home, and the curl creating expectation rules fails when I use the cable and the wifi at the same time. I tested it for a while, and it would be nice if you can run the new version of the script-curl.sh with only wifi/cable.
Also, I notice that even if the curl does not create the rule in the mock server side, we can still verify the issue with the Nginx and the chunked transfer. I added my findings here. The tcpdump-ingress-chunk.pcap gets traffic from the incoming traffic to the mock server. This is good enough to see that the Nginx is changing the transfer encoding header. However, as described in the intro description, you can see the same results in the Nginx logs (also attached). For example, search for this id in the logs file "jesus-test-56913", and you will see that the Nginx is changing the header.
OscarNeira
on 17 Feb 2020
Hi @aledbf ,
Just checking if you got the example running in your minikube or have time to review the attached logs from the previous message?
OscarNeira
on 2 Mar 2020
Just checking if you got the example running in your minikube
No, still same issue.
or have time to review the attached logs from the previous message?
Reading the log I see
client sent invalid header line: "x-spectrum-meta-fs2_headers_test-name: ["test_createSingleSmallCall"]" while reading client request headers, client: 192.168.64.1, server: _, request: "POST /api/1/space/spectrum-aa-1/objects/jesus-test-21560%2Ftest_createSingleSmallCall%2Fload-test-1/payload HTTP/1.1"
Please set enable-underscores-in-headers: "true" in the configuration configmap
https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#enable-underscores-in-headers
Also, please update to 0.30.0. After that, please post the log again with these two changes.
aledbf
on 2 Mar 2020
Hi @aledbf , the issue remains the same with image 0.30.0 and (enable-underscores-in-headers: "true"). Here are the logs and tcp dump file.
tcpdump-ingress-chunk.pcap.zip
nginx-controller-result-underscore.log
Here how to repro the issue:
- Deleting and starting minikube
- Enable minikube addons ingress
- Add enable-underscores-in-headers: "true" to configmap
- Update deployment and change image version 0.30.0 and modify RunAsUser to (RunAsUser 101)
- Run ./k8s/start.curl.sh (deploys and setup mock server)
- The script will run the java client image 5 times
Results:
- Got OK four out of five requests with the correct Transfer encoding.
- You verify this in the logs search for "/jesus-test-27731"
OscarNeira
on 3 Mar 2020
@OscarNeira please add the annotation
nginx.ingress.kubernetes.io/proxy-request-buffering: "off" in the ingress definition
aledbf
on 3 Mar 2020
@aledbf Yep, it already had these
metadata:
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/backend-protocol: HTTP
nginx.ingress.kubernetes.io/proxy-body-size: "0"
nginx.ingress.kubernetes.io/proxy-buffering: "off"
nginx.ingress.kubernetes.io/proxy-max-temp-file-size: "0"
nginx.ingress.kubernetes.io/proxy-request-buffering: "off"
OscarNeira
on 3 Mar 2020
@OscarNeira ok, then remove all the annotations, run the tests and post the log, please.
aledbf
on 3 Mar 2020
@OscarNeira ok, then remove all the annotations, run the tests and post the log, please.
ok
OscarNeira
on 3 Mar 2020
OscarNeira
on 3 Mar 2020
@aledbf any luck?
OscarNeira
on 12 Mar 2020
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale
fejta-bot
on 10 Jun 2020
This is still a problem
OscarNeira
on 10 Jun 2020
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale
fejta-bot
on 8 Sep 2020
Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
fejta-bot
on 8 Oct 2020
Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close
fejta-bot
on 7 Nov 2020
@fejta-bot: Closing this issue.
In response to this:
Rotten issues close after 30d of inactivity.
Reopen the issue with/reopen.
Mark the issue as fresh with/remove-lifecycle rotten.Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
k8s-ci-robot
on 7 Nov 2020
never fix/ still and issue
OscarNeira
on 9 Nov 2020
Related issues
oilbeater
Β·
3Comments
yuyang0
Β·
3Comments
jwfang
Β·
3Comments
whereisaaron
Β·
3Comments
natemurthy
Β·
3Comments