Ambassador: 'Not Found 404' after a freshly installed Helm chart

[caiobegotti]

Describe the bug

Just followed the Helm chart installation instructions on GKE using version 0.38.0 in a namespace called api of my cluster. After everything is running and ready I go to either the public IP of the service or to the admin IP (after port-mapping it to localhost) and I get a Not Found 404 page. There are no special steps to reproduce it.

Expected behavior
At least something not like a Not Found 404 message, any other behavior would be ok after a fresh installation of the official Helm chart:

Not Found
The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.

Versions (please complete the following information):

• Ambassador: 0.38.0
• Kubernetes environment: GKE
• Version: v1.9.7-gke.5

Additional context

Pod description:

Name:           ambassador-6f4c85b89b-vbwc7
Namespace:      api
Node:           gke-nymos-dev-default-pool-58f1925b-18bf/10.154.0.4
Start Time:     Thu, 16 Aug 2018 09:34:32 -0300
Labels:         app=ambassador
                pod-template-hash=2907416456
                release=ambassador
Annotations:    prometheus.io/port=9102
                prometheus.io/scrape=true
Status:         Running
IP:             10.12.5.158
Controlled By:  ReplicaSet/ambassador-6f4c85b89b
Containers:
  statsd-sink:
    Container ID:   docker://20ecfa85dc8aef60f91ba487415dc253328b54c111a3df29153f529b55aba348
    Image:          datawire/prom-statsd-exporter:0.6.0
    Image ID:       docker-pullable://datawire/prom-statsd-exporter@sha256:761767e0d969219d71244f9bc35954a2f0c788bd311e52a06cea69fc2eacdf89
    Port:           9102/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Thu, 16 Aug 2018 09:34:37 -0300
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from ambassador-token-fjswq (ro)
  ambassador:
    Container ID:   docker://4d2710606ffa277237f3cda983c7164fda56a7c761030e69642f3c0d18b7d23f
    Image:          quay.io/datawire/ambassador:0.38.0
    Image ID:       docker-pullable://quay.io/datawire/ambassador@sha256:20160bfa8eaa48abbf9ee4c57543119270f897ee1e3e65ee29f4bf951fd92cea
    Ports:          80/TCP, 443/TCP, 8877/TCP
    Host Ports:     0/TCP, 0/TCP, 0/TCP
    State:          Running
      Started:      Thu, 16 Aug 2018 09:34:48 -0300
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:admin/ambassador/v0/check_alive delay=30s timeout=1s period=3s #success=1 #failure=3
    Readiness:      http-get http://:admin/ambassador/v0/check_ready delay=30s timeout=1s period=3s #success=1 #failure=3
    Environment:
      AMBASSADOR_NAMESPACE:  api (v1:metadata.namespace)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from ambassador-token-fjswq (ro)
Conditions:
  Type           Status
  Initialized    True 
  Ready          True 
  PodScheduled   True 
Volumes:
  ambassador-token-fjswq:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ambassador-token-fjswq
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason                 Age   From                                               Message
  ----    ------                 ----  ----                                               -------
  Normal  Scheduled              7m    default-scheduler                                  Successfully assigned ambassador-6f4c85b89b-vbwc7 to gke-nymos-dev-default-pool-58f1925b-18bf
  Normal  SuccessfulMountVolume  7m    kubelet, gke-nymos-dev-default-pool-58f1925b-18bf  MountVolume.SetUp succeeded for volume "ambassador-token-fjswq"
  Normal  Pulling                7m    kubelet, gke-nymos-dev-default-pool-58f1925b-18bf  pulling image "datawire/prom-statsd-exporter:0.6.0"
  Normal  Pulled                 6m    kubelet, gke-nymos-dev-default-pool-58f1925b-18bf  Successfully pulled image "datawire/prom-statsd-exporter:0.6.0"
  Normal  Created                6m    kubelet, gke-nymos-dev-default-pool-58f1925b-18bf  Created container
  Normal  Started                6m    kubelet, gke-nymos-dev-default-pool-58f1925b-18bf  Started container
  Normal  Pulling                6m    kubelet, gke-nymos-dev-default-pool-58f1925b-18bf  pulling image "quay.io/datawire/ambassador:0.38.0"
  Normal  Pulled                 6m    kubelet, gke-nymos-dev-default-pool-58f1925b-18bf  Successfully pulled image "quay.io/datawire/ambassador:0.38.0"
  Normal  Created                6m    kubelet, gke-nymos-dev-default-pool-58f1925b-18bf  Created container
  Normal  Started                6m    kubelet, gke-nymos-dev-default-pool-58f1925b-18bf  Started container

Pod logs of the main container:

2018-08-16 12:34:50 kubewatch 0.38.0 INFO: generating config with gencount 1 (0 changes)
/usr/lib/python3.6/site-packages/pkg_resources/__init__.py:1298: UserWarning: /ambassador is writable by group/others and vulnerable to attack when used with get_resource_filename. Consider a more secure location (set with .set_extraction_path or the PYTHON_EGG_CACHE environment variable).
  warnings.warn(msg, UserWarning)
2018-08-16 12:34:50 kubewatch 0.38.0 INFO: Scout reports {"latest_version": "0.38.0", "application": "ambassador", "notices": [], "cached": false, "timestamp": 1534422890.548433}
[2018-08-16 12:34:51.707][12][info][config] source/server/configuration_impl.cc:50] loading 0 static secret(s)
[2018-08-16 12:34:51.734][12][info][upstream] source/common/upstream/cluster_manager_impl.cc:132] cm init: all clusters initialized
[2018-08-16 12:34:51.734][12][info][config] source/server/configuration_impl.cc:60] loading 1 listener(s)
[2018-08-16 12:34:51.760][12][info][config] source/server/configuration_impl.cc:94] loading tracing configuration
[2018-08-16 12:34:51.760][12][info][config] source/server/configuration_impl.cc:116] loading stats sink configuration
AMBASSADOR: starting diagd
AMBASSADOR: starting Envoy
AMBASSADOR: waiting
PIDS: 16:diagd 17:envoy 18:kubewatch
starting hot-restarter with target: /ambassador/start-envoy.sh
forking and execing new child process at epoch 0
forked new child process with PID=19
[2018-08-16 12:34:52.228][19][info][main] source/server/server.cc:183] initializing epoch 0 (hot restart version=10.200.16384.127.options=capacity=16384, num_slots=8209 hash=228984379728933363 size=2654312)
[2018-08-16 12:34:52.228][19][info][main] source/server/server.cc:185] statically linked extensions:
[2018-08-16 12:34:52.228][19][info][main] source/server/server.cc:187]   access_loggers: envoy.file_access_log,envoy.http_grpc_access_log
[2018-08-16 12:34:52.229][19][info][main] source/server/server.cc:190]   filters.http: envoy.buffer,envoy.cors,envoy.ext_authz,envoy.fault,envoy.filters.http.header_to_metadata,envoy.filters.http.jwt_authn,envoy.filters.http.rbac,envoy.grpc_http1_bridge,envoy.grpc_json_transcoder,envoy.grpc_web,envoy.gzip,envoy.health_check,envoy.http_dynamo_filter,envoy.ip_tagging,envoy.lua,envoy.rate_limit,envoy.router,envoy.squash,extauth
[2018-08-16 12:34:52.229][19][info][main] source/server/server.cc:193]   filters.listener: envoy.listener.original_dst,envoy.listener.proxy_protocol,envoy.listener.tls_inspector
[2018-08-16 12:34:52.229][19][info][main] source/server/server.cc:196]   filters.network: envoy.client_ssl_auth,envoy.echo,envoy.ext_authz,envoy.filters.network.thrift_proxy,envoy.http_connection_manager,envoy.mongo_proxy,envoy.ratelimit,envoy.redis_proxy,envoy.tcp_proxy
[2018-08-16 12:34:52.229][19][info][main] source/server/server.cc:198]   stat_sinks: envoy.dog_statsd,envoy.metrics_service,envoy.statsd
[2018-08-16 12:34:52.229][19][info][main] source/server/server.cc:200]   tracers: envoy.dynamic.ot,envoy.lightstep,envoy.zipkin
[2018-08-16 12:34:52.229][19][info][main] source/server/server.cc:203]   transport_sockets.downstream: envoy.transport_sockets.capture,raw_buffer,tls
[2018-08-16 12:34:52.229][19][info][main] source/server/server.cc:206]   transport_sockets.upstream: envoy.transport_sockets.capture,raw_buffer,tls
[2018-08-16 12:34:52.269][19][info][config] source/server/configuration_impl.cc:50] loading 0 static secret(s)
[2018-08-16 12:34:52.280][19][info][upstream] source/common/upstream/cluster_manager_impl.cc:132] cm init: all clusters initialized
[2018-08-16 12:34:52.280][19][info][config] source/server/configuration_impl.cc:60] loading 1 listener(s)
[2018-08-16 12:34:52.298][19][info][config] source/server/configuration_impl.cc:94] loading tracing configuration
[2018-08-16 12:34:52.298][19][info][config] source/server/configuration_impl.cc:116] loading stats sink configuration
[2018-08-16 12:34:52.299][19][info][main] source/server/server.cc:378] all clusters initialized. initializing init manager
[2018-08-16 12:34:52.299][19][info][config] source/server/listener_manager_impl.cc:781] all dependencies initialized. starting workers
[2018-08-16 12:34:52.300][19][info][main] source/server/server.cc:398] starting main dispatch loop
/usr/lib/python3.6/site-packages/pkg_resources/__init__.py:1298: UserWarning: /ambassador is writable by group/others and vulnerable to attack when used with get_resource_filename. Consider a more secure location (set with .set_extraction_path or the PYTHON_EGG_CACHE environment variable).
  warnings.warn(msg, UserWarning)
2018-08-16 12:34:55 diagd 0.38.0 [P16TMainThread] INFO: thread count 5, listening on 0.0.0.0:8877
[2018-08-16 12:34:55 +0000] [16] [INFO] Starting gunicorn 19.8.1
[2018-08-16 12:34:55 +0000] [16] [INFO] Listening at: http://0.0.0.0:8877 (16)
[2018-08-16 12:34:55 +0000] [16] [INFO] Using worker: threads
[2018-08-16 12:34:55 +0000] [44] [INFO] Booting worker with pid: 44
2018-08-16 12:34:55 diagd 0.38.0 [P44TMainThread] INFO: Starting periodic updates
[2018-08-16 12:35:02.300][19][info][main] source/server/drain_manager_impl.cc:63] shutting down parent after drain
ACCESS [2018-08-16T12:35:30.814Z] "GET / HTTP/1.1" 404 NR 0 0 0 - "-" "curl/7.54.0" "ba3dbc27-0dde-40fd-b231-7d747bf04605" "35.234.131.150" "-"
ACCESS [2018-08-16T12:35:59.750Z] "GET / HTTP/1.1" 404 NR 0 0 0 - "-" "curl/7.54.0" "e8517420-8a5d-44e7-b888-a05daff4a580" "35.234.131.150" "-"
ACCESS [2018-08-16T12:36:25.082Z] "HEAD / HTTP/1.1" 404 NR 0 0 0 - "-" "curl/7.54.0" "b2a29702-b229-4e30-af04-340732dcbf35" "35.234.131.150" "-"
ACCESS [2018-08-16T12:36:53.180Z] "GET / HTTP/1.1" 404 NR 0 0 0 - "-" "curl/7.54.0" "26479127-7b62-4769-a651-aed7f64da944" "35.234.131.150" "-"
ACCESS [2018-08-16T12:36:56.690Z] "GET / HTTP/1.1" 404 NR 0 0 0 - "-" "curl/7.54.0" "bc427559-0d50-4f37-b83e-ac9116842735" "35.234.131.150" "-"
ACCESS [2018-08-16T12:36:59.848Z] "GET / HTTP/1.1" 404 NR 0 0 0 - "-" "curl/7.54.0" "d1726392-ab70-4a33-ad94-4570a808eaf2" "35.234.131.150" "-"

Public service (up until some point in the future, for debugging if you need):

kubectl get svc ambassador -n api
NAME         TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)                      AGE
ambassador   LoadBalancer   10.15.245.39   35.234.131.150   80:32627/TCP,443:32689/TCP   13m

[nbkrause]

Hi @caio1982,

Thanks for reaching out. Nothing looks out of the ordinary with the deployment.

I went ahead and tested the helm install just in case and it completed without issue. I was able to create and map services to and access then through ambassador.

After everything is running and ready I go to either the public IP of the service or to the admin IP (after port-mapping it to localhost) and I get a Not Found 404 page.

From this it sounds like you are expecting to be able to get a response from curl http://<ambassador-external-ip>/ without any / level mapping. This will result in a 404 as ambassador has no service to point the request to.

I would love to help you set up ambassador. Join us on slack https://d6e.co/slack and we can help get you up and running, or help you troubleshoot this further if my understanding of your issue is wrong.

Cheers!

[harindaka]

@caiobegotti did you find a solution to this?

[caiobegotti]

@harindaka yeah... I moved on to Istio, as to be quite frank I found support and community around Ambassador pretty lacking compared to Istio, which is more complicated by a huge margin but you can get things done eventually with some actual help. Sorry for not being really helpful :-(

[harindaka]

@caiobegotti and anyone who's facing the same issue in the future you are probably trying to access the admin ui via root (/). Instead try /ambassador/v0/diag/. This fixed it for me.