Pomerium: Error DeadlineExceeded (context deadline exceeded)

[chickahoona]

What happened?

I followed the documentation. I am properly authenticated against google, yet then I receive

rpc error: code = DeadlineExceeded desc = context deadline exceeded
Request 7e0361f8-ed67-f560-e2a2-8af0f8d19955

What did you expect to happen?

I hoped I would see the dummy docker image

How'd it happen?

1. Ran docker-compose up
2. Opened https://internal.doc.something.com/
3. Saw error the error mentioned above:

What's your environment like?

• Pomerium version (retrieve with pomerium --version or /ping endpoint):
  We are using the latest docker image. Version v0.4.0+28eae36
• Server Operating System/Architecture/Cloud:
  We are using a droplet from digital ocean. As base image we used the docker image, which comes with docker-compose and so on preconfigured

What's your config.yaml?

authenticate_service_url: https://authenticate.doc.something.com
authorize_service_url: https://authorize.doc.something.com

certificate_file: "./cert.pem"
certificate_key_file: "./privkey.pem"
certificate_authority_file: "./cert.pem"

idp_provider: google
idp_client_id: 12345656-ihavebeenreplaced.apps.googleusercontent.com
idp_client_secret: IHAVEBEENREPLACED


policy:
  - from: https://internal.doc.something.com
    to: http://httpbin:80
    allowed_domains:
      - something.com

Our docker-compose.yml looks like this:

version: "3"
services:
  pomerium:
    image: pomerium/pomerium:v0.4.0
    environment:
      # Generate new secret keys. e.g. `head -c32 /dev/urandom | base64`
      - COOKIE_SECRET=IHAVEBEENREPLACED
    volumes:
      # Mount your domain's certificates : https://www.pomerium.io/docs/reference/certificates
      - /root/.acme.sh/*.doc.something.com_ecc/fullchain.cer:/pomerium/cert.pem:ro
      - /root/.acme.sh/*.doc.something.com_ecc/*.doc.something.com.key:/pomerium/privkey.pem:ro
      # Mount your config file : https://www.pomerium.io/docs/reference/reference/
      - /root/config.yaml:/pomerium/config.yaml:ro
    links:
      - "httpbin:httpbin"
    ports:
      - 443:443

  httpbin:
    image: karthequian/helloworld:latest
    ports:
      - 80:80

What did you see in the logs?

# docker-compose up
Starting root_httpbin_1 ... done

Starting root_pomerium_1 ... done

Attaching to root_httpbin_1, root_pomerium_1
pomerium_1  | {"level":"info","version":"v0.4.0+28eae36","time":"2019-10-08T12:02:49Z","message":"cmd/pomerium"}
pomerium_1  | {"level":"warn","time":"2019-10-08T12:02:49Z","message":"identity/google: no service account, cannot retrieve groups"}
pomerium_1  | {"level":"debug","route":"internal.doc.something.com","domain":"something.com","time":"2019-10-08T12:02:49Z","message":"add domain"}
pomerium_1  | {"level":"warn","addr":":5443","time":"2019-10-08T12:02:49Z","message":"internal/grpcutil: insecure server"}
pomerium_1  | {"level":"info","addr":"authorize.doc.something.com:443","time":"2019-10-08T12:02:49Z","message":"proxy/clients: grpc with insecure"}
pomerium_1  | {"level":"info","addr":":443","time":"2019-10-08T12:02:49Z","message":"internal/httputil: http server started"}
pomerium_1  | {"level":"error","ip":"79.225.95.250","user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36","req_id":"647c5aaf-9b22-ef6d-925e-59aefe9205
17","error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded","http-message":"rpc error: code = DeadlineExceeded desc = context deadline exceeded","http-code":500,"time":"2019-10-08T12:03:04Z","message":"ht
tp-error"}
pomerium_1  | {"level":"debug","ip":"79.225.95.250","user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36","req_id":"647c5aaf-9b22-ef6d-925e-59aefe9205
17","duration":10002.246291,"size":7949,"status":500,"email":"[email protected]","group":"","method":"GET","service":"all","host":"internal.doc.something.com","path":"/","time":"2019-10-08T12:03:04Z","message":"http-requ
est"}
pomerium_1  | {"level":"error","ip":"79.225.95.250","user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36","referer":"https://internal.doc.something.com/"
,"req_id":"bf084d1c-1461-dfd9-90e5-42309b42825b","error":"rpc error: code = DeadlineExceeded desc = latest connection error: connection error: desc = \"transport: Error while dialing dial tcp 68.183.76.124:443: i/o timeout\"",
"http-message":"rpc error: code = DeadlineExceeded desc = latest connection error: connection error: desc = \"transport: Error while dialing dial tcp 68.183.76.124:443: i/o timeout\"","http-code":500,"time":"2019-10-08T12:03:1
5Z","message":"http-error"}
pomerium_1  | {"level":"debug","ip":"79.225.95.250","user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36","referer":"https://internal.doc.something.com/"
,"req_id":"bf084d1c-1461-dfd9-90e5-42309b42825b","duration":10001.932749,"size":8054,"status":500,"email":"[email protected]","group":"","method":"GET","service":"all","host":"internal.doc.something.com","path":"/favicon
.ico","time":"2019-10-08T12:03:15Z","message":"http-request"}
httpbin_1   | 78.187.125.163 - - [08/Oct/2019:12:06:01 +0000] "GET / HTTP/1.1" 200 4369 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36"
httpbin_1   | 78.187.125.163 - - [08/Oct/2019:12:06:01 +0000] "GET / HTTP/1.1" 200 4369 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36"
^CGracefully stopping... (press Ctrl+C again to force)

[desimone]

Since you are running in all in one mode, try commenting out the authorize_service_url: https://authorize.doc.something.com.

[nitper]

Since you are running in all in one mode, try commenting out the authorize_service_url: https://authorize.doc.something.com.

I had the same issue after upgrading, and this fixed it for me.

[chickahoona]

Amazing, this did the trick. Thank you both for your help. As a small followup to prevent this for other people to happen:

I followed the documentation here: https://www.pomerium.io/docs/quick-start/#configure which has this line there too. One should probably remove that line from the doc.

Thank you very much again!