Zero-to-jupyterhub-k8s: Upload Error Invalid response: 413

[julienchastang]

I administer a JupyterHub running this project. I have a user running into the problem below when trying to upload a file.

Large file size warning: The file size is 34 MB. Do you still want to upload it?
When she selects "Upload", she gets the following message:
Upload Error: Invalid response: 413 Request Entity Too Large

I searched to find there were people reporting similar problems when trying to save a large notebook. I tried the solutions associated with that problem (e.g., ingress.kubernetes.io/proxy-body-size: 64m) but that did not solve the issue.

Ping @zonca

Contents of config.yaml

hub:
  cookieSecret: "xxx"


ingress:
  enabled: true
  annotations:
    kubernetes.io/tls-acme: "true"
  hosts:
    - jupyterhub.example.com
  tls:
      - hosts:
         - jupyterhub.example.com
        secretName: secret

proxy:
  secretToken: "xxx"

singleuser:
  startTimeout: 600
  memory:
    guarantee: 2G
    limit: 2G
  cpu:
    guarantee: 0.5
    limit: 0.75
  defaultUrl: "/lab"
  image:
    name: hub
    tag: ffffff
  lifecycleHooks:
    postStart:
      exec:
          command:
            - "sh"
            - "-c"
            - >
              cp /README_FIRST.ipynb /home/jovyan

hub:
  extraConfig: |-
    c.Spawner.cmd = ['jupyter-labhub']

auth:
  type: github
  github:
    clientId: "xxx"
    clientSecret: "xxx"
    callbackUrl: "https://jupyterhub.example.com:443/oauth_callback"
  admin:
    users:
      - admin

[betatim]

Hi there 👋!

Can move this discussion to http://discourse.jupyter.org/ (where most of the people who hang out here also hang out, as well as more people). We are trying to streamline the different discussion places and want to use the issues on GitHub repos for technical discussions on how to change the contents of the repo. More general discussions should go to discourse so that they are easier to find, better indexed by google and generally more accessible than being hidden in the bowls of a GitHub repository ;)

Thanks!

[manics]

What ingress controller are you using? Some of the ingress docs are unclear. For instance for the current Nginx ingress you need nginx.ingress.kubernetes.io/proxy-body-size instead of ingress.kubernetes.io/proxy-body-size: https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#custom-max-body-size

[julienchastang]

Hey that worked!

nginx.ingress.kubernetes.io/proxy-body-size: 64m

is what you need. You still have the large file upload warning, but that's not necessarily a bad thing.

Thanks for your help!

[desmondyeoh]

Hey that worked!

nginx.ingress.kubernetes.io/proxy-body-size: 64m

is what you need. You still have the large file upload warning, but that's not necessarily a bad thing.

Thanks for your help!

Hey @julienchastang I'm struggling with this issue now..

I'm curious where did u set this?
nginx.ingress.kubernetes.io/proxy-body-size: 64m

Is it in jupyterhub's config.yaml u showed above?

Like so?

ingress:
  enabled: true
  annotations:
    kubernetes.io/tls-acme: "true"
    nginx.ingress.kubernetes.io/proxy-body-size: 64m

I tried this but it isn't working. From kubectl describe ingress -n jhub, I can see that the annotation is there. But I'm still getting 413 when uploading 2MB notebook.

It's been a while since you last posted, thanks a lot if you're reading this.

[julienchastang]

Here is the full config file. Does that help?

[desmondyeoh]

Hi @julienchastang , thank you so much for the config file. It helped me to reflect the problem to my manager.
So it turns out that there were 2 nginx in our config. I.e. our infra has an nginx before reaching the k8s ingress-nginx.
After configuring the first nginx, it's working fine now. :D

[flasheras]

Adding nginx.ingress.kubernetes.io/proxy-body-size: 64m to the config.yaml solved the problem for me. It was tested with 0.8.2 and 0.9.1 helm charts