Skaffold: `debug` should support debugging `python -m module`-style command

[briandealwis]

debug doesn't properly support debugging Python modules: we transform the container command-line to insert -mptvsd with some ptvsd-specific arguments. We don't rewrite the command-line if it already had a -m argument to start a module, such as in our hot-reload/python example.

[briandealwis]

Already supported by ptvsd. Move along, nothing to see here.

[briandealwis]

I've dug into it more and now understand my original reason for this issue.  The problem can be seen when running skaffold debug on examples/hot-reload/python where the Python image fails with

/usr/local/bin/python: No module named flask

The problem is that skaffold debug sets PYTHONUSERBASE to the debug-helpers that have been installed in /dbg and then rewrites the command line to run

    python -mptvsd --host localhost -m flask run --host=0.0.0.0

But in this image, flask and its dependencies are installed in ~/.local/lib/python3.7/site-packages/. PEP-370 normally causes $HOME/.local to be used as the user site (and thus added to sys.path), unless PYTHONUSERBASE is set. And so flask is not found.

[lboudard]

@briandealwis, encountering same issue on our end, do you have any wokraround this?
having set PYTHONPATH env to site-packages doesn't seem to help
Thanks!

[briandealwis]

You can decorate the pod / pod-template by hand.

To cause debug to skip the pod, add an annotation like the following:

  annotations:
    debug.cloud.google.com/config: "{}"

To enable Python debugging using ptvsd, add ptvsd to your requirements.txt, and then amend your k8s manifest to invoke ptvsd with your app. You can use skaffold debug -v debug to see the amended manifests.

For example, with Skaffold's hot-reload/python example, I made the following changes to the Python's deployment:

diff --git examples/hot-reload/k8s/python.yaml examples/hot-reload/k8s/python.yaml
index 64bb21d9b..01b8f88f5 100644
--- examples/hot-reload/k8s/python.yaml
+++ examples/hot-reload/k8s/python.yaml
@@ -21,11 +21,16 @@ spec:
     metadata:
       labels:
         app: python
+      annotations:
+        debug.cloud.google.com/config: '{"python":{"artifact":"python-reload","runtime":"python","workingDir":"/home/python","ports":{"dap":5678}}}'
     spec:
       securityContext:
         runAsUser: 1000
       containers:
       - name: python
         image: python-reload
+        args: [python, -mptvsd, --host, 0.0.0.0, --port, "5678", -m, flask, run, --host=0.0.0.0]
         ports:
         - containerPort: 5000
+        - containerPort: 5678
+          name: dap
diff --git examples/hot-reload/python/requirements.txt examples/hot-reload/python/requirements.txt
index 2c34d09e1..a0aa0bd4a 100644
--- examples/hot-reload/python/requirements.txt
+++ examples/hot-reload/python/requirements.txt
@@ -1 +1,2 @@
 Flask==1.0
+ptvsd

[briandealwis]

Not forgotten.

[gsquared94]

@briandealwis if you're not prioritizing this, we can downgrade the priority

[lallinger-arbeit]

For anyone stumbling across this using OpenShift: You may not remove the securityContext! Just add the default servicaccount of the namespace you are using to the anyuid SCC.
In addition to briandealwis solution i needed to specify ptvsd==5.0.0a8 in the requirements.txt to make it work.
For easy use also specify this in the ports section of the python service:

  - port: 5000
    name: web
  - port: 5678
    name: dap

My working launch.json for VSC:

{
    "configurations": [

        {
            "name": "Python: attach",
            "type": "python",
            "request": "attach",
            "port": 5678,
            "host": "localhost",
            "pathMappings": [
                {
                    "localRoot": "${workspaceFolder}/python",
                    "remoteRoot": "/home/python"
                }
            ]
        }
    ]
}

Then just use skaffold dev --port-forward and you should be fine

[tejal29]

This is nearing slo am wondering if we further reduce the priority if not actively worked upon.

[mr-bjerre]

is it possible to debug FastAPI applications?

[briandealwis]

@mr-bjerre Python offers many corner cases when configuring for debugging 😢 In the mean time, you can configure your image to start your app manually using ptvsd or debugpy, and then modify your resource's podspec to include an annotation like the following (modified appropriately to match your container configuration):

metadata:
      annotations:
        debug.cloud.google.com/config: '{"python":{"artifact":"python-reload","runtime":"python","workingDir":"/home/python","ports":{"dap":5678}}}'

skaffold debug will honour and propagate those settings to the IDE.

[mr-bjerre]

@briandealwis thanks for getting back quickly.

I actually tried your hot reload example with the edits you provided in an earlier comment but without any luck.

Does there exist some documentation on the annotation that you are suggesting?

EDIT:

Link to docs on the annotations: https://skaffold.dev/docs/workflows/debug/#workload-annotations

[mr-bjerre]

Ah wait I put the annotations on root metadata and not in the template metadata. Now the application works but

1. vscode does not apply the correct runtime (autocompletion is using my local python runtime)
2. breakpoints in vscode are ignored
3. no hot reloading

Is that expected?

❯ skaffold debug                                                           
Listing files to watch...
 - node-example
 - python-reload
Generating tags...
 - node-example -> node-example:v1.17.1-40-ge9c9bdb7f
 - python-reload -> python-reload:v1.17.1-40-ge9c9bdb7f-dirty
Checking cache...
 - node-example: Found Locally
 - python-reload: Found Locally
Tags used in deployment:
 - node-example -> node-example:12b854a5f1edeb68266c18c5716a1267d533e995df824d54d994127a577262a3
 - python-reload -> python-reload:da2b0442896f05d419a55194af827c362f605ea5fe5ac576c525a01e34a87010
Starting deploy...
 - service/node created
 - deployment.apps/node created
 - service/python created
 - deployment.apps/python created
Waiting for deployments to stabilize...
 - deployment/python is ready. [1/2 deployment(s) still pending]
 - deployment/node is ready.
Deployments stabilized in 3.551932779s
Press Ctrl+C to exit
Not watching for changes...
[install-nodejs-debug-support] Installing runtime debugging support files in /dbg
[install-nodejs-debug-support] Installation complete
[python]  * Serving Flask app "src/app.py"
[python]  * Environment: production
[python]    WARNING: Do not use the development server in a production environment.
[python]    Use a production WSGI server instead.
[python]  * Debug mode: off
[python]  * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
[node] 
[node] > [email protected] production /home/node/app
[node] > node src/index.js
[node] 
[node] Debugger listening on ws://0.0.0.0:9229/ef4acf03-b97a-47cc-a024-4a97617f1b2e
[node] For help, see: https://nodejs.org/en/docs/inspector
[node] Example app listening on port 3000!
[python] 172.17.0.1 - - [18/Dec/2020 10:45:45] "GET / HTTP/1.1" 200 -
[python] 172.17.0.1 - - [18/Dec/2020 10:45:45] "GET /favicon.ico HTTP/1.1" 404 -
[python] 172.17.0.1 - - [18/Dec/2020 10:45:57] "GET / HTTP/1.1" 200 -