Skaffold: `debug` should provide some notice that file changes are ignored

[fouadkada]

Expected behavior

When code change, a code reload is made and the changed code reflects in the output.

Actual behavior

by running skaffold with verbosity DEBUG, I can see that skaffold detected some file changed but the result of the code doesn't appear as if the changes were applied.

Information

• Skaffold version: 1.12.0
• Operating system: Ubuntu 20.04 with minikube 1.11and kubectl 1.18
• Contents of skaffold.yaml:

apiVersion: skaffold/v2beta5
kind: Config
metadata:
  name: url-shortener
build:
  artifacts:
    - image: url_shortener
      context: shortener
      docker:
        dockerfile: build/docker/Dockerfile.dev
        noCache: false
deploy:
  kubectl:
    manifests:
      - shortener/deployments/kubernetes/shortener.yaml

• Contents of Dockerfile.dev:

FROM golang:1.14 AS builder

WORKDIR /app

COPY go.mod go.sum ./
RUN go mod download
COPY . .

RUN CGO_ENABLED=0 go build -o /app.o ./cmd/shortener/shortener.go

FROM alpine:3.12
COPY --from=builder /app.o ./
COPY --from=builder /app ./
EXPOSE 3000
ENV GOTRACEBACK=all
CMD ["./app.o"]

• Contents of shortener.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: url-shortener-deployment
spec:
  selector:
    matchLabels:
      app: url-shortener
  template:
    metadata:
      labels:
        app: url-shortener
    spec:
      containers:
        - name: url-shortener
          image: url_shortener
          ports:
            - containerPort: 3000
---
apiVersion: v1
kind: Service
metadata:
  name: url-shortener-service
spec:
  selector:
    app: url-shortener
  ports:
    - port: 3000
      nodePort: 30000
  type: NodePort

skaffold debug message output

time="2020-07-05T22:51:08+02:00" level=debug msg="Found dependencies for dockerfile: [{go.mod /app true} {go.sum /app true} {. /app true}]"
time="2020-07-05T22:51:08+02:00" level=info msg="files modified: [shortener/internal/handler/rest/rest.go]"

Steps to reproduce the behavior

1. run skaffold dev
2. make some code changes
3. check the logs to see that skaffold detected the change
4. run part of the code to see that the output is the newly changes code

[briandealwis]

I believe you are accessing your service via the service port-forward (port 3000)? I've noticed a staleness issue around services and deployments too, but I think it's working as intended by k8s. Basically Skaffold pushes the built container and redeploys the deployment; k8s notices the change and starts the new container, and then tears down the old container. But until the old container is terminated, the service continues sending requests to the old container.

[fouadkada]

After more testing, I feel that the problem is with intellij.
During a debug session, using the "develop on kubernetes" configuration provided by the cloud code plugin it seems that there's no rebuild happening after code changes.

If you take this repository for example skaffold_test and run it in the CLI using skaffold dev then whenever code changes happen, a rebuild is triggered and changes are reflected.
When debugging from the IDE (intellij idea), code changes don't seem to trigger a rebuild.

[briandealwis]

The behaviour you’re observing is working as intended.

Cloud Code’s debugging uses skaffold debug, not skaffold dev. Although they are similar, debug deliberately disables Skaffold’s file-watcher as it leads to puzzling and frustrating situations where debug sessions are torn down as containers are rebuilt and redeployed from seemingly innocuous changes.

We’re looking to exposing ways to allow triggering rebuilds and redeploys during debugging.

[briandealwis]

I've separated my theory on stale data from services to #4522 and retitled this issue. It appears that @fouadkada is using skaffold debug under the hood. We should be able to inform the user that file changes are ignored when the file-watcher is disabled as is the default with debug.

[wojtek-viirtue]

@briandealwis Is there a way to specifically opt-in for the file-watcher when running a skaffold debug ? I think it's pretty common to know to restart a debugging session after a code change given "staleness".

Currently, I have to stop skaffold and restart after a code change and then restart the debugging session anyway.

One can work around this for now by running skaffold dev and bake the debugging prerequisites into the image which isn't ideal.

[briandealwis]

@wojtek-viirtue yes, this ability was added in #4089. You can use skaffold debug --auto-build=true --auto-deploy=true, and if your builds are set up for it --auto-sync=true too.

[wojtek-viirtue]

@briandealwis Thank you!

[Rio]

I've run into the --auto-* flags just now. Any particular reason they're hidden? Are they not the recommended way to enable this typo of functionality?

[wojtek-viirtue]

I agree with @Rio, it would be great if this was part of the skaffold debug --help output