Falco: Crash with Invalid JSON error while parsing container info

[Srinivas11789]

What happened:
Falco crashes with Runtime error: Invalid JSON encountered while parsing container info resulting in CrashLoopBackOff pod state

What you expected to happen:

• Parse container info without error
• Throw error and run without crash ( possible fallback? )

How to reproduce it (as minimally and precisely as possible):

• Make a k8s deployment with large number of ports (> 1000)
• Example nginx deployment [This is a dumb example configuration just to recreate the issue]

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deploy
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx
          ports:
          - { containerPort: 8080, name: server1}
          - { containerPort: 8081, name: server2}
          - { containerPort: 8082, name: server3}
          - { containerPort: 8083, name: server4}
          - { containerPort: 50000, hostPort: 50000, protocol: UDP, name: port1 }
          - { containerPort: 50001, hostPort: 50001, protocol: UDP, name: port2 }
          - { containerPort: 50002, hostPort: 50002, protocol: UDP, name: port3 }
          - { containerPort: 50003, hostPort: 50003, protocol: UDP, name: port4 }
          - { containerPort: 50004, hostPort: 50004, protocol: UDP, name: port5 }
          - { containerPort: 50005, hostPort: 50005, protocol: UDP, name: port6 }
          - { containerPort: 50006, hostPort: 50006, protocol: UDP, name: port7 }
          - { containerPort: 50007, hostPort: 50007, protocol: UDP, name: port8 }
          - { containerPort: 50008, hostPort: 50008, protocol: UDP, name: port9 }
          - { containerPort: 50009, hostPort: 50009, protocol: UDP, name: port10 }
          ...
          ...
          - { containerPort: 50998, hostPort: 50998, protocol: UDP, name: port999 }

• Deploy Falco on the same node and check falco logs
• fyi References,
  
  
  
  • We need to explicitly list all the ports as mentioned at https://github.com/kubernetes/kubernetes/issues/23864
  
  
  • Example: https://kubernetes.io/docs/tasks/run-application/run-stateless-application-deployment/
  
  

Anything else we need to know?:

• values.yaml for parameters

ebpf:
  # Enable eBPF support for Falco - This allows Falco to run on Google COS.
  enabled: true

  settings:
    # Needed to enable eBPF JIT at runtime for performance reasons.
    # Can be skipped if eBPF JIT is enabled from outside the container
    hostNetwork: true
    # Needed to correctly detect the kernel version for the eBPF program
    # Set to false if not running on Google COS
    mountEtcVolume: true

falco:
  # Output format
  jsonOutput: true
  logLevel: notice
  # Slack alerts
  programOutput:
    enabled: true
    keepAlive: false
    program: "\" jq '{text: .output}' | curl -d @- -X POST https://hooks.slack.com/services/XXXX\""

Environment:

• Falco version (use falco --version): falco version 0.15.3
• System info

{
  "machine": "x86_64",
  "nodename": "gke-test-default-pool-3d67c0cd-n8b4",
  "release": "4.14.119+",
  "sysname": "Linux",
  "version": "#1 SMP Tue May 14 21:04:23 PDT 2019"
}

• Cloud provider or hardware configuration: GCP
• OS (e.g: cat /etc/os-release):

BUILD_ID=10895.242.0
NAME="Container-Optimized OS"

• Kernel (e.g. uname -a):

Linux gke-test-default-pool-3d67c0cd-dlng 4.14.119+ #1 SMP Tue May 14 21:04:23 PDT 2019 x86_64 Intel(R) Xeon(R) CPU
 @ 2.20GHz GenuineIntel GNU/Linux

• Install tools (e.g. in kubernetes, rpm, deb, from source): Kubernetes (helm)
• Others:

[fntlnz]

Hi @Srinivas11789 good catch! Thanks for opening the issue, we will try to reproduce it.

[fntlnz]

/assign @fntlnz
/assign @leodido

[vsimon]

Hi I'm hitting this issue as well. Thanks for looking in to it.

[vsimon]

any update on this?

[fntlnz]

@vsimon @Srinivas11789 this is in the backlog, will address shortly, in the meanwhile if anyone has more details please post here! ❤️

[fntlnz]

@Srinivas11789 I am not able to reproduce the error you are reporting for the parser, however I acknowledge that not checking the error at the event loop level can lead to falco crashing.

I couldn't try with ports because k8s didn't allow me to create a container with that many ports caused by a network sandboxing error.

The first definition contains around 3k env variables, the other around 10k but k8s didn't allow me to load it.

As you suggested we need two fixes for this:

• The first is handling parsing errors, those are not handled right now and lead falco to crash and die, the fix for this is in #746
• The second is to understand the root cause of why the JSON parser breaks for you and fix it.

So since I was not able to reproduce the parsing error you are reporting I can't address the second fix we have to do, please continue providing feedback to help fixing this 👼

Having a complete reproducible yaml definition that breaks falco would help.

@vsimon you can probably help too.

---

My kubernetes version (compiled from master):

Client Version: version.Info{Major:"", Minor:"", GitVersion:"v0.0.0-master+$Format:%h$", GitCommit:"81a61ae0e37143299ee5947a6c2c5195ec5f72ae", GitTreeState:"clean", BuildDate:"2019-05-20T03:59:28Z", GoVersion:"go1.12.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.0", GitCommit:"641856db18352033a0d96dbc99153fa3b27298e5", GitTreeState:"clean", BuildDate:"2019-06-09T08:06:25Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}

[Srinivas11789]

@fntlnz Thanks for the update and a possible fix. 👍

Agree that the first fix would solve the crash but having this deployment configuration would keep triggering the JSON parse errors for that container. So I think it would be better if we also try to fix the root cause. I can help with more feedback.

I tried to reproduce this again today (same environment as mentioned before) and still see the issue occurring, I added some issue reproducible ready files here that I used. Let me know if that helps. 🤔

@vsimon Thanks for the follow up.

[fntlnz]

Thanks for the updated files to reproduce @Srinivas11789 - will try to see if I can trigger that case on my environment.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[vsimon]

/no-stale

[leodido]

/milestone 1.0.0

Moving this to 1.0.0 because we are re-designing the input interface (via gRPC). Once we have that we'll use the k8s go client directly plus go json package. Which in turn means that we'll use the same code k8s uses solving this bug.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[vsimon]

/no-stale

[santi-asapp]

Hi, I have a similar issue with my EKS cluster (v1.14) when trying to parse a JSON from helm chart deployment. "Runtime error: Invalid JSON encountered while parsing container info:"

I'm running>
Falco version: 0.21.0-23+35691b0
Driver version: be1ea2d9482d0e6e2cb14a0fd7e08cbecf517f94

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Srinivas11789]

/no-stale

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Issues labeled "cncf", "roadmap" and "help wanted" will not be automatically closed. Please refer to a maintainer to get such label added if you think this should be kept open.

[vsimon]

/no-stale

[leogr]

/help

[poiana]

@leogr:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[vpharabot]

Hello
I have same issue while container info json (with inspect) is correct. But i get same "Crash with Invalid JSON error while parsing container info" error. Is there any workaround for that ?

[leogr]

Hello
I have same issue while container info json (with inspect) is correct. But i get same "Crash with Invalid JSON error while parsing container info" error. Is there any workaround for that ?

What's Falco version? Could you provide detailed steps to reproduce the problem?

[vpharabot]

I'm using latest release falco:0.25.0
I'm not currently sure how to reproduce in my case yet.

[vpharabot]

I'm able to reproduce
If you have a pod with 62K character in annotations, when falco try to parse the container info, falco will crash
The limit might be lower, but at least with 62K characters, i'm able to reproduce

[adamzr]

This is causing me an issue with a Java Spring Boot image produced by Spring Boot Maven Plugin's Docker Image generation feature. That feature uses Paketo buildpacks. This is likely a problem for every Java image produced by Paketo buildpacks.

[adamzr]

@fntlnz @leodido Try the Docker image nebhale/spring-music that's a typical Java Spring Docker image created by Poketo buildpacks. There is a lot of JSON in the labels. I think this will cause Falco to crash.

[PhilipSchmid]

Hi guys,

I just run into the same issue 😢.

Working image:

user@node1:~$ docker inspect image_with_io_buildpacks_build_metadata_label:v1 -f '{{json .Config.Labels}}' | wc -m
58378

NOT working image:

user@node1:~$ docker inspect image_with_io_buildpacks_build_metadata_label:v2 -f '{{json .Config.Labels}}' | wc -m 
596846

@leodido Is there any possibility this will be fixed prior to Falco release 1.0.0?

Thanks & regards,
Philip

[Annegies]

On our container platform we also have some containers running that were built with some kind of buildpack resulting in insanely huge labels on the docker images. And Falco crashes when trying to parse them.
These labels are ridiculous but Falco should also be able to handle it in my opinion.

What is weird though is that this starting happening when we upgraded from 0.26.2 to 0.27.0. It's running fine with 0.26.2.
I couldn't find a change in de changelog that could explain this?

[rbkaspr]

I'm also encountering this error in Falco 0.27.0 running on EKS 1.18.9-eks-d1db3c. Has there been any progress made towards solving this?

[rbkaspr]

Additional context, the only container that seemed to be triggering the issue was any instance of the micrometermetrics/prometheus-rsocket-proxy image. Removing all pods running that image allows Falco to run normally.

Hopefully that helps