Falco: SSL server failed to start

[WeiZhang555]

What happened:

I enabled SSL support in Falco configuration file but then the server failed to start. Its logs:

** CONFIG_BPF_JIT enabled and/or use --net=host if      **
** running inside a container.                          **
**********************************************************
* BPF probe located, it's now possible to start sysdig
Thu Sep 26 06:37:19 2019: Falco initialized with configuration file /etc/falco/falco.yaml
Thu Sep 26 06:37:19 2019: Loading rules from file /etc/falco/falco_rules.yaml:
Thu Sep 26 06:37:20 2019: Loading rules from file /etc/falco/falco_rules.local.yaml:
Thu Sep 26 06:37:20 2019: Loading rules from file /etc/falco/k8s_audit_rules.yaml:
Thu Sep 26 06:37:20 2019: Starting internal webserver, listening on port 443 (SSL)
Thu Sep 26 06:37:20 2019: Runtime error: Could not create embedded webserver: null context when constructing CivetServer. Possible problem binding to port.. Exiting.

What you expected to happen:

Falco server can start successfully with SSL support.

How to reproduce it (as minimally and precisely as possible):

1. Following official docs: https://falco.org/docs/installation/ to install Falco in daemonset .
2. modify falco-config/falco.yaml and open the SSL flag:

webserver:
  enabled: true
  listen_port: 443 
  k8s_audit_endpoint: /k8s-audit
  ssl_enabled: true
  ssl_certificate: /etc/falco/falco.pem

The SSL certificate is generated following the comment:

$ openssl req -newkey rsa:2048 -nodes -keyout key.pem -x509 -days 365 -out certificate.pem
$ cat certificate.pem key.pem > falco.pem
$ sudo cp falco.pem /etc/falco/falco.pem

After that the daemonset can't start successfully, but it can start after I set "ssl_enabled: false".

And the falco.pem is passed to POD via ConfigMap "falco-config"

Anything else we need to know?:

Environment:

• Falco version (use falco --version): image "falcosecurity/falco:latest"
• System info
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release): 18.04.1 LTS (Bionic Beaver)
• Kernel (e.g. uname -a): 4.15.0-51-generic
• Install tools (e.g. in kubernetes, rpm, deb, from source): Kubernetes
• Others:

[fntlnz]

Thanks for reporting @WeiZhang555 - Looking into this

[fntlnz]

Debugged this for a bit and found out that there's a problem linking the openssl library.

[1569924431] [error] [client ] initialize_ssl: error loading library libcrypto.so

[leodido]

We confirm the bug exists. Me and @fntlnz working on a solution.

[fntlnz]

Ok me and @leodido have a fix ready for this! Sending a PR soon.

[fntlnz]

Ok taking this back, it's not that simple. Civetweb doesn't like our openssl statically built. Need more time to fix this.

[fntlnz]

Working on this branch: https://github.com/falcosecurity/falco/compare/860-fix-civet-ssl

[mfdii]

@WeiZhang555 please ensure there is nothing listening on port 443.

SSL works for me with 0.17.1

# Server Side
vagrant@vagrant:~$ sudo falco
Tue Oct  1 18:45:59 2019: Falco initialized with configuration file /etc/falco/falco.yaml
Tue Oct  1 18:45:59 2019: Loading rules from file /etc/falco/falco_rules.yaml:
Tue Oct  1 18:45:59 2019: Loading rules from file /etc/falco/falco_rules.local.yaml:
Tue Oct  1 18:45:59 2019: Loading rules from file /etc/falco/k8s_audit_rules.yaml:
Tue Oct  1 18:45:59 2019: Starting internal webserver, listening on port 8765 (SSL)

# CLient Side
openssl s_client -connect 127.0.0.1:8765
CONNECTED(00000003)
depth=0 C = US, ST = OH, L = WHEREVS, O = Stuff INC, CN = 127.0.0.1
verify error:num=18:self signed certificate
verify return:1
depth=0 C = US, ST = OH, L = WHEREVS, O = Stuff INC, CN = 127.0.0.1
verify return:1
---
Certificate chain
 0 s:/C=US/ST=OH/L=WHEREVS/O=Stuff INC/CN=127.0.0.1
   i:/C=US/ST=OH/L=WHEREVS/O=Stuff INC/CN=127.0.0.1
---
....
---
GET /k8s_audit

HTTP/1.1 400 Bad Request
Cache-Control: no-cache, no-store, must-revalidate, private, max-age=0
Pragma: no-cache
Expires: 0
Content-Type: text/plain; charset=utf-8
Date: Tue, 01 Oct 2019 18:48:58 GMT
Connection: close

Error 400: Bad Request
Bad requestclosed

[fntlnz]

@mfdii that happens when libcrypto is present in the system at the right path . However since we embed a static copy of openssl which contains libcrypto I don't think we should use both the system one and our embedded one, we've got to choose.

[leodido]

/area build

[WeiZhang555]

Thanks for your quick response 😄

[mac-abdon]

Is it possible to get an update on this? It looks like the issue is still present in falcosecurity/falco:0.18.0. This issue prevents operators from running falco in an SSL everywhere type of environment.

Here is what I'm seeing:

2019-12-06T01:32:56+0000: Runtime error: Could not create embedded webserver: null context when constructing CivetServer. Possible problem binding to port.. Exiting.

[fntlnz]

I believe I created confusion on this by mixing the ssl linking problem I found with this issue.

The error both you @mac-abdon and you @WeiZhang555 seems related to another service bound on the same port - would you please check that there's nothing else listening on 443 or just leave the default port 8765 - @mfdii was reporting success doing that https://github.com/falcosecurity/falco/issues/860#issuecomment-537177446

[leodido]

Maybe I am wrong but this seems to work to me too.

Also we are rewriting inputs so it is likely that webserver will be deprecated in favor of the new API.

[WeiZhang555]

The bug still exists for me after I changed the port to 8765

$ kubectl get svc
NAME            TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
falco-service   ClusterIP   10.68.60.248   <none>        8765/TCP   8m39s

$ kubectl logs -f falco-daemonset-m79qf
* Setting up /usr/src links from host
* Mounting debugfs
Found kernel config at /host/boot/config-4.15.0-65-generic
* Trying to compile BPF probe falco-probe-bpf (falco-probe-bpf-0.18.0-x86_64-4.15.0-65-generic-0f4107e0067519c431f6d4e50493a152.o)
**********************************************************
** BPF doesn't have JIT enabled, performance might be   **
** degraded. Please ensure to run on a kernel with      **
** CONFIG_BPF_JIT enabled and/or use --net=host if      **
** running inside a container.                          **
**********************************************************
* BPF probe located, it's now possible to start sysdig
Tue Dec 31 03:55:44 2019: Falco initialized with configuration file /etc/falco/falco.yaml
Tue Dec 31 03:55:44 2019: Loading rules from file /etc/falco/falco_rules.yaml:
Tue Dec 31 03:55:44 2019: Loading rules from file /etc/falco/falco_rules.local.yaml:
Tue Dec 31 03:55:44 2019: Loading rules from file /etc/falco/k8s_audit_rules.yaml:
Tue Dec 31 03:55:44 2019: Starting internal webserver, listening on port 8765 (SSL)
Tue Dec 31 03:55:44 2019: Runtime error: Could not create embedded webserver: null context when constructing CivetServer. Possible problem binding to port.. Exiting.

$ sudo netstat -lnp | grep 8765
<nothing output>

I'm sure no other process is binding to 8765.

[fntlnz]

ok, @WeiZhang555 I debugged this a bit more today and I'm not aware of why that happens.

The dependency we use for the webserver, Civetweb, figures out libssl.so and libcrypto.so at runtime using libdl.
Also, the error management of that dependency internally doesn't advise about that directly, resulting in that error you posted if it can't find libcrypto.so or a core dump if it can't find libssl.so.

The inability to not find those library doesn't mean that they are not installed, it's just that because it looks for the libraries at a different location.

How to fix

To make it work, you can symlink your libcrypto to the location where Falco looks for it.

ln -s /lib64/libcrypto.so.10 /lib64/libcrypto.so
ln -s /lib64/libssl.so.10 /lib64/libssl.so

How to debug the problem

If you start Falco using strace it will show that it can't find libcrypto.so

open("/lib64/tls/libcrypto.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/lib64/libcrypto.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib64/tls/libcrypto.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib64/libcrypto.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
munmap(0x7f9c1ea50000, 13318)           = 0
stat("/tmp/error.log", {st_mode=S_IFREG|0644, st_size=246, ...}) = 0
open("/tmp/error.log", O_WRONLY|O_CREAT|O_APPEND, 0666) = 12
lseek(12, 0, SEEK_END)                  = 246
fstat(12, {st_mode=S_IFREG|0644, st_size=246, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9c1ea53000
write(12, "[1579170429] [error] [client ] i"..., 82) = 82
close(12)                               = 0
munmap(0x7f9c1ea53000, 4096)            = 0
close(3)                                = 0
socket(AF_LOCAL, SOCK_DGRAM|SOCK_CLOEXEC, 0) = 3
connect(3, {sa_family=AF_LOCAL, sun_path="/dev/log"}, 110) = -1 ENOENT (No such file or directory)
close(3)                                = 0
stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
write(2, "Thu Jan 16 10:27:09 2020: Runtim"..., 166Thu Jan 16 10:27:09 2020: Runtime error: Could not create embedded webserver: null context when constructing CivetServer. Possible problem binding to port.. Exiting.

Once you fix libcrypto it will do the same for libssl.

Action items
In my opinion, we want to solve this, there are many ways, however:

• Wait for the input API and rewrite the webserver
• Change the webserver to something else that is smarter with libssl and libcrypto
• Try to patch civetweb to avoid dynamic linking

Additional notes:
In civetweb, there's a compile option that allows changing the name for the libraries. It's CRYPTO_LIB, by default that value is libssl.so as defined in civetweb.c. However, changing that default will mean that when we compile package all the other system with the old default will stop working.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Issues labeled "cncf", "roadmap" and "help wanted" will not be automatically closed. Please refer to a maintainer to get such label added if you think this should be kept open.

[leogr]

Has this been fixed?

[kevin90n]

Following up on what @fntlnz suggested, I have tried the following workaround in the Falco daemonset to fix this issue.

    lifecycle:
      postStart:
        exec:
          command:
          - /bin/sh
          - -c
          - ln -s /host/usr/lib64/libcrypto.so.10  /lib/libcrypto.so && ln -s
            /host/usr/lib64/libssl.so.10   /lib/libssl.so

I have only tested this out in the following image: docker.io/falcosecurity/falco:0.18.0. Hope it helps someone.

[czunker]

I couldn't get the workaround running with falco 0.25.0.

I understand the above comment by @fntlnz, that there might be a replacement for the current webserver. Any update on that?

Just asking to get a sense of direction...

[czunker]

Another workaround:
I configured an Ingress for the falco webserver and used the Ingress URL for the AuditSink. But this leaves the communication between Ingress Controller (traefik in our case) and falco webserver unencrypted. So it might not be applicable to every setup.

[antoinedeschenes]

Is openssl now statically linked in the falco build? The problem probably happens on systems without libssl 1.0 installed (the falcosecurity/falco docker image, for example)

[leogr]

Is openssl now statically linked in the falco build? The problem probably happens on systems without libssl 1.0 installed (the falcosecurity/falco docker image, for example)

Since 0.26.1, only the binary package (ie. the .tar.gz) and the falcosecurity/falco-no-driver ship the 100% statically-linked build.
Hopefully, we will use the 100% statically-linked build for all other artifacts within the next few releases.

[nibalizer]

I tried with 0.26.2 falco-no-driver and still got:

Wed Dec  2 00:13:53 2020: Starting internal webserver, listening on port 8765 (SSL)
Wed Dec  2 00:13:53 2020: Runtime error: Could not create embedded webserver: null context when constructing CivetServer. Possible problem binding to port.. Exiting.

[ashutoshbhatt1]

hi
I am getting same error when trying to integrate with Prometheus. Is there a workaround for this? Can I still use falco-exporter without fixing this? Just started using falco last week so I am very new to this tool.

[leogr]

hi
I am getting same error when trying to integrate with Prometheus. Is there a workaround for this? Can I still use falco-exporter without fixing this? Just started using falco last week so I am very new to this tool.

Hi @ashutoshbhatt1

This issue is not related to falco-exporter (that relies on the gRPC API, not the webserver for the audit log feature). So, you can use falco-exporter without any issue. Furthermore, although the gRPC API has its own mutual TLS configuration, my advice is to the Unix Socket configuration. You can find more details here, and if you prefer to use helm the falco-exporter's chart is here.

[poiana]

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale