Ingress-nginx: nginx: [error] Failed to load tracing library /usr/local/lib/libjaegertracing.so

[mikebryant]

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/.): No

What keywords did you search in NGINX Ingress controller issues before filing this one? (If you have found any duplicates, you should instead reply there.): Failed to load tracing library

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

NGINX Ingress controller version: 0.16.2

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.7", GitCommit:"b30876a5539f09684ff9fde266fda10b37738c9c", GitTreeState:"clean", BuildDate:"2018-01-16T21:59:57Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.7+coreos.0", GitCommit:"768e049ab5230010251f30475e0e785e2e999566", GitTreeState:"clean", BuildDate:"2018-01-18T00:17:18Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

What happened:

Tried to upgrade to 0.16.2 from 0.15.0. Have opentracing with the jaeger backend configured. It fails to load, with the following output

I0703 13:32:58.009633       7 controller.go:169] Configuration changes detected, backend reload required.
E0703 13:32:58.628312       7 controller.go:175] Unexpected failure reloading the backend:

-------------------------------------------------------------------------------
Error: exit status 1
2018/07/03 13:32:58 [error] 111#111: Failed to load tracing library /usr/local/lib/libjaegertracing.so: H�  
nginx: [error] Failed to load tracing library /usr/local/lib/libjaegertracing.so: H�  
nginx: configuration file /tmp/nginx-cfg156395180 test failed

-------------------------------------------------------------------------------
W0703 13:32:58.628337       7 queue.go:130] requeuing kube-system/kube-dns, err 
-------------------------------------------------------------------------------
Error: exit status 1
2018/07/03 13:32:58 [error] 111#111: Failed to load tracing library /usr/local/lib/libjaegertracing.so: H�  
nginx: [error] Failed to load tracing library /usr/local/lib/libjaegertracing.so: H�  
nginx: configuration file /tmp/nginx-cfg156395180 test failed

-------------------------------------------------------------------------------

What you expected to happen:
To be able to load the config

How to reproduce it (as minimally and precisely as possible):

m@hotppc17686:~$ docker run -it --user 0 --rm --entrypoint=/bin/bash mirror-quay.docker.tech.lastmile
.com/kubernetes-ingress-controller/nginx-ingress-controller:0.16.2 
root@be9ef4ddd0da:/etc/nginx# cat /etc/nginx/opentracing.json
{                        
          "service_name": "nginxingresscontroller",
            "sampler": { 
                            "type": "remote",     
                                    "param": 1
                                      },
                                        "reporter": {
                                                        "localAgentHostPort": "127.0.0.1:6831"
                                                          }
}

root@be9ef4ddd0da:/etc/nginx# cat /etc/nginx/nginx.conf                                              
# A very simple nginx configuration file that forces nginx to start.                                 
pid /tmp/nginx.pid;                                                                                  

load_module /etc/nginx/modules/ngx_http_opentracing_module.so;                                       
events {}                                                                    
# Load the OpenTracing dynamic module.                                                               

http {                                                                                               
  # Load a vendor tracer                                                                             
  #opentracing_load_tracer /usr/local/lib/libjaegertracing_plugin.so /etc/jaeger-nginx-config.json;  
  opentracing_load_tracer /usr/local/lib/libjaegertracing.so       /etc/nginx/opentracing.json;      

  # or                                                                                               
  #   opentracing_load_tracer /usr/local/lib/liblightstep_tracer_plugin.so /path/to/config;          
  # or                                                                                               
  #   opentracing_load_tracer /usr/local/lib/libzipkin_opentracing_plugin.so /path/to/config;

  # Enable tracing for all requests.                             
  opentracing on;                                                                                    

  # Optionally, set additional tags.                                                                 
  opentracing_tag http_user_agent $http_user_agent;                                                  

  upstream backend {                                                                          
    server app-service:9001;                                                                         
  }                                                                        

  location ~ {                                                                                      
    # The operation name used for spans defaults to the name of the location                  
    # block, but you can use this directive to customize it.                                        
    opentracing_operation_name $uri;                                               

    # Propagate the active span context upstream, so that the trace can be
    # continued by the backend.                                                                     
    # See http://opentracing.io/documentation/pages/api/cross-process-tracing.html              
    opentracing_propagate_context;                                                                  

    proxy_pass http://backend;                                                                       
  }                                                                                                  
}                                                                                                    
daemon off;                
root@be9ef4ddd0da:/etc/nginx# nginx -c /etc/nginx/nginx.conf
2018/07/04 11:03:37 [error] 320#320: Failed to load tracing library /usr/local/lib/libjaegertracing.so: H  
nginx: [error] Failed to load tracing library /usr/local/lib/libjaegertracing.so: H  
root@be9ef4ddd0da:/etc/nginx# 

Anything else we need to know:

[mikebryant]

Related to #2676?

[mikebryant]

Added a minimal reproduction using the image, and an example here: https://github.com/opentracing-contrib/nginx-opentracing/tree/master/example/trivial/ubuntu-x86_64

[mikebryant]

Following https://github.com/opentracing-contrib/nginx-opentracing/tree/master/example/trivial/ubuntu-x86_64 more closely, and downloading the binary releases explicitly (wget -O libjaegertracing_plugin.so https://github.com/jaegertracing/jaeger-client-cpp/releases/download/v0.4.1/libjaegertracing_plugin.linux_amd64.so) fixes the issue...

[mikebryant]

Looks like the built version of /usr/local/lib/libjaegertracing.so in the ingress-nginx docker image is the problem

[aledbf]

@mikebryant apologies for this issue. This weekend I will take a look at the generated library in the nginx image

[mikebryant]

FYI I'm currently working on a fix, I'll hopefully have a PR for this tomorrow. I believe the issue is we need the plugin version of jaeger instead of the shared library, which is a different build option. I'm just waiting for it to build and I can test it later tonight

[mikebryant]

@aledbf see #2744

[aledbf]

Closing. Fixed in #2744