Cloud-on-k8s: APM agents do not connect

[APuertaSales]

Bug Report

What did you do?
We have been using your operator to deploy APM but when we try to connect a Java application it fails.

What did you expect to see?
In /app/kibana#/home/tutorial/apm as I can check the APM server status, I should check agent status and see there my application.
It should be easy following your example YAMLs:
https://github.com/elastic/cloud-on-k8s/blob/master/config/samples/apm/apm_es_kibana.yaml

What did you see instead? Under which circumstances?
Change APM-HTTP service to nodeport and try to connect from outside the K8s cluster:
java -javaagent:c:\devel\tmp\elastic-apm-agent-1.13.0.jar -Delastic.apm.service_name=ms-gisauditlogdomain -Delastic.apm.server_urls=http://10.138.132.6:31744 -Delastic.apm.secret_token= -D
elastic.apm.application_packages=com.cor -jar ms-gisauditlogdomain-application-0.0.1-SNAPSHOT.jar

But we get this error:
[elastic-apm-remote-config-poller] ERROR co.elastic.apm.agent.configuration.ApmServerConfigurationSource - Connection reset

Checking the service we see that it is configured by default as HTTPS, not HTTP. Try to connect again:
java -javaagent:c:\devel\tmp\elastic-apm-agent-1.13.0.jar -Delastic.apm.service_name=ms-gisauditlogdomain -Delastic.apm.server_urls=https://10.138.132.6:30042 -Delastic.apm.secret_token= -
Delastic.apm.application_packages=com.cor -jar ms-gisauditlogdomain-application-0.0.1-SNAPSHOT.jar

And got this error:
```2020-02-28 17:09:00.492 [elastic-apm-server-healthcheck] WARN co.elastic.apm.agent.report.ApmServerHealthChecker - Elastic APM server https://10.138.132.6:30042/ is not available (sun.security.validator.ValidatorException: PKIX path building failed: sun.security.
provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target)
2020-02-28 17:09:00.534 [elastic-apm-remote-config-poller] ERROR co.elastic.apm.agent.configuration.ApmServerConfigurationSource - sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Then we configure APM server not to use SSL:
```  config:
    apm-server:
      ssl:
        enabled: false 

And have to remove the readinessProbe from the deployment. We try to connect again:
java -javaagent:c:\devel\tmp\elastic-apm-agent-1.13.0.jar -Delastic.apm.service_name=ms-gisauditlogdomain -Delastic.apm.server_urls=http://10.138.132.6:30042 -Delastic.apm.secret_token= -D
elastic.apm.application_packages=com.cor -jar ms-gisauditlogdomain-application-0.0.1-SNAPSHOT.jar

And got this error:
[elastic-apm-remote-config-poller] ERROR co.elastic.apm.agent.configuration.ApmServerConfigurationSource - Unexpected status 401 while fetching configuration

Adding the token to the agent I get this log in the agent:

[elastic-apm-server-healthcheck] INFO co.elastic.apm.agent.report.ApmServerHealthChecker - Elastic APM server is available: { "build_date": "2020-02-05T22:13:54Z", "build_sha": "c32e23164f2df31cdc976618de60ed2a97717462", "version": "7.6
.0"}

But this error in APM server:

ERROR [request] middleware/log_middleware.go:95 forbidden request {"request_id": "c67c3548-fa48-4389-96fc-598f6aa53e5c", "method": "POST", "URL": "/config/v1/agents", "content_length": 415, "remote_address": "192.168.12.0", "user-agent": "elasticapm-java/1.13.0", "response_code": 403, "error": "forbidden request: Agent remote configuration is disabled. Configure the `apm-server.kibana` section in apm-server.yml to enable it. If you are using a RUM agent, you also need to configure the `apm-server.rum` section. If you are not using remote configuration, you can safely ignore this error."}

Then I enable Kibana in APM Server:

config:
    apm-server:
      ssl:
        enabled: false
      kibana:
        enabled: true

And I get this error in APM server:

ERROR [kibana] kibana/connecting_client.go:76 failed to obtain connection to Kibana: fail to get the Kibana version: HTTP GET request to http://localhost:5601/api/status fails: fail to execute the HTTP GET request: Get http://localhost:5601/api/status: dial tcp [::1]:5601: connect: cannot assign requested address. Response: .

Then I add the Kibana host to the configuration:
``` kibana:
enabled: true
host: https://kb-apm-kb-http:5601

Then I get a different error:

ERROR [kibana] kibana/connecting_client.go:76 failed to obtain connection to Kibana: fail to get the Kibana version: HTTP GET request to https://kb-apm-kb-http:5601/api/status fails: fail to execute the HTTP GET request: Get https://kb-apm-kb-http:5601/api/status: x509: certificate signed by unknown authority. Response: .
```

Environment

• ECK version:

{
"name" : "es-apm-es-default-0",
"cluster_name" : "es-apm",
"cluster_uuid" : "UWTp1skpTg-4rDxQS5-B-Q",
"version" : {
"number" : "7.6.0",
"build_flavor" : "default",
"build_type" : "docker",
"build_hash" : "7f634e9f44834fbc12724506cc1da681b0c3b1e3",
"build_date" : "2020-02-06T00:09:00.449973Z",
"build_snapshot" : false,
"lucene_version" : "8.4.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}

• Kubernetes information:

  On premise, Kubernetes v1.15.3 vanilla.

  Thanks in advance!!!!

[pebrc]

ECK does enable TLS by default on all Elastic stack applications and it uses self-signed certificates unless you change that yourself. This is documented here https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-apm-server.html and here https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-accessing-elastic-services.html

Adding the token to the agent I get this log in the agent:

It seems that you got a working configuration at this point. You then proceeded to configure APM agent configuration via Kibana

But you ran into the same problem here, by default we use self-signed certificates in Kibana, so you either have to turn TLS off or use a certificate issued by a well known authority or configure the ECK generated certificates authorities as trusted.

Please find below an adjusted version of our APM/Kibana/Elasticsearch sample configuration with APM agent configuration via Kibana turned on and configured. The example assumes you create a custom user called apm-kibana-user to access Kibana and put the password in a secret called my-kibana-users-password

apiVersion: apm.k8s.elastic.co/v1                                                                                                                                          
kind: ApmServer                                                                                                                                                            
metadata:                                                                                                                                                                  
  name: apm-apm-sample                                                                                                                                                     
spec:                                                                                                                                                                      
  version: 7.6.0                                                                                                                                                           
  count: 1                                                                                                                                                                 
  secureSettings:                                                                                                                                                          
  - secretName: my-kibana-users-password                                                                                                                             
  http:                                                                                                                                                                    
    tls:                                                                                                                                                                   
      selfSignedCertificate:                                                                                                                                               
        disabled: true                                                                                                                                                     
  config:                                                                                                                                                                  
    apm-server:                                                                                                                                                            
      kibana:                                                                                                                                                              
        host: "https://kb-apm-sample-kb-http:5601"                                                                                                                         
        username: apm-kibana-user                                                                                                                                                  
        password: ${apm-kibana-user}                                                                                                                                               
        ssl.certificate_authorities:                                                                                                                                       
        - /usr/share/apm-server/config/kibana-ca/ca.crt                                                                                                                    
        enabled: true                                                                                                                                                      
  podTemplate:                                                                                                                                                             
    spec:                                                                                                                                                                  
      containers:                                                                                                                                                          
      - name: apm-server                                                                                                                                                   
        volumeMounts:                                                                                                                                                      
         - mountPath: /usr/share/apm-server/config/kibana-ca                                                                                                               
           name: kibana-ca                                                                                                                                                 
           readOnly: true                                                                                                                                                  
      volumes:                                                                                                                                                             
      - name: kibana-ca                                                                                                                                                    
        secret:                                                                                                                                                            
          defaultMode: 420                                                                                                                                                 
          optional: false                                                                                                                                                  
          secretName: kb-apm-sample-kb-http-certs-public                                                                                                                   
  elasticsearchRef:                                                                                                                                                        
    name: "es-apm-sample"  

Please note that we are working on making this setup easier in this issue https://github.com/elastic/cloud-on-k8s/issues/1264

Please use our discuss forum at https://discuss.elastic.co/c/eck for any questions and let's reserve Github for bug reports and feature requests.

So unless there is some additional bug/feature requesst related to this I would close this issue.