Druid: k8s integration tests don't stop containers after tests

[jihoonson]

Affected Version

All versions since k8s tests were added.

Description

A list of containers after a successful run of an integration test:

$ docker ps
CONTAINER ID   IMAGE                  COMMAND                  CREATED              STATUS                  PORTS     NAMES
9ced87b70c46   d1ccdd18e6ed           "kube-controller-man…"   1 second ago         Up Less than a second             k8s_kube-controller-manager_kube-controller-manager-jihoon-super-pc_kube-system_d03fa667719c85732e2eac4f277a2040_2
780fab0a24c3   6c9320041a7b           "kube-scheduler --au…"   3 seconds ago        Up 2 seconds                      k8s_kube-scheduler_kube-scheduler-jihoon-super-pc_kube-system_363a5bee1d59c51a98e345162db75755_2
f3a3b96d0efd   4e68534e24f6           "/usr/local/bin/kube…"   5 seconds ago        Up 4 seconds                      k8s_kube-proxy_kube-proxy-2fsgf_kube-system_c5660130-8654-43d3-932e-2d128e706b8e_2
55bc43485fc0   67da37a9a360           "/coredns -conf /etc…"   13 seconds ago       Up 12 seconds                     k8s_coredns_coredns-66bff467f8-g4dxt_kube-system_61e14cfb-8af0-4cda-9b2d-428ef15a7dce_2
69300cc8fe1a   67da37a9a360           "/coredns -conf /etc…"   13 seconds ago       Up 12 seconds                     k8s_coredns_coredns-66bff467f8-wsrbm_kube-system_bb2e3ba1-49bc-45fd-b0e5-10a29cea7f80_2
3eea87f2aab0   303ce5db0e90           "etcd --advertise-cl…"   18 seconds ago       Up 17 seconds                     k8s_etcd_etcd-jihoon-super-pc_kube-system_ef1e4625843f8e8bd5edba2fd63deefe_2
365af554eda8   k8s.gcr.io/pause:3.2   "/pause"                 About a minute ago   Up About a minute                 k8s_POD_coredns-66bff467f8-wsrbm_kube-system_bb2e3ba1-49bc-45fd-b0e5-10a29cea7f80_1
17f09235a22e   k8s.gcr.io/pause:3.2   "/pause"                 About a minute ago   Up About a minute                 k8s_POD_coredns-66bff467f8-g4dxt_kube-system_61e14cfb-8af0-4cda-9b2d-428ef15a7dce_1
5ed6e7d80f7b   k8s.gcr.io/pause:3.2   "/pause"                 About a minute ago   Up About a minute                 k8s_POD_kube-proxy-2fsgf_kube-system_c5660130-8654-43d3-932e-2d128e706b8e_1
4d3041ffe418   k8s.gcr.io/pause:3.2   "/pause"                 About a minute ago   Up About a minute                 k8s_POD_etcd-jihoon-super-pc_kube-system_ef1e4625843f8e8bd5edba2fd63deefe_1
4c56c18e3751   k8s.gcr.io/pause:3.2   "/pause"                 About a minute ago   Up About a minute                 k8s_POD_kube-apiserver-jihoon-super-pc_kube-system_e8550d1d08f5080bcd7eb646431c19fc_1
2b8b8de6c656   k8s.gcr.io/pause:3.2   "/pause"                 About a minute ago   Up About a minute                 k8s_POD_kube-controller-manager-jihoon-super-pc_kube-system_d03fa667719c85732e2eac4f277a2040_1
fe954c15a0aa   k8s.gcr.io/pause:3.2   "/pause"                 About a minute ago   Up About a minute                 k8s_POD_kube-scheduler-jihoon-super-pc_kube-system_363a5bee1d59c51a98e345162db75755_1
86d3939cea1c   7c4bea8d0a72           "/druid.sh router"       29 minutes ago       Up 29 minutes                     k8s_druid-tiny-cluster-routers_druid-tiny-cluster-routers-0_default_e6a0e69f-5c3f-4a4f-88b2-6411725b11dc_0
bf3452774527   7c4bea8d0a72           "/druid.sh broker"       29 minutes ago       Up 29 minutes                     k8s_druid-tiny-cluster-brokers_druid-tiny-cluster-brokers-0_default_87dab512-9ff2-44f7-9e2d-b2a9c6631eb7_0
acb608756a89   7c4bea8d0a72           "/druid.sh coordinat…"   29 minutes ago       Up 29 minutes                     k8s_druid-tiny-cluster-coordinators_druid-tiny-cluster-coordinators-0_default_4735b0cb-9cfd-42b5-bd8c-23ba2f204e99_0
817c0a98bb79   7c4bea8d0a72           "/druid.sh historical"   29 minutes ago       Up 29 minutes                     k8s_druid-tiny-cluster-historicals_druid-tiny-cluster-historicals-0_default_4847b0c2-8c56-4a58-8451-f9991438b3c4_0
97b2be8eb6b3   7c4bea8d0a72           "/druid.sh middleMan…"   29 minutes ago       Up 29 minutes                     k8s_druid-tiny-cluster-middlemanagers_druid-tiny-cluster-middlemanagers-0_default_0074b105-9ec8-4503-89a4-42c8a5f607da_0
64b3682642f2   k8s.gcr.io/pause:3.2   "/pause"                 29 minutes ago       Up 29 minutes                     k8s_POD_druid-tiny-cluster-routers-0_default_e6a0e69f-5c3f-4a4f-88b2-6411725b11dc_0
38094392b2ae   k8s.gcr.io/pause:3.2   "/pause"                 29 minutes ago       Up 29 minutes                     k8s_POD_druid-tiny-cluster-coordinators-0_default_4735b0cb-9cfd-42b5-bd8c-23ba2f204e99_0
ceaf6beb10ac   k8s.gcr.io/pause:3.2   "/pause"                 29 minutes ago       Up 29 minutes                     k8s_POD_druid-tiny-cluster-brokers-0_default_87dab512-9ff2-44f7-9e2d-b2a9c6631eb7_0
46d29948eb71   k8s.gcr.io/pause:3.2   "/pause"                 29 minutes ago       Up 29 minutes                     k8s_POD_druid-tiny-cluster-middlemanagers-0_default_0074b105-9ec8-4503-89a4-42c8a5f607da_0
5abd38e80c75   k8s.gcr.io/pause:3.2   "/pause"                 29 minutes ago       Up 29 minutes                     k8s_POD_druid-tiny-cluster-historicals-0_default_4847b0c2-8c56-4a58-8451-f9991438b3c4_0
7b198d00b69d   k8s.gcr.io/pause:3.2   "/pause"                 35 minutes ago       Up 35 minutes                     k8s_POD_druid-operator-5cf74dd88c-lgfm6_default_9b4364cf-f4d3-4cfd-8189-cf4369687bb1_0
8bf2938a7abf   k8s.gcr.io/pause:3.2   "/pause"                 37 minutes ago       Up 37 minutes                     k8s_POD_storage-provisioner_kube-system_38150797-6b44-43fb-80a7-6ea7e8a48bb7_0

[himanshug]

I thought https://github.com/apache/druid/blob/master/.travis.yml#L575 does the trick, how did you get above from travis ?

[jihoonson]

Hmm, I'm not sure why the script didn't kill the containers. I ran them in my local machine.

[himanshug]

that script actually is supposed to destroy the whole k8s cluster setup by minikube .. can you manually run the script and see if it really works... maybe the script itself is buggy or else, maybe after_script feature doesn't do what we think it should or maybe after_script feature is just broken for local travis runner

[himanshug]

CC @zhangyue19921010 as He added it and might know something.

[jihoonson]

Oh, I didn't run on travis or didn't run the travis script. I just ran the integration test manually and expected to kill all those containers after all tests passed because it spawns them automatically.

[jihoonson]

I can test the script later, but I believe it would work (I didn't know that exists). Maybe we should just call the script somewhere inside of the integration test framework instead of after_script.

[zhangyue19921010]

Sure I will investigate this problem. And will respone later.

[himanshug]

because it spawns them automatically.

actually, k8s/druid cluster is not created by the integration test... you must have manually run following scripts...

        - integration-tests/script/setup_k8s_cluster.sh
        - integration-tests/script/setup_druid_operator_on_k8s.sh
        - integration-tests/script/setup_druid_on_k8s.sh

if not, then I am puzzled how the druid cluster came into existence 🤔

[jihoonson]

Oops, I think you are right. It has been a while since I ran the test. Then, can we automate the creation and cleanup of the cluster in integration tests instead of relying on the Travis script? I think that would be better since we may want to run the same test in other environments than Travis and we can reduce the migration cost in that case.

[zhangyue19921010]

Hi @jihoonson and @himanshug . I did some test on my ec2.

As you can see, the stop script works fine and remove all the pods.

Also I agree with jihoonson that automate do the creation and cleanup of the cluster in integration tests instead of relying on the Travis script.

I will make another PR to finish it soon :)

[maytasm]

I think you can have build_run_cluster.sh and stop_cluster.sh run your script instead. In build_run_cluster.sh and stop_cluster.sh, you can check the $DRUID_INTEGRATION_TEST_GROUP variable and determine if the k8 scripts have to be run or not. (The build_run_cluster.sh and stop_cluster.sh is run from the mvn integration-tests profile.)

[zhangyue19921010]

Hi guys. I just make a PR https://github.com/apache/druid/pull/10785, trying to solve this problem. Job 73(K8s related) is successful and I believe this PR is ready for a review.

On the other hand. I run this Job on my local env, it will do the Druid/K8s cluster creation and cleanup itself instead of relying on the Travis script. After running

export CONFIG_FILE='k8s_run_config_file.json' 
export IT_TEST='-Dit.test=ITNestedQueryPushDownTest' 
export POD_NAME=int-test 
export POD_NAMESPACE=default 
export BUILD_DRUID_CLSUTER=true
export MAVEN_SKIP="-Pskip-static-checks -Ddruid.console.skip=true -Dmaven.javadoc.skip=true"
mvn -B clean install -q -ff -Pskip-static-checks -Ddruid.console.skip=true -Dmaven.javadoc.skip=true -Pskip-tests -T1C
mvn verify -pl integration-tests -P int-tests-config-file ${IT_TEST} ${MAVEN_SKIP} -Dpod.name=${POD_NAME} -Dpod.namespace=${POD_NAMESPACE} -Dbuild.druid.cluster=${BUILD_DRUID_CLSUTER}

IT is successful and docker ps shows no containers left.

!1377 $ docker ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

PTAL :)