Nomad: alloc stuck in pending and zombie allocs
Nomad version
Nomad v0.8.6 (ab54ebcfcde062e9482558b7c052702d4cb8aa1b+CHANGES)
Operating system and Environment details
Ubuntu 16.04.5 LTS (GNU/Linux 4.4.0-1073-aws x86_64)- Job runs using docker driver
Issue
We upgraded our nomad environment 0.8.4 -> 0.8.6 around 2 month ago, and we noticed a weird behavior it in the last month or so (might start after the upgrade but we are not entirely sure)
after a deploy we noticed that we have some allocations stuck in "pending" status, it happens on some (on a few, like 1 or 2 clients of 50 in this nomad client class)
after some investigation we noticed that on those problematic nodes with pending allocs we have some docker containers running that nomad lost contact with and doesn't know about those allocation ids
Reproduction steps
massive register of jobs (around 100 jobs in 1 min )
Nomad Client logs (if appropriate)
/var/log/syslog.1:Feb 19 13:46:30 ip-10-1-3-243 nomad[2349]: 2019/02/19 13:46:30.016752 [INFO] client: task "productimagesaver" for alloc "e1f1f9df-0269-43f4-a71a-0e8bc9b61a4a" completed successfully
/var/log/syslog.1:Feb 19 13:46:30 ip-10-1-3-243 nomad[2349]: 2019/02/19 13:46:30.016803 [INFO] client: Restarting task "productimagesaver" for alloc "e1f1f9df-0269-43f4-a71a-0e8bc9b61a4a" in 16.424790056s
/var/log/syslog.1:Feb 19 13:46:46 ip-10-1-3-243 nomad[2349]: 2019-02-19T13:46:46.789Z [DEBUG] plugin: starting plugin: path=/usr/local/bin/nomad args="[/usr/local/bin/nomad executor {"LogFile":"/var/lib/nomad/alloc/e1f1f9df-0269-43f4-a71a-0e8bc9b61a4a/productimagesaver/executor.out","LogLevel":"INFO"}]"
/var/log/syslog.1:Feb 19 13:48:04 ip-10-1-3-243 nomad[2349]: 2019/02/19 13:48:04.681464 [INFO] client: task "productimagesaver" for alloc "e1f1f9df-0269-43f4-a71a-0e8bc9b61a4a" completed successfully
/var/log/syslog.1:Feb 19 13:48:04 ip-10-1-3-243 nomad[2349]: 2019/02/19 13:48:04.681516 [INFO] client: Restarting task "productimagesaver" for alloc "e1f1f9df-0269-43f4-a71a-0e8bc9b61a4a" in 18.290144711s
/var/log/syslog.1:Feb 19 13:48:23 ip-10-1-3-243 nomad[2349]: 2019-02-19T13:48:23.162Z [DEBUG] plugin: starting plugin: path=/usr/local/bin/nomad args="[/usr/local/bin/nomad executor {"LogFile":"/var/lib/nomad/alloc/e1f1f9df-0269-43f4-a71a-0e8bc9b61a4a/productimagesaver/executor.out","LogLevel":"INFO"}]"
ubuntu@ip-10-1-3-243:/var/lib/nomad/alloc$ tail -n 2500 /var/log/syslog | grep nomad | grep -v "initiating run"
Feb 20 07:37:07 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:07.989149 [INFO] driver.docker: stopped container b3e17862a9a61e720abc5cc4d7bce44d6c2c952b4acce739ea3e16ccd54a30f7
Feb 20 07:37:07 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:07.996Z [DEBUG] plugin.nomad: 2019/02/20 07:37:07 [ERR] plugin: plugin server: accept unix /tmp/plugin285849355: use of closed network connection
Feb 20 07:37:07 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:07.996Z [DEBUG] plugin: plugin process exited: path=/usr/local/bin/nomad
Feb 20 07:37:08 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:08.006889 [INFO] (runner) stopping
Feb 20 07:37:08 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:08.007108 [INFO] (runner) received finish
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.508091 [WARN] client: failed to broadcast update to allocation "c61a7fac-94b2-ff0a-5c40-c2ed9443d834"
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.539603 [INFO] client.gc: garbage collecting allocation e2fcccfc-99b3-c9c5-b56b-442a1d08dafd due to new allocations and over max (50)
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.589743 [ERR] client.vault: failed to derive vault tokens: Allocation "010dab43-5870-a1e1-3eca-348e7f725aae" does not exist
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.590267 [ERR] client.vault: failed to derive token for allocation "010dab43-5870-a1e1-3eca-348e7f725aae" and tasks [elasticsearchdelete]: Allocation "010dab43-5870-a1e1-3eca-348e7f725aae" does not exist
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.591707 [ERR] client: failed to derive Vault token for task elasticsearchdelete on alloc "010dab43-5870-a1e1-3eca-348e7f725aae": Allocation "010dab43-5870-a1e1-3eca-348e7f725aae" does not exist
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.615886 [INFO] client.gc: garbage collecting allocation 66a890f5-08ae-fc44-c188-72ef5bfebe76 due to new allocations and over max (50)
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.618244 [INFO] client.gc: marking allocation 010dab43-5870-a1e1-3eca-348e7f725aae for GC
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.656436 [ERR] client.vault: failed to derive vault tokens: Allocation "1373725a-72bc-c6e3-672e-5ce0ad367a92" does not exist
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.660591 [ERR] client.vault: failed to derive token for allocation "1373725a-72bc-c6e3-672e-5ce0ad367a92" and tasks [elasticsearchreviewersupdate]: Allocation "1373725a-72bc-c6e3-672e-5ce0ad367a92" does not exist
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.661075 [ERR] client: failed to derive Vault token for task elasticsearchreviewersupdate on alloc "1373725a-72bc-c6e3-672e-5ce0ad367a92": Allocation "1373725a-72bc-c6e3-672e-5ce0ad367a92" does not exist
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.671440 [INFO] client.gc: garbage collecting allocation 47188c2d-0b9f-b8fd-dc90-f738f41c24fc due to new allocations and over max (50)
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.673560 [INFO] client.gc: marking allocation 1373725a-72bc-c6e3-672e-5ce0ad367a92 for GC
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.715556 [ERR] client.vault: failed to derive vault tokens: Allocation "7d70b65d-3ee3-a11d-7d7a-ba2cffc9b725" does not exist
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.716162 [ERR] client.vault: failed to derive token for allocation "7d70b65d-3ee3-a11d-7d7a-ba2cffc9b725" and tasks [featureactionasync]: Allocation "7d70b65d-3ee3-a11d-7d7a-ba2cffc9b725" does not exist
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.716627 [ERR] client: failed to derive Vault token for task featureactionasync on alloc "7d70b65d-3ee3-a11d-7d7a-ba2cffc9b725": Allocation "7d70b65d-3ee3-a11d-7d7a-ba2cffc9b725" does not exist
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.723720 [INFO] client.gc: marking allocation 4a25c260-083a-9334-621f-838ce32ec367 for GC
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.724369 [INFO] client.gc: marking allocation 8fff9f09-3c3f-d31e-d581-0c546599db9c for GC
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.724816 [INFO] client.gc: marking allocation 47188c2d-0b9f-b8fd-dc90-f738f41c24fc for GC
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.725347 [INFO] client.gc: marking allocation 6c707f7b-493e-0edc-eda5-bdc7d805c152 for GC
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.725762 [INFO] client.gc: marking allocation 755bb66f-d1df-0ad1-8b74-c41bb39bda44 for GC
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.731260 [INFO] client.gc: marking allocation a93f9e0c-3cf6-8643-c6bc-452dac0d720b for GC
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.731351 [INFO] client.gc: marking allocation e2fcccfc-99b3-c9c5-b56b-442a1d08dafd for GC
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.731402 [INFO] client.gc: marking allocation e1f1f9df-0269-43f4-a71a-0e8bc9b61a4a for GC
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.731511 [INFO] client.gc: marking allocation 10c1eca1-c212-e834-0cd5-2c7ef8e0455e for GC
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.731611 [INFO] client.gc: marking allocation b898faca-95fd-2d58-0fa8-658c39994533 for GC
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.731687 [INFO] client.gc: marking allocation ff28eec9-f4e8-00ea-20ad-99400bb8c740 for GC
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.731730 [INFO] client.gc: marking allocation fe751977-49f0-faae-85d2-010b6c0910d2 for GC
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.731838 [INFO] client.gc: marking allocation c618839d-0deb-666a-d240-2550f6fa7859 for GC
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.731893 [INFO] client.gc: marking allocation 62e7f7ad-2cb4-b336-0792-0a391293fd4a for GC
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.732308 [INFO] client.gc: marking allocation 5c4cdadc-8975-317e-9574-6fa9e6eb3c69 for GC
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.732440 [INFO] client.gc: marking allocation d29fa168-f746-6797-1d1e-77fcb5695fd0 for GC
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.732516 [INFO] client.gc: marking allocation fd5650e8-bf17-fb39-dfc4-1486e580f2a3 for GC
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.732623 [INFO] client.gc: marking allocation 50a6d500-38e2-bf39-6125-af5f1a8ac5a7 for GC
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.732838 [INFO] client.gc: marking allocation 82a1ada6-93ba-97ca-14b1-dfa4c25d247d for GC
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.732928 [INFO] client.gc: marking allocation 253da826-1b49-6d40-be48-e1d2f9f5042a for GC
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.732983 [INFO] client.gc: marking allocation cc02cd53-0d96-91dd-d4ad-71d31da10e9f for GC
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.733063 [INFO] client.gc: marking allocation 66a890f5-08ae-fc44-c188-72ef5bfebe76 for GC
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.733114 [INFO] client.gc: marking allocation 4f133899-967c-a017-7b60-ba800a1336fb for GC
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.733155 [INFO] client.gc: marking allocation ca8967e0-c71c-ca4b-767d-0eb9909dba30 for GC
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.733291 [INFO] client.gc: marking allocation 843d0060-ca1c-06b3-f8cd-196c045b4b37 for GC
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.733417 [INFO] client.gc: garbage collecting allocation 010dab43-5870-a1e1-3eca-348e7f725aae due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.726477 [INFO] client.gc: garbage collecting allocation 26129e97-dd6b-6ec8-0933-32aaaeff9579 due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.726555 [INFO] client.gc: marking allocation 7d70b65d-3ee3-a11d-7d7a-ba2cffc9b725 for GC
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.733654 [INFO] client.gc: garbage collecting allocation 755bb66f-d1df-0ad1-8b74-c41bb39bda44 due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.733701 [INFO] client.gc: garbage collecting allocation 4fccdc8f-1d15-0b9d-7ddf-8c4375b29d3d due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.733741 [INFO] client.gc: garbage collecting allocation 1e6003a5-e7db-0597-9136-80bb587061a8 due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.733774 [INFO] client.gc: garbage collecting allocation a93f9e0c-3cf6-8643-c6bc-452dac0d720b due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.733817 [INFO] client.gc: garbage collecting allocation e2fcccfc-99b3-c9c5-b56b-442a1d08dafd due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.733851 [INFO] client.gc: garbage collecting allocation e1f1f9df-0269-43f4-a71a-0e8bc9b61a4a due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.733888 [INFO] client.gc: garbage collecting allocation f50856b2-19ff-2131-981b-a4b6033af692 due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.733929 [INFO] client.gc: garbage collecting allocation 10c1eca1-c212-e834-0cd5-2c7ef8e0455e due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.733971 [INFO] client.gc: garbage collecting allocation aa1f9e06-88ea-700d-fa80-99fa791940d8 due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.734006 [INFO] client.gc: garbage collecting allocation b898faca-95fd-2d58-0fa8-658c39994533 due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.734048 [INFO] client.gc: garbage collecting allocation ff28eec9-f4e8-00ea-20ad-99400bb8c740 due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.734078 [INFO] client.gc: garbage collecting allocation fe751977-49f0-faae-85d2-010b6c0910d2 due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.734113 [INFO] client.gc: garbage collecting allocation 0aa58005-8102-be75-99c0-2078f5053f69 due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.734151 [INFO] client.gc: garbage collecting allocation 0ba0acce-8768-86a5-d771-f8e5e4720cf4 due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.734179 [INFO] client.gc: garbage collecting allocation c618839d-0deb-666a-d240-2550f6fa7859 due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.734214 [INFO] client.gc: garbage collecting allocation 62e7f7ad-2cb4-b336-0792-0a391293fd4a due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.734253 [INFO] client.gc: garbage collecting allocation 1e48f64b-f9e7-cbef-f058-83ea5832e316 due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.734299 [INFO] client.gc: garbage collecting allocation 916a9304-8312-587a-5616-788f765fde9e due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.734342 [INFO] client.gc: garbage collecting allocation 4c63e5bf-9dc6-1401-58f6-2b42e867f0af due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.734385 [INFO] client.gc: garbage collecting allocation a1ffc952-fad8-dfab-abfb-0b15ba273c3b due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.726695 [INFO] client.gc: garbage collecting allocation 8fff9f09-3c3f-d31e-d581-0c546599db9c due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.727249 [INFO] client.gc: garbage collecting allocation 01ad35fc-3184-d995-6b27-59b9b52e39fe due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.727276 [INFO] client.gc: garbage collecting allocation 0cf745cb-e63d-019e-be93-4a33883c6293 due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.727313 [INFO] client.gc: garbage collecting allocation 9353c7ce-a892-7859-d505-42105df579ff due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.727343 [INFO] client.gc: garbage collecting allocation 4a25c260-083a-9334-621f-838ce32ec367 due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.727374 [INFO] client.gc: garbage collecting allocation e4b9456e-423d-2568-2437-a1d8fd30915a due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.727408 [INFO] client.gc: garbage collecting allocation b09eb284-b626-d7b9-f065-512cb928c7c8 due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.727440 [INFO] client.gc: garbage collecting allocation 0cddf75c-3292-fc95-d268-fad26f8bffaa due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.727473 [INFO] client.gc: garbage collecting allocation 47188c2d-0b9f-b8fd-dc90-f738f41c24fc due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.727506 [INFO] client.gc: garbage collecting allocation 6c707f7b-493e-0edc-eda5-bdc7d805c152 due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.727537 [INFO] client.gc: garbage collecting allocation 7d37efea-9722-0ef6-7c8c-5f51ab37f582 due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.727562 [INFO] client.gc: garbage collecting allocation 4131be35-fed6-3e36-a5f4-a02846a5d18b due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.734880 [INFO] client.gc: garbage collecting allocation 76ebc02d-c145-35af-cbf6-c2a69c6211ba due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.734926 [INFO] client.gc: garbage collecting allocation 8e7e338f-433f-2ae8-33d1-274ce837c782 due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.734966 [INFO] client.gc: garbage collecting allocation 5c4cdadc-8975-317e-9574-6fa9e6eb3c69 due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.739250 [INFO] client.gc: garbage collecting allocation 5c83189f-5d54-934b-b2a0-8b64cd9dc7ec due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.739325 [INFO] client.gc: garbage collecting allocation 1373725a-72bc-c6e3-672e-5ce0ad367a92 due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.739362 [INFO] client.gc: garbage collecting allocation d29fa168-f746-6797-1d1e-77fcb5695fd0 due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.739406 [INFO] client.gc: garbage collecting allocation 518fb661-3b61-f544-1bcc-e5d0bb482f05 due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.739442 [INFO] client.gc: garbage collecting allocation fd5650e8-bf17-fb39-dfc4-1486e580f2a3 due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.739486 [INFO] client.gc: garbage collecting allocation 0ad682d3-f7dd-0f5a-3eb4-9e6546351e3e due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.739531 [INFO] client.gc: garbage collecting allocation 50a6d500-38e2-bf39-6125-af5f1a8ac5a7 due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.739583 [INFO] client.gc: garbage collecting allocation bc577863-b80a-1414-4250-1d3d76563f21 due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.740779 [INFO] client.gc: garbage collecting allocation cf534d3d-5b3f-df6a-c83f-0b35b4706459 due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.740837 [INFO] client.gc: garbage collecting allocation 8a304a33-fe4e-b46a-6b1c-49bf771aec58 due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.740876 [INFO] client.gc: garbage collecting allocation 4c6072c0-4299-a53b-e4be-eea7a1ab04c6 due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.740913 [INFO] client.gc: garbage collecting allocation 82a1ada6-93ba-97ca-14b1-dfa4c25d247d due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.740965 [INFO] client.gc: garbage collecting allocation e296aaa8-afe7-5744-41df-5a21c8a8564b due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.741000 [INFO] client.gc: garbage collecting allocation 253da826-1b49-6d40-be48-e1d2f9f5042a due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.741084 [INFO] client.gc: garbage collecting allocation cc02cd53-0d96-91dd-d4ad-71d31da10e9f due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.741151 [INFO] client.gc: garbage collecting allocation 66a890f5-08ae-fc44-c188-72ef5bfebe76 due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.741194 [INFO] client.gc: garbage collecting allocation 4f133899-967c-a017-7b60-ba800a1336fb due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.741229 [INFO] client.gc: garbage collecting allocation ca8967e0-c71c-ca4b-767d-0eb9909dba30 due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.741268 [INFO] client.gc: garbage collecting allocation 295ba95c-4aca-3b59-4c55-24bf2fc1c9b7 due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.741321 [INFO] client.gc: garbage collecting allocation 729c062f-a3b7-33cc-0964-60535f6eb296 due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.741378 [INFO] client.gc: garbage collecting allocation 843d0060-ca1c-06b3-f8cd-196c045b4b37 due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.741419 [INFO] client.gc: garbage collecting allocation 34498163-b600-8346-c3a9-3de1d3e0fc91 due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.741460 [INFO] client.gc: garbage collecting allocation 7d70b65d-3ee3-a11d-7d7a-ba2cffc9b725 due to forced collection
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.757961 [ERR] client.vault: failed to derive vault tokens: Allocation "fd5650e8-bf17-fb39-dfc4-1486e580f2a3" does not exist
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.757997 [ERR] client.vault: failed to derive token for allocation "fd5650e8-bf17-fb39-dfc4-1486e580f2a3" and tasks [importedimagecsvvalidatorjobs]: Allocation "fd5650e8-bf17-fb39-dfc4-1486e580f2a3" does not exist
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.758011 [ERR] client: failed to derive Vault token for task importedimagecsvvalidatorjobs on alloc "fd5650e8-bf17-fb39-dfc4-1486e580f2a3": Allocation "fd5650e8-bf17-fb39-dfc4-1486e580f2a3" does not exist
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.898845 [INFO] client.gc: marking allocation 62d2d93f-7583-3d6a-2418-ed13a33c6162 for GC
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.931825 [INFO] (runner) creating new runner (dry: false, once: false)
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.933422 [INFO] (runner) creating watcher
Feb 20 07:37:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:09.933583 [INFO] (runner) starting
Feb 20 07:37:10 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:10.186120 [INFO] (runner) rendered "(dynamic)" => "/var/lib/nomad/alloc/18520f7b-5fe4-5397-736e-51ac3a9f2452/subsriptionhandler/secrets/envvars"
Feb 20 07:37:10 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:10.489Z [DEBUG] plugin: starting plugin: path=/usr/local/bin/nomad args="[/usr/local/bin/nomad executor {"LogFile":"/var/lib/nomad/alloc/18520f7b-5fe4-5397-736e-51ac3a9f2452/subsriptionhandler/executor.out","LogLevel":"INFO"}]"
Feb 20 07:37:10 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:10.489Z [DEBUG] plugin: waiting for RPC address: path=/usr/local/bin/nomad
Feb 20 07:37:10 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:10.541Z [DEBUG] plugin.nomad: plugin address: timestamp=2019-02-20T07:37:10.538Z address=/tmp/plugin294085799 network=unix
Feb 20 07:37:10 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:10.654719 [INFO] driver.docker: created container 4a49ff026ea707ee77e3e5895ea38fe0a62f6352c93d91afa6c7b4677e5fd859
Feb 20 07:37:10 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:10.894849 [INFO] (runner) creating new runner (dry: false, once: false)
Feb 20 07:37:10 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:10.895402 [INFO] (runner) creating watcher
Feb 20 07:37:10 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:10.895543 [INFO] (runner) starting
Feb 20 07:37:10 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:10.947394 [INFO] (runner) rendered "(dynamic)" => "/var/lib/nomad/alloc/5faecd69-39f4-fbec-2316-c39629dfaaff/syndicationmanager/secrets/envvars"
Feb 20 07:37:11 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:11.097Z [DEBUG] plugin: starting plugin: path=/usr/local/bin/nomad args="[/usr/local/bin/nomad executor {"LogFile":"/var/lib/nomad/alloc/5faecd69-39f4-fbec-2316-c39629dfaaff/syndicationmanager/executor.out","LogLevel":"INFO"}]"
Feb 20 07:37:11 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:11.098Z [DEBUG] plugin: waiting for RPC address: path=/usr/local/bin/nomad
Feb 20 07:37:11 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:11.169Z [DEBUG] plugin.nomad: plugin address: timestamp=2019-02-20T07:37:11.167Z network=unix address=/tmp/plugin281458751
Feb 20 07:37:11 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:11.237619 [WARN] client: failed to broadcast update to allocation "010dab43-5870-a1e1-3eca-348e7f725aae"
Feb 20 07:37:11 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:11.290531 [INFO] driver.docker: created container 866122ff4832679e5ad0bea5088c3e670ba2c9c328ea8204ffcc10f74915b031
Feb 20 07:37:11 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:11.408075 [INFO] driver.docker: started container 4a49ff026ea707ee77e3e5895ea38fe0a62f6352c93d91afa6c7b4677e5fd859
Feb 20 07:37:11 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:11.679083 [INFO] (runner) creating new runner (dry: false, once: false)
Feb 20 07:37:11 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:11.679290 [INFO] (runner) creating watcher
Feb 20 07:37:11 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:11.679655 [INFO] (runner) starting
Feb 20 07:37:11 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:11.762083 [INFO] (runner) creating new runner (dry: false, once: false)
Feb 20 07:37:11 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:11.764533 [INFO] (runner) creating watcher
Feb 20 07:37:11 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:11.771536 [INFO] (runner) starting
Feb 20 07:37:11 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:11.853934 [INFO] (runner) creating new runner (dry: false, once: false)
Feb 20 07:37:11 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:11.854274 [INFO] (runner) creating watcher
Feb 20 07:37:11 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:11.859115 [INFO] (runner) starting
Feb 20 07:37:11 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:11.861086 [INFO] (runner) rendered "(dynamic)" => "/var/lib/nomad/alloc/a8e27d6f-08b9-0cea-0851-a35dda527a70/affiliationmethod/secrets/envvars"
Feb 20 07:37:11 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:11.966370 [INFO] (runner) rendered "(dynamic)" => "/var/lib/nomad/alloc/561de838-cc2c-0986-1f95-52bd1ba97a05/orderlimitstatsresetjobs/secrets/envvars"
Feb 20 07:37:12 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:12.025543 [INFO] (runner) creating new runner (dry: false, once: false)
Feb 20 07:37:12 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:12.027836 [INFO] (runner) creating watcher
Feb 20 07:37:12 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:12.028278 [INFO] (runner) starting
Feb 20 07:37:12 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:12.441035 [INFO] (runner) creating new runner (dry: false, once: false)
Feb 20 07:37:12 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:12.444546 [INFO] (runner) creating watcher
Feb 20 07:37:12 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:12.447570 [INFO] (runner) starting
Feb 20 07:37:12 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:12.595556 [INFO] driver.docker: stopped container f365a8fd8437b619ac5725bfc64d8ac1b6ac49b4ba7148859adac4c41b381168
Feb 20 07:37:12 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:12.675708 [INFO] driver.docker: started container 866122ff4832679e5ad0bea5088c3e670ba2c9c328ea8204ffcc10f74915b031
Feb 20 07:37:12 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:12.677190 [INFO] (runner) rendered "(dynamic)" => "/var/lib/nomad/alloc/e3b8f19d-c308-c917-d45d-6fc9c09c5f95/reviewconfirmationsenderjobs/secrets/envvars"
Feb 20 07:37:12 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:12.688694 [INFO] (runner) rendered "(dynamic)" => "/var/lib/nomad/alloc/84426f8f-d6df-eeac-6e9d-c926d323a324/orderproductsappinvalidationcheckerjobs/secrets/envvars"
Feb 20 07:37:12 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:12.754183 [INFO] (runner) creating new runner (dry: false, once: false)
Feb 20 07:37:12 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:12.755708 [INFO] (runner) creating watcher
Feb 20 07:37:12 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:12.756337 [INFO] (runner) starting
Feb 20 07:37:12 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:12.772Z [DEBUG] plugin.nomad: 2019/02/20 07:37:12 [ERR] plugin: plugin server: accept unix /tmp/plugin572065213: use of closed network connection
Feb 20 07:37:12 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:12.774Z [DEBUG] plugin: starting plugin: path=/usr/local/bin/nomad args="[/usr/local/bin/nomad executor {"LogFile":"/var/lib/nomad/alloc/a8e27d6f-08b9-0cea-0851-a35dda527a70/affiliationmethod/executor.out","LogLevel":"INFO"}]"
Feb 20 07:37:12 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:12.780Z [DEBUG] plugin: plugin process exited: path=/usr/local/bin/nomad
Feb 20 07:37:12 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:12.788Z [DEBUG] plugin: waiting for RPC address: path=/usr/local/bin/nomad
Feb 20 07:37:12 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:12.789Z [DEBUG] plugin: starting plugin: path=/usr/local/bin/nomad args="[/usr/local/bin/nomad executor {"LogFile":"/var/lib/nomad/alloc/561de838-cc2c-0986-1f95-52bd1ba97a05/orderlimitstatsresetjobs/executor.out","LogLevel":"INFO"}]"
Feb 20 07:37:12 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:12.842820 [INFO] (runner) rendered "(dynamic)" => "/var/lib/nomad/alloc/0a0e36a8-e882-da70-7c4b-f44004baf678/ordersitereviewrequestsenderjobs/secrets/envvars"
Feb 20 07:37:12 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:12.885Z [DEBUG] plugin: waiting for RPC address: path=/usr/local/bin/nomad
Feb 20 07:37:13 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:13.029872 [INFO] (runner) stopping
Feb 20 07:37:13 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:13.047227 [INFO] (runner) received finish
Feb 20 07:37:13 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:13.120981 [INFO] (runner) creating new runner (dry: false, once: false)
Feb 20 07:37:13 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:13.145486 [INFO] (runner) creating watcher
Feb 20 07:37:13 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:13.145708 [INFO] (runner) starting
Feb 20 07:37:13 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:13.291826 [INFO] (runner) rendered "(dynamic)" => "/var/lib/nomad/alloc/50d2d0ef-5ed9-0ab5-c44b-369986bde064/advocatorsmatcher/secrets/envvars"
Feb 20 07:37:13 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:13.333962 [INFO] (runner) creating new runner (dry: false, once: false)
Feb 20 07:37:13 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:13.335109 [INFO] (runner) creating watcher
Feb 20 07:37:13 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:13.335526 [INFO] (runner) starting
Feb 20 07:37:13 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:13.387Z [DEBUG] plugin: starting plugin: path=/usr/local/bin/nomad args="[/usr/local/bin/nomad executor {"LogFile":"/var/lib/nomad/alloc/84426f8f-d6df-eeac-6e9d-c926d323a324/orderproductsappinvalidationcheckerjobs/executor.out","LogLevel":"INFO"}]"
Feb 20 07:37:13 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:13.397Z [DEBUG] plugin: waiting for RPC address: path=/usr/local/bin/nomad
Feb 20 07:37:13 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:13.403Z [DEBUG] plugin: starting plugin: path=/usr/local/bin/nomad args="[/usr/local/bin/nomad executor {"LogFile":"/var/lib/nomad/alloc/e3b8f19d-c308-c917-d45d-6fc9c09c5f95/reviewconfirmationsenderjobs/executor.out","LogLevel":"INFO"}]"
Feb 20 07:37:13 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:13.414Z [DEBUG] plugin: waiting for RPC address: path=/usr/local/bin/nomad
Feb 20 07:37:13 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:13.504Z [DEBUG] plugin.nomad: plugin address: timestamp=2019-02-20T07:37:13.451Z address=/tmp/plugin193333889 network=unix
Feb 20 07:37:13 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:13.558123 [INFO] (runner) rendered "(dynamic)" => "/var/lib/nomad/alloc/47346154-f79f-8aa9-78a4-328a7d5fd639/reviewsproductsfixer/secrets/envvars"
Feb 20 07:37:13 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:13.628875 [INFO] driver.docker: stopped container c7a2fb3ab02472f222559aa6831a9692c07969f135c54b31fcd51fbbe28181df
Feb 20 07:37:13 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:13.641Z [DEBUG] plugin.nomad: plugin address: timestamp=2019-02-20T07:37:13.625Z address=/tmp/plugin146965599 network=unix
Feb 20 07:37:13 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:13.666984 [INFO] (runner) creating new runner (dry: false, once: false)
Feb 20 07:37:13 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:13.671103 [INFO] (runner) creating watcher
Feb 20 07:37:13 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:13.673Z [DEBUG] plugin.nomad: 2019/02/20 07:37:13 [ERR] plugin: plugin server: accept unix /tmp/plugin292152716: use of closed network connection
Feb 20 07:37:13 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:13.700Z [DEBUG] plugin: plugin process exited: path=/usr/local/bin/nomad
Feb 20 07:37:13 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:13.712084 [INFO] (runner) starting
Feb 20 07:37:13 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:13.770884 [INFO] (runner) rendered "(dynamic)" => "/var/lib/nomad/alloc/32f6c9a4-ca9d-6ab4-7457-69b999688731/qandaretarget/secrets/envvars"
Feb 20 07:37:13 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:13.865Z [DEBUG] plugin: starting plugin: path=/usr/local/bin/nomad args="[/usr/local/bin/nomad executor {"LogFile":"/var/lib/nomad/alloc/50d2d0ef-5ed9-0ab5-c44b-369986bde064/advocatorsmatcher/executor.out","LogLevel":"INFO"}]"
Feb 20 07:37:13 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:13.913Z [DEBUG] plugin: starting plugin: path=/usr/local/bin/nomad args="[/usr/local/bin/nomad executor {"LogFile":"/var/lib/nomad/alloc/0a0e36a8-e882-da70-7c4b-f44004baf678/ordersitereviewrequestsenderjobs/executor.out","LogLevel":"INFO"}]"
Feb 20 07:37:13 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:13.939272 [INFO] driver.docker: created container 73d29ea833c586332fe62f6d750fbd8c3a733856ef335dc2d81c6b84de22d067
Feb 20 07:37:14 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:13.964189 [INFO] (runner) creating new runner (dry: false, once: false)2019-02-20T07:37:13.991Z [DEBUG] plugin: waiting for RPC address: path=/usr/local/bin/nomad
Feb 20 07:37:14 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:14.039Z [DEBUG] plugin: waiting for RPC address: path=/usr/local/bin/nomad
Feb 20 07:37:14 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:14.047877 [INFO] (runner) creating watcher
Feb 20 07:37:14 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:14.048655 [INFO] (runner) starting
Feb 20 07:37:14 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:13.973946 [INFO] driver.docker: created container 95a4a7c5e589d79527b266e94aced1f83260d30ba9b89f880081db2964cc8cd7
Feb 20 07:37:14 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:14.140899 [INFO] (runner) stopping
Feb 20 07:37:14 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:14.192695 [INFO] (runner) received finish
Feb 20 07:37:14 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:14.200Z [DEBUG] plugin.nomad: plugin address: timestamp=2019-02-20T07:37:14.175Z address=/tmp/plugin415365179 network=unix
Feb 20 07:37:14 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:14.209924 [INFO] (runner) rendered "(dynamic)" => "/var/lib/nomad/alloc/4ee62344-ac78-79b1-6f2d-637831aa6c7b/updatecampaignmonitor/secrets/envvars"
Feb 20 07:37:14 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:14.210465 [INFO] (runner) creating new runner (dry: false, once: false)
Feb 20 07:37:14 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:14.211574 [INFO] (runner) creating watcher
Feb 20 07:37:14 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:14.276378 [INFO] (runner) starting
Feb 20 07:37:14 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:14.253724 [INFO] driver.docker: stopped container 878b0a967d00b8a6115bf31de1b54f01a6f82c584c31f9c475133ec334df0857
Feb 20 07:37:14 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:14.416Z [DEBUG] plugin.nomad: plugin address: timestamp=2019-02-20T07:37:14.335Z address=/tmp/plugin728456263 network=unix
Feb 20 07:37:14 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:14.515539 [INFO] (runner) creating new runner (dry: false, once: false)
Feb 20 07:37:14 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:14.519717 [INFO] (runner) creating watcher
Feb 20 07:37:14 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:14.523498 [INFO] (runner) starting
Feb 20 07:37:14 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:14.554Z [DEBUG] plugin.nomad: 2019/02/20 07:37:14 [ERR] plugin: plugin server: accept unix /tmp/plugin584178317: use of closed network connection
Feb 20 07:37:14 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:14.615Z [DEBUG] plugin: plugin process exited: path=/usr/local/bin/nomad
Feb 20 07:37:14 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:14.655Z [WARN ] plugin: error closing client during Kill: err="unexpected EOF"
Feb 20 07:37:14 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:14.685875 [INFO] driver.docker: created container 6c8104143b49e9d7a4e727766ca9e55c2f751739f7e287ec79bad1e2394a01e6
Feb 20 07:37:14 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:14.831255 [WARN] client: failed to broadcast update to allocation "755bb66f-d1df-0ad1-8b74-c41bb39bda44"
Feb 20 07:37:15 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:14.996467 [INFO] (runner) rendered "(dynamic)" => "/var/lib/nomad/alloc/b2089158-b912-f25e-5344-d862bce481f5/shopifyflowhandler/secrets/envvars"
Feb 20 07:37:15 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:15.082Z [DEBUG] plugin.nomad: plugin address: timestamp=2019-02-20T07:37:14.975Z address=/tmp/plugin252511358 network=unix
Feb 20 07:37:15 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:15.175213 [INFO] (runner) creating new runner (dry: false, once: false)
Feb 20 07:37:15 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:15.183916 [INFO] (runner) creating watcher
Feb 20 07:37:15 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:15.196931 [INFO] (runner) starting
Feb 20 07:37:15 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:15.053644 [INFO] (runner) stopping
Feb 20 07:37:15 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:15.265442 [INFO] (runner) received finish
Feb 20 07:37:15 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:15.266Z [DEBUG] plugin: starting plugin: path=/usr/local/bin/nomad args="[/usr/local/bin/nomad executor {"LogFile":"/var/lib/nomad/alloc/32f6c9a4-ca9d-6ab4-7457-69b999688731/qandaretarget/executor.out","LogLevel":"INFO"}]"
Feb 20 07:37:15 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:15.297Z [DEBUG] plugin: starting plugin: path=/usr/local/bin/nomad args="[/usr/local/bin/nomad executor {"LogFile":"/var/lib/nomad/alloc/47346154-f79f-8aa9-78a4-328a7d5fd639/reviewsproductsfixer/executor.out","LogLevel":"INFO"}]"
Feb 20 07:37:15 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:15.363671 [INFO] driver.docker: created container 207960c5032dd69558dadcd780ce9b5f5bbca493d5de08a186386ac71e48fad12019-02-20T07:37:15.318Z [DEBUG] plugin.nomad: plugin address: timestamp=2019-02-20T07:37:15.161Z network=unix address=/tmp/plugin946674775
Feb 20 07:37:15 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:15.582Z [DEBUG] plugin: waiting for RPC address: path=/usr/local/bin/nomad
Feb 20 07:37:15 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:15.650Z [DEBUG] plugin: waiting for RPC address: path=/usr/local/bin/nomad
Feb 20 07:37:15 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:15.720216 [INFO] (runner) rendered "(dynamic)" => "/var/lib/nomad/alloc/f80c02af-6f8d-7be8-50f9-5b40a375f0cd/afterservicesitereviewrequestsenderjobs/secrets/envvars"2019-02-20T07:37:15.735Z [DEBUG] plugin.nomad: plugin address: timestamp=2019-02-20T07:37:15.615Z address=/tmp/plugin799028695 network=unix
Feb 20 07:37:15 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:15.819577 [INFO] (runner) rendered "(dynamic)" => "/var/lib/nomad/alloc/6325910e-b942-3330-4209-ccd91c5aab70/featurecallbacks/secrets/envvars"
Feb 20 07:37:15 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:15.871514 [INFO] driver.docker: started container 73d29ea833c586332fe62f6d750fbd8c3a733856ef335dc2d81c6b84de22d067
Feb 20 07:37:15 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:15.895149 [INFO] (runner) creating new runner (dry: false, once: false)
Feb 20 07:37:15 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:15.903Z [DEBUG] plugin.nomad: plugin address: timestamp=2019-02-20T07:37:15.854Z address=/tmp/plugin014428725 network=unix
Feb 20 07:37:15 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:15.904414 [INFO] (runner) creating watcher
Feb 20 07:37:15 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:15.904934 [INFO] (runner) starting
Feb 20 07:37:16 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:16.135187 [INFO] driver.docker: created container 528e0568f2d4b38c0d5a1979b1d515d9e574e377fa51c4cb2ac2ed5d1175088b
Feb 20 07:37:16 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:16.196185 [INFO] driver.docker: created container 6b3a476559c1cd408d07dbcfda1c8cc33564d45bcf0ee9500b04e0288d448a02
Feb 20 07:37:16 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:16.298Z [DEBUG] plugin: starting plugin: path=/usr/local/bin/nomad args="[/usr/local/bin/nomad executor {"LogFile":"/var/lib/nomad/alloc/b2089158-b912-f25e-5344-d862bce481f5/shopifyflowhandler/executor.out","LogLevel":"INFO"}]"
Feb 20 07:37:16 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:16.312133 [INFO] driver.docker: started container 95a4a7c5e589d79527b266e94aced1f83260d30ba9b89f880081db2964cc8cd7
Feb 20 07:37:16 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:16.356739 [INFO] (runner) rendered "(dynamic)" => "/var/lib/nomad/alloc/6547d340-b459-ffa9-2e8d-828a07bd8f54/unsubscribersmassactionjobs/secrets/envvars"
Feb 20 07:37:16 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:16.390Z [DEBUG] plugin: starting plugin: path=/usr/local/bin/nomad args="[/usr/local/bin/nomad executor {"LogFile":"/var/lib/nomad/alloc/4ee62344-ac78-79b1-6f2d-637831aa6c7b/updatecampaignmonitor/executor.out","LogLevel":"INFO"}]"
Feb 20 07:37:16 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:16.397Z [DEBUG] plugin: waiting for RPC address: path=/usr/local/bin/nomad
Feb 20 07:37:16 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:16.467264 [INFO] (runner) rendered "(dynamic)" => "/var/lib/nomad/alloc/2cb570fe-4827-e739-84a3-4e960ac41f19/mailafterpurchasepreparer/secrets/envvars"
Feb 20 07:37:16 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:16.480Z [DEBUG] plugin: waiting for RPC address: path=/usr/local/bin/nomad
Feb 20 07:37:16 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:16.525148 [INFO] driver.docker: created container 4e4b631780105c6d16648a2a9ab8462c5b4eb4301f6b9c5def4f61280235d938
Feb 20 07:37:16 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:16.795294 [INFO] driver.docker: created container bf3d06da730e87856a39968a419484ddee4a73084c0e7928c9e6617442931f4f
Feb 20 07:37:16 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:16.863Z [DEBUG] plugin: starting plugin: path=/usr/local/bin/nomad args="[/usr/local/bin/nomad executor {"LogFile":"/var/lib/nomad/alloc/2cb570fe-4827-e739-84a3-4e960ac41f19/mailafterpurchasepreparer/executor.out","LogLevel":"INFO"}]"
Feb 20 07:37:16 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:16.863Z [DEBUG] plugin: starting plugin: path=/usr/local/bin/nomad args="[/usr/local/bin/nomad executor {"LogFile":"/var/lib/nomad/alloc/6325910e-b942-3330-4209-ccd91c5aab70/featurecallbacks/executor.out","LogLevel":"INFO"}]"
Feb 20 07:37:16 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:16.865Z [DEBUG] plugin: starting plugin: path=/usr/local/bin/nomad args="[/usr/local/bin/nomad executor {"LogFile":"/var/lib/nomad/alloc/6547d340-b459-ffa9-2e8d-828a07bd8f54/unsubscribersmassactionjobs/executor.out","LogLevel":"INFO"}]"
Feb 20 07:37:16 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:16.871Z [DEBUG] plugin: starting plugin: path=/usr/local/bin/nomad args="[/usr/local/bin/nomad executor {"LogFile":"/var/lib/nomad/alloc/f80c02af-6f8d-7be8-50f9-5b40a375f0cd/afterservicesitereviewrequestsenderjobs/executor.out","LogLevel":"INFO"}]"
Feb 20 07:37:16 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:16.894Z [DEBUG] plugin: waiting for RPC address: path=/usr/local/bin/nomad
Feb 20 07:37:16 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:16.912Z [DEBUG] plugin: waiting for RPC address: path=/usr/local/bin/nomad
Feb 20 07:37:16 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:16.913Z [DEBUG] plugin: waiting for RPC address: path=/usr/local/bin/nomad
Feb 20 07:37:16 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:16.926Z [DEBUG] plugin: waiting for RPC address: path=/usr/local/bin/nomad
Feb 20 07:37:16 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:16.939967 [INFO] driver.docker: started container 6c8104143b49e9d7a4e727766ca9e55c2f751739f7e287ec79bad1e2394a01e6
Feb 20 07:37:17 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:17.185Z [DEBUG] plugin.nomad: plugin address: timestamp=2019-02-20T07:37:17.171Z address=/tmp/plugin102981300 network=unix
Feb 20 07:37:17 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:17.280Z [DEBUG] plugin.nomad: plugin address: timestamp=2019-02-20T07:37:17.280Z network=unix address=/tmp/plugin820746033
Feb 20 07:37:17 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:17.344969 [INFO] driver.docker: created container 6eb02e26d4d3a75a03c7862991969972141df1df2ab4d50510195e9ae254570f
Feb 20 07:37:17 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:17.573252 [INFO] driver.docker: created container 5ef8f9b0c405378dfc3353cbaaab9366d529ecf93cf4eec80d45a0e0058e79fc
Feb 20 07:37:17 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:17.676Z [DEBUG] plugin.nomad: plugin address: timestamp=2019-02-20T07:37:17.662Z address=/tmp/plugin867742797 network=unix
Feb 20 07:37:17 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:17.699Z [DEBUG] plugin.nomad: plugin address: timestamp=2019-02-20T07:37:17.688Z network=unix address=/tmp/plugin403442121
Feb 20 07:37:17 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:17.724Z [DEBUG] plugin.nomad: plugin address: timestamp=2019-02-20T07:37:17.719Z address=/tmp/plugin242351714 network=unix
Feb 20 07:37:17 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:17.884Z [DEBUG] plugin.nomad: plugin address: timestamp=2019-02-20T07:37:17.884Z address=/tmp/plugin002876504 network=unix
Feb 20 07:37:17 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:17.959060 [WARN] client: failed to broadcast update to allocation "e1f1f9df-0269-43f4-a71a-0e8bc9b61a4a"
Feb 20 07:37:18 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:18.004740 [INFO] driver.docker: created container 08e10c7d6f7e2f7b200b947724d73ba1584b5281890f34e9a4dc6a0fa9e6c342
Feb 20 07:37:18 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:18.055614 [INFO] driver.docker: created container 881dd45dcba3d6ae48d4a66f5d9541ecdf9d11b09e402bcb5659a3974534c2dd
Feb 20 07:37:18 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:18.131685 [INFO] driver.docker: created container 0ac71b8b2e784d2a2390df82daa3c03df57c89fca5366b0a7cff97503b1fd242
Feb 20 07:37:18 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:18.142352 [INFO] driver.docker: created container bd9a3de39e26e1bb7a6d713b095fa8e9ff515dafcfc446084dda31b889b0d8f2
Feb 20 07:37:19 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:19.214475 [INFO] driver.docker: started container 207960c5032dd69558dadcd780ce9b5f5bbca493d5de08a186386ac71e48fad1
Feb 20 07:37:20 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:20.578055 [INFO] driver.docker: started container 528e0568f2d4b38c0d5a1979b1d515d9e574e377fa51c4cb2ac2ed5d1175088b
Feb 20 07:37:20 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:20.937812 [INFO] driver.docker: started container bf3d06da730e87856a39968a419484ddee4a73084c0e7928c9e6617442931f4f
Feb 20 07:37:20 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:20.993695 [INFO] driver.docker: started container 4e4b631780105c6d16648a2a9ab8462c5b4eb4301f6b9c5def4f61280235d938
Feb 20 07:37:21 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:21.151662 [INFO] driver.docker: started container 6b3a476559c1cd408d07dbcfda1c8cc33564d45bcf0ee9500b04e0288d448a02
Feb 20 07:37:21 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:21.510023 [INFO] driver.docker: stopped container c46f469f5a5682ee67ecf27ac36fc9285f30c0e94252a1f440fdf9eff01ebb16
Feb 20 07:37:21 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:21.528Z [DEBUG] plugin.nomad: 2019/02/20 07:37:21 [ERR] plugin: plugin server: accept unix /tmp/plugin038419676: use of closed network connection
Feb 20 07:37:21 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:21.531Z [WARN ] plugin: error closing client during Kill: err="unexpected EOF"
Feb 20 07:37:21 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:21.531Z [DEBUG] plugin: plugin process exited: path=/usr/local/bin/nomad
Feb 20 07:37:21 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:21.624061 [INFO] driver.docker: started container 6eb02e26d4d3a75a03c7862991969972141df1df2ab4d50510195e9ae254570f
Feb 20 07:37:21 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:21.996136 [INFO] driver.docker: started container bd9a3de39e26e1bb7a6d713b095fa8e9ff515dafcfc446084dda31b889b0d8f2
Feb 20 07:37:22 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:22.056118 [INFO] driver.docker: started container 881dd45dcba3d6ae48d4a66f5d9541ecdf9d11b09e402bcb5659a3974534c2dd
Feb 20 07:37:22 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:22.056118 [INFO] driver.docker: started container 0ac71b8b2e784d2a2390df82daa3c03df57c89fca5366b0a7cff97503b1fd242
Feb 20 07:37:22 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:22.120173 [INFO] driver.docker: started container 5ef8f9b0c405378dfc3353cbaaab9366d529ecf93cf4eec80d45a0e0058e79fc
Feb 20 07:37:22 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:22.155460 [INFO] driver.docker: started container 08e10c7d6f7e2f7b200b947724d73ba1584b5281890f34e9a4dc6a0fa9e6c342
Feb 20 07:37:22 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:22.451432 [INFO] (runner) stopping
Feb 20 07:37:22 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:22.454031 [INFO] (runner) received finish
Feb 20 07:37:23 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:23.954804 [WARN] client: failed to broadcast update to allocation "e1f1f9df-0269-43f4-a71a-0e8bc9b61a4a"
Feb 20 07:37:29 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:29.882536 [INFO] driver.docker: stopped container 2288f5ca7e2e2b8a7050d29460cb2565b82d14f604b2ac8a91fc532cb235c81a
Feb 20 07:37:29 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:29.893Z [DEBUG] plugin.nomad: 2019/02/20 07:37:29 [ERR] plugin: plugin server: accept unix /tmp/plugin353106220: use of closed network connection
Feb 20 07:37:29 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:29.893Z [DEBUG] plugin: plugin process exited: path=/usr/local/bin/nomad
Feb 20 07:37:29 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:29.958249 [INFO] (runner) stopping
Feb 20 07:37:29 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:29.958631 [INFO] (runner) received finish
Feb 20 07:37:31 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:31.459202 [WARN] client: failed to broadcast update to allocation "b898faca-95fd-2d58-0fa8-658c39994533"
Feb 20 07:37:54 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:54.097403 [INFO] driver.docker: stopped container 8c265723dfdd975e450f724b34bb88652dceb5af9b3966dcf25108051456cd95
Feb 20 07:37:54 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:54.108Z [DEBUG] plugin.nomad: 2019/02/20 07:37:54 [ERR] plugin: plugin server: accept unix /tmp/plugin066221560: use of closed network connection
Feb 20 07:37:54 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:54.108Z [DEBUG] plugin: plugin process exited: path=/usr/local/bin/nomad
Feb 20 07:37:54 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:54.144412 [INFO] (runner) stopping
Feb 20 07:37:54 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:54.144946 [INFO] (runner) received finish
Feb 20 07:37:55 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:55.645521 [WARN] client: failed to broadcast update to allocation "fe751977-49f0-faae-85d2-010b6c0910d2"
Feb 20 07:37:58 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:58.072308 [INFO] driver.docker: stopped container b386b77206061703950cf510cd4480223e94658c7f000f75d22f0aa8fe85d927
Feb 20 07:37:58 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:58.078Z [DEBUG] plugin.nomad: 2019/02/20 07:37:58 [ERR] plugin: plugin server: accept unix /tmp/plugin359109521: use of closed network connection
Feb 20 07:37:58 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:37:58.083Z [DEBUG] plugin: plugin process exited: path=/usr/local/bin/nomad
Feb 20 07:37:58 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:58.112175 [INFO] (runner) stopping
Feb 20 07:37:58 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:58.112570 [INFO] (runner) received finish
Feb 20 07:37:59 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:37:59.613620 [WARN] client: failed to broadcast update to allocation "c618839d-0deb-666a-d240-2550f6fa7859"
Feb 20 07:38:01 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:38:01.808070 [INFO] driver.docker: stopped container 1bc9640ec829aaba3bd429439f503178733bca4dd014b45835c3bf5c26a3000d
Feb 20 07:38:01 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:38:01.820Z [DEBUG] plugin.nomad: 2019/02/20 07:38:01 [ERR] plugin: plugin server: accept unix /tmp/plugin539917924: use of closed network connection
Feb 20 07:38:01 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:38:01.820Z [DEBUG] plugin: plugin process exited: path=/usr/local/bin/nomad
Feb 20 07:38:01 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:38:01.860070 [INFO] (runner) stopping
Feb 20 07:38:01 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:38:01.860530 [INFO] (runner) received finish
Feb 20 07:38:03 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:38:03.361148 [WARN] client: failed to broadcast update to allocation "8fff9f09-3c3f-d31e-d581-0c546599db9c"
Feb 20 07:38:04 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:38:04.901020 [WARN] client: failed to broadcast update to allocation "1373725a-72bc-c6e3-672e-5ce0ad367a92"
Feb 20 07:38:09 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:38:09.975309 [INFO] driver.docker: stopped container 2252a0231819588cfa598100ca6e6993940007e4d25232afea05cb76234374a8
Feb 20 07:38:09 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:38:09.981Z [DEBUG] plugin.nomad: 2019/02/20 07:38:09 [ERR] plugin: plugin server: accept unix /tmp/plugin186207916: use of closed network connection
Feb 20 07:38:09 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:38:09.982Z [DEBUG] plugin: plugin process exited: path=/usr/local/bin/nomad
Feb 20 07:38:10 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:38:10.017545 [INFO] (runner) stopping
Feb 20 07:38:10 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:38:10.018034 [INFO] (runner) received finish
Feb 20 07:38:11 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:38:11.519635 [WARN] client: failed to broadcast update to allocation "d29fa168-f746-6797-1d1e-77fcb5695fd0"
Feb 20 07:38:13 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:38:13.039918 [WARN] client: failed to broadcast update to allocation "fd5650e8-bf17-fb39-dfc4-1486e580f2a3"
Feb 20 07:38:16 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:38:16.237971 [INFO] driver.docker: stopped container cafac167491ec9a0509f45472c97268790778f4b261b242b2b13a69310983fa7
Feb 20 07:38:16 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:38:16.241Z [DEBUG] plugin.nomad: 2019/02/20 07:38:16 [ERR] plugin: plugin server: accept unix /tmp/plugin699539517: use of closed network connection
Feb 20 07:38:16 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:38:16.243Z [DEBUG] plugin: plugin process exited: path=/usr/local/bin/nomad
Feb 20 07:38:16 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:38:16.260128 [INFO] (runner) stopping
Feb 20 07:38:16 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:38:16.260533 [INFO] (runner) received finish
Feb 20 07:38:17 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:38:17.761410 [WARN] client: failed to broadcast update to allocation "253da826-1b49-6d40-be48-e1d2f9f5042a"
Feb 20 07:38:20 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:38:20.303275 [INFO] driver.docker: stopped container b293939952a391ac70ed0bf92c3247becba955dad0d4eec34cddd5415cd10be2
Feb 20 07:38:20 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:38:20.316Z [DEBUG] plugin.nomad: 2019/02/20 07:38:20 [ERR] plugin: plugin server: accept unix /tmp/plugin388259153: use of closed network connection
Feb 20 07:38:20 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:38:20.317Z [DEBUG] plugin: plugin process exited: path=/usr/local/bin/nomad
Feb 20 07:38:20 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:38:20.392496 [INFO] (runner) stopping
Feb 20 07:38:20 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:38:20.392927 [INFO] (runner) received finish
Feb 20 07:38:21 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:38:21.893506 [WARN] client: failed to broadcast update to allocation "4f133899-967c-a017-7b60-ba800a1336fb"
Feb 20 07:38:24 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:38:24.040213 [INFO] driver.docker: stopped container c7c61b04ec81d2ded34635567f1035f43bb90b843d1947b5a76133e26f26e93f
Feb 20 07:38:24 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:38:24.046Z [DEBUG] plugin.nomad: 2019/02/20 07:38:24 [ERR] plugin: plugin server: accept unix /tmp/plugin198075200: use of closed network connection
Feb 20 07:38:24 ip-10-1-3-243 nomad[2349]: 2019-02-20T07:38:24.049Z [DEBUG] plugin: plugin process exited: path=/usr/local/bin/nomad
Feb 20 07:38:24 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:38:24.082589 [INFO] (runner) stopping
Feb 20 07:38:24 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:38:24.083695 [INFO] (runner) received finish
Feb 20 07:38:25 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:38:25.584269 [WARN] client: failed to broadcast update to allocation "ca8967e0-c71c-ca4b-767d-0eb9909dba30"
Feb 20 07:38:27 ip-10-1-3-243 nomad[2349]: 2019/02/20 07:38:27.100568 [WARN] client: failed to broadcast update to allocation "7d70b65d-3ee3-a11d-7d7a-ba2cffc9b725"
ubuntu@ip-10-1-3-243:~$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c46f469f5a56 8e2120191012 "rake resque:work" 18 hours ago Up 18 hours productimagesaver-e1f1f9df-0269-43f4-a71a-0e8bc9b61a4a
b293939952a3 a68250211150 "rake resque:work" 20 hours ago Up 20 hours collectionsimportjobs-4f133899-967c-a017-7b60-ba800a1336fb
8c265723dfdd 8e2120191012 "rake resque:work" 20 hours ago Up 20 hours newreviewhandlerjobs-fe751977-49f0-faae-85d2-010b6c0910d2
2288f5ca7e2e 8e2120191012 "rake resque:work" 20 hours ago Up 20 hours trackconversion-b898faca-95fd-2d58-0fa8-658c39994533
793b918fad42 8e2120191012 "rake resque:work" 21 hours ago Up 21 hours orderproductsappsinvalidatorjobs-7fb0f298-0d13-3560-ac36-368aa740f929
cafac167491e 8e2120191012 "rake resque:work" 21 hours ago Up 21 hours productsappscreator-253da826-1b49-6d40-be48-e1d2f9f5042a
1bc9640ec829 8e2120191012 "rake resque:work" 22 hours ago Up 22 hours unsubscribersactionjobs-8fff9f09-3c3f-d31e-d581-0c546599db9c
b3e17862a9a6 8e2120191012 "rake resque:work" 24 hours ago Up 24 hours productsgroupingsyncjobs-c61a7fac-94b2-ff0a-5c40-c2ed9443d834
b386b7720606 8e2120191012 "rake resque:work" 24 hours ago Up 24 hours orderproductsappinvalidationcheckerjobs-c618839d-0deb-666a-d240-2550f6fa7859
2252a0231819 8e2120191012 "rake resque:work" 2 days ago Up 2 days reviewcreate-d29fa168-f746-6797-1d1e-77fcb5695fd0
c7a2fb3ab024 8e2120191012 "rake resque:work" 2 days ago Up 2 days elasticsearchupdate-82a1ada6-93ba-97ca-14b1-dfa4c25d247d
f365a8fd8437 8e2120191012 "rake resque:work" 2 days ago Up 2 days elasticsearchproductsappsbulkupdate-755bb66f-d1df-0ad1-8b74-c41bb39bda44
c7c61b04ec81 8e2120191012 "rake resque:work" 2 days ago Up 2 days summaryreporttasks-ca8967e0-c71c-ca4b-767d-0eb9909dba30
878b0a967d00 8e2120191012 "rake resque:work" 2 days ago Up 2 days imagereprocess-843d0060-ca1c-06b3-f8cd-196c045b4b37
195ea90eec95 3c54c7f1da10 "/entrypoint.sh -con…" 3 days ago Up 3 days 8092/udp, 8125/udp, 10.1.3.243:8094->8094/tcp, 10.1.3.243:8094->8094/udp metrics-874081b8-7e1f-aa76-bb46-670b565893d0
ubuntu@ip-10-1-3-243:~$ nomad node status -verbose -self
ID = cf3d924d-217c-ffc6-cbb0-3fda8889ecc5
Name = i-00173cb51ba882572
Class = api-resque
DC = us-east-1
Drain = false
Eligibility = eligible
Status = ready
Uptime = 78h41m56s
Drivers
Driver Detected Healthy Message Time
docker true true Driver is available and responsive 2019-02-17T00:50:52Z
exec true true Driver exec is detected 2019-02-17T00:50:52Z
java false false Driver java is not detected 2019-02-17T00:49:51Z
qemu false false Driver qemu is not detected 2019-02-17T00:49:51Z
raw_exec false false Driver raw_exec is not detected 2019-02-17T00:49:51Z
rkt false false Driver rkt is not detected 2019-02-17T00:50:52Z
Node Events
Time Subsystem Message Details
2019-02-17T00:49:52Z Cluster Node registered <none>
Allocated Resources
CPU Memory Disk IOPS
200/6000 MHz 128 MiB/15 GiB 600 MiB/60 GiB 0/0
Allocation Resource Utilization
CPU Memory
89/6000 MHz 43 MiB/15 GiB
Host Resource Utilization
CPU Memory Disk
5939/6000 MHz 5.2 GiB/15 GiB 4.4 GiB/62 GiB
Allocations
ID Eval ID Node ID Task Group Version Desired Status Created Modified
4ee62344-ac78-79b1-6f2d-637831aa6c7b baef93fd-8bc3-5f15-c416-1dfd6f9818d4 cf3d924d-217c-ffc6-cbb0-3fda8889ecc5 updatecampaignmonitor 24 run pending 2019-02-19T13:11:58Z 2019-02-19T13:11:58Z
6547d340-b459-ffa9-2e8d-828a07bd8f54 1e0417e0-2526-2775-ad96-3e4926a2c49d cf3d924d-217c-ffc6-cbb0-3fda8889ecc5 unsubscribersmassactionjobs 270 run pending 2019-02-19T13:11:56Z 2019-02-20T06:30:09Z
5faecd69-39f4-fbec-2316-c39629dfaaff a0da0e8d-844f-3039-500b-76278906ebbc cf3d924d-217c-ffc6-cbb0-3fda8889ecc5 syndicationmanager 8777 run pending 2019-02-19T13:11:46Z 2019-02-20T07:24:33Z
18520f7b-5fe4-5397-736e-51ac3a9f2452 dff80efe-1976-7152-ebd3-218d6bac756b cf3d924d-217c-ffc6-cbb0-3fda8889ecc5 subsriptionhandler 448 run pending 2019-02-19T13:11:42Z 2019-02-20T06:37:08Z
b2089158-b912-f25e-5344-d862bce481f5 3a1ffaef-6e40-8c25-b7e7-ea1cc3c423d5 cf3d924d-217c-ffc6-cbb0-3fda8889ecc5 shopifyflowhandler 11266 run pending 2019-02-19T13:11:38Z 2019-02-20T07:23:33Z
47346154-f79f-8aa9-78a4-328a7d5fd639 b61d31d1-4c8f-3374-5f82-fcbb308cee01 cf3d924d-217c-ffc6-cbb0-3fda8889ecc5 reviewsproductsfixer 24 run pending 2019-02-19T13:11:31Z 2019-02-19T13:11:31Z
84426f8f-d6df-eeac-6e9d-c926d323a324 c090d889-aa79-0ef6-ae20-92192caf6a1b cf3d924d-217c-ffc6-cbb0-3fda8889ecc5 orderproductsappinvalidationcheckerjobs 16489 run pending 2019-02-19T13:11:30Z 2019-02-20T07:30:03Z
e3b8f19d-c308-c917-d45d-6fc9c09c5f95 d13cda14-80eb-2da5-290d-119828a84ae4 cf3d924d-217c-ffc6-cbb0-3fda8889ecc5 reviewconfirmationsenderjobs 4004 run pending 2019-02-19T13:11:22Z 2019-02-20T07:28:36Z
32f6c9a4-ca9d-6ab4-7457-69b999688731 4065e2b5-ab59-47e5-92d6-040844e6c7a6 cf3d924d-217c-ffc6-cbb0-3fda8889ecc5 qandaretarget 24 run pending 2019-02-19T13:11:11Z 2019-02-19T13:11:11Z
0a0e36a8-e882-da70-7c4b-f44004baf678 42fd1abf-8c0e-7a94-ca84-b1c88083d7bb cf3d924d-217c-ffc6-cbb0-3fda8889ecc5 ordersitereviewrequestsenderjobs 732 run pending 2019-02-19T13:10:43Z 2019-02-20T05:29:31Z
561de838-cc2c-0986-1f95-52bd1ba97a05 00112529-6e29-9a75-8c50-1f49f6873c1b cf3d924d-217c-ffc6-cbb0-3fda8889ecc5 orderlimitstatsresetjobs 2843 run pending 2019-02-19T13:10:39Z 2019-02-20T06:33:39Z
2cb570fe-4827-e739-84a3-4e960ac41f19 78276ae4-d4e8-b524-271d-2815f2ae2cba cf3d924d-217c-ffc6-cbb0-3fda8889ecc5 mailafterpurchasepreparer 161 run pending 2019-02-19T13:10:28Z 2019-02-20T02:20:00Z
6325910e-b942-3330-4209-ccd91c5aab70 739fcd33-cc11-8c71-35b4-cb90966526c5 cf3d924d-217c-ffc6-cbb0-3fda8889ecc5 featurecallbacks 2473 run pending 2019-02-19T13:10:00Z 2019-02-20T07:23:04Z
f80c02af-6f8d-7be8-50f9-5b40a375f0cd b83f3a8b-bd9f-dbde-8fe8-429a0a30c742 cf3d924d-217c-ffc6-cbb0-3fda8889ecc5 afterservicesitereviewrequestsenderjobs 439 run pending 2019-02-19T13:09:14Z 2019-02-20T06:18:09Z
a8e27d6f-08b9-0cea-0851-a35dda527a70 1db9183e-a75e-fffb-64b4-40fdb0263d4b cf3d924d-217c-ffc6-cbb0-3fda8889ecc5 affiliationmethod 116 run pending 2019-02-19T13:09:09Z 2019-02-20T07:26:02Z
50d2d0ef-5ed9-0ab5-c44b-369986bde064 02f974ca-adab-b8f4-6096-259a7ef87ecd cf3d924d-217c-ffc6-cbb0-3fda8889ecc5 advocatorsmatcher 24 run pending 2019-02-19T13:09:08Z 2019-02-19T13:09:08Z
62d2d93f-7583-3d6a-2418-ed13a33c6162 be5f908e-785c-8aee-a98d-64172a8c4fd8 cf3d924d-217c-ffc6-cbb0-3fda8889ecc5 trackconversion 18025 stop pending 2019-02-19T13:04:28Z 2019-02-19T13:11:53Z
8bc1de6e-c634-e94c-213d-723ba4d00442 54557e4e-c759-038f-a986-637195cd516f cf3d924d-217c-ffc6-cbb0-3fda8889ecc5 protologbeats 17 run running 2019-02-17T00:49:58Z 2019-02-17T00:50:08Z
874081b8-7e1f-aa76-bb46-670b565893d0 d187b3d7-dbc6-6085-ddeb-b99a830abe0f cf3d924d-217c-ffc6-cbb0-3fda8889ecc5 metrics 6 run running 2019-02-17T00:49:58Z 2019-02-17T00:50:10Z
Attributes
consul.datacenter = us-east-1
consul.revision = 28141971
consul.server = false
consul.version = 1.2.0
cpu.arch = amd64
cpu.frequency = 3000
cpu.modelname = Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz
cpu.numcores = 2
cpu.totalcompute = 6000
driver.docker = 1
driver.docker.bridge_ip = 172.17.0.1
driver.docker.version = 18.09.0
driver.docker.volumes.enabled = 1
driver.exec = 1
kernel.name = linux
kernel.version = 4.4.0-1073-aws
memory.totalbytes = 16032980992
nomad.revision = ab54ebcfcde062e9482558b7c052702d4cb8aa1b+CHANGES
nomad.version = 0.8.6
os.name = ubuntu
os.signals = SIGFPE,SIGTERM,SIGTTIN,SIGUSR2,SIGBUS,SIGIOT,SIGIO,SIGURG,SIGXFSZ,SIGSEGV,SIGWINCH,SIGPROF,SIGXCPU,SIGCONT,SIGTRAP,SIGUSR1,SIGQUIT,SIGSYS,SIGABRT,SIGALRM,SIGCHLD,SIGILL,SIGKILL,SIGTTOU,SIGHUP,SIGINT,SIGPIPE,SIGSTOP,SIGTSTP
os.version = 16.04
platform.aws.ami-id = ami-08e7627605445d6ac
platform.aws.instance-type = r4.large
platform.aws.placement.availability-zone = us-east-1a
unique.cgroup.mountpoint = /sys/fs/cgroup
unique.consul.name = ip-10-1-3-243
unique.hostname = ip-10-1-3-243
unique.network.ip-address = 10.1.3.243
unique.platform.aws.hostname = ip-10-1-3-243.ec2.internal
unique.platform.aws.instance-id = i-00173cb51ba882572
unique.platform.aws.local-hostname = ip-10-1-3-243.ec2.internal
unique.platform.aws.local-ipv4 = 10.1.3.243
unique.storage.bytesfree = 64524410880
unique.storage.bytestotal = 66549686272
unique.storage.volume = /dev/xvda1
vault.accessible = true
vault.cluster_id = 5b79f440-9f53-50ad-3920-61775f74a8a3
vault.cluster_name = vault-cluster-d089cf67
vault.version = 0.11.6
Meta
ubuntu@ip-10-1-3-243:~$ nomad agent-info
client
heartbeat_ttl = 16.824670832s
known_servers = 10.1.1.116:4647,10.1.1.154:4647,10.1.3.43:4647,10.1.7.139:4647,10.1.9.50:4647
last_heartbeat = 11.668983365s
node_id = cf3d924d-217c-ffc6-cbb0-3fda8889ecc5
num_allocations = 50
runtime
arch = amd64
cpu_count = 2
goroutines = 920
kernel.name = linux
max_procs = 2
version = go1.10.4
danaps
👀4
👍3
All 10 comments
I've run into this as well. I can see the nomad agent has received the SIGINT signal; however the task remains in a running state. The kill_timeout (defaults to 5s) doesn't seem to be kicking in.
Worse; when kill -KILL <pid of nomad agent> for that running task; nomad job status <job> now reports the wrong result and still thinks the old alloc is still running. -- Is this expected that Nomad is unaware of system-level changes (killing processes outside of Nomad's control?)
prologic
on 11 Mar 2019
/cc @johnzhanghua (my colleague)
prologic
on 11 Mar 2019
But really on further reading my issue also seems to be related to #5395
prologic
on 11 Mar 2019
Hey there
Since this issue hasn't had any activity in a while - we're going to automatically close it in 30 days. If you're still seeing this issue with the latest version of Nomad, please respond here and we'll keep this open and take another look at this.
Thanks!
![stale[bot] picture](https://avatars3.githubusercontent.com/in/1724?v=4&s=40)
stale[bot]
on 9 Jun 2019
We're running 0.9.3, and we can see the following:
nomad sets desired state to stop, but the container is never stopped. It stays in this state forever.
We can stop it with docker stop <container id> without any problem, but nomad still reports running.
Also tried stopping it manually nomad alloc stop <alloc ID>, and nomad says OK, but it never changes to "complete", i.e it's stuck in state running even when desired is stop.
Restarting the nomad service on the client resolves the issue.
parberge
on 1 Jul 2019
i see this same issue sporadically on my home server and at my work. only way to fix sometimes is to purge the job entirely then re-run.
dcarbone
on 4 Jul 2019
Stumbled into this issue as well. Could i be an issue with the Docker driver?
Nomad v0.9.3 (c5e8b66c3789e4e7f9a83b4e188e9a937eea43ce)
Docker version 18.09.7, build 2d0083d
Ubuntu 18.04.2 LTS (GNU/Linux 4.15.0-1043-aws x86_64)
Can confirm killing the job itself and re-applying works. Haven't tried rebooting the Nomad client itself as that would cause a prod outage 😬
After looking at the syslog running on the affected nomad client I see this line throughout the log:
Jul 22 13:54:31 ip-10-132-35-14 dockerd[1048]: time="2019-07-22T13:54:31.955402151Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Cross-posted here as well: https://discuss.hashicorp.com/t/0-9-x-zombie-allocations/2092/2
mikeblum
on 15 Jul 2019
Same issue with exec driver, nomad server/agents version 0.9.3.
transferkraM
on 22 Aug 2019
Experiencing the same on v0.8.7 using the docker driver. Task remains in pending state however all containers are removed.
alleeclark
on 22 Aug 2019
@preetapan , is there anything new about this issue?
I still have the same problem(0.10.0)
elifish4
on 3 Oct 2019
👍9
Was this page helpful?
0 / 5 - 0 ratings
Related issues
joliver
·
3Comments
mancusogmu
·
3Comments
Garagoth
·
3Comments
bdclark
·
3Comments
hynek
·
3Comments
Most helpful comment
@preetapan , is there anything new about this issue?
I still have the same problem(0.10.0)