Aws-parallelcluster: Dead compute nodes not removed from SGE host list
Related
https://forums.aws.amazon.com/thread.jspa?threadID=241553
https://github.com/awslabs/cfncluster/issues/208
Trying to delete the node gives the error
sudo -u sgeadmin -i qconf -de ip-XXXXX.compute.internal
Host object "ip-XXXXX.compute.internal" is still referenced in cluster queue "all.q"
awacs
All 21 comments
What version of cfncluster are you working with?
mohanasudhan
on 7 Mar 2018
1.4.2
awacs
on 7 Mar 2018
I am also getting this error with 1.4.2
I've been able to remove a host when this happens by following two commands I found here:
http://arc.liv.ac.uk/pipermail/gridengine-users/2008-September/020174.html
I had to handle edit the given file editor of these commands.
# edit the file to remove your hostname(s)
sudo -E -u sgeadmin /opt/sge/bin/lx-amd64/qconf -mq all.q
sudo -E -u sgeadmin /opt/sge/bin/lx-amd64/qconf -mhgrp @allhosts
sudo -E -u sgeadmin /opt/sge/bin/lx-amd64/qconf -de ip-172-31-xx-xxx
This script: /opt/sge/bin/dead_nodes is also handy.
rmilne
on 16 Mar 2018
I am using 1.4.2 and also seeing the same problem. A bunch of "dr" jobs in the sge.
yanpanlau
on 7 May 2018
Sorry to chase, any update on this?
yanpanlau
on 18 Jul 2018
Hi guys. I'm facing the related issue:
I've started noticing "zombie" hosts alongside the cluster. How It looks...
I have a registered host in SGE and assigned job on it, but actual node is terminated.
At logs I see this:
2018-08-07 06:48:07,324 INFO [sqswatcher:pollQueue] eventType=autoscaling:EC2_INSTANCE_TERMINATE
2018-08-07 06:48:07,324 INFO [sqswatcher:pollQueue] instanceId=i-XXXXXXXX
2018-08-07 06:48:07,356 INFO [sge:removeHost] Removing ip-xxx-xx-xx-xx
[email protected] removed "ip-xxx-xx-xx-xx.eu-west-1.compute.internal" from administrative host list
root@hostname modified "all.q" in cluster queue list
root@hostname modified "@allhosts" in host group list
Host object "ip-xxx-xx-xx-xx" is still referenced in cluster queue "all.q".
2018-08-07 06:48:07,413 ERROR [sge:__runSgeCommand] Failed to run ['/opt/sge/bin/lx-amd64/qconf', '-de', 'ip-xxx-xx-xx-xx']
root@hostname removed "ip-xxx-xx-xx-xx.eu-west-1.compute.internal" from submit host list
So the same error as you've guys mentioned.
What have I found how the node terminates:
- Nodewatcher checks if no running jobs on 95th% of an hour
- If no jobs - it sends the signal to terminate the instance
- SQSwatcher monitors status of the machine and when it is terminated it sends the signal to head-node to remove the worker host from SGE
- SGE removes the worker host from it's list with the error above.
What are my thoughts?
Probably in the time range between stage 2 and 4 job scheduler on head-node assigns a new job to the terminated node, but it never runs cause worker does not physically exist. And then the only way to remove this host from the SGE workers-list is:
HOST_TO_REMOVE=ip-xxx-xx-xx-xx
# first, you need to delete jobs on this host
qdel -f <job id>
# disable the host from queue to avoid any jobs to be allocated to this host
qmod -d all.q@$HOST_TO_REMOVE
# wait for jobs to be finished execution on this host, then kill the execution script
qconf -ke $HOST_TO_REMOVE
# remove it from the cluster, this opens an editor, just remove the lines referring to this host
qconf -mq all.q
# remove it from allhosts group, this also opens an editor, remove lines referring to this host
qconf -mhgrp @allhosts
# remove it from execution host list
qconf -de $HOST_TO_REMOVE
#If the host isn't removed then:
#Remove worker's files
rm -rf /opt/sge/default/common/local_conf/$HOST_TO_REMOVE.eu-west-1.compute.internal
rm -rf /opt/sge/default/spool/qmaster/qinstances/all.q/$HOST_TO_REMOVE.eu-west-1.compute.internal
rm -rf /opt/sge/default/spool/qmaster/exec_hosts/$HOST_TO_REMOVE.eu-west-1.compute.internal
#restart sge
/etc/init.d/sgemaster.p6444 restart
Probably you've got any other workaround on this issue?
Probably it's better idea to change the cfn-cluster code that firstly it removes the node from SGE and then terminates it?
All the comments are appreciated.
Thanks!
pro100frol
on 20 Aug 2018
@pro100frol thanks for the detailed analysis. We're working on fixing this in the next release of cfncluster 1.6.0
sean-smith
on 21 Aug 2018
@sean-smith Thank you for the response. Do we know any approx date of the release?
Thanks!
pro100frol
on 29 Aug 2018
This bug has been resolved in the upcoming 1.6.0 release:
https://github.com/awslabs/cfncluster/blob/develop/CHANGELOG.rst#160
If you are willing to test the release candindate you can install it with:
pip install -Iv cfncluster==1.6.0rc1
Please keep us updated if you have the opportunity to test the rc1 or please stay tuned for the upcoming 1.6.0 release.
enrico-usai
on 3 Sep 2018
We are running v1.6.1 and still see this issue.
[root@ip-172-31-128-116 log]# grep cfncluster_version /var/log/messages
Nov 1 00:42:56 ip-172-31-128-116 cloud-init: + export cfncluster_version=cfncluster-1.6.1
Nov 1 00:42:56 ip-172-31-128-116 cloud-init: + cfncluster_version=cfncluster-1.6.1
In the qhost output below, the ones with no LOAD/MEMUSE/SWAPUS are terminated nodes:
[root@ip-172-31-128-116 log]# qhost
HOSTNAME ARCH NCPU NSOC NCOR NTHR LOAD MEMTOT MEMUSE SWAPTO SWAPUS
----------------------------------------------------------------------------------------------
global - - - - - - - - - -
ip-172-31-128-103 lx-amd64 16 1 8 16 0.84 61.3G 4.9G 0.0 0.0
ip-172-31-128-119 lx-amd64 16 1 8 16 0.32 61.3G 4.5G 0.0 0.0
ip-172-31-128-128 lx-amd64 16 1 8 16 0.30 61.3G 1006.9M 0.0 0.0
ip-172-31-128-134 lx-amd64 16 1 8 16 0.20 61.3G 1.3G 0.0 0.0
ip-172-31-128-142 lx-amd64 16 1 8 16 - 61.3G - 0.0 -
ip-172-31-128-174 lx-amd64 16 1 8 16 - 61.3G - 0.0 -
ip-172-31-128-18 lx-amd64 16 1 8 16 0.28 61.3G 1.3G 0.0 0.0
ip-172-31-128-19 lx-amd64 16 1 8 16 0.91 61.3G 44.2G 0.0 0.0
ip-172-31-128-20 lx-amd64 16 1 8 16 - 61.3G - 0.0 -
ip-172-31-128-203 lx-amd64 16 1 8 16 0.18 61.3G 1.3G 0.0 0.0
ip-172-31-128-207 lx-amd64 16 1 8 16 0.29 61.3G 1.4G 0.0 0.0
ip-172-31-128-217 lx-amd64 16 1 8 16 - 61.3G - 0.0 -
ip-172-31-128-218 lx-amd64 16 1 8 16 0.71 61.3G 3.1G 0.0 0.0
ip-172-31-128-240 lx-amd64 16 1 8 16 0.22 61.3G 1.4G 0.0 0.0
ip-172-31-128-247 lx-amd64 16 1 8 16 - 61.3G - 0.0 -
ip-172-31-128-37 lx-amd64 16 1 8 16 - 61.3G - 0.0 -
ip-172-31-128-44 lx-amd64 16 1 8 16 - 61.3G - 0.0 -
ip-172-31-128-53 lx-amd64 16 1 8 16 - 61.3G - 0.0 -
ip-172-31-128-58 lx-amd64 16 1 8 16 - 61.3G - 0.0 -
ip-172-31-128-6 lx-amd64 16 1 8 16 0.58 61.3G 1.3G 0.0 0.0
ip-172-31-128-60 lx-amd64 16 1 8 16 1.03 61.3G 3.3G 0.0 0.0
ip-172-31-128-78 lx-amd64 16 1 8 16 - 61.3G - 0.0 -
ip-172-31-128-89 lx-amd64 16 1 8 16 - 61.3G - 0.0 -
ip-172-31-128-96 lx-amd64 16 1 8 16 0.31 61.3G 1.3G 0.0 0.0
Looking specifically at one node:
[root@ip-172-31-128-116 log]# qhost -h ip-172-31-128-20
HOSTNAME ARCH NCPU NSOC NCOR NTHR LOAD MEMTOT MEMUSE SWAPTO SWAPUS
----------------------------------------------------------------------------------------------
global - - - - - - - - - -
ip-172-31-128-20 lx-amd64 16 1 8 16 - 61.3G - 0.0 -
sqswatcher history of this node:
2018-11-01 09:04:54,790 INFO [sqswatcher:pollQueue] eventType=cfncluster:COMPUTE_READY
2018-11-01 09:04:54,790 INFO [sqswatcher:pollQueue] instanceId=i-07d49141430c93288
2018-11-01 09:04:55,506 INFO [sqswatcher:pollQueue] Adding Hostname: ip-172-31-128-20
2018-11-01 09:04:55,507 INFO [sge:addHost] Adding ip-172-31-128-20 with 8 slots
ip-172-31-128-20.us-west-2.compute.internal added to administrative host list
ip-172-31-128-20.us-west-2.compute.internal added to submit host list
[email protected] added "ip-172-31-128-20.us-west-2.compute.internal" to exechost list
2018-11-01 09:04:56,507 INFO [sge:addHost] Connecting to host: ip-172-31-128-20 iter: 0
2018-11-01 09:04:56,613 INFO [transport:_log] Connected (version 2.0, client OpenSSH_7.4)
2018-11-01 09:04:56,897 INFO [transport:_log] Authentication (publickey) successful!
[email protected] modified "@allhosts" in host group list
[email protected] modified "all.q" in cluster queue list
2018-11-01 09:33:10,339 INFO [sqswatcher:pollQueue] eventType=autoscaling:EC2_INSTANCE_TERMINATE
2018-11-01 09:33:10,340 INFO [sqswatcher:pollQueue] instanceId=i-07d49141430c93288
2018-11-01 09:33:10,383 INFO [sge:removeHost] Removing ip-172-31-128-20
[email protected] removed "ip-172-31-128-20.us-west-2.compute.internal" from administrative host list
[email protected] modified "all.q" in cluster queue list
[email protected] modified "@allhosts" in host group list
Host object "ip-172-31-128-20" is still referenced in cluster queue "all.q".
2018-11-01 09:33:10,564 ERROR [sge:__runSgeCommand] Failed to run ['/opt/sge/bin/lx-amd64/qconf', '-de', 'ip-172-31-128-20']
[email protected] removed "ip-172-31-128-20.us-west-2.compute.internal" from submit host list
There is a job that still appears running on ip-172-31-128-20, though the node no longer exists:
[root@ip-172-31-128-116 log]# qstat -u '*' | grep '\<172.31.128.20\>'
3101 0.25500 dvsim build r 11/01/2018 09:31:51 [email protected] 1
Looking at the times, the node came up at 09:04. A job was scheduled at 09:31. sqswatcher shows that an attempt was made to remove the node at 09:33.
The terminated nodes do not exist in all.q or @allhosts:
[root@ip-172-31-128-116 log]# qconf -sq all.q
qname all.q
hostlist @allhosts
seq_no 0
load_thresholds np_load_avg=1.75
suspend_thresholds NONE
nsuspend 1
suspend_interval 00:05:00
priority 0
min_cpu_interval 00:05:00
processors UNDEFINED
qtype BATCH INTERACTIVE
ckpt_list NONE
pe_list make smp mpi
rerun TRUE
slots 1,[ip-172-31-128-103.us-west-2.compute.internal=8], \
[ip-172-31-128-240.us-west-2.compute.internal=8], \
[ip-172-31-128-128.us-west-2.compute.internal=8], \
[ip-172-31-128-96.us-west-2.compute.internal=8], \
[ip-172-31-128-44.us-west-2.compute.internal=8], \
[ip-172-31-128-60.us-west-2.compute.internal=8], \
[ip-172-31-128-6.us-west-2.compute.internal=8], \
[ip-172-31-128-203.us-west-2.compute.internal=8], \
[ip-172-31-128-18.us-west-2.compute.internal=8], \
[ip-172-31-128-207.us-west-2.compute.internal=8], \
[ip-172-31-128-19.us-west-2.compute.internal=8], \
[ip-172-31-128-134.us-west-2.compute.internal=8], \
[ip-172-31-128-119.us-west-2.compute.internal=8]
tmpdir /tmp
shell /bin/sh
prolog NONE
epilog NONE
shell_start_mode posix_compliant
starter_method NONE
suspend_method NONE
resume_method NONE
terminate_method NONE
notify 00:00:60
owner_list NONE
user_lists NONE
xuser_lists NONE
subordinate_list NONE
complex_values NONE
projects NONE
xprojects NONE
calendar NONE
initial_state default
s_rt INFINITY
h_rt INFINITY
s_cpu INFINITY
h_cpu INFINITY
s_fsize INFINITY
h_fsize INFINITY
s_data INFINITY
h_data INFINITY
s_stack INFINITY
h_stack INFINITY
s_core INFINITY
h_core INFINITY
s_rss INFINITY
h_rss INFINITY
s_vmem INFINITY
h_vmem INFINITY
[root@ip-172-31-128-116 log]# qconf -shgrp @allhosts
group_name @allhosts
hostlist ip-172-31-128-103.us-west-2.compute.internal \
ip-172-31-128-240.us-west-2.compute.internal \
ip-172-31-128-128.us-west-2.compute.internal \
ip-172-31-128-96.us-west-2.compute.internal \
ip-172-31-128-44.us-west-2.compute.internal \
ip-172-31-128-60.us-west-2.compute.internal \
ip-172-31-128-6.us-west-2.compute.internal \
ip-172-31-128-203.us-west-2.compute.internal \
ip-172-31-128-18.us-west-2.compute.internal \
ip-172-31-128-207.us-west-2.compute.internal \
ip-172-31-128-19.us-west-2.compute.internal \
ip-172-31-128-134.us-west-2.compute.internal \
ip-172-31-128-119.us-west-2.compute.internal
The node cannot be removed:
[root@ip-172-31-128-116 log]# qconf -de ip-172-31-128-20.compute.internal
Host object "ip-172-31-128-20.compute.internal" is still referenced in cluster queue "all.q".
Please let me know if there is any more information I can provide. Thanks!
tkuzuhara
on 1 Nov 2018
Hi tkuzuhara,
thank you very much for the detailed and helpful report!
If you still have the possibility I'd kindly ask if you can verify with a couple more dead nodes, e.g. ip-172-31-128-247 ip-172-31-128-37 ip-172-31-128-78, if there were jobs scheduled to those nodes few minutes before the instance termination message caught by the sqswatcher.
Thanks
mauri-melato
on 6 Nov 2018
Unfortunately it hasn't fixed. I can confirm that the bug is still there.
pro100frol
on 12 Nov 2018
Ok thank you for reporting this, we'll check asap.
enrico-usai
on 12 Nov 2018
@mauri-melato, sorry to have not followed up on your earlier request for more info. It appears clear that this issue is still happening for a lot of people (based on @pro100frol 's comment, and #644 seems to be another duplicate). This is by far the biggest problem we are having with cfncluster, because it leaves jobs and nodes hanging indefinitely, requiring frequent manual intervention, including complete teardown of clusters when there are too many dead compute nodes that cannot be removed.
Are you guys able to replicate and troubleshoot this issue? I'm also wondering if there is any evidence that parallelcluster 2.0.2+ addresses this issue? Any update would be appreciated. Thanks a lot!
tkuzuhara
on 4 Jan 2019
FYI I'm on parallelcluster 2.1.1 and still seeing this issue.
100% reproducible just by submitting a job to the cluster, then sudo shutdown -h now the node that that job is running on
root@ip-172-31-128-23 /var/log $ qconf -shgrp @allhosts
group_name @allhosts
hostlist ip-172-31-128-112.us-west-2.compute.internal
root@ip-172-31-128-23 /var/log $ qhost
HOSTNAME ARCH NCPU NSOC NCOR NTHR LOAD MEMTOT MEMUSE SWAPTO SWAPUS
----------------------------------------------------------------------------------------------
global - - - - - - - - - -
ip-172-31-128-112 lx-amd64 72 2 36 72 0.26 137.3G 1.6G 0.0 0.0
ip-172-31-128-245 lx-amd64 72 2 36 72 - 137.3G - 0.0 -
root@ip-172-31-128-23 /var/log $ qstat -u '*'
job-ID prior name user state submit/start at queue slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
6 0.55500 sleep keien r 01/16/2019 16:42:45 [email protected] 1
root@ip-172-31-128-23 /var/log $ qconf -de ip-172-31-128-245
Host object "ip-172-31-128-245" is still referenced in cluster queue "all.q".
the node that I killed manually in the above is ip-172-31-128-245 (this was after trying to qdel -f the job that was still on the dead host)
keien
on 17 Jan 2019
Hi all,
this issue has been fixed by: aws/aws-parallelcluster-node#98
When all the jobs referencing the terminated instance will be delete, then the host will be removed from the scheduler. We have also plans to change the default SGE configuration in order to allow non-admin users to be able to force delete jobs aws/aws-parallelcluster-cookbook#259
I'm going to close this case since the patch has been merged in the develop branch.
It will be available in the next version of AWS ParallelCluster.
Please let us know if you have any questions.
lukeseawalker
on 13 Feb 2019
We appreciate the update but this doesn't address the core issue - if the node goes down and the jobs are still left in the r state, they will never get taken off of the queue, thus the host will never be removed from the scheduler anyway.
keien
on 15 Feb 2019
Hi,
in a normal condition, there shouldn't be host going down with job still running. If this occurs we need to investigate as different issue.
On the other side, when a manual termination of the host is done, we are now providing a way to bring the cluster back to a clean state, requiring a manual termination of the job too.
lukeseawalker
on 19 Feb 2019
What about spot instances being taken away by AWS? Wouldn't that run into this same situation?
keien
on 25 Feb 2019
Absolutely yes, spot and manual termination will fall into the same situation.
But so far we do not do any operation on the jobs, we let the scheduler or the user be the only authoritative sources for job status.
lukeseawalker
on 1 Mar 2019
I agree with @keien There are cases where an instance can fail even if there are running jobs. Working with spot instances is probably the most important one. It would be very useful if AWS ParallelCluster could deal with this situation
tkainrad
on 16 Apr 2019
Related issues
calebwin
路
4Comments
microbioticajon
路
7Comments
tom-dyar
路
5Comments
huribeir
路
3Comments
floswald
路
5Comments
Most helpful comment
This bug has been resolved in the upcoming 1.6.0 release:
https://github.com/awslabs/cfncluster/blob/develop/CHANGELOG.rst#160
If you are willing to test the release candindate you can install it with:
pip install -Iv cfncluster==1.6.0rc1Please keep us updated if you have the opportunity to test the rc1 or please stay tuned for the upcoming 1.6.0 release.