[prev in list] [next in list] [prev in thread] [next in thread]
List: npaci-rocks-discussion
Subject: [Rocks-Discuss] Re: ssh: connect to host compute-0-0 port 22: Connection refused
From: Juan Eduardo Ramirez Vargas <juaneduardo.ramirez () upr ! edu>
Date: 2017-07-19 15:41:29
Message-ID: CA+EmpbCUyKigz4bJiWOqkBiR7HsW82=FY9ZaFdLKAX6Zftxf7w () mail ! gmail ! com
[Download RAW message or body]
Hi,
Sounds that your nodes most likely were power off improperly. And they
run rocks-grub.
When this happen the compute nodes at boot try to reinstall themselves.
If you want to avoid reinstall process. When the boot reaches grub for
reinstall just type any key to get the grub menu, then you should see the
second option to normal boot.
If you don't want to force reinstall nodes at power failure. It requires to
turn of rocks-grub 'process'.
/etc/init.d/rocks-grub stop (if already set)
chkconfig rocks-grub off (permanently set it off).
By default rocks do 'chkconfig rocks-grub on'
You may want eventually put in your extent-compute.xml a line to turn off
rocks-grub, in case don't want this rocks feature.
I hope this help you
Eduardo
On Tue, Jul 18, 2017 at 6:13 PM, John Price <jprice@csudh.edu> wrote:
> Oh, yeah. That happens sometimes too. :)
>
> I've found that tripwire is usually the culprit here. I have not spent
> any time trying to fix this; I just go in once in a while and delete the
> log files, and all is well.
>
> I should have mentioned that; I think someone else did somewhere in the
> thread.
>
> John
>
> On Tue, 2017-07-18 at 21:55 +0000, Info wrote:
> > Thanks for all your help. The issue is now resolved. /var on the
> headnode was 100% full. So that is what caused all those python errors when
> running rocks list...
> >
> > I found this out because I noticed mysqld was stopped. When I tried to
> restart it , it said, "no space on device". I then cleared out the log
> files in /var/log . Now everything seems to be working fine now!
> >
> > From: John Price <jprice@csudh.edu>
> > To: Discussion of Rocks Clusters <npaci-rocks-discussion@sdsc.edu>
> > Sent: Tuesday, July 18, 2017 5:10 PM
> > Subject: [Rocks-Discuss] Re: ssh: connect to host compute-0-0 port 22:
> Connection refused
> >
> > I suspect that there is not. In your message at 12:06 PDT (from a
> > different branch of this thread), you say:
> >
> > =====
> > I tried monitoring the boot up process. The messages are:
> >
> > Trying to load: pxelinux.cfg/0A1FFFE
> > Booting from local disk
> > PXE-MOF: Exiting Qlogic PXE ROM
> >
> > Booting from Hard Drive C:
> > Press any key to enter the menu
> >
> > Booting Rocks Reinstall in 0 secionds....
> > Greetings
> > ... bunch of messages on mounting /proc, /dev/, /sys/, /tmp etc.
> > =====
> >
> > The line "Booting from local disk" means that the compute node has asked
> > the head node whether or not it needs to download a new system (if, for
> > instance, something had changed), and was told "no" by the head node
> > (which is why it then tries to boot from the local disk).
> >
> > This contradicts your previous statements that it's asking you where the
> > installation images are, which is what led me to defer to Philip et al.
> >
> > I now find your situation fairly confusing. On the one hand, you don't
> > seem to have a system installed on the compute node. On the other hand,
> > you get some information back from the "rocks list profile" command; if
> > there were no system on compute-0-0, then the head node shouldn't even
> > recognize the name, and should tell you accordingly. For instance, on
> > my system, I get (my highest node is compute-0-43):
> > =====
> > [root@hadlab ~]# rocks list host profile compute-0-44
> > Error: cannot resolve host "compute-0-44"
> > [host]...
> > [root@hadlab ~]#
> > =====
> >
> > You might need a "rocks sync config" in there somewhere, but I would
> > wait for one of the real gurus to chime in. I'm sure that building
> > everything from scratch will work, but it may be going down a rabbit
> > hole you're not ready to go down just yet.
> >
> > John
> >
> > On Tue, 2017-07-18 at 20:42 +0000, Info wrote:
> > > Thanks John. Before I try the "nuclear" option, is there a way to boot
> compute-0-0 to it's last working state? I.e, I don't want it to reinstall
> the Rocks image. I just want to get it to where it was a few days ago when
> it was working fine. I tried hitting F11 in the boot process but am not
> sure what to do.
> > >
> > >
> > > From: John Price <jprice@csudh.edu>
> > > To: Discussion of Rocks Clusters <npaci-rocks-discussion@sdsc.edu>
> > > Sent: Tuesday, July 18, 2017 3:37 PM
> > > Subject: [Rocks-Discuss] Re: ssh: connect to host compute-0-0 port
> 22: Connection refused
> > >
> > > We're now past my level of expertise here. If (a) you have no
> > > extend-compute.xml file, and (b) you're still getting kickstart errors
> > > (this is what you're seeing in this message), there's something deeper
> > > going wrong here. What I would do here is give up and start over by
> > > reinstalling rocks on the head node from scratch, which isn't nearly as
> > > painful as it sounds. Actually, it will probably go much faster than
> > > trying to sort this out.
> > >
> > > If you go this route, make certain that you have all your network
> > > parameters written down before you start:
> > >
> > > IP address
> > > IP netmask
> > > IP gateway
> > > DNS servers
> > >
> > > It's also a good idea to back up your entire /export directory
> someplace
> > > so you can get to the old files after you're done
> > > (especially /export/home, but it's good to have a backup of everything
> > > in /export/rocks/install as well).
> > >
> > > This "nuclear option" is not without its risks, but those risks are
> > > mostly easy to understand, and therefore circumvent (largely related to
> > > not backing up files that you will need later). At the end of the
> > > process, you'll have a clean system that you can then proceed to
> destroy
> > > on your own terms. :)
> > >
> > > John
> > >
> > > On Tue, 2017-07-18 at 19:09 +0000, Info wrote:
> > > > I have tried those commands on the head node. Unfortunately, the
> last one gives errors:
> > > >
> > > > "rocks list host profile compute-0-0"
> > > >
> > > > Traceback (most recent call last):
> > > > File "/opt/rocks/bin/rocks", line 259, in <module>
> > > > command.runWrapper(name, args[i:])
> > > > File "/opt/rocks/lib/python2.6/site-packages/rocks/commands/__init__.py",
> line 1899, in runWrapper
> > > > self.run(self._params, self._args)
> > > > File "/opt/rocks/lib/python2.6/site-packages/rocks/commands/
> list/host/profile/__init__.py", line 300, in run
> > > > for host in self.getHostnames(args):
> > > > File "/opt/rocks/lib/python2.6/site-packages/rocks/commands/__init__.py",
> line 747, in getHostnames
> > > > min,max = self.db.fetchone()
> > > > ValueError: need more than 0 values to unpack
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > ________________________________
> > > > From: Ian Kaufman <ikaufman@eng.ucsd.edu>
> > > > To: Discussion of Rocks Clusters <npaci-rocks-discussion@sdsc.edu>
> > > > Sent: Tuesday, July 18, 2017 2:40 PM
> > > > Subject: [Rocks-Discuss] Re: ssh: connect to host compute-0-0 port
> 22: Connection refused
> > > >
> > > >
> > > >
> > > > Have you tried the following suggestion from John Price:
> > > >
> > > > cd /export/rocks/install
> > > > rocks create distro
> > > > rocks list host profile compute-0-0
> > > >
> > > > and see a kickstart file, or, if errors, can you post the errors?
> > > >
> > > > Ian
> > > >
> > > >
> > > >
> > > > On Tue, Jul 18, 2017 at 11:32 AM, Josue Sanchez <
> > > > Josue.Sanchez@faradayfuture.com> wrote:
> > > >
> > > > > When it reboots, can you monitor the boot up process and see if the
> > > > > pxeboot screen provides any errors before it takes you to the
> screen you
> > > > > see now?
> > > > >
> > > > >
> > > > >
> > > > > -----Original Message-----
> > > > > From: npaci-rocks-discussion-bounces@sdsc.edu [mailto:npaci-rocks-
> > > > > discussion-bounces@sdsc.edu] On Behalf Of Info
> > > > > Sent: Tuesday, July 18, 2017 10:57 AM
> > > > > To: Discussion of Rocks Clusters <npaci-rocks-discussion@sdsc.edu>
> > > > > Subject: [Rocks-Discuss] Re: ssh: connect to host compute-0-0 port
> 22:
> > > > > Connection refused
> > > > >
> > > > > I just tried rebooting, but unfortunately it comes back to the
> same page
> > > > > asking for "Installation Method".
> > > > >
> > > > >
> > > > > From: Josue Sanchez <Josue.Sanchez@faradayfuture.com>
> > > > > To: Discussion of Rocks Clusters <npaci-rocks-discussion@sdsc.edu
> >
> > > > > Sent: Tuesday, July 18, 2017 1:44 PM
> > > > > Subject: [Rocks-Discuss] Re: ssh: connect to host compute-0-0
> port 22:
> > > > > Connection refused
> > > > >
> > > > > Did you try rebooting it to see if it lands there again?
> > > > >
> > > > > Maybe the headnode wasn't fully up so it didn't get the image, now
> that
> > > > > headnode is fully up it might find it.
> > > > >
> > > > > -----Original Message-----
> > > > > From: npaci-rocks-discussion-bounces@sdsc.edu [mailto:npaci-rocks-
> > > > > discussion-bounces@sdsc.edu] On Behalf Of Info
> > > > > Sent: Tuesday, July 18, 2017 10:35 AM
> > > > > To: Discussion of Rocks Clusters <npaci-rocks-discussion@sdsc.edu>
> > > > > Subject: [Rocks-Discuss] Re: ssh: connect to host compute-0-0 port
> 22:
> > > > > Connection refused
> > > > >
> > > > > Thanks for all the help. I figured out how to access the virtual
> console
> > > > > using Dell Remote Access Controller 8. I can see that compute-0-0
> is stuck
> > > > > on this:
> > > > >
> > > > > Welcome to Rocks for x86_64
> > > > >
> > > > > Installation Method
> > > > >
> > > > > What type of media contains the installation image?
> > > > > --various options--
> > > > >
> > > > > Any ideas on where the image would be?
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > ________________________________
> > > > > From: Ian Kaufman <ikaufman@eng.ucsd.edu>
> > > > > To: Discussion of Rocks Clusters <npaci-rocks-discussion@sdsc.edu>
> > > > > Sent: Tuesday, July 18, 2017 12:33 PM
> > > > > Subject: [Rocks-Discuss] Re: ssh: connect to host compute-0-0 port
> 22:
> > > > > Connection refused
> > > > >
> > > > >
> > > > >
> > > > > It's a possibility. We really don't know the state of the compute
> nodes.
> > > > > They may have broken installs from before the head going down, or
> the
> > > > > reinstallation failed at the same time.
> > > > >
> > > > > Not being able to get to the compute nodes locally, or through a
> remote
> > > > > management console, is going to make analysis very difficult.
> > > > >
> > > > > On Tue, Jul 18, 2017 at 9:22 AM, Josue Sanchez <
> > > > > Josue.Sanchez@faradayfuture.com> wrote:
> > > > >
> > > > > > So if the DHCP server did hand out the ip, which If I'm correct
> runs on
> > > > > > the head node, that means the head node would have been up
> before the
> > > > > > compute node so there shouldn't be an issue with the compute
> node getting
> > > > > > the image?
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: npaci-rocks-discussion-bounces@sdsc.edu [mailto:
> npaci-rocks-
> > > > > > discussion-bounces@sdsc.edu] On Behalf Of Ian Kaufman
> > > > > > Sent: Tuesday, July 18, 2017 9:17 AM
> > > > > > To: Discussion of Rocks Clusters <npaci-rocks-discussion@sdsc.
> edu>
> > > > > > Subject: [Rocks-Discuss] Re: ssh: connect to host compute-0-0
> port 22:
> > > > > > Connection refused
> > > > > >
> > > > > > Yes, it is possible. During the boot process, thr DHCP server
> will hand
> > > > > > out the IP address so that the rest of the install process can
> take
> > > > > place.
> > > > > >
> > > > > > On Jul 18, 2017 9:14 AM, "Josue Sanchez"
> <Josue.Sanchez@faradayfuture.
> > > > > com>
> > > > > > wrote:
> > > > > >
> > > > > > > Would nodes still be able to get pinged if they didn't pxeboot
> or get
> > > > > > > the rocks image?
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: npaci-rocks-discussion-bounces@sdsc.edu [mailto:
> npaci-rocks-
> > > > > > > discussion-bounces@sdsc.edu] On Behalf Of Ian Kaufman
> > > > > > > Sent: Tuesday, July 18, 2017 8:31 AM
> > > > > > > To: Discussion of Rocks Clusters <npaci-rocks-discussion@sdsc.
> edu>
> > > > > > > Subject: [Rocks-Discuss] Re: ssh: connect to host compute-0-0
> port 22:
> > > > > > > Connection refused
> > > > > > >
> > > > > > > I think John is going down the right path. If the head node was
> > > > > > > rebooted this morning, for no apparent reason, I suspect the
> compute
> > > > > > > nodes may have been as well. When the compute nodes reject SSH
> > > > > > > connections, it is usually either because SSH isn't running,
> or the
> > > > > > > system is overloaded. If the head and compute nodes were
> rebooted
> > > > > > > simultaneously, it is possible the head node was not fully up
> while
> > > > > > > the compute nodes were trying to PXEboot or get the ROCKS
> image. Or,
> > > > > > > as John pointed out, the extend-compute.xml file is missing
> and should
> > > > > > be there.
> > > > > > >
> > > > > > > Ian
> > > > > > >
> > > > > > > On Mon, Jul 17, 2017 at 5:50 PM, John W. Price <
> jprice@csudh.edu>
> > > > > wrote:
> > > > > > >
> > > > > > > > If extend-compute.xml does not exist (important note: this
> is on the
> > > > > > > > head node!), then this is *not* the problem. Rocks will
> just skip
> > > > > > > > over it and happily move on.
> > > > > > > >
> > > > > > > > My next best guess is that, at some point in the past (you
> said you
> > > > > > > > inherited the cluster, IIRC), there *was* an
> extend-compute.xml
> > > > > > > > file, but that it was deleted somehow (perhaps in an effort
> to
> > > > > > > > "start from scratch", although there are much better ways to
> do
> > > > > > > > that). In that case, you probably want to do the following
> (again,
> > > > > on
> > > > > > the head node):
> > > > > > > >
> > > > > > > > cd /export/rocks/install
> > > > > > > > rocks create distro
> > > > > > > > rocks list host profile compute-0-0
> > > > > > > >
> > > > > > > > at which point you should see something that looks like an
> actual
> > > > > > > > kickstart file (instead of the error messages you're seeing
> now).
> > > > > > > > Then you can reboot your compute nodes, and that should get
> you back
> > > > > > > > to what you had before.
> > > > > > > >
> > > > > > > > Note that if there was an extend-compute.xml file in the
> past, there
> > > > > > > > was probably a good reason for it, and that if you can't
> find it,
> > > > > > > > you may have some fun reconstructing what used to be in it.
> > > > > > > >
> > > > > > > > John
> > > > > > > >
> > > > > > > > --
> > > > > > > > John W. Price
> > > > > > > > Professor and Chair of Physics, CSU Dominguez Hills
> Director, Office
> > > > > > > > of Undergraduate Research, Scholarship, and Creative Activity
> > > > > > > > 310-243-3403
> > > > > > > >
> > > > > > > > ________________________________________
> > > > > > > > From: npaci-rocks-discussion-bounces@sdsc.edu
> > > > > > > > <npaci-rocks-discussion- bounces@sdsc.edu> on behalf of Info
> > > > > > > > <rentalcondo2012-info@yahoo.com>
> > > > > > > > Sent: Monday, July 17, 2017 15:18
> > > > > > > > To: Discussion of Rocks Clusters
> > > > > > > > Subject: [Rocks-Discuss] Re: ssh: connect to host
> compute-0-0 port
> > > > > 22:
> > > > > > > > Connection refused
> > > > > > > >
> > > > > > > > Thanks for the information. I checked my head node here:
> > > > > > > >
> > > > > > > > /export/rocks/install/site-profiles/6.2/nodes/
> > > > > > > >
> > > > > > > > and I only see a skeleton.xml file . I also noticed the head
> node
> > > > > > > > was rebooted by the system early this morning. So I guess
> that has
> > > > > > > > something to do with the problem. I checked
> /var/log/messages but
> > > > > > > > can not find any reason why it rebooted.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > ________________________________
> > > > > > > > From: John Price <jprice@csudh.edu>
> > > > > > > > To: Discussion of Rocks Clusters <
> npaci-rocks-discussion@sdsc.edu>
> > > > > > > > Sent: Monday, July 17, 2017 6:08 PM
> > > > > > > > Subject: [Rocks-Discuss] Re: ssh: connect to host
> compute-0-0 port
> > > > > 22:
> > > > > > > > Connection refused
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > I don't think this is it. From the OP, it looks like they
> aren't
> > > > > > > > being kickstarted properly. Having rebuilt my own cluster
> several
> > > > > > > > times in the last month (on purpose; I'm teaching my
> students how to
> > > > > > > > do it...), this is almost always an issue with the
> > > > > > > > extend-compute.xml file, contained in
> > > > > > > > /export/rocks/install/site-profiles/{rocks-release-
> > > > > > > > version}/nodes/extend-compute.xml
> > > > > > > >
> > > > > > > > where {rocks-release-version} is the number listed in
> > > > > > > > /etc/rocks-release.
> > > > > > > >
> > > > > > > > The "easy" way to fix this is to rename the current version
> of
> > > > > > > > extend-compute.xml to old-extend-compute.xml (or something
> similar
> > > > > > > > that won't be parsed when you run the rocks create distro
> command),
> > > > > > > > and rebuild it from scratch, starting with skeleton.xml.
> > > > > > > >
> > > > > > > > It's not the fastest way, but until you've done this a few
> hundred
> > > > > > > > times, it will certainly be the easiest.
> > > > > > > >
> > > > > > > > John
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, 2017-07-17 at 21:53 +0000, Josue Sanchez wrote:
> > > > > > > > > Can you confirm the nodes are on? Can you ping the nodes?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > -----Original Message-----
> > > > > > > > > From: npaci-rocks-discussion-bounces@sdsc.edu [mailto:
> npaci-rocks-
> > > > > > > > discussion-bounces@sdsc.edu] On Behalf Of Info
> > > > > > > > > Sent: Monday, July 17, 2017 2:15 PM
> > > > > > > > > To: npaci-rocks-discussion@sdsc.edu
> > > > > > > > > Subject: [Rocks-Discuss] ssh: connect to host compute-0-0
> port 22:
> > > > > > > > Connection refused
> > > > > > > > >
> > > > > > > > > I inherited a four node cluster from someone who left our
> company
> > > > > > > > > last
> > > > > > > > week. The first thing I noticed is that were was no nfs
> filesystem
> > > > > > > > between the nodes. So I manually created that by editing
> > > > > > > > /etc/exports on the head node and mounted the folder on the
> compute
> > > > > > > > node with the
> > > > > > > "mount" command.
> > > > > > > > This worked well and I was able to run some simple
> hello_world mpi
> > > > > > > > programs! Passwordless ssh worked perfectly between all the
> compute
> > > > > > > > nodes and the head nodes.
> > > > > > > > >
> > > > > > > > > But today when I came to work, I find I can not ssh to any
> of the
> > > > > > > > compute nodes! I get the error:
> > > > > > > > >
> > > > > > > > > ssh: connect to host compute-0-0 port 22: Connection
> refused
> > > > > > > > >
> > > > > > > > > I am curious what happened overnight to cause this? It
> seems like
> > > > > > > > > all
> > > > > > > > the nodes rebooted themselves overnight and sshd is not
> running?
> > > > > > > > Unfortunately the machines are in a different state so I can
> not
> > > > > > > > easily connect a monitor to see the issue.
> > > > > > > > >
> > > > > > > > > I tried doing this on the head node, "rocks list host
> profile
> > > > > > > > compute-0-0", but got these errors:
> > > > > > > > >
> > > > > > > > > Traceback (most recent call last):
> > > > > > > > > File "/opt/rocks/bin/rocks", line 259, in <module>
> > > > > > > > > command.runWrapper(name, args[i:])
> > > > > > > > > File
> > > > > > > > > "/opt/rocks/lib/python2.6/site-packages/rocks/commands/_
> _init__.py
> > > > > > > > > ",
> > > > > > > > line 1899, in runWrapper
> > > > > > > > > self.run(self._params, self._args)
> > > > > > > > > File "/opt/rocks/lib/python2.6/
> site-packages/rocks/commands/
> > > > > > > > list/host/profile/__init__.py", line 300, in run
> > > > > > > > > for host in self.getHostnames(args):
> > > > > > > > > File
> > > > > > > > > "/opt/rocks/lib/python2.6/site-packages/rocks/commands/_
> _init__.py
> > > > > > > > > ",
> > > > > > > > line 747, in getHostnames
> > > > > > > > > min,max = self.db.fetchone()
> > > > > > > > >
> > > > > > > > > ValueError: need more than 0 values to unpack
> > > > > > > > >
> > > > > > > > > Any tips on a quick fix to the issue?
> > > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > John W. Price
> > > > > > > > Professor and Chair, CSUDH Department of Physics Director,
> Office of
> > > > > > > > Undergraduate Research, Scholarship, and Creative Activity
> > > > > > > > 310-243-3403
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Ian Kaufman
> > > > > > > Research Systems Administrator
> > > > > > > UC San Diego, Jacobs School of Engineering ikaufman AT ucsd
> DOT edu
> > > > > > > -------------- next part -------------- An HTML attachment was
> > > > > > > scrubbed...
> > > > > > > URL:
> > > > > > > http://lists.sdsc.edu/pipermail/npaci-rocks-
> discussion/attachments/
> > > > > > > 20170718/fdaeeb23/attachment.html
> > > > > > >
> > > > > > >
> > > > > > -------------- next part --------------
> > > > > > An HTML attachment was scrubbed...
> > > > > > URL: http://lists.sdsc.edu/pipermail/npaci-rocks-
> discussion/attachments/
> > > > > > 20170718/ab115ad1/attachment.html
> > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Ian Kaufman
> > > > > Research Systems Administrator
> > > > > UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu
> > > > > -------------- next part --------------
> > > > > An HTML attachment was scrubbed...
> > > > > URL: http://lists.sdsc.edu/pipermail/npaci-rocks-
> discussion/attachments/
> > > > > 20170718/e474c41d/attachment.html
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > -------------- next part --------------
> > > > > An HTML attachment was scrubbed...
> > > > > URL: http://lists.sdsc.edu/pipermail/npaci-rocks-
> discussion/attachments/
> > > > > 20170718/9cfd76be/attachment.html
> > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> >
>
> --
> John W. Price
> Professor and Chair, CSUDH Department of Physics
> Director, Office of Undergraduate Research, Scholarship, and Creative
> Activity
> 310-243-3403
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20170719/a0b41420/attachment.html \
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic