[prev in list] [next in list] [prev in thread] [next in thread]
List: aix-l
Subject: Re: HACMP: umount fails and fuser finds nothing - Fix and Work-around
From: Christopher Baker <cbaker () GOODYEAR ! COM>
Date: 2007-01-23 22:22:57
Message-ID: OF66CCF956.AB7EEE3A-ON8525726C.0079665B-8525726C.007AF306 () goodyear ! com
[Download RAW message or body]
This is a multipart message in MIME format.
--=_alternative 007AF3048525726C_=
Content-Type: text/plain; charset="US-ASCII"
Folks,
This is a BUG in AIX. There is a APAR for it:
APAR IY84689. NFS SERVER DOES NOT RELEASE LOCKS AFTER CLIENT REBOOT
Below is a link to APAR IY84689
http://www-1.ibm.com/support/docview.wss?uid=isg1IY84689
We had a limited maintenance window this past weekend before we got this
info on the APAR. Taking a clue from the HACMP script
"/usr/es/sbin/cluster/events/utils/cl_deativate_fs", we saw that HACMP
tries three times to umount a local filesystem. With that it tries
"fuser" too. If it still is not unmounted, they do a "stopsrc -s
rpc.lockd", then try 57 more times to umount.
Later, they restart the rpc.lockd.
Due to the need to handle quota'ed filesystems and control the order that
filesystems are exported, we do not have HACMP manage the exports. So,
this "stopsrc -s rpc.lockd" step is skipped in the "cl_deactive_fs" script
because it first checks to see if it controls the exporting of the
filesystem in HACMP by doing a "odmget" for it.
odmget -q "name=EXPORT_FILESYSTEM AND group=$GROUPNAME" HACMPresource |
grep value | awk '{print $3}' | sed 's/"//g' | grep -w $FS
Since it is not there, it never ran the stopsrc command but just tried
umount 60 times and fails.
We first did a "stopsrc -g nfs" in our HACMP stop script. That worked,
but it was too radical... Especially if you were failing back from the 2nd
NFS server back to your own server. We then found that just refreshing
the rpc.lockd was all that was needed. That is when we found that HACMP
was trying that too.
We will try this APAR in the distant future when we have another
maintenance day.
If some one else tries it, please let us know if there is any "side
effects" and if does it works.
Thanks,
Chris Baker
IBM AIX Discussion List <aix-l@Princeton.EDU> wrote on 01/10/2007 04:56:03
PM:
>
> No, all VG's (except rootvg) are set to "No" for Auto Vary On.
>
> So, if I do a lsvg, I see all the VG's from both servers. But, if I
> do a lsvg -o, I only see the local ones that HACMP varied on. Of
> course, if NFS2 is failed over to NFS1, than a lsvg -o is the same
> as lsvg on NFS1 because NFS2's VG's have been taken over by NFS1.
>
> But, again, I am having the problem before the VG's are varied off.
>
> Thanks
>
>
> IBM AIX Discussion List <aix-l@Princeton.EDU> wrote on 01/10/2007
01:33:13 PM:
>
> > is the volume group set to chvg -an "vgname" do lsvg
> > -o to see is set to auto vsryon if set to autovary on
> > set to no to auto varyon
> >
> > --- Christopher Baker <cbaker@GOODYEAR.COM> wrote:
> >
> > > Robert,
> > >
> > > Thank you for your reply, but I am not having
> > > trouble shutting down the
> > > AIX system. I am having the trouble just shutting
> > > down gracefully the
> > > HACMP environment or failing over.
> > >
> > > Looking through the hacmp.out, it is quite clear
> > > that HACMP is unable to
> > > unmount local filesystems that need to be unmounted
> > > so that the other
> > > system can varyonvg and mount the same filesystems
> > > and export them as the
> > > first box.
> > >
> > > If I open a shell window at this hung point, I am
> > > not able to umount the
> > > filesystem manually. If I do an lsof or a fuser,
> > > there does not appear to
> > > be anything holding the mount.
> > >
> > > Thanks,
> > >
> > > Christopher M. Baker
> > > Senior Technical Support Analyst
> > > HPCE - Linux Cluster Development and Support
> > > Goodyear Tire and Rubber Company
> > > 330.796.1725
> > >
> > > =================================================
> > > Contains Confidential and/or Proprietary
> > > Information.
> > > May not be copied or disseminated without the
> > > expressed
> > > written consent of The Goodyear Tire & Rubber
> > > Company.
> > > =================================================
> > >
> > >
> > >
> > >
> > > Robert Binkley <leebinkley@YAHOO.COM>
> > > Sent by: IBM AIX Discussion List
> > > <aix-l@Princeton.EDU>
> > > 01/09/2007 09:41 AM
> > > Please respond to
> > > IBM AIX Discussion List <aix-l@Princeton.EDU>
> > >
> > >
> > > To
> > > aix-l@Princeton.EDU
> > > cc
> > >
> > > Subject
> > > Re: HACMP: umount fails and fuser finds nothing
> > >
> > >
> > >
> > >
> > >
> > >
> > > 1: when you run cd /usr/es/sbin/cluster/cllsserv
> > > ;this
> > > will show what the cluster knows about.
> > > 2: Do you use an rc.shutdown script to restart you
> > > systems
> > > 3: If ther are entries in the /etc/inittab that the
> > > cluster controls it may fail to shutdown due to not
> > > able to completely exit 0.
> > > 4: Have you ran an snap -r ; snap -e at the time of
> > > the error, and sent it to the ibm hacmp engineers.
> > > 5: when Looking through you hacmp.out file and
> > > greping
> > > for exit 1
> > >
> > > --- Christopher Baker <cbaker@GOODYEAR.COM> wrote:
> > >
> > > > Folks,
> > > >
> > > > Has anyone had this problem. We have two servers
> > > > (LPARS) on two different
> > > > P570's that are the two systems in a HACMP
> > > cluster.
> > > > We cannot
> > > > successfully fail over or gracefully stop HACMP
> > > > because the system is
> > > > unable to umount one or more local filesystems.
> > > In
> > > > the hacmp.out file we
> > > > see that the umount gets a "Device busy". It then
> > > > does a
> > > >
> > > > fuser -k -u -x /dev/nfs1lv11
> > > >
> > > > That returns nothing
> > > >
> > > > /dev/nfs1lv11:
> > > >
> > > > It then tries to umount the filesystem "/s" that
> > > is
> > > > on LV /dev/nfs1lv11.
> > > > This continues forever.
> > > >
> > > > At the command line, an lsof and a "fuser -d
> > > > /dev/nfs1lv11" come back
> > > > blank.
> > > >
> > > > INFO: These are NFS servers and the filesystems
> > > we
> > > > have trouble with are
> > > > exported to many IBM workstations as well as
> > > non-IBM
> > > > systems and SAMBA
> > > > shared.
> > > > We un-export all filesystems and stop the SAMBA
> > > > processes before HACMP
> > > > tries to umount them.
> > > >
> > > >
> > > > We have a similar, but much less frequent problem
> > > on
> > > > other (non-NFS
> > > > server) clusters where a zombied process will not
> > > > "kill -9" and it is
> > > > using a local or NFS mounted filesystem. If we
> > > need
> > > > to fail that over or
> > > > just stop HACMP so we can reboot, we have to do a
> > > > "shutdown".
> > > >
> > > > Any assistance would be of help.
> > > >
> > > >
> > > > Christopher M. Baker
> > > > Senior Technical Support Analyst
> > > > HPCE - Linux Cluster Development and Support
> > > > Goodyear Tire and Rubber Company
> > > > 330.796.1725
> > > >
> > > > =================================================
> > > > Contains Confidential and/or Proprietary
> > > > Information.
> > > > May not be copied or disseminated without the
> > > > expressed
> > > > written consent of The Goodyear Tire & Rubber
> > > > Company.
> > > > =================================================
> > > >
> > >
> > >
--=_alternative 007AF3048525726C_=
Content-Type: text/html; charset="US-ASCII"
<br><font size=2 face="sans-serif">Folks,</font>
<br>
<br><font size=2 face="sans-serif">This is a BUG in AIX. There is
a APAR for it:</font>
<br>
<br><font size=3>APAR IY84689. NFS SERVER DOES NOT RELEASE LOCKS AFTER
CLIENT REBOOT</font>
<br><font size=2 face="sans-serif"><br>
Below is a link to APAR IY84689</font>
<br><font size=2 face="sans-serif">http://www-1.ibm.com/support/docview.wss?uid=isg1IY84689</font>
<br>
<br>
<br><font size=2 face="sans-serif">We had a limited maintenance window
this past weekend before we got this info on the APAR. Taking a clue
from the HACMP script "/usr/es/sbin/cluster/events/utils/cl_deativate_fs",
we saw that HACMP tries three times to umount a local filesystem. With
that it tries "fuser" too. If it still is not unmounted,
they do a "stopsrc -s rpc.lockd", then try 57 more times to umount.</font>
<br>
<br><font size=2 face="sans-serif">Later, they restart the rpc.lockd.</font>
<br>
<br><font size=2 face="sans-serif">Due to the need to handle quota'ed filesystems
and control the order that filesystems are exported, we do not have HACMP
manage the exports. So, this "stopsrc -s rpc.lockd" step
is skipped in the "cl_deactive_fs" script because it first checks
to see if it controls the exporting of the filesystem in HACMP by doing
a "odmget" for it. </font>
<br>
<br><font size=2 face="sans-serif">odmget -q "name=EXPORT_FILESYSTEM
AND group=$GROUPNAME" HACMPresource | grep value | awk '{print $3}'
| sed 's/"//g' | grep -w $FS</font>
<br>
<br><font size=2 face="sans-serif">Since it is not there, it never ran
the stopsrc command but just tried umount 60 times and fails.</font>
<br>
<br><font size=2 face="sans-serif">We first did a "stopsrc -g nfs"
in our HACMP stop script. That worked, but it was too radical...
Especially if you were failing back from the 2nd NFS server back to your
own server. We then found that just refreshing the rpc.lockd was
all that was needed. That is when we found that HACMP was trying
that too.</font>
<br>
<br><font size=2 face="sans-serif">We will try this APAR in the distant
future when we have another maintenance day.</font>
<br>
<br><font size=2 face="sans-serif">If some one else tries it, please let
us know if there is any "side effects" and if does it works.</font>
<br>
<br><font size=2 face="sans-serif">Thanks,<br>
</font>
<br><font size=2 face="sans-serif"> Chris Baker</font>
<br>
<br><font size=2><tt>IBM AIX Discussion List <aix-l@Princeton.EDU>
wrote on 01/10/2007 04:56:03 PM:<br>
<br>
> <br>
> No, all VG's (except rootvg) are set to "No" for Auto Vary
On. <br>
> <br>
> So, if I do a lsvg, I see all the VG's from both servers. But,
if I<br>
> do a lsvg -o, I only see the local ones that HACMP varied on. Of
<br>
> course, if NFS2 is failed over to NFS1, than a lsvg -o is the same
<br>
> as lsvg on NFS1 because NFS2's VG's have been taken over by NFS1.
<br>
> <br>
> But, again, I am having the problem before the VG's are varied off.
<br>
> <br>
> Thanks<br>
> <br>
> <br>
> IBM AIX Discussion List <aix-l@Princeton.EDU> wrote on 01/10/2007
01:33:13 PM:<br>
> <br>
> > is the volume group set to chvg -an "vgname" do lsvg<br>
> > -o to see is set to auto vsryon if set to autovary on<br>
> > set to no to auto varyon <br>
> > <br>
> > --- Christopher Baker <cbaker@GOODYEAR.COM> wrote:<br>
> > <br>
> > > Robert,<br>
> > > <br>
> > > Thank you for your reply, but I am not having<br>
> > > trouble shutting down the <br>
> > > AIX system. I am having the trouble just shutting<br>
> > > down gracefully the <br>
> > > HACMP environment or failing over.<br>
> > > <br>
> > > Looking through the hacmp.out, it is quite clear<br>
> > > that HACMP is unable to <br>
> > > unmount local filesystems that need to be unmounted<br>
> > > so that the other <br>
> > > system can varyonvg and mount the same filesystems<br>
> > > and export them as the <br>
> > > first box.<br>
> > > <br>
> > > If I open a shell window at this hung point, I am<br>
> > > not able to umount the <br>
> > > filesystem manually. If I do an lsof or a fuser,<br>
> > > there does not appear to <br>
> > > be anything holding the mount.<br>
> > > <br>
> > > Thanks,<br>
> > > <br>
> > > Christopher M. Baker<br>
> > > Senior Technical Support Analyst<br>
> > > HPCE - Linux Cluster Development and Support<br>
> > > Goodyear Tire and Rubber Company<br>
> > > 330.796.1725<br>
> > > <br>
> > > =================================================<br>
> > > Contains Confidential and/or Proprietary<br>
> > > Information.<br>
> > > May not be copied or disseminated without the<br>
> > > expressed<br>
> > > written consent of The Goodyear Tire & Rubber<br>
> > > Company.<br>
> > > =================================================<br>
> > > <br>
> > > <br>
> > > <br>
> > > <br>
> > > Robert Binkley <leebinkley@YAHOO.COM> <br>
> > > Sent by: IBM AIX Discussion List<br>
> > > <aix-l@Princeton.EDU><br>
> > > 01/09/2007 09:41 AM<br>
> > > Please respond to<br>
> > > IBM AIX Discussion List <aix-l@Princeton.EDU><br>
> > > <br>
> > > <br>
> > > To<br>
> > > aix-l@Princeton.EDU<br>
> > > cc<br>
> > > <br>
> > > Subject<br>
> > > Re: HACMP: umount fails and fuser finds nothing<br>
> > > <br>
> > > <br>
> > > <br>
> > > <br>
> > > <br>
> > > <br>
> > > 1: when you run cd /usr/es/sbin/cluster/cllsserv<br>
> > > ;this<br>
> > > will show what the cluster knows about. <br>
> > > 2: Do you use an rc.shutdown script to restart you<br>
> > > systems<br>
> > > 3: If ther are entries in the /etc/inittab that the<br>
> > > cluster controls it may fail to shutdown due to not<br>
> > > able to completely exit 0.<br>
> > > 4: Have you ran an snap -r ; snap -e at the time of<br>
> > > the error, and sent it to the ibm hacmp engineers.<br>
> > > 5: when Looking through you hacmp.out file and<br>
> > > greping<br>
> > > for exit 1 <br>
> > > <br>
> > > --- Christopher Baker <cbaker@GOODYEAR.COM> wrote:<br>
> > > <br>
> > > > Folks,<br>
> > > > <br>
> > > > Has anyone had this problem. We have two servers<br>
> > > > (LPARS) on two different <br>
> > > > P570's that are the two systems in a HACMP<br>
> > > cluster. <br>
> > > > We cannot <br>
> > > > successfully fail over or gracefully stop HACMP<br>
> > > > because the system is <br>
> > > > unable to umount one or more local filesystems. <br>
> > > In<br>
> > > > the hacmp.out file we <br>
> > > > see that the umount gets a "Device busy".
It then<br>
> > > > does a <br>
> > > > <br>
> > > > fuser -k -u -x /dev/nfs1lv11<br>
> > > > <br>
> > > > That returns nothing<br>
> > > > <br>
> > > > /dev/nfs1lv11:<br>
> > > > <br>
> > > > It then tries to umount the filesystem "/s"
that<br>
> > > is<br>
> > > > on LV /dev/nfs1lv11. <br>
> > > > This continues forever.<br>
> > > > <br>
> > > > At the command line, an lsof and a "fuser -d<br>
> > > > /dev/nfs1lv11" come back <br>
> > > > blank.<br>
> > > > <br>
> > > > INFO: These are NFS servers and the filesystems<br>
> > > we<br>
> > > > have trouble with are <br>
> > > > exported to many IBM workstations as well as<br>
> > > non-IBM<br>
> > > > systems and SAMBA <br>
> > > > shared.<br>
> > > > We un-export all filesystems and stop the SAMBA<br>
> > > > processes before HACMP <br>
> > > > tries to umount them.<br>
> > > > <br>
> > > > <br>
> > > > We have a similar, but much less frequent problem<br>
> > > on<br>
> > > > other (non-NFS <br>
> > > > server) clusters where a zombied process will not<br>
> > > > "kill -9" and it is <br>
> > > > using a local or NFS mounted filesystem. If we<br>
> > > need<br>
> > > > to fail that over or <br>
> > > > just stop HACMP so we can reboot, we have to do a<br>
> > > > "shutdown".<br>
> > > > <br>
> > > > Any assistance would be of help.<br>
> > > > <br>
> > > > <br>
> > > > Christopher M. Baker<br>
> > > > Senior Technical Support Analyst<br>
> > > > HPCE - Linux Cluster Development and Support<br>
> > > > Goodyear Tire and Rubber Company<br>
> > > > 330.796.1725<br>
> > > > <br>
> > > > =================================================<br>
> > > > Contains Confidential and/or Proprietary<br>
> > > > Information.<br>
> > > > May not be copied or disseminated without the<br>
> > > > expressed<br>
> > > > written consent of The Goodyear Tire & Rubber<br>
> > > > Company.<br>
> > > > =================================================<br>
> > > > <br>
> > > <br>
> > > </tt></font>
--=_alternative 007AF3048525726C_=--
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic