[prev in list] [next in list] [prev in thread] [next in thread]
List: aix-l
Subject: Re: What process is causing high disk io?
From: "Weinmann, Robert" <robert.weinmann () INGENTRA ! COM>
Date: 2008-04-03 19:39:56
Message-ID: DC842A42AA10F04987EA1CFFF1080D480334BB () oldexchange ! ingentra ! com
[Download RAW message or body]
Thanks for all the suggestions, gents.
I was able to narrow it down to one particular Oracle database. There was a
long running process on the db from a external jdbc connection from an
Oracle Application Server. I was able to recycle the app server and the io
activity went away.
I'm not sure if any of the sugggested tools would have given me that level
of detail. I'll have to wait to see if it reoccurs and try again.
Bob
-----Original Message-----
From: Mills, John T [mailto:John.Mills@GBE.COM]
Sent: Thursday, April 03, 2008 2:44 PM
To: aix-l@Princeton.EDU
Subject: Re: What process is causing high disk io?
One of my favorites is lsof as well.
lsof /dev/hd2
But filemon is the second step.
John T. Mills
_____
From: IBM AIX Discussion List [mailto:aix-l@Princeton.EDU] On Behalf Of
Davignon, Edward
Sent: Thursday, April 03, 2008 12:54 PM
To: aix-l@Princeton.EDU
Subject: Re: What process is causing high disk io?
Bob,
To find out what process or what file is using the disk, the turn on the
file trace facility with the filemon command.
filemon -u -o fmon.out; sleep 10; trcstop
view fmon.out
(make sure you turn this off with "trcstop" when you are done.)
----------------
Another option is to monitor it at the LVM level like this:
lvmstat -v whatevervg -e
lvmstat -v whatevervg
lvmstat -v whatevervg -d
(make sure you turn this off with the "lvmstat -v whatevervg -d" when you
are done.)
----------
-Ed
_____
From: IBM AIX Discussion List [mailto:aix-l@Princeton.EDU] On Behalf Of
Weinmann, Robert
Sent: Thursday, April 03, 2008 1:01 PM
To: aix-l@Princeton.EDU
Subject: What process is causing high disk io?
Hello all,
I have a raid disk that is under high read activity. Disk utilization is
90-100%. It is a raid group that contains several Oracle databases. How
can I identify the process that is causing the high usage?
I don't see any runaway processes, and cpu usage is in single digits.
Thanks,
Bob
[Attachment #3 (text/html)]
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML xmlns="http://www.w3.org/TR/REC-html40" xmlns:v =
"urn:schemas-microsoft-com:vml" xmlns:o =
"urn:schemas-microsoft-com:office:office" xmlns:w =
"urn:schemas-microsoft-com:office:word"><HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<TITLE>What process is causing high disk io?</TITLE>
<META content="MSHTML 6.00.2800.1528" name=GENERATOR><!--[if !mso]>
<STYLE>v\:* {
BEHAVIOR: url(#default#VML)
}
o\:* {
BEHAVIOR: url(#default#VML)
}
w\:* {
BEHAVIOR: url(#default#VML)
}
.shape {
BEHAVIOR: url(#default#VML)
}
</STYLE>
<![endif]-->
<STYLE>@font-face {
font-family: Tahoma;
}
@page Section1 {size: 8.5in 11.0in; margin: 1.0in 1.25in 1.0in 1.25in; }
P.MsoNormal {
FONT-SIZE: 12pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Times New Roman"
}
LI.MsoNormal {
FONT-SIZE: 12pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Times New Roman"
}
DIV.MsoNormal {
FONT-SIZE: 12pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Times New Roman"
}
A:link {
COLOR: blue; TEXT-DECORATION: underline
}
SPAN.MsoHyperlink {
COLOR: blue; TEXT-DECORATION: underline
}
A:visited {
COLOR: purple; TEXT-DECORATION: underline
}
SPAN.MsoHyperlinkFollowed {
COLOR: purple; TEXT-DECORATION: underline
}
P {
FONT-SIZE: 12pt; MARGIN-LEFT: 0in; MARGIN-RIGHT: 0in; FONT-FAMILY: "Times New \
Roman"; mso-margin-top-alt: auto; mso-margin-bottom-alt: auto }
SPAN.EmailStyle18 {
COLOR: navy; FONT-FAMILY: Arial; mso-style-type: personal-reply
}
DIV.Section1 {
page: Section1
}
</STYLE>
</HEAD>
<BODY lang=EN-US vLink=purple link=blue>
<DIV><SPAN class=718163119-03042008><FONT face=Arial color=#0000ff size=2>Thanks
for all the suggestions, gents.</FONT></SPAN></DIV>
<DIV><SPAN class=718163119-03042008><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=718163119-03042008><FONT face=Arial color=#0000ff size=2>I was
able to narrow it down to one particular Oracle database. There was a long
running process on the db from a external jdbc connection from an Oracle
Application Server. I was able to recycle the app server and the io
activity went away.</FONT></SPAN></DIV>
<DIV><SPAN class=718163119-03042008><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=718163119-03042008><FONT face=Arial color=#0000ff size=2>I'm
not sure if any of the sugggested tools would have given me that level of
detail. I'll have to wait to see if it reoccurs and try
again.</FONT></SPAN></DIV>
<DIV><SPAN class=718163119-03042008><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=718163119-03042008><FONT face=Arial color=#0000ff
size=2>Bob</FONT></SPAN></DIV>
<BLOCKQUOTE dir=ltr style="MARGIN-RIGHT: 0px">
<DIV class=OutlookMessageHeader dir=ltr align=left><FONT face=Tahoma
size=2>-----Original Message-----<BR><B>From:</B> Mills, John T
[mailto:John.Mills@GBE.COM]<BR><B>Sent:</B> Thursday, April 03, 2008 2:44
PM<BR><B>To:</B> aix-l@Princeton.EDU<BR><B>Subject:</B> Re: What process is
causing high disk io?<BR><BR></FONT></DIV>
<DIV dir=ltr align=left><SPAN class=929464118-03042008><FONT
face="Courier New" color=#0000ff size=2>One of my favorites is lsof as
well.</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=929464118-03042008><FONT
face="Courier New" color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=929464118-03042008><FONT
face="Courier New" color=#0000ff size=2>lsof /dev/hd2</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=929464118-03042008><FONT
face="Courier New" color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=929464118-03042008><FONT
face="Courier New" color=#0000ff size=2>But filemon is the second
step.</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=929464118-03042008><FONT
face="Courier New" color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=929464118-03042008><FONT
face="Courier New" color=#0000ff size=2>John T. Mills</FONT></SPAN></DIV><BR>
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left>
<HR tabIndex=-1>
<FONT face=Tahoma size=2><B>From:</B> IBM AIX Discussion List
[mailto:aix-l@Princeton.EDU] <B>On Behalf Of </B>Davignon,
Edward<BR><B>Sent:</B> Thursday, April 03, 2008 12:54 PM<BR><B>To:</B>
aix-l@Princeton.EDU<BR><B>Subject:</B> Re: What process is causing high disk
io?<BR></FONT><BR></DIV>
<DIV></DIV>
<DIV class=Section1>
<P class=MsoNormal><FONT face=Arial color=navy size=2><SPAN
style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: \
Arial">Bob,<o:p></o:p></SPAN></FONT></P> <P class=MsoNormal><FONT face=Arial \
color=navy size=2><SPAN style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial">To \
find out what process or what file is using the disk, the turn on the file trace \
facility with the filemon command.<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial color=navy size=2><SPAN
style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: \
Arial"><o:p> </o:p></SPAN></FONT></P> <P class=MsoNormal><FONT face=Arial \
color=navy size=2><SPAN style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: \
Arial">filemon -u -o fmon.out; sleep 10; trcstop<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial color=navy size=2><SPAN
style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: \
Arial"><o:p> </o:p></SPAN></FONT></P> <P class=MsoNormal><FONT face=Arial \
color=navy size=2><SPAN style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: \
Arial">view fmon.out<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial color=navy size=2><SPAN
style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: \
Arial"><o:p> </o:p></SPAN></FONT></P> <P class=MsoNormal><FONT face=Arial \
color=navy size=2><SPAN style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: \
Arial">(make sure you turn this off with "trcstop" when you are \
done.)<o:p></o:p></SPAN></FONT></P> <P class=MsoNormal><FONT face=Arial color=navy \
size=2><SPAN style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: \
Arial"><o:p> </o:p></SPAN></FONT></P> <P class=MsoNormal><FONT face=Arial \
color=navy size=2><SPAN style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: \
Arial">----------------<o:p></o:p></SPAN></FONT></P> <P class=MsoNormal><FONT \
face=Arial color=navy size=2><SPAN style="FONT-SIZE: 10pt; COLOR: navy; \
FONT-FAMILY: Arial"><o:p> </o:p></SPAN></FONT></P> <P class=MsoNormal><FONT \
face=Arial color=navy size=2><SPAN style="FONT-SIZE: 10pt; COLOR: navy; \
FONT-FAMILY: Arial">Another option is to monitor it at the LVM level like \
this:<o:p></o:p></SPAN></FONT></P> <P class=MsoNormal><FONT face=Arial color=navy \
size=2><SPAN style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: \
Arial"><o:p> </o:p></SPAN></FONT></P> <P class=MsoNormal><FONT face=Arial \
color=navy size=2><SPAN style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: \
Arial">lvmstat -v whatevervg
-e<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial color=navy size=2><SPAN
style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial">lvmstat -v
whatevervg<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial color=navy size=2><SPAN
style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial">lvmstat -v whatevervg
-d<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial color=navy size=2><SPAN
style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: \
Arial"><o:p> </o:p></SPAN></FONT></P> <P class=MsoNormal><FONT face=Arial \
color=navy size=2><SPAN style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: \
Arial">(make sure you turn this off with the "lvmstat -v whatevervg -d" when you \
are done.)<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial color=navy size=2><SPAN
style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: \
Arial"><o:p> </o:p></SPAN></FONT></P> <P class=MsoNormal><FONT face=Arial \
color=navy size=2><SPAN style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: \
Arial">----------<o:p></o:p></SPAN></FONT></P> <P class=MsoNormal><FONT face=Arial \
color=navy size=2><SPAN style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: \
Arial"><o:p> </o:p></SPAN></FONT></P> <P class=MsoNormal><FONT face=Arial \
color=navy size=2><SPAN style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: \
Arial">-Ed<o:p></o:p></SPAN></FONT></P> <P class=MsoNormal><FONT face=Arial \
color=navy size=2><SPAN style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: \
Arial"><o:p> </o:p></SPAN></FONT></P> <P class=MsoNormal><FONT face=Arial \
color=navy size=2><SPAN style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: \
Arial"><o:p> </o:p></SPAN></FONT></P> <DIV>
<DIV class=MsoNormal style="TEXT-ALIGN: center" align=center><FONT
face="Times New Roman" size=3><SPAN style="FONT-SIZE: 12pt">
<HR tabIndex=-1 align=center width="100%" SIZE=2>
</SPAN></FONT></DIV>
<P class=MsoNormal><B><FONT face=Tahoma size=2><SPAN
style="FONT-WEIGHT: bold; FONT-SIZE: 10pt; FONT-FAMILY: \
Tahoma">From:</SPAN></FONT></B><FONT face=Tahoma size=2><SPAN style="FONT-SIZE: \
10pt; FONT-FAMILY: Tahoma"> IBM AIX Discussion List [mailto:aix-l@Princeton.EDU] \
<B><SPAN style="FONT-WEIGHT: bold">On Behalf Of </SPAN></B>Weinmann, \
Robert<BR><B><SPAN style="FONT-WEIGHT: bold">Sent:</SPAN></B> Thursday, April 03, \
2008 1:01 PM<BR><B><SPAN style="FONT-WEIGHT: bold">To:</SPAN></B>
aix-l@Princeton.EDU<BR><B><SPAN style="FONT-WEIGHT: bold">Subject:</SPAN></B>
What process is causing high disk io?</SPAN></FONT><o:p></o:p></P></DIV>
<P class=MsoNormal><FONT face="Times New Roman" size=3><SPAN
style="FONT-SIZE: 12pt"><o:p> </o:p></SPAN></FONT></P>
<P><FONT face=Arial size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Hello all,</SPAN></FONT>
<o:p></o:p></P>
<P><FONT face=Arial size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">I
have a raid disk that is under high read activity. Disk utilization is
90-100%. It is a raid group that contains several Oracle
databases. How can I identify the process that is causing the high
usage?</SPAN></FONT><o:p></o:p></P>
<P><FONT face=Arial size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">I
don't see any runaway processes, and cpu usage is in single
digits.</SPAN></FONT> <o:p></o:p></P>
<P><FONT face=Arial size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Thanks,</SPAN></FONT>
<o:p></o:p></P>
<P><FONT face=Arial size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Bob</SPAN></FONT>
<o:p></o:p></P></DIV></BLOCKQUOTE></BODY></HTML>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic