[prev in list] [next in list] [prev in thread] [next in thread] 

List:       illumos-discuss
Subject:    Re: [discuss] netstat -a hanging on domain sockets
From:       "Jim Klimov" <jimklimov () cos ! ru>
Date:       2016-04-22 13:09:58
Message-ID: 70246F2F-F852-4356-BAF3-452A6F9BCBBC () cos ! ru
[Download RAW message or body]

21 апреля 2016  г. 20:24:37 CEST, Gabriele Bulfon <gbulfon@sonicle.com> \
пишет:
> I see...but I feel cpu was normal, memory was ok, disk space too...what
> may be affecting unix domain sockets and hangs?
> BTW, this is a sample output from the same zone, now that is responding
> fine.
> Would you please give me some insights to what are the interesting
> numbers in these cases?
> sonicle@cloudserver:~$ prstat -mL   PID USERNAME USR SYS TRP TFL DFL
> LCK SLP LAT VCX ICX SCL SIG PROCESS/LWPID 26840 sonicle  5.3 0.0 0.0
> 0.0 0.0 0.0  95 0.0   6   3 111   0 imapd/1 23455 sonicle  3.1 0.2 0.0
> 0.0 0.0 0.0  97 0.0  82   3 843   0 imapd/1  5952 sonicle  0.2 0.5 0.0
> 0.0 0.0  99 0.1 0.0  1K   1 13K   0 slapd/7sonicle@cloudserver:~$
> vmstat kthr      memory            page            disk          faults
> cpu r b w   swap  free  re  mf pi po fr de sr s0 s1 s2 s3   in   sy  
> cs us sy id 0 0 0 20345404 1559380 416 1750 0 0 0 0 0 -0 18  4 99 5798
> 49801 17818 3 2 95sonicle@cloudserver:~$ mpstatCPU minf mjf xcal  intr
> ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl  0  190   0   32  
> 592  124 1826   11  101   81    0  2519    2   3   0  95  1  113   0  
> 48   318   32 1411    3   69   71    0  4228    4   2   0  94  2   96  
> 0   44   537   34 1157    3   61   67    0  4213    4   3   0  93  3  
> 89   0   39   338   78 1220    3   68   75    0  3165    3   2   0  95 
> 4  272   0   17   269   31 1174    5   48   64    0  2047    1   2   0 
> 97  5  115   0   43   327   46 1271    3   57   69    0  3488    4   2 
> 0  94  6   79   0   30   260   33  881    2   46   58    0  2622    3  
> 1   0  96  7   91   0   49   336   51 1267    3   56   68    0  2930   
> 4   2   0  95  8   77   0   25   257   39  719    3   42   48    0 
> 2280    2   1   0  96  9   87   0   40   271   37  878    2   50   56  
> 0  3109    4   1   0  95 10   95   0   48   284   45  950    2   52  
> 58    0  3392    4   2   0  95 11   81   0   27   233   33  696    2  
> 39   47    0  2836    3   1   0  96 12   83   0   47   327   46 1237   
> 3   56   67    0  3383    4   2   0  95 13   88   0   31   346  134 
> 899    2   46   57    0  3022    4   1   0  95 14  117   0   47   410 
> 152 1264    3   57   68    0  4247    5   2   0  93 15   76   0   27  
> 692  508  968    3   47   75    0  2319    2   2   0  96
> ----------------------------------------------------------------------------------------
>  Sonicle S.r.l.
> > 
> http://www.sonicle.com
> Music:
> http://www.gabrielebulfon.com
> Quantum Mechanics :
> http://www.cdbaby.com/cd/gabrielebulfon
> ----------------------------------------------------------------------------------
> Da:   Robert Mustacchi
> discuss@lists.illumos.org
> Data: 21 aprile 2016 20.09.28 CEST
> Oggetto: Re: [discuss] netstat -a hanging on domain sockets
> On 4/21/16 11:07 , Gabriele Bulfon wrote:
> Hi,
> I had a strange situation on a server starting to respond very slow on
> almost any deamon (tomcat/apache/ssh/cyrus).
> While prstat/top did not show any high cpu usage (under 5%) and prstat
> -Z didn't show any very high memory usage, and
> swap look there unused...I tried running netstat -a to see how many
> sockets where there serving.
> The result was a very fast output of all the normal sockets, and then
> hanging for 10-30 seconds before the output of active UNIX domain
> sockets.
> I repeated the command some times, but didn't think about saving the
> output somewhere, so I don't know what the domain sockets state was
> (which files, where).
> I can say the machine runs amavis / spamd / lmtpd /saslauthd / cyrus /
> postfix / apache / tomcat / postgres  for more than 600 users.
> Is there anything I should look deeper about the domain socket usage?
> Is it possible that they were so much busy slowing down
> every socket (even ssh took 1 to 2 minutes to take me to the bash).
> In general, you should use the -n option to netstat whenever you're
> debugging so you're not generating a lot of dns resolutions. But when
> you're looking at that kind of thing, you should start with your CPU
> microstates for processes (usually the -mL options to prstat) and tools
> like mpstat and vmstat.
> Robert
> 

In vmstat looking for lag reasons, for me often there is correlation with pi/po \
(swapping page-in/out activity), sr (scanrate - how often kernel looks for pages to \
evict) and cs (context switching - good/bad values depend on amount and performance \
of cores, but bad impact reflects that the system is overloaded with active \
processes).

Small free RAM (under 128-64-32mb depending on don't know what) is also a problem - \
processes can fail to fork, etc.

But I more often deal with small old servers, where 2-4gb ram is norm, 16gb is \
supersized ;)

Jim
--
Typos courtesy of K-9 Mail on my Samsung Android


-------------------------------------------
illumos-discuss
Archives: https://www.listbox.com/member/archive/182180/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182180/25758063-6f7f4185
Modify Your Subscription: \
https://www.listbox.com/member/?member_id=25758063&id_secret=25758063-83fb4fd4 \
Powered by Listbox: http://www.listbox.com


[prev in list] [next in list] [prev in thread] [next in thread]