[prev in list] [next in list] [prev in thread] [next in thread] 

List:       postgresql-general
Subject:    Re: [GENERAL] Server hitting 100% CPU usage, system comes to a crawl.
From:       David Kerr <dmk () mr-paradox ! net>
Date:       2011-10-31 23:54:49
Message-ID: 20111031235449.GA57676 () mr-paradox ! net
[Download RAW message or body]

On Thu, Oct 27, 2011 at 02:09:51PM -0600, Brian Fehrle wrote:
- On 10/27/2011 01:48 PM, Scott Marlowe wrote:
- >On Thu, Oct 27, 2011 at 12:39 PM, Brian Fehrle
- ><brianf@consistentstate.com>  wrote:
- >>Looking at top, I see no SWAP usage, very little IOWait, and there are a
- >>large number of postmaster processes at 100% cpu usage (makes sense, at 
- >>this
- >>point there are 150 or so queries currently executing on the database).
- >>
- >>  Tasks: 713 total,  44 running, 668 sleeping,   0 stopped,   1 zombie
- >>Cpu(s):  4.4%us, 92.0%sy,  0.0%ni,  3.0%id,  0.0%wa,  0.0%hi,  0.3%si,
- >>  0.2%st
- >>Mem:  134217728k total, 131229972k used,  2987756k free,   462444k buffers
- >>Swap:  8388600k total,      296k used,  8388304k free, 119029580k cached
- >OK, a few points.  1: You've got a zombie process.  Find out what's
- >causing that, it could be a trigger of some type for this behaviour.
- >2: You're 92% sys.  That's bad.  It means the OS is chewing up 92% of
- >your 32 cores doing something.  what tasks are at the top of the list
- >in top?
- >
- Out of the top 50 processes in top, 48 of them are postmasters, one is 
- syslog, and one is psql. Each of the postmasters have a high %CPU, the 
- top ones being 80% and higher, the rest being anywhere between 30% - 
- 60%. Would postmaster 'queries' that are running attribute to the sys 
- CPU usage, or should they be under the 'us' CPU usage?

total spitball here but - I had something similar happen once and it 
was syslog causing the problem.

Are you using regular vanilla syslog? or syslog-ng/rsyslog? my problem
was vanilla syslog. When I moved to -ng/rsyslog or logging to a file 
my problem went away.

Dave

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general
[prev in list] [next in list] [prev in thread] [next in thread]