[prev in list] [next in list] [prev in thread] [next in thread]
List: postgresql-admin
Subject: Re: [ADMIN] Postgres will not allow new connections, suspended process, waiting error
From: Magnus Hagander <magnus () hagander ! net>
Date: 2017-07-02 16:07:36
Message-ID: CABUevEx_3h4T+9AuzgCHm+78qudwZjYDdw2kLpJiFjWQv9EMqg () mail ! gmail ! com
[Download RAW message or body]
On Sat, Jul 1, 2017 at 12:59 AM, Prateek Mahajan <prateekm99@gmail.com>
wrote:
> More details.
>
> Environment
> PostgreSQL 9.5, EnterpriseDB Postgres installer
> Windows Server 2012R2 with Active Directory
> Symantec End Point Protection
>
> Symptom:
>
> After about 1 week of running, one of PostgreSQL process (postgres.exe)
> showed "suspended" in task manager, and I cannot kill it in the task
> manager ("Access Denied" error message appeared). This "suspended" process
> was not the master PID as indicated in postmaster.pid file.
> Current live connections still work but one cannot establish new
> connections. The only solution that I have is to restart the Server
> Other information:
>
> The PostgreSQL service is run under a domain account.
> The maximum connection was never reached as it is set as 1000 and we only
> had about 10 connections.
> There was plenty of available memory there. The total memory is 288GB and
> only 8% was used
> There were minimum hard drive activities as it occurred. The C drive where
> PostgreSQL was installed had about 86GB of free space.
> There are additional 4 table spaces that are not on C drive but spread
> over 4 hard drives. Each of 4 hard drives has more than 500GB of space.
> we have been using the same configuration files for years and the same
> file is also used on a second PostgreSQL server, which does not have the
> issue at all.
> The PostgreSQL logs had something like this when this happened and it
> continues to produce this warning message every minute or so:
>
> 2017-06-28 19:40:21 CDT WARNING: worker took too long to start; canceled
> 2017-06-28 19:41:21 CDT WARNING: worker took too long to start; canceled
> 2017-06-28 19:42:21 CDT WARNING: worker took too long to start; canceled
> 2017-06-28 19:43:21 CDT WARNING: worker took too long to start; canceled
>
>
Those are autovacuum workers trying to start. My guess is that's a symptom
of the same basic problem, which is that your machine behaves as if it's
heavily overloaded.
As a first try I'd attempt removing the Symantec Endpoint stuff and see if
that helps. It's very common that software like that breaks the database.
And being unable to kill things in the task manager clearly indicates the
problem lies outside the control of Postgres.
--
Magnus Hagander
Me: https://www.hagander.net/ <http://www.hagander.net/>
Work: https://www.redpill-linpro.com/ <http://www.redpill-linpro.com/>
[Attachment #3 (text/html)]
<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Jul \
1, 2017 at 12:59 AM, Prateek Mahajan <span dir="ltr"><<a \
href="mailto:prateekm99@gmail.com" \
target="_blank">prateekm99@gmail.com</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr"><div>More \
details.</div><div><br></div><div>Environment</div><div>PostgreSQL 9.5, EnterpriseDB \
Postgres installer</div><div>Windows Server 2012R2 with Active \
Directory</div><div>Symantec End Point \
Protection</div><div><br></div><div>Symptom:</div><div><br></div><div>After about 1 \
week of running, one of PostgreSQL process (postgres.exe) showed \
"suspended" in task manager, and I cannot kill it in the task manager \
("Access Denied" error message appeared). This "suspended" \
process was not the master PID as indicated in postmaster.pid file. \
</div><div>Current live connections still work but one cannot establish new \
connections. The only solution that I have is to restart the Server </div><div>Other \
information:</div><div><br></div><div>The PostgreSQL service is run under a domain \
account.</div><div>The maximum connection was never reached as it is set as 1000 and \
we only had about 10 connections.</div><div>There was plenty of available memory \
there. The total memory is 288GB and only 8% was used</div><div>There were minimum \
hard drive activities as it occurred. The C drive where PostgreSQL was installed had \
about 86GB of free space.</div><div>There are additional 4 table spaces that are not \
on C drive but spread over 4 hard drives. Each of 4 hard drives has more than 500GB \
of space.</div><div>we have been using the same configuration files for years and the \
same file is also used on a second PostgreSQL server, which does not have the issue \
at all.</div><div>The PostgreSQL logs had something like this when this happened and \
it continues to produce this warning message every minute or \
so:</div><div><br></div><div>2017-06-28 19:40:21 CDT WARNING: worker took too long \
to start; canceled</div><div>2017-06-28 19:41:21 CDT WARNING: worker took too long \
to start; canceled</div><div>2017-06-28 19:42:21 CDT WARNING: worker took too long \
to start; canceled</div><div>2017-06-28 19:43:21 CDT WARNING: worker took too long \
to start; canceled</div><div><div class="h5"><div class="gmail_extra"><div><div \
class="m_-8665029563452853401gmail_signature" data-smartmail="gmail_signature"><div \
dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div \
dir="ltr"><div dir="ltr"><div dir="ltr"><div \
dir="ltr"><br></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></blockquote><div><br></div><div>Those \
are autovacuum workers trying to start. My guess is that's a symptom of the same \
basic problem, which is that your machine behaves as if it's heavily \
overloaded.</div><div><br></div><div>As a first try I'd attempt removing the \
Symantec Endpoint stuff and see if that helps. It's very common that software \
like that breaks the database. And being unable to kill things in the task manager \
clearly indicates the problem lies outside the control of Postgres. \
</div></div><div><br></div>-- <br><div class="gmail_signature" \
data-smartmail="gmail_signature"><div dir="ltr"><div> Magnus Hagander<br> Me: <a \
href="http://www.hagander.net/" target="_blank">https://www.hagander.net/</a><br> \
Work: <a href="http://www.redpill-linpro.com/" \
target="_blank">https://www.redpill-linpro.com/</a></div></div></div> </div></div>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic