[prev in list] [next in list] [prev in thread] [next in thread] 

List:       postgresql-admin
Subject:    [ADMIN] Database corruption
From:       David Haube <dch () frolix ! muddywaters ! org>
Date:       2010-06-24 18:28:31
Message-ID: 4C23A3CF.4080506 () frolix ! muddywaters ! org
[Download RAW message or body]

Postgres version 8.4.4

Hardware:
12 cpu intel
sda  15K 600GB raid 1
sdb   ssd  173GB raid 1 (most tables and indexes on sdb)
adaptec controller
24G of memory
shared buffers 12GB
Machine age 3 months
Normal load 2-4
200-300 Transactions per second

We had a database failure last night after one of the tables had a 
corrupted block.  After we noticed the corruption all available memory  
was used up plus swap.  The database died (or rather killed by the 
kernel) with an out of memory error.  We switched to the warm standby 
which doesn't have any corruption.

On the postmortem I found 4 tables with corruption.  Only thing that 
links these tables was there was autovacuum (to prevent wraparound) 
either running or had run on those tables.  All tables are in the Gig 
range or multi Gig range.  The vacuum of some of the tables had been 
going on for days.

The errors from the log file were in the form of :
ERROR:  invalid page header in block 290125 of relation 
pg_tblspc/16385/18674/205612

After an attempted vaccum we had this error:
2010-06-24 17:31:09 UTC [31766]: [36-1]WARNING:  PD_ALL_VISIBLE flag was 
incorrectly set in relation "org_crawl_page_scrape_result" page 128902


The first error was logged at 10:15pm
At 1 am a pg_dump was run from cron and failed after 20 minutes while 
try ing to allocate an immenense amount of memory while attempting to 
dump one of the corrupted tables.

At 2:00 am All memory was used up and cpu was maxed and a load average 
of 56.

We transferred to the standby and rebooted the machine.  At this time 
the database is sitting there though we'll need to remake the database 
and turn it into the warm standby.




-- 
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin
[prev in list] [next in list] [prev in thread] [next in thread]