vCenter Server Appliance - Full Database Partition

May 01, 2015

Issues with your vCSA appliance locking up or crashing due to a full database volume? Before you go too crazy looking at the database itself, or adding more space to the volume, see this bug.

After upgrading to vCenter Server Appliance 5.5 Update 2, pg_log file reports this error: WARNING: there is already a transaction in progress (2092127)

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2092127

SSH into vCenter

vcenter:/ # df -h
Filesystem      Size Used Avail Use% Mounted on
/dev/sda3       9.8G 3.7G 5.6G 41% /
udev            4.0G 108K 4.0G   1% /dev
tmpfs           4.0G     0 4.0G   0% /dev/shm
/dev/sda1       128M   21M 101M 18% /boot
/dev/sdb1        20G 8.3G   11G 45% /storage/core
/dev/sdb2        20G 3.6G   16G 19% /storage/log
/dev/sdb3       256G 243G   51M 100% /storage/db

vcenter:~ # cd /storage/db/vpostgres/

vcentereast:/storage/db/vpostgres # ls -lh
total 49G
-rw------- 1 postgres users    4 Feb 20 2014 PG_VERSION
drwx------ 7 postgres users 4.0K May 18 2014 base
drwx------ 2 postgres users 4.0K Apr 30 23:28 global
drwx------ 2 postgres users 4.0K Apr 30 20:06 pg_clog
-rw------- 1 postgres users 3.7K Feb 4 18:40 pg_hba.conf
-rw------- 1 postgres users 1.6K Feb 20 2014 pg_ident.conf
drwx------ 2 postgres users 1.1M Apr 4 12:30 pg_log
drwx------ 4 postgres users 4.0K Feb 20 2014 pg_multixact
drwx------ 2 postgres users 4.0K Apr 30 23:26 pg_notify
drwx------ 2 postgres users 4.0K Apr 30 23:30 pg_stat_tmp
drwx------ 2 postgres users 4.0K Apr 30 23:30 pg_subtrans
drwx------ 2 postgres users 4.0K Feb 20 2014 pg_tblspc
drwx------ 2 postgres users 4.0K Feb 20 2014 pg_twophase
drwx------ 3 postgres users 40K Apr 30 23:26 pg_xlog
-rw------- 1 postgres users 188 Apr 30 23:26 postgresql.conf
-rw------- 1 postgres users   68 Apr 30 23:26 postmaster.opts
-rw------- 1 postgres users   47 Apr 30 23:26 postmaster.pid
-rw------- 1 postgres users 49G Apr 30 23:25 serverlog

vcenter:/storage/db/vpostgres # rm serverlog

vcenter:/storage/db/vpostgres # sudo shutdown -r now

There's also a massive amount of log files in /storage/db/vpostgres/pg_log that also look to be diagnostic logs as opposed to anything useful.

vcenter:/storage/db/vpostgres # du -h
12K     ./pg_notify
49M     ./pg_clog
5.4M    ./base/11821
5.5M    ./base/11829
12G     ./base/16385
20K     ./base/pgsql_tmp
5.4M    ./base/1
12G     ./base
160K    ./pg_stat_tmp
696K    ./global
4.0K    ./pg_twophase
4.0K    ./pg_xlog/archive_status
273M    ./pg_xlog
200K    ./pg_multixact/members
232K    ./pg_multixact/offsets
436K    ./pg_multixact
4.0K    ./pg_tblspc
183G    ./pg_log
120K    ./pg_subtrans
194G    .

Original find courtesy of this blog post:

http://www.vcloudnine.de/vcenter-server-appliance-troubleshooting-full-database-partition/