phatqFrom our monitoring we received a disk full alert for our configuration backup server, which is a CentOS VM.
Nagios XI Alert
This is an escalated notification
Type: PROBLEM
Servicecheck:/ Disk Usage
Hostname: *CentOS_Server*
State: WARNING
service_output:
/: 95%used(25740MB/27091MB) (95%) : WARNING
service_long_output:
So we needed to increase the size of this volume. In our case the volume /dev/mapper/centos_centos7-root was on a physical disk with that has a total capacity of 60GB. A mere 26Gb was in use so there was more then 30GB of free space to be allocated:
The file system sits on top while the physical disk is the first tier if you will.
4. File system
3. Logical Volumes
2. Logical Volume group
1. Physical disk
[root@conf ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos_centos7-root 27G 26G 1.4G 96% /
devtmpfs 475M 0 475M 0% /dev
tmpfs 487M 0 487M 0% /dev/shm
tmpfs 487M 51M 437M 11% /run
tmpfs 487M 0 487M 0% /sys/fs/cgroup
/dev/mapper/VolGroup01-VCSA--Backup 250G 18G 233G 8% /mnt/VCSA-Backup
/dev/sda1 497M 229M 268M 47% /boot
tmpfs 98M 0 98M 0% /run/user/0
[root@conf ~]# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
VCSA-Backup VolGroup01 -wi-ao---- <250.00g
root centos_centos7 -wi-ao---- <26.47g
swap centos_centos7 -wi-ao---- 3.00g
[root@confbackup ~]# fdisk -l
Disk /dev/sda: 64.4 GB, 64424509440 bytes, 125829120 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000a5cdd
Device Boot Start End Blocks Id System
/dev/sda1 * 2048 1026047 512000 83 Linux
/dev/sda2 1026048 62914559 30944256 8e Linux LVM
Disk /dev/sdb: 268.4 GB, 268435456000 bytes, 524288000 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x00000000
Device Boot Start End Blocks Id System
/dev/sdb1 63 524287999 262143968+ 8e Linux LVM
Disk /dev/mapper/centos_centos7-swap: 3221 MB, 3221225472 bytes, 6291456 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/centos_centos7-root: 28.4 GB, 28420603904 bytes, 55508992 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/VolGroup01-VCSA--Backup: 268.4 GB, 268431261696 bytes, 524279808 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
volume group display (vgdisplay) to see the size of the volume groups
[root@confbackup ~]# vgdisplay
--- Volume group ---
VG Name VolGroup01
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 2
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 1
Open LV 1
Max PV 0
Cur PV 1
Act PV 1
VG Size <250.00 GiB
PE Size 4.00 MiB
Total PE 63999
Alloc PE / Size 63999 / <250.00 GiB
Free PE / Size 0 / 0
--- Volume group ---
VG Name centos_centos7
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 3
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 2
Open LV 2
Max PV 0
Cur PV 1
Act PV 1
VG Size <29.51 GiB
PE Size 4.00 MiB
Total PE 7554
Alloc PE / Size 7544 / <29.47 GiB
Free PE / Size 10 / 40.00 MiB
Do a lvdisplay to list the size of the logical volumes, important is the root partition. You can see that the size is a mere 26GB (other lv’s on the list is filtered out)
[root@confbackup ~]# lvdisplay
--- Logical volume ---
LV Path /dev/centos_centos7/root
LV Name root
VG Name centos_centos7
LV Write Access read/write
LV Creation host, time localhost.localdomain, 2015-04-08 18:19:04 +0200
LV Status available
# open 1
LV Size <26.47 GiB
Current LE 6776
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 8192
Block device 253:1
Do a pvdisplay to list the size of the physical volume, in our case /dev/sda2 needs te be extended
[root@confbackup ~]# pvdisplay
--- Physical volume ---
PV Name /dev/sdb1
VG Name VolGroup01
PV Size <250.00 GiB / not usable <3.97 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 63999
Free PE 0
Allocated PE 63999
--- Physical volume ---
PV Name /dev/sda2
VG Name centos_centos7
PV Size 29.51 GiB / not usable 3.00 MiB
Allocatable yes
PE Size 4.00 MiB
Total PE 7554
Free PE 10
Allocated PE 7544
Extend the partition
[root@confbackup ~]# fdisk /dev/sda
Welcome to fdisk (util-linux 2.23.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Command (m for help): m
Command action
a toggle a bootable flag
b edit bsd disklabel
c toggle the dos compatibility flag
d delete a partition
g create a new empty GPT partition table
G create an IRIX (SGI) partition table
l list known partition types
m print this menu
n add a new partition
o create a new empty DOS partition table
p print the partition table
q quit without saving changes
s create a new empty Sun disklabel
t change a partition's system id
u change display/entry units
v verify the partition table
w write table to disk and exit
x extra functionality (experts only)
Command (m for help): p
Disk /dev/sda: 64.4 GB, 64424509440 bytes, 125829120 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000a5cdd
Device Boot Start End Blocks Id System
/dev/sda1 * 2048 1026047 512000 83 Linux
/dev/sda2 1026048 62914559 30944256 8e Linux LVM
Command (m for help): n
Partition type:
p primary (2 primary, 0 extended, 2 free)
e extended
Select (default p): p
Partition number (3,4, default 3):
First sector (62914560-125829119, default 62914560):
Using default value 62914560
Last sector, +sectors or +size{K,M,G} (62914560-125829119, default 125829119):
Using default value 125829119
Partition 3 of type Linux and of size 30 GiB is set
Command (m for help): p
Disk /dev/sda: 64.4 GB, 64424509440 bytes, 125829120 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000a5cdd
Device Boot Start End Blocks Id System
/dev/sda1 * 2048 1026047 512000 83 Linux
/dev/sda2 1026048 62914559 30944256 8e Linux LVM
/dev/sda3 62914560 125829119 31457280 83 Linux
Command (m for help): t
Partition number (1-3, default 3):
Hex code (type L to list all codes): 8e
Changed type of partition 'Linux' to 'Linux LVM'
Command (m for help): p
Disk /dev/sda: 64.4 GB, 64424509440 bytes, 125829120 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000a5cdd
Device Boot Start End Blocks Id System
/dev/sda1 * 2048 1026047 512000 83 Linux
/dev/sda2 1026048 62914559 30944256 8e Linux LVM
/dev/sda3 62914560 125829119 31457280 8e Linux LVM
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.
[root@confbackup ~]# partprobe
[root@confbackup ~]# ls -l
total 0
[root@confbackup ~]# ls -l /dev/sd*
brw-rw----. 1 root disk 8, 0 Oct 10 12:24 /dev/sda
brw-rw----. 1 root disk 8, 1 Oct 10 12:24 /dev/sda1
brw-rw----. 1 root disk 8, 2 Oct 10 12:24 /dev/sda2
brw-rw----. 1 root disk 8, 3 Oct 10 12:24 /dev/sda3
brw-rw----. 1 root disk 8, 16 Oct 10 12:24 /dev/sdb
brw-rw----. 1 root disk 8, 17 Oct 10 12:24 /dev/sdb1
[root@confbackup ~]# pvcreate /dev/sda3
Physical volume "/dev/sda3" successfully created.
[root@confbackup ~]# vgdisplay
--- Volume group ---
VG Name VolGroup01
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 2
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 1
Open LV 1
Max PV 0
Cur PV 1
Act PV 1
VG Size <250.00 GiB
PE Size 4.00 MiB
Total PE 63999
Alloc PE / Size 63999 / <250.00 GiB
Free PE / Size 0 / 0
VG UUID IN49ef-10HQ-5i4K-nvzA-zpzW-xn5U-ohesf0
--- Volume group ---
VG Name centos_centos7
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 3
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 2
Open LV 2
Max PV 0
Cur PV 1
Act PV 1
VG Size <29.51 GiB
PE Size 4.00 MiB
Total PE 7554
Alloc PE / Size 7544 / <29.47 GiB
Free PE / Size 10 / 40.00 MiB
VG UUID Mlwe8D-BeAr-wV8v-e332-vXpX-LmdD-DBsvWo
[root@confbackup ~]# vgextend centos_centos7 /dev/sda3
Volume group "centos_centos7" successfully extended
[root@confbackup ~]# vgdisplay
--- Volume group ---
VG Name VolGroup01
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 2
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 1
Open LV 1
Max PV 0
Cur PV 1
Act PV 1
VG Size <250.00 GiB
PE Size 4.00 MiB
Total PE 63999
Alloc PE / Size 63999 / <250.00 GiB
Free PE / Size 0 / 0
VG UUID IN49ef-10HQ-5i4K-nvzA-zpzW-xn5U-ohesf0
--- Volume group ---
VG Name centos_centos7
System ID
Format lvm2
Metadata Areas 2
Metadata Sequence No 5
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 2
Open LV 2
Max PV 0
Cur PV 2
Act PV 2
VG Size 59.50 GiB
PE Size 4.00 MiB
Total PE 15233
Alloc PE / Size 7544 / <29.47 GiB
Free PE / Size 7689 / <30.04 GiB
VG UUID Mlwe8D-BeAr-wV8v-e332-vXpX-LmdD-DBsvWo
Now extend the volume with 10GB
[root@confbackup ~]# lvextend /dev/centos_centos7/root -L +10G
Size of logical volume centos_centos7/root changed from <26.47 GiB (6776
extents) to <36.47 GiB (9336 extents).
Logical volume centos_centos7/root successfully resized.
Resize the file system
[root@confbackup ~]# resize2fs /dev/centos_centos7/root
resize2fs 1.42.9 (28-Dec-2013)
resize2fs: Bad magic number in super-block while trying to open /dev/centos_centos7/root
Couldn't find valid filesystem superblock.
[root@confbackup ~]# mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
[root@confbackup ~]# xfs_growfs /
meta-data=/dev/mapper/centos_centos7-root isize=256 agcount=4, agsize=1734656 blks
= sectsz=512 attr=2, projid32bit=1
= crc=0 finobt=0 spinodes=0
data = bsize=4096 blocks=6938624, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=0
log =internal bsize=4096 blocks=3388, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
data blocks changed from 6938624 to 9560064
/dev/mapper/centos_centos7-root is increased with 10GB to 37GB
[root@confbackup ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos_centos7-root 37G 26G 12G 69% /
devtmpfs 475M 0 475M 0% /dev
tmpfs 487M 0 487M 0% /dev/shm
tmpfs 487M 51M 437M 11% /run
tmpfs 487M 0 487M 0% /sys/fs/cgroup
/dev/mapper/VolGroup01-VCSA--Backup 250G 18G 233G 8% /mnt/VCSA-Backup
/dev/sda1 497M 229M 268M 47% /boot
tmpfs 98M 0 98M 0% /run/user/0