Logical volumes are inactive at boot time

Question

I resized my logical volume and filesystem and all went smoothly. I installed new kernel and after reboot I can't boot neither current nor former one. I get volume group not found error after selecting grub(2) option. Inspection from busy box reveals the volumes are not registered with device mapper and that they are inactive. I wasn't able to mount them after activating, I got file not found error (mount /dev/mapper/all-root /mnt).

Any ideas how to proceed or make them active at the boot time? Or why the volumes are all of sudden inactive at boot time?

Regards,

Marek

EDIT: Further investigation revealed that this had nothing to do with the resizing of logical volumes. The fact that logical volumes had to be activated manually in ash shell after failed boot and possible solution to this problem is covered in my reply below.

Answer 1

So I managed to solve this eventually. There is a problem (bug) with detecting logical volumes, which is some sort of race condition (maybe in my case regarding the fact that this happens inside KVM). This is covered in the following discussion. In my particular case (Debian Squeeze ) the solution is as follows:

• backup the script /usr/share/initramfs-tools/scripts/local-top/lvm2
• apply the patch from mentioned bug report
• run update-initramfs -u

This helped me, hope it'll help others (strangely, this is not part of mainstream yet).

Link to patch: https://bugs.debian.org/cgi-bin/bugreport.cgi?msg=10;filename=lvm2_wait-lvm.patch;att=1;bug=568838

Below is a copy for posterity:

--- /usr/share/initramfs-tools/scripts/local-top/lvm2 2009-08-17 19:28:09.000000000 +0200
+++ /usr/share/initramfs-tools/scripts/local-top/lvm2 2010-02-19 23:22:14.000000000 +0100
@@ -45,12 +45,30 @@
eval $(dmsetup splitname --nameprefixes --noheadings --rows "$dev")

if [ "$DM_VG_NAME" ] && [ "$DM_LV_NAME" ]; then
lvm lvchange -aly --ignorelockingfailure "$DM_VG_NAME/$DM_LV_NAME"
rc=$?
if [ $rc = 5 ]; then
echo "Unable to find LVM volume $DM_VG_NAME/$DM_LV_NAME"


fi


Make sure that we have non-empty volume group and logical volume

if [ -z "$DM_VG_NAME" ] || [ -z "$DM_LV_NAME" ]; then
return 1
fi

If the logical volume hasn't shown up yet, give it a little while

to deal with LVM on removable devices (inspired from scripts/local)

fulldev="/dev/$DM_VG_NAME/$DM_LV_NAME"
if [ -z "lvm lvscan -a --ignorelockingfailure |grep $fulldev" ]; then
Use default root delay

slumber=$(( ${ROOTDELAY:-180} * 10 ))

while [ -z "lvm lvscan -a --ignorelockingfailure |grep $fulldev" ]; do
/bin/sleep 0.1


slumber=$(( ${slumber} - 1 ))


[ ${slumber} -gt 0 ] || break


done
fi

Activate logical volume

lvm lvchange -aly --ignorelockingfailure "$DM_VG_NAME/$DM_LV_NAME"
rc=$?
if [ $rc = 5 ]; then
echo "Unable to find LVM volume $DM_VG_NAME/$DM_LV_NAME"
fi

}

Answer 2

Create a startup script in /etc/init.d/lvm containing the following:

#!/bin/sh

case "$1" in
 start)
    /sbin/vgscan
    /sbin/vgchange -ay
    ;;
  stop)
    /sbin/vgchange -an
    ;;
  restart|force-reload)
    ;;
esac

exit 0

Then execute the commands:

chmod 0755 /etc/init.d/lvm
update-rc.d lvm start 26 S . stop 82 1 .

Should do the trick for Debian systems.

Answer 3

If vgscan "finds" the volumes, you should be able to activate them with vgchange -ay /dev/volumegroupname

$ sudo vgscan
[sudo] password for username: 
  Reading all physical volumes.  This may take a while...
  Found volume group "vg02" using metadata type lvm2
  Found volume group "vg00" using metadata type lvm2

$ sudo vgchange -ay /dev/vg02
  7 logical volume(s) in volume group "vg00" now active

I am not sure what would cause them to go inactive after a reboot though.

Answer 4

I had this problem too. In the end this is what seemed to fix it:

diff -u /usr/share/initramfs-tools/scripts/local-top/lvm2-backup /usr/share/initramfs-tools/scripts/local-top/lvm2
--- /usr/share/initramfs-tools/scripts/local-top/lvm2-backup    2014-06-06 19:55:19.249857946 -0400
+++ /usr/share/initramfs-tools/scripts/local-top/lvm2   2014-06-21 01:26:01.015289945 -0400
@@ -60,6 +60,7 @@

 modprobe -q dm-mod

+lvm vgchange -ay
 activate_vg "$ROOT"
 activate_vg "$resume"

Other things I tried:

1. your patch
2. diffing /etc/lvm/lvm.conf
3. GRUB_PRELOAD_MODULES="lvm"
4. GRUB_CMDLINE_LINUX="scsi_mod.scan=sync"
5. sudo grub-install /dev/sda && sudo grub-install /dev/sdb && sudo update-grub && sudo update-initramfs -u -k all
6. sudo apt-get install --reinstall lvm2 grub-pc grub-common

I went through and undid the other changes, this is the only one that mattered for me, though it's probably the least elegant.

Answer 5

we ran into this problem and found that disabling lvmetad by setting use_lvmetad=0 in /etc/lvm/lvm.conf forced the volumes to be found and mae accessible at boot.

Answer 6

Without any of the configuration details or error messages we'd need to give an actual answer, I'll take a stab in the dark with grub-mkdevicemap as a solution.

Answer 7

Assuming your system uses initramfs, there's probably a configuration problem there. You should update your initramfs image that's started at boot time by grub (in Debian you do this with update-initramfs, don't know about other distros).

You could also do this by hand by unpacking initramfs and changing /etc/lvm/lvm.conf (or something like it) in your initramfs image and then repacking it again.

Answer 8

I've got the same problem in my environment running Red Hat 7.4 as a KVM guest. I'm running qemu-kvm-1.5.3-141 and virt-manager 1.4.1. At first I was running Red Hat 7.2 as guest without any problem, but after upgrading minor release from 7.2 to 7.4 and kernel to latest version 3.10.0-693.5.2, something went wrong and I couldn't boot my /var LV partition any more. The system went to emergency mode asking for the root password. Entering with root password and running the commands lvm vgchange -ay and systemctl default I was able to activate my /var LV and boot the system.

I haven't figured out what causes this issue, but my workaround was to include the LV /var in /etc/default/grub as you see below:

GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=vg_local/root rd.lvm.lv=vg_local/var rd.lvm.lv=vg_local/swap rhgb quiet biosdevname=0 net.ifnames=0 ipv6.disable=1"

Then I had to run grub2-mkconfig -o /boot/grub2/grub.cfg and check if the rd.lvm.lv=vg_local/var was included in vmlinuz line of /boot/grub2/grub.cfg. After rebooting the system, I didn't get the error for activating my /var LV anymore and the system completes the boot up process with success.

Answer 9

I figured out in my case that the grub root was root=/dev/vgname/root.

So the test in /usr/share/initramfs-tools/scripts/local-top/lvm2:

  # Make sure that we have a d-m path
  dev="${dev#/dev/mapper/}"          
  if [ "$dev" = "$1" ]; then         
    return 1                         
  fi      

was always false, and the root volume never activated.

Updating /etc/fstab from

/dev/vgname/root        /

to

/dev/mapper/vgname-root   /

and doing:

update-grub
grub-install /dev/sda

solved my problem.