Gentoo installation with raid, lvm, luks and systemd

It's that time again. I have once again been contemplating switching away from Gentoo to Debian or Arch. But I just can't do it. The reason this issue came up again is that I once again tried to do a quick update this time moving to systemd. But I just put too little thought into it an messed my system up quite a bit. After some consideration I decided however that I won't move away from Gentoo. I have been using it for years, got so used to it and still have lot to learn so I'll stick with it. Another reason that is even better became crystal clear to me after this video Gentoo Linux, or Why in the World You Should Compile Everything. A source based distribution is not about speed but all about control over what gets onto and happens on your system. The idea of having all the parts out there and having them magically combine into a complete system just appeals to me. It is very much like I think of my own programming projects. You lay out all the pieces and then see them sort of self assemble anytime you want. This blog post will document my new fresh Gentoo install from within my currently running system.

Planned Configuration

  • GPT partition layout for UEFI boot
  • RAID1 mirror with 2 identical disks. I will start with just one disk and add the other one later.
  • LVM on top to easily manage how the space is used
  • LUKS encrypted LVM volumes for user home directories and swap
  • bare system with custom kernel and systemd working

After all that is working I will probably go with my trusty XFCE desktop again. So I will only install a minimal system to start with.


I start this installation from within my current gentoo system. But you can of course use pretty much any Linux LiveCD or USB stick. If you do so I suggest to boot this CD or USB stick in UEFI mode and you won't need the awkward reboot in the middle. I recommend SystemRescueCD.

Since I am going to have encrypted volumes it is a good idea to initialize the drive with random data. Like this:

dd if=/dev/urandom bs=4096 | pv > /dev/sdb
65.6MiB 0:00:04 [16.3MiB/s] [>                                ]  0% ETA 32:19:40

This will take some time like the Pipe Viewer pv shows us. I will just fill the first 200 GB or so and the rest later. Why random data? First of all the drive I am using is used so I do not want any old stuff from whomever lurking around there. Secondly if we were to just zero the drive it would then later be much easier so see how much data is on the encrypted home volumes. Since we are using encryption we might as well do it right. Filled with random data first the volumes will just look like big binary blobs if you do not have the key. Regardless of whether just one byte is used or the whole volume.



We will use UEFI with a GPT partition table. To create it we use parted.

parted /dev/sdb
(parted) mklabel GPT  
(parted) mkpart ESI fat32 0% 500m
(parted) set 1 boot on   
(parted) print                                                            
Model: ATA WDC WD2002FAEX-0 (scsi)
Disk /dev/sdb: 2000GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End    Size   File system  Name  Flags
 1      1049kB  500MB  499MB               ESI   boot

This is the UEFI boot partition, where the UEFI capable BIOS will look for bootable files (in our case GRUB2). The "1049kB" confused me for a short while but is effectively sector 2048. So there is some free space in front of it should you need to squeeze GRUB in there for non-UEFI booting. We continue with the swap partition.


(parted) mkpart raid1-swap 500m 35g
(parted) set 2 raid on

I used 35 GB as an endpoint because this way I have a swap size of 32 GiB and a tiny bit extra. This is the maximum RAM size I expect this system to ever have (8 GiB at the moment). Why swap at all? Strictly speaking your system would most likely work fine without it since the 8 GiB in my case are plenty for most things. But there is no harm in having swap and I plan to use hibernation as well. You can choose the partition name "raid1-swap" freely. I toggled the "raid" flag as well to indicate to the kernel automounter that this will be a RAID partition. Now comes the root partition.


(parted) mkpart raid1-root 35g 36g
(parted) set 3 raid on

1 GB for root should be plenty since I will have /usr, /home, /var and if necessary others on LVM. Now the partition for the LVM.


(parted) mkpart raid1-lvm 36g 200g
(parted) set 4 raid on

Simple enough. I used just 200 GB of this 2 TB disk because the rest of it has not yet been overwritten with random data.

A final look:

(parted) p                                                                
Model: ATA WDC WD2002FAEX-0 (scsi)
Disk /dev/sdb: 2000GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name        Flags
 1      1049kB  500MB   499MB                ESI         boot
 2      500MB   35.0GB  34.5GB               raid1-swap  raid
 3      35.0GB  36.0GB  999MB                raid1-root  raid
 4      36.0GB  200GB   164GB                raid1-lvm   raid


File System, RAID and LVM preparation


The UEFI system partition needs FAT32.

mkfs.vfat -F 32 /dev/sdb1


Now we set up the RAID superblocks for our swap, root and LVM partitions. You may note that I do not start with md0. I am configuring these from within my current Gentoo system which also happens to have a RAID. So I cannot start with md0. But that does not matter, they are not permanent names. If you run through this guide a second time or have the RAID partitions set up before for another reason you may get a note like "mdadm: Note: this array has metadata at the start". Make sure everything is ok and then you can override the old partitions. Also note that I start with just one drive and specify the second as "missing". I will add the other when my installation is complete and sync them.

mdadm --create /dev/md1 --level=mirror --raid-devices=2 /dev/sdb2 missing
mdadm --create /dev/md2 --level=mirror --raid-devices=2 /dev/sdb3 missing
mdadm --create /dev/md3 --level=mirror --raid-devices=2 /dev/sdb4 missing

Note: The arrays will be assembled in the initramfs so you can safely pick the default metadata version >= 1.2 even though the kernel itself does not understand it.

Check it:

cat /proc/mdstat
Personalities : [raid1] 
md3 : active raid1 sdb4[0]
      160025472 blocks super 1.2 [2/1] [U_]

md2 : active raid1 sdb3[0]
      975296 blocks super 1.2 [2/1] [U_]

md1 : active raid1 sdb2[0]
      33658752 blocks super 1.2 [2/1] [U_]

Our root file system is simple ext4.

mkfs.ext4 /dev/md2


First select the physical volumes for the volume groups. This is the complete md3.

pvcreate /dev/md3

Add it to a new volume group.

vgcreate vg /dev/md3

"vg" is just the name for the volume group. You can of course pick your own.

Now we have a place to set up some logical volumes. We will make some for /usr, /var and /home/user1.

lvcreate -L 20G -n usr vg
mkfs.ext4 /dev/vg/usr

lvcreate -L 20G -n var vg
lvcreate -L 40G -n home-user1 vg

/usr does not need to be encrypted because there is no actual user data in it. Just the applications which are open source anyway. I am not paranoid enough to encrypt which applications I installed. /var however usually contains mail and printer spool as well as database stuff.

Crypto Preparation

At this point we have three partitions that have to be encrypted:

1. swap (/dev/md1)
2. /var (/dev/vg/var)
3. /home/user1 (/dev/vg/home-user1)

Swap needs to be encrypted so that unencrypted user data does not get stored on the disk in case of swapping or hibernation. Pretty much the same goes for /var which holds database data as well as mail and printer spools. However if your system wakes up from hibernation you will not want an extra password but instead the password from a user such as user1 should work. The same goes for unlocking /var.

To achieve this is not that difficult because you can have more than one (8 to be exact) decryption keys per LUKS encrypted volume. So swap and /var get the root key (root password) as well as the regular users passwords (user1 and possibly others). The home partitions should of course only have one password.

This setup has however the one flaw that you will have to enter the password two or three times. Once or twice during boot for unlocking /var and swap. And once for logging in with your user. This can maybe be mitigated by using graphical overlays during the boot process such as plymouth which can cache the password temporarily.

Encrypting the Volumes

Now we initialize the partitions. I use the default key parameters. You will be asked for a password. For now swap and var will get my root password. The user user1 gets his respective password.

cryptsetup luksFormat /dev/md1
cryptsetup luksFormat /dev/vg/var
cryptsetup luksFormat /dev/vg/home-user1

The users should however also be able to open swap and /var with their user password. This way they can reboot or wake up from hibernation without root access. So we add user1's password to swap and /var.

cryptsetup luksAddKey /dev/md1
[enter root password and then enter user1 password]
cryptsetup luksAddKey /dev/vg/var
[enter root password and then enter user1 password]

Initializing File Systems

At this step we will open the encrypted volumes and initialize them with the respective file systems.

cryptsetup open /dev/md1 swap
mkswap /dev/mapper/swap

cryptsetup open /dev/vg/var var
mkfs.ext4 /dev/mapper/var

cryptsetup open /dev/vg/home-user1 home-user1
mkfs.ext4 /dev/mapper/home-user1

Of course you will need to enter the respective passwords when opening the partitions.

Gentoo Installation

We can now switch over to the regular Gentoo installation and start like this:

# activate swap
swapon /dev/mapper/swap
# mount root
mkdir /mnt/gentoo
mount /dev/md2 /mnt/gentoo
# mount usr
mkdir /mnt/gentoo/usr
mount /dev/vg/usr /mnt/gentoo/usr
# mount UEFI partition as boot
mkdir /mnt/gentoo/boot
mount /dev/sdb1 /mnt/gentoo/boot
# mounting var
mkdir /mnt/gentoo/var
mount /dev/mapper/var /mnt/gentoo/var

Continue the installation with downloading and unpacking the stage tarball as explained in the Gentoo handbook chapter 5 until you reach chapter 5.c.

Before we move on we quickly add a tmpfs for /tmp and /var/tmp. /tmp because we do not want to have any resources used during installation cropping up on the unencrypted root. And /var/tmp because it is used heavily by portage and there is no need to have it encrypting and decrypting all the time during installation.

# creating a tmpfs for /tmp
mount -t tmpfs tmpfs /mnt/gentoo/tmp
# and another one for /var/tmp
mount -t tmpfs tmpfs /mnt/gentoo/var/tmp

The default maximum size for a tmpfs is half your RAM. It will only be used as needed. Depending on the size of your RAM you may actually have to disable it and use the real /var/tmp for the really big package installations like LibreOffice an so forth.

Continue with chapter 5.c. In chapter 6.b I chose the profile "default/linux/amd64/13.0" for a minimal system. When you have reached paragraph about setting the timezone skip it and continue here. We will set the timezone with systemctl later on.


The locale will be set later on with systemctl but we will generate the desired locales here first. Edit /etc/locale.gen and activate the desired locales. I selected English and German.

en_US.UTF-8 UTF-8
de_DE.UTF-8 UTF-8

Then generate them with:



While in the chroot don't forget to set your root password:


RAID Configuration

While dracut will auto assemble RAID arrays in our setup it is good idea to have an mdadm.conf to describe the current and desired layout.

emerge -av mdadm

Then set up an /etc/mdadm.conf file like this:

DEVICE /dev/sd[a-e]*
ARRAY /dev/md1 metadata=1.2 UUID=fa009f73:b43fab9c:42b39f4a:91e28858
ARRAY /dev/md2 metadata=1.2 UUID=6cf87b94:25775da3:7fdeefeb:b66dd3d4
ARRAY /dev/md3 metadata=1.2 UUID=139f2034:4005ccf4:b0f0b842:54796b0f

You can get the UUIDs with:

mdadm --misc -D /dev/md1

Of course you have to adapt this for your setup. To make it easy in case you are copy and pasting from this post anyway you can use this:

echo "DEVICE /dev/sd[a-e]*" >> /etc/mdadm.conf
for i in 1 2 3; do echo "ARRAY /dev/md$i metadata=1.2 UUID=$(mdadm --misc -D /dev/md$i | egrep -o '([0-9a-f]{8}:?){4}')" >> /etc/mdadm.conf ; done

This configuration will be read and used by dracut later on.

LVM Configuration

First of all install the lvm tools.

emerge -av sys-fs/lvm2

To enable the general automatic activation of we enable listening to lvmetad in the global section of /etc/lvm/lvm.conf

use_lvmetad = 1

The systemd unit lvm2-lvmetad.service will also have to be activated later on for this to work.

You can also download my lvm.conf for comparison.

We only want our volume group vg to be activated by default, so add the following in the activation section of /etc/lvm/lvm.conf

volume_list = [ "vg" ]

And to speed up the scanning for LVM volume groups in the devices section with:

filter = [ "r|/dev/nbd.*|", "a|/dev/md.*|" ]

This assumes all your physical LVM volumes are on /dev/md (i.e. RAID) devices.


Here we define a proper fstab for our setup. I prefer using UUIDs here to avoid any confusion. You can easily find the UUIDs like this:

ls -l /dev/disk/by-uuid/
# or

For me this results in an fstab like this:

UUID=04C0-82C6                                  /boot           vfat            noauto,noatime  1 2
UUID=f202b721-03c7-4589-8898-1f4b7e882db2       /               ext4            noatime         0 1
UUID=2e554de3-cb35-470e-b542-446eeeeb24da       /usr            ext4            noatime         0 0
UUID=4c61fe1f-13ad-403a-b95f-3f7fa0596c4c       none            swap            sw              0 0
UUID=570592fc-fa46-4846-9d7f-418eb2f6dd95       /var            ext4            noatime         0 0
tmpfs                                           /var/tmp        tmpfs           nodev,nosuid    0 0

Remember to use the mapped (i.e.) opened UUIDs for LUKS encrypted volumes such as /var and swap. For example:

blkid | grep swap
/dev/mapper/swap: UUID="4c61fe1f-13ad-403a-b95f-3f7fa0596c4c" TYPE="swap"

I also tired to use /dev/mapper/var for var but that led to problems while trying to mount it during boot. It seems it is not a good idea to mix UUIDs and device paths. So I stick to UUIDs only.


Some of our encrypted partitions should be loaded at boot time. To enforce this we put them in the /etc/crypttab.

var     UUID=f531cc7f-a201-4bf1-80f2-e7a8621e7ff7

Emerging systemd

There are several things to consider for systemd. Check with the systemd wiki. I have copied the necessary parts for my setup below.

At the time of writing this step was recommended:

ln -sf /proc/self/mounts /etc/mtab

First of all we need systemd. Since we are using crypto we will need the cryptsetup useflag. Add it to your /etc/portage/make.conf. This is best done with euse from app-portage/gentoolkit.

emerge -av app-portage/gentoolkit
euse -E cryptsetup systemd gudev dbus

I added gudev and dbus because they will be required by NetworkManager anyway.

There is a circular dependency with dbus, so proceed like this:

USE="-systemd" emerge -av sys-apps/dbus
emerge -av sys-apps/systemd
emerge -av sys-apps/dbus

We also add the networkmanager, so that you can get your network up and running quickly.

emerge -av net-misc/networkmanager

Depending on your selected profile this may pull quite a bit of stuff.

Configuring systemd

To get our /var detected and activated we need to enable the lvm2-lvmetad.service. But since we are still in a chroot we cannot use systemctl directly. So we do it manually.

cd /etc/systemd/system
ln -s /usr/lib/systemd/system/lvm2-lvmetad.service

Configuring the Kernel

We are building our own kernel here. So you should especially know which device drivers to pick. If you are unsure also check with the handbook chapter 7. Make sure you build your kernel with UEFI options as well as RAID1 and LVM enabled.

emerge -av gentoo-sources
cd /usr/src/linux
make menuconfig

Then make sure to select:

Gentoo Linux > Support for init systems, system and service managers > [*] systemd
General setup > [*] Initial RAM filesystem and RAM disk (initramfs/initrd) support
Enable the block layer > Partition Types > [*]   EFI GUID Partition support
Processor type and features > [*] EFI runtime service support
Device Drivers > Generic Driver Options > () path to uevent helper
Device Drivers > Multiple devices driver support (RAID and LVM) > <*>   RAID support 
Device Drivers > Multiple devices driver support (RAID and LVM) > <*>   RAID-1 (mirroring) mode
Device Drivers > Multiple devices driver support (RAID and LVM) > <*>   Device mapper support > <*>     Crypt target support
Device Drivers > Multiple devices driver support (RAID and LVM) > <*>   Device mapper support > <*>     Mirror target
Device Drivers > Graphics support > Support for frame buffer devices > [*]   Enable firmware EDID
Device Drivers > Graphics support > Support for frame buffer devices > [*]   EFI-based Framebuffer Support
Device Drivers > Graphics support > Console display driver support > <*> Framebuffer Console support
Firmware Drivers > EFI (Extensible Firmware Interface)  Support > <*> EFI Variable Support via sysfs

You can also have a look at my .config file.

For the encryption I activated the following as well:

Cryptographic API > <*>   SHA256 digest algorithm (SSSE3/AVX/AVX2)
Cryptographic API > <*>   SHA512 digest algorithm (SSSE3/AVX/AVX2)
Cryptographic API > <*>   AES cipher algorithms (AES-NI)

If you forget the EFI based framebuffer it may later on appear as if your kernel hangs at boot because you won't see any further output.

Compile the kernel and copy it to /boot

make && make modules_install && make install


Since our /usr is an LVM volume and /swap which may be needed for resuming from hibernation is encrypted we need an initramfs. In the old days we would have compiled whatever tools we need for the initramfs statically and fiddled them together. This makes upgrades rather painful. As a solution we now have is dracut.

Dracut basically inspects you environment and creates an initramfs to ensure all the essentials are there to mount the rootfs. And since we have a separate /usr and are using systemd we also have to make sure that gets mounted properly.

At the time of writing dracut was still in ~amd64 so we emerge it like this:

echo "sys-kernel/dracut" >> /etc/portage/package.keywords
echo 'DRACUT_MODULES="crypt lvm mdraid"' >> /etc/portage/make.conf
euse -E device-mapper
emerge -av dracut

Now we will add some configuration options to /etc/dracut.conf. Since Dracut does not assemble RAID or LVM by default we will select just the volumes we need. Activating everything with '' would be slower. The first 3 items are our RAID md devices. You can get the UUIDs like this:

emerge -av sys-fs/mdadm
mdadm --misc -D /dev/md1 | grep UUID

They have to be in the mdadm format with ':' and not '-'. The fourth line just tells dracut to activate the vg/usr LVM volume.

kernel_cmdline+=" "  
kernel_cmdline+=" "  
kernel_cmdline+=" "  
kernel_cmdline+=" "  
kernel_cmdline+=" rd.luks.uuid=49130618-5eeb-4746-8e68-4237149eafe4 "
kernel_cmdline+=" resume=UUID=4c61fe1f-13ad-403a-b95f-3f7fa0596c4c "
add_dracutmodules+="crypt lvm mdraid resume"

The spaces at between " and content are important !

The next 2 lines are the encrypted swap partition and the unencrypted swap volume. Listing the encrypted partition here with rd.luks.uuid will instruct dracut to ask for the password. The kernel will then see the open/unencrypted version as the swap partition to use for resuming from hibernation.

You can get them like this:

blkid | egrep '(/md1|/swap)'

Then we add the needed modules with add_dracutmodules+="crypt lvm mdraid resume".

And finally we enable our local mdadm.conf in the last line.

Then we generate the initramfs with the following command:

dracut --force --hostonly --add-device=/dev/disk/by-uuid/49130618-5eeb-4746-8e68-4237149eafe4 '' 3.10.17-gentoo

Adapt to your kernel version of course. Without the explicit version number it may not work if you are just in a chroot booted from a different kernel version. The --add-device option points to the raid device with the swap partition on it. It is necessary because otherwise dracut might not pick it up and booting will fail with something like dracut-initqueue: Failed to issue method call: Unit systemd-cryptsetup@luks\x2d...service failed to load: No such file or directory.

Or if you don't like to type the kernel version:

dracut --force --hostonly --add-device=/dev/disk/by-uuid/49130618-5eeb-4746-8e68-4237149eafe4 '' $(ls -l /usr/src/linux | sed 's/.*linux-//')

Note: when your system is completely set up with keyboard layout and so forth you may want to regenerate the dracut initramfs once more. Otherwise it can happen that you have the wrong keyboard layout when you try to enter the passwords for /var and swap during boot.

Installing GRUB2

Add the desired platform to your make.conf and emerge grub2

echo 'GRUB_PLATFORMS="efi-64"' >> /etc/portage/make.conf
emerge -av sys-boot/grub:2
mkdir /boot/grub

Now uncomment and modify the default command line in /etc/default/grub:


Build the configuration:

grub2-mkconfig -o /boot/grub/grub.cfg

The output should contain something like this:

Found linux image: /boot/vmlinuz-3.10.17-gentoo
Found initrd image: /boot/initramfs-3.10.17-gentoo.img

I got some errors like "/usr/sbin/grub2-probe: warning: Couldn't find physical volume '(null)'", but that does not seem to be a problem.

Now comes the annoying part. I am currently working in a chroot started from a non-UEFI system. You cannot really install grub2 UEFI properly from a non UEFI system. So we have to use an UEFI capable Linux live disk to boot. I always have a SystemRescueCD USB stick lying around. So I will use that one. Just be sure to select UEFI boot. On my mainboard this means pressing F11 during boot and then selecting UEFI: USB .. .

You will of course have to manually put all the parts of your new system together and chroot. For me this means:

mount /dev/md2 /mnt/gentoo
mount /dev/vg/usr /mnt/gentoo/usr
mount /dev/sdb1 /mnt/gentoo/boot
cryptsetup open /dev/vg/var var
mount /dev/mapper/var /mnt/gentoo/var/
mount -t tmpfs none /mnt/gentoo/tmp
mount -t tmpfs none /mnt/gentoo/var/tmp
cp -L /etc/resolv.conf /mnt/gentoo/etc/
mount -t proc none /mnt/gentoo/proc
mount --rbind /sys /mnt/gentoo/sys
mount --rbind /dev /mnt/gentoo/dev
chroot /mnt/gentoo /bin/bash

If the live stick is a bit older you might need to use cryptsetup luksOpen /dev/vg/var var.

And in the chroot:

source /etc/profile
export PS1="(chroot) $PS1"

Then you can install grub2 like this:

grub2-install --efi-directory=/boot

This should not produce any errors, but instead just show the current boot order. In /boot you should find an EFI directory now and a new boot entry 'gentoo' has been added to the NVRAM of the UEFI BIOS.

The Reboot

During the installation you may have seen some complaints about an unset hostname or a missing locale. This is because they should be set through systemd. That is however only possible if the system was booted with systemd. So it is time to to reboot now. If you have followed the steps closely your system should boot properly. You will be asked the password for swap and /var during boot. Should something fail, reboot your old system or LiveCD and chroot into your new installation. Even though systemd will not work in the chroot you can use 'journalctl' to diagnose boot errors.

Note: I had some issues with the lvm2-lvmetad.service not starting correctly and therfore /var not being mounted. The system then hangs after "Reached target swap". This seems somehow related to creating the link for starting it manually (see "Configuring systemd" above). If that happens wait 30-60 seconds and you will be offered a root shell for maintenance. Login and do the following:

rm -rf /etc/systemd/system/
systemctl enable lvm2-lvmetad.service

Configuring systemd

Systemd is new territory for me so I summed up my most important settings in the following paragraphs.

I have watched quite a few videos and read some articles on systemd before this new install. There was some controversy, but I personnally really like it. I suggest to just give it a try.

Setting the Locale

localectl set-locale LANG=en_US.utf8
localectl set-keymap de-latin1-nodeadkeys

If you have skipped the locale section earlier you have to generate the locales here first. You can do this as usual by editing /etc/locale.gen and running locale-gen.

Setting the Hostname

hostnamectl set-hostname testhost

Replace 'testhost' with your desired hostname.

Setting the Timezone

timedatectl set-timezone Europe/Berlin

Getting a Network Connection

systemctl enable NetworkManager
systemctl start NetworkManager

This will symlink the NetworkManager.service file from /usr/lib/systemd/system to /etc/systemd/system. This also means you can see all available service files in /usr/lib/systemd/system and if you like put your own in /etc/systemd/system.

You can check what the NetworkManager has logged like this:

journalctl -u NetworkManager

In my case it complained about the missing NetworkManager.conf but got my wired connection working with dhcp anyway.

Regular Users

Until now we have done everything as root and do of course need to add regular users for everyday work. Since each of them has an own LUKS encrypted LVM volume and should also be able to unlock swap and /var some additional steps are required.

Automounting /home

The base system is now up and running. We do however still need to add some functionality so that our user home directories will be mounted on login.

For this we will use pam_mount.

emerge -av pam_mount

And in /etc/pam.d/system-auth add the following in the top block:

auth            optional

And this in the bottom block:

session         optional

Adding a User

Adding a user requires some extra work. The following assumes the user name 'user1'.

Add a Regular User

useradd -m user1
passwd user1

Add Encrypted Home Volume

Use the same password you used with passwd here.

lvcreate -L 10G -n home-user1 vg
cryptsetup luksFormat /dev/vg/home-user1
cryptsetup open /dev/vg/home-user1 home-user1
mkfs.ext4 /dev/mapper/home-user1
mount /dev/mapper/home-user1 /home/user1
chmod 0700 /home/user1
chown user1:user1 /home/user1
umount /home/user1
cryptsetup luksClose /dev/mapper/home-user1

Automounting Home Volume

Now edit /etc/security/pam_mount.conf.xml and add:

<volume user="user1" mountpoint="/home/user1" path="/dev/vg/home-user1" fstype="crypt" options="noatime" />

Put it before the last '</pam_mount>'.

Now logout or switch to another tty and log in as 'user1'. It should automatically mount the encrypted lvm volume as your home. Since your home is in the root of an ext4 volume so you should see the default 'lost+found' directory in there.

If you want a newly added userto ba able to unlock /var and swap during boot you need to add that password to the /var and swap volumes as shown in the paragraph "Encrypting the Volumes" above.

Changing the Password

Changing Keys

If you want to change a users password you also need to change it on to the swap and /var LUKS volumes.

cryptsetup luksChangeKey /dev/md1
[enter old and new password]
cryptsetup luksChangeKey /dev/vg/var
[enter old and new password]

Changing Password

The user can do this himself with the regular:


Removing a User

Removing a user also requires some steps. Be sure not to forget to remove the key from swap and /var*. Also the user needs to be logged out. The following steps assume the user name 'user1'.

Removing Keys

/dev/md1 ist the swap partition.

cryptsetup luksRemoveKey /dev/md1
[enter user1 password]
cryptsetup luksRemoveKey /dev/vg/var
[enter user1 password]

If you do not have the password you can remove a key with luksKillSlot. You do however need to know which keyslot it is. If you are expecting this to happen you should keep track of the assigned keyslots when creating the users.

Removing LVM volume and /home

lvremove vg/home-user1

Removing User Account

userdel -r user1


Hibernation (suspend to disk) should work out of the box with this setup. You can try it with:

systemctl hibernate

On reboot you will then be asked the password (i.e. one of the passwords) for the swap partition by dracut and the system should resume from hibernation.


Remember the beginning of this post when I filled just part of the disk with random data to initialize it because it took so long? Well I have filled the rest with random data and now it is time to use the rest of the space. This means:

1. increasing the size of the RAID1 array 
2. increasing the size of the pysical volume belonging to the volume group
3. add some space to a users home volume
4. resize the LUKS volume on it
5. resize the contained ext4 volume

Since /usr is on the LVM on this RAID you will need to resize it by booting from another system. For example a LiveCD.

Stopping LUKS, LVM and RAID

If you decide to resize directly after you have made all the steps above or are using a LiveCD which activates most of the volumes automatically you will need to close LUKS volumes, deactivate volume groups and stop the RAID array.

cryptsetup close /dev/mapper/var
# check which drives (i.e. RAID array) belong to our volume group
# deactivate volume group
vgchange -a n vg
# stop raid
mdadm --manage -S /dev/md3

Changing the RAID Partitions

If your RAID already has 2 drives you will need to do the following steps for both drives.

parted /dev/sdb
(parted) print
Model: ATA WDC WD2002FAEX-0 (scsi)
Disk /dev/sdb: 2000GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name     Flags
 1      1049kB  500MB   499MB   fat32        ESI      boot
 2      500MB   35.0GB  34.5GB               primary  raid
 3      35.0GB  50.0GB  15.0GB               primary  raid
 4      50.0GB  200GB   150GB                primary  raid
(parted) rm 4
(parted) mkpart raid1-lvm 50g 250g
(parted) set 4 raid on
(parted) quit

Re-assembling and Resizing the RAID Array

mdadm --assemble /dev/md3 /dev/sdb4
mdadm --grow /dev/md3 -z max --assume-clean

The grow step is important, because mdadm will not automatically use the new size of the underlying partition on reassembly.

Resizing the LVM volume

Resize the physical LVM volume to the partition size.

pvresize /dev/md3

Enlarging a home Volume

Adding 10GB to user1's home volume

lvresize -L +10G vg/home-user1

And then activate the volume

lvchange -a y vg/home-user1

Enlarging a LUKS Volume

cryptsetup resize /dev/vg/home-user1

Enlarging the ext4 File System

Open the encrypted volume:

cryptsetup open /dev/vg/home-user1 home-user1

Check it:

e2fsck -f /dev/mapper/home-user1

And resize the file system:

resize2fs /dev/mapper/home-user1

You see there are quite a few steps necessary but all the tools are there and well established.

A Quick Performance Test

After the whole installation I thought to myself: "So many layers? This must all be slow."

emerge -av bonnie++

On the bare RAID (i.e root):

bonnie++ -d /root -u root -s 10g -r 0

For this test I had an installation with a significantly bigger root. If you have followed the steps above exactly "10g" will not be possible.

I do not recommend to use '-u root' on a system with important data on it. My test system is still empty at this point. So it is ok.

Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
testhost       10G  1062  98 142701   7 65568   4  5765  94 207092   7 711.3   9

With /home/user1 opened and mounted:

bonnie++ -d /home/user1 -u root -s 10g -r 0

And the result:

Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
testhost       10G  1000  97 128191   7 67962   5  5517  91 176171   6 692.1   8

Thats around 10% slower than without encryption. So it barely matters. 10% are well worth the added security I think.

List of used Software Versions

  • gentoo-sources-3.10.17
  • sys-fs/lvm2-2.02.103
  • sys-fs/cryptsetup-1.6.2
  • sys-kernel/dracut-034-r4
  • sys-apps/systemd-208-r2
  • sys-boot/grub-2.00_p5107-r2
  • sys-auth/pam_mount-2.14

All on amd64 architecture.

Conclusion / Known Issues

I guess this post is long enough and I better stop now before I squeeze in more stuff. There is still a lot to do. The next step will probably be to mess around a bit more with systemd, go through the logs and check everything.

At this time there are the following issues:

  • the dracut (swap) and (var) password inputs get overdrawn which is somewhat confusing
  • some logging for example from NetworkManager gets written to the console
  • pam_mount complains about something when a user logs in

These do however not seem very severe issues. I will probably address them in my next post.