Running Systems

Hi. As some know now, I have an article about encrypting and protecting Linux – while using TPM to prevent the need to enter the encryption passphrase every time the system boots. This is one of the most popular article in this blog, and for a very good reason – people want their computers (especially portable devices) to be secure and well protected against data theft (let’s assume that by trivial means. I tend to believe that if the NSA or any country-class intelligence service is targeting you, your data will be compromised, so don’t get targeted by countries please), but they want to be the least-bothered with entering and re-entering security credentials along the way. Eating the cake while leaving it whole, if you would like.

Using TPM (which is welded into the motherboard of the computer) to keep the encryption passphrase is a great solution – you get to have an encrypted disk, but don’t have to reuse the passphrase on every reboot. Also – you can rest assured that when/if your hard drive is pulled out of the computer, there is nothing to do with it, because the key is kept on the motherboard.

However – this is not enough, and my article does not cover the scenario where a malicious user pulls out our hard drive and puts a different hard drive instead of it. The current process is of keeping the passphrase directly in the TPM2 chip, not as an SSL challenge/response mechanism, but just as a storage for the key. As such – any user with physical access to the machine from within the OS level can obtain it – even by replacing the disk and pulling the key out, or even worse – by patching the boot sequence (which partially resided on /boot, which is not encrypted) and by replacing initramfs/initrd or by modifying GRUB boot flow, and just asking the TPM nicely for the key. So – our system is not that secure. At least not as much as we aim at. I have mentioned that in the disk encryption article, but yet to have provided a solution.

The reason I have not provided a solution has to do with how Canonical (Ubuntu) take a look at the chain of trust, and the different way I look at it, and to bring this issue up front, I am writing this article, which is not really a technical, but more of a discussion of options. If some of my readers respond and enlighten me – I would be grateful for that, as I am truly looking for a feasible solution to my Ubuntu – this or the next one.

A chain of trust is a workflow where every step is trusted by the previous step, on one side, and can trust the next step. My desirable chain of trust in this case would look like this:

• The BIOS is locked and cannot be modified without credentials
• The BIOS allows booting only from a single disk device, and only a specific boot file
• Using SecureBoot, only the bootable binary (GRUB2 in our case) – a signed and un-tempered file, can be called
• GRUB2 EFI executable has its configuration embedded, so that you cannot modify it, add additional parameters, or stop its boot process mid-way.
• GRUB2 is password protected, and allows only a single entry (default) without additional parameters, without authentication.
• GRUB2 loader trusts the kernel it boots to be one pre-signed by some trusted authority.
• GRUB2 loader trusts initramfs/initrd as it is signed in a manner GRUB2 understands and respects, aka – it is unmodified
• Initramfs/initrd is calling the TPM2 to obtain the key, and by doing so, does two things: Opens the disk encryption by using the key, and trusting the BIOS to be un-tempered with, because resetting the BIOS password would reset the TPM2 contents as well (this is good for us).
• The system continues boot correctly.

You can notice that every step of the boot process has the current step trusted by the previous one, and has a trust mechanism towards the next step. This can work by using self-signed certificate to sign GRUB2 EFI executable, and saving this certificate (and its verification!) in the SecureBoot section of the BIOS, overriding any ‘common’ certificates the BIOS knows of. Then – using grub-standalone instead of the regular GRUB2, which means that GRUB configuration and modules are embedded into a single, signed, executable – we prevent injection of GRUB modules (drivers) or change of parameters. Also – GRUB is password protected against non-default boot. Following that, GRUB trusts the kernel, which is signed, and initramfs/initrd, which is also signed (GRUB used to trust GPG signed files, however, and this is the source of this discussion, Canonical changed some things there, and in my opinion – not for the better). And from here and onwards – the boot process works fine.

This chain of trust seems reasonable. It will prevent boot sequence intervention, and will make sure that the unencrypted part of the partition gets files replaced. You cannot load any different GRUB (even if replaced) due to BIOS trust issues, and you cannot change the configuration, kernel of initramfs/initrd, and therefore – you are bounded to this predefined flow.

This does not follow the ‘trusted computing’ flow. Trusted computing is a mechanism that is supposed to prevent the kernel (and the hardware assisting the process) loading unsigned kernel drivers, and it should be enforced by the TPM/SecureBoot as well. This is nice and pretty, but the vulnerable areas in the trusted computing areas are with the loaders. You might try to inject kernel drivers or malicious code when running under privileged context – when GRUB gets called in.

To mitigate this, Ubuntu enforced (starting with grub version 2.04-1ubuntu26.2) using SHIM, which is a pre-loader mechanism. The chain of trust works slightly differently here:

• BIOS trusts SHIM, using Microsoft certificate
• SHIM is signed using Microsoft certificate trusts a different set of certificates – self-signed or Canonical certificates. SHIM is pre-GRUB loader.
• SHIM trusts GRUB EFI binary, which is signed by Canonical
• GRUB trusts the kernel using Canonical certificate (all stock kernels are provided pre-signed by Canonical), or custom kernel (and modules) by self-signed certificate, but it needs to be used for every kernel used, and every module used by this kernel.
• GRUB trusts initramfs/initrd by whatever means (no trust is checked by default there)

The problem with this flow is that it has external sources in the chain of trust, where different components might get replaced or patched. The problems, as I see them (and I would like to get a different opinion about it, so please feel free to share) are:

• SHIM can be replaced with another EFI binary trusted by Microsoft (like MS Windows loader, or some other tool used by whatever 3rd party I do not know). This will result in skipping the entire trust chain, into a different flow, of which we have no control.
• SHIM might be able to get compiled with a self-signed certificate (which then be entered into the BIOS SecureBoot as a trusted certificate), however – this process will prevent easy/automated deployment of SHIM, or will require re-compiling it on every update.
• Using grub-standalone (reviewed above) will require signing it with Canonical certificate (we cannot do this), or with our self-signed. If we use self-signed, we will remove Canonical certificate from SHIM, but then we will have to use our certificate to re-sign the kernel, or we add our self-signed certificate to SHIM, but then, our locked-down GRUB can be replaced with a general GRUB2, signed by Canonical (the default) and worked around the lockdown easily.
• Now GRUB2 (also grub-standalone, as this is part of the change introduced in version 2.04-1ubuntu26.2) will not trust GPG signature for the kernel (allowing us to avoid re-signing the kernel using SSL, but adding a GPG signature, thus – easier automation of processes and trust chains), but only the SSL, which has to be known to SHIM.
• By default – initramfs signature is not checked. At this stage with all the chain breaks along the way – it doesn’t matter much now. An attacker can just hook-in anywhere before, and pull our keys out already, using off-the-shelf security loaders of GRUB EFI binaries.

I was able to recomplie the latest GRUB packages for Ubuntu omitting these three patches, without code modifications, to regain the original security flow I presented above (pending tests. Not done yet):

• 0096-linuxefi-fail-kernel-validation-without-shim-protoco.patch
• 0099-chainloader-Avoid-a-double-free-when-validation-fail.patch
• ubuntu-linuxefi-arm64.patch

Is it to be trusted? I do not know. Is it safe “enough”? Maybe. Would I have to continue maintaining a non-official GRUB package? Maybe. How do I maintain it for Ubuntu 22.04? It seems Canonical has their mind set on their flow, which allows for trusted computing – I cannot ignore that – but fails, to my understanding, to provide “good enough” physical access protection when some of the disk is not encrypted (to be exact: /boot and /boot/efi).

You can take a look at the discussion in this Ubuntu bug thread. You can also get the workflow of using SecureBoot and SHIM in Ubuntu in their official wiki here. I did not discuss MokManager, as a tool used to add certificates to SecureBoot or SHIM since I need to dig deeper into this, but to my belief – it only eases management for some of the tasks which might be set easier from within a working OS.

I would love to hear where my analysing of Canonical workflow is wrong, and where their offered security works better than my (previous) workflow – in regards to physical access to data protection.

I will also link this article to the encrypted disk with TPM2 article, in hope it will draw more interaction with this post.

Thanks for reading this far.

An update: I am adding a link to a post where I share my thoughts about the additional steps required to protect a Linux system against physical access to data. I add it at the top of this article, because I want some insight from you – the people who might read this post. It deals with the chain of trust beyond the scope of this article – the entire boot process. Please proceed reading to handle only the technical aspects of obtaining your protected system’s encryption keys using TPM2, and remember that this is not enough to protect a system entirely against this attack vector. You key can be obtained if the remaining of the tasks, which are out of the scope of this article, are not followed. End of update

Following the conversation around a past post about Ubuntu 18.04 and TPM2 encrypted system disk (can be found here), I have added a post about using a more modern Ubuntu (20.04) with TPM2. This post includes all the updates (many thanks to the commenter kelderek for setting up a very well working setup!) and has just been tested on a virtual Ubuntu with (virtual) TPM2 device. A note about how to get this setup – sometime in the future. Please be advised that this process works with LVM block device encryption – LUKS, and not with other methods, such as BTRFS or ZFS. If you’re using either – this is not the guide for you, although you might be able to create a working flow derived from this guide (and if you do – please do post a link!)

A very important note, before we begin – This is not enough! To fully prevent your data from being stolen by a bad person with a screwdriver, you will need to add some additional measures, like locking up your BIOS with a password (resetting the BIOS password resets TPM2 data, which is good) ; Signing and using your own embedded GRUB2 (which is not covered here, and maybe one day will be), with your own certificate ; Protect GRUB2 from booth changed using password ; Lock down boot devices in the BIOS (so the evil hacker cannot boot from a USB or CDROM without unlocking the BIOS first) ; Lock the hard drive (or SSD, or NVME) with a security to the BIOS (possible on some models of laptops and hard drives). All these steps are required to complete the security process in order to make the life of a cracker/hacker harder to the point of impossible, when attempting to obtain your data. Without them, the effort is great, but not impossible.
Also note that all the above described actions might lock you out of your own system in case of any hardware failure or forgotten password, so make sure you have a (secure!) backup of your data, or else…

I should also add about the above mentioned GRUB2 lockdown (using grub-standalone) – Ubuntu has changed starting with grub2 versions later than 2.04-1ubuntu26 (both on Ubuntu 18.04 and 20.04. I do not know how it is on 22.04, but I will probably soon find out) their internal workflow, and they are basing their kernel verification on SHIM certificate, and not on GPG signature – which they find imperative (they provide signed kernels), but at this time, I find it relaxed compared to the method I am working on (meaning – working on providing a guide) because it allows for other Microsoft-signed loaders, where I would have none other than self-signed files. A guide would be provided soon, and maybe even a patched GRUB2 package, to undo their change (which has a lot of advancements, but in my small real-life usecase – mostly disadvantages).

Now – to the process. The idea is this: We add a new key to the cryptsetup – a long one, and this key is stored in TPM2. We add scripts which pull this key out of TPM2 store whenever the system boots. Thanks to some additional comments by Kelderek, we also add some failback, in case of an incorrect key, to allow up to recover and boot using manual key.

I will not wrap all my actions with ‘sudo’ command. All commands need to be called as root, so you can prefix this guide with ‘sudo -s’ or ‘sudo bash’ if you like. I can also assume you ‘cd’ to the root’s directory to protect the root.key file while it is still there. So – ‘sudo’ yourselves, and let’s begin.

Install required packages:

apt install tpm2-tools

Define TPM2 memory space to hold the key:

tpm2_nvdefine -s 64 0x1500016
# This command will define a 64 byte memory space in TPM2, at the above mentioned address

Create a random 64 byte key file:

cat /dev/urandom | tr -dc ‘a-zA-Z0-9’ | head -c 64 > root.key

Save the contents of the key file to TPM2:

tpm2_nvwrite -i root.key 0x1500016

Compare the contents of the TPM and the file, to verify that they are exactly the same:

echo root.key file contents: `cat root.key`
echo The value stored in TPM: `tpm2_nvread 0x1500016`
tpm2_nvread 0x1500016 2> /dev/null | diff root.key – && echo The root.key file matches what is stored in the TPM. It is ok to proceed! || echo The root.key file does not match what is stored in the TPM. Do NOT proceed until the two match!

Identify the encrypted device, and add the key to the LUKS:

lsblk # Identify the owner of 'crypt'. We will assume it is /dev/sda3 in our example
cryptsetup luksAddKey /dev/sda3 root.key

Create a script to pull the key from TPM2:

cat << EOF > /usr/local/sbin/tpm2-getkey
#!/bin/sh
if [ -f ".tpm2-getkey.tmp" ]; then
# tmp file exists, meaning we tried the TPM this boot, but it didn’t work for the drive and this must be the second
# or later pass for the drive. Either the TPM is failed/missing, or has the wrong key stored in it.
/lib/cryptsetup/askpass "Automatic disk unlock via TPM failed for  () Enter passphrase: "
exit
fi
# No tmp, so it is the first time trying the script. Create a tmp file and try the TPM
touch .tpm2-getkey.tmp
 
tpm2_nvread 0x1500016
EOF

This script will be embedded into the future initramfs (or initrd), and will pull the key from TPM2. Now, we need to set its permissions and ownerships:

chown root: /usr/local/sbin/tpm2-getkey
chmod 750 /usr/local/sbin/tpm2-getkey

Create a hook script to initramfs:

cat << EOF > /etc/initramfs-tools/hooks/tpm2-decryptkey
#!/bin/sh
PREREQ=""
prereqs()
{
echo ""
}
case \$1 in
prereqs)
prereqs
exit 0
;;
esac
. /usr/share/initramfs-tools/hook-functions
copy_exec `which tpm2_nvread`
copy_exec /usr/lib/x86_64-linux-gnu/libtss2-tcti-device.so.0.0.0
copy_exec /usr/lib/x86_64-linux-gnu/libtss2-tcti-device.so.0
copy_exec /lib/cryptsetup/askpass
exit 0
EOF

Set file permissions:

chown root: /etc/initramfs-tools/hooks/tpm2-decryptkey
chmod 755 /etc/initramfs-tools/hooks/tpm2-decryptkey

Edit crypttab:

The file /etc/crypttab needs to have an entry added at the end of the line containing the boot volume. For example (taken from the previous post’s comments):

sda3_crypt UUID=d4a5a9a4-a2da-4c2e-a24c-1c1f764a66d2 none luks,discard

should become:

sda3_crypt UUID=d4a5a9a4-a2da-4c2e-a24c-1c1f764a66d2 none luks,discard,keyscript=/usr/local/sbin/tpm2-getkey

First – backup your existing crypttab file:

cp /etc/crypttab /etc/crypttab.backup

If you have a single line in the file (only the boot volume is encrypted), you can use the following command:

sed -i 's%$%,keyscript=/usr/local/sbin/tpm2-getkey%' /etc/crypttab

Otherwise, you will have to append it manually.

Backup the original initrd and create a new one

cp /boot/initrd.img-$(uname -r) /boot/initrd.img-$(uname -r).orig
mkinitramfs -o /boot/initrd.img-$(uname -r) $(uname -r)

Reboot.

If all works well – you can now delete the file root.key holding a clear text key. You will still have your original manually defined decryption key working, and the initrd will fail-over to handle manual askpass if the automatic TPM2 fails to work, so you should be rather safe there, as well as update-proof.

Notes:

• The procedure might work on older Ubuntu systems, although it was not tested. A comment in the previous post suggested that the addition of the source
  deb http://cz.archive.ubuntu.com/ubuntu focal main universe
  to your Ubuntu 18.04, the same procedure should work. I did not test it yet.
• The security of the system is not complete until you handle the other tasks mentioned at the beginning of the text.
• This procedure does not cover TPM1.2 systems, and will not work without modification on them. I might document such process in the future.

Good luck, and post here if it works for you, or if you have any suggestions!